2024-04-24T23:04:10.509 INFO:root:teuthology version: 0.0.1.dev269+g9481b1d 2024-04-24T23:04:10.509 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672187 --name teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi --verbose --description fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/ffsb}} --owner scheduled_teuthology@teuthology -- /home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672187/orig.config.yaml 2024-04-24T23:04:10.677 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-24T23:04:10.771 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672187 branch: squid description: fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/ffsb}} email: ceph-qa@ceph.com first_in_suite: false job_id: '7672187' kernel: client: branch: testing kdb: 1 sha1: distro ktype: upstream last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules name: teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd max_mds: 5 session_timeout: 300 conf: client: client mount timeout: 600 client_shutdown_timeout: 120 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds oft prefetch dirfrags: true mds op complaint time: 180 mds verify scatter: true mds_dir_prefetch: true mds_sleep_rank_change: 5000000.0 mon client directed command retry: 5 ms inject delay max: 1 ms inject delay probability: 0.005 ms inject delay type: client mds osd ms inject socket failures: 2500 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 filestore flush min: 0 mon client directed command retry: 5 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 ms inject delay max: 1 ms inject delay probability: 0.005 ms inject delay type: client mds ms inject socket failures: 2500 osd blocked scrub grace period: 3600 osd failsafe full ratio: 0.95 osd max markdown count: 1000 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - MON_DOWN - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED - Replacing daemon mds - SLOW_OPS - slow request sha1: d2c45d977142e024a59b68a8f1dce8f0b9840551 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs check-counter: counters: mds: - mds.exported - mds.imported install: ceph: flavor: default sha1: d2c45d977142e024a59b68a8f1dce8f0b9840551 kclient: syntax: v1 selinux: allowlist: - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: squid sha1: d2c45d977142e024a59b68a8f1dce8f0b9840551 owner: scheduled_teuthology@teuthology priority: 100 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mgr.y - mds.a - mds.c - mds.e - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - - mon.b - mon.c - mgr.x - mds.b - mds.d - mds.f - osd.4 - osd.5 - osd.6 - osd.7 seed: 8528 sha1: d2c45d977142e024a59b68a8f1dce8f0b9840551 sleep_before_teardown: 0 subset: 122/512 suite: fs suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_d2c45d977142e024a59b68a8f1dce8f0b9840551/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: d2c45d977142e024a59b68a8f1dce8f0b9840551 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi006.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGBKLf8afP4hWXBS+othavKra26AYnE5js76yYD3eC1YHMuoyt7cFszoUSA4h7yqlHM6zZArkdCBujyHtkcJk58= smithi190.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBXNxoUEc6Gg1c7NEunVZBhXYLxME0phfSwBV3OmsDx9E5JQRWCdibkjfmVsoa2HLbGn80qNSwiVu0SWaYaV3Zo= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: create_rbd_pool: false - sequential: - mgrmodules - kclient: null - mds_thrash: null - workunit: clients: all: - suites/ffsb.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: 9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17 timestamp: 2024-04-24_21:24:02 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-24T23:04:10.771 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_d2c45d977142e024a59b68a8f1dce8f0b9840551/qa; will attempt to use it 2024-04-24T23:04:10.772 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_d2c45d977142e024a59b68a8f1dce8f0b9840551/qa/tasks 2024-04-24T23:04:10.772 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-24T23:04:10.969 INFO:teuthology.task.internal:Checking packages... 2024-04-24T23:04:10.993 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'd2c45d977142e024a59b68a8f1dce8f0b9840551' 2024-04-24T23:04:10.993 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-24T23:04:10.993 INFO:teuthology.packaging:ref: None 2024-04-24T23:04:10.994 INFO:teuthology.packaging:tag: None 2024-04-24T23:04:10.994 INFO:teuthology.packaging:branch: squid 2024-04-24T23:04:10.994 INFO:teuthology.packaging:sha1: d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-24T23:04:10.994 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-04-24T23:04:11.322 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2445.gd2c45d97 2024-04-24T23:04:11.324 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-24T23:04:11.333 INFO:teuthology.task.internal:no buildpackages task found 2024-04-24T23:04:11.333 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-24T23:04:11.366 INFO:teuthology.task.internal:Saving configuration 2024-04-24T23:04:11.388 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-24T23:04:11.402 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-24T23:04:11.426 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi006.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672187', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-24 22:59:08.288705', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGBKLf8afP4hWXBS+othavKra26AYnE5js76yYD3eC1YHMuoyt7cFszoUSA4h7yqlHM6zZArkdCBujyHtkcJk58='} 2024-04-24T23:04:11.447 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi190.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672187', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-24 22:59:08.290663', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBXNxoUEc6Gg1c7NEunVZBhXYLxME0phfSwBV3OmsDx9E5JQRWCdibkjfmVsoa2HLbGn80qNSwiVu0SWaYaV3Zo='} 2024-04-24T23:04:11.447 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-24T23:04:11.549 INFO:teuthology.task.internal:roles: ubuntu@smithi006.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2024-04-24T23:04:11.550 INFO:teuthology.task.internal:roles: ubuntu@smithi190.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2024-04-24T23:04:11.550 INFO:teuthology.run_tasks:Running task console_log... 2024-04-24T23:04:11.644 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f659c04a0d0>, signals=[15]) 2024-04-24T23:04:11.644 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-24T23:04:11.652 INFO:teuthology.task.internal:Opening connections... 2024-04-24T23:04:11.652 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi006.front.sepia.ceph.com 2024-04-24T23:04:11.654 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi006.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-24T23:04:11.736 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi190.front.sepia.ceph.com 2024-04-24T23:04:11.738 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi190.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-24T23:04:11.821 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-24T23:04:11.829 DEBUG:teuthology.orchestra.run.smithi006:> uname -m 2024-04-24T23:04:11.849 INFO:teuthology.orchestra.run.smithi006.stdout:x86_64 2024-04-24T23:04:11.849 DEBUG:teuthology.orchestra.run.smithi006:> cat /etc/os-release 2024-04-24T23:04:11.907 INFO:teuthology.orchestra.run.smithi006.stdout:NAME="CentOS Stream" 2024-04-24T23:04:11.907 INFO:teuthology.orchestra.run.smithi006.stdout:VERSION="9" 2024-04-24T23:04:11.907 INFO:teuthology.orchestra.run.smithi006.stdout:ID="centos" 2024-04-24T23:04:11.907 INFO:teuthology.orchestra.run.smithi006.stdout:ID_LIKE="rhel fedora" 2024-04-24T23:04:11.907 INFO:teuthology.orchestra.run.smithi006.stdout:VERSION_ID="9" 2024-04-24T23:04:11.907 INFO:teuthology.orchestra.run.smithi006.stdout:PLATFORM_ID="platform:el9" 2024-04-24T23:04:11.907 INFO:teuthology.orchestra.run.smithi006.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-24T23:04:11.907 INFO:teuthology.orchestra.run.smithi006.stdout:ANSI_COLOR="0;31" 2024-04-24T23:04:11.908 INFO:teuthology.orchestra.run.smithi006.stdout:LOGO="fedora-logo-icon" 2024-04-24T23:04:11.908 INFO:teuthology.orchestra.run.smithi006.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-24T23:04:11.908 INFO:teuthology.orchestra.run.smithi006.stdout:HOME_URL="https://centos.org/" 2024-04-24T23:04:11.908 INFO:teuthology.orchestra.run.smithi006.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-24T23:04:11.908 INFO:teuthology.orchestra.run.smithi006.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-24T23:04:11.908 INFO:teuthology.orchestra.run.smithi006.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-24T23:04:11.908 INFO:teuthology.lock.ops:Updating smithi006.front.sepia.ceph.com on lock server 2024-04-24T23:04:11.932 DEBUG:teuthology.orchestra.run.smithi190:> uname -m 2024-04-24T23:04:11.949 INFO:teuthology.orchestra.run.smithi190.stdout:x86_64 2024-04-24T23:04:11.950 DEBUG:teuthology.orchestra.run.smithi190:> cat /etc/os-release 2024-04-24T23:04:12.007 INFO:teuthology.orchestra.run.smithi190.stdout:NAME="CentOS Stream" 2024-04-24T23:04:12.007 INFO:teuthology.orchestra.run.smithi190.stdout:VERSION="9" 2024-04-24T23:04:12.007 INFO:teuthology.orchestra.run.smithi190.stdout:ID="centos" 2024-04-24T23:04:12.007 INFO:teuthology.orchestra.run.smithi190.stdout:ID_LIKE="rhel fedora" 2024-04-24T23:04:12.007 INFO:teuthology.orchestra.run.smithi190.stdout:VERSION_ID="9" 2024-04-24T23:04:12.007 INFO:teuthology.orchestra.run.smithi190.stdout:PLATFORM_ID="platform:el9" 2024-04-24T23:04:12.007 INFO:teuthology.orchestra.run.smithi190.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-24T23:04:12.007 INFO:teuthology.orchestra.run.smithi190.stdout:ANSI_COLOR="0;31" 2024-04-24T23:04:12.008 INFO:teuthology.orchestra.run.smithi190.stdout:LOGO="fedora-logo-icon" 2024-04-24T23:04:12.008 INFO:teuthology.orchestra.run.smithi190.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-24T23:04:12.008 INFO:teuthology.orchestra.run.smithi190.stdout:HOME_URL="https://centos.org/" 2024-04-24T23:04:12.008 INFO:teuthology.orchestra.run.smithi190.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-24T23:04:12.008 INFO:teuthology.orchestra.run.smithi190.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-24T23:04:12.008 INFO:teuthology.orchestra.run.smithi190.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-24T23:04:12.008 INFO:teuthology.lock.ops:Updating smithi190.front.sepia.ceph.com on lock server 2024-04-24T23:04:12.031 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-24T23:04:12.046 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-24T23:04:12.066 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-24T23:04:12.067 DEBUG:teuthology.orchestra.run.smithi006:> test '!' -e /home/ubuntu/cephtest 2024-04-24T23:04:12.069 DEBUG:teuthology.orchestra.run.smithi190:> test '!' -e /home/ubuntu/cephtest 2024-04-24T23:04:12.084 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-24T23:04:12.099 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-24T23:04:12.100 DEBUG:teuthology.orchestra.run.smithi006:> test -z $(ls -A /var/lib/ceph) 2024-04-24T23:04:12.127 DEBUG:teuthology.orchestra.run.smithi190:> test -z $(ls -A /var/lib/ceph) 2024-04-24T23:04:12.222 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-24T23:04:12.278 INFO:teuthology.run_tasks:Running task kernel... 2024-04-24T23:04:12.295 INFO:teuthology.task.kernel:normalize config orig: {'client': {'branch': 'testing'}, 'kdb': 1, 'sha1': 'distro'} 2024-04-24T23:04:12.296 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'branch': 'testing'}} 2024-04-24T23:04:12.296 INFO:teuthology.task.kernel:config {'client.0': {'branch': 'testing'}}, timeout 300 2024-04-24T23:04:12.296 DEBUG:teuthology.orchestra.run.smithi006:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-24T23:04:12.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T23:04:12.314 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&ref=testing 2024-04-24T23:04:12.711 DEBUG:teuthology.task.kernel:sha1 for client.0 is 806ec36b21129f113ce356c007dd59694d340b37 2024-04-24T23:04:12.711 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "806ec36b21129f113ce356c007dd59694d340b37"... 2024-04-24T23:04:12.711 DEBUG:teuthology.orchestra.run.smithi006:> uname -r 2024-04-24T23:04:12.730 INFO:teuthology.orchestra.run.smithi006.stdout:5.14.0-437.el9.x86_64 2024-04-24T23:04:12.730 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-437.el9.x86_64 vs 806ec36b21129f113ce356c007dd59694d340b37 2024-04-24T23:04:12.730 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-04-24T23:04:12.731 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum upgrade -y linux-firmware 2024-04-24T23:04:14.967 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS Stream 9 - BaseOS 14 MB/s | 8.0 MB 00:00 2024-04-24T23:04:18.684 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS Stream 9 - AppStream 7.3 MB/s | 19 MB 00:02 2024-04-24T23:04:23.602 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS Stream 9 - CRB 5.0 MB/s | 6.2 MB 00:01 2024-04-24T23:04:25.102 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS Stream 9 - Extras packages 53 kB/s | 16 kB 00:00 2024-04-24T23:04:28.004 INFO:teuthology.orchestra.run.smithi006.stdout:Extra Packages for Enterprise Linux 7.5 MB/s | 21 MB 00:02 2024-04-24T23:04:33.270 INFO:teuthology.orchestra.run.smithi006.stdout:lab-extras 24 kB/s | 1.7 kB 00:00 2024-04-24T23:04:35.377 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-04-24T23:04:35.378 INFO:teuthology.orchestra.run.smithi006.stdout:Nothing to do. 2024-04-24T23:04:35.378 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-04-24T23:04:35.480 INFO:teuthology.task.kernel:Downloading kernel 806ec36b21129f113ce356c007dd59694d340b37 on client.0... 2024-04-24T23:04:35.481 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=806ec36b21129f113ce356c007dd59694d340b37 2024-04-24T23:04:35.621 INFO:teuthology.task.kernel:fetching, builder baseurl is https://4.chacra.ceph.com/r/kernel/testing/806ec36b21129f113ce356c007dd59694d340b37/centos/9/flavors/default/x86_64/ 2024-04-24T23:04:35.621 DEBUG:teuthology.orchestra.run.smithi006:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.9.0_rc3_g806ec36b2112-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://4.chacra.ceph.com/r/kernel/testing/806ec36b21129f113ce356c007dd59694d340b37/centos/9/flavors/default/x86_64/ --input-file=- 2024-04-24T23:04:35.624 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi006.front.sepia.ceph.com to complete... 2024-04-24T23:04:47.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24 23:04:47 URL:https://4.chacra.ceph.com/r/kernel/testing/806ec36b21129f113ce356c007dd59694d340b37/centos/9/flavors/default/x86_64/kernel-6.9.0_rc3_g806ec36b2112-1.x86_64.rpm [357367792/357367792] -> "/tmp/kernel.x86_64.rpm" [1] 2024-04-24T23:04:47.752 INFO:teuthology.orchestra.run.smithi006.stderr:FINISHED --2024-04-24 23:04:47-- 2024-04-24T23:04:47.752 INFO:teuthology.orchestra.run.smithi006.stderr:Total wall clock time: 12s 2024-04-24T23:04:47.752 INFO:teuthology.orchestra.run.smithi006.stderr:Downloaded: 1 files, 341M in 12s (28.8 MB/s) 2024-04-24T23:04:47.755 INFO:teuthology.task.kernel:Installing kernel 806ec36b21129f113ce356c007dd59694d340b37 on client.0... 2024-04-24T23:04:47.755 DEBUG:teuthology.orchestra.run.smithi006:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-04-24T23:04:49.962 INFO:teuthology.orchestra.run.smithi006.stdout:Verifying... ######################################## 2024-04-24T23:04:50.085 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing... ######################################## 2024-04-24T23:04:50.204 INFO:teuthology.orchestra.run.smithi006.stdout:Updating / installing... 2024-04-24T23:04:58.557 INFO:teuthology.orchestra.run.smithi006.stdout:kernel-6.9.0_rc3_g806ec36b2112-1 ######################################## 2024-04-24T23:05:38.803 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi006.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-04-24T23:05:38.803 DEBUG:teuthology.orchestra.run.smithi006:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-04-24T23:05:38.896 INFO:teuthology.orchestra.run.smithi006.stdout:/boot/System.map-6.9.0-rc3-g806ec36b2112 2024-04-24T23:05:38.896 INFO:teuthology.orchestra.run.smithi006.stdout:/boot/config-6.9.0-rc3-g806ec36b2112 2024-04-24T23:05:38.896 INFO:teuthology.orchestra.run.smithi006.stdout:/boot/vmlinuz-6.9.0-rc3-g806ec36b2112 2024-04-24T23:05:38.896 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/System.map 2024-04-24T23:05:38.897 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/config 2024-04-24T23:05:38.897 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel 2024-04-24T23:05:38.897 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch 2024-04-24T23:05:38.897 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86 2024-04-24T23:05:38.897 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto 2024-04-24T23:05:38.897 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-04-24T23:05:38.897 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-04-24T23:05:38.897 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-04-24T23:05:38.897 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-04-24T23:05:38.897 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-04-24T23:05:38.897 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-04-24T23:05:38.897 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-04-24T23:05:38.897 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/crc32c-intel.ko 2024-04-24T23:05:38.897 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-04-24T23:05:38.898 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-04-24T23:05:38.898 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-04-24T23:05:38.898 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-04-24T23:05:38.898 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/serpent-avx2.ko 2024-04-24T23:05:38.898 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-04-24T23:05:38.898 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-04-24T23:05:38.898 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-04-24T23:05:38.898 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-04-24T23:05:38.898 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-04-24T23:05:38.898 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-04-24T23:05:38.898 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-04-24T23:05:38.898 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kernel 2024-04-24T23:05:38.898 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kernel/cpu 2024-04-24T23:05:38.898 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kernel/cpu/mce 2024-04-24T23:05:38.899 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-04-24T23:05:38.899 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kvm 2024-04-24T23:05:38.899 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kvm/kvm-amd.ko 2024-04-24T23:05:38.899 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kvm/kvm-intel.ko 2024-04-24T23:05:38.899 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kvm/kvm.ko 2024-04-24T23:05:38.899 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto 2024-04-24T23:05:38.899 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/ansi_cprng.ko 2024-04-24T23:05:38.899 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/anubis.ko 2024-04-24T23:05:38.899 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/arc4.ko 2024-04-24T23:05:38.899 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx 2024-04-24T23:05:38.899 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_memcpy.ko 2024-04-24T23:05:38.899 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_pq.ko 2024-04-24T23:05:38.899 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_raid6_recov.ko 2024-04-24T23:05:38.900 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_tx.ko 2024-04-24T23:05:38.900 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_xor.ko 2024-04-24T23:05:38.900 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/raid6test.ko 2024-04-24T23:05:38.900 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/authenc.ko 2024-04-24T23:05:38.900 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/authencesn.ko 2024-04-24T23:05:38.900 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/blake2b_generic.ko 2024-04-24T23:05:38.900 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/blowfish_common.ko 2024-04-24T23:05:38.900 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/blowfish_generic.ko 2024-04-24T23:05:38.900 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/camellia_generic.ko 2024-04-24T23:05:38.900 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cast5_generic.ko 2024-04-24T23:05:38.900 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cast6_generic.ko 2024-04-24T23:05:38.900 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cast_common.ko 2024-04-24T23:05:38.900 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/ccm.ko 2024-04-24T23:05:38.900 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cmac.ko 2024-04-24T23:05:38.901 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/crc32_generic.ko 2024-04-24T23:05:38.901 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/crypto_engine.ko 2024-04-24T23:05:38.901 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/crypto_user.ko 2024-04-24T23:05:38.901 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cts.ko 2024-04-24T23:05:38.901 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/des_generic.ko 2024-04-24T23:05:38.901 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/dh_generic.ko 2024-04-24T23:05:38.901 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/echainiv.ko 2024-04-24T23:05:38.901 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/essiv.ko 2024-04-24T23:05:38.901 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/fcrypt.ko 2024-04-24T23:05:38.901 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/khazad.ko 2024-04-24T23:05:38.901 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/lz4.ko 2024-04-24T23:05:38.901 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/lz4hc.ko 2024-04-24T23:05:38.901 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/md4.ko 2024-04-24T23:05:38.901 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/michael_mic.ko 2024-04-24T23:05:38.901 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/pcbc.ko 2024-04-24T23:05:38.902 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/pcrypt.ko 2024-04-24T23:05:38.902 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/rmd160.ko 2024-04-24T23:05:38.902 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/seed.ko 2024-04-24T23:05:38.902 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/serpent_generic.ko 2024-04-24T23:05:38.902 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/tcrypt.ko 2024-04-24T23:05:38.902 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/tea.ko 2024-04-24T23:05:38.902 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/twofish_common.ko 2024-04-24T23:05:38.902 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/twofish_generic.ko 2024-04-24T23:05:38.902 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/vmac.ko 2024-04-24T23:05:38.902 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/wp512.ko 2024-04-24T23:05:38.902 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/xcbc.ko 2024-04-24T23:05:38.903 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/xor.ko 2024-04-24T23:05:38.903 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/xxhash_generic.ko 2024-04-24T23:05:38.903 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers 2024-04-24T23:05:38.903 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi 2024-04-24T23:05:38.903 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/acpi_ipmi.ko 2024-04-24T23:05:38.903 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/acpi_pad.ko 2024-04-24T23:05:38.903 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/ec_sys.ko 2024-04-24T23:05:38.903 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/platform_profile.ko 2024-04-24T23:05:38.903 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/sbs.ko 2024-04-24T23:05:38.903 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/sbshc.ko 2024-04-24T23:05:38.903 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/video.ko 2024-04-24T23:05:38.903 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata 2024-04-24T23:05:38.903 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/acard-ahci.ko 2024-04-24T23:05:38.903 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/ahci_platform.ko 2024-04-24T23:05:38.903 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/ata_generic.ko 2024-04-24T23:05:38.904 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/libahci_platform.ko 2024-04-24T23:05:38.904 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_acpi.ko 2024-04-24T23:05:38.904 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_ali.ko 2024-04-24T23:05:38.904 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_amd.ko 2024-04-24T23:05:38.904 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_artop.ko 2024-04-24T23:05:38.904 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_atiixp.ko 2024-04-24T23:05:38.904 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_atp867x.ko 2024-04-24T23:05:38.904 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_cmd640.ko 2024-04-24T23:05:38.904 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_cmd64x.ko 2024-04-24T23:05:38.904 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_cypress.ko 2024-04-24T23:05:38.904 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_efar.ko 2024-04-24T23:05:38.904 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_hpt366.ko 2024-04-24T23:05:38.904 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_hpt37x.ko 2024-04-24T23:05:38.904 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_hpt3x2n.ko 2024-04-24T23:05:38.904 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_hpt3x3.ko 2024-04-24T23:05:38.905 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_it8213.ko 2024-04-24T23:05:38.905 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_it821x.ko 2024-04-24T23:05:38.905 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_jmicron.ko 2024-04-24T23:05:38.905 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_marvell.ko 2024-04-24T23:05:38.905 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_mpiix.ko 2024-04-24T23:05:38.905 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_netcell.ko 2024-04-24T23:05:38.905 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_ninja32.ko 2024-04-24T23:05:38.905 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_ns87410.ko 2024-04-24T23:05:38.905 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_ns87415.ko 2024-04-24T23:05:38.905 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_oldpiix.ko 2024-04-24T23:05:38.905 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_opti.ko 2024-04-24T23:05:38.905 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_optidma.ko 2024-04-24T23:05:38.905 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_pcmcia.ko 2024-04-24T23:05:38.905 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_pdc2027x.ko 2024-04-24T23:05:38.906 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-04-24T23:05:38.906 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_piccolo.ko 2024-04-24T23:05:38.906 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_rdc.ko 2024-04-24T23:05:38.906 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_sch.ko 2024-04-24T23:05:38.906 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_serverworks.ko 2024-04-24T23:05:38.906 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_sil680.ko 2024-04-24T23:05:38.906 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_sis.ko 2024-04-24T23:05:38.906 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_sl82c105.ko 2024-04-24T23:05:38.906 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_triflex.ko 2024-04-24T23:05:38.906 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_via.ko 2024-04-24T23:05:38.906 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pdc_adma.ko 2024-04-24T23:05:38.907 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_inic162x.ko 2024-04-24T23:05:38.907 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_mv.ko 2024-04-24T23:05:38.907 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_nv.ko 2024-04-24T23:05:38.907 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_promise.ko 2024-04-24T23:05:38.907 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_qstor.ko 2024-04-24T23:05:38.907 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_sil.ko 2024-04-24T23:05:38.907 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_sil24.ko 2024-04-24T23:05:38.907 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_sis.ko 2024-04-24T23:05:38.907 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_svw.ko 2024-04-24T23:05:38.907 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_sx4.ko 2024-04-24T23:05:38.907 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_uli.ko 2024-04-24T23:05:38.907 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_via.ko 2024-04-24T23:05:38.907 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_vsc.ko 2024-04-24T23:05:38.907 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/auxdisplay 2024-04-24T23:05:38.908 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/auxdisplay/ks0108.ko 2024-04-24T23:05:38.908 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/base 2024-04-24T23:05:38.908 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/base/regmap 2024-04-24T23:05:38.908 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/base/regmap/regmap-i2c.ko 2024-04-24T23:05:38.908 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/base/regmap/regmap-mmio.ko 2024-04-24T23:05:38.908 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/bcma 2024-04-24T23:05:38.908 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/bcma/bcma.ko 2024-04-24T23:05:38.908 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block 2024-04-24T23:05:38.908 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/aoe 2024-04-24T23:05:38.908 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/aoe/aoe.ko 2024-04-24T23:05:38.908 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/brd.ko 2024-04-24T23:05:38.908 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/drbd 2024-04-24T23:05:38.908 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/drbd/drbd.ko 2024-04-24T23:05:38.908 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/floppy.ko 2024-04-24T23:05:38.909 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/loop.ko 2024-04-24T23:05:38.909 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/mtip32xx 2024-04-24T23:05:38.909 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-04-24T23:05:38.909 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/nbd.ko 2024-04-24T23:05:38.909 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/null_blk 2024-04-24T23:05:38.909 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/null_blk/null_blk.ko 2024-04-24T23:05:38.909 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/pktcdvd.ko 2024-04-24T23:05:38.909 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/rbd.ko 2024-04-24T23:05:38.909 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/virtio_blk.ko 2024-04-24T23:05:38.909 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/xen-blkback 2024-04-24T23:05:38.909 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-04-24T23:05:38.909 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/xen-blkfront.ko 2024-04-24T23:05:38.909 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/zram 2024-04-24T23:05:38.909 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/zram/zram.ko 2024-04-24T23:05:38.909 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char 2024-04-24T23:05:38.910 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hangcheck-timer.ko 2024-04-24T23:05:38.910 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random 2024-04-24T23:05:38.910 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/amd-rng.ko 2024-04-24T23:05:38.910 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/intel-rng.ko 2024-04-24T23:05:38.910 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-04-24T23:05:38.910 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/via-rng.ko 2024-04-24T23:05:38.910 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/virtio-rng.ko 2024-04-24T23:05:38.910 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi 2024-04-24T23:05:38.910 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-04-24T23:05:38.910 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-04-24T23:05:38.910 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-04-24T23:05:38.910 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_si.ko 2024-04-24T23:05:38.910 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-04-24T23:05:38.910 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-04-24T23:05:38.910 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/lp.ko 2024-04-24T23:05:38.911 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/mwave 2024-04-24T23:05:38.911 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/mwave/mwave.ko 2024-04-24T23:05:38.911 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ppdev.ko 2024-04-24T23:05:38.911 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tlclk.ko 2024-04-24T23:05:38.911 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm 2024-04-24T23:05:38.911 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm.ko 2024-04-24T23:05:38.911 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_atmel.ko 2024-04-24T23:05:38.911 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_infineon.ko 2024-04-24T23:05:38.911 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_nsc.ko 2024-04-24T23:05:38.911 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_tis.ko 2024-04-24T23:05:38.911 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-04-24T23:05:38.911 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/uv_mmtimer.ko 2024-04-24T23:05:38.911 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/virtio_console.ko 2024-04-24T23:05:38.911 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq 2024-04-24T23:05:38.912 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-04-24T23:05:38.912 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-04-24T23:05:38.912 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/p4-clockmod.ko 2024-04-24T23:05:38.912 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-04-24T23:05:38.912 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/powernow-k8.ko 2024-04-24T23:05:38.912 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/speedstep-lib.ko 2024-04-24T23:05:38.912 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto 2024-04-24T23:05:38.912 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/ccp 2024-04-24T23:05:38.912 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-04-24T23:05:38.912 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/ccp/ccp.ko 2024-04-24T23:05:38.912 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel 2024-04-24T23:05:38.912 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat 2024-04-24T23:05:38.912 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat/qat_common 2024-04-24T23:05:38.912 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-04-24T23:05:38.912 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-04-24T23:05:38.913 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-04-24T23:05:38.913 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/padlock-aes.ko 2024-04-24T23:05:38.913 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/padlock-sha.ko 2024-04-24T23:05:38.913 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/virtio 2024-04-24T23:05:38.913 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-04-24T23:05:38.913 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dca 2024-04-24T23:05:38.913 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dca/dca.ko 2024-04-24T23:05:38.913 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/devfreq 2024-04-24T23:05:38.913 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-04-24T23:05:38.913 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma 2024-04-24T23:05:38.913 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma/dw 2024-04-24T23:05:38.913 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma/dw/dw_dmac.ko 2024-04-24T23:05:38.913 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma/ioat 2024-04-24T23:05:38.913 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma/ioat/ioatdma.ko 2024-04-24T23:05:38.913 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac 2024-04-24T23:05:38.914 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/amd64_edac.ko 2024-04-24T23:05:38.914 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/e752x_edac.ko 2024-04-24T23:05:38.914 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/edac_mce_amd.ko 2024-04-24T23:05:38.914 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i3000_edac.ko 2024-04-24T23:05:38.914 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i3200_edac.ko 2024-04-24T23:05:38.914 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i5100_edac.ko 2024-04-24T23:05:38.914 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i5400_edac.ko 2024-04-24T23:05:38.914 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i7300_edac.ko 2024-04-24T23:05:38.914 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i7core_edac.ko 2024-04-24T23:05:38.914 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i82975x_edac.ko 2024-04-24T23:05:38.914 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/ie31200_edac.ko 2024-04-24T23:05:38.914 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/sb_edac.ko 2024-04-24T23:05:38.914 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/x38_edac.ko 2024-04-24T23:05:38.914 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/firmware 2024-04-24T23:05:38.915 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/firmware/edd.ko 2024-04-24T23:05:38.915 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/firmware/iscsi_ibft.ko 2024-04-24T23:05:38.915 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/gpio 2024-04-24T23:05:38.915 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/gpio/gpio-ich.ko 2024-04-24T23:05:38.915 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/gpio/gpio-viperboard.ko 2024-04-24T23:05:38.915 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid 2024-04-24T23:05:38.915 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-appleir.ko 2024-04-24T23:05:38.915 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-aureal.ko 2024-04-24T23:05:38.915 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-axff.ko 2024-04-24T23:05:38.915 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-dr.ko 2024-04-24T23:05:38.915 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-elecom.ko 2024-04-24T23:05:38.915 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-elo.ko 2024-04-24T23:05:38.915 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-emsff.ko 2024-04-24T23:05:38.915 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-gaff.ko 2024-04-24T23:05:38.915 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-gt683r.ko 2024-04-24T23:05:38.916 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-gyration.ko 2024-04-24T23:05:38.916 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-holtek-kbd.ko 2024-04-24T23:05:38.916 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-holtek-mouse.ko 2024-04-24T23:05:38.916 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-holtekff.ko 2024-04-24T23:05:38.916 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-hyperv.ko 2024-04-24T23:05:38.916 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-icade.ko 2024-04-24T23:05:38.916 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-keytouch.ko 2024-04-24T23:05:38.916 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-kye.ko 2024-04-24T23:05:38.916 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-lcpower.ko 2024-04-24T23:05:38.916 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-led.ko 2024-04-24T23:05:38.916 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-lenovo.ko 2024-04-24T23:05:38.916 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-logitech-dj.ko 2024-04-24T23:05:38.916 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-04-24T23:05:38.916 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-multitouch.ko 2024-04-24T23:05:38.916 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-ortek.ko 2024-04-24T23:05:38.917 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-penmount.ko 2024-04-24T23:05:38.917 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-petalynx.ko 2024-04-24T23:05:38.917 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-picolcd.ko 2024-04-24T23:05:38.917 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-pl.ko 2024-04-24T23:05:38.917 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-plantronics.ko 2024-04-24T23:05:38.917 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-primax.ko 2024-04-24T23:05:38.917 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-rmi.ko 2024-04-24T23:05:38.917 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-arvo.ko 2024-04-24T23:05:38.917 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-common.ko 2024-04-24T23:05:38.917 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-isku.ko 2024-04-24T23:05:38.917 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-kone.ko 2024-04-24T23:05:38.917 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-04-24T23:05:38.917 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-konepure.ko 2024-04-24T23:05:38.917 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-04-24T23:05:38.917 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-lua.ko 2024-04-24T23:05:38.918 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-pyra.ko 2024-04-24T23:05:38.918 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-ryos.ko 2024-04-24T23:05:38.918 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-savu.ko 2024-04-24T23:05:38.918 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat.ko 2024-04-24T23:05:38.918 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-saitek.ko 2024-04-24T23:05:38.918 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-samsung.ko 2024-04-24T23:05:38.918 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-sensor-hub.ko 2024-04-24T23:05:38.918 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-sjoy.ko 2024-04-24T23:05:38.918 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-sony.ko 2024-04-24T23:05:38.918 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-speedlink.ko 2024-04-24T23:05:38.918 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-steelseries.ko 2024-04-24T23:05:38.918 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-sunplus.ko 2024-04-24T23:05:38.918 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-thrustmaster.ko 2024-04-24T23:05:38.918 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-tivo.ko 2024-04-24T23:05:38.918 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-tmff.ko 2024-04-24T23:05:38.919 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-topseed.ko 2024-04-24T23:05:38.919 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-twinhan.ko 2024-04-24T23:05:38.919 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-uclogic.ko 2024-04-24T23:05:38.919 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-waltop.ko 2024-04-24T23:05:38.919 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-wiimote.ko 2024-04-24T23:05:38.919 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-xinmo.ko 2024-04-24T23:05:38.919 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-zpff.ko 2024-04-24T23:05:38.919 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-zydacron.ko 2024-04-24T23:05:38.919 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/uhid.ko 2024-04-24T23:05:38.919 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/wacom.ko 2024-04-24T23:05:38.919 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hv 2024-04-24T23:05:38.919 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hv/hv_balloon.ko 2024-04-24T23:05:38.919 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hv/hv_utils.ko 2024-04-24T23:05:38.919 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hv/hv_vmbus.ko 2024-04-24T23:05:38.919 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon 2024-04-24T23:05:38.920 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/abituguru.ko 2024-04-24T23:05:38.920 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/abituguru3.ko 2024-04-24T23:05:38.920 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/acpi_power_meter.ko 2024-04-24T23:05:38.920 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ad7414.ko 2024-04-24T23:05:38.920 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ad7418.ko 2024-04-24T23:05:38.920 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adc128d818.ko 2024-04-24T23:05:38.920 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm1025.ko 2024-04-24T23:05:38.920 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm1026.ko 2024-04-24T23:05:38.920 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm1029.ko 2024-04-24T23:05:38.920 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm1031.ko 2024-04-24T23:05:38.920 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm9240.ko 2024-04-24T23:05:38.920 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ads7828.ko 2024-04-24T23:05:38.920 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7410.ko 2024-04-24T23:05:38.920 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7411.ko 2024-04-24T23:05:38.920 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7462.ko 2024-04-24T23:05:38.921 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7470.ko 2024-04-24T23:05:38.921 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7475.ko 2024-04-24T23:05:38.921 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7x10.ko 2024-04-24T23:05:38.921 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/amc6821.ko 2024-04-24T23:05:38.921 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/applesmc.ko 2024-04-24T23:05:38.921 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/asb100.ko 2024-04-24T23:05:38.921 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/asc7621.ko 2024-04-24T23:05:38.921 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/asus_atk0110.ko 2024-04-24T23:05:38.921 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/atxp1.ko 2024-04-24T23:05:38.921 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/coretemp.ko 2024-04-24T23:05:38.921 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-04-24T23:05:38.921 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/dme1737.ko 2024-04-24T23:05:38.921 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ds1621.ko 2024-04-24T23:05:38.921 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ds620.ko 2024-04-24T23:05:38.921 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/emc1403.ko 2024-04-24T23:05:38.922 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/emc6w201.ko 2024-04-24T23:05:38.922 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/f71805f.ko 2024-04-24T23:05:38.922 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/f71882fg.ko 2024-04-24T23:05:38.922 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/f75375s.ko 2024-04-24T23:05:38.922 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/fam15h_power.ko 2024-04-24T23:05:38.922 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/fschmd.ko 2024-04-24T23:05:38.922 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/g760a.ko 2024-04-24T23:05:38.922 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/g762.ko 2024-04-24T23:05:38.922 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/gl518sm.ko 2024-04-24T23:05:38.922 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/gl520sm.ko 2024-04-24T23:05:38.922 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/hwmon-vid.ko 2024-04-24T23:05:38.922 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/i5500_temp.ko 2024-04-24T23:05:38.922 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/i5k_amb.ko 2024-04-24T23:05:38.922 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ibmaem.ko 2024-04-24T23:05:38.922 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ibmpex.ko 2024-04-24T23:05:38.923 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ina209.ko 2024-04-24T23:05:38.923 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ina2xx.ko 2024-04-24T23:05:38.923 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/it87.ko 2024-04-24T23:05:38.923 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/k10temp.ko 2024-04-24T23:05:38.923 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/k8temp.ko 2024-04-24T23:05:38.923 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lineage-pem.ko 2024-04-24T23:05:38.923 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm63.ko 2024-04-24T23:05:38.923 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm73.ko 2024-04-24T23:05:38.923 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm75.ko 2024-04-24T23:05:38.923 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm77.ko 2024-04-24T23:05:38.923 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm78.ko 2024-04-24T23:05:38.923 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm80.ko 2024-04-24T23:05:38.923 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm83.ko 2024-04-24T23:05:38.924 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm85.ko 2024-04-24T23:05:38.924 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm87.ko 2024-04-24T23:05:38.924 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm90.ko 2024-04-24T23:05:38.924 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm92.ko 2024-04-24T23:05:38.924 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm93.ko 2024-04-24T23:05:38.924 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm95234.ko 2024-04-24T23:05:38.924 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm95241.ko 2024-04-24T23:05:38.924 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm95245.ko 2024-04-24T23:05:38.924 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc2945.ko 2024-04-24T23:05:38.924 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4151.ko 2024-04-24T23:05:38.924 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4215.ko 2024-04-24T23:05:38.924 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4222.ko 2024-04-24T23:05:38.924 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4245.ko 2024-04-24T23:05:38.925 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4260.ko 2024-04-24T23:05:38.925 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4261.ko 2024-04-24T23:05:38.925 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max16065.ko 2024-04-24T23:05:38.925 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max1619.ko 2024-04-24T23:05:38.925 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max1668.ko 2024-04-24T23:05:38.925 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max197.ko 2024-04-24T23:05:38.925 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max6639.ko 2024-04-24T23:05:38.925 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max6650.ko 2024-04-24T23:05:38.925 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max6697.ko 2024-04-24T23:05:38.925 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/mcp3021.ko 2024-04-24T23:05:38.925 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/nct6683.ko 2024-04-24T23:05:38.925 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/nct6775-core.ko 2024-04-24T23:05:38.925 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/nct6775.ko 2024-04-24T23:05:38.925 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/nct7802.ko 2024-04-24T23:05:38.926 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pc87360.ko 2024-04-24T23:05:38.926 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pc87427.ko 2024-04-24T23:05:38.926 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pcf8591.ko 2024-04-24T23:05:38.926 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus 2024-04-24T23:05:38.926 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-04-24T23:05:38.926 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-04-24T23:05:38.926 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-04-24T23:05:38.926 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/max16064.ko 2024-04-24T23:05:38.926 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/max34440.ko 2024-04-24T23:05:38.926 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/max8688.ko 2024-04-24T23:05:38.926 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-04-24T23:05:38.926 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-04-24T23:05:38.926 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-04-24T23:05:38.926 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-04-24T23:05:38.926 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-04-24T23:05:38.927 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-04-24T23:05:38.927 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/powr1220.ko 2024-04-24T23:05:38.927 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sch5627.ko 2024-04-24T23:05:38.927 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sch5636.ko 2024-04-24T23:05:38.927 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sch56xx-common.ko 2024-04-24T23:05:38.927 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sht15.ko 2024-04-24T23:05:38.927 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sht21.ko 2024-04-24T23:05:38.927 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/shtc1.ko 2024-04-24T23:05:38.927 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sis5595.ko 2024-04-24T23:05:38.927 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/smsc47b397.ko 2024-04-24T23:05:38.927 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/smsc47m1.ko 2024-04-24T23:05:38.927 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/smsc47m192.ko 2024-04-24T23:05:38.927 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/thmc50.ko 2024-04-24T23:05:38.927 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/tmp102.ko 2024-04-24T23:05:38.927 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/tmp103.ko 2024-04-24T23:05:38.928 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/tmp401.ko 2024-04-24T23:05:38.928 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/tmp421.ko 2024-04-24T23:05:38.928 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/via-cputemp.ko 2024-04-24T23:05:38.928 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/via686a.ko 2024-04-24T23:05:38.928 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/vt1211.ko 2024-04-24T23:05:38.928 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/vt8231.ko 2024-04-24T23:05:38.928 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83627ehf.ko 2024-04-24T23:05:38.928 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83627hf.ko 2024-04-24T23:05:38.928 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83781d.ko 2024-04-24T23:05:38.928 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83791d.ko 2024-04-24T23:05:38.929 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83792d.ko 2024-04-24T23:05:38.929 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83793.ko 2024-04-24T23:05:38.929 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83795.ko 2024-04-24T23:05:38.929 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83l785ts.ko 2024-04-24T23:05:38.929 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83l786ng.ko 2024-04-24T23:05:38.929 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c 2024-04-24T23:05:38.929 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/algos 2024-04-24T23:05:38.929 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-04-24T23:05:38.929 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-04-24T23:05:38.929 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses 2024-04-24T23:05:38.929 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-04-24T23:05:38.929 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-04-24T23:05:38.929 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-04-24T23:05:38.930 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-04-24T23:05:38.930 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-04-24T23:05:38.930 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-04-24T23:05:38.930 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-04-24T23:05:38.930 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-04-24T23:05:38.930 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-i801.ko 2024-04-24T23:05:38.930 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-isch.ko 2024-04-24T23:05:38.930 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-04-24T23:05:38.930 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-04-24T23:05:38.930 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-04-24T23:05:38.930 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-parport.ko 2024-04-24T23:05:38.930 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-04-24T23:05:38.930 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-04-24T23:05:38.930 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-04-24T23:05:38.931 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-04-24T23:05:38.931 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-04-24T23:05:38.931 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-04-24T23:05:38.931 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-via.ko 2024-04-24T23:05:38.931 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-04-24T23:05:38.931 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-04-24T23:05:38.931 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-dev.ko 2024-04-24T23:05:38.931 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-mux.ko 2024-04-24T23:05:38.931 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-04-24T23:05:38.931 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-smbus.ko 2024-04-24T23:05:38.931 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-stub.ko 2024-04-24T23:05:38.931 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband 2024-04-24T23:05:38.931 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core 2024-04-24T23:05:38.931 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/ib_cm.ko 2024-04-24T23:05:38.931 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/ib_core.ko 2024-04-24T23:05:38.932 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/ib_umad.ko 2024-04-24T23:05:38.932 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-04-24T23:05:38.932 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/iw_cm.ko 2024-04-24T23:05:38.932 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/rdma_cm.ko 2024-04-24T23:05:38.932 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-04-24T23:05:38.932 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw 2024-04-24T23:05:38.932 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/cxgb4 2024-04-24T23:05:38.932 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-04-24T23:05:38.932 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mlx4 2024-04-24T23:05:38.932 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-04-24T23:05:38.932 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mlx5 2024-04-24T23:05:38.932 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-04-24T23:05:38.932 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mthca 2024-04-24T23:05:38.932 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-04-24T23:05:38.932 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp 2024-04-24T23:05:38.933 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/ipoib 2024-04-24T23:05:38.933 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-04-24T23:05:38.933 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/iser 2024-04-24T23:05:38.933 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-04-24T23:05:38.933 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/isert 2024-04-24T23:05:38.933 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-04-24T23:05:38.933 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/srp 2024-04-24T23:05:38.933 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-04-24T23:05:38.933 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/srpt 2024-04-24T23:05:38.933 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-04-24T23:05:38.933 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input 2024-04-24T23:05:38.933 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/gameport 2024-04-24T23:05:38.933 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-04-24T23:05:38.933 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/gameport/fm801-gp.ko 2024-04-24T23:05:38.933 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/gameport/gameport.ko 2024-04-24T23:05:38.934 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/joydev.ko 2024-04-24T23:05:38.934 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/keyboard 2024-04-24T23:05:38.934 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/keyboard/gpio_keys.ko 2024-04-24T23:05:38.934 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse 2024-04-24T23:05:38.934 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/appletouch.ko 2024-04-24T23:05:38.934 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/bcm5974.ko 2024-04-24T23:05:38.934 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/cyapatp.ko 2024-04-24T23:05:38.934 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/elan_i2c.ko 2024-04-24T23:05:38.934 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/sermouse.ko 2024-04-24T23:05:38.934 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-04-24T23:05:38.934 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/synaptics_usb.ko 2024-04-24T23:05:38.934 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/vsxxxaa.ko 2024-04-24T23:05:38.934 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/rmi4 2024-04-24T23:05:38.934 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/rmi4/rmi_core.ko 2024-04-24T23:05:38.934 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio 2024-04-24T23:05:38.935 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio/altera_ps2.ko 2024-04-24T23:05:38.935 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio/arc_ps2.ko 2024-04-24T23:05:38.935 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-04-24T23:05:38.935 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio/serio_raw.ko 2024-04-24T23:05:38.935 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/sparse-keymap.ko 2024-04-24T23:05:38.935 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds 2024-04-24T23:05:38.935 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/leds-blinkm.ko 2024-04-24T23:05:38.935 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/leds-lm3530.ko 2024-04-24T23:05:38.935 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/leds-lp3944.ko 2024-04-24T23:05:38.935 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/leds-ss4200.ko 2024-04-24T23:05:38.935 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger 2024-04-24T23:05:38.935 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-04-24T23:05:38.935 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-04-24T23:05:38.935 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-04-24T23:05:38.935 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-04-24T23:05:38.936 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-04-24T23:05:38.936 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-04-24T23:05:38.936 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-04-24T23:05:38.936 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-04-24T23:05:38.936 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-04-24T23:05:38.936 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md 2024-04-24T23:05:38.936 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/bcache 2024-04-24T23:05:38.936 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/bcache/bcache.ko 2024-04-24T23:05:38.936 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-bio-prison.ko 2024-04-24T23:05:38.936 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-cache-smq.ko 2024-04-24T23:05:38.936 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-cache.ko 2024-04-24T23:05:38.936 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-crypt.ko 2024-04-24T23:05:38.936 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-delay.ko 2024-04-24T23:05:38.936 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-flakey.ko 2024-04-24T23:05:38.937 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-log-userspace.ko 2024-04-24T23:05:38.937 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-multipath.ko 2024-04-24T23:05:38.937 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-queue-length.ko 2024-04-24T23:05:38.937 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-raid.ko 2024-04-24T23:05:38.937 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-round-robin.ko 2024-04-24T23:05:38.937 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-service-time.ko 2024-04-24T23:05:38.937 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-switch.ko 2024-04-24T23:05:38.937 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-thin-pool.ko 2024-04-24T23:05:38.937 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-verity.ko 2024-04-24T23:05:38.937 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/persistent-data 2024-04-24T23:05:38.937 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-04-24T23:05:38.937 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/raid0.ko 2024-04-24T23:05:38.937 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/raid1.ko 2024-04-24T23:05:38.937 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/raid10.ko 2024-04-24T23:05:38.937 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/raid456.ko 2024-04-24T23:05:38.938 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message 2024-04-24T23:05:38.938 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion 2024-04-24T23:05:38.938 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptbase.ko 2024-04-24T23:05:38.938 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptctl.ko 2024-04-24T23:05:38.938 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptfc.ko 2024-04-24T23:05:38.938 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptlan.ko 2024-04-24T23:05:38.938 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptsas.ko 2024-04-24T23:05:38.938 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptscsih.ko 2024-04-24T23:05:38.938 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptspi.ko 2024-04-24T23:05:38.938 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd 2024-04-24T23:05:38.938 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/lpc_ich.ko 2024-04-24T23:05:38.938 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/lpc_sch.ko 2024-04-24T23:05:38.938 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/mfd-core.ko 2024-04-24T23:05:38.939 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/sm501.ko 2024-04-24T23:05:38.939 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/viperboard.ko 2024-04-24T23:05:38.939 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/vx855.ko 2024-04-24T23:05:38.939 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/wl1273-core.ko 2024-04-24T23:05:38.939 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc 2024-04-24T23:05:38.939 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/altera-stapl 2024-04-24T23:05:38.939 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-04-24T23:05:38.939 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/apds9802als.ko 2024-04-24T23:05:38.939 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/apds990x.ko 2024-04-24T23:05:38.939 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/bh1770glc.ko 2024-04-24T23:05:38.939 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/cb710 2024-04-24T23:05:38.939 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/cb710/cb710.ko 2024-04-24T23:05:38.939 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/eeprom 2024-04-24T23:05:38.940 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/eeprom/at24.ko 2024-04-24T23:05:38.940 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-04-24T23:05:38.940 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/eeprom/max6875.ko 2024-04-24T23:05:38.940 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/enclosure.ko 2024-04-24T23:05:38.940 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/hpilo.ko 2024-04-24T23:05:38.940 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/isl29003.ko 2024-04-24T23:05:38.940 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/isl29020.ko 2024-04-24T23:05:38.940 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/lis3lv02d 2024-04-24T23:05:38.940 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-04-24T23:05:38.940 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-04-24T23:05:38.940 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/mei 2024-04-24T23:05:38.940 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/mei/mei-me.ko 2024-04-24T23:05:38.940 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/mei/mei-txe.ko 2024-04-24T23:05:38.941 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/mei/mei.ko 2024-04-24T23:05:38.941 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-gru 2024-04-24T23:05:38.941 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-gru/gru.ko 2024-04-24T23:05:38.941 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-xp 2024-04-24T23:05:38.941 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-xp/xp.ko 2024-04-24T23:05:38.941 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-xp/xpc.ko 2024-04-24T23:05:38.941 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-04-24T23:05:38.941 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/tifm_7xx1.ko 2024-04-24T23:05:38.941 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/tifm_core.ko 2024-04-24T23:05:38.941 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/tsl2550.ko 2024-04-24T23:05:38.941 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/vmw_balloon.ko 2024-04-24T23:05:38.941 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/vmw_vmci 2024-04-24T23:05:38.941 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-04-24T23:05:38.941 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net 2024-04-24T23:05:38.941 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/bonding 2024-04-24T23:05:38.942 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/bonding/bonding.ko 2024-04-24T23:05:38.942 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa 2024-04-24T23:05:38.942 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/b53 2024-04-24T23:05:38.942 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/b53/b53_common.ko 2024-04-24T23:05:38.942 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/bcm-sf2.ko 2024-04-24T23:05:38.942 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/mv88e6060.ko 2024-04-24T23:05:38.942 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/mv88e6xxx 2024-04-24T23:05:38.942 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-04-24T23:05:38.942 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dummy.ko 2024-04-24T23:05:38.942 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/eql.ko 2024-04-24T23:05:38.942 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet 2024-04-24T23:05:38.942 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com 2024-04-24T23:05:38.942 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-04-24T23:05:38.942 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-04-24T23:05:38.943 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-04-24T23:05:38.943 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-04-24T23:05:38.943 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390 2024-04-24T23:05:38.943 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390/8390.ko 2024-04-24T23:05:38.943 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-04-24T23:05:38.943 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-04-24T23:05:38.943 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-04-24T23:05:38.943 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/adaptec 2024-04-24T23:05:38.943 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-04-24T23:05:38.943 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/agere 2024-04-24T23:05:38.943 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/agere/et131x.ko 2024-04-24T23:05:38.943 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/alteon 2024-04-24T23:05:38.943 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-04-24T23:05:38.943 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/altera 2024-04-24T23:05:38.943 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-04-24T23:05:38.944 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/amd 2024-04-24T23:05:38.944 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-04-24T23:05:38.944 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-04-24T23:05:38.944 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-04-24T23:05:38.944 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros 2024-04-24T23:05:38.944 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/alx 2024-04-24T23:05:38.944 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-04-24T23:05:38.944 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atl1c 2024-04-24T23:05:38.944 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-04-24T23:05:38.944 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atl1e 2024-04-24T23:05:38.944 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-04-24T23:05:38.944 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atlx 2024-04-24T23:05:38.944 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-04-24T23:05:38.944 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-04-24T23:05:38.944 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom 2024-04-24T23:05:38.945 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-04-24T23:05:38.945 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-04-24T23:05:38.945 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-04-24T23:05:38.945 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-04-24T23:05:38.945 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-04-24T23:05:38.945 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/genet 2024-04-24T23:05:38.945 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-04-24T23:05:38.945 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-04-24T23:05:38.945 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/brocade 2024-04-24T23:05:38.945 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/brocade/bna 2024-04-24T23:05:38.945 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-04-24T23:05:38.945 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio 2024-04-24T23:05:38.945 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb 2024-04-24T23:05:38.945 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-04-24T23:05:38.946 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-04-24T23:05:38.946 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-04-24T23:05:38.946 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-04-24T23:05:38.946 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-04-24T23:05:38.946 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-04-24T23:05:38.946 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-04-24T23:05:38.946 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-04-24T23:05:38.946 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-04-24T23:05:38.946 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/cisco 2024-04-24T23:05:38.946 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/cisco/enic 2024-04-24T23:05:38.946 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-04-24T23:05:38.946 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec 2024-04-24T23:05:38.946 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip 2024-04-24T23:05:38.946 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-04-24T23:05:38.947 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-04-24T23:05:38.947 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-04-24T23:05:38.947 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-04-24T23:05:38.947 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-04-24T23:05:38.947 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-04-24T23:05:38.947 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dlink 2024-04-24T23:05:38.947 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-04-24T23:05:38.947 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-04-24T23:05:38.947 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dnet.ko 2024-04-24T23:05:38.947 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/emulex 2024-04-24T23:05:38.947 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/emulex/benet 2024-04-24T23:05:38.947 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-04-24T23:05:38.947 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/ethoc.ko 2024-04-24T23:05:38.947 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/fealnx.ko 2024-04-24T23:05:38.947 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel 2024-04-24T23:05:38.948 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e100.ko 2024-04-24T23:05:38.948 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e1000 2024-04-24T23:05:38.948 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-04-24T23:05:38.948 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e1000e 2024-04-24T23:05:38.948 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-04-24T23:05:38.948 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/fm10k 2024-04-24T23:05:38.948 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-04-24T23:05:38.948 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/i40e 2024-04-24T23:05:38.948 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-04-24T23:05:38.948 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/iavf 2024-04-24T23:05:38.948 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-04-24T23:05:38.948 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/igb 2024-04-24T23:05:38.948 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-04-24T23:05:38.948 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/igbvf 2024-04-24T23:05:38.948 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-04-24T23:05:38.949 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/ixgbe 2024-04-24T23:05:38.949 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-04-24T23:05:38.949 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/ixgbevf 2024-04-24T23:05:38.949 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-04-24T23:05:38.949 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/jme.ko 2024-04-24T23:05:38.949 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/marvell 2024-04-24T23:05:38.949 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-04-24T23:05:38.949 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/marvell/skge.ko 2024-04-24T23:05:38.949 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-04-24T23:05:38.949 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox 2024-04-24T23:05:38.949 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx4 2024-04-24T23:05:38.949 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-04-24T23:05:38.949 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-04-24T23:05:38.949 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx5 2024-04-24T23:05:38.950 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-04-24T23:05:38.950 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-04-24T23:05:38.950 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/micrel 2024-04-24T23:05:38.950 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-04-24T23:05:38.950 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/myricom 2024-04-24T23:05:38.950 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/myricom/myri10ge 2024-04-24T23:05:38.950 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-04-24T23:05:38.950 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/natsemi 2024-04-24T23:05:38.950 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-04-24T23:05:38.950 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-04-24T23:05:38.950 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/neterion 2024-04-24T23:05:38.950 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-04-24T23:05:38.950 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/nvidia 2024-04-24T23:05:38.950 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-04-24T23:05:38.950 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/packetengines 2024-04-24T23:05:38.951 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-04-24T23:05:38.951 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-04-24T23:05:38.951 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic 2024-04-24T23:05:38.951 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/netxen 2024-04-24T23:05:38.951 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-04-24T23:05:38.951 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-04-24T23:05:38.951 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-04-24T23:05:38.951 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-04-24T23:05:38.951 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/rdc 2024-04-24T23:05:38.951 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-04-24T23:05:38.951 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek 2024-04-24T23:05:38.951 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-04-24T23:05:38.951 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-04-24T23:05:38.951 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek/atp.ko 2024-04-24T23:05:38.952 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-04-24T23:05:38.952 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/rocker 2024-04-24T23:05:38.952 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-04-24T23:05:38.952 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sfc 2024-04-24T23:05:38.952 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-04-24T23:05:38.952 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/silan 2024-04-24T23:05:38.952 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-04-24T23:05:38.952 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sis 2024-04-24T23:05:38.952 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sis/sis190.ko 2024-04-24T23:05:38.952 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sis/sis900.ko 2024-04-24T23:05:38.952 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc 2024-04-24T23:05:38.953 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-04-24T23:05:38.953 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-04-24T23:05:38.953 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-04-24T23:05:38.953 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-04-24T23:05:38.953 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/stmicro 2024-04-24T23:05:38.953 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/stmicro/stmmac 2024-04-24T23:05:38.953 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-04-24T23:05:38.953 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-04-24T23:05:38.953 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun 2024-04-24T23:05:38.953 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun/cassini.ko 2024-04-24T23:05:38.953 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun/niu.ko 2024-04-24T23:05:38.953 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun/sungem.ko 2024-04-24T23:05:38.953 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-04-24T23:05:38.953 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/tehuti 2024-04-24T23:05:38.954 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-04-24T23:05:38.954 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/ti 2024-04-24T23:05:38.954 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/ti/tlan.ko 2024-04-24T23:05:38.954 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/via 2024-04-24T23:05:38.954 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-04-24T23:05:38.954 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-04-24T23:05:38.954 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/wiznet 2024-04-24T23:05:38.954 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-04-24T23:05:38.954 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-04-24T23:05:38.954 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/xircom 2024-04-24T23:05:38.954 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-04-24T23:05:38.954 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/geneve.ko 2024-04-24T23:05:38.954 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/hyperv 2024-04-24T23:05:38.955 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-04-24T23:05:38.955 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ieee802154 2024-04-24T23:05:38.955 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ieee802154/fakelb.ko 2024-04-24T23:05:38.955 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ifb.ko 2024-04-24T23:05:38.955 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ipvlan 2024-04-24T23:05:38.955 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ipvlan/ipvlan.ko 2024-04-24T23:05:38.955 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/macvlan.ko 2024-04-24T23:05:38.955 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/macvtap.ko 2024-04-24T23:05:38.955 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio 2024-04-24T23:05:38.955 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio.ko 2024-04-24T23:05:38.955 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-04-24T23:05:38.955 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-04-24T23:05:38.955 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio/mdio-regmap.ko 2024-04-24T23:05:38.955 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mii.ko 2024-04-24T23:05:38.955 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/net_failover.ko 2024-04-24T23:05:38.956 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/netconsole.ko 2024-04-24T23:05:38.956 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/nlmon.ko 2024-04-24T23:05:38.956 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/pcs 2024-04-24T23:05:38.956 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/pcs/pcs-lynx.ko 2024-04-24T23:05:38.956 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-04-24T23:05:38.956 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy 2024-04-24T23:05:38.956 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/amd.ko 2024-04-24T23:05:38.956 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-04-24T23:05:38.956 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-04-24T23:05:38.956 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/bcm7xxx.ko 2024-04-24T23:05:38.956 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/bcm87xx.ko 2024-04-24T23:05:38.956 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/broadcom.ko 2024-04-24T23:05:38.956 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/cicada.ko 2024-04-24T23:05:38.956 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/davicom.ko 2024-04-24T23:05:38.956 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/dp83640.ko 2024-04-24T23:05:38.957 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/et1011c.ko 2024-04-24T23:05:38.957 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/icplus.ko 2024-04-24T23:05:38.957 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/lxt.ko 2024-04-24T23:05:38.957 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/marvell.ko 2024-04-24T23:05:38.957 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/micrel.ko 2024-04-24T23:05:38.957 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/national.ko 2024-04-24T23:05:38.957 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/phylink.ko 2024-04-24T23:05:38.957 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/qsemi.ko 2024-04-24T23:05:38.957 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/realtek.ko 2024-04-24T23:05:38.957 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/smsc.ko 2024-04-24T23:05:38.957 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/ste10Xp.ko 2024-04-24T23:05:38.957 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/vitesse.ko 2024-04-24T23:05:38.957 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp 2024-04-24T23:05:38.957 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/bsd_comp.ko 2024-04-24T23:05:38.958 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_async.ko 2024-04-24T23:05:38.958 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_deflate.ko 2024-04-24T23:05:38.958 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_generic.ko 2024-04-24T23:05:38.958 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_mppe.ko 2024-04-24T23:05:38.958 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_synctty.ko 2024-04-24T23:05:38.958 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/pppoe.ko 2024-04-24T23:05:38.958 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/pppox.ko 2024-04-24T23:05:38.958 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/pptp.ko 2024-04-24T23:05:38.958 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/slip 2024-04-24T23:05:38.958 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/slip/slhc.ko 2024-04-24T23:05:38.958 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/slip/slip.ko 2024-04-24T23:05:38.958 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/sungem_phy.ko 2024-04-24T23:05:38.958 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/tap.ko 2024-04-24T23:05:38.958 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team 2024-04-24T23:05:38.959 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team.ko 2024-04-24T23:05:38.959 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_activebackup.ko 2024-04-24T23:05:38.959 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_broadcast.ko 2024-04-24T23:05:38.959 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-04-24T23:05:38.959 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_random.ko 2024-04-24T23:05:38.959 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-04-24T23:05:38.959 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/tun.ko 2024-04-24T23:05:38.959 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/veth.ko 2024-04-24T23:05:38.959 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/virtio_net.ko 2024-04-24T23:05:38.959 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/vmxnet3 2024-04-24T23:05:38.959 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-04-24T23:05:38.959 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/vxlan 2024-04-24T23:05:38.959 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/vxlan/vxlan.ko 2024-04-24T23:05:38.959 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/xen-netback 2024-04-24T23:05:38.959 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/xen-netback/xen-netback.ko 2024-04-24T23:05:38.959 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/xen-netfront.ko 2024-04-24T23:05:38.960 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ntb 2024-04-24T23:05:38.960 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ntb/ntb.ko 2024-04-24T23:05:38.960 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/nvme 2024-04-24T23:05:38.960 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/nvme/host 2024-04-24T23:05:38.960 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/nvme/host/nvme-core.ko 2024-04-24T23:05:38.960 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/nvme/host/nvme.ko 2024-04-24T23:05:38.960 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport 2024-04-24T23:05:38.960 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport/parport.ko 2024-04-24T23:05:38.960 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport/parport_cs.ko 2024-04-24T23:05:38.960 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport/parport_pc.ko 2024-04-24T23:05:38.960 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport/parport_serial.ko 2024-04-24T23:05:38.960 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci 2024-04-24T23:05:38.960 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/hotplug 2024-04-24T23:05:38.960 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-04-24T23:05:38.960 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/pcie 2024-04-24T23:05:38.961 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/pcie/aer_inject.ko 2024-04-24T23:05:38.961 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/xen-pcifront.ko 2024-04-24T23:05:38.961 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pcmcia 2024-04-24T23:05:38.961 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pcmcia/i82092.ko 2024-04-24T23:05:38.961 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pcmcia/pd6729.ko 2024-04-24T23:05:38.961 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pcmcia/yenta_socket.ko 2024-04-24T23:05:38.961 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pinctrl 2024-04-24T23:05:38.961 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pinctrl/intel 2024-04-24T23:05:38.961 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-04-24T23:05:38.961 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform 2024-04-24T23:05:38.961 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86 2024-04-24T23:05:38.961 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/acer-wmi.ko 2024-04-24T23:05:38.961 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/acerhdf.ko 2024-04-24T23:05:38.961 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/apple-gmux.ko 2024-04-24T23:05:38.962 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/asus-laptop.ko 2024-04-24T23:05:38.962 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-04-24T23:05:38.962 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/asus-wmi.ko 2024-04-24T23:05:38.962 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/classmate-laptop.ko 2024-04-24T23:05:38.962 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-04-24T23:05:38.962 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-04-24T23:05:38.962 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-04-24T23:05:38.962 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-04-24T23:05:38.962 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/hdaps.ko 2024-04-24T23:05:38.962 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/intel 2024-04-24T23:05:38.962 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-04-24T23:05:38.962 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/intel_ips.ko 2024-04-24T23:05:38.962 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/msi-wmi.ko 2024-04-24T23:05:38.962 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/mxm-wmi.ko 2024-04-24T23:05:38.962 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-04-24T23:05:38.963 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/samsung-laptop.ko 2024-04-24T23:05:38.963 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/samsung-q10.ko 2024-04-24T23:05:38.963 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/topstar-laptop.ko 2024-04-24T23:05:38.963 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-04-24T23:05:38.963 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/toshiba_haps.ko 2024-04-24T23:05:38.963 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/wmi-bmof.ko 2024-04-24T23:05:38.963 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/wmi.ko 2024-04-24T23:05:38.963 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/powercap 2024-04-24T23:05:38.963 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/powercap/intel_rapl_common.ko 2024-04-24T23:05:38.963 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/powercap/intel_rapl_msr.ko 2024-04-24T23:05:38.963 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps 2024-04-24T23:05:38.963 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/clients 2024-04-24T23:05:38.963 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/clients/pps-gpio.ko 2024-04-24T23:05:38.963 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/clients/pps-ldisc.ko 2024-04-24T23:05:38.963 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/clients/pps_parport.ko 2024-04-24T23:05:38.963 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/pps_core.ko 2024-04-24T23:05:38.964 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ptp 2024-04-24T23:05:38.964 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ptp/ptp.ko 2024-04-24T23:05:38.964 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ptp/ptp_kvm.ko 2024-04-24T23:05:38.964 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras 2024-04-24T23:05:38.964 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras/amd 2024-04-24T23:05:38.964 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras/amd/atl 2024-04-24T23:05:38.964 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras/amd/atl/amd_atl.ko 2024-04-24T23:05:38.964 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras/amd/fmpm.ko 2024-04-24T23:05:38.964 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc 2024-04-24T23:05:38.964 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-bq32k.ko 2024-04-24T23:05:38.964 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1286.ko 2024-04-24T23:05:38.964 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1307.ko 2024-04-24T23:05:38.964 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1374.ko 2024-04-24T23:05:38.964 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1511.ko 2024-04-24T23:05:38.964 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1553.ko 2024-04-24T23:05:38.965 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1672.ko 2024-04-24T23:05:38.965 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1742.ko 2024-04-24T23:05:38.965 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds2404.ko 2024-04-24T23:05:38.965 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds3232.ko 2024-04-24T23:05:38.965 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-em3027.ko 2024-04-24T23:05:38.965 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-fm3130.ko 2024-04-24T23:05:38.965 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-isl12022.ko 2024-04-24T23:05:38.965 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-isl1208.ko 2024-04-24T23:05:38.965 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-m41t80.ko 2024-04-24T23:05:38.965 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-m48t35.ko 2024-04-24T23:05:38.965 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-m48t59.ko 2024-04-24T23:05:38.965 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-max6900.ko 2024-04-24T23:05:38.965 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-msm6242.ko 2024-04-24T23:05:38.965 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf2127.ko 2024-04-24T23:05:38.965 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf85063.ko 2024-04-24T23:05:38.965 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf8523.ko 2024-04-24T23:05:38.966 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf8563.ko 2024-04-24T23:05:38.966 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf8583.ko 2024-04-24T23:05:38.966 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rp5c01.ko 2024-04-24T23:05:38.966 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rs5c372.ko 2024-04-24T23:05:38.966 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-04-24T23:05:38.966 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rx8025.ko 2024-04-24T23:05:38.966 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rx8581.ko 2024-04-24T23:05:38.966 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-04-24T23:05:38.966 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-x1205.ko 2024-04-24T23:05:38.966 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi 2024-04-24T23:05:38.966 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/3w-9xxx.ko 2024-04-24T23:05:38.966 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/3w-sas.ko 2024-04-24T23:05:38.966 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/3w-xxxx.ko 2024-04-24T23:05:38.966 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/BusLogic.ko 2024-04-24T23:05:38.967 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/a100u2w.ko 2024-04-24T23:05:38.967 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aacraid 2024-04-24T23:05:38.967 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aacraid/aacraid.ko 2024-04-24T23:05:38.967 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/advansys.ko 2024-04-24T23:05:38.967 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aic7xxx 2024-04-24T23:05:38.967 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-04-24T23:05:38.967 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-04-24T23:05:38.967 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/am53c974.ko 2024-04-24T23:05:38.967 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/arcmsr 2024-04-24T23:05:38.967 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-04-24T23:05:38.967 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/atp870u.ko 2024-04-24T23:05:38.967 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/be2iscsi 2024-04-24T23:05:38.967 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-04-24T23:05:38.967 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bfa 2024-04-24T23:05:38.967 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bfa/bfa.ko 2024-04-24T23:05:38.967 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bnx2fc 2024-04-24T23:05:38.968 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-04-24T23:05:38.968 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bnx2i 2024-04-24T23:05:38.968 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-04-24T23:05:38.968 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/ch.ko 2024-04-24T23:05:38.968 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/csiostor 2024-04-24T23:05:38.968 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/csiostor/csiostor.ko 2024-04-24T23:05:38.968 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi 2024-04-24T23:05:38.968 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/cxgb3i 2024-04-24T23:05:38.968 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-04-24T23:05:38.968 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/cxgb4i 2024-04-24T23:05:38.968 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-04-24T23:05:38.968 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-04-24T23:05:38.968 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/dc395x.ko 2024-04-24T23:05:38.968 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler 2024-04-24T23:05:38.968 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-04-24T23:05:38.969 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-04-24T23:05:38.969 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-04-24T23:05:38.969 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-04-24T23:05:38.969 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/dmx3191d.ko 2024-04-24T23:05:38.969 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/esas2r 2024-04-24T23:05:38.969 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/esas2r/esas2r.ko 2024-04-24T23:05:38.969 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/esp_scsi.ko 2024-04-24T23:05:38.969 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fcoe 2024-04-24T23:05:38.969 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fcoe/fcoe.ko 2024-04-24T23:05:38.969 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-04-24T23:05:38.969 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fnic 2024-04-24T23:05:38.969 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fnic/fnic.ko 2024-04-24T23:05:38.969 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/hpsa.ko 2024-04-24T23:05:38.969 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/hptiop.ko 2024-04-24T23:05:38.969 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/hv_storvsc.ko 2024-04-24T23:05:38.970 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/initio.ko 2024-04-24T23:05:38.970 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/ipr.ko 2024-04-24T23:05:38.970 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/ips.ko 2024-04-24T23:05:38.970 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/isci 2024-04-24T23:05:38.970 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/isci/isci.ko 2024-04-24T23:05:38.970 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-04-24T23:05:38.970 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/iscsi_tcp.ko 2024-04-24T23:05:38.970 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libfc 2024-04-24T23:05:38.970 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libfc/libfc.ko 2024-04-24T23:05:38.970 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libiscsi.ko 2024-04-24T23:05:38.970 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libiscsi_tcp.ko 2024-04-24T23:05:38.970 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libsas 2024-04-24T23:05:38.971 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libsas/libsas.ko 2024-04-24T23:05:38.971 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/lpfc 2024-04-24T23:05:38.971 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/lpfc/lpfc.ko 2024-04-24T23:05:38.971 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid 2024-04-24T23:05:38.971 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid.ko 2024-04-24T23:05:38.971 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-04-24T23:05:38.971 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-04-24T23:05:38.971 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-04-24T23:05:38.971 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mpt3sas 2024-04-24T23:05:38.971 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-04-24T23:05:38.971 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mvsas 2024-04-24T23:05:38.971 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mvsas/mvsas.ko 2024-04-24T23:05:38.971 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mvumi.ko 2024-04-24T23:05:38.971 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/pm8001 2024-04-24T23:05:38.972 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-04-24T23:05:38.972 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/pmcraid.ko 2024-04-24T23:05:38.972 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla1280.ko 2024-04-24T23:05:38.972 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla2xxx 2024-04-24T23:05:38.972 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-04-24T23:05:38.972 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-04-24T23:05:38.972 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla4xxx 2024-04-24T23:05:38.972 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-04-24T23:05:38.972 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/raid_class.ko 2024-04-24T23:05:38.972 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_debug.ko 2024-04-24T23:05:38.972 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_fc.ko 2024-04-24T23:05:38.972 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-04-24T23:05:38.972 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_sas.ko 2024-04-24T23:05:38.972 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_spi.ko 2024-04-24T23:05:38.972 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_srp.ko 2024-04-24T23:05:38.973 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/ses.ko 2024-04-24T23:05:38.973 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/st.ko 2024-04-24T23:05:38.973 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/stex.ko 2024-04-24T23:05:38.973 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/sym53c8xx_2 2024-04-24T23:05:38.973 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-04-24T23:05:38.973 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/virtio_scsi.ko 2024-04-24T23:05:38.973 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/vmw_pvscsi.ko 2024-04-24T23:05:38.973 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/wd719x.ko 2024-04-24T23:05:38.973 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ssb 2024-04-24T23:05:38.973 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ssb/ssb.ko 2024-04-24T23:05:38.973 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target 2024-04-24T23:05:38.973 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/iscsi 2024-04-24T23:05:38.973 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-04-24T23:05:38.973 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/loopback 2024-04-24T23:05:38.974 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/loopback/tcm_loop.ko 2024-04-24T23:05:38.974 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_file.ko 2024-04-24T23:05:38.974 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_iblock.ko 2024-04-24T23:05:38.974 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_mod.ko 2024-04-24T23:05:38.974 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_pscsi.ko 2024-04-24T23:05:38.974 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_user.ko 2024-04-24T23:05:38.974 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/tcm_fc 2024-04-24T23:05:38.974 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-04-24T23:05:38.974 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal 2024-04-24T23:05:38.974 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel 2024-04-24T23:05:38.974 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal 2024-04-24T23:05:38.974 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-04-24T23:05:38.974 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-04-24T23:05:38.974 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-04-24T23:05:38.975 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-04-24T23:05:38.975 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-04-24T23:05:38.975 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-04-24T23:05:38.975 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-04-24T23:05:38.975 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-04-24T23:05:38.975 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-04-24T23:05:38.975 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-04-24T23:05:38.975 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-04-24T23:05:38.975 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-04-24T23:05:38.975 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-04-24T23:05:38.975 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-04-24T23:05:38.975 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-04-24T23:05:38.975 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-04-24T23:05:38.975 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-04-24T23:05:38.976 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-04-24T23:05:38.976 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty 2024-04-24T23:05:38.976 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/ipwireless 2024-04-24T23:05:38.976 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-04-24T23:05:38.976 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/n_gsm.ko 2024-04-24T23:05:38.976 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/n_hdlc.ko 2024-04-24T23:05:38.976 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/nozomi.ko 2024-04-24T23:05:38.976 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial 2024-04-24T23:05:38.976 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/8250 2024-04-24T23:05:38.976 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-04-24T23:05:38.976 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/arc_uart.ko 2024-04-24T23:05:38.976 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/jsm 2024-04-24T23:05:38.976 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/jsm/jsm.ko 2024-04-24T23:05:38.976 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs 2024-04-24T23:05:38.977 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs/core 2024-04-24T23:05:38.977 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs/core/ufshcd-core.ko 2024-04-24T23:05:38.977 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs/host 2024-04-24T23:05:38.977 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-04-24T23:05:38.977 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio 2024-04-24T23:05:38.977 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio.ko 2024-04-24T23:05:38.977 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio_aec.ko 2024-04-24T23:05:38.977 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio_cif.ko 2024-04-24T23:05:38.977 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio_pci_generic.ko 2024-04-24T23:05:38.977 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio_sercos3.ko 2024-04-24T23:05:38.977 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb 2024-04-24T23:05:38.977 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class 2024-04-24T23:05:38.977 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class/cdc-acm.ko 2024-04-24T23:05:38.977 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class/cdc-wdm.ko 2024-04-24T23:05:38.977 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class/usblp.ko 2024-04-24T23:05:38.978 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class/usbtmc.ko 2024-04-24T23:05:38.978 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/host 2024-04-24T23:05:38.978 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/host/sl811-hcd.ko 2024-04-24T23:05:38.978 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/image 2024-04-24T23:05:38.978 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/image/mdc800.ko 2024-04-24T23:05:38.978 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/image/microtek.ko 2024-04-24T23:05:38.978 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc 2024-04-24T23:05:38.978 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/adutux.ko 2024-04-24T23:05:38.978 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/appledisplay.ko 2024-04-24T23:05:38.978 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/emi26.ko 2024-04-24T23:05:38.978 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/emi62.ko 2024-04-24T23:05:38.978 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/ezusb.ko 2024-04-24T23:05:38.978 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/idmouse.ko 2024-04-24T23:05:38.978 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/iowarrior.ko 2024-04-24T23:05:38.979 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/isight_firmware.ko 2024-04-24T23:05:38.979 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/ldusb.ko 2024-04-24T23:05:38.979 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/legousbtower.ko 2024-04-24T23:05:38.979 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/sisusbvga 2024-04-24T23:05:38.979 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-04-24T23:05:38.979 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/trancevibrator.ko 2024-04-24T23:05:38.979 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/usb3503.ko 2024-04-24T23:05:38.979 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/usblcd.ko 2024-04-24T23:05:38.979 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/usbsevseg.ko 2024-04-24T23:05:38.979 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/uss720.ko 2024-04-24T23:05:38.979 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/yurex.ko 2024-04-24T23:05:38.979 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/phy 2024-04-24T23:05:38.979 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/phy/phy-generic.ko 2024-04-24T23:05:38.979 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial 2024-04-24T23:05:38.979 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/aircable.ko 2024-04-24T23:05:38.980 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ark3116.ko 2024-04-24T23:05:38.980 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/belkin_sa.ko 2024-04-24T23:05:38.980 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ch341.ko 2024-04-24T23:05:38.980 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/cp210x.ko 2024-04-24T23:05:38.980 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/cyberjack.ko 2024-04-24T23:05:38.980 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/cypress_m8.ko 2024-04-24T23:05:38.980 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/digi_acceleport.ko 2024-04-24T23:05:38.980 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/empeg.ko 2024-04-24T23:05:38.980 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ftdi_sio.ko 2024-04-24T23:05:38.980 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/garmin_gps.ko 2024-04-24T23:05:38.980 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/io_edgeport.ko 2024-04-24T23:05:38.980 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/io_ti.ko 2024-04-24T23:05:38.980 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ipaq.ko 2024-04-24T23:05:38.980 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ipw.ko 2024-04-24T23:05:38.981 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ir-usb.ko 2024-04-24T23:05:38.981 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-04-24T23:05:38.981 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/keyspan.ko 2024-04-24T23:05:38.981 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/keyspan_pda.ko 2024-04-24T23:05:38.981 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/kl5kusb105.ko 2024-04-24T23:05:38.981 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/kobil_sct.ko 2024-04-24T23:05:38.981 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/mct_u232.ko 2024-04-24T23:05:38.981 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/mos7720.ko 2024-04-24T23:05:38.981 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/mos7840.ko 2024-04-24T23:05:38.981 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/navman.ko 2024-04-24T23:05:38.981 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/omninet.ko 2024-04-24T23:05:38.981 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/opticon.ko 2024-04-24T23:05:38.981 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/option.ko 2024-04-24T23:05:38.981 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/oti6858.ko 2024-04-24T23:05:38.981 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/pl2303.ko 2024-04-24T23:05:38.982 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/qcaux.ko 2024-04-24T23:05:38.982 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/qcserial.ko 2024-04-24T23:05:38.982 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/quatech2.ko 2024-04-24T23:05:38.982 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/safe_serial.ko 2024-04-24T23:05:38.982 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/sierra.ko 2024-04-24T23:05:38.982 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/spcp8x5.ko 2024-04-24T23:05:38.982 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ssu100.ko 2024-04-24T23:05:38.982 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/symbolserial.ko 2024-04-24T23:05:38.982 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-04-24T23:05:38.982 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-04-24T23:05:38.982 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/usb_debug.ko 2024-04-24T23:05:38.982 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/usb_wwan.ko 2024-04-24T23:05:38.982 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/visor.ko 2024-04-24T23:05:38.982 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/whiteheat.ko 2024-04-24T23:05:38.983 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/xsens_mt.ko 2024-04-24T23:05:38.983 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage 2024-04-24T23:05:38.983 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/uas.ko 2024-04-24T23:05:38.983 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-alauda.ko 2024-04-24T23:05:38.983 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-cypress.ko 2024-04-24T23:05:38.983 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-datafab.ko 2024-04-24T23:05:38.983 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-04-24T23:05:38.983 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-freecom.ko 2024-04-24T23:05:38.983 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-isd200.ko 2024-04-24T23:05:38.983 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-04-24T23:05:38.983 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-karma.ko 2024-04-24T23:05:38.983 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-onetouch.ko 2024-04-24T23:05:38.984 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-realtek.ko 2024-04-24T23:05:38.984 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-sddr09.ko 2024-04-24T23:05:38.984 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-sddr55.ko 2024-04-24T23:05:38.984 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-usbat.ko 2024-04-24T23:05:38.984 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/usb-storage.ko 2024-04-24T23:05:38.984 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/usbip 2024-04-24T23:05:38.984 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/usbip/usbip-core.ko 2024-04-24T23:05:38.984 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/usbip/usbip-host.ko 2024-04-24T23:05:38.984 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-04-24T23:05:38.984 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio 2024-04-24T23:05:38.984 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/pci 2024-04-24T23:05:38.984 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-04-24T23:05:38.984 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/pci/vfio-pci.ko 2024-04-24T23:05:38.984 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/vfio.ko 2024-04-24T23:05:38.985 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-04-24T23:05:38.985 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost 2024-04-24T23:05:38.985 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost/vhost.ko 2024-04-24T23:05:38.985 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost/vhost_iotlb.ko 2024-04-24T23:05:38.985 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost/vhost_net.ko 2024-04-24T23:05:38.985 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost/vhost_scsi.ko 2024-04-24T23:05:38.985 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video 2024-04-24T23:05:38.985 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video/backlight 2024-04-24T23:05:38.985 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video/backlight/apple_bl.ko 2024-04-24T23:05:38.985 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video/backlight/lcd.ko 2024-04-24T23:05:38.985 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video/backlight/platform_lcd.ko 2024-04-24T23:05:38.985 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio 2024-04-24T23:05:38.985 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio.ko 2024-04-24T23:05:38.985 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_balloon.ko 2024-04-24T23:05:38.985 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_mmio.ko 2024-04-24T23:05:38.986 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_pci.ko 2024-04-24T23:05:38.986 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-04-24T23:05:38.986 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-04-24T23:05:38.986 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_ring.ko 2024-04-24T23:05:38.986 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1 2024-04-24T23:05:38.986 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/masters 2024-04-24T23:05:38.986 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/masters/ds2482.ko 2024-04-24T23:05:38.986 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/masters/ds2490.ko 2024-04-24T23:05:38.987 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves 2024-04-24T23:05:38.987 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-04-24T23:05:38.987 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-04-24T23:05:38.987 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-04-24T23:05:38.987 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-04-24T23:05:38.987 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-04-24T23:05:38.988 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-04-24T23:05:38.988 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-04-24T23:05:38.988 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-04-24T23:05:38.988 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-04-24T23:05:38.988 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_smem.ko 2024-04-24T23:05:38.988 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_therm.ko 2024-04-24T23:05:38.989 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/wire.ko 2024-04-24T23:05:38.989 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog 2024-04-24T23:05:38.989 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/alim1535_wdt.ko 2024-04-24T23:05:38.989 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/alim7101_wdt.ko 2024-04-24T23:05:38.989 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/f71808e_wdt.ko 2024-04-24T23:05:38.989 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/hpwdt.ko 2024-04-24T23:05:38.990 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/i6300esb.ko 2024-04-24T23:05:38.990 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-04-24T23:05:38.990 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/iTCO_wdt.ko 2024-04-24T23:05:38.990 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/ib700wdt.ko 2024-04-24T23:05:38.990 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/ibmasr.ko 2024-04-24T23:05:38.990 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-04-24T23:05:38.991 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/it8712f_wdt.ko 2024-04-24T23:05:38.991 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/it87_wdt.ko 2024-04-24T23:05:38.991 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/machzwd.ko 2024-04-24T23:05:38.991 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/nv_tco.ko 2024-04-24T23:05:38.991 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/pcwd_pci.ko 2024-04-24T23:05:38.991 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/pcwd_usb.ko 2024-04-24T23:05:38.992 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-04-24T23:05:38.992 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/sch311x_wdt.ko 2024-04-24T23:05:38.992 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/softdog.ko 2024-04-24T23:05:38.992 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/sp5100_tco.ko 2024-04-24T23:05:38.992 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/via_wdt.ko 2024-04-24T23:05:38.992 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-04-24T23:05:38.993 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/w83877f_wdt.ko 2024-04-24T23:05:38.993 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/w83977f_wdt.ko 2024-04-24T23:05:38.993 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/wdt_pci.ko 2024-04-24T23:05:38.993 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/xen_wdt.ko 2024-04-24T23:05:38.993 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen 2024-04-24T23:05:38.993 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-acpi-processor.ko 2024-04-24T23:05:38.994 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-evtchn.ko 2024-04-24T23:05:38.994 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-gntalloc.ko 2024-04-24T23:05:38.994 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-gntdev.ko 2024-04-24T23:05:38.994 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-pciback 2024-04-24T23:05:38.994 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-04-24T23:05:38.994 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-privcmd.ko 2024-04-24T23:05:38.995 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xenfs 2024-04-24T23:05:38.995 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xenfs/xenfs.ko 2024-04-24T23:05:38.995 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs 2024-04-24T23:05:38.995 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/9p 2024-04-24T23:05:38.995 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/9p/9p.ko 2024-04-24T23:05:38.995 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/affs 2024-04-24T23:05:38.996 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/affs/affs.ko 2024-04-24T23:05:38.996 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/befs 2024-04-24T23:05:38.996 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/befs/befs.ko 2024-04-24T23:05:38.996 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/binfmt_misc.ko 2024-04-24T23:05:38.996 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/btrfs 2024-04-24T23:05:38.997 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/btrfs/btrfs.ko 2024-04-24T23:05:38.997 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/cachefiles 2024-04-24T23:05:38.997 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/cachefiles/cachefiles.ko 2024-04-24T23:05:38.997 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ceph 2024-04-24T23:05:38.997 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ceph/ceph.ko 2024-04-24T23:05:38.997 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/coda 2024-04-24T23:05:38.998 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/coda/coda.ko 2024-04-24T23:05:38.998 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/cramfs 2024-04-24T23:05:38.998 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/cramfs/cramfs.ko 2024-04-24T23:05:38.998 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/dlm 2024-04-24T23:05:38.998 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/dlm/dlm.ko 2024-04-24T23:05:38.998 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ecryptfs 2024-04-24T23:05:38.999 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ecryptfs/ecryptfs.ko 2024-04-24T23:05:38.999 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/f2fs 2024-04-24T23:05:38.999 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/f2fs/f2fs.ko 2024-04-24T23:05:38.999 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fat 2024-04-24T23:05:38.999 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fat/fat.ko 2024-04-24T23:05:38.999 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fat/msdos.ko 2024-04-24T23:05:39.000 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fat/vfat.ko 2024-04-24T23:05:39.000 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fuse 2024-04-24T23:05:39.000 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fuse/cuse.ko 2024-04-24T23:05:39.000 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fuse/fuse.ko 2024-04-24T23:05:39.000 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/gfs2 2024-04-24T23:05:39.001 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/gfs2/gfs2.ko 2024-04-24T23:05:39.001 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/hfs 2024-04-24T23:05:39.001 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/hfs/hfs.ko 2024-04-24T23:05:39.001 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/hfsplus 2024-04-24T23:05:39.001 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/hfsplus/hfsplus.ko 2024-04-24T23:05:39.001 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/isofs 2024-04-24T23:05:39.002 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/isofs/isofs.ko 2024-04-24T23:05:39.002 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/jfs 2024-04-24T23:05:39.002 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/jfs/jfs.ko 2024-04-24T23:05:39.002 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/lockd 2024-04-24T23:05:39.002 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/lockd/lockd.ko 2024-04-24T23:05:39.003 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/minix 2024-04-24T23:05:39.003 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/minix/minix.ko 2024-04-24T23:05:39.003 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/netfs 2024-04-24T23:05:39.003 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/netfs/netfs.ko 2024-04-24T23:05:39.003 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs 2024-04-24T23:05:39.003 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/blocklayout 2024-04-24T23:05:39.004 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-04-24T23:05:39.004 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/filelayout 2024-04-24T23:05:39.004 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-04-24T23:05:39.004 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/flexfilelayout 2024-04-24T23:05:39.004 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-04-24T23:05:39.004 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/nfs.ko 2024-04-24T23:05:39.005 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/nfsv3.ko 2024-04-24T23:05:39.005 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/nfsv4.ko 2024-04-24T23:05:39.005 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs_common 2024-04-24T23:05:39.005 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs_common/grace.ko 2024-04-24T23:05:39.005 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs_common/nfs_acl.ko 2024-04-24T23:05:39.005 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfsd 2024-04-24T23:05:39.006 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfsd/nfsd.ko 2024-04-24T23:05:39.006 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nilfs2 2024-04-24T23:05:39.006 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nilfs2/nilfs2.ko 2024-04-24T23:05:39.006 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls 2024-04-24T23:05:39.006 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-celtic.ko 2024-04-24T23:05:39.006 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-centeuro.ko 2024-04-24T23:05:39.007 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-croatian.ko 2024-04-24T23:05:39.007 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-cyrillic.ko 2024-04-24T23:05:39.007 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-gaelic.ko 2024-04-24T23:05:39.007 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-greek.ko 2024-04-24T23:05:39.007 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-iceland.ko 2024-04-24T23:05:39.008 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-inuit.ko 2024-04-24T23:05:39.008 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-roman.ko 2024-04-24T23:05:39.008 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-romanian.ko 2024-04-24T23:05:39.008 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-turkish.ko 2024-04-24T23:05:39.008 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp1250.ko 2024-04-24T23:05:39.008 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp1251.ko 2024-04-24T23:05:39.008 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp1255.ko 2024-04-24T23:05:39.008 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp737.ko 2024-04-24T23:05:39.008 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp775.ko 2024-04-24T23:05:39.008 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp850.ko 2024-04-24T23:05:39.009 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp852.ko 2024-04-24T23:05:39.009 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp855.ko 2024-04-24T23:05:39.009 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp857.ko 2024-04-24T23:05:39.009 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp860.ko 2024-04-24T23:05:39.009 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp861.ko 2024-04-24T23:05:39.009 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp862.ko 2024-04-24T23:05:39.009 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp863.ko 2024-04-24T23:05:39.009 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp864.ko 2024-04-24T23:05:39.009 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp865.ko 2024-04-24T23:05:39.009 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp866.ko 2024-04-24T23:05:39.009 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp869.ko 2024-04-24T23:05:39.009 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp874.ko 2024-04-24T23:05:39.009 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp932.ko 2024-04-24T23:05:39.009 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp936.ko 2024-04-24T23:05:39.009 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp949.ko 2024-04-24T23:05:39.010 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp950.ko 2024-04-24T23:05:39.010 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_euc-jp.ko 2024-04-24T23:05:39.010 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-1.ko 2024-04-24T23:05:39.010 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-13.ko 2024-04-24T23:05:39.010 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-14.ko 2024-04-24T23:05:39.010 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-15.ko 2024-04-24T23:05:39.010 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-2.ko 2024-04-24T23:05:39.010 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-3.ko 2024-04-24T23:05:39.010 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-4.ko 2024-04-24T23:05:39.010 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-5.ko 2024-04-24T23:05:39.010 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-6.ko 2024-04-24T23:05:39.010 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-7.ko 2024-04-24T23:05:39.010 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-9.ko 2024-04-24T23:05:39.011 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_koi8-r.ko 2024-04-24T23:05:39.011 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_koi8-ru.ko 2024-04-24T23:05:39.011 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_koi8-u.ko 2024-04-24T23:05:39.011 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_ucs2_utils.ko 2024-04-24T23:05:39.011 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_utf8.ko 2024-04-24T23:05:39.011 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2 2024-04-24T23:05:39.011 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/cluster 2024-04-24T23:05:39.011 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-04-24T23:05:39.011 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/dlm 2024-04-24T23:05:39.011 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-04-24T23:05:39.011 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/dlmfs 2024-04-24T23:05:39.011 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-04-24T23:05:39.011 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/ocfs2.ko 2024-04-24T23:05:39.011 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-04-24T23:05:39.011 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-04-24T23:05:39.012 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-04-24T23:05:39.012 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/overlayfs 2024-04-24T23:05:39.012 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/overlayfs/overlay.ko 2024-04-24T23:05:39.012 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/pstore 2024-04-24T23:05:39.012 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/pstore/ramoops.ko 2024-04-24T23:05:39.012 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/reiserfs 2024-04-24T23:05:39.012 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/reiserfs/reiserfs.ko 2024-04-24T23:05:39.012 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/romfs 2024-04-24T23:05:39.012 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/romfs/romfs.ko 2024-04-24T23:05:39.012 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb 2024-04-24T23:05:39.012 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/client 2024-04-24T23:05:39.012 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/client/cifs.ko 2024-04-24T23:05:39.012 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/common 2024-04-24T23:05:39.012 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/common/cifs_arc4.ko 2024-04-24T23:05:39.012 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/common/cifs_md4.ko 2024-04-24T23:05:39.013 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/squashfs 2024-04-24T23:05:39.013 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/squashfs/squashfs.ko 2024-04-24T23:05:39.013 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/sysv 2024-04-24T23:05:39.013 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/sysv/sysv.ko 2024-04-24T23:05:39.013 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/udf 2024-04-24T23:05:39.013 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/udf/udf.ko 2024-04-24T23:05:39.013 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ufs 2024-04-24T23:05:39.013 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ufs/ufs.ko 2024-04-24T23:05:39.013 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/xfs 2024-04-24T23:05:39.013 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/xfs/xfs.ko 2024-04-24T23:05:39.013 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel 2024-04-24T23:05:39.013 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/rcu 2024-04-24T23:05:39.013 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/rcu/rcutorture.ko 2024-04-24T23:05:39.013 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/torture.ko 2024-04-24T23:05:39.013 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/trace 2024-04-24T23:05:39.014 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-04-24T23:05:39.014 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib 2024-04-24T23:05:39.014 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/asn1_encoder.ko 2024-04-24T23:05:39.014 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crc-itu-t.ko 2024-04-24T23:05:39.014 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crc8.ko 2024-04-24T23:05:39.014 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crypto 2024-04-24T23:05:39.014 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crypto/libarc4.ko 2024-04-24T23:05:39.014 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crypto/libdes.ko 2024-04-24T23:05:39.014 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/libcrc32c.ko 2024-04-24T23:05:39.014 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/lru_cache.ko 2024-04-24T23:05:39.014 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/lz4 2024-04-24T23:05:39.014 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/lz4/lz4_compress.ko 2024-04-24T23:05:39.014 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/lz4/lz4hc_compress.ko 2024-04-24T23:05:39.015 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/math 2024-04-24T23:05:39.015 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/math/cordic.ko 2024-04-24T23:05:39.015 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/raid6 2024-04-24T23:05:39.015 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/raid6/raid6_pq.ko 2024-04-24T23:05:39.015 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/reed_solomon 2024-04-24T23:05:39.015 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/reed_solomon/reed_solomon.ko 2024-04-24T23:05:39.015 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/ts_bm.ko 2024-04-24T23:05:39.015 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/ts_fsm.ko 2024-04-24T23:05:39.015 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/ts_kmp.ko 2024-04-24T23:05:39.015 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/zstd 2024-04-24T23:05:39.015 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/zstd/zstd_compress.ko 2024-04-24T23:05:39.015 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/mm 2024-04-24T23:05:39.015 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/mm/hwpoison-inject.ko 2024-04-24T23:05:39.016 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net 2024-04-24T23:05:39.016 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan 2024-04-24T23:05:39.016 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/6lowpan.ko 2024-04-24T23:05:39.016 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_dest.ko 2024-04-24T23:05:39.016 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_fragment.ko 2024-04-24T23:05:39.016 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_hop.ko 2024-04-24T23:05:39.016 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_ipv6.ko 2024-04-24T23:05:39.016 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_mobility.ko 2024-04-24T23:05:39.016 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_routing.ko 2024-04-24T23:05:39.016 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_udp.ko 2024-04-24T23:05:39.016 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802 2024-04-24T23:05:39.016 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/garp.ko 2024-04-24T23:05:39.016 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/mrp.ko 2024-04-24T23:05:39.016 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/p8022.ko 2024-04-24T23:05:39.016 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/psnap.ko 2024-04-24T23:05:39.017 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/stp.ko 2024-04-24T23:05:39.017 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/8021q 2024-04-24T23:05:39.017 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/8021q/8021q.ko 2024-04-24T23:05:39.017 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p 2024-04-24T23:05:39.017 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p/9pnet.ko 2024-04-24T23:05:39.017 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p/9pnet_fd.ko 2024-04-24T23:05:39.017 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p/9pnet_rdma.ko 2024-04-24T23:05:39.017 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p/9pnet_virtio.ko 2024-04-24T23:05:39.017 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/batman-adv 2024-04-24T23:05:39.017 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/batman-adv/batman-adv.ko 2024-04-24T23:05:39.017 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge 2024-04-24T23:05:39.017 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/br_netfilter.ko 2024-04-24T23:05:39.017 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/bridge.ko 2024-04-24T23:05:39.017 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter 2024-04-24T23:05:39.017 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-04-24T23:05:39.018 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_among.ko 2024-04-24T23:05:39.018 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_arp.ko 2024-04-24T23:05:39.018 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-04-24T23:05:39.018 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-04-24T23:05:39.018 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_ip.ko 2024-04-24T23:05:39.018 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-04-24T23:05:39.018 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_limit.ko 2024-04-24T23:05:39.018 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_log.ko 2024-04-24T23:05:39.018 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_mark.ko 2024-04-24T23:05:39.018 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-04-24T23:05:39.018 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-04-24T23:05:39.018 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-04-24T23:05:39.018 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-04-24T23:05:39.018 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_snat.ko 2024-04-24T23:05:39.019 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_stp.ko 2024-04-24T23:05:39.019 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-04-24T23:05:39.019 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-04-24T23:05:39.019 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-04-24T23:05:39.019 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-04-24T23:05:39.019 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebtables.ko 2024-04-24T23:05:39.019 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-04-24T23:05:39.019 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-04-24T23:05:39.019 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-04-24T23:05:39.019 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ceph 2024-04-24T23:05:39.019 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ceph/libceph.ko 2024-04-24T23:05:39.019 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/core 2024-04-24T23:05:39.019 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/core/failover.ko 2024-04-24T23:05:39.019 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/core/pktgen.ko 2024-04-24T23:05:39.019 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp 2024-04-24T23:05:39.020 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp/dccp.ko 2024-04-24T23:05:39.020 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp/dccp_diag.ko 2024-04-24T23:05:39.020 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp/dccp_ipv4.ko 2024-04-24T23:05:39.020 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp/dccp_ipv6.ko 2024-04-24T23:05:39.020 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dns_resolver 2024-04-24T23:05:39.020 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dns_resolver/dns_resolver.ko 2024-04-24T23:05:39.020 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa 2024-04-24T23:05:39.020 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/dsa_core.ko 2024-04-24T23:05:39.020 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/tag_brcm.ko 2024-04-24T23:05:39.020 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/tag_dsa.ko 2024-04-24T23:05:39.020 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/tag_none.ko 2024-04-24T23:05:39.020 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/tag_trailer.ko 2024-04-24T23:05:39.020 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154 2024-04-24T23:05:39.020 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154/6lowpan 2024-04-24T23:05:39.020 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-04-24T23:05:39.021 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154/ieee802154.ko 2024-04-24T23:05:39.021 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154/ieee802154_socket.ko 2024-04-24T23:05:39.021 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4 2024-04-24T23:05:39.021 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ah4.ko 2024-04-24T23:05:39.021 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/esp4.ko 2024-04-24T23:05:39.021 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/fou.ko 2024-04-24T23:05:39.021 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/gre.ko 2024-04-24T23:05:39.021 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/inet_diag.ko 2024-04-24T23:05:39.021 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ip_gre.ko 2024-04-24T23:05:39.021 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ip_tunnel.ko 2024-04-24T23:05:39.021 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ip_vti.ko 2024-04-24T23:05:39.021 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ipcomp.ko 2024-04-24T23:05:39.021 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ipip.ko 2024-04-24T23:05:39.021 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter 2024-04-24T23:05:39.021 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/arp_tables.ko 2024-04-24T23:05:39.021 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-04-24T23:05:39.022 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-04-24T23:05:39.022 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-04-24T23:05:39.022 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-04-24T23:05:39.022 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-04-24T23:05:39.022 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-04-24T23:05:39.022 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-04-24T23:05:39.022 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-04-24T23:05:39.022 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-04-24T23:05:39.022 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/iptable_security.ko 2024-04-24T23:05:39.022 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-04-24T23:05:39.022 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-04-24T23:05:39.022 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-04-24T23:05:39.022 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-04-24T23:05:39.023 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-04-24T23:05:39.023 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-04-24T23:05:39.023 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-04-24T23:05:39.023 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-04-24T23:05:39.023 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-04-24T23:05:39.023 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-04-24T23:05:39.023 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_bic.ko 2024-04-24T23:05:39.023 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_dctcp.ko 2024-04-24T23:05:39.023 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_diag.ko 2024-04-24T23:05:39.023 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_highspeed.ko 2024-04-24T23:05:39.023 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_htcp.ko 2024-04-24T23:05:39.023 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_hybla.ko 2024-04-24T23:05:39.023 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_illinois.ko 2024-04-24T23:05:39.023 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_lp.ko 2024-04-24T23:05:39.024 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_scalable.ko 2024-04-24T23:05:39.024 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_vegas.ko 2024-04-24T23:05:39.024 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_veno.ko 2024-04-24T23:05:39.024 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_westwood.ko 2024-04-24T23:05:39.024 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_yeah.ko 2024-04-24T23:05:39.024 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tunnel4.ko 2024-04-24T23:05:39.024 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/udp_diag.ko 2024-04-24T23:05:39.024 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/udp_tunnel.ko 2024-04-24T23:05:39.024 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/xfrm4_tunnel.ko 2024-04-24T23:05:39.024 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6 2024-04-24T23:05:39.024 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ah6.ko 2024-04-24T23:05:39.024 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/esp6.ko 2024-04-24T23:05:39.024 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/fou6.ko 2024-04-24T23:05:39.024 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ip6_tunnel.ko 2024-04-24T23:05:39.025 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-04-24T23:05:39.025 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ip6_vti.ko 2024-04-24T23:05:39.025 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ipcomp6.ko 2024-04-24T23:05:39.025 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter 2024-04-24T23:05:39.025 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-04-24T23:05:39.025 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-04-24T23:05:39.025 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-04-24T23:05:39.025 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-04-24T23:05:39.025 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-04-24T23:05:39.025 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-04-24T23:05:39.025 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-04-24T23:05:39.025 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-04-24T23:05:39.025 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-04-24T23:05:39.025 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-04-24T23:05:39.026 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-04-24T23:05:39.026 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-04-24T23:05:39.026 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-04-24T23:05:39.026 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-04-24T23:05:39.026 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-04-24T23:05:39.026 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-04-24T23:05:39.026 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-04-24T23:05:39.026 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-04-24T23:05:39.026 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-04-24T23:05:39.026 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-04-24T23:05:39.026 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-04-24T23:05:39.026 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-04-24T23:05:39.026 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-04-24T23:05:39.027 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-04-24T23:05:39.027 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/sit.ko 2024-04-24T23:05:39.027 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/tunnel6.ko 2024-04-24T23:05:39.027 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/xfrm6_tunnel.ko 2024-04-24T23:05:39.027 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/key 2024-04-24T23:05:39.027 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/key/af_key.ko 2024-04-24T23:05:39.027 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp 2024-04-24T23:05:39.027 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_core.ko 2024-04-24T23:05:39.027 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_debugfs.ko 2024-04-24T23:05:39.027 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_eth.ko 2024-04-24T23:05:39.027 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_ip.ko 2024-04-24T23:05:39.027 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_ip6.ko 2024-04-24T23:05:39.027 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_netlink.ko 2024-04-24T23:05:39.027 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_ppp.ko 2024-04-24T23:05:39.027 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/llc 2024-04-24T23:05:39.028 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/llc/llc.ko 2024-04-24T23:05:39.028 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/mac802154 2024-04-24T23:05:39.028 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/mac802154/mac802154.ko 2024-04-24T23:05:39.028 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/mpls 2024-04-24T23:05:39.028 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/mpls/mpls_gso.ko 2024-04-24T23:05:39.028 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter 2024-04-24T23:05:39.028 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset 2024-04-24T23:05:39.028 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set.ko 2024-04-24T23:05:39.028 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-04-24T23:05:39.028 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-04-24T23:05:39.028 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-04-24T23:05:39.028 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-04-24T23:05:39.028 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-04-24T23:05:39.028 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-04-24T23:05:39.028 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-04-24T23:05:39.029 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-04-24T23:05:39.029 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-04-24T23:05:39.029 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-04-24T23:05:39.029 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-04-24T23:05:39.029 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-04-24T23:05:39.029 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-04-24T23:05:39.029 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-04-24T23:05:39.029 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-04-24T23:05:39.029 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs 2024-04-24T23:05:39.029 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs.ko 2024-04-24T23:05:39.029 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-04-24T23:05:39.029 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-04-24T23:05:39.029 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-04-24T23:05:39.030 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-04-24T23:05:39.030 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-04-24T23:05:39.030 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-04-24T23:05:39.030 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-04-24T23:05:39.030 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-04-24T23:05:39.030 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-04-24T23:05:39.030 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-04-24T23:05:39.030 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-04-24T23:05:39.030 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-04-24T23:05:39.030 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-04-24T23:05:39.030 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conncount.ko 2024-04-24T23:05:39.030 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack.ko 2024-04-24T23:05:39.030 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-04-24T23:05:39.031 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-04-24T23:05:39.031 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-04-24T23:05:39.031 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_h323.ko 2024-04-24T23:05:39.031 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_irc.ko 2024-04-24T23:05:39.031 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-04-24T23:05:39.031 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-04-24T23:05:39.031 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-04-24T23:05:39.031 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_sane.ko 2024-04-24T23:05:39.031 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_sip.ko 2024-04-24T23:05:39.031 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-04-24T23:05:39.031 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-04-24T23:05:39.031 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_dup_netdev.ko 2024-04-24T23:05:39.031 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_flow_table.ko 2024-04-24T23:05:39.031 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_flow_table_inet.ko 2024-04-24T23:05:39.031 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_log_syslog.ko 2024-04-24T23:05:39.032 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat.ko 2024-04-24T23:05:39.032 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_amanda.ko 2024-04-24T23:05:39.032 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_ftp.ko 2024-04-24T23:05:39.032 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_irc.ko 2024-04-24T23:05:39.032 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_sip.ko 2024-04-24T23:05:39.032 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_tftp.ko 2024-04-24T23:05:39.032 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_synproxy_core.ko 2024-04-24T23:05:39.032 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_tables.ko 2024-04-24T23:05:39.032 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink.ko 2024-04-24T23:05:39.032 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink_acct.ko 2024-04-24T23:05:39.032 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink_log.ko 2024-04-24T23:05:39.032 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink_osf.ko 2024-04-24T23:05:39.032 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink_queue.ko 2024-04-24T23:05:39.032 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_chain_nat.ko 2024-04-24T23:05:39.032 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_compat.ko 2024-04-24T23:05:39.033 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_ct.ko 2024-04-24T23:05:39.033 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_dup_netdev.ko 2024-04-24T23:05:39.033 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_fib.ko 2024-04-24T23:05:39.033 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_fwd_netdev.ko 2024-04-24T23:05:39.033 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_hash.ko 2024-04-24T23:05:39.033 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_limit.ko 2024-04-24T23:05:39.033 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_log.ko 2024-04-24T23:05:39.033 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_masq.ko 2024-04-24T23:05:39.033 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_nat.ko 2024-04-24T23:05:39.033 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_numgen.ko 2024-04-24T23:05:39.033 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_queue.ko 2024-04-24T23:05:39.033 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_quota.ko 2024-04-24T23:05:39.033 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_redir.ko 2024-04-24T23:05:39.033 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_reject.ko 2024-04-24T23:05:39.034 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_reject_inet.ko 2024-04-24T23:05:39.034 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_socket.ko 2024-04-24T23:05:39.034 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_tunnel.ko 2024-04-24T23:05:39.034 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_AUDIT.ko 2024-04-24T23:05:39.034 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_CHECKSUM.ko 2024-04-24T23:05:39.034 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_CLASSIFY.ko 2024-04-24T23:05:39.034 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-04-24T23:05:39.034 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_CT.ko 2024-04-24T23:05:39.034 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_DSCP.ko 2024-04-24T23:05:39.034 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_HL.ko 2024-04-24T23:05:39.034 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_HMARK.ko 2024-04-24T23:05:39.034 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_IDLETIMER.ko 2024-04-24T23:05:39.034 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_LED.ko 2024-04-24T23:05:39.034 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_LOG.ko 2024-04-24T23:05:39.035 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_MASQUERADE.ko 2024-04-24T23:05:39.035 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_NETMAP.ko 2024-04-24T23:05:39.035 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_NFLOG.ko 2024-04-24T23:05:39.035 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_NFQUEUE.ko 2024-04-24T23:05:39.035 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_RATEEST.ko 2024-04-24T23:05:39.035 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_REDIRECT.ko 2024-04-24T23:05:39.035 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_SECMARK.ko 2024-04-24T23:05:39.035 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TCPMSS.ko 2024-04-24T23:05:39.035 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-04-24T23:05:39.035 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TEE.ko 2024-04-24T23:05:39.035 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TPROXY.ko 2024-04-24T23:05:39.035 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TRACE.ko 2024-04-24T23:05:39.035 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_addrtype.ko 2024-04-24T23:05:39.035 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_bpf.ko 2024-04-24T23:05:39.035 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_cgroup.ko 2024-04-24T23:05:39.036 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_cluster.ko 2024-04-24T23:05:39.036 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_comment.ko 2024-04-24T23:05:39.036 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_connbytes.ko 2024-04-24T23:05:39.036 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_connlabel.ko 2024-04-24T23:05:39.036 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_connlimit.ko 2024-04-24T23:05:39.036 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_connmark.ko 2024-04-24T23:05:39.036 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_conntrack.ko 2024-04-24T23:05:39.036 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_cpu.ko 2024-04-24T23:05:39.036 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_dccp.ko 2024-04-24T23:05:39.036 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_devgroup.ko 2024-04-24T23:05:39.036 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_dscp.ko 2024-04-24T23:05:39.036 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_ecn.ko 2024-04-24T23:05:39.036 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_esp.ko 2024-04-24T23:05:39.036 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_hashlimit.ko 2024-04-24T23:05:39.036 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_helper.ko 2024-04-24T23:05:39.036 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_hl.ko 2024-04-24T23:05:39.037 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_ipcomp.ko 2024-04-24T23:05:39.037 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_iprange.ko 2024-04-24T23:05:39.037 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_ipvs.ko 2024-04-24T23:05:39.037 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_l2tp.ko 2024-04-24T23:05:39.037 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_length.ko 2024-04-24T23:05:39.037 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_limit.ko 2024-04-24T23:05:39.037 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_mac.ko 2024-04-24T23:05:39.037 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_mark.ko 2024-04-24T23:05:39.037 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_multiport.ko 2024-04-24T23:05:39.037 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_nat.ko 2024-04-24T23:05:39.037 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_nfacct.ko 2024-04-24T23:05:39.037 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_osf.ko 2024-04-24T23:05:39.037 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_owner.ko 2024-04-24T23:05:39.037 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_physdev.ko 2024-04-24T23:05:39.037 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_pkttype.ko 2024-04-24T23:05:39.038 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_policy.ko 2024-04-24T23:05:39.038 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_quota.ko 2024-04-24T23:05:39.038 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_rateest.ko 2024-04-24T23:05:39.038 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_realm.ko 2024-04-24T23:05:39.038 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_recent.ko 2024-04-24T23:05:39.038 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_sctp.ko 2024-04-24T23:05:39.038 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_set.ko 2024-04-24T23:05:39.038 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_socket.ko 2024-04-24T23:05:39.038 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_state.ko 2024-04-24T23:05:39.038 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_statistic.ko 2024-04-24T23:05:39.038 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_string.ko 2024-04-24T23:05:39.038 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_tcpmss.ko 2024-04-24T23:05:39.038 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_time.ko 2024-04-24T23:05:39.039 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_u32.ko 2024-04-24T23:05:39.039 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netlink 2024-04-24T23:05:39.039 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netlink/netlink_diag.ko 2024-04-24T23:05:39.039 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/nsh 2024-04-24T23:05:39.039 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/nsh/nsh.ko 2024-04-24T23:05:39.039 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch 2024-04-24T23:05:39.039 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch/openvswitch.ko 2024-04-24T23:05:39.039 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch/vport-geneve.ko 2024-04-24T23:05:39.039 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch/vport-gre.ko 2024-04-24T23:05:39.039 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch/vport-vxlan.ko 2024-04-24T23:05:39.039 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/packet 2024-04-24T23:05:39.039 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/packet/af_packet_diag.ko 2024-04-24T23:05:39.039 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/rds 2024-04-24T23:05:39.040 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/rds/rds.ko 2024-04-24T23:05:39.040 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/rds/rds_rdma.ko 2024-04-24T23:05:39.040 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/rds/rds_tcp.ko 2024-04-24T23:05:39.040 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched 2024-04-24T23:05:39.040 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_csum.ko 2024-04-24T23:05:39.040 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_gact.ko 2024-04-24T23:05:39.040 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_mirred.ko 2024-04-24T23:05:39.040 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_nat.ko 2024-04-24T23:05:39.040 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_pedit.ko 2024-04-24T23:05:39.040 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_police.ko 2024-04-24T23:05:39.040 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_simple.ko 2024-04-24T23:05:39.040 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_skbedit.ko 2024-04-24T23:05:39.041 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_vlan.ko 2024-04-24T23:05:39.041 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_basic.ko 2024-04-24T23:05:39.041 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_bpf.ko 2024-04-24T23:05:39.041 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_flow.ko 2024-04-24T23:05:39.041 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_fw.ko 2024-04-24T23:05:39.041 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_route.ko 2024-04-24T23:05:39.041 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_u32.ko 2024-04-24T23:05:39.041 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_cmp.ko 2024-04-24T23:05:39.041 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_ipset.ko 2024-04-24T23:05:39.041 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_meta.ko 2024-04-24T23:05:39.041 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_nbyte.ko 2024-04-24T23:05:39.041 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_text.ko 2024-04-24T23:05:39.041 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_u32.ko 2024-04-24T23:05:39.041 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_choke.ko 2024-04-24T23:05:39.042 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_codel.ko 2024-04-24T23:05:39.042 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_drr.ko 2024-04-24T23:05:39.042 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_fq.ko 2024-04-24T23:05:39.042 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_gred.ko 2024-04-24T23:05:39.042 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_hfsc.ko 2024-04-24T23:05:39.042 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_hhf.ko 2024-04-24T23:05:39.042 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_htb.ko 2024-04-24T23:05:39.042 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_ingress.ko 2024-04-24T23:05:39.042 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_mqprio.ko 2024-04-24T23:05:39.042 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_mqprio_lib.ko 2024-04-24T23:05:39.042 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_multiq.ko 2024-04-24T23:05:39.042 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_netem.ko 2024-04-24T23:05:39.042 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_pie.ko 2024-04-24T23:05:39.042 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_plug.ko 2024-04-24T23:05:39.042 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_prio.ko 2024-04-24T23:05:39.043 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_qfq.ko 2024-04-24T23:05:39.043 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_red.ko 2024-04-24T23:05:39.043 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_sfb.ko 2024-04-24T23:05:39.043 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_sfq.ko 2024-04-24T23:05:39.043 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_tbf.ko 2024-04-24T23:05:39.043 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_teql.ko 2024-04-24T23:05:39.043 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sctp 2024-04-24T23:05:39.043 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sctp/sctp.ko 2024-04-24T23:05:39.043 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sctp/sctp_diag.ko 2024-04-24T23:05:39.043 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc 2024-04-24T23:05:39.043 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/auth_gss 2024-04-24T23:05:39.043 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-04-24T23:05:39.043 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-04-24T23:05:39.043 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/sunrpc.ko 2024-04-24T23:05:39.043 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/xprtrdma 2024-04-24T23:05:39.044 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-04-24T23:05:39.044 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/tipc 2024-04-24T23:05:39.044 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/tipc/tipc.ko 2024-04-24T23:05:39.044 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/tipc/tipc_diag.ko 2024-04-24T23:05:39.044 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/unix 2024-04-24T23:05:39.044 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/unix/unix_diag.ko 2024-04-24T23:05:39.044 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock 2024-04-24T23:05:39.044 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-04-24T23:05:39.044 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-04-24T23:05:39.044 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vsock.ko 2024-04-24T23:05:39.044 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vsock_diag.ko 2024-04-24T23:05:39.044 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vsock_loopback.ko 2024-04-24T23:05:39.044 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/xfrm 2024-04-24T23:05:39.044 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/xfrm/xfrm_ipcomp.ko 2024-04-24T23:05:39.044 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security 2024-04-24T23:05:39.044 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys 2024-04-24T23:05:39.045 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys/encrypted-keys 2024-04-24T23:05:39.045 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-04-24T23:05:39.045 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys/trusted-keys 2024-04-24T23:05:39.045 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys/trusted-keys/trusted.ko 2024-04-24T23:05:39.045 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.alias 2024-04-24T23:05:39.045 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.alias.bin 2024-04-24T23:05:39.045 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.builtin 2024-04-24T23:05:39.045 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.builtin.alias.bin 2024-04-24T23:05:39.045 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.builtin.bin 2024-04-24T23:05:39.045 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.builtin.modinfo 2024-04-24T23:05:39.045 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.dep 2024-04-24T23:05:39.045 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.dep.bin 2024-04-24T23:05:39.045 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.devname 2024-04-24T23:05:39.045 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.order 2024-04-24T23:05:39.045 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.softdep 2024-04-24T23:05:39.046 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.symbols 2024-04-24T23:05:39.046 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.symbols.bin 2024-04-24T23:05:39.046 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/vmlinuz 2024-04-24T23:05:39.049 DEBUG:teuthology.task.kernel:get_image_version: 6.9.0-rc3-g806ec36b2112 2024-04-24T23:05:39.050 DEBUG:teuthology.orchestra.run.smithi006:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-04-24T23:05:39.073 INFO:teuthology.orchestra.run.smithi006.stdout:postinstall scriptlet (using /bin/sh): 2024-04-24T23:05:39.074 INFO:teuthology.orchestra.run.smithi006.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-04-24T23:05:39.074 INFO:teuthology.orchestra.run.smithi006.stdout: /usr/bin/kernel-install add 6.9.0-rc3-g806ec36b2112 /lib/modules/6.9.0-rc3-g806ec36b2112/vmlinuz 2024-04-24T23:05:39.074 INFO:teuthology.orchestra.run.smithi006.stdout:fi 2024-04-24T23:05:39.074 INFO:teuthology.orchestra.run.smithi006.stdout:for file in vmlinuz System.map config; do 2024-04-24T23:05:39.074 INFO:teuthology.orchestra.run.smithi006.stdout: if ! cmp --silent "/lib/modules/6.9.0-rc3-g806ec36b2112/${file}" "/boot/${file}-6.9.0-rc3-g806ec36b2112"; then 2024-04-24T23:05:39.075 INFO:teuthology.orchestra.run.smithi006.stdout: cp "/lib/modules/6.9.0-rc3-g806ec36b2112/${file}" "/boot/${file}-6.9.0-rc3-g806ec36b2112" 2024-04-24T23:05:39.075 INFO:teuthology.orchestra.run.smithi006.stdout: fi 2024-04-24T23:05:39.075 INFO:teuthology.orchestra.run.smithi006.stdout:done 2024-04-24T23:05:39.075 INFO:teuthology.orchestra.run.smithi006.stdout:if [ -d "/lib/modules/6.9.0-rc3-g806ec36b2112/dtb" ] && \ 2024-04-24T23:05:39.075 INFO:teuthology.orchestra.run.smithi006.stdout: ! diff -rq "/lib/modules/6.9.0-rc3-g806ec36b2112/dtb" "/boot/dtb-6.9.0-rc3-g806ec36b2112" >/dev/null 2>&1; then 2024-04-24T23:05:39.075 INFO:teuthology.orchestra.run.smithi006.stdout: rm -rf "/boot/dtb-6.9.0-rc3-g806ec36b2112" 2024-04-24T23:05:39.075 INFO:teuthology.orchestra.run.smithi006.stdout: cp -r "/lib/modules/6.9.0-rc3-g806ec36b2112/dtb" "/boot/dtb-6.9.0-rc3-g806ec36b2112" 2024-04-24T23:05:39.075 INFO:teuthology.orchestra.run.smithi006.stdout:fi 2024-04-24T23:05:39.075 INFO:teuthology.orchestra.run.smithi006.stdout:if [ ! -e "/lib/modules/6.9.0-rc3-g806ec36b2112/modules.dep" ]; then 2024-04-24T23:05:39.075 INFO:teuthology.orchestra.run.smithi006.stdout: /usr/sbin/depmod 6.9.0-rc3-g806ec36b2112 2024-04-24T23:05:39.076 INFO:teuthology.orchestra.run.smithi006.stdout:fi 2024-04-24T23:05:39.076 INFO:teuthology.orchestra.run.smithi006.stdout:preuninstall scriptlet (using /bin/sh): 2024-04-24T23:05:39.076 INFO:teuthology.orchestra.run.smithi006.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-04-24T23:05:39.076 INFO:teuthology.orchestra.run.smithi006.stdout:kernel-install remove 6.9.0-rc3-g806ec36b2112 2024-04-24T23:05:39.076 INFO:teuthology.orchestra.run.smithi006.stdout:fi 2024-04-24T23:05:39.077 DEBUG:teuthology.orchestra.run.smithi006:> sudo rpm -qi grub2-tools 2024-04-24T23:05:39.160 INFO:teuthology.orchestra.run.smithi006.stdout:Name : grub2-tools 2024-04-24T23:05:39.160 INFO:teuthology.orchestra.run.smithi006.stdout:Epoch : 1 2024-04-24T23:05:39.160 INFO:teuthology.orchestra.run.smithi006.stdout:Version : 2.06 2024-04-24T23:05:39.160 INFO:teuthology.orchestra.run.smithi006.stdout:Release : 68.el9 2024-04-24T23:05:39.160 INFO:teuthology.orchestra.run.smithi006.stdout:Architecture: x86_64 2024-04-24T23:05:39.160 INFO:teuthology.orchestra.run.smithi006.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-04-24T23:05:39.160 INFO:teuthology.orchestra.run.smithi006.stdout:Group : Unspecified 2024-04-24T23:05:39.160 INFO:teuthology.orchestra.run.smithi006.stdout:Size : 8264254 2024-04-24T23:05:39.161 INFO:teuthology.orchestra.run.smithi006.stdout:License : GPLv3+ 2024-04-24T23:05:39.161 INFO:teuthology.orchestra.run.smithi006.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-04-24T23:05:39.161 INFO:teuthology.orchestra.run.smithi006.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-04-24T23:05:39.161 INFO:teuthology.orchestra.run.smithi006.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-04-24T23:05:39.161 INFO:teuthology.orchestra.run.smithi006.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-04-24T23:05:39.161 INFO:teuthology.orchestra.run.smithi006.stdout:Packager : builder@centos.org 2024-04-24T23:05:39.161 INFO:teuthology.orchestra.run.smithi006.stdout:Vendor : CentOS 2024-04-24T23:05:39.161 INFO:teuthology.orchestra.run.smithi006.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-24T23:05:39.161 INFO:teuthology.orchestra.run.smithi006.stdout:Summary : Support tools for GRUB. 2024-04-24T23:05:39.161 INFO:teuthology.orchestra.run.smithi006.stdout:Description : 2024-04-24T23:05:39.161 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:05:39.161 INFO:teuthology.orchestra.run.smithi006.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-24T23:05:39.161 INFO:teuthology.orchestra.run.smithi006.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-24T23:05:39.161 INFO:teuthology.orchestra.run.smithi006.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-24T23:05:39.162 INFO:teuthology.orchestra.run.smithi006.stdout:hardware devices. 2024-04-24T23:05:39.162 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:05:39.162 INFO:teuthology.orchestra.run.smithi006.stdout:This subpackage provides tools for support of all platforms. 2024-04-24T23:05:39.163 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-24T23:05:39.163 INFO:teuthology.task.kernel:Updating grub on smithi006 to boot 6.9.0-rc3-g806ec36b2112 2024-04-24T23:05:39.163 DEBUG:teuthology.orchestra.run.smithi006:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-24T23:05:39.758 INFO:teuthology.orchestra.run.smithi006.stderr:Generating grub configuration file ... 2024-04-24T23:05:41.250 INFO:teuthology.orchestra.run.smithi006.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-04-24T23:05:41.290 INFO:teuthology.orchestra.run.smithi006.stderr:done 2024-04-24T23:05:41.293 DEBUG:teuthology.orchestra.run.smithi006:> mktemp 2024-04-24T23:05:41.326 INFO:teuthology.orchestra.run.smithi006.stdout:/tmp/tmp.9zIpS4bwer 2024-04-24T23:05:41.327 DEBUG:teuthology.orchestra.run.smithi006:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.9zIpS4bwer 2024-04-24T23:05:41.392 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 0666 /tmp/tmp.9zIpS4bwer 2024-04-24T23:05:41.526 DEBUG:teuthology.orchestra.remote:smithi006:/tmp/tmp.9zIpS4bwer is 6KB 2024-04-24T23:05:41.575 DEBUG:teuthology.orchestra.run.smithi006:> rm -fr /tmp/tmp.9zIpS4bwer 2024-04-24T23:05:41.590 DEBUG:teuthology.orchestra.run.smithi006:> sudo /bin/ls /boot/loader/entries || true 2024-04-24T23:05:41.652 INFO:teuthology.orchestra.run.smithi006.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-04-24T23:05:41.652 INFO:teuthology.orchestra.run.smithi006.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-04-24T23:05:41.652 INFO:teuthology.orchestra.run.smithi006.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-04-24T23:05:41.653 INFO:teuthology.orchestra.run.smithi006.stdout:7fc8828e06d0482694bcf4a447d13cb6-6.9.0-rc3-g806ec36b2112.conf 2024-04-24T23:05:41.654 DEBUG:teuthology.orchestra.run.smithi006:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-6.9.0-rc3-g806ec36b2112 2024-04-24T23:05:41.828 DEBUG:teuthology.orchestra.run.smithi006:> sudo shutdown -r now 2024-04-24T23:06:11.832 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-04-24T23:06:11.833 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi006.front.sepia.ceph.com' 2024-04-24T23:06:11.834 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi006.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-24T23:07:11.868 DEBUG:teuthology.orchestra.remote:timed out 2024-04-24T23:07:20.869 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi006.front.sepia.ceph.com' 2024-04-24T23:07:20.870 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi006.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-24T23:07:21.521 DEBUG:teuthology.orchestra.run.smithi006:> true 2024-04-24T23:07:22.143 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi006.front.sepia.ceph.com' 2024-04-24T23:07:22.143 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "6.9.0-rc3-g806ec36b2112"... 2024-04-24T23:07:22.143 DEBUG:teuthology.orchestra.run.smithi006:> uname -r 2024-04-24T23:07:22.183 INFO:teuthology.orchestra.run.smithi006.stdout:6.9.0-rc3-g806ec36b2112 2024-04-24T23:07:22.183 DEBUG:teuthology.task.kernel:current kernel version is 6.9.0-rc3-g806ec36b2112 vs 6.9.0-rc3-g806ec36b2112 2024-04-24T23:07:22.183 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-24T23:07:22.183 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-24T23:07:23.186 DEBUG:teuthology.parallel:result is None 2024-04-24T23:07:23.186 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-24T23:07:23.193 INFO:teuthology.task.internal:Creating test directory... 2024-04-24T23:07:23.193 DEBUG:teuthology.orchestra.run.smithi006:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-24T23:07:23.200 DEBUG:teuthology.orchestra.run.smithi190:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-24T23:07:23.239 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-24T23:07:23.257 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-24T23:07:23.289 INFO:teuthology.task.internal:Creating archive directory... 2024-04-24T23:07:23.289 DEBUG:teuthology.orchestra.run.smithi006:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-24T23:07:23.294 DEBUG:teuthology.orchestra.run.smithi190:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-24T23:07:23.391 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-24T23:07:23.397 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-24T23:07:23.397 DEBUG:teuthology.orchestra.run.smithi190:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-24T23:07:23.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T23:07:23.413 DEBUG:teuthology.orchestra.run.smithi006:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-24T23:07:23.424 DEBUG:teuthology.orchestra.run.smithi190:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-24T23:07:23.498 INFO:teuthology.orchestra.run.smithi190.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-24T23:07:23.528 INFO:teuthology.orchestra.run.smithi190.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-24T23:07:23.588 INFO:teuthology.orchestra.run.smithi006.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-24T23:07:23.653 INFO:teuthology.orchestra.run.smithi006.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-24T23:07:23.658 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-24T23:07:23.665 INFO:teuthology.task.internal:Configuring sudo... 2024-04-24T23:07:23.666 DEBUG:teuthology.orchestra.run.smithi006:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-24T23:07:23.670 DEBUG:teuthology.orchestra.run.smithi190:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-24T23:07:23.737 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-24T23:07:23.745 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-24T23:07:23.746 DEBUG:teuthology.orchestra.run.smithi006:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-24T23:07:23.749 DEBUG:teuthology.orchestra.run.smithi190:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-24T23:07:23.784 DEBUG:teuthology.orchestra.run.smithi006:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-24T23:07:23.898 DEBUG:teuthology.orchestra.run.smithi006:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-24T23:07:23.974 DEBUG:teuthology.orchestra.run.smithi006:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-24T23:07:24.051 DEBUG:teuthology.orchestra.run.smithi006:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-24T23:07:24.124 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-04-24T23:07:24.124 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-24T23:07:24.206 DEBUG:teuthology.orchestra.run.smithi190:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-24T23:07:24.260 DEBUG:teuthology.orchestra.run.smithi190:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-24T23:07:24.301 DEBUG:teuthology.orchestra.run.smithi190:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-24T23:07:24.394 DEBUG:teuthology.orchestra.run.smithi190:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-24T23:07:24.429 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-04-24T23:07:24.429 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-24T23:07:24.496 DEBUG:teuthology.orchestra.run.smithi006:> sudo service rsyslog restart 2024-04-24T23:07:24.504 DEBUG:teuthology.orchestra.run.smithi190:> sudo service rsyslog restart 2024-04-24T23:07:24.582 INFO:teuthology.orchestra.run.smithi190.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-24T23:07:24.599 INFO:teuthology.orchestra.run.smithi006.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-24T23:07:24.938 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-24T23:07:24.945 INFO:teuthology.task.internal:Starting timer... 2024-04-24T23:07:24.946 INFO:teuthology.run_tasks:Running task pcp... 2024-04-24T23:07:24.989 INFO:teuthology.run_tasks:Running task selinux... 2024-04-24T23:07:25.022 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:getty_t:s0']} 2024-04-24T23:07:25.023 DEBUG:teuthology.orchestra.run.smithi006:> sudo service auditd rotate 2024-04-24T23:07:25.145 INFO:teuthology.orchestra.run.smithi006.stdout:Rotating logs: 2024-04-24T23:07:25.150 DEBUG:teuthology.orchestra.run.smithi190:> sudo service auditd rotate 2024-04-24T23:07:25.210 INFO:teuthology.orchestra.run.smithi190.stdout:Rotating logs: 2024-04-24T23:07:25.212 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-24T23:07:25.212 DEBUG:teuthology.orchestra.run.smithi006:> /usr/sbin/getenforce 2024-04-24T23:07:25.265 INFO:teuthology.orchestra.run.smithi006.stdout:Permissive 2024-04-24T23:07:25.265 DEBUG:teuthology.orchestra.run.smithi190:> /usr/sbin/getenforce 2024-04-24T23:07:25.285 INFO:teuthology.orchestra.run.smithi190.stdout:Permissive 2024-04-24T23:07:25.285 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi006.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi190.front.sepia.ceph.com': 'permissive'} 2024-04-24T23:07:25.285 DEBUG:teuthology.orchestra.run.smithi006:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-24T23:07:25.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T23:07:25.362 DEBUG:teuthology.orchestra.run.smithi190:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-24T23:07:25.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T23:07:25.387 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-24T23:07:25.387 DEBUG:teuthology.orchestra.run.smithi006:> sudo /usr/sbin/setenforce permissive 2024-04-24T23:07:25.468 DEBUG:teuthology.orchestra.run.smithi190:> sudo /usr/sbin/setenforce permissive 2024-04-24T23:07:25.493 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-24T23:07:25.503 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-04-24T23:07:25.508 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-04-24T23:07:25.653 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-24T23:07:25.669 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-24T23:07:25.670 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi006.front.sepia.ceph.com,smithi190.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-24T23:12:36.625 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi006.front.sepia.ceph.com'), Remote(name='ubuntu@smithi190.front.sepia.ceph.com')] 2024-04-24T23:12:36.626 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi006.front.sepia.ceph.com' 2024-04-24T23:12:36.627 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi006.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-24T23:12:36.748 DEBUG:teuthology.orchestra.run.smithi006:> true 2024-04-24T23:12:36.901 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi006.front.sepia.ceph.com' 2024-04-24T23:12:36.902 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi190.front.sepia.ceph.com' 2024-04-24T23:12:36.902 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi190.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-24T23:12:36.999 DEBUG:teuthology.orchestra.run.smithi190:> true 2024-04-24T23:12:37.077 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi190.front.sepia.ceph.com' 2024-04-24T23:12:37.077 INFO:teuthology.run_tasks:Running task clock... 2024-04-24T23:12:37.088 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-24T23:12:37.088 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-24T23:12:37.089 DEBUG:teuthology.orchestra.run.smithi006:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-24T23:12:37.097 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-24T23:12:37.097 DEBUG:teuthology.orchestra.run.smithi190:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-24T23:12:37.145 INFO:teuthology.orchestra.run.smithi190.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-24T23:12:37.159 INFO:teuthology.orchestra.run.smithi190.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-24T23:12:37.181 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: ntpd: command not found 2024-04-24T23:12:37.187 INFO:teuthology.orchestra.run.smithi006.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-24T23:12:37.191 INFO:teuthology.orchestra.run.smithi190.stdout:506 Cannot talk to daemon 2024-04-24T23:12:37.204 INFO:teuthology.orchestra.run.smithi190.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-24T23:12:37.217 INFO:teuthology.orchestra.run.smithi190.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-24T23:12:37.230 INFO:teuthology.orchestra.run.smithi006.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-24T23:12:37.269 INFO:teuthology.orchestra.run.smithi190.stderr:bash: line 1: ntpq: command not found 2024-04-24T23:12:37.272 INFO:teuthology.orchestra.run.smithi190.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-24T23:12:37.272 INFO:teuthology.orchestra.run.smithi190.stdout:=============================================================================== 2024-04-24T23:12:37.272 INFO:teuthology.orchestra.run.smithi190.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-24T23:12:37.273 INFO:teuthology.orchestra.run.smithi190.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-24T23:12:37.273 INFO:teuthology.orchestra.run.smithi190.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-24T23:12:37.273 INFO:teuthology.orchestra.run.smithi190.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-24T23:12:37.304 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: ntpd: command not found 2024-04-24T23:12:37.338 INFO:teuthology.orchestra.run.smithi006.stdout:506 Cannot talk to daemon 2024-04-24T23:12:37.379 INFO:teuthology.orchestra.run.smithi006.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-24T23:12:37.418 INFO:teuthology.orchestra.run.smithi006.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-24T23:12:37.555 INFO:teuthology.orchestra.run.smithi006.stderr:bash: line 1: ntpq: command not found 2024-04-24T23:12:37.564 INFO:teuthology.orchestra.run.smithi006.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-24T23:12:37.564 INFO:teuthology.orchestra.run.smithi006.stdout:=============================================================================== 2024-04-24T23:12:37.564 INFO:teuthology.orchestra.run.smithi006.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-24T23:12:37.564 INFO:teuthology.orchestra.run.smithi006.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-24T23:12:37.564 INFO:teuthology.orchestra.run.smithi006.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-24T23:12:37.564 INFO:teuthology.orchestra.run.smithi006.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-24T23:12:37.566 INFO:teuthology.run_tasks:Running task install... 2024-04-24T23:12:37.575 DEBUG:teuthology.task.install:project ceph 2024-04-24T23:12:37.575 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'd2c45d977142e024a59b68a8f1dce8f0b9840551'}} 2024-04-24T23:12:37.575 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'd2c45d977142e024a59b68a8f1dce8f0b9840551'} 2024-04-24T23:12:37.575 INFO:teuthology.task.install:Using flavor: default 2024-04-24T23:12:37.581 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-24T23:12:37.581 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-04-24T23:12:37.582 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd2c45d977142e024a59b68a8f1dce8f0b9840551', 'tag': None, 'wait_for_package': False} 2024-04-24T23:12:37.582 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-24T23:12:37.584 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd2c45d977142e024a59b68a8f1dce8f0b9840551', 'tag': None, 'wait_for_package': False} 2024-04-24T23:12:37.585 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-24T23:12:37.832 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/flavors/default/ 2024-04-24T23:12:37.832 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2445.gd2c45d97 2024-04-24T23:12:37.846 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/flavors/default/ 2024-04-24T23:12:37.846 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2445.gd2c45d97 2024-04-24T23:12:37.957 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-24T23:12:37.958 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-04-24T23:12:37.958 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-24T23:12:37.984 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-24T23:12:37.984 DEBUG:teuthology.orchestra.run.smithi190:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d2c45d977142e024a59b68a8f1dce8f0b9840551/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-24T23:12:38.023 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-24T23:12:38.023 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-04-24T23:12:38.023 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-24T23:12:38.050 DEBUG:teuthology.orchestra.run.smithi190:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-24T23:12:38.120 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-24T23:12:38.120 DEBUG:teuthology.orchestra.run.smithi006:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d2c45d977142e024a59b68a8f1dce8f0b9840551/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-24T23:12:38.124 DEBUG:teuthology.orchestra.run.smithi190:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-24T23:12:38.187 INFO:teuthology.orchestra.run.smithi190.stdout:check_obsoletes = 1 2024-04-24T23:12:38.189 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum clean all 2024-04-24T23:12:38.254 DEBUG:teuthology.orchestra.run.smithi006:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-24T23:12:38.368 DEBUG:teuthology.orchestra.run.smithi006:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-24T23:12:38.412 INFO:teuthology.orchestra.run.smithi190.stdout:48 files removed 2024-04-24T23:12:38.439 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-24T23:12:38.448 INFO:teuthology.orchestra.run.smithi006.stdout:check_obsoletes = 1 2024-04-24T23:12:38.453 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum clean all 2024-04-24T23:12:38.789 INFO:teuthology.orchestra.run.smithi006.stdout:48 files removed 2024-04-24T23:12:38.828 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-24T23:12:39.116 INFO:teuthology.orchestra.run.smithi190.stdout:ceph packages for x86_64 200 kB/s | 83 kB 00:00 2024-04-24T23:12:39.550 INFO:teuthology.orchestra.run.smithi190.stdout:ceph noarch packages 29 kB/s | 11 kB 00:00 2024-04-24T23:12:39.741 INFO:teuthology.orchestra.run.smithi006.stdout:ceph packages for x86_64 178 kB/s | 83 kB 00:00 2024-04-24T23:12:39.917 INFO:teuthology.orchestra.run.smithi190.stdout:ceph source packages 5.7 kB/s | 1.9 kB 00:00 2024-04-24T23:12:40.317 INFO:teuthology.orchestra.run.smithi006.stdout:ceph noarch packages 24 kB/s | 11 kB 00:00 2024-04-24T23:12:40.859 INFO:teuthology.orchestra.run.smithi190.stdout:CentOS Stream 9 - BaseOS 8.9 MB/s | 8.0 MB 00:00 2024-04-24T23:12:40.893 INFO:teuthology.orchestra.run.smithi006.stdout:ceph source packages 3.9 kB/s | 1.9 kB 00:00 2024-04-24T23:12:41.732 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.0 MB 00:00 2024-04-24T23:12:43.858 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS Stream 9 - AppStream 21 MB/s | 19 MB 00:00 2024-04-24T23:12:48.526 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS Stream 9 - CRB 9.0 MB/s | 6.2 MB 00:00 2024-04-24T23:12:50.235 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS Stream 9 - Extras packages 45 kB/s | 16 kB 00:00 2024-04-24T23:12:52.417 INFO:teuthology.orchestra.run.smithi006.stdout:Extra Packages for Enterprise Linux 11 MB/s | 21 MB 00:01 2024-04-24T23:12:58.317 INFO:teuthology.orchestra.run.smithi190.stdout:CentOS Stream 9 - AppStream 1.2 MB/s | 19 MB 00:16 2024-04-24T23:12:58.547 INFO:teuthology.orchestra.run.smithi006.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-04-24T23:13:01.147 INFO:teuthology.orchestra.run.smithi006.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-24T23:13:01.148 INFO:teuthology.orchestra.run.smithi006.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-24T23:13:01.170 INFO:teuthology.orchestra.run.smithi006.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-24T23:13:01.171 INFO:teuthology.orchestra.run.smithi006.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-24T23:13:01.172 INFO:teuthology.orchestra.run.smithi006.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-24T23:13:01.174 INFO:teuthology.orchestra.run.smithi006.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-24T23:13:01.176 INFO:teuthology.orchestra.run.smithi006.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-24T23:13:01.180 INFO:teuthology.orchestra.run.smithi006.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-24T23:13:01.181 INFO:teuthology.orchestra.run.smithi006.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-24T23:13:01.258 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-04-24T23:13:01.273 INFO:teuthology.orchestra.run.smithi006.stdout:======================================================================================= 2024-04-24T23:13:01.273 INFO:teuthology.orchestra.run.smithi006.stdout: Package Arch Version Repository Size 2024-04-24T23:13:01.273 INFO:teuthology.orchestra.run.smithi006.stdout:======================================================================================= 2024-04-24T23:13:01.273 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2024-04-24T23:13:01.274 INFO:teuthology.orchestra.run.smithi006.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-24T23:13:01.274 INFO:teuthology.orchestra.run.smithi006.stdout: ceph x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 6.6 k 2024-04-24T23:13:01.274 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-base x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 5.4 M 2024-04-24T23:13:01.274 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-fuse x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 890 k 2024-04-24T23:13:01.274 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 145 k 2024-04-24T23:13:01.274 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 1.5 M 2024-04-24T23:13:01.274 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 141 k 2024-04-24T23:13:01.274 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 3.5 M 2024-04-24T23:13:01.274 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 7.4 M 2024-04-24T23:13:01.274 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-rook noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 49 k 2024-04-24T23:13:01.274 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-radosgw x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 10 M 2024-04-24T23:13:01.275 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-test x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 48 M 2024-04-24T23:13:01.275 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-volume noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 264 k 2024-04-24T23:13:01.275 INFO:teuthology.orchestra.run.smithi006.stdout: cephadm noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 762 k 2024-04-24T23:13:01.275 INFO:teuthology.orchestra.run.smithi006.stdout: cephfs-mirror x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 222 k 2024-04-24T23:13:01.275 INFO:teuthology.orchestra.run.smithi006.stdout: cephfs-shell x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 27 k 2024-04-24T23:13:01.275 INFO:teuthology.orchestra.run.smithi006.stdout: cephfs-top noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 18 k 2024-04-24T23:13:01.275 INFO:teuthology.orchestra.run.smithi006.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-24T23:13:01.275 INFO:teuthology.orchestra.run.smithi006.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-24T23:13:01.275 INFO:teuthology.orchestra.run.smithi006.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-24T23:13:01.275 INFO:teuthology.orchestra.run.smithi006.stdout: libcephfs-devel x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 32 k 2024-04-24T23:13:01.275 INFO:teuthology.orchestra.run.smithi006.stdout: libcephfs2 x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 745 k 2024-04-24T23:13:01.276 INFO:teuthology.orchestra.run.smithi006.stdout: librados-devel x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 127 k 2024-04-24T23:13:01.276 INFO:teuthology.orchestra.run.smithi006.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-24T23:13:01.276 INFO:teuthology.orchestra.run.smithi006.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-24T23:13:01.276 INFO:teuthology.orchestra.run.smithi006.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-04-24T23:13:01.276 INFO:teuthology.orchestra.run.smithi006.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-24T23:13:01.276 INFO:teuthology.orchestra.run.smithi006.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-24T23:13:01.276 INFO:teuthology.orchestra.run.smithi006.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-24T23:13:01.276 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cephfs x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 162 k 2024-04-24T23:13:01.276 INFO:teuthology.orchestra.run.smithi006.stdout: python3-rados x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 321 k 2024-04-24T23:13:01.276 INFO:teuthology.orchestra.run.smithi006.stdout: python3-rbd x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 299 k 2024-04-24T23:13:01.276 INFO:teuthology.orchestra.run.smithi006.stdout: python3-rgw x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 99 k 2024-04-24T23:13:01.277 INFO:teuthology.orchestra.run.smithi006.stdout: rbd-fuse x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 85 k 2024-04-24T23:13:01.277 INFO:teuthology.orchestra.run.smithi006.stdout: rbd-mirror x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 3.1 M 2024-04-24T23:13:01.277 INFO:teuthology.orchestra.run.smithi006.stdout: rbd-nbd x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 171 k 2024-04-24T23:13:01.277 INFO:teuthology.orchestra.run.smithi006.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-24T23:13:01.277 INFO:teuthology.orchestra.run.smithi006.stdout:Upgrading: 2024-04-24T23:13:01.277 INFO:teuthology.orchestra.run.smithi006.stdout: librados2 x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 3.4 M 2024-04-24T23:13:01.277 INFO:teuthology.orchestra.run.smithi006.stdout: librbd1 x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 3.1 M 2024-04-24T23:13:01.277 INFO:teuthology.orchestra.run.smithi006.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-04-24T23:13:01.277 INFO:teuthology.orchestra.run.smithi006.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-04-24T23:13:01.277 INFO:teuthology.orchestra.run.smithi006.stdout:Installing dependencies: 2024-04-24T23:13:01.277 INFO:teuthology.orchestra.run.smithi006.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-24T23:13:01.278 INFO:teuthology.orchestra.run.smithi006.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-24T23:13:01.278 INFO:teuthology.orchestra.run.smithi006.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-24T23:13:01.278 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-common x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 21 M 2024-04-24T23:13:01.278 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 32 k 2024-04-24T23:13:01.278 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mds x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 2.4 M 2024-04-24T23:13:01.278 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 249 k 2024-04-24T23:13:01.278 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mon x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 4.7 M 2024-04-24T23:13:01.278 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-osd x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 17 M 2024-04-24T23:13:01.278 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 15 k 2024-04-24T23:13:01.278 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-selinux x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 25 k 2024-04-24T23:13:01.278 INFO:teuthology.orchestra.run.smithi006.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-24T23:13:01.278 INFO:teuthology.orchestra.run.smithi006.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-24T23:13:01.279 INFO:teuthology.orchestra.run.smithi006.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-24T23:13:01.279 INFO:teuthology.orchestra.run.smithi006.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-24T23:13:01.279 INFO:teuthology.orchestra.run.smithi006.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-24T23:13:01.279 INFO:teuthology.orchestra.run.smithi006.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-24T23:13:01.279 INFO:teuthology.orchestra.run.smithi006.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-24T23:13:01.279 INFO:teuthology.orchestra.run.smithi006.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-24T23:13:01.279 INFO:teuthology.orchestra.run.smithi006.stdout: libcephsqlite x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 163 k 2024-04-24T23:13:01.279 INFO:teuthology.orchestra.run.smithi006.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-24T23:13:01.279 INFO:teuthology.orchestra.run.smithi006.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-24T23:13:01.279 INFO:teuthology.orchestra.run.smithi006.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-24T23:13:01.279 INFO:teuthology.orchestra.run.smithi006.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-24T23:13:01.280 INFO:teuthology.orchestra.run.smithi006.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-24T23:13:01.280 INFO:teuthology.orchestra.run.smithi006.stdout: libradosstriper1 x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 501 k 2024-04-24T23:13:01.280 INFO:teuthology.orchestra.run.smithi006.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-24T23:13:01.280 INFO:teuthology.orchestra.run.smithi006.stdout: librgw2 x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 5.3 M 2024-04-24T23:13:01.280 INFO:teuthology.orchestra.run.smithi006.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-24T23:13:01.280 INFO:teuthology.orchestra.run.smithi006.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-24T23:13:01.280 INFO:teuthology.orchestra.run.smithi006.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-24T23:13:01.280 INFO:teuthology.orchestra.run.smithi006.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-24T23:13:01.280 INFO:teuthology.orchestra.run.smithi006.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-24T23:13:01.280 INFO:teuthology.orchestra.run.smithi006.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-24T23:13:01.280 INFO:teuthology.orchestra.run.smithi006.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-24T23:13:01.280 INFO:teuthology.orchestra.run.smithi006.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-24T23:13:01.281 INFO:teuthology.orchestra.run.smithi006.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-24T23:13:01.281 INFO:teuthology.orchestra.run.smithi006.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-24T23:13:01.281 INFO:teuthology.orchestra.run.smithi006.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-24T23:13:01.281 INFO:teuthology.orchestra.run.smithi006.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-24T23:13:01.281 INFO:teuthology.orchestra.run.smithi006.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-24T23:13:01.281 INFO:teuthology.orchestra.run.smithi006.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-24T23:13:01.281 INFO:teuthology.orchestra.run.smithi006.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-24T23:13:01.281 INFO:teuthology.orchestra.run.smithi006.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-24T23:13:01.281 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-24T23:13:01.281 INFO:teuthology.orchestra.run.smithi006.stdout: python3-ceph-argparse x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 45 k 2024-04-24T23:13:01.281 INFO:teuthology.orchestra.run.smithi006.stdout: python3-ceph-common x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 133 k 2024-04-24T23:13:01.281 INFO:teuthology.orchestra.run.smithi006.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-24T23:13:01.282 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-24T23:13:01.282 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-24T23:13:01.282 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-24T23:13:01.282 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-24T23:13:01.282 INFO:teuthology.orchestra.run.smithi006.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-24T23:13:01.282 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-24T23:13:01.282 INFO:teuthology.orchestra.run.smithi006.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-24T23:13:01.282 INFO:teuthology.orchestra.run.smithi006.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-24T23:13:01.282 INFO:teuthology.orchestra.run.smithi006.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-24T23:13:01.282 INFO:teuthology.orchestra.run.smithi006.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-24T23:13:01.282 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-24T23:13:01.283 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-24T23:13:01.283 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-24T23:13:01.283 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-24T23:13:01.283 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-24T23:13:01.283 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-24T23:13:01.283 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-24T23:13:01.283 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-24T23:13:01.283 INFO:teuthology.orchestra.run.smithi006.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-24T23:13:01.283 INFO:teuthology.orchestra.run.smithi006.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-24T23:13:01.283 INFO:teuthology.orchestra.run.smithi006.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-24T23:13:01.283 INFO:teuthology.orchestra.run.smithi006.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-24T23:13:01.283 INFO:teuthology.orchestra.run.smithi006.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-24T23:13:01.284 INFO:teuthology.orchestra.run.smithi006.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-24T23:13:01.284 INFO:teuthology.orchestra.run.smithi006.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-24T23:13:01.284 INFO:teuthology.orchestra.run.smithi006.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-24T23:13:01.284 INFO:teuthology.orchestra.run.smithi006.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-24T23:13:01.284 INFO:teuthology.orchestra.run.smithi006.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-24T23:13:01.284 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-24T23:13:01.284 INFO:teuthology.orchestra.run.smithi006.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-24T23:13:01.284 INFO:teuthology.orchestra.run.smithi006.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-24T23:13:01.284 INFO:teuthology.orchestra.run.smithi006.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-24T23:13:01.284 INFO:teuthology.orchestra.run.smithi006.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-24T23:13:01.284 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-24T23:13:01.284 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-24T23:13:01.285 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-24T23:13:01.285 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-24T23:13:01.285 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-24T23:13:01.285 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-24T23:13:01.285 INFO:teuthology.orchestra.run.smithi006.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-24T23:13:01.285 INFO:teuthology.orchestra.run.smithi006.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-24T23:13:01.285 INFO:teuthology.orchestra.run.smithi006.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-24T23:13:01.285 INFO:teuthology.orchestra.run.smithi006.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-24T23:13:01.285 INFO:teuthology.orchestra.run.smithi006.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-24T23:13:01.286 INFO:teuthology.orchestra.run.smithi006.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-24T23:13:01.286 INFO:teuthology.orchestra.run.smithi006.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-24T23:13:01.286 INFO:teuthology.orchestra.run.smithi006.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-24T23:13:01.286 INFO:teuthology.orchestra.run.smithi006.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-24T23:13:01.286 INFO:teuthology.orchestra.run.smithi006.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-24T23:13:01.286 INFO:teuthology.orchestra.run.smithi006.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-24T23:13:01.286 INFO:teuthology.orchestra.run.smithi006.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-24T23:13:01.286 INFO:teuthology.orchestra.run.smithi006.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-24T23:13:01.286 INFO:teuthology.orchestra.run.smithi006.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-24T23:13:01.287 INFO:teuthology.orchestra.run.smithi006.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-24T23:13:01.287 INFO:teuthology.orchestra.run.smithi006.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-24T23:13:01.287 INFO:teuthology.orchestra.run.smithi006.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-24T23:13:01.287 INFO:teuthology.orchestra.run.smithi006.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-24T23:13:01.287 INFO:teuthology.orchestra.run.smithi006.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-24T23:13:01.287 INFO:teuthology.orchestra.run.smithi006.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-24T23:13:01.287 INFO:teuthology.orchestra.run.smithi006.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-24T23:13:01.287 INFO:teuthology.orchestra.run.smithi006.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-24T23:13:01.287 INFO:teuthology.orchestra.run.smithi006.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-24T23:13:01.288 INFO:teuthology.orchestra.run.smithi006.stdout:Installing weak dependencies: 2024-04-24T23:13:01.288 INFO:teuthology.orchestra.run.smithi006.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-24T23:13:01.288 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:13:01.288 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2024-04-24T23:13:01.288 INFO:teuthology.orchestra.run.smithi006.stdout:======================================================================================= 2024-04-24T23:13:01.288 INFO:teuthology.orchestra.run.smithi006.stdout:Install 148 Packages 2024-04-24T23:13:01.288 INFO:teuthology.orchestra.run.smithi006.stdout:Upgrade 4 Packages 2024-04-24T23:13:01.288 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:13:01.289 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 220 M 2024-04-24T23:13:01.289 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2024-04-24T23:13:02.451 INFO:teuthology.orchestra.run.smithi190.stdout:CentOS Stream 9 - CRB 11 MB/s | 6.2 MB 00:00 2024-04-24T23:13:02.595 INFO:teuthology.orchestra.run.smithi006.stdout:(1/152): ceph-19.0.0-2445.gd2c45d97.el9.x86_64. 18 kB/s | 6.6 kB 00:00 2024-04-24T23:13:02.914 INFO:teuthology.orchestra.run.smithi006.stdout:(2/152): ceph-base-19.0.0-2445.gd2c45d97.el9.x8 7.9 MB/s | 5.4 MB 00:00 2024-04-24T23:13:02.964 INFO:teuthology.orchestra.run.smithi006.stdout:(3/152): ceph-immutable-object-cache-19.0.0-244 2.9 MB/s | 145 kB 00:00 2024-04-24T23:13:03.173 INFO:teuthology.orchestra.run.smithi006.stdout:(4/152): ceph-mds-19.0.0-2445.gd2c45d97.el9.x86 11 MB/s | 2.4 MB 00:00 2024-04-24T23:13:03.215 INFO:teuthology.orchestra.run.smithi006.stdout:(5/152): ceph-fuse-19.0.0-2445.gd2c45d97.el9.x8 1.4 MB/s | 890 kB 00:00 2024-04-24T23:13:03.306 INFO:teuthology.orchestra.run.smithi006.stdout:(6/152): ceph-mgr-19.0.0-2445.gd2c45d97.el9.x86 11 MB/s | 1.5 MB 00:00 2024-04-24T23:13:04.294 INFO:teuthology.orchestra.run.smithi190.stdout:CentOS Stream 9 - Extras packages 26 kB/s | 16 kB 00:00 2024-04-24T23:13:04.315 INFO:teuthology.orchestra.run.smithi006.stdout:(7/152): ceph-osd-19.0.0-2445.gd2c45d97.el9.x86 17 MB/s | 17 MB 00:01 2024-04-24T23:13:04.907 INFO:teuthology.orchestra.run.smithi006.stdout:(8/152): ceph-mon-19.0.0-2445.gd2c45d97.el9.x86 2.8 MB/s | 4.7 MB 00:01 2024-04-24T23:13:05.091 INFO:teuthology.orchestra.run.smithi006.stdout:(9/152): ceph-radosgw-19.0.0-2445.gd2c45d97.el9 13 MB/s | 10 MB 00:00 2024-04-24T23:13:05.116 INFO:teuthology.orchestra.run.smithi006.stdout:(10/152): ceph-selinux-19.0.0-2445.gd2c45d97.el 121 kB/s | 25 kB 00:00 2024-04-24T23:13:05.220 INFO:teuthology.orchestra.run.smithi190.stdout:Extra Packages for Enterprise Linux 26 MB/s | 21 MB 00:00 2024-04-24T23:13:05.224 INFO:teuthology.orchestra.run.smithi006.stdout:(11/152): cephfs-mirror-19.0.0-2445.gd2c45d97.e 2.0 MB/s | 222 kB 00:00 2024-04-24T23:13:05.553 INFO:teuthology.orchestra.run.smithi006.stdout:(12/152): ceph-common-19.0.0-2445.gd2c45d97.el9 6.3 MB/s | 21 MB 00:03 2024-04-24T23:13:05.583 INFO:teuthology.orchestra.run.smithi006.stdout:(13/152): cephfs-shell-19.0.0-2445.gd2c45d97.el 76 kB/s | 27 kB 00:00 2024-04-24T23:13:05.617 INFO:teuthology.orchestra.run.smithi006.stdout:(14/152): libcephfs-devel-19.0.0-2445.gd2c45d97 507 kB/s | 32 kB 00:00 2024-04-24T23:13:05.692 INFO:teuthology.orchestra.run.smithi006.stdout:(15/152): libcephsqlite-19.0.0-2445.gd2c45d97.e 2.2 MB/s | 163 kB 00:00 2024-04-24T23:13:05.768 INFO:teuthology.orchestra.run.smithi006.stdout:(16/152): libcephfs2-19.0.0-2445.gd2c45d97.el9. 4.0 MB/s | 745 kB 00:00 2024-04-24T23:13:05.801 INFO:teuthology.orchestra.run.smithi006.stdout:(17/152): librados-devel-19.0.0-2445.gd2c45d97. 1.1 MB/s | 127 kB 00:00 2024-04-24T23:13:05.901 INFO:teuthology.orchestra.run.smithi006.stdout:(18/152): libradosstriper1-19.0.0-2445.gd2c45d9 3.7 MB/s | 501 kB 00:00 2024-04-24T23:13:06.010 INFO:teuthology.orchestra.run.smithi006.stdout:(19/152): python3-ceph-argparse-19.0.0-2445.gd2 419 kB/s | 45 kB 00:00 2024-04-24T23:13:06.127 INFO:teuthology.orchestra.run.smithi006.stdout:(20/152): python3-ceph-common-19.0.0-2445.gd2c4 1.1 MB/s | 133 kB 00:00 2024-04-24T23:13:06.244 INFO:teuthology.orchestra.run.smithi006.stdout:(21/152): python3-cephfs-19.0.0-2445.gd2c45d97. 1.4 MB/s | 162 kB 00:00 2024-04-24T23:13:06.369 INFO:teuthology.orchestra.run.smithi006.stdout:(22/152): librgw2-19.0.0-2445.gd2c45d97.el9.x86 9.4 MB/s | 5.3 MB 00:00 2024-04-24T23:13:06.411 INFO:teuthology.orchestra.run.smithi006.stdout:(23/152): python3-rados-19.0.0-2445.gd2c45d97.e 1.9 MB/s | 321 kB 00:00 2024-04-24T23:13:06.453 INFO:teuthology.orchestra.run.smithi006.stdout:(24/152): python3-rbd-19.0.0-2445.gd2c45d97.el9 3.5 MB/s | 299 kB 00:00 2024-04-24T23:13:06.487 INFO:teuthology.orchestra.run.smithi006.stdout:(25/152): python3-rgw-19.0.0-2445.gd2c45d97.el9 1.3 MB/s | 99 kB 00:00 2024-04-24T23:13:06.545 INFO:teuthology.orchestra.run.smithi006.stdout:(26/152): rbd-fuse-19.0.0-2445.gd2c45d97.el9.x8 939 kB/s | 85 kB 00:00 2024-04-24T23:13:06.621 INFO:teuthology.orchestra.run.smithi006.stdout:(27/152): rbd-nbd-19.0.0-2445.gd2c45d97.el9.x86 2.3 MB/s | 171 kB 00:00 2024-04-24T23:13:06.688 INFO:teuthology.orchestra.run.smithi006.stdout:(28/152): ceph-grafana-dashboards-19.0.0-2445.g 483 kB/s | 32 kB 00:00 2024-04-24T23:13:06.788 INFO:teuthology.orchestra.run.smithi006.stdout:(29/152): ceph-mgr-cephadm-19.0.0-2445.gd2c45d9 1.4 MB/s | 141 kB 00:00 2024-04-24T23:13:07.130 INFO:teuthology.orchestra.run.smithi006.stdout:(30/152): rbd-mirror-19.0.0-2445.gd2c45d97.el9. 4.8 MB/s | 3.1 MB 00:00 2024-04-24T23:13:07.238 INFO:teuthology.orchestra.run.smithi006.stdout:(31/152): ceph-mgr-dashboard-19.0.0-2445.gd2c45 7.8 MB/s | 3.5 MB 00:00 2024-04-24T23:13:07.322 INFO:teuthology.orchestra.run.smithi006.stdout:(32/152): ceph-mgr-modules-core-19.0.0-2445.gd2 3.0 MB/s | 249 kB 00:00 2024-04-24T23:13:07.406 INFO:teuthology.orchestra.run.smithi006.stdout:(33/152): ceph-mgr-rook-19.0.0-2445.gd2c45d97.e 596 kB/s | 49 kB 00:00 2024-04-24T23:13:07.506 INFO:teuthology.orchestra.run.smithi006.stdout:(34/152): ceph-prometheus-alerts-19.0.0-2445.gd 154 kB/s | 15 kB 00:00 2024-04-24T23:13:07.589 INFO:teuthology.orchestra.run.smithi006.stdout:(35/152): ceph-volume-19.0.0-2445.gd2c45d97.el9 3.1 MB/s | 264 kB 00:00 2024-04-24T23:13:07.698 INFO:teuthology.orchestra.run.smithi006.stdout:(36/152): cephadm-19.0.0-2445.gd2c45d97.el9.noa 6.9 MB/s | 762 kB 00:00 2024-04-24T23:13:07.782 INFO:teuthology.orchestra.run.smithi006.stdout:(37/152): cephfs-top-19.0.0-2445.gd2c45d97.el9. 221 kB/s | 18 kB 00:00 2024-04-24T23:13:08.357 INFO:teuthology.orchestra.run.smithi006.stdout:(38/152): ceph-test-19.0.0-2445.gd2c45d97.el9.x 15 MB/s | 48 MB 00:03 2024-04-24T23:13:08.391 INFO:teuthology.orchestra.run.smithi006.stdout:(39/152): fuse-2.9.9-15.el9.x86_64.rpm 132 kB/s | 80 kB 00:00 2024-04-24T23:13:08.574 INFO:teuthology.orchestra.run.smithi006.stdout:(40/152): ceph-mgr-diskprediction-local-19.0.0- 5.1 MB/s | 7.4 MB 00:01 2024-04-24T23:13:08.816 INFO:teuthology.orchestra.run.smithi006.stdout:(41/152): libquadmath-11.4.1-3.el9.x86_64.rpm 460 kB/s | 195 kB 00:00 2024-04-24T23:13:09.117 INFO:teuthology.orchestra.run.smithi006.stdout:(42/152): libgfortran-11.4.1-3.el9.x86_64.rpm 1.0 MB/s | 803 kB 00:00 2024-04-24T23:13:09.158 INFO:teuthology.orchestra.run.smithi006.stdout:(43/152): python3-cffi-1.14.5-5.el9.x86_64.rpm 435 kB/s | 253 kB 00:00 2024-04-24T23:13:09.231 INFO:teuthology.orchestra.run.smithi006.stdout:(44/152): python3-ply-3.11-14.el9.noarch.rpm 940 kB/s | 106 kB 00:00 2024-04-24T23:13:09.268 INFO:teuthology.orchestra.run.smithi006.stdout:(45/152): python3-pycparser-2.20-6.el9.noarch.r 1.2 MB/s | 135 kB 00:00 2024-04-24T23:13:09.343 INFO:teuthology.orchestra.run.smithi006.stdout:(46/152): python3-cryptography-36.0.1-4.el9.x86 2.3 MB/s | 1.2 MB 00:00 2024-04-24T23:13:09.368 INFO:teuthology.orchestra.run.smithi006.stdout:(47/152): python3-requests-2.25.1-8.el9.noarch. 915 kB/s | 125 kB 00:00 2024-04-24T23:13:09.477 INFO:teuthology.orchestra.run.smithi006.stdout:(48/152): python3-urllib3-1.26.5-5.el9.noarch.r 1.0 MB/s | 215 kB 00:00 2024-04-24T23:13:09.544 INFO:teuthology.orchestra.run.smithi006.stdout:(49/152): automake-1.16.2-8.el9.noarch.rpm 3.4 MB/s | 689 kB 00:00 2024-04-24T23:13:09.586 INFO:teuthology.orchestra.run.smithi006.stdout:(50/152): bison-3.7.4-5.el9.x86_64.rpm 4.3 MB/s | 943 kB 00:00 2024-04-24T23:13:09.611 INFO:teuthology.orchestra.run.smithi006.stdout:(51/152): elfutils-libelf-devel-0.190-2.el9.x86 326 kB/s | 21 kB 00:00 2024-04-24T23:13:09.636 INFO:teuthology.orchestra.run.smithi006.stdout:(52/152): boost-program-options-1.75.0-8.el9.x8 676 kB/s | 107 kB 00:00 2024-04-24T23:13:09.661 INFO:teuthology.orchestra.run.smithi006.stdout:(53/152): flexiblas-3.0.4-8.el9.x86_64.rpm 663 kB/s | 33 kB 00:00 2024-04-24T23:13:09.695 INFO:teuthology.orchestra.run.smithi006.stdout:(54/152): flex-2.6.4-9.el9.x86_64.rpm 2.8 MB/s | 313 kB 00:00 2024-04-24T23:13:09.720 INFO:teuthology.orchestra.run.smithi006.stdout:(55/152): flexiblas-openblas-openmp-3.0.4-8.el9 316 kB/s | 18 kB 00:00 2024-04-24T23:13:09.754 INFO:teuthology.orchestra.run.smithi006.stdout:(56/152): libaio-devel-0.3.111-13.el9.x86_64.rp 193 kB/s | 11 kB 00:00 2024-04-24T23:13:09.788 INFO:teuthology.orchestra.run.smithi006.stdout:(57/152): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.4 MB/s | 160 kB 00:00 2024-04-24T23:13:09.813 INFO:teuthology.orchestra.run.smithi006.stdout:(58/152): librabbitmq-0.11.0-7.el9.x86_64.rpm 784 kB/s | 45 kB 00:00 2024-04-24T23:13:09.863 INFO:teuthology.orchestra.run.smithi006.stdout:(59/152): librdkafka-1.6.1-102.el9.x86_64.rpm 8.7 MB/s | 662 kB 00:00 2024-04-24T23:13:09.913 INFO:teuthology.orchestra.run.smithi006.stdout:(60/152): libtool-2.4.6-45.el9.x86_64.rpm 5.7 MB/s | 582 kB 00:00 2024-04-24T23:13:09.939 INFO:teuthology.orchestra.run.smithi006.stdout:(61/152): libuuid-devel-2.37.4-18.el9.x86_64.rp 343 kB/s | 25 kB 00:00 2024-04-24T23:13:09.964 INFO:teuthology.orchestra.run.smithi006.stdout:(62/152): libzstd-devel-1.5.1-2.el9.x86_64.rpm 962 kB/s | 47 kB 00:00 2024-04-24T23:13:09.998 INFO:teuthology.orchestra.run.smithi006.stdout:(63/152): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.0 MB/s | 292 kB 00:00 2024-04-24T23:13:10.031 INFO:teuthology.orchestra.run.smithi006.stdout:(64/152): lua-5.4.4-4.el9.x86_64.rpm 2.8 MB/s | 188 kB 00:00 2024-04-24T23:13:10.056 INFO:teuthology.orchestra.run.smithi006.stdout:(65/152): openblas-0.3.21-2.el9.x86_64.rpm 612 kB/s | 35 kB 00:00 2024-04-24T23:13:10.149 INFO:teuthology.orchestra.run.smithi006.stdout:(66/152): flexiblas-netlib-3.0.4-8.el9.x86_64.r 5.9 MB/s | 3.0 MB 00:00 2024-04-24T23:13:10.257 INFO:teuthology.orchestra.run.smithi006.stdout:(67/152): postgresql-13.14-1.el9.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2024-04-24T23:13:10.286 INFO:teuthology.orchestra.run.smithi190.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-04-24T23:13:10.399 INFO:teuthology.orchestra.run.smithi006.stdout:(68/152): openblas-openmp-0.3.21-2.el9.x86_64.r 13 MB/s | 4.7 MB 00:00 2024-04-24T23:13:10.441 INFO:teuthology.orchestra.run.smithi006.stdout:(69/152): postgresql-contrib-13.14-1.el9.x86_64 4.9 MB/s | 911 kB 00:00 2024-04-24T23:13:10.474 INFO:teuthology.orchestra.run.smithi006.stdout:(70/152): postgresql-private-libs-13.14-1.el9.x 1.7 MB/s | 133 kB 00:00 2024-04-24T23:13:10.578 INFO:teuthology.orchestra.run.smithi006.stdout:(71/152): openssl-devel-3.2.1-1.el9.x86_64.rpm 8.6 MB/s | 4.4 MB 00:00 2024-04-24T23:13:10.617 INFO:teuthology.orchestra.run.smithi006.stdout:(72/152): python3-attrs-20.3.0-7.el9.noarch.rpm 2.1 MB/s | 87 kB 00:00 2024-04-24T23:13:10.659 INFO:teuthology.orchestra.run.smithi006.stdout:(73/152): protobuf-3.14.0-13.el9.x86_64.rpm 5.5 MB/s | 1.0 MB 00:00 2024-04-24T23:13:10.709 INFO:teuthology.orchestra.run.smithi006.stdout:(74/152): python3-devel-3.9.18-3.el9.x86_64.rpm 4.9 MB/s | 244 kB 00:00 2024-04-24T23:13:10.759 INFO:teuthology.orchestra.run.smithi006.stdout:(75/152): python3-jinja2-2.11.3-5.el9.noarch.rp 4.9 MB/s | 248 kB 00:00 2024-04-24T23:13:10.801 INFO:teuthology.orchestra.run.smithi006.stdout:(76/152): python3-jsonpatch-1.21-16.el9.noarch. 631 kB/s | 26 kB 00:00 2024-04-24T23:13:10.843 INFO:teuthology.orchestra.run.smithi006.stdout:(77/152): python3-jsonpointer-2.0-4.el9.noarch. 471 kB/s | 19 kB 00:00 2024-04-24T23:13:10.893 INFO:teuthology.orchestra.run.smithi006.stdout:(78/152): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 172 kB 00:00 2024-04-24T23:13:11.027 INFO:teuthology.orchestra.run.smithi006.stdout:(79/152): postgresql-server-13.14-1.el9.x86_64. 9.9 MB/s | 5.8 MB 00:00 2024-04-24T23:13:11.052 INFO:teuthology.orchestra.run.smithi006.stdout:(80/152): python3-markupsafe-1.1.1-12.el9.x86_6 220 kB/s | 35 kB 00:00 2024-04-24T23:13:11.102 INFO:teuthology.orchestra.run.smithi006.stdout:(81/152): python3-numpy-f2py-1.20.1-5.el9.x86_6 5.0 MB/s | 250 kB 00:00 2024-04-24T23:13:11.228 INFO:teuthology.orchestra.run.smithi006.stdout:(82/152): python3-babel-2.9.1-2.el9.noarch.rpm 9.8 MB/s | 6.0 MB 00:00 2024-04-24T23:13:11.261 INFO:teuthology.orchestra.run.smithi006.stdout:(83/152): python3-oauthlib-3.1.1-5.el9.noarch.r 1.4 MB/s | 222 kB 00:00 2024-04-24T23:13:11.435 INFO:teuthology.orchestra.run.smithi006.stdout:(84/152): python3-packaging-20.9-5.el9.noarch.r 375 kB/s | 77 kB 00:00 2024-04-24T23:13:11.462 INFO:teuthology.orchestra.run.smithi006.stdout:(85/152): python3-prettytable-0.7.2-27.el9.noar 212 kB/s | 42 kB 00:00 2024-04-24T23:13:11.496 INFO:teuthology.orchestra.run.smithi006.stdout:(86/152): python3-protobuf-3.14.0-13.el9.noarch 4.4 MB/s | 269 kB 00:00 2024-04-24T23:13:11.521 INFO:teuthology.orchestra.run.smithi006.stdout:(87/152): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.7 MB/s | 159 kB 00:00 2024-04-24T23:13:11.555 INFO:teuthology.orchestra.run.smithi006.stdout:(88/152): python3-pyasn1-modules-0.4.8-6.el9.no 4.7 MB/s | 279 kB 00:00 2024-04-24T23:13:11.571 INFO:teuthology.orchestra.run.smithi006.stdout:(89/152): python3-pytz-2021.1-5.el9.noarch.rpm 1.0 MB/s | 51 kB 00:00 2024-04-24T23:13:11.588 INFO:teuthology.orchestra.run.smithi006.stdout:(90/152): python3-requests-oauthlib-1.3.0-12.el 1.6 MB/s | 54 kB 00:00 2024-04-24T23:13:11.622 INFO:teuthology.orchestra.run.smithi006.stdout:(91/152): python3-toml-0.10.2-6.el9.noarch.rpm 1.2 MB/s | 42 kB 00:00 2024-04-24T23:13:11.656 INFO:teuthology.orchestra.run.smithi006.stdout:(92/152): python3-wcwidth-0.2.5-8.el9.noarch.rp 1.3 MB/s | 44 kB 00:00 2024-04-24T23:13:11.706 INFO:teuthology.orchestra.run.smithi006.stdout:(93/152): qatlib-24.02.0-1.el9.x86_64.rpm 4.4 MB/s | 221 kB 00:00 2024-04-24T23:13:11.815 INFO:teuthology.orchestra.run.smithi006.stdout:(94/152): python3-numpy-1.20.1-5.el9.x86_64.rpm 6.5 MB/s | 5.1 MB 00:00 2024-04-24T23:13:11.831 INFO:teuthology.orchestra.run.smithi006.stdout:(95/152): qatlib-service-24.02.0-1.el9.x86_64.r 288 kB/s | 36 kB 00:00 2024-04-24T23:13:11.857 INFO:teuthology.orchestra.run.smithi006.stdout:(96/152): qatzip-libs-1.2.0-1.el9.x86_64.rpm 1.1 MB/s | 47 kB 00:00 2024-04-24T23:13:11.890 INFO:teuthology.orchestra.run.smithi006.stdout:(97/152): socat-1.7.4.1-5.el9.x86_64.rpm 5.2 MB/s | 305 kB 00:00 2024-04-24T23:13:11.916 INFO:teuthology.orchestra.run.smithi006.stdout:(98/152): uuid-1.6.2-55.el9.x86_64.rpm 1.0 MB/s | 58 kB 00:00 2024-04-24T23:13:11.933 INFO:teuthology.orchestra.run.smithi006.stdout:(99/152): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 2.2 MB/s | 94 kB 00:00 2024-04-24T23:13:11.958 INFO:teuthology.orchestra.run.smithi006.stdout:(100/152): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 64 kB 00:00 2024-04-24T23:13:11.983 INFO:teuthology.orchestra.run.smithi006.stdout:(101/152): zlib-devel-1.2.11-41.el9.x86_64.rpm 912 kB/s | 45 kB 00:00 2024-04-24T23:13:12.275 INFO:teuthology.orchestra.run.smithi006.stdout:(102/152): python3-scipy-1.6.2-8.el9.x86_64.rpm 23 MB/s | 16 MB 00:00 2024-04-24T23:13:12.300 INFO:teuthology.orchestra.run.smithi006.stdout:(103/152): lua-devel-5.4.4-4.el9.x86_64.rpm 65 kB/s | 22 kB 00:00 2024-04-24T23:13:12.404 INFO:teuthology.orchestra.run.smithi190.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-24T23:13:12.405 INFO:teuthology.orchestra.run.smithi190.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-24T23:13:12.416 INFO:teuthology.orchestra.run.smithi190.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-24T23:13:12.416 INFO:teuthology.orchestra.run.smithi190.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-24T23:13:12.417 INFO:teuthology.orchestra.run.smithi190.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-24T23:13:12.418 INFO:teuthology.orchestra.run.smithi190.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-24T23:13:12.422 INFO:teuthology.orchestra.run.smithi190.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-24T23:13:12.422 INFO:teuthology.orchestra.run.smithi190.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-24T23:13:12.422 INFO:teuthology.orchestra.run.smithi190.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-24T23:13:12.483 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-04-24T23:13:12.499 INFO:teuthology.orchestra.run.smithi190.stdout:======================================================================================= 2024-04-24T23:13:12.499 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repository Size 2024-04-24T23:13:12.499 INFO:teuthology.orchestra.run.smithi190.stdout:======================================================================================= 2024-04-24T23:13:12.499 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-04-24T23:13:12.499 INFO:teuthology.orchestra.run.smithi190.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-24T23:13:12.499 INFO:teuthology.orchestra.run.smithi190.stdout: ceph x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 6.6 k 2024-04-24T23:13:12.499 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-base x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 5.4 M 2024-04-24T23:13:12.499 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-fuse x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 890 k 2024-04-24T23:13:12.499 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 145 k 2024-04-24T23:13:12.499 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 1.5 M 2024-04-24T23:13:12.500 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 141 k 2024-04-24T23:13:12.500 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 3.5 M 2024-04-24T23:13:12.500 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 7.4 M 2024-04-24T23:13:12.500 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-rook noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 49 k 2024-04-24T23:13:12.500 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-radosgw x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 10 M 2024-04-24T23:13:12.500 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-test x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 48 M 2024-04-24T23:13:12.500 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-volume noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 264 k 2024-04-24T23:13:12.500 INFO:teuthology.orchestra.run.smithi190.stdout: cephadm noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 762 k 2024-04-24T23:13:12.500 INFO:teuthology.orchestra.run.smithi190.stdout: cephfs-mirror x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 222 k 2024-04-24T23:13:12.500 INFO:teuthology.orchestra.run.smithi190.stdout: cephfs-shell x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 27 k 2024-04-24T23:13:12.500 INFO:teuthology.orchestra.run.smithi190.stdout: cephfs-top noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 18 k 2024-04-24T23:13:12.500 INFO:teuthology.orchestra.run.smithi190.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-24T23:13:12.501 INFO:teuthology.orchestra.run.smithi190.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-24T23:13:12.501 INFO:teuthology.orchestra.run.smithi190.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-24T23:13:12.501 INFO:teuthology.orchestra.run.smithi190.stdout: libcephfs-devel x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 32 k 2024-04-24T23:13:12.501 INFO:teuthology.orchestra.run.smithi190.stdout: libcephfs2 x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 745 k 2024-04-24T23:13:12.501 INFO:teuthology.orchestra.run.smithi190.stdout: librados-devel x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 127 k 2024-04-24T23:13:12.501 INFO:teuthology.orchestra.run.smithi190.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-24T23:13:12.501 INFO:teuthology.orchestra.run.smithi190.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-24T23:13:12.501 INFO:teuthology.orchestra.run.smithi190.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-04-24T23:13:12.501 INFO:teuthology.orchestra.run.smithi190.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-24T23:13:12.501 INFO:teuthology.orchestra.run.smithi190.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-24T23:13:12.501 INFO:teuthology.orchestra.run.smithi190.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-24T23:13:12.501 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cephfs x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 162 k 2024-04-24T23:13:12.502 INFO:teuthology.orchestra.run.smithi190.stdout: python3-rados x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 321 k 2024-04-24T23:13:12.502 INFO:teuthology.orchestra.run.smithi190.stdout: python3-rbd x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 299 k 2024-04-24T23:13:12.502 INFO:teuthology.orchestra.run.smithi190.stdout: python3-rgw x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 99 k 2024-04-24T23:13:12.502 INFO:teuthology.orchestra.run.smithi190.stdout: rbd-fuse x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 85 k 2024-04-24T23:13:12.502 INFO:teuthology.orchestra.run.smithi190.stdout: rbd-mirror x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 3.1 M 2024-04-24T23:13:12.502 INFO:teuthology.orchestra.run.smithi190.stdout: rbd-nbd x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 171 k 2024-04-24T23:13:12.502 INFO:teuthology.orchestra.run.smithi190.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-24T23:13:12.502 INFO:teuthology.orchestra.run.smithi190.stdout:Upgrading: 2024-04-24T23:13:12.502 INFO:teuthology.orchestra.run.smithi190.stdout: librados2 x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 3.4 M 2024-04-24T23:13:12.502 INFO:teuthology.orchestra.run.smithi190.stdout: librbd1 x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 3.1 M 2024-04-24T23:13:12.503 INFO:teuthology.orchestra.run.smithi190.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-04-24T23:13:12.503 INFO:teuthology.orchestra.run.smithi190.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-04-24T23:13:12.503 INFO:teuthology.orchestra.run.smithi190.stdout:Installing dependencies: 2024-04-24T23:13:12.503 INFO:teuthology.orchestra.run.smithi190.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-24T23:13:12.503 INFO:teuthology.orchestra.run.smithi190.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-24T23:13:12.503 INFO:teuthology.orchestra.run.smithi190.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-24T23:13:12.503 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-common x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 21 M 2024-04-24T23:13:12.503 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 32 k 2024-04-24T23:13:12.503 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mds x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 2.4 M 2024-04-24T23:13:12.503 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 249 k 2024-04-24T23:13:12.503 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mon x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 4.7 M 2024-04-24T23:13:12.503 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-osd x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 17 M 2024-04-24T23:13:12.503 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 15 k 2024-04-24T23:13:12.503 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-selinux x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 25 k 2024-04-24T23:13:12.504 INFO:teuthology.orchestra.run.smithi190.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-24T23:13:12.504 INFO:teuthology.orchestra.run.smithi190.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-24T23:13:12.504 INFO:teuthology.orchestra.run.smithi190.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-24T23:13:12.504 INFO:teuthology.orchestra.run.smithi190.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-24T23:13:12.504 INFO:teuthology.orchestra.run.smithi190.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-24T23:13:12.504 INFO:teuthology.orchestra.run.smithi190.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-24T23:13:12.504 INFO:teuthology.orchestra.run.smithi190.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-24T23:13:12.504 INFO:teuthology.orchestra.run.smithi190.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-24T23:13:12.504 INFO:teuthology.orchestra.run.smithi190.stdout: libcephsqlite x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 163 k 2024-04-24T23:13:12.504 INFO:teuthology.orchestra.run.smithi190.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-24T23:13:12.504 INFO:teuthology.orchestra.run.smithi190.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-24T23:13:12.504 INFO:teuthology.orchestra.run.smithi190.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-24T23:13:12.504 INFO:teuthology.orchestra.run.smithi190.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-24T23:13:12.504 INFO:teuthology.orchestra.run.smithi190.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-24T23:13:12.505 INFO:teuthology.orchestra.run.smithi190.stdout: libradosstriper1 x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 501 k 2024-04-24T23:13:12.505 INFO:teuthology.orchestra.run.smithi190.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-24T23:13:12.505 INFO:teuthology.orchestra.run.smithi190.stdout: librgw2 x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 5.3 M 2024-04-24T23:13:12.505 INFO:teuthology.orchestra.run.smithi190.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-24T23:13:12.505 INFO:teuthology.orchestra.run.smithi190.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-24T23:13:12.505 INFO:teuthology.orchestra.run.smithi190.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-24T23:13:12.505 INFO:teuthology.orchestra.run.smithi190.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-24T23:13:12.505 INFO:teuthology.orchestra.run.smithi190.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-24T23:13:12.505 INFO:teuthology.orchestra.run.smithi190.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-24T23:13:12.505 INFO:teuthology.orchestra.run.smithi190.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-24T23:13:12.505 INFO:teuthology.orchestra.run.smithi190.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-24T23:13:12.505 INFO:teuthology.orchestra.run.smithi190.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-24T23:13:12.506 INFO:teuthology.orchestra.run.smithi190.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-24T23:13:12.506 INFO:teuthology.orchestra.run.smithi190.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-24T23:13:12.506 INFO:teuthology.orchestra.run.smithi190.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-24T23:13:12.506 INFO:teuthology.orchestra.run.smithi190.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-24T23:13:12.506 INFO:teuthology.orchestra.run.smithi190.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-24T23:13:12.506 INFO:teuthology.orchestra.run.smithi190.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-24T23:13:12.506 INFO:teuthology.orchestra.run.smithi190.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-24T23:13:12.506 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-24T23:13:12.506 INFO:teuthology.orchestra.run.smithi190.stdout: python3-ceph-argparse x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 45 k 2024-04-24T23:13:12.506 INFO:teuthology.orchestra.run.smithi190.stdout: python3-ceph-common x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 133 k 2024-04-24T23:13:12.506 INFO:teuthology.orchestra.run.smithi190.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-24T23:13:12.506 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-24T23:13:12.507 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-24T23:13:12.507 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-24T23:13:12.507 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-24T23:13:12.507 INFO:teuthology.orchestra.run.smithi190.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-24T23:13:12.507 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-24T23:13:12.507 INFO:teuthology.orchestra.run.smithi190.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-24T23:13:12.507 INFO:teuthology.orchestra.run.smithi190.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-24T23:13:12.507 INFO:teuthology.orchestra.run.smithi190.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-24T23:13:12.507 INFO:teuthology.orchestra.run.smithi190.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-24T23:13:12.507 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-24T23:13:12.508 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-24T23:13:12.508 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-24T23:13:12.508 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-24T23:13:12.508 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-24T23:13:12.508 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-24T23:13:12.508 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-24T23:13:12.508 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-24T23:13:12.508 INFO:teuthology.orchestra.run.smithi190.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-24T23:13:12.508 INFO:teuthology.orchestra.run.smithi190.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-24T23:13:12.508 INFO:teuthology.orchestra.run.smithi190.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-24T23:13:12.508 INFO:teuthology.orchestra.run.smithi190.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-24T23:13:12.508 INFO:teuthology.orchestra.run.smithi190.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-24T23:13:12.508 INFO:teuthology.orchestra.run.smithi190.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-24T23:13:12.509 INFO:teuthology.orchestra.run.smithi190.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-24T23:13:12.509 INFO:teuthology.orchestra.run.smithi190.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-24T23:13:12.509 INFO:teuthology.orchestra.run.smithi190.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-24T23:13:12.509 INFO:teuthology.orchestra.run.smithi190.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-24T23:13:12.509 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-24T23:13:12.509 INFO:teuthology.orchestra.run.smithi190.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-24T23:13:12.509 INFO:teuthology.orchestra.run.smithi190.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-24T23:13:12.509 INFO:teuthology.orchestra.run.smithi190.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-24T23:13:12.509 INFO:teuthology.orchestra.run.smithi190.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-24T23:13:12.509 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-24T23:13:12.509 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-24T23:13:12.509 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-24T23:13:12.509 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-24T23:13:12.510 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-24T23:13:12.510 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-24T23:13:12.510 INFO:teuthology.orchestra.run.smithi190.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-24T23:13:12.510 INFO:teuthology.orchestra.run.smithi190.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-24T23:13:12.510 INFO:teuthology.orchestra.run.smithi190.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-24T23:13:12.510 INFO:teuthology.orchestra.run.smithi190.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-24T23:13:12.510 INFO:teuthology.orchestra.run.smithi190.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-24T23:13:12.510 INFO:teuthology.orchestra.run.smithi190.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-24T23:13:12.510 INFO:teuthology.orchestra.run.smithi190.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-24T23:13:12.510 INFO:teuthology.orchestra.run.smithi190.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-24T23:13:12.510 INFO:teuthology.orchestra.run.smithi190.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-24T23:13:12.510 INFO:teuthology.orchestra.run.smithi190.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-24T23:13:12.510 INFO:teuthology.orchestra.run.smithi190.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-24T23:13:12.510 INFO:teuthology.orchestra.run.smithi190.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-24T23:13:12.511 INFO:teuthology.orchestra.run.smithi190.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-24T23:13:12.511 INFO:teuthology.orchestra.run.smithi190.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-24T23:13:12.511 INFO:teuthology.orchestra.run.smithi190.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-24T23:13:12.511 INFO:teuthology.orchestra.run.smithi190.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-24T23:13:12.511 INFO:teuthology.orchestra.run.smithi190.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-24T23:13:12.511 INFO:teuthology.orchestra.run.smithi190.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-24T23:13:12.511 INFO:teuthology.orchestra.run.smithi190.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-24T23:13:12.511 INFO:teuthology.orchestra.run.smithi190.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-24T23:13:12.511 INFO:teuthology.orchestra.run.smithi190.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-24T23:13:12.511 INFO:teuthology.orchestra.run.smithi190.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-24T23:13:12.511 INFO:teuthology.orchestra.run.smithi190.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-24T23:13:12.511 INFO:teuthology.orchestra.run.smithi190.stdout:Installing weak dependencies: 2024-04-24T23:13:12.511 INFO:teuthology.orchestra.run.smithi190.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-24T23:13:12.511 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-04-24T23:13:12.512 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-04-24T23:13:12.512 INFO:teuthology.orchestra.run.smithi190.stdout:======================================================================================= 2024-04-24T23:13:12.512 INFO:teuthology.orchestra.run.smithi190.stdout:Install 148 Packages 2024-04-24T23:13:12.512 INFO:teuthology.orchestra.run.smithi190.stdout:Upgrade 4 Packages 2024-04-24T23:13:12.512 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-04-24T23:13:12.512 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 220 M 2024-04-24T23:13:12.512 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-04-24T23:13:12.551 INFO:teuthology.orchestra.run.smithi006.stdout:(104/152): protobuf-compiler-3.14.0-13.el9.x86_ 1.5 MB/s | 863 kB 00:00 2024-04-24T23:13:12.651 INFO:teuthology.orchestra.run.smithi006.stdout:(105/152): gperftools-libs-2.9.1-2.el9.x86_64.r 883 kB/s | 309 kB 00:00 2024-04-24T23:13:12.693 INFO:teuthology.orchestra.run.smithi006.stdout:(106/152): abseil-cpp-20211102.0-3.el9.x86_64.r 1.3 MB/s | 548 kB 00:00 2024-04-24T23:13:12.719 INFO:teuthology.orchestra.run.smithi006.stdout:(107/152): grpc-data-1.46.7-10.el9.noarch.rpm 117 kB/s | 19 kB 00:00 2024-04-24T23:13:12.777 INFO:teuthology.orchestra.run.smithi006.stdout:(108/152): libarrow-doc-9.0.0-10.el9.noarch.rpm 311 kB/s | 26 kB 00:00 2024-04-24T23:13:12.844 INFO:teuthology.orchestra.run.smithi006.stdout:(109/152): liboath-2.6.7-2.el9.x86_64.rpm 390 kB/s | 49 kB 00:00 2024-04-24T23:13:12.861 INFO:teuthology.orchestra.run.smithi006.stdout:(110/152): libunwind-1.6.2-1.el9.x86_64.rpm 814 kB/s | 67 kB 00:00 2024-04-24T23:13:12.995 INFO:teuthology.orchestra.run.smithi006.stdout:(111/152): libarrow-9.0.0-10.el9.x86_64.rpm 13 MB/s | 4.4 MB 00:00 2024-04-24T23:13:13.028 INFO:teuthology.orchestra.run.smithi006.stdout:(112/152): luarocks-3.9.2-1.el9.noarch.rpm 826 kB/s | 151 kB 00:00 2024-04-24T23:13:13.079 INFO:teuthology.orchestra.run.smithi006.stdout:(113/152): parquet-libs-9.0.0-10.el9.x86_64.rpm 3.8 MB/s | 838 kB 00:00 2024-04-24T23:13:13.104 INFO:teuthology.orchestra.run.smithi006.stdout:(114/152): python3-bcrypt-3.2.2-1.el9.x86_64.rp 582 kB/s | 43 kB 00:00 2024-04-24T23:13:13.154 INFO:teuthology.orchestra.run.smithi006.stdout:(115/152): python3-asyncssh-2.13.2-5.el9.noarch 3.4 MB/s | 548 kB 00:00 2024-04-24T23:13:13.179 INFO:teuthology.orchestra.run.smithi006.stdout:(116/152): python3-cachetools-4.2.4-1.el9.noarc 323 kB/s | 32 kB 00:00 2024-04-24T23:13:13.205 INFO:teuthology.orchestra.run.smithi006.stdout:(117/152): python3-certifi-2021.10.8-2.el9.noar 150 kB/s | 15 kB 00:00 2024-04-24T23:13:13.238 INFO:teuthology.orchestra.run.smithi006.stdout:(118/152): python3-cheroot-8.6.0-4.el9.noarch.r 2.0 MB/s | 172 kB 00:00 2024-04-24T23:13:13.272 INFO:teuthology.orchestra.run.smithi006.stdout:(119/152): python3-cherrypy-18.6.1-2.el9.noarch 3.8 MB/s | 358 kB 00:00 2024-04-24T23:13:13.314 INFO:teuthology.orchestra.run.smithi006.stdout:(120/152): python3-colorama-0.4.4-12.el9.noarch 474 kB/s | 35 kB 00:00 2024-04-24T23:13:13.348 INFO:teuthology.orchestra.run.smithi006.stdout:(121/152): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.1 MB/s | 303 kB 00:00 2024-04-24T23:13:13.354 INFO:teuthology.orchestra.run.smithi190.stdout:(1/152): ceph-19.0.0-2445.gd2c45d97.el9.x86_64. 37 kB/s | 6.6 kB 00:00 2024-04-24T23:13:13.399 INFO:teuthology.orchestra.run.smithi006.stdout:(122/152): python3-google-auth-2.29.0-1.el9.noa 1.6 MB/s | 206 kB 00:00 2024-04-24T23:13:13.448 INFO:teuthology.orchestra.run.smithi006.stdout:(123/152): python3-grpcio-tools-1.46.7-10.el9.x 1.4 MB/s | 144 kB 00:00 2024-04-24T23:13:13.473 INFO:teuthology.orchestra.run.smithi006.stdout:(124/152): python3-jaraco-8.2.1-3.el9.noarch.rp 145 kB/s | 11 kB 00:00 2024-04-24T23:13:13.541 INFO:teuthology.orchestra.run.smithi006.stdout:(125/152): python3-grpcio-1.46.7-10.el9.x86_64. 9.0 MB/s | 2.0 MB 00:00 2024-04-24T23:13:13.546 INFO:teuthology.orchestra.run.smithi190.stdout:(2/152): ceph-fuse-19.0.0-2445.gd2c45d97.el9.x8 4.5 MB/s | 890 kB 00:00 2024-04-24T23:13:13.566 INFO:teuthology.orchestra.run.smithi006.stdout:(126/152): python3-jaraco-classes-3.2.1-5.el9.n 152 kB/s | 18 kB 00:00 2024-04-24T23:13:13.591 INFO:teuthology.orchestra.run.smithi006.stdout:(127/152): python3-jaraco-collections-3.0.0-8.e 199 kB/s | 23 kB 00:00 2024-04-24T23:13:13.596 INFO:teuthology.orchestra.run.smithi190.stdout:(3/152): ceph-immutable-object-cache-19.0.0-244 2.8 MB/s | 145 kB 00:00 2024-04-24T23:13:13.616 INFO:teuthology.orchestra.run.smithi006.stdout:(128/152): python3-jaraco-functools-3.5.0-2.el9 261 kB/s | 19 kB 00:00 2024-04-24T23:13:13.667 INFO:teuthology.orchestra.run.smithi006.stdout:(129/152): python3-jaraco-text-3.2.0-6.el9.noar 266 kB/s | 20 kB 00:00 2024-04-24T23:13:13.692 INFO:teuthology.orchestra.run.smithi006.stdout:(130/152): python3-logutils-0.3.5-21.el9.noarch 620 kB/s | 46 kB 00:00 2024-04-24T23:13:13.750 INFO:teuthology.orchestra.run.smithi006.stdout:(131/152): python3-kubernetes-26.1.0-2.el9.noar 6.5 MB/s | 1.0 MB 00:00 2024-04-24T23:13:13.775 INFO:teuthology.orchestra.run.smithi006.stdout:(132/152): python3-natsort-7.1.1-5.el9.noarch.r 696 kB/s | 58 kB 00:00 2024-04-24T23:13:13.809 INFO:teuthology.orchestra.run.smithi006.stdout:(133/152): python3-more-itertools-8.12.0-2.el9. 556 kB/s | 79 kB 00:00 2024-04-24T23:13:13.843 INFO:teuthology.orchestra.run.smithi006.stdout:(134/152): python3-pecan-1.4.2-3.el9.noarch.rpm 2.9 MB/s | 272 kB 00:00 2024-04-24T23:13:13.868 INFO:teuthology.orchestra.run.smithi006.stdout:(135/152): python3-portend-3.1.0-2.el9.noarch.r 179 kB/s | 16 kB 00:00 2024-04-24T23:13:13.894 INFO:teuthology.orchestra.run.smithi006.stdout:(136/152): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.1 MB/s | 90 kB 00:00 2024-04-24T23:13:13.919 INFO:teuthology.orchestra.run.smithi006.stdout:(137/152): python3-pyperclip-1.8.2-2.el9.noarch 377 kB/s | 28 kB 00:00 2024-04-24T23:13:13.930 INFO:teuthology.orchestra.run.smithi190.stdout:(4/152): ceph-mds-19.0.0-2445.gd2c45d97.el9.x86 7.1 MB/s | 2.4 MB 00:00 2024-04-24T23:13:13.952 INFO:teuthology.orchestra.run.smithi006.stdout:(138/152): python3-repoze-lru-0.7-16.el9.noarch 370 kB/s | 31 kB 00:00 2024-04-24T23:13:13.994 INFO:teuthology.orchestra.run.smithi006.stdout:(139/152): python3-routes-2.5.1-5.el9.noarch.rp 1.8 MB/s | 188 kB 00:00 2024-04-24T23:13:14.020 INFO:teuthology.orchestra.run.smithi006.stdout:(140/152): python3-rsa-4.9-2.el9.noarch.rpm 592 kB/s | 59 kB 00:00 2024-04-24T23:13:14.045 INFO:teuthology.orchestra.run.smithi006.stdout:(141/152): python3-tempora-5.0.0-2.el9.noarch.r 390 kB/s | 36 kB 00:00 2024-04-24T23:13:14.050 INFO:teuthology.orchestra.run.smithi190.stdout:(5/152): ceph-base-19.0.0-2445.gd2c45d97.el9.x8 6.2 MB/s | 5.4 MB 00:00 2024-04-24T23:13:14.078 INFO:teuthology.orchestra.run.smithi006.stdout:(142/152): python3-typing-extensions-4.4.0-2.el 613 kB/s | 51 kB 00:00 2024-04-24T23:13:14.112 INFO:teuthology.orchestra.run.smithi006.stdout:(143/152): python3-webob-1.8.7-6.el9.noarch.rpm 2.5 MB/s | 230 kB 00:00 2024-04-24T23:13:14.138 INFO:teuthology.orchestra.run.smithi006.stdout:(144/152): python3-websocket-client-1.2.3-2.el9 979 kB/s | 90 kB 00:00 2024-04-24T23:13:14.139 INFO:teuthology.orchestra.run.smithi190.stdout:(6/152): ceph-mgr-19.0.0-2445.gd2c45d97.el9.x86 7.2 MB/s | 1.5 MB 00:00 2024-04-24T23:13:14.171 INFO:teuthology.orchestra.run.smithi006.stdout:(145/152): python3-werkzeug-2.0.3-3.el9.1.noarc 4.6 MB/s | 427 kB 00:00 2024-04-24T23:13:14.196 INFO:teuthology.orchestra.run.smithi006.stdout:(146/152): python3-zc-lockfile-2.0-10.el9.noarc 240 kB/s | 20 kB 00:00 2024-04-24T23:13:14.238 INFO:teuthology.orchestra.run.smithi006.stdout:(147/152): re2-20211101-3.el9.x86_64.rpm 1.9 MB/s | 192 kB 00:00 2024-04-24T23:13:14.408 INFO:teuthology.orchestra.run.smithi006.stdout:(148/152): thrift-0.15.0-2.el9.x86_64.rpm 6.7 MB/s | 1.6 MB 00:00 2024-04-24T23:13:14.690 INFO:teuthology.orchestra.run.smithi190.stdout:(7/152): ceph-mon-19.0.0-2445.gd2c45d97.el9.x86 7.3 MB/s | 4.7 MB 00:00 2024-04-24T23:13:14.998 INFO:teuthology.orchestra.run.smithi006.stdout:(149/152): librbd1-19.0.0-2445.gd2c45d97.el9.x8 4.1 MB/s | 3.1 MB 00:00 2024-04-24T23:13:15.389 INFO:teuthology.orchestra.run.smithi006.stdout:(150/152): openssl-3.2.1-1.el9.x86_64.rpm 1.4 MB/s | 1.3 MB 00:00 2024-04-24T23:13:15.506 INFO:teuthology.orchestra.run.smithi006.stdout:(151/152): librados2-19.0.0-2445.gd2c45d97.el9. 2.6 MB/s | 3.4 MB 00:01 2024-04-24T23:13:15.581 INFO:teuthology.orchestra.run.smithi190.stdout:(8/152): ceph-common-19.0.0-2445.gd2c45d97.el9. 8.7 MB/s | 21 MB 00:02 2024-04-24T23:13:15.639 INFO:teuthology.orchestra.run.smithi190.stdout:(9/152): ceph-selinux-19.0.0-2445.gd2c45d97.el9 433 kB/s | 25 kB 00:00 2024-04-24T23:13:15.964 INFO:teuthology.orchestra.run.smithi006.stdout:(152/152): openssl-libs-3.2.1-1.el9.x86_64.rpm 2.5 MB/s | 2.4 MB 00:00 2024-04-24T23:13:15.970 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2024-04-24T23:13:15.971 INFO:teuthology.orchestra.run.smithi006.stdout:Total 15 MB/s | 220 MB 00:14 2024-04-24T23:13:16.431 INFO:teuthology.orchestra.run.smithi190.stdout:(10/152): ceph-osd-19.0.0-2445.gd2c45d97.el9.x8 7.3 MB/s | 17 MB 00:02 2024-04-24T23:13:16.540 INFO:teuthology.orchestra.run.smithi190.stdout:(11/152): ceph-radosgw-19.0.0-2445.gd2c45d97.el 5.5 MB/s | 10 MB 00:01 2024-04-24T23:13:16.565 INFO:teuthology.orchestra.run.smithi190.stdout:(12/152): cephfs-mirror-19.0.0-2445.gd2c45d97.e 1.6 MB/s | 222 kB 00:00 2024-04-24T23:13:16.607 INFO:teuthology.orchestra.run.smithi190.stdout:(13/152): cephfs-shell-19.0.0-2445.gd2c45d97.el 405 kB/s | 27 kB 00:00 2024-04-24T23:13:16.633 INFO:teuthology.orchestra.run.smithi190.stdout:(14/152): libcephfs-devel-19.0.0-2445.gd2c45d97 471 kB/s | 32 kB 00:00 2024-04-24T23:13:16.691 INFO:teuthology.orchestra.run.smithi190.stdout:(15/152): libcephsqlite-19.0.0-2445.gd2c45d97.e 2.7 MB/s | 163 kB 00:00 2024-04-24T23:13:16.741 INFO:teuthology.orchestra.run.smithi190.stdout:(16/152): libcephfs2-19.0.0-2445.gd2c45d97.el9. 5.4 MB/s | 745 kB 00:00 2024-04-24T23:13:16.767 INFO:teuthology.orchestra.run.smithi190.stdout:(17/152): librados-devel-19.0.0-2445.gd2c45d97. 1.6 MB/s | 127 kB 00:00 2024-04-24T23:13:16.825 INFO:teuthology.orchestra.run.smithi190.stdout:(18/152): libradosstriper1-19.0.0-2445.gd2c45d9 5.9 MB/s | 501 kB 00:00 2024-04-24T23:13:16.893 INFO:teuthology.orchestra.run.smithi190.stdout:(19/152): python3-ceph-argparse-19.0.0-2445.gd2 671 kB/s | 45 kB 00:00 2024-04-24T23:13:16.985 INFO:teuthology.orchestra.run.smithi190.stdout:(20/152): python3-ceph-common-19.0.0-2445.gd2c4 1.4 MB/s | 133 kB 00:00 2024-04-24T23:13:17.077 INFO:teuthology.orchestra.run.smithi190.stdout:(21/152): python3-cephfs-19.0.0-2445.gd2c45d97. 1.7 MB/s | 162 kB 00:00 2024-04-24T23:13:17.169 INFO:teuthology.orchestra.run.smithi190.stdout:(22/152): python3-rados-19.0.0-2445.gd2c45d97.e 3.4 MB/s | 321 kB 00:00 2024-04-24T23:13:17.252 INFO:teuthology.orchestra.run.smithi190.stdout:(23/152): python3-rbd-19.0.0-2445.gd2c45d97.el9 3.5 MB/s | 299 kB 00:00 2024-04-24T23:13:17.353 INFO:teuthology.orchestra.run.smithi190.stdout:(24/152): librgw2-19.0.0-2445.gd2c45d97.el9.x86 9.1 MB/s | 5.3 MB 00:00 2024-04-24T23:13:17.378 INFO:teuthology.orchestra.run.smithi190.stdout:(25/152): python3-rgw-19.0.0-2445.gd2c45d97.el9 794 kB/s | 99 kB 00:00 2024-04-24T23:13:17.403 INFO:teuthology.orchestra.run.smithi190.stdout:(26/152): rbd-fuse-19.0.0-2445.gd2c45d97.el9.x8 1.7 MB/s | 85 kB 00:00 2024-04-24T23:13:17.478 INFO:teuthology.orchestra.run.smithi190.stdout:(27/152): rbd-nbd-19.0.0-2445.gd2c45d97.el9.x86 2.2 MB/s | 171 kB 00:00 2024-04-24T23:13:17.546 INFO:teuthology.orchestra.run.smithi190.stdout:(28/152): ceph-grafana-dashboards-19.0.0-2445.g 476 kB/s | 32 kB 00:00 2024-04-24T23:13:17.604 INFO:teuthology.orchestra.run.smithi190.stdout:(29/152): ceph-mgr-cephadm-19.0.0-2445.gd2c45d9 2.4 MB/s | 141 kB 00:00 2024-04-24T23:13:17.788 INFO:teuthology.orchestra.run.smithi190.stdout:(30/152): rbd-mirror-19.0.0-2445.gd2c45d97.el9. 7.5 MB/s | 3.1 MB 00:00 2024-04-24T23:13:18.080 INFO:teuthology.orchestra.run.smithi190.stdout:(31/152): ceph-mgr-dashboard-19.0.0-2445.gd2c45 7.4 MB/s | 3.5 MB 00:00 2024-04-24T23:13:18.155 INFO:teuthology.orchestra.run.smithi190.stdout:(32/152): ceph-mgr-modules-core-19.0.0-2445.gd2 3.2 MB/s | 249 kB 00:00 2024-04-24T23:13:18.214 INFO:teuthology.orchestra.run.smithi190.stdout:(33/152): ceph-mgr-rook-19.0.0-2445.gd2c45d97.e 844 kB/s | 49 kB 00:00 2024-04-24T23:13:18.272 INFO:teuthology.orchestra.run.smithi190.stdout:(34/152): ceph-prometheus-alerts-19.0.0-2445.gd 263 kB/s | 15 kB 00:00 2024-04-24T23:13:18.356 INFO:teuthology.orchestra.run.smithi190.stdout:(35/152): ceph-volume-19.0.0-2445.gd2c45d97.el9 3.1 MB/s | 264 kB 00:00 2024-04-24T23:13:18.489 INFO:teuthology.orchestra.run.smithi190.stdout:(36/152): cephadm-19.0.0-2445.gd2c45d97.el9.noa 5.6 MB/s | 762 kB 00:00 2024-04-24T23:13:18.540 INFO:teuthology.orchestra.run.smithi190.stdout:(37/152): cephfs-top-19.0.0-2445.gd2c45d97.el9. 364 kB/s | 18 kB 00:00 2024-04-24T23:13:18.673 INFO:teuthology.orchestra.run.smithi190.stdout:(38/152): ceph-mgr-diskprediction-local-19.0.0- 8.3 MB/s | 7.4 MB 00:00 2024-04-24T23:13:18.744 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2024-04-24T23:13:19.004 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2024-04-24T23:13:19.004 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2024-04-24T23:13:19.307 INFO:teuthology.orchestra.run.smithi190.stdout:(39/152): ceph-test-19.0.0-2445.gd2c45d97.el9.x 13 MB/s | 48 MB 00:03 2024-04-24T23:13:19.334 INFO:teuthology.orchestra.run.smithi190.stdout:(40/152): fuse-2.9.9-15.el9.x86_64.rpm 101 kB/s | 80 kB 00:00 2024-04-24T23:13:19.684 INFO:teuthology.orchestra.run.smithi190.stdout:(41/152): python3-cffi-1.14.5-5.el9.x86_64.rpm 723 kB/s | 253 kB 00:00 2024-04-24T23:13:19.826 INFO:teuthology.orchestra.run.smithi190.stdout:(42/152): libgfortran-11.4.1-3.el9.x86_64.rpm 696 kB/s | 803 kB 00:01 2024-04-24T23:13:19.852 INFO:teuthology.orchestra.run.smithi190.stdout:(43/152): libquadmath-11.4.1-3.el9.x86_64.rpm 359 kB/s | 195 kB 00:00 2024-04-24T23:13:19.969 INFO:teuthology.orchestra.run.smithi190.stdout:(44/152): python3-ply-3.11-14.el9.noarch.rpm 750 kB/s | 106 kB 00:00 2024-04-24T23:13:19.994 INFO:teuthology.orchestra.run.smithi190.stdout:(45/152): python3-pycparser-2.20-6.el9.noarch.r 954 kB/s | 135 kB 00:00 2024-04-24T23:13:20.086 INFO:teuthology.orchestra.run.smithi190.stdout:(46/152): python3-cryptography-36.0.1-4.el9.x86 3.1 MB/s | 1.2 MB 00:00 2024-04-24T23:13:20.111 INFO:teuthology.orchestra.run.smithi190.stdout:(47/152): python3-requests-2.25.1-8.el9.noarch. 878 kB/s | 125 kB 00:00 2024-04-24T23:13:20.136 INFO:teuthology.orchestra.run.smithi190.stdout:(48/152): python3-urllib3-1.26.5-5.el9.noarch.r 1.5 MB/s | 215 kB 00:00 2024-04-24T23:13:20.270 INFO:teuthology.orchestra.run.smithi190.stdout:(49/152): boost-program-options-1.75.0-8.el9.x8 801 kB/s | 107 kB 00:00 2024-04-24T23:13:20.295 INFO:teuthology.orchestra.run.smithi190.stdout:(50/152): automake-1.16.2-8.el9.noarch.rpm 3.2 MB/s | 689 kB 00:00 2024-04-24T23:13:20.320 INFO:teuthology.orchestra.run.smithi190.stdout:(51/152): elfutils-libelf-devel-0.190-2.el9.x86 429 kB/s | 21 kB 00:00 2024-04-24T23:13:20.354 INFO:teuthology.orchestra.run.smithi190.stdout:(52/152): flex-2.6.4-9.el9.x86_64.rpm 5.2 MB/s | 313 kB 00:00 2024-04-24T23:13:20.379 INFO:teuthology.orchestra.run.smithi190.stdout:(53/152): flexiblas-3.0.4-8.el9.x86_64.rpm 559 kB/s | 33 kB 00:00 2024-04-24T23:13:20.413 INFO:teuthology.orchestra.run.smithi190.stdout:(54/152): bison-3.7.4-5.el9.x86_64.rpm 3.1 MB/s | 943 kB 00:00 2024-04-24T23:13:20.438 INFO:teuthology.orchestra.run.smithi190.stdout:(55/152): flexiblas-openblas-openmp-3.0.4-8.el9 310 kB/s | 18 kB 00:00 2024-04-24T23:13:20.463 INFO:teuthology.orchestra.run.smithi190.stdout:(56/152): libaio-devel-0.3.111-13.el9.x86_64.rp 223 kB/s | 11 kB 00:00 2024-04-24T23:13:20.489 INFO:teuthology.orchestra.run.smithi190.stdout:(57/152): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 160 kB 00:00 2024-04-24T23:13:20.514 INFO:teuthology.orchestra.run.smithi190.stdout:(58/152): librabbitmq-0.11.0-7.el9.x86_64.rpm 901 kB/s | 45 kB 00:00 2024-04-24T23:13:20.547 INFO:teuthology.orchestra.run.smithi190.stdout:(59/152): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 662 kB 00:00 2024-04-24T23:13:20.623 INFO:teuthology.orchestra.run.smithi190.stdout:(60/152): flexiblas-netlib-3.0.4-8.el9.x86_64.r 11 MB/s | 3.0 MB 00:00 2024-04-24T23:13:20.648 INFO:teuthology.orchestra.run.smithi190.stdout:(61/152): libuuid-devel-2.37.4-18.el9.x86_64.rp 253 kB/s | 25 kB 00:00 2024-04-24T23:13:20.673 INFO:teuthology.orchestra.run.smithi190.stdout:(62/152): libzstd-devel-1.5.1-2.el9.x86_64.rpm 947 kB/s | 47 kB 00:00 2024-04-24T23:13:20.699 INFO:teuthology.orchestra.run.smithi190.stdout:(63/152): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.7 MB/s | 292 kB 00:00 2024-04-24T23:13:20.732 INFO:teuthology.orchestra.run.smithi190.stdout:(64/152): libtool-2.4.6-45.el9.x86_64.rpm 2.6 MB/s | 582 kB 00:00 2024-04-24T23:13:20.749 INFO:teuthology.orchestra.run.smithi190.stdout:(65/152): openblas-0.3.21-2.el9.x86_64.rpm 703 kB/s | 35 kB 00:00 2024-04-24T23:13:20.775 INFO:teuthology.orchestra.run.smithi190.stdout:(66/152): lua-5.4.4-4.el9.x86_64.rpm 1.8 MB/s | 188 kB 00:00 2024-04-24T23:13:20.850 INFO:teuthology.orchestra.run.smithi190.stdout:(67/152): postgresql-13.14-1.el9.x86_64.rpm 21 MB/s | 1.6 MB 00:00 2024-04-24T23:13:20.917 INFO:teuthology.orchestra.run.smithi190.stdout:(68/152): openssl-devel-3.2.1-1.el9.x86_64.rpm 26 MB/s | 4.4 MB 00:00 2024-04-24T23:13:20.956 INFO:teuthology.orchestra.run.smithi190.stdout:(69/152): postgresql-contrib-13.14-1.el9.x86_64 8.4 MB/s | 911 kB 00:00 2024-04-24T23:13:21.018 INFO:teuthology.orchestra.run.smithi190.stdout:(70/152): postgresql-private-libs-13.14-1.el9.x 1.3 MB/s | 133 kB 00:00 2024-04-24T23:13:21.085 INFO:teuthology.orchestra.run.smithi190.stdout:(71/152): openblas-openmp-0.3.21-2.el9.x86_64.r 13 MB/s | 4.7 MB 00:00 2024-04-24T23:13:21.127 INFO:teuthology.orchestra.run.smithi190.stdout:(72/152): protobuf-3.14.0-13.el9.x86_64.rpm 9.3 MB/s | 1.0 MB 00:00 2024-04-24T23:13:21.152 INFO:teuthology.orchestra.run.smithi190.stdout:(73/152): python3-attrs-20.3.0-7.el9.noarch.rpm 1.3 MB/s | 87 kB 00:00 2024-04-24T23:13:21.236 INFO:teuthology.orchestra.run.smithi190.stdout:(74/152): postgresql-server-13.14-1.el9.x86_64. 21 MB/s | 5.8 MB 00:00 2024-04-24T23:13:21.261 INFO:teuthology.orchestra.run.smithi190.stdout:(75/152): python3-devel-3.9.18-3.el9.x86_64.rpm 2.2 MB/s | 244 kB 00:00 2024-04-24T23:13:21.294 INFO:teuthology.orchestra.run.smithi190.stdout:(76/152): python3-jinja2-2.11.3-5.el9.noarch.rp 4.2 MB/s | 248 kB 00:00 2024-04-24T23:13:21.319 INFO:teuthology.orchestra.run.smithi190.stdout:(77/152): python3-jsonpatch-1.21-16.el9.noarch. 439 kB/s | 26 kB 00:00 2024-04-24T23:13:21.386 INFO:teuthology.orchestra.run.smithi190.stdout:(78/152): python3-babel-2.9.1-2.el9.noarch.rpm 23 MB/s | 6.0 MB 00:00 2024-04-24T23:13:21.411 INFO:teuthology.orchestra.run.smithi190.stdout:(79/152): python3-jsonpointer-2.0-4.el9.noarch. 164 kB/s | 19 kB 00:00 2024-04-24T23:13:21.437 INFO:teuthology.orchestra.run.smithi190.stdout:(80/152): python3-mako-1.1.4-6.el9.noarch.rpm 1.4 MB/s | 172 kB 00:00 2024-04-24T23:13:21.462 INFO:teuthology.orchestra.run.smithi190.stdout:(81/152): python3-markupsafe-1.1.1-12.el9.x86_6 460 kB/s | 35 kB 00:00 2024-04-24T23:13:21.521 INFO:teuthology.orchestra.run.smithi190.stdout:(82/152): python3-numpy-f2py-1.20.1-5.el9.x86_6 2.9 MB/s | 250 kB 00:00 2024-04-24T23:13:21.546 INFO:teuthology.orchestra.run.smithi190.stdout:(83/152): python3-oauthlib-3.1.1-5.el9.noarch.r 2.6 MB/s | 222 kB 00:00 2024-04-24T23:13:21.613 INFO:teuthology.orchestra.run.smithi190.stdout:(84/152): python3-numpy-1.20.1-5.el9.x86_64.rpm 25 MB/s | 5.1 MB 00:00 2024-04-24T23:13:21.638 INFO:teuthology.orchestra.run.smithi190.stdout:(85/152): python3-prettytable-0.7.2-27.el9.noar 459 kB/s | 42 kB 00:00 2024-04-24T23:13:21.664 INFO:teuthology.orchestra.run.smithi190.stdout:(86/152): python3-packaging-20.9-5.el9.noarch.r 542 kB/s | 77 kB 00:00 2024-04-24T23:13:21.697 INFO:teuthology.orchestra.run.smithi190.stdout:(87/152): python3-protobuf-3.14.0-13.el9.noarch 3.1 MB/s | 269 kB 00:00 2024-04-24T23:13:21.756 INFO:teuthology.orchestra.run.smithi190.stdout:(88/152): python3-pyasn1-modules-0.4.8-6.el9.no 3.0 MB/s | 279 kB 00:00 2024-04-24T23:13:21.781 INFO:teuthology.orchestra.run.smithi190.stdout:(89/152): python3-pytz-2021.1-5.el9.noarch.rpm 610 kB/s | 51 kB 00:00 2024-04-24T23:13:21.784 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2024-04-24T23:13:21.784 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2024-04-24T23:13:21.806 INFO:teuthology.orchestra.run.smithi190.stdout:(90/152): python3-requests-oauthlib-1.3.0-12.el 1.0 MB/s | 54 kB 00:00 2024-04-24T23:13:21.898 INFO:teuthology.orchestra.run.smithi190.stdout:(91/152): python3-toml-0.10.2-6.el9.noarch.rpm 454 kB/s | 42 kB 00:00 2024-04-24T23:13:21.974 INFO:teuthology.orchestra.run.smithi190.stdout:(92/152): python3-wcwidth-0.2.5-8.el9.noarch.rp 588 kB/s | 44 kB 00:00 2024-04-24T23:13:22.050 INFO:teuthology.orchestra.run.smithi190.stdout:(93/152): qatlib-24.02.0-1.el9.x86_64.rpm 2.9 MB/s | 221 kB 00:00 2024-04-24T23:13:22.083 INFO:teuthology.orchestra.run.smithi190.stdout:(94/152): python3-pyasn1-0.4.8-6.el9.noarch.rpm 358 kB/s | 159 kB 00:00 2024-04-24T23:13:22.108 INFO:teuthology.orchestra.run.smithi190.stdout:(95/152): qatlib-service-24.02.0-1.el9.x86_64.r 611 kB/s | 36 kB 00:00 2024-04-24T23:13:22.266 INFO:teuthology.orchestra.run.smithi190.stdout:(96/152): python3-scipy-1.6.2-8.el9.x86_64.rpm 34 MB/s | 16 MB 00:00 2024-04-24T23:13:22.291 INFO:teuthology.orchestra.run.smithi190.stdout:(97/152): qatzip-libs-1.2.0-1.el9.x86_64.rpm 227 kB/s | 47 kB 00:00 2024-04-24T23:13:22.317 INFO:teuthology.orchestra.run.smithi190.stdout:(98/152): uuid-1.6.2-55.el9.x86_64.rpm 1.1 MB/s | 58 kB 00:00 2024-04-24T23:13:22.350 INFO:teuthology.orchestra.run.smithi190.stdout:(99/152): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 1.6 MB/s | 94 kB 00:00 2024-04-24T23:13:22.376 INFO:teuthology.orchestra.run.smithi190.stdout:(100/152): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.1 MB/s | 64 kB 00:00 2024-04-24T23:13:22.401 INFO:teuthology.orchestra.run.smithi190.stdout:(101/152): socat-1.7.4.1-5.el9.x86_64.rpm 1.0 MB/s | 305 kB 00:00 2024-04-24T23:13:22.426 INFO:teuthology.orchestra.run.smithi190.stdout:(102/152): zlib-devel-1.2.11-41.el9.x86_64.rpm 593 kB/s | 45 kB 00:00 2024-04-24T23:13:22.568 INFO:teuthology.orchestra.run.smithi190.stdout:(103/152): lua-devel-5.4.4-4.el9.x86_64.rpm 116 kB/s | 22 kB 00:00 2024-04-24T23:13:22.927 INFO:teuthology.orchestra.run.smithi190.stdout:(104/152): abseil-cpp-20211102.0-3.el9.x86_64.r 1.1 MB/s | 548 kB 00:00 2024-04-24T23:13:22.961 INFO:teuthology.orchestra.run.smithi190.stdout:(105/152): gperftools-libs-2.9.1-2.el9.x86_64.r 790 kB/s | 309 kB 00:00 2024-04-24T23:13:22.986 INFO:teuthology.orchestra.run.smithi190.stdout:(106/152): grpc-data-1.46.7-10.el9.noarch.rpm 333 kB/s | 19 kB 00:00 2024-04-24T23:13:23.128 INFO:teuthology.orchestra.run.smithi190.stdout:(107/152): protobuf-compiler-3.14.0-13.el9.x86_ 1.2 MB/s | 863 kB 00:00 2024-04-24T23:13:23.161 INFO:teuthology.orchestra.run.smithi190.stdout:(108/152): libarrow-doc-9.0.0-10.el9.noarch.rpm 148 kB/s | 26 kB 00:00 2024-04-24T23:13:23.228 INFO:teuthology.orchestra.run.smithi190.stdout:(109/152): libarrow-9.0.0-10.el9.x86_64.rpm 17 MB/s | 4.4 MB 00:00 2024-04-24T23:13:23.254 INFO:teuthology.orchestra.run.smithi190.stdout:(110/152): liboath-2.6.7-2.el9.x86_64.rpm 395 kB/s | 49 kB 00:00 2024-04-24T23:13:23.279 INFO:teuthology.orchestra.run.smithi190.stdout:(111/152): libunwind-1.6.2-1.el9.x86_64.rpm 579 kB/s | 67 kB 00:00 2024-04-24T23:13:23.304 INFO:teuthology.orchestra.run.smithi190.stdout:(112/152): luarocks-3.9.2-1.el9.noarch.rpm 2.0 MB/s | 151 kB 00:00 2024-04-24T23:13:23.371 INFO:teuthology.orchestra.run.smithi190.stdout:(113/152): parquet-libs-9.0.0-10.el9.x86_64.rpm 9.0 MB/s | 838 kB 00:00 2024-04-24T23:13:23.396 INFO:teuthology.orchestra.run.smithi190.stdout:(114/152): python3-asyncssh-2.13.2-5.el9.noarch 4.6 MB/s | 548 kB 00:00 2024-04-24T23:13:23.421 INFO:teuthology.orchestra.run.smithi190.stdout:(115/152): python3-bcrypt-3.2.2-1.el9.x86_64.rp 371 kB/s | 43 kB 00:00 2024-04-24T23:13:23.447 INFO:teuthology.orchestra.run.smithi190.stdout:(116/152): python3-cachetools-4.2.4-1.el9.noarc 433 kB/s | 32 kB 00:00 2024-04-24T23:13:23.489 INFO:teuthology.orchestra.run.smithi190.stdout:(117/152): python3-certifi-2021.10.8-2.el9.noar 225 kB/s | 15 kB 00:00 2024-04-24T23:13:23.542 INFO:teuthology.orchestra.run.smithi190.stdout:(118/152): python3-cheroot-8.6.0-4.el9.noarch.r 1.4 MB/s | 172 kB 00:00 2024-04-24T23:13:23.572 INFO:teuthology.orchestra.run.smithi190.stdout:(119/152): python3-cherrypy-18.6.1-2.el9.noarch 2.8 MB/s | 358 kB 00:00 2024-04-24T23:13:23.606 INFO:teuthology.orchestra.run.smithi190.stdout:(120/152): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.5 MB/s | 303 kB 00:00 2024-04-24T23:13:23.631 INFO:teuthology.orchestra.run.smithi190.stdout:(121/152): python3-colorama-0.4.4-12.el9.noarch 405 kB/s | 35 kB 00:00 2024-04-24T23:13:23.681 INFO:teuthology.orchestra.run.smithi190.stdout:(122/152): python3-google-auth-2.29.0-1.el9.noa 2.7 MB/s | 206 kB 00:00 2024-04-24T23:13:23.757 INFO:teuthology.orchestra.run.smithi190.stdout:(123/152): python3-grpcio-1.46.7-10.el9.x86_64. 14 MB/s | 2.0 MB 00:00 2024-04-24T23:13:23.782 INFO:teuthology.orchestra.run.smithi190.stdout:(124/152): python3-grpcio-tools-1.46.7-10.el9.x 960 kB/s | 144 kB 00:00 2024-04-24T23:13:23.807 INFO:teuthology.orchestra.run.smithi190.stdout:(125/152): python3-jaraco-8.2.1-3.el9.noarch.rp 87 kB/s | 11 kB 00:00 2024-04-24T23:13:23.832 INFO:teuthology.orchestra.run.smithi190.stdout:(126/152): python3-jaraco-classes-3.2.1-5.el9.n 235 kB/s | 18 kB 00:00 2024-04-24T23:13:23.874 INFO:teuthology.orchestra.run.smithi190.stdout:(127/152): python3-jaraco-collections-3.0.0-8.e 347 kB/s | 23 kB 00:00 2024-04-24T23:13:23.900 INFO:teuthology.orchestra.run.smithi190.stdout:(128/152): python3-jaraco-functools-3.5.0-2.el9 212 kB/s | 19 kB 00:00 2024-04-24T23:13:23.925 INFO:teuthology.orchestra.run.smithi190.stdout:(129/152): python3-jaraco-text-3.2.0-6.el9.noar 213 kB/s | 20 kB 00:00 2024-04-24T23:13:23.992 INFO:teuthology.orchestra.run.smithi190.stdout:(130/152): python3-kubernetes-26.1.0-2.el9.noar 11 MB/s | 1.0 MB 00:00 2024-04-24T23:13:24.017 INFO:teuthology.orchestra.run.smithi190.stdout:(131/152): python3-logutils-0.3.5-21.el9.noarch 396 kB/s | 46 kB 00:00 2024-04-24T23:13:24.042 INFO:teuthology.orchestra.run.smithi190.stdout:(132/152): python3-more-itertools-8.12.0-2.el9. 673 kB/s | 79 kB 00:00 2024-04-24T23:13:24.068 INFO:teuthology.orchestra.run.smithi190.stdout:(133/152): python3-natsort-7.1.1-5.el9.noarch.r 765 kB/s | 58 kB 00:00 2024-04-24T23:13:24.118 INFO:teuthology.orchestra.run.smithi190.stdout:(134/152): python3-pecan-1.4.2-3.el9.noarch.rpm 3.5 MB/s | 272 kB 00:00 2024-04-24T23:13:24.143 INFO:teuthology.orchestra.run.smithi190.stdout:(135/152): python3-portend-3.1.0-2.el9.noarch.r 163 kB/s | 16 kB 00:00 2024-04-24T23:13:24.168 INFO:teuthology.orchestra.run.smithi190.stdout:(136/152): python3-pyOpenSSL-21.0.0-1.el9.noarc 896 kB/s | 90 kB 00:00 2024-04-24T23:13:24.194 INFO:teuthology.orchestra.run.smithi190.stdout:(137/152): python3-pyperclip-1.8.2-2.el9.noarch 376 kB/s | 28 kB 00:00 2024-04-24T23:13:24.244 INFO:teuthology.orchestra.run.smithi190.stdout:(138/152): python3-repoze-lru-0.7-16.el9.noarch 410 kB/s | 31 kB 00:00 2024-04-24T23:13:24.269 INFO:teuthology.orchestra.run.smithi190.stdout:(139/152): python3-routes-2.5.1-5.el9.noarch.rp 1.8 MB/s | 188 kB 00:00 2024-04-24T23:13:24.294 INFO:teuthology.orchestra.run.smithi190.stdout:(140/152): python3-rsa-4.9-2.el9.noarch.rpm 589 kB/s | 59 kB 00:00 2024-04-24T23:13:24.319 INFO:teuthology.orchestra.run.smithi190.stdout:(141/152): python3-tempora-5.0.0-2.el9.noarch.r 479 kB/s | 36 kB 00:00 2024-04-24T23:13:24.370 INFO:teuthology.orchestra.run.smithi190.stdout:(142/152): python3-typing-extensions-4.4.0-2.el 682 kB/s | 51 kB 00:00 2024-04-24T23:13:24.395 INFO:teuthology.orchestra.run.smithi190.stdout:(143/152): python3-webob-1.8.7-6.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-04-24T23:13:24.429 INFO:teuthology.orchestra.run.smithi190.stdout:(144/152): python3-websocket-client-1.2.3-2.el9 823 kB/s | 90 kB 00:00 2024-04-24T23:13:24.454 INFO:teuthology.orchestra.run.smithi190.stdout:(145/152): python3-werkzeug-2.0.3-3.el9.1.noarc 5.0 MB/s | 427 kB 00:00 2024-04-24T23:13:24.479 INFO:teuthology.orchestra.run.smithi190.stdout:(146/152): python3-zc-lockfile-2.0-10.el9.noarc 238 kB/s | 20 kB 00:00 2024-04-24T23:13:24.513 INFO:teuthology.orchestra.run.smithi190.stdout:(147/152): re2-20211101-3.el9.x86_64.rpm 2.3 MB/s | 192 kB 00:00 2024-04-24T23:13:24.571 INFO:teuthology.orchestra.run.smithi190.stdout:(148/152): thrift-0.15.0-2.el9.x86_64.rpm 14 MB/s | 1.6 MB 00:00 2024-04-24T23:13:25.355 INFO:teuthology.orchestra.run.smithi190.stdout:(149/152): openssl-3.2.1-1.el9.x86_64.rpm 1.7 MB/s | 1.3 MB 00:00 2024-04-24T23:13:25.405 INFO:teuthology.orchestra.run.smithi190.stdout:(150/152): librados2-19.0.0-2445.gd2c45d97.el9. 3.7 MB/s | 3.4 MB 00:00 2024-04-24T23:13:25.647 INFO:teuthology.orchestra.run.smithi190.stdout:(151/152): openssl-libs-3.2.1-1.el9.x86_64.rpm 8.2 MB/s | 2.4 MB 00:00 2024-04-24T23:13:25.764 INFO:teuthology.orchestra.run.smithi190.stdout:(152/152): librbd1-19.0.0-2445.gd2c45d97.el9.x8 2.5 MB/s | 3.1 MB 00:01 2024-04-24T23:13:25.768 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-04-24T23:13:25.768 INFO:teuthology.orchestra.run.smithi190.stdout:Total 17 MB/s | 220 MB 00:13 2024-04-24T23:13:26.830 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-04-24T23:13:26.972 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-04-24T23:13:26.972 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-04-24T23:13:28.313 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2024-04-24T23:13:28.423 INFO:teuthology.orchestra.run.smithi006.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/156 2024-04-24T23:13:28.662 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/156 2024-04-24T23:13:28.731 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-04-24T23:13:28.731 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-04-24T23:13:28.924 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/156 2024-04-24T23:13:28.987 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/156 2024-04-24T23:13:29.032 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/156 2024-04-24T23:13:29.163 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/156 2024-04-24T23:13:29.611 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/156 2024-04-24T23:13:29.713 INFO:teuthology.orchestra.run.smithi006.stdout: Upgrading : librados2-2:19.0.0-2445.gd2c45d97.el9.x86_64 8/156 2024-04-24T23:13:29.894 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: librados2-2:19.0.0-2445.gd2c45d97.el9.x86_64 8/156 2024-04-24T23:13:29.923 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libcephfs2-2:19.0.0-2445.gd2c45d97.el9.x86_64 9/156 2024-04-24T23:13:30.057 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: libcephfs2-2:19.0.0-2445.gd2c45d97.el9.x86_64 9/156 2024-04-24T23:13:30.105 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-rados-2:19.0.0-2445.gd2c45d97.el9.x86_64 10/156 2024-04-24T23:13:30.471 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/156 2024-04-24T23:13:30.535 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/156 2024-04-24T23:13:30.562 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libcephsqlite-2:19.0.0-2445.gd2c45d97.el9.x86_64 13/156 2024-04-24T23:13:30.697 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2445.gd2c45d97.el9.x86_64 13/156 2024-04-24T23:13:30.721 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libradosstriper1-2:19.0.0-2445.gd2c45d97.el9.x86 14/156 2024-04-24T23:13:30.833 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2445.gd2c45d97.el9.x86 14/156 2024-04-24T23:13:30.864 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-24T23:13:30.942 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-24T23:13:31.241 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-24T23:13:31.299 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-24T23:13:31.522 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : re2-1:20211101-3.el9.x86_64 17/156 2024-04-24T23:13:31.641 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 18/156 2024-04-24T23:13:31.681 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/156 2024-04-24T23:13:31.730 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/156 2024-04-24T23:13:31.861 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/156 2024-04-24T23:13:31.920 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 22/156 2024-04-24T23:13:31.986 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 23/156 2024-04-24T23:13:32.033 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 24/156 2024-04-24T23:13:32.063 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : lua-5.4.4-4.el9.x86_64 25/156 2024-04-24T23:13:32.183 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/156 2024-04-24T23:13:32.251 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 27/156 2024-04-24T23:13:32.290 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 28/156 2024-04-24T23:13:32.346 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/156 2024-04-24T23:13:32.457 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/156 2024-04-24T23:13:32.501 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-ceph-common-2:19.0.0-2445.gd2c45d97.el9. 31/156 2024-04-24T23:13:32.556 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-ceph-argparse-2:19.0.0-2445.gd2c45d97.el 32/156 2024-04-24T23:13:32.592 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-cephfs-2:19.0.0-2445.gd2c45d97.el9.x86_6 33/156 2024-04-24T23:13:32.717 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/156 2024-04-24T23:13:32.788 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/156 2024-04-24T23:13:32.900 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 36/156 2024-04-24T23:13:33.084 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 37/156 2024-04-24T23:13:33.149 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 38/156 2024-04-24T23:13:33.864 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-rsa-4.9-2.el9.noarch 39/156 2024-04-24T23:13:33.976 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 40/156 2024-04-24T23:13:34.015 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 41/156 2024-04-24T23:13:34.059 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 42/156 2024-04-24T23:13:34.101 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : librados-devel-2:19.0.0-2445.gd2c45d97.el9.x86_6 43/156 2024-04-24T23:13:34.161 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 44/156 2024-04-24T23:13:34.424 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 45/156 2024-04-24T23:13:34.485 INFO:teuthology.orchestra.run.smithi006.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 46/156 2024-04-24T23:13:34.569 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 47/156 2024-04-24T23:13:34.666 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 48/156 2024-04-24T23:13:34.709 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 49/156 2024-04-24T23:13:34.746 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 50/156 2024-04-24T23:13:34.818 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 51/156 2024-04-24T23:13:34.864 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 52/156 2024-04-24T23:13:34.870 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-04-24T23:13:34.927 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 53/156 2024-04-24T23:13:34.951 INFO:teuthology.orchestra.run.smithi190.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/156 2024-04-24T23:13:34.989 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/156 2024-04-24T23:13:34.994 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/156 2024-04-24T23:13:35.042 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/156 2024-04-24T23:13:35.077 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/156 2024-04-24T23:13:35.102 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/156 2024-04-24T23:13:35.153 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/156 2024-04-24T23:13:35.185 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/156 2024-04-24T23:13:35.246 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/156 2024-04-24T23:13:35.285 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 57/156 2024-04-24T23:13:35.344 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 58/156 2024-04-24T23:13:35.487 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 59/156 2024-04-24T23:13:35.524 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 60/156 2024-04-24T23:13:35.565 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/156 2024-04-24T23:13:35.574 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 61/156 2024-04-24T23:13:35.620 INFO:teuthology.orchestra.run.smithi190.stdout: Upgrading : librados2-2:19.0.0-2445.gd2c45d97.el9.x86_64 8/156 2024-04-24T23:13:35.631 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 62/156 2024-04-24T23:13:35.665 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 63/156 2024-04-24T23:13:35.742 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: librados2-2:19.0.0-2445.gd2c45d97.el9.x86_64 8/156 2024-04-24T23:13:35.771 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libcephfs2-2:19.0.0-2445.gd2c45d97.el9.x86_64 9/156 2024-04-24T23:13:35.856 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 64/156 2024-04-24T23:13:35.862 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: libcephfs2-2:19.0.0-2445.gd2c45d97.el9.x86_64 9/156 2024-04-24T23:13:35.891 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-rados-2:19.0.0-2445.gd2c45d97.el9.x86_64 10/156 2024-04-24T23:13:36.045 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/156 2024-04-24T23:13:36.048 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 65/156 2024-04-24T23:13:36.096 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 66/156 2024-04-24T23:13:36.097 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/156 2024-04-24T23:13:36.125 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libcephsqlite-2:19.0.0-2445.gd2c45d97.el9.x86_64 13/156 2024-04-24T23:13:36.144 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 67/156 2024-04-24T23:13:36.192 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/156 2024-04-24T23:13:36.230 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2445.gd2c45d97.el9.x86_64 13/156 2024-04-24T23:13:36.260 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libradosstriper1-2:19.0.0-2445.gd2c45d97.el9.x86 14/156 2024-04-24T23:13:36.260 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/156 2024-04-24T23:13:36.322 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2445.gd2c45d97.el9.x86 14/156 2024-04-24T23:13:36.329 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 70/156 2024-04-24T23:13:36.344 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-24T23:13:36.365 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-24T23:13:36.376 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/156 2024-04-24T23:13:36.430 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/156 2024-04-24T23:13:36.490 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 73/156 2024-04-24T23:13:36.557 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-24T23:13:36.565 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 74/156 2024-04-24T23:13:36.594 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-24T23:13:36.619 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 75/156 2024-04-24T23:13:36.705 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : re2-1:20211101-3.el9.x86_64 17/156 2024-04-24T23:13:36.766 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 76/156 2024-04-24T23:13:36.797 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 18/156 2024-04-24T23:13:36.826 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/156 2024-04-24T23:13:36.832 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 77/156 2024-04-24T23:13:36.869 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/156 2024-04-24T23:13:36.876 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 78/156 2024-04-24T23:13:36.942 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/156 2024-04-24T23:13:36.947 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/156 2024-04-24T23:13:36.985 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 22/156 2024-04-24T23:13:37.028 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 23/156 2024-04-24T23:13:37.061 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 24/156 2024-04-24T23:13:37.089 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : lua-5.4.4-4.el9.x86_64 25/156 2024-04-24T23:13:37.098 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-packaging-20.9-5.el9.noarch 80/156 2024-04-24T23:13:37.157 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/156 2024-04-24T23:13:37.164 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 81/156 2024-04-24T23:13:37.211 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 27/156 2024-04-24T23:13:37.250 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 28/156 2024-04-24T23:13:37.285 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/156 2024-04-24T23:13:37.372 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/156 2024-04-24T23:13:37.404 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-ceph-common-2:19.0.0-2445.gd2c45d97.el9. 31/156 2024-04-24T23:13:37.447 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-ceph-argparse-2:19.0.0-2445.gd2c45d97.el 32/156 2024-04-24T23:13:37.476 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-cephfs-2:19.0.0-2445.gd2c45d97.el9.x86_6 33/156 2024-04-24T23:13:37.544 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/156 2024-04-24T23:13:37.590 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/156 2024-04-24T23:13:37.664 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 36/156 2024-04-24T23:13:37.805 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 37/156 2024-04-24T23:13:37.859 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 38/156 2024-04-24T23:13:38.162 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 82/156 2024-04-24T23:13:38.214 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 83/156 2024-04-24T23:13:38.247 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 84/156 2024-04-24T23:13:38.370 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 85/156 2024-04-24T23:13:38.374 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-rsa-4.9-2.el9.noarch 39/156 2024-04-24T23:13:38.435 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 86/156 2024-04-24T23:13:38.572 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 87/156 2024-04-24T23:13:38.615 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 88/156 2024-04-24T23:13:38.675 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 40/156 2024-04-24T23:13:38.710 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 41/156 2024-04-24T23:13:38.752 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 42/156 2024-04-24T23:13:38.790 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : librados-devel-2:19.0.0-2445.gd2c45d97.el9.x86_6 43/156 2024-04-24T23:13:38.832 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 44/156 2024-04-24T23:13:38.860 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : openblas-0.3.21-2.el9.x86_64 89/156 2024-04-24T23:13:38.895 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 90/156 2024-04-24T23:13:38.987 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 45/156 2024-04-24T23:13:39.004 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 91/156 2024-04-24T23:13:39.043 INFO:teuthology.orchestra.run.smithi190.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 46/156 2024-04-24T23:13:39.098 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 47/156 2024-04-24T23:13:39.170 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 48/156 2024-04-24T23:13:39.202 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 49/156 2024-04-24T23:13:39.238 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 50/156 2024-04-24T23:13:39.285 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 51/156 2024-04-24T23:13:39.329 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 52/156 2024-04-24T23:13:39.381 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 53/156 2024-04-24T23:13:39.423 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/156 2024-04-24T23:13:39.557 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/156 2024-04-24T23:13:39.612 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/156 2024-04-24T23:13:39.642 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 57/156 2024-04-24T23:13:39.683 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 58/156 2024-04-24T23:13:39.777 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 59/156 2024-04-24T23:13:39.817 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 60/156 2024-04-24T23:13:39.866 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 61/156 2024-04-24T23:13:39.891 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 92/156 2024-04-24T23:13:39.900 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 62/156 2024-04-24T23:13:39.997 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 63/156 2024-04-24T23:13:39.998 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 64/156 2024-04-24T23:13:40.048 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 93/156 2024-04-24T23:13:40.125 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 65/156 2024-04-24T23:13:40.157 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 66/156 2024-04-24T23:13:40.185 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 67/156 2024-04-24T23:13:40.219 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/156 2024-04-24T23:13:40.258 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/156 2024-04-24T23:13:40.289 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 70/156 2024-04-24T23:13:40.320 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/156 2024-04-24T23:13:40.362 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/156 2024-04-24T23:13:40.404 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 73/156 2024-04-24T23:13:40.507 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 74/156 2024-04-24T23:13:40.664 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 75/156 2024-04-24T23:13:40.816 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 76/156 2024-04-24T23:13:40.861 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 77/156 2024-04-24T23:13:40.892 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 78/156 2024-04-24T23:13:40.939 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/156 2024-04-24T23:13:41.044 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-packaging-20.9-5.el9.noarch 80/156 2024-04-24T23:13:41.101 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 81/156 2024-04-24T23:13:41.725 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 94/156 2024-04-24T23:13:41.805 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 82/156 2024-04-24T23:13:41.827 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 95/156 2024-04-24T23:13:41.857 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 83/156 2024-04-24T23:13:41.861 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 96/156 2024-04-24T23:13:41.891 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 84/156 2024-04-24T23:13:41.896 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 97/156 2024-04-24T23:13:41.988 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 85/156 2024-04-24T23:13:42.048 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 86/156 2024-04-24T23:13:42.128 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 87/156 2024-04-24T23:13:42.176 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 88/156 2024-04-24T23:13:42.309 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 98/156 2024-04-24T23:13:42.352 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : openblas-0.3.21-2.el9.x86_64 89/156 2024-04-24T23:13:42.382 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 90/156 2024-04-24T23:13:42.390 INFO:teuthology.orchestra.run.smithi006.stdout: Upgrading : librbd1-2:19.0.0-2445.gd2c45d97.el9.x86_64 99/156 2024-04-24T23:13:42.468 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 91/156 2024-04-24T23:13:42.617 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: librbd1-2:19.0.0-2445.gd2c45d97.el9.x86_64 99/156 2024-04-24T23:13:42.656 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-rbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 100/156 2024-04-24T23:13:42.718 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 101/156 2024-04-24T23:13:43.098 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 92/156 2024-04-24T23:13:43.212 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 93/156 2024-04-24T23:13:43.307 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 102/156 2024-04-24T23:13:43.359 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : librgw2-2:19.0.0-2445.gd2c45d97.el9.x86_64 103/156 2024-04-24T23:13:43.460 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: librgw2-2:19.0.0-2445.gd2c45d97.el9.x86_64 103/156 2024-04-24T23:13:43.486 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-rgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 104/156 2024-04-24T23:13:44.425 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 94/156 2024-04-24T23:13:44.511 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 95/156 2024-04-24T23:13:44.548 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 96/156 2024-04-24T23:13:44.589 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 97/156 2024-04-24T23:13:44.907 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 98/156 2024-04-24T23:13:44.944 INFO:teuthology.orchestra.run.smithi190.stdout: Upgrading : librbd1-2:19.0.0-2445.gd2c45d97.el9.x86_64 99/156 2024-04-24T23:13:45.033 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: librbd1-2:19.0.0-2445.gd2c45d97.el9.x86_64 99/156 2024-04-24T23:13:45.067 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-rbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 100/156 2024-04-24T23:13:45.101 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 101/156 2024-04-24T23:13:45.570 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 102/156 2024-04-24T23:13:45.597 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : librgw2-2:19.0.0-2445.gd2c45d97.el9.x86_64 103/156 2024-04-24T23:13:45.668 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: librgw2-2:19.0.0-2445.gd2c45d97.el9.x86_64 103/156 2024-04-24T23:13:45.690 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-rgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 104/156 2024-04-24T23:13:46.220 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 105/156 2024-04-24T23:13:46.293 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 105/156 2024-04-24T23:13:46.981 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 105/156 2024-04-24T23:13:47.029 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-base-2:19.0.0-2445.gd2c45d97.el9.x86_64 106/156 2024-04-24T23:13:47.224 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-base-2:19.0.0-2445.gd2c45d97.el9.x86_64 106/156 2024-04-24T23:13:47.224 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-24T23:13:47.224 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-24T23:13:47.224 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:13:47.279 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-selinux-2:19.0.0-2445.gd2c45d97.el9.x86_64 107/156 2024-04-24T23:13:48.014 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 105/156 2024-04-24T23:13:48.044 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 105/156 2024-04-24T23:13:48.582 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 105/156 2024-04-24T23:13:48.631 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-base-2:19.0.0-2445.gd2c45d97.el9.x86_64 106/156 2024-04-24T23:13:48.728 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-base-2:19.0.0-2445.gd2c45d97.el9.x86_64 106/156 2024-04-24T23:13:48.728 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-24T23:13:48.728 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-24T23:13:48.728 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-04-24T23:13:48.757 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-selinux-2:19.0.0-2445.gd2c45d97.el9.x86_64 107/156 2024-04-24T23:14:00.248 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2445.gd2c45d97.el9.x86_64 107/156 2024-04-24T23:14:00.248 INFO:teuthology.orchestra.run.smithi190.stdout:skipping the directory /sys 2024-04-24T23:14:00.248 INFO:teuthology.orchestra.run.smithi190.stdout:skipping the directory /proc 2024-04-24T23:14:00.249 INFO:teuthology.orchestra.run.smithi190.stdout:skipping the directory /mnt 2024-04-24T23:14:00.249 INFO:teuthology.orchestra.run.smithi190.stdout:skipping the directory /var/tmp 2024-04-24T23:14:00.249 INFO:teuthology.orchestra.run.smithi190.stdout:skipping the directory /home 2024-04-24T23:14:00.249 INFO:teuthology.orchestra.run.smithi190.stdout:skipping the directory /root 2024-04-24T23:14:00.249 INFO:teuthology.orchestra.run.smithi190.stdout:skipping the directory /tmp 2024-04-24T23:14:00.249 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-04-24T23:14:00.419 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-volume-2:19.0.0-2445.gd2c45d97.el9.noarch 108/156 2024-04-24T23:14:00.437 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-volume-2:19.0.0-2445.gd2c45d97.el9.noarch 108/156 2024-04-24T23:14:00.437 INFO:teuthology.orchestra.run.smithi190.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T23:14:00.437 INFO:teuthology.orchestra.run.smithi190.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-24T23:14:00.437 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-04-24T23:14:01.010 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2445.gd2c45d97.el9.x86_64 107/156 2024-04-24T23:14:01.011 INFO:teuthology.orchestra.run.smithi006.stdout:skipping the directory /sys 2024-04-24T23:14:01.011 INFO:teuthology.orchestra.run.smithi006.stdout:skipping the directory /proc 2024-04-24T23:14:01.011 INFO:teuthology.orchestra.run.smithi006.stdout:skipping the directory /mnt 2024-04-24T23:14:01.011 INFO:teuthology.orchestra.run.smithi006.stdout:skipping the directory /var/tmp 2024-04-24T23:14:01.011 INFO:teuthology.orchestra.run.smithi006.stdout:skipping the directory /home 2024-04-24T23:14:01.011 INFO:teuthology.orchestra.run.smithi006.stdout:skipping the directory /root 2024-04-24T23:14:01.011 INFO:teuthology.orchestra.run.smithi006.stdout:skipping the directory /tmp 2024-04-24T23:14:01.011 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:14:01.275 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-volume-2:19.0.0-2445.gd2c45d97.el9.noarch 108/156 2024-04-24T23:14:01.338 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-volume-2:19.0.0-2445.gd2c45d97.el9.noarch 108/156 2024-04-24T23:14:01.338 INFO:teuthology.orchestra.run.smithi006.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T23:14:01.338 INFO:teuthology.orchestra.run.smithi006.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-24T23:14:01.338 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:14:01.905 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-osd-2:19.0.0-2445.gd2c45d97.el9.x86_64 109/156 2024-04-24T23:14:01.930 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-osd-2:19.0.0-2445.gd2c45d97.el9.x86_64 109/156 2024-04-24T23:14:01.930 INFO:teuthology.orchestra.run.smithi190.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T23:14:01.930 INFO:teuthology.orchestra.run.smithi190.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-24T23:14:01.930 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-24T23:14:01.930 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-24T23:14:01.930 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-04-24T23:14:02.156 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-mds-2:19.0.0-2445.gd2c45d97.el9.x86_64 110/156 2024-04-24T23:14:02.178 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-mds-2:19.0.0-2445.gd2c45d97.el9.x86_64 110/156 2024-04-24T23:14:02.178 INFO:teuthology.orchestra.run.smithi190.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T23:14:02.178 INFO:teuthology.orchestra.run.smithi190.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-24T23:14:02.178 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-24T23:14:02.178 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-24T23:14:02.179 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-04-24T23:14:02.615 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-mon-2:19.0.0-2445.gd2c45d97.el9.x86_64 111/156 2024-04-24T23:14:02.637 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-mon-2:19.0.0-2445.gd2c45d97.el9.x86_64 111/156 2024-04-24T23:14:02.637 INFO:teuthology.orchestra.run.smithi190.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T23:14:02.638 INFO:teuthology.orchestra.run.smithi190.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-24T23:14:02.638 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-24T23:14:02.638 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-24T23:14:02.638 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-04-24T23:14:02.736 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : automake-1.16.2-8.el9.noarch 112/156 2024-04-24T23:14:02.784 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-ply-3.11-14.el9.noarch 113/156 2024-04-24T23:14:02.848 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 114/156 2024-04-24T23:14:03.031 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 115/156 2024-04-24T23:14:03.042 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-osd-2:19.0.0-2445.gd2c45d97.el9.x86_64 109/156 2024-04-24T23:14:03.074 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 116/156 2024-04-24T23:14:03.126 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-osd-2:19.0.0-2445.gd2c45d97.el9.x86_64 109/156 2024-04-24T23:14:03.126 INFO:teuthology.orchestra.run.smithi006.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T23:14:03.126 INFO:teuthology.orchestra.run.smithi006.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-24T23:14:03.126 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-24T23:14:03.126 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-24T23:14:03.126 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:14:03.151 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 117/156 2024-04-24T23:14:03.247 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-24T23:14:03.385 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-24T23:14:03.402 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-mds-2:19.0.0-2445.gd2c45d97.el9.x86_64 110/156 2024-04-24T23:14:03.432 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 120/156 2024-04-24T23:14:03.464 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 121/156 2024-04-24T23:14:03.475 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-mds-2:19.0.0-2445.gd2c45d97.el9.x86_64 110/156 2024-04-24T23:14:03.475 INFO:teuthology.orchestra.run.smithi006.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T23:14:03.475 INFO:teuthology.orchestra.run.smithi006.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-24T23:14:03.475 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-24T23:14:03.475 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-24T23:14:03.475 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:14:03.492 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : fuse-2.9.9-15.el9.x86_64 122/156 2024-04-24T23:14:03.924 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: cephadm-2:19.0.0-2445.gd2c45d97.el9.noarch 123/156 2024-04-24T23:14:03.948 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : cephadm-2:19.0.0-2445.gd2c45d97.el9.noarch 123/156 2024-04-24T23:14:03.980 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2445.gd2c45d97.e 124/156 2024-04-24T23:14:03.994 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-mon-2:19.0.0-2445.gd2c45d97.el9.x86_64 111/156 2024-04-24T23:14:04.042 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2445.gd2c45d97. 125/156 2024-04-24T23:14:04.066 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-mon-2:19.0.0-2445.gd2c45d97.el9.x86_64 111/156 2024-04-24T23:14:04.066 INFO:teuthology.orchestra.run.smithi006.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T23:14:04.066 INFO:teuthology.orchestra.run.smithi006.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-24T23:14:04.066 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-24T23:14:04.067 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-24T23:14:04.067 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:14:04.080 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2445.gd2c45d97.el9.noa 126/156 2024-04-24T23:14:04.204 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : automake-1.16.2-8.el9.noarch 112/156 2024-04-24T23:14:04.267 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-ply-3.11-14.el9.noarch 113/156 2024-04-24T23:14:04.342 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 114/156 2024-04-24T23:14:04.606 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 115/156 2024-04-24T23:14:04.667 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 116/156 2024-04-24T23:14:04.757 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 117/156 2024-04-24T23:14:04.869 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-24T23:14:05.055 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-24T23:14:05.103 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2445.gd2c45d97.el9.noa 126/156 2024-04-24T23:14:05.141 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 120/156 2024-04-24T23:14:05.148 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2445.gd2c45d97.el9.n 127/156 2024-04-24T23:14:05.182 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 121/156 2024-04-24T23:14:05.214 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : fuse-2.9.9-15.el9.x86_64 122/156 2024-04-24T23:14:05.864 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: cephadm-2:19.0.0-2445.gd2c45d97.el9.noarch 123/156 2024-04-24T23:14:05.899 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : cephadm-2:19.0.0-2445.gd2c45d97.el9.noarch 123/156 2024-04-24T23:14:05.958 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2445.gd2c45d97.e 124/156 2024-04-24T23:14:06.073 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2445.gd2c45d97. 125/156 2024-04-24T23:14:06.102 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2445.gd2c45d97.el9.noa 126/156 2024-04-24T23:14:06.109 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2445.gd2c45d97.el9.n 127/156 2024-04-24T23:14:06.140 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2445.gd2c 128/156 2024-04-24T23:14:06.241 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2445.gd2c 128/156 2024-04-24T23:14:06.403 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2445.gd2c45d97.el 129/156 2024-04-24T23:14:06.442 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-mgr-2:19.0.0-2445.gd2c45d97.el9.x86_64 130/156 2024-04-24T23:14:06.466 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2445.gd2c45d97.el9.x86_64 130/156 2024-04-24T23:14:06.467 INFO:teuthology.orchestra.run.smithi190.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T23:14:06.467 INFO:teuthology.orchestra.run.smithi190.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-24T23:14:06.467 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-24T23:14:06.467 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-24T23:14:06.467 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-04-24T23:14:06.510 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-mgr-rook-2:19.0.0-2445.gd2c45d97.el9.noarch 131/156 2024-04-24T23:14:06.521 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2445.gd2c45d97.el9.noarch 131/156 2024-04-24T23:14:06.619 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-2:19.0.0-2445.gd2c45d97.el9.x86_64 132/156 2024-04-24T23:14:06.692 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 133/156 2024-04-24T23:14:07.466 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2445.gd2c45d97.el9.noa 126/156 2024-04-24T23:14:07.512 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2445.gd2c45d97.el9.n 127/156 2024-04-24T23:14:07.593 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libtool-2.4.6-45.el9.x86_64 134/156 2024-04-24T23:14:07.629 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-radosgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 135/156 2024-04-24T23:14:07.652 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 135/156 2024-04-24T23:14:07.652 INFO:teuthology.orchestra.run.smithi190.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T23:14:07.652 INFO:teuthology.orchestra.run.smithi190.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-24T23:14:07.652 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-24T23:14:07.653 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-24T23:14:07.653 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-04-24T23:14:07.697 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2445.gd2c45 136/156 2024-04-24T23:14:07.719 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2445.gd2c45 136/156 2024-04-24T23:14:07.719 INFO:teuthology.orchestra.run.smithi190.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T23:14:07.719 INFO:teuthology.orchestra.run.smithi190.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-24T23:14:07.719 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-04-24T23:14:07.764 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : cephfs-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 137/156 2024-04-24T23:14:07.786 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 137/156 2024-04-24T23:14:07.787 INFO:teuthology.orchestra.run.smithi190.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T23:14:07.787 INFO:teuthology.orchestra.run.smithi190.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-24T23:14:07.787 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-24T23:14:07.787 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-24T23:14:07.787 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-04-24T23:14:08.098 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : rbd-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 138/156 2024-04-24T23:14:08.120 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 138/156 2024-04-24T23:14:08.121 INFO:teuthology.orchestra.run.smithi190.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T23:14:08.121 INFO:teuthology.orchestra.run.smithi190.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-24T23:14:08.121 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-24T23:14:08.121 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-24T23:14:08.121 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-04-24T23:14:08.683 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2445.gd2c45d97.el9.n 127/156 2024-04-24T23:14:08.712 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2445.gd2c 128/156 2024-04-24T23:14:08.867 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2445.gd2c 128/156 2024-04-24T23:14:09.062 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2445.gd2c45d97.el 129/156 2024-04-24T23:14:09.098 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-mgr-2:19.0.0-2445.gd2c45d97.el9.x86_64 130/156 2024-04-24T23:14:09.183 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2445.gd2c45d97.el9.x86_64 130/156 2024-04-24T23:14:09.183 INFO:teuthology.orchestra.run.smithi006.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T23:14:09.183 INFO:teuthology.orchestra.run.smithi006.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-24T23:14:09.183 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-24T23:14:09.184 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-24T23:14:09.184 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:14:09.248 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-mgr-rook-2:19.0.0-2445.gd2c45d97.el9.noarch 131/156 2024-04-24T23:14:09.288 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2445.gd2c45d97.el9.noarch 131/156 2024-04-24T23:14:09.442 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-2:19.0.0-2445.gd2c45d97.el9.x86_64 132/156 2024-04-24T23:14:09.520 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 133/156 2024-04-24T23:14:10.588 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libtool-2.4.6-45.el9.x86_64 134/156 2024-04-24T23:14:10.617 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-radosgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 135/156 2024-04-24T23:14:10.697 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 135/156 2024-04-24T23:14:10.697 INFO:teuthology.orchestra.run.smithi006.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T23:14:10.697 INFO:teuthology.orchestra.run.smithi006.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-24T23:14:10.697 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-24T23:14:10.697 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-24T23:14:10.697 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:14:10.760 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2445.gd2c45 136/156 2024-04-24T23:14:10.853 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2445.gd2c45 136/156 2024-04-24T23:14:10.901 INFO:teuthology.orchestra.run.smithi006.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T23:14:10.901 INFO:teuthology.orchestra.run.smithi006.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-24T23:14:10.901 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:14:10.953 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : cephfs-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 137/156 2024-04-24T23:14:11.060 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 137/156 2024-04-24T23:14:11.061 INFO:teuthology.orchestra.run.smithi006.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T23:14:11.061 INFO:teuthology.orchestra.run.smithi006.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-24T23:14:11.061 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-24T23:14:11.061 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-24T23:14:11.061 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:14:11.470 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : rbd-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 138/156 2024-04-24T23:14:11.556 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 138/156 2024-04-24T23:14:11.556 INFO:teuthology.orchestra.run.smithi006.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T23:14:11.556 INFO:teuthology.orchestra.run.smithi006.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-24T23:14:11.556 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-24T23:14:11.556 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-24T23:14:11.556 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:14:12.627 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-test-2:19.0.0-2445.gd2c45d97.el9.x86_64 139/156 2024-04-24T23:14:12.676 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : rbd-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 140/156 2024-04-24T23:14:12.725 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : rbd-nbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 141/156 2024-04-24T23:14:12.757 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 142/156 2024-04-24T23:14:12.785 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 143/156 2024-04-24T23:14:12.963 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : cephfs-shell-2:19.0.0-2445.gd2c45d97.el9.x86_64 144/156 2024-04-24T23:14:13.014 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 145/156 2024-04-24T23:14:13.044 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libcephfs-devel-2:19.0.0-2445.gd2c45d97.el9.x86_ 146/156 2024-04-24T23:14:13.690 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-24T23:14:13.725 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-24T23:14:13.745 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-24T23:14:14.573 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : cephfs-top-2:19.0.0-2445.gd2c45d97.el9.noarch 148/156 2024-04-24T23:14:14.646 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 149/156 2024-04-24T23:14:14.687 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 150/156 2024-04-24T23:14:14.767 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : flex-2.6.4-9.el9.x86_64 151/156 2024-04-24T23:14:14.800 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : bison-3.7.4-5.el9.x86_64 152/156 2024-04-24T23:14:14.800 INFO:teuthology.orchestra.run.smithi190.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-24T23:14:14.884 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-24T23:14:14.925 INFO:teuthology.orchestra.run.smithi190.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-24T23:14:14.925 INFO:teuthology.orchestra.run.smithi190.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-24T23:14:15.021 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-24T23:14:15.090 INFO:teuthology.orchestra.run.smithi190.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-24T23:14:16.810 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-test-2:19.0.0-2445.gd2c45d97.el9.x86_64 139/156 2024-04-24T23:14:16.867 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : rbd-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 140/156 2024-04-24T23:14:16.949 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : rbd-nbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 141/156 2024-04-24T23:14:16.997 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 142/156 2024-04-24T23:14:17.047 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 143/156 2024-04-24T23:14:17.390 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : cephfs-shell-2:19.0.0-2445.gd2c45d97.el9.x86_64 144/156 2024-04-24T23:14:17.456 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 145/156 2024-04-24T23:14:17.492 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libcephfs-devel-2:19.0.0-2445.gd2c45d97.el9.x86_ 146/156 2024-04-24T23:14:18.425 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-24T23:14:18.474 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-24T23:14:18.528 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-24T23:14:19.948 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : cephfs-top-2:19.0.0-2445.gd2c45d97.el9.noarch 148/156 2024-04-24T23:14:20.059 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 149/156 2024-04-24T23:14:20.136 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 150/156 2024-04-24T23:14:20.322 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : flex-2.6.4-9.el9.x86_64 151/156 2024-04-24T23:14:20.365 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : bison-3.7.4-5.el9.x86_64 152/156 2024-04-24T23:14:20.365 INFO:teuthology.orchestra.run.smithi006.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-24T23:14:20.490 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-24T23:14:20.554 INFO:teuthology.orchestra.run.smithi006.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-24T23:14:20.555 INFO:teuthology.orchestra.run.smithi006.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-24T23:14:20.695 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-24T23:14:20.831 INFO:teuthology.orchestra.run.smithi006.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-24T23:14:20.978 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-24T23:14:20.978 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-2:19.0.0-2445.gd2c45d97.el9.x86_64 1/156 2024-04-24T23:14:20.978 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-base-2:19.0.0-2445.gd2c45d97.el9.x86_64 2/156 2024-04-24T23:14:20.978 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 3/156 2024-04-24T23:14:20.978 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 4/156 2024-04-24T23:14:20.978 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2445.gd2c45 5/156 2024-04-24T23:14:20.979 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-mds-2:19.0.0-2445.gd2c45d97.el9.x86_64 6/156 2024-04-24T23:14:20.979 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-mgr-2:19.0.0-2445.gd2c45d97.el9.x86_64 7/156 2024-04-24T23:14:20.979 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-mon-2:19.0.0-2445.gd2c45d97.el9.x86_64 8/156 2024-04-24T23:14:20.979 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-osd-2:19.0.0-2445.gd2c45d97.el9.x86_64 9/156 2024-04-24T23:14:20.979 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-radosgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 10/156 2024-04-24T23:14:20.979 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-selinux-2:19.0.0-2445.gd2c45d97.el9.x86_64 11/156 2024-04-24T23:14:20.979 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-test-2:19.0.0-2445.gd2c45d97.el9.x86_64 12/156 2024-04-24T23:14:20.979 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : cephfs-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 13/156 2024-04-24T23:14:20.979 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : cephfs-shell-2:19.0.0-2445.gd2c45d97.el9.x86_64 14/156 2024-04-24T23:14:20.979 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libcephfs-devel-2:19.0.0-2445.gd2c45d97.el9.x86_ 15/156 2024-04-24T23:14:20.979 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libcephfs2-2:19.0.0-2445.gd2c45d97.el9.x86_64 16/156 2024-04-24T23:14:20.979 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libcephsqlite-2:19.0.0-2445.gd2c45d97.el9.x86_64 17/156 2024-04-24T23:14:20.979 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : librados-devel-2:19.0.0-2445.gd2c45d97.el9.x86_6 18/156 2024-04-24T23:14:20.980 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libradosstriper1-2:19.0.0-2445.gd2c45d97.el9.x86 19/156 2024-04-24T23:14:20.980 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : librgw2-2:19.0.0-2445.gd2c45d97.el9.x86_64 20/156 2024-04-24T23:14:20.980 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2445.gd2c45d97.el 21/156 2024-04-24T23:14:20.980 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-ceph-common-2:19.0.0-2445.gd2c45d97.el9. 22/156 2024-04-24T23:14:20.980 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-cephfs-2:19.0.0-2445.gd2c45d97.el9.x86_6 23/156 2024-04-24T23:14:20.980 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-rados-2:19.0.0-2445.gd2c45d97.el9.x86_64 24/156 2024-04-24T23:14:20.980 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-rbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 25/156 2024-04-24T23:14:20.980 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-rgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 26/156 2024-04-24T23:14:20.980 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : rbd-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 27/156 2024-04-24T23:14:20.980 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : rbd-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 28/156 2024-04-24T23:14:20.980 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : rbd-nbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 29/156 2024-04-24T23:14:20.980 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2445.gd2c45d97. 30/156 2024-04-24T23:14:20.980 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2445.gd2c45d97.el9.noa 31/156 2024-04-24T23:14:20.980 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2445.gd2c45d97.el9.n 32/156 2024-04-24T23:14:20.981 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2445.gd2c 33/156 2024-04-24T23:14:20.981 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2445.gd2c45d97.el 34/156 2024-04-24T23:14:20.981 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2445.gd2c45d97.el9.noarch 35/156 2024-04-24T23:14:20.981 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2445.gd2c45d97.e 36/156 2024-04-24T23:14:20.981 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-volume-2:19.0.0-2445.gd2c45d97.el9.noarch 37/156 2024-04-24T23:14:20.981 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : cephadm-2:19.0.0-2445.gd2c45d97.el9.noarch 38/156 2024-04-24T23:14:20.981 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : cephfs-top-2:19.0.0-2445.gd2c45d97.el9.noarch 39/156 2024-04-24T23:14:20.981 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/156 2024-04-24T23:14:20.981 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/156 2024-04-24T23:14:20.981 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/156 2024-04-24T23:14:20.981 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/156 2024-04-24T23:14:20.981 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/156 2024-04-24T23:14:20.981 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/156 2024-04-24T23:14:20.981 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/156 2024-04-24T23:14:20.981 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/156 2024-04-24T23:14:20.982 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/156 2024-04-24T23:14:20.982 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/156 2024-04-24T23:14:20.982 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/156 2024-04-24T23:14:20.982 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/156 2024-04-24T23:14:20.982 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/156 2024-04-24T23:14:20.982 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/156 2024-04-24T23:14:20.982 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/156 2024-04-24T23:14:20.982 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/156 2024-04-24T23:14:20.982 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/156 2024-04-24T23:14:20.982 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/156 2024-04-24T23:14:20.982 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/156 2024-04-24T23:14:20.982 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/156 2024-04-24T23:14:20.982 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/156 2024-04-24T23:14:20.982 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/156 2024-04-24T23:14:20.983 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/156 2024-04-24T23:14:20.983 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/156 2024-04-24T23:14:20.983 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/156 2024-04-24T23:14:20.983 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/156 2024-04-24T23:14:20.983 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/156 2024-04-24T23:14:20.983 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/156 2024-04-24T23:14:20.983 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/156 2024-04-24T23:14:20.983 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/156 2024-04-24T23:14:20.983 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/156 2024-04-24T23:14:20.983 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/156 2024-04-24T23:14:20.983 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/156 2024-04-24T23:14:20.983 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/156 2024-04-24T23:14:20.983 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/156 2024-04-24T23:14:20.983 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/156 2024-04-24T23:14:20.984 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/156 2024-04-24T23:14:20.984 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/156 2024-04-24T23:14:20.984 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/156 2024-04-24T23:14:20.984 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/156 2024-04-24T23:14:20.984 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/156 2024-04-24T23:14:20.984 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/156 2024-04-24T23:14:20.984 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/156 2024-04-24T23:14:20.984 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/156 2024-04-24T23:14:20.984 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/156 2024-04-24T23:14:20.984 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/156 2024-04-24T23:14:20.984 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/156 2024-04-24T23:14:20.984 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/156 2024-04-24T23:14:20.984 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/156 2024-04-24T23:14:20.984 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/156 2024-04-24T23:14:20.985 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/156 2024-04-24T23:14:20.985 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/156 2024-04-24T23:14:20.985 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/156 2024-04-24T23:14:20.985 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/156 2024-04-24T23:14:20.985 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/156 2024-04-24T23:14:20.985 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/156 2024-04-24T23:14:20.985 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/156 2024-04-24T23:14:20.985 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/156 2024-04-24T23:14:20.985 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/156 2024-04-24T23:14:20.985 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/156 2024-04-24T23:14:20.985 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/156 2024-04-24T23:14:20.985 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/156 2024-04-24T23:14:20.985 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/156 2024-04-24T23:14:20.985 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/156 2024-04-24T23:14:20.986 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/156 2024-04-24T23:14:20.986 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/156 2024-04-24T23:14:20.986 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/156 2024-04-24T23:14:20.986 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/156 2024-04-24T23:14:20.986 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 108/156 2024-04-24T23:14:20.986 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 109/156 2024-04-24T23:14:20.986 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/156 2024-04-24T23:14:20.986 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/156 2024-04-24T23:14:20.986 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/156 2024-04-24T23:14:20.986 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 113/156 2024-04-24T23:14:20.986 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/156 2024-04-24T23:14:20.990 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/156 2024-04-24T23:14:20.990 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/156 2024-04-24T23:14:20.990 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/156 2024-04-24T23:14:20.990 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-24T23:14:20.990 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-24T23:14:20.990 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/156 2024-04-24T23:14:20.990 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/156 2024-04-24T23:14:20.991 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/156 2024-04-24T23:14:20.991 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/156 2024-04-24T23:14:20.991 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/156 2024-04-24T23:14:20.991 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/156 2024-04-24T23:14:20.991 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/156 2024-04-24T23:14:20.991 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/156 2024-04-24T23:14:20.991 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/156 2024-04-24T23:14:20.991 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/156 2024-04-24T23:14:20.991 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/156 2024-04-24T23:14:20.991 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/156 2024-04-24T23:14:20.992 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/156 2024-04-24T23:14:20.992 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/156 2024-04-24T23:14:20.992 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/156 2024-04-24T23:14:20.992 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/156 2024-04-24T23:14:20.992 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/156 2024-04-24T23:14:20.992 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/156 2024-04-24T23:14:20.993 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/156 2024-04-24T23:14:20.993 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/156 2024-04-24T23:14:20.993 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/156 2024-04-24T23:14:20.993 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/156 2024-04-24T23:14:20.993 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/156 2024-04-24T23:14:20.993 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/156 2024-04-24T23:14:20.993 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/156 2024-04-24T23:14:20.993 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/156 2024-04-24T23:14:20.993 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/156 2024-04-24T23:14:20.993 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/156 2024-04-24T23:14:20.993 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/156 2024-04-24T23:14:20.993 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : librados2-2:19.0.0-2445.gd2c45d97.el9.x86_64 149/156 2024-04-24T23:14:20.993 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/156 2024-04-24T23:14:20.994 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : librbd1-2:19.0.0-2445.gd2c45d97.el9.x86_64 151/156 2024-04-24T23:14:20.994 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/156 2024-04-24T23:14:20.994 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 153/156 2024-04-24T23:14:20.994 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-24T23:14:20.994 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 155/156 2024-04-24T23:14:21.125 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-24T23:14:21.125 INFO:teuthology.orchestra.run.smithi006.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-04-24T23:14:21.125 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:14:23.018 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-24T23:14:23.018 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-04-24T23:14:23.018 INFO:teuthology.orchestra.run.smithi190.stdout:Upgraded: 2024-04-24T23:14:23.018 INFO:teuthology.orchestra.run.smithi190.stdout: librados2-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:14:23.019 INFO:teuthology.orchestra.run.smithi190.stdout: librbd1-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:14:23.019 INFO:teuthology.orchestra.run.smithi190.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-04-24T23:14:23.019 INFO:teuthology.orchestra.run.smithi190.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-04-24T23:14:23.019 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-04-24T23:14:23.019 INFO:teuthology.orchestra.run.smithi190.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-24T23:14:23.019 INFO:teuthology.orchestra.run.smithi190.stdout: automake-1.16.2-8.el9.noarch 2024-04-24T23:14:23.019 INFO:teuthology.orchestra.run.smithi190.stdout: bison-3.7.4-5.el9.x86_64 2024-04-24T23:14:23.019 INFO:teuthology.orchestra.run.smithi190.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-24T23:14:23.019 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:14:23.019 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-base-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:14:23.019 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:14:23.019 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:14:23.019 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-grafana-dashboards-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-24T23:14:23.020 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-immutable-object-cache-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:14:23.020 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mds-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:14:23.020 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:14:23.020 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-cephadm-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-24T23:14:23.020 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-dashboard-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-24T23:14:23.020 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-24T23:14:23.020 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-modules-core-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-24T23:14:23.020 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-rook-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-24T23:14:23.020 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mon-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:14:23.020 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-osd-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:14:23.020 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-prometheus-alerts-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-24T23:14:23.020 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-radosgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:14:23.020 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-selinux-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:14:23.021 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-test-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:14:23.021 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-volume-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-24T23:14:23.021 INFO:teuthology.orchestra.run.smithi190.stdout: cephadm-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-24T23:14:23.021 INFO:teuthology.orchestra.run.smithi190.stdout: cephfs-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:14:23.021 INFO:teuthology.orchestra.run.smithi190.stdout: cephfs-shell-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:14:23.021 INFO:teuthology.orchestra.run.smithi190.stdout: cephfs-top-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-24T23:14:23.021 INFO:teuthology.orchestra.run.smithi190.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-24T23:14:23.021 INFO:teuthology.orchestra.run.smithi190.stdout: flex-2.6.4-9.el9.x86_64 2024-04-24T23:14:23.021 INFO:teuthology.orchestra.run.smithi190.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-24T23:14:23.021 INFO:teuthology.orchestra.run.smithi190.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-24T23:14:23.021 INFO:teuthology.orchestra.run.smithi190.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-24T23:14:23.021 INFO:teuthology.orchestra.run.smithi190.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-24T23:14:23.021 INFO:teuthology.orchestra.run.smithi190.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-24T23:14:23.022 INFO:teuthology.orchestra.run.smithi190.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-24T23:14:23.022 INFO:teuthology.orchestra.run.smithi190.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-24T23:14:23.022 INFO:teuthology.orchestra.run.smithi190.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-24T23:14:23.022 INFO:teuthology.orchestra.run.smithi190.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-24T23:14:23.022 INFO:teuthology.orchestra.run.smithi190.stdout: libcephfs-devel-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:14:23.022 INFO:teuthology.orchestra.run.smithi190.stdout: libcephfs2-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:14:23.022 INFO:teuthology.orchestra.run.smithi190.stdout: libcephsqlite-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:14:23.022 INFO:teuthology.orchestra.run.smithi190.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-24T23:14:23.022 INFO:teuthology.orchestra.run.smithi190.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-24T23:14:23.022 INFO:teuthology.orchestra.run.smithi190.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-24T23:14:23.022 INFO:teuthology.orchestra.run.smithi190.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-24T23:14:23.022 INFO:teuthology.orchestra.run.smithi190.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-24T23:14:23.022 INFO:teuthology.orchestra.run.smithi190.stdout: librados-devel-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:14:23.022 INFO:teuthology.orchestra.run.smithi190.stdout: libradosstriper1-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:14:23.023 INFO:teuthology.orchestra.run.smithi190.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-24T23:14:23.023 INFO:teuthology.orchestra.run.smithi190.stdout: librgw2-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:14:23.023 INFO:teuthology.orchestra.run.smithi190.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-24T23:14:23.023 INFO:teuthology.orchestra.run.smithi190.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-24T23:14:23.023 INFO:teuthology.orchestra.run.smithi190.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-24T23:14:23.023 INFO:teuthology.orchestra.run.smithi190.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-24T23:14:23.023 INFO:teuthology.orchestra.run.smithi190.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-24T23:14:23.023 INFO:teuthology.orchestra.run.smithi190.stdout: lua-5.4.4-4.el9.x86_64 2024-04-24T23:14:23.023 INFO:teuthology.orchestra.run.smithi190.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-24T23:14:23.023 INFO:teuthology.orchestra.run.smithi190.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-24T23:14:23.023 INFO:teuthology.orchestra.run.smithi190.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-24T23:14:23.023 INFO:teuthology.orchestra.run.smithi190.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-24T23:14:23.023 INFO:teuthology.orchestra.run.smithi190.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-04-24T23:14:23.023 INFO:teuthology.orchestra.run.smithi190.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-24T23:14:23.023 INFO:teuthology.orchestra.run.smithi190.stdout: postgresql-13.14-1.el9.x86_64 2024-04-24T23:14:23.024 INFO:teuthology.orchestra.run.smithi190.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-24T23:14:23.024 INFO:teuthology.orchestra.run.smithi190.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-24T23:14:23.024 INFO:teuthology.orchestra.run.smithi190.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-24T23:14:23.024 INFO:teuthology.orchestra.run.smithi190.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-24T23:14:23.024 INFO:teuthology.orchestra.run.smithi190.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-24T23:14:23.024 INFO:teuthology.orchestra.run.smithi190.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-24T23:14:23.024 INFO:teuthology.orchestra.run.smithi190.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-24T23:14:23.024 INFO:teuthology.orchestra.run.smithi190.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-24T23:14:23.024 INFO:teuthology.orchestra.run.smithi190.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-24T23:14:23.024 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-24T23:14:23.024 INFO:teuthology.orchestra.run.smithi190.stdout: python3-ceph-argparse-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:14:23.024 INFO:teuthology.orchestra.run.smithi190.stdout: python3-ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:14:23.024 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cephfs-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:14:23.024 INFO:teuthology.orchestra.run.smithi190.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-24T23:14:23.025 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-24T23:14:23.025 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-24T23:14:23.025 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-24T23:14:23.025 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-24T23:14:23.025 INFO:teuthology.orchestra.run.smithi190.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-24T23:14:23.025 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-24T23:14:23.025 INFO:teuthology.orchestra.run.smithi190.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-24T23:14:23.025 INFO:teuthology.orchestra.run.smithi190.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-24T23:14:23.025 INFO:teuthology.orchestra.run.smithi190.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-24T23:14:23.025 INFO:teuthology.orchestra.run.smithi190.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-24T23:14:23.025 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-24T23:14:23.025 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-24T23:14:23.025 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-24T23:14:23.025 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-24T23:14:23.025 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-24T23:14:23.026 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-24T23:14:23.026 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-24T23:14:23.026 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-24T23:14:23.026 INFO:teuthology.orchestra.run.smithi190.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-24T23:14:23.026 INFO:teuthology.orchestra.run.smithi190.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-24T23:14:23.026 INFO:teuthology.orchestra.run.smithi190.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-24T23:14:23.026 INFO:teuthology.orchestra.run.smithi190.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-24T23:14:23.026 INFO:teuthology.orchestra.run.smithi190.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-24T23:14:23.026 INFO:teuthology.orchestra.run.smithi190.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-24T23:14:23.026 INFO:teuthology.orchestra.run.smithi190.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-24T23:14:23.026 INFO:teuthology.orchestra.run.smithi190.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-24T23:14:23.026 INFO:teuthology.orchestra.run.smithi190.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-24T23:14:23.026 INFO:teuthology.orchestra.run.smithi190.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-24T23:14:23.027 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-24T23:14:23.027 INFO:teuthology.orchestra.run.smithi190.stdout: python3-ply-3.11-14.el9.noarch 2024-04-24T23:14:23.027 INFO:teuthology.orchestra.run.smithi190.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-24T23:14:23.027 INFO:teuthology.orchestra.run.smithi190.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-24T23:14:23.027 INFO:teuthology.orchestra.run.smithi190.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-24T23:14:23.027 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-24T23:14:23.027 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-24T23:14:23.027 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-24T23:14:23.027 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-24T23:14:23.027 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-24T23:14:23.027 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-24T23:14:23.027 INFO:teuthology.orchestra.run.smithi190.stdout: python3-rados-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:14:23.028 INFO:teuthology.orchestra.run.smithi190.stdout: python3-rbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:14:23.028 INFO:teuthology.orchestra.run.smithi190.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-24T23:14:23.028 INFO:teuthology.orchestra.run.smithi190.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-24T23:14:23.028 INFO:teuthology.orchestra.run.smithi190.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-24T23:14:23.028 INFO:teuthology.orchestra.run.smithi190.stdout: python3-rgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:14:23.028 INFO:teuthology.orchestra.run.smithi190.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-24T23:14:23.028 INFO:teuthology.orchestra.run.smithi190.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-24T23:14:23.028 INFO:teuthology.orchestra.run.smithi190.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-24T23:14:23.028 INFO:teuthology.orchestra.run.smithi190.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-24T23:14:23.028 INFO:teuthology.orchestra.run.smithi190.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-24T23:14:23.028 INFO:teuthology.orchestra.run.smithi190.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-24T23:14:23.028 INFO:teuthology.orchestra.run.smithi190.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-24T23:14:23.028 INFO:teuthology.orchestra.run.smithi190.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-24T23:14:23.028 INFO:teuthology.orchestra.run.smithi190.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-24T23:14:23.028 INFO:teuthology.orchestra.run.smithi190.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-24T23:14:23.029 INFO:teuthology.orchestra.run.smithi190.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-24T23:14:23.029 INFO:teuthology.orchestra.run.smithi190.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-24T23:14:23.029 INFO:teuthology.orchestra.run.smithi190.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-24T23:14:23.029 INFO:teuthology.orchestra.run.smithi190.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-24T23:14:23.029 INFO:teuthology.orchestra.run.smithi190.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-24T23:14:23.029 INFO:teuthology.orchestra.run.smithi190.stdout: rbd-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:14:23.029 INFO:teuthology.orchestra.run.smithi190.stdout: rbd-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:14:23.029 INFO:teuthology.orchestra.run.smithi190.stdout: rbd-nbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:14:23.029 INFO:teuthology.orchestra.run.smithi190.stdout: re2-1:20211101-3.el9.x86_64 2024-04-24T23:14:23.029 INFO:teuthology.orchestra.run.smithi190.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-24T23:14:23.029 INFO:teuthology.orchestra.run.smithi190.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-24T23:14:23.029 INFO:teuthology.orchestra.run.smithi190.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-24T23:14:23.029 INFO:teuthology.orchestra.run.smithi190.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-24T23:14:23.030 INFO:teuthology.orchestra.run.smithi190.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-24T23:14:23.030 INFO:teuthology.orchestra.run.smithi190.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-24T23:14:23.030 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-04-24T23:14:23.030 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-04-24T23:14:23.402 DEBUG:teuthology.parallel:result is None 2024-04-24T23:14:26.783 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-2:19.0.0-2445.gd2c45d97.el9.x86_64 1/156 2024-04-24T23:14:26.783 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-base-2:19.0.0-2445.gd2c45d97.el9.x86_64 2/156 2024-04-24T23:14:26.783 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 3/156 2024-04-24T23:14:26.784 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 4/156 2024-04-24T23:14:26.784 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2445.gd2c45 5/156 2024-04-24T23:14:26.784 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-mds-2:19.0.0-2445.gd2c45d97.el9.x86_64 6/156 2024-04-24T23:14:26.784 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-mgr-2:19.0.0-2445.gd2c45d97.el9.x86_64 7/156 2024-04-24T23:14:26.784 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-mon-2:19.0.0-2445.gd2c45d97.el9.x86_64 8/156 2024-04-24T23:14:26.784 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-osd-2:19.0.0-2445.gd2c45d97.el9.x86_64 9/156 2024-04-24T23:14:26.784 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-radosgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 10/156 2024-04-24T23:14:26.784 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-selinux-2:19.0.0-2445.gd2c45d97.el9.x86_64 11/156 2024-04-24T23:14:26.784 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-test-2:19.0.0-2445.gd2c45d97.el9.x86_64 12/156 2024-04-24T23:14:26.784 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : cephfs-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 13/156 2024-04-24T23:14:26.784 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : cephfs-shell-2:19.0.0-2445.gd2c45d97.el9.x86_64 14/156 2024-04-24T23:14:26.784 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libcephfs-devel-2:19.0.0-2445.gd2c45d97.el9.x86_ 15/156 2024-04-24T23:14:26.784 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libcephfs2-2:19.0.0-2445.gd2c45d97.el9.x86_64 16/156 2024-04-24T23:14:26.784 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libcephsqlite-2:19.0.0-2445.gd2c45d97.el9.x86_64 17/156 2024-04-24T23:14:26.785 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : librados-devel-2:19.0.0-2445.gd2c45d97.el9.x86_6 18/156 2024-04-24T23:14:26.785 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libradosstriper1-2:19.0.0-2445.gd2c45d97.el9.x86 19/156 2024-04-24T23:14:26.785 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : librgw2-2:19.0.0-2445.gd2c45d97.el9.x86_64 20/156 2024-04-24T23:14:26.785 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2445.gd2c45d97.el 21/156 2024-04-24T23:14:26.785 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-ceph-common-2:19.0.0-2445.gd2c45d97.el9. 22/156 2024-04-24T23:14:26.785 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-cephfs-2:19.0.0-2445.gd2c45d97.el9.x86_6 23/156 2024-04-24T23:14:26.785 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-rados-2:19.0.0-2445.gd2c45d97.el9.x86_64 24/156 2024-04-24T23:14:26.785 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-rbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 25/156 2024-04-24T23:14:26.798 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-rgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 26/156 2024-04-24T23:14:26.798 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : rbd-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 27/156 2024-04-24T23:14:26.798 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : rbd-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 28/156 2024-04-24T23:14:26.798 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : rbd-nbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 29/156 2024-04-24T23:14:26.798 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2445.gd2c45d97. 30/156 2024-04-24T23:14:26.798 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2445.gd2c45d97.el9.noa 31/156 2024-04-24T23:14:26.798 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2445.gd2c45d97.el9.n 32/156 2024-04-24T23:14:26.798 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2445.gd2c 33/156 2024-04-24T23:14:26.798 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2445.gd2c45d97.el 34/156 2024-04-24T23:14:26.798 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2445.gd2c45d97.el9.noarch 35/156 2024-04-24T23:14:26.798 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2445.gd2c45d97.e 36/156 2024-04-24T23:14:26.799 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-volume-2:19.0.0-2445.gd2c45d97.el9.noarch 37/156 2024-04-24T23:14:26.799 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : cephadm-2:19.0.0-2445.gd2c45d97.el9.noarch 38/156 2024-04-24T23:14:26.799 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : cephfs-top-2:19.0.0-2445.gd2c45d97.el9.noarch 39/156 2024-04-24T23:14:26.799 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/156 2024-04-24T23:14:26.799 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/156 2024-04-24T23:14:26.799 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/156 2024-04-24T23:14:26.799 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/156 2024-04-24T23:14:26.799 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/156 2024-04-24T23:14:26.799 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/156 2024-04-24T23:14:26.799 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/156 2024-04-24T23:14:26.799 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/156 2024-04-24T23:14:26.799 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/156 2024-04-24T23:14:26.799 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/156 2024-04-24T23:14:26.799 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/156 2024-04-24T23:14:26.800 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/156 2024-04-24T23:14:26.800 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/156 2024-04-24T23:14:26.800 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/156 2024-04-24T23:14:26.800 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/156 2024-04-24T23:14:26.800 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/156 2024-04-24T23:14:26.800 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/156 2024-04-24T23:14:26.800 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/156 2024-04-24T23:14:26.800 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/156 2024-04-24T23:14:26.800 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/156 2024-04-24T23:14:26.800 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/156 2024-04-24T23:14:26.800 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/156 2024-04-24T23:14:26.801 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/156 2024-04-24T23:14:26.801 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/156 2024-04-24T23:14:26.801 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/156 2024-04-24T23:14:26.801 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/156 2024-04-24T23:14:26.801 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/156 2024-04-24T23:14:26.801 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/156 2024-04-24T23:14:26.801 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/156 2024-04-24T23:14:26.801 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/156 2024-04-24T23:14:26.801 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/156 2024-04-24T23:14:26.801 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/156 2024-04-24T23:14:26.801 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/156 2024-04-24T23:14:26.801 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/156 2024-04-24T23:14:26.801 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/156 2024-04-24T23:14:26.801 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/156 2024-04-24T23:14:26.802 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/156 2024-04-24T23:14:26.802 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/156 2024-04-24T23:14:26.802 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/156 2024-04-24T23:14:26.802 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/156 2024-04-24T23:14:26.802 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/156 2024-04-24T23:14:26.802 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/156 2024-04-24T23:14:26.802 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/156 2024-04-24T23:14:26.802 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/156 2024-04-24T23:14:26.802 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/156 2024-04-24T23:14:26.802 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/156 2024-04-24T23:14:26.802 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/156 2024-04-24T23:14:26.802 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/156 2024-04-24T23:14:26.803 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/156 2024-04-24T23:14:26.803 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/156 2024-04-24T23:14:26.803 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/156 2024-04-24T23:14:26.803 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/156 2024-04-24T23:14:26.803 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/156 2024-04-24T23:14:26.804 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/156 2024-04-24T23:14:26.804 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/156 2024-04-24T23:14:26.804 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/156 2024-04-24T23:14:26.804 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/156 2024-04-24T23:14:26.804 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/156 2024-04-24T23:14:26.804 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/156 2024-04-24T23:14:26.804 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/156 2024-04-24T23:14:26.805 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/156 2024-04-24T23:14:26.805 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/156 2024-04-24T23:14:26.805 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/156 2024-04-24T23:14:26.805 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/156 2024-04-24T23:14:26.805 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/156 2024-04-24T23:14:26.805 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/156 2024-04-24T23:14:26.805 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/156 2024-04-24T23:14:26.805 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/156 2024-04-24T23:14:26.805 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 108/156 2024-04-24T23:14:26.805 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 109/156 2024-04-24T23:14:26.806 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/156 2024-04-24T23:14:26.806 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/156 2024-04-24T23:14:26.806 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/156 2024-04-24T23:14:26.806 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 113/156 2024-04-24T23:14:26.806 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/156 2024-04-24T23:14:26.806 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/156 2024-04-24T23:14:26.806 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/156 2024-04-24T23:14:26.806 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/156 2024-04-24T23:14:26.806 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-24T23:14:26.807 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-24T23:14:26.807 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/156 2024-04-24T23:14:26.807 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/156 2024-04-24T23:14:26.807 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/156 2024-04-24T23:14:26.807 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/156 2024-04-24T23:14:26.807 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/156 2024-04-24T23:14:26.807 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/156 2024-04-24T23:14:26.807 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/156 2024-04-24T23:14:26.807 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/156 2024-04-24T23:14:26.808 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/156 2024-04-24T23:14:26.808 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/156 2024-04-24T23:14:26.808 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/156 2024-04-24T23:14:26.808 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/156 2024-04-24T23:14:26.808 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/156 2024-04-24T23:14:26.808 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/156 2024-04-24T23:14:26.808 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/156 2024-04-24T23:14:26.808 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/156 2024-04-24T23:14:26.809 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/156 2024-04-24T23:14:26.809 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/156 2024-04-24T23:14:26.809 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/156 2024-04-24T23:14:26.809 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/156 2024-04-24T23:14:26.809 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/156 2024-04-24T23:14:26.809 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/156 2024-04-24T23:14:26.809 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/156 2024-04-24T23:14:26.809 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/156 2024-04-24T23:14:26.809 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/156 2024-04-24T23:14:26.810 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/156 2024-04-24T23:14:26.810 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/156 2024-04-24T23:14:26.810 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/156 2024-04-24T23:14:26.810 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/156 2024-04-24T23:14:26.810 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : librados2-2:19.0.0-2445.gd2c45d97.el9.x86_64 149/156 2024-04-24T23:14:26.810 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/156 2024-04-24T23:14:26.810 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : librbd1-2:19.0.0-2445.gd2c45d97.el9.x86_64 151/156 2024-04-24T23:14:26.810 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/156 2024-04-24T23:14:26.811 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 153/156 2024-04-24T23:14:26.811 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-24T23:14:26.811 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 155/156 2024-04-24T23:14:29.272 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-24T23:14:29.272 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:14:29.273 INFO:teuthology.orchestra.run.smithi006.stdout:Upgraded: 2024-04-24T23:14:29.273 INFO:teuthology.orchestra.run.smithi006.stdout: librados2-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:14:29.273 INFO:teuthology.orchestra.run.smithi006.stdout: librbd1-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:14:29.273 INFO:teuthology.orchestra.run.smithi006.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-04-24T23:14:29.273 INFO:teuthology.orchestra.run.smithi006.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-04-24T23:14:29.273 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2024-04-24T23:14:29.273 INFO:teuthology.orchestra.run.smithi006.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-24T23:14:29.273 INFO:teuthology.orchestra.run.smithi006.stdout: automake-1.16.2-8.el9.noarch 2024-04-24T23:14:29.273 INFO:teuthology.orchestra.run.smithi006.stdout: bison-3.7.4-5.el9.x86_64 2024-04-24T23:14:29.273 INFO:teuthology.orchestra.run.smithi006.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-24T23:14:29.273 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:14:29.273 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-base-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:14:29.273 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:14:29.274 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:14:29.274 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-grafana-dashboards-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-24T23:14:29.274 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-immutable-object-cache-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:14:29.274 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mds-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:14:29.274 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:14:29.274 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-cephadm-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-24T23:14:29.274 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-dashboard-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-24T23:14:29.274 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-24T23:14:29.274 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-modules-core-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-24T23:14:29.274 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-rook-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-24T23:14:29.274 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mon-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:14:29.274 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-osd-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:14:29.274 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-prometheus-alerts-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-24T23:14:29.275 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-radosgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:14:29.275 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-selinux-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:14:29.275 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-test-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:14:29.275 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-volume-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-24T23:14:29.275 INFO:teuthology.orchestra.run.smithi006.stdout: cephadm-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-24T23:14:29.275 INFO:teuthology.orchestra.run.smithi006.stdout: cephfs-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:14:29.275 INFO:teuthology.orchestra.run.smithi006.stdout: cephfs-shell-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:14:29.275 INFO:teuthology.orchestra.run.smithi006.stdout: cephfs-top-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-24T23:14:29.275 INFO:teuthology.orchestra.run.smithi006.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-24T23:14:29.275 INFO:teuthology.orchestra.run.smithi006.stdout: flex-2.6.4-9.el9.x86_64 2024-04-24T23:14:29.275 INFO:teuthology.orchestra.run.smithi006.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-24T23:14:29.275 INFO:teuthology.orchestra.run.smithi006.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-24T23:14:29.275 INFO:teuthology.orchestra.run.smithi006.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-24T23:14:29.276 INFO:teuthology.orchestra.run.smithi006.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-24T23:14:29.276 INFO:teuthology.orchestra.run.smithi006.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-24T23:14:29.276 INFO:teuthology.orchestra.run.smithi006.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-24T23:14:29.276 INFO:teuthology.orchestra.run.smithi006.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-24T23:14:29.276 INFO:teuthology.orchestra.run.smithi006.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-24T23:14:29.276 INFO:teuthology.orchestra.run.smithi006.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-24T23:14:29.276 INFO:teuthology.orchestra.run.smithi006.stdout: libcephfs-devel-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:14:29.276 INFO:teuthology.orchestra.run.smithi006.stdout: libcephfs2-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:14:29.276 INFO:teuthology.orchestra.run.smithi006.stdout: libcephsqlite-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:14:29.276 INFO:teuthology.orchestra.run.smithi006.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-24T23:14:29.276 INFO:teuthology.orchestra.run.smithi006.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-24T23:14:29.276 INFO:teuthology.orchestra.run.smithi006.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-24T23:14:29.276 INFO:teuthology.orchestra.run.smithi006.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-24T23:14:29.277 INFO:teuthology.orchestra.run.smithi006.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-24T23:14:29.277 INFO:teuthology.orchestra.run.smithi006.stdout: librados-devel-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:14:29.277 INFO:teuthology.orchestra.run.smithi006.stdout: libradosstriper1-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:14:29.277 INFO:teuthology.orchestra.run.smithi006.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-24T23:14:29.277 INFO:teuthology.orchestra.run.smithi006.stdout: librgw2-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:14:29.277 INFO:teuthology.orchestra.run.smithi006.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-24T23:14:29.277 INFO:teuthology.orchestra.run.smithi006.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-24T23:14:29.277 INFO:teuthology.orchestra.run.smithi006.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-24T23:14:29.277 INFO:teuthology.orchestra.run.smithi006.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-24T23:14:29.277 INFO:teuthology.orchestra.run.smithi006.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-24T23:14:29.277 INFO:teuthology.orchestra.run.smithi006.stdout: lua-5.4.4-4.el9.x86_64 2024-04-24T23:14:29.277 INFO:teuthology.orchestra.run.smithi006.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-24T23:14:29.277 INFO:teuthology.orchestra.run.smithi006.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-24T23:14:29.278 INFO:teuthology.orchestra.run.smithi006.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-24T23:14:29.278 INFO:teuthology.orchestra.run.smithi006.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-24T23:14:29.278 INFO:teuthology.orchestra.run.smithi006.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-04-24T23:14:29.278 INFO:teuthology.orchestra.run.smithi006.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-24T23:14:29.278 INFO:teuthology.orchestra.run.smithi006.stdout: postgresql-13.14-1.el9.x86_64 2024-04-24T23:14:29.278 INFO:teuthology.orchestra.run.smithi006.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-24T23:14:29.278 INFO:teuthology.orchestra.run.smithi006.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-24T23:14:29.278 INFO:teuthology.orchestra.run.smithi006.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-24T23:14:29.278 INFO:teuthology.orchestra.run.smithi006.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-24T23:14:29.278 INFO:teuthology.orchestra.run.smithi006.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-24T23:14:29.278 INFO:teuthology.orchestra.run.smithi006.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-24T23:14:29.278 INFO:teuthology.orchestra.run.smithi006.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-24T23:14:29.278 INFO:teuthology.orchestra.run.smithi006.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-24T23:14:29.279 INFO:teuthology.orchestra.run.smithi006.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-24T23:14:29.279 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-24T23:14:29.279 INFO:teuthology.orchestra.run.smithi006.stdout: python3-ceph-argparse-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:14:29.279 INFO:teuthology.orchestra.run.smithi006.stdout: python3-ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:14:29.279 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cephfs-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:14:29.279 INFO:teuthology.orchestra.run.smithi006.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-24T23:14:29.279 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-24T23:14:29.279 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-24T23:14:29.279 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-24T23:14:29.279 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-24T23:14:29.279 INFO:teuthology.orchestra.run.smithi006.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-24T23:14:29.279 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-24T23:14:29.279 INFO:teuthology.orchestra.run.smithi006.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-24T23:14:29.279 INFO:teuthology.orchestra.run.smithi006.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-24T23:14:29.280 INFO:teuthology.orchestra.run.smithi006.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-24T23:14:29.280 INFO:teuthology.orchestra.run.smithi006.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-24T23:14:29.280 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-24T23:14:29.280 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-24T23:14:29.280 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-24T23:14:29.280 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-24T23:14:29.280 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-24T23:14:29.280 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-24T23:14:29.280 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-24T23:14:29.280 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-24T23:14:29.280 INFO:teuthology.orchestra.run.smithi006.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-24T23:14:29.280 INFO:teuthology.orchestra.run.smithi006.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-24T23:14:29.280 INFO:teuthology.orchestra.run.smithi006.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-24T23:14:29.280 INFO:teuthology.orchestra.run.smithi006.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-24T23:14:29.281 INFO:teuthology.orchestra.run.smithi006.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-24T23:14:29.281 INFO:teuthology.orchestra.run.smithi006.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-24T23:14:29.281 INFO:teuthology.orchestra.run.smithi006.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-24T23:14:29.281 INFO:teuthology.orchestra.run.smithi006.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-24T23:14:29.281 INFO:teuthology.orchestra.run.smithi006.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-24T23:14:29.281 INFO:teuthology.orchestra.run.smithi006.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-24T23:14:29.281 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-24T23:14:29.281 INFO:teuthology.orchestra.run.smithi006.stdout: python3-ply-3.11-14.el9.noarch 2024-04-24T23:14:29.281 INFO:teuthology.orchestra.run.smithi006.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-24T23:14:29.281 INFO:teuthology.orchestra.run.smithi006.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-24T23:14:29.281 INFO:teuthology.orchestra.run.smithi006.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-24T23:14:29.281 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-24T23:14:29.281 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-24T23:14:29.282 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-24T23:14:29.282 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-24T23:14:29.282 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-24T23:14:29.282 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-24T23:14:29.282 INFO:teuthology.orchestra.run.smithi006.stdout: python3-rados-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:14:29.282 INFO:teuthology.orchestra.run.smithi006.stdout: python3-rbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:14:29.282 INFO:teuthology.orchestra.run.smithi006.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-24T23:14:29.282 INFO:teuthology.orchestra.run.smithi006.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-24T23:14:29.282 INFO:teuthology.orchestra.run.smithi006.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-24T23:14:29.282 INFO:teuthology.orchestra.run.smithi006.stdout: python3-rgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:14:29.282 INFO:teuthology.orchestra.run.smithi006.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-24T23:14:29.282 INFO:teuthology.orchestra.run.smithi006.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-24T23:14:29.282 INFO:teuthology.orchestra.run.smithi006.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-24T23:14:29.282 INFO:teuthology.orchestra.run.smithi006.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-24T23:14:29.283 INFO:teuthology.orchestra.run.smithi006.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-24T23:14:29.283 INFO:teuthology.orchestra.run.smithi006.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-24T23:14:29.283 INFO:teuthology.orchestra.run.smithi006.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-24T23:14:29.283 INFO:teuthology.orchestra.run.smithi006.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-24T23:14:29.283 INFO:teuthology.orchestra.run.smithi006.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-24T23:14:29.283 INFO:teuthology.orchestra.run.smithi006.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-24T23:14:29.283 INFO:teuthology.orchestra.run.smithi006.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-24T23:14:29.283 INFO:teuthology.orchestra.run.smithi006.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-24T23:14:29.283 INFO:teuthology.orchestra.run.smithi006.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-24T23:14:29.283 INFO:teuthology.orchestra.run.smithi006.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-24T23:14:29.283 INFO:teuthology.orchestra.run.smithi006.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-24T23:14:29.283 INFO:teuthology.orchestra.run.smithi006.stdout: rbd-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:14:29.283 INFO:teuthology.orchestra.run.smithi006.stdout: rbd-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:14:29.283 INFO:teuthology.orchestra.run.smithi006.stdout: rbd-nbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:14:29.284 INFO:teuthology.orchestra.run.smithi006.stdout: re2-1:20211101-3.el9.x86_64 2024-04-24T23:14:29.284 INFO:teuthology.orchestra.run.smithi006.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-24T23:14:29.284 INFO:teuthology.orchestra.run.smithi006.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-24T23:14:29.284 INFO:teuthology.orchestra.run.smithi006.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-24T23:14:29.284 INFO:teuthology.orchestra.run.smithi006.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-24T23:14:29.284 INFO:teuthology.orchestra.run.smithi006.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-24T23:14:29.284 INFO:teuthology.orchestra.run.smithi006.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-24T23:14:29.284 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:14:29.284 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-04-24T23:14:29.897 DEBUG:teuthology.parallel:result is None 2024-04-24T23:14:29.898 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-24T23:14:30.056 DEBUG:teuthology.orchestra.run.smithi006:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-24T23:14:30.113 INFO:teuthology.orchestra.run.smithi006.stdout:19.0.0-2445.gd2c45d97.el9 2024-04-24T23:14:30.113 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2445.gd2c45d97.el9 2024-04-24T23:14:30.113 INFO:teuthology.task.install:The correct ceph version 19.0.0-2445.gd2c45d97 is installed. 2024-04-24T23:14:30.116 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-24T23:14:30.344 DEBUG:teuthology.orchestra.run.smithi190:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-24T23:14:30.369 INFO:teuthology.orchestra.run.smithi190.stdout:19.0.0-2445.gd2c45d97.el9 2024-04-24T23:14:30.369 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2445.gd2c45d97.el9 2024-04-24T23:14:30.369 INFO:teuthology.task.install:The correct ceph version 19.0.0-2445.gd2c45d97 is installed. 2024-04-24T23:14:30.371 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-24T23:14:30.371 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-04-24T23:14:30.371 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-24T23:14:30.444 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-04-24T23:14:30.445 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-24T23:14:30.475 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-24T23:14:30.475 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-04-24T23:14:30.475 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/usr/bin/daemon-helper 2024-04-24T23:14:30.550 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-24T23:14:30.661 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-04-24T23:14:30.661 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/usr/bin/daemon-helper 2024-04-24T23:14:30.690 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-24T23:14:30.756 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-24T23:14:30.757 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-04-24T23:14:30.757 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-24T23:14:30.844 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-24T23:14:30.949 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-04-24T23:14:30.950 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-24T23:14:30.976 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-24T23:14:31.043 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-24T23:14:31.043 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-04-24T23:14:31.043 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/usr/bin/stdin-killer 2024-04-24T23:14:31.117 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-24T23:14:31.225 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-04-24T23:14:31.226 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/usr/bin/stdin-killer 2024-04-24T23:14:31.252 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-24T23:14:31.320 INFO:teuthology.run_tasks:Running task ceph... 2024-04-24T23:14:31.405 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-04-24T23:14:31.405 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 777 /var/log/ceph 2024-04-24T23:14:31.411 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod 777 /var/log/ceph 2024-04-24T23:14:31.478 INFO:tasks.ceph:Disabling ceph logrotate... 2024-04-24T23:14:31.479 DEBUG:teuthology.orchestra.run.smithi006:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-24T23:14:31.483 DEBUG:teuthology.orchestra.run.smithi190:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-24T23:14:31.546 INFO:tasks.ceph:Creating extra log directories... 2024-04-24T23:14:31.547 DEBUG:teuthology.orchestra.run.smithi006:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-24T23:14:31.551 DEBUG:teuthology.orchestra.run.smithi190:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-24T23:14:31.620 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2024-04-24T23:14:31.620 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2024-04-24T23:14:31.620 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2024-04-24T23:14:31.621 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-04-24T23:14:31.621 DEBUG:teuthology.orchestra.run.smithi006:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-04-24T23:14:31.662 DEBUG:teuthology.orchestra.run.smithi006:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-04-24T23:14:31.860 DEBUG:teuthology.orchestra.run.smithi006:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-04-24T23:14:31.926 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-04-24T23:14:31.926 DEBUG:teuthology.orchestra.run.smithi190:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-04-24T23:14:31.944 DEBUG:teuthology.orchestra.run.smithi190:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-04-24T23:14:32.064 DEBUG:teuthology.orchestra.run.smithi190:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-04-24T23:14:32.095 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-04-24T23:14:32.096 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'client_shutdown_timeout': 120, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds oft prefetch dirfrags': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_dir_prefetch': True, 'mds_sleep_rank_change': 5000000.0, 'mon client directed command retry': 5, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds osd', 'ms inject socket failures': 2500, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'filestore flush min': 0, 'mon client directed command retry': 5, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds', 'ms inject socket failures': 2500, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'Replacing daemon mds', 'SLOW_OPS', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-04-24T23:14:32.096 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672187', 'branch': 'squid', 'description': 'fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/ffsb}}', 'email': 'ceph-qa@ceph.com', 'first_in_suite': False, 'job_id': '7672187', 'kernel': {'client': {'branch': 'testing'}, 'kdb': 1, 'sha1': 'distro'}, 'ktype': 'upstream', 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'mgrmodules': {'sequential': [{'print': 'Enabling mgr modules'}]}, 'name': 'teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'squid'}, 'ceph': {'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 5, 'session_timeout': 300}, 'conf': {'client': {'client mount timeout': 600, 'client_shutdown_timeout': 120, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds oft prefetch dirfrags': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_dir_prefetch': True, 'mds_sleep_rank_change': 5000000.0, 'mon client directed command retry': 5, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds osd', 'ms inject socket failures': 2500, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'filestore flush min': 0, 'mon client directed command retry': 5, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds', 'ms inject socket failures': 2500, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'Replacing daemon mds', 'SLOW_OPS', 'slow request'], 'sha1': 'd2c45d977142e024a59b68a8f1dce8f0b9840551'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'check-counter': {'counters': {'mds': ['mds.exported', 'mds.imported']}}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'd2c45d977142e024a59b68a8f1dce8f0b9840551'}}, 'kclient': {'syntax': 'v1'}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'squid', 'sha1': 'd2c45d977142e024a59b68a8f1dce8f0b9840551'}}, 'owner': 'scheduled_teuthology@teuthology', 'priority': 100, 'repo': 'https://git.ceph.com/ceph.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7']], 'seed': 8528, 'sha1': 'd2c45d977142e024a59b68a8f1dce8f0b9840551', 'sleep_before_teardown': 0, 'subset': '122/512', 'suite': 'fs', 'suite_branch': 'squid', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph_d2c45d977142e024a59b68a8f1dce8f0b9840551/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph.git', 'suite_sha1': 'd2c45d977142e024a59b68a8f1dce8f0b9840551', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi006.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGBKLf8afP4hWXBS+othavKra26AYnE5js76yYD3eC1YHMuoyt7cFszoUSA4h7yqlHM6zZArkdCBujyHtkcJk58=', 'smithi190.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBXNxoUEc6Gg1c7NEunVZBhXYLxME0phfSwBV3OmsDx9E5JQRWCdibkjfmVsoa2HLbGn80qNSwiVu0SWaYaV3Zo='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'client': {'branch': 'testing'}, 'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'd2c45d977142e024a59b68a8f1dce8f0b9840551'}}, {'ceph': {'create_rbd_pool': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 5, 'session_timeout': 300}, 'conf': {'client': {'client mount timeout': 600, 'client_shutdown_timeout': 120, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds oft prefetch dirfrags': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_dir_prefetch': True, 'mds_sleep_rank_change': 5000000.0, 'mon client directed command retry': 5, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds osd', 'ms inject socket failures': 2500, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'filestore flush min': 0, 'mon client directed command retry': 5, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds', 'ms inject socket failures': 2500, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'Replacing daemon mds', 'SLOW_OPS', 'slow request'], 'sha1': 'd2c45d977142e024a59b68a8f1dce8f0b9840551', 'cluster': 'ceph'}}, {'sequential': ['mgrmodules']}, {'kclient': None}, {'mds_thrash': None}, {'workunit': {'clients': {'all': ['suites/ffsb.sh']}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ["local function is_kupstream() return yaml.ktype == 'upstream' end", "local function is_kdistro() return yaml.ktype == 'distro' end", 'local function is_fuse() return false end', 'local function syntax_version() return yaml.overrides.kclient.syntax end']}, 'teuthology_branch': 'main', 'teuthology_sha1': '9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17', 'timestamp': '2024-04-24_21:24:02', 'tube': 'smithi', 'user': 'teuthology', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-04-24T23:14:32.096 DEBUG:teuthology.orchestra.run.smithi006:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-24T23:14:32.101 DEBUG:teuthology.orchestra.run.smithi190:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-24T23:14:32.146 DEBUG:teuthology.orchestra.run.smithi006:> sudo install -d -m0777 -- /var/run/ceph 2024-04-24T23:14:32.185 DEBUG:teuthology.orchestra.run.smithi190:> sudo install -d -m0777 -- /var/run/ceph 2024-04-24T23:14:32.246 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-04-24T23:14:32.246 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/scratch_devs of=/dev/stdout 2024-04-24T23:14:32.287 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-24T23:14:32.287 DEBUG:teuthology.orchestra.run.smithi006:> stat /dev/vg_nvme/lv_1 2024-04-24T23:14:32.370 INFO:teuthology.orchestra.run.smithi006.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-24T23:14:32.371 INFO:teuthology.orchestra.run.smithi006.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-24T23:14:32.371 INFO:teuthology.orchestra.run.smithi006.stdout:Device: 6h/6d Inode: 632 Links: 1 2024-04-24T23:14:32.371 INFO:teuthology.orchestra.run.smithi006.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-24T23:14:32.371 INFO:teuthology.orchestra.run.smithi006.stdout:Context: system_u:object_r:device_t:s0 2024-04-24T23:14:32.371 INFO:teuthology.orchestra.run.smithi006.stdout:Access: 2024-04-24 23:14:21.603702909 +0000 2024-04-24T23:14:32.371 INFO:teuthology.orchestra.run.smithi006.stdout:Modify: 2024-04-24 23:12:09.430979393 +0000 2024-04-24T23:14:32.371 INFO:teuthology.orchestra.run.smithi006.stdout:Change: 2024-04-24 23:12:09.430979393 +0000 2024-04-24T23:14:32.371 INFO:teuthology.orchestra.run.smithi006.stdout: Birth: 2024-04-24 23:12:09.430979393 +0000 2024-04-24T23:14:32.371 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-24T23:14:32.484 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records in 2024-04-24T23:14:32.484 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records out 2024-04-24T23:14:32.484 INFO:teuthology.orchestra.run.smithi006.stderr:512 bytes copied, 0.000475288 s, 1.1 MB/s 2024-04-24T23:14:32.484 DEBUG:teuthology.orchestra.run.smithi006:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-24T23:14:32.533 DEBUG:teuthology.orchestra.run.smithi006:> stat /dev/vg_nvme/lv_2 2024-04-24T23:14:32.573 INFO:teuthology.orchestra.run.smithi006.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-24T23:14:32.573 INFO:teuthology.orchestra.run.smithi006.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-24T23:14:32.573 INFO:teuthology.orchestra.run.smithi006.stdout:Device: 6h/6d Inode: 651 Links: 1 2024-04-24T23:14:32.573 INFO:teuthology.orchestra.run.smithi006.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-24T23:14:32.573 INFO:teuthology.orchestra.run.smithi006.stdout:Context: system_u:object_r:device_t:s0 2024-04-24T23:14:32.574 INFO:teuthology.orchestra.run.smithi006.stdout:Access: 2024-04-24 23:14:21.604702897 +0000 2024-04-24T23:14:32.574 INFO:teuthology.orchestra.run.smithi006.stdout:Modify: 2024-04-24 23:12:10.033972304 +0000 2024-04-24T23:14:32.574 INFO:teuthology.orchestra.run.smithi006.stdout:Change: 2024-04-24 23:12:10.033972304 +0000 2024-04-24T23:14:32.574 INFO:teuthology.orchestra.run.smithi006.stdout: Birth: 2024-04-24 23:12:10.033972304 +0000 2024-04-24T23:14:32.574 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-24T23:14:32.678 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records in 2024-04-24T23:14:32.679 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records out 2024-04-24T23:14:32.679 INFO:teuthology.orchestra.run.smithi006.stderr:512 bytes copied, 0.00038026 s, 1.3 MB/s 2024-04-24T23:14:32.681 DEBUG:teuthology.orchestra.run.smithi006:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-24T23:14:32.738 DEBUG:teuthology.orchestra.run.smithi006:> stat /dev/vg_nvme/lv_3 2024-04-24T23:14:32.784 INFO:teuthology.orchestra.run.smithi006.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-24T23:14:32.784 INFO:teuthology.orchestra.run.smithi006.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-24T23:14:32.784 INFO:teuthology.orchestra.run.smithi006.stdout:Device: 6h/6d Inode: 669 Links: 1 2024-04-24T23:14:32.785 INFO:teuthology.orchestra.run.smithi006.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-24T23:14:32.785 INFO:teuthology.orchestra.run.smithi006.stdout:Context: system_u:object_r:device_t:s0 2024-04-24T23:14:32.785 INFO:teuthology.orchestra.run.smithi006.stdout:Access: 2024-04-24 23:14:21.605702885 +0000 2024-04-24T23:14:32.785 INFO:teuthology.orchestra.run.smithi006.stdout:Modify: 2024-04-24 23:12:10.656964981 +0000 2024-04-24T23:14:32.785 INFO:teuthology.orchestra.run.smithi006.stdout:Change: 2024-04-24 23:12:10.656964981 +0000 2024-04-24T23:14:32.785 INFO:teuthology.orchestra.run.smithi006.stdout: Birth: 2024-04-24 23:12:10.656964981 +0000 2024-04-24T23:14:32.785 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-24T23:14:32.854 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records in 2024-04-24T23:14:32.855 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records out 2024-04-24T23:14:32.855 INFO:teuthology.orchestra.run.smithi006.stderr:512 bytes copied, 0.00034703 s, 1.5 MB/s 2024-04-24T23:14:32.858 DEBUG:teuthology.orchestra.run.smithi006:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-24T23:14:32.905 DEBUG:teuthology.orchestra.run.smithi006:> stat /dev/vg_nvme/lv_4 2024-04-24T23:14:32.986 INFO:teuthology.orchestra.run.smithi006.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-24T23:14:32.986 INFO:teuthology.orchestra.run.smithi006.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-24T23:14:32.986 INFO:teuthology.orchestra.run.smithi006.stdout:Device: 6h/6d Inode: 690 Links: 1 2024-04-24T23:14:32.986 INFO:teuthology.orchestra.run.smithi006.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-24T23:14:32.987 INFO:teuthology.orchestra.run.smithi006.stdout:Context: system_u:object_r:device_t:s0 2024-04-24T23:14:32.987 INFO:teuthology.orchestra.run.smithi006.stdout:Access: 2024-04-24 23:14:21.605702885 +0000 2024-04-24T23:14:32.987 INFO:teuthology.orchestra.run.smithi006.stdout:Modify: 2024-04-24 23:12:11.289957539 +0000 2024-04-24T23:14:32.987 INFO:teuthology.orchestra.run.smithi006.stdout:Change: 2024-04-24 23:12:11.289957539 +0000 2024-04-24T23:14:32.987 INFO:teuthology.orchestra.run.smithi006.stdout: Birth: 2024-04-24 23:12:11.289957539 +0000 2024-04-24T23:14:32.987 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-24T23:14:33.109 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records in 2024-04-24T23:14:33.110 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records out 2024-04-24T23:14:33.110 INFO:teuthology.orchestra.run.smithi006.stderr:512 bytes copied, 0.000400935 s, 1.3 MB/s 2024-04-24T23:14:33.114 DEBUG:teuthology.orchestra.run.smithi006:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-24T23:14:33.168 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-24T23:14:33.168 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-04-24T23:14:33.168 DEBUG:teuthology.orchestra.run.smithi190:> dd if=/scratch_devs of=/dev/stdout 2024-04-24T23:14:33.186 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-24T23:14:33.186 DEBUG:teuthology.orchestra.run.smithi190:> stat /dev/vg_nvme/lv_1 2024-04-24T23:14:33.244 INFO:teuthology.orchestra.run.smithi190.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-24T23:14:33.244 INFO:teuthology.orchestra.run.smithi190.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-24T23:14:33.244 INFO:teuthology.orchestra.run.smithi190.stdout:Device: 5h/5d Inode: 664 Links: 1 2024-04-24T23:14:33.244 INFO:teuthology.orchestra.run.smithi190.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-24T23:14:33.244 INFO:teuthology.orchestra.run.smithi190.stdout:Context: system_u:object_r:device_t:s0 2024-04-24T23:14:33.244 INFO:teuthology.orchestra.run.smithi190.stdout:Access: 2024-04-24 23:14:15.446158679 +0000 2024-04-24T23:14:33.244 INFO:teuthology.orchestra.run.smithi190.stdout:Modify: 2024-04-24 23:10:54.139232280 +0000 2024-04-24T23:14:33.245 INFO:teuthology.orchestra.run.smithi190.stdout:Change: 2024-04-24 23:10:54.139232280 +0000 2024-04-24T23:14:33.245 INFO:teuthology.orchestra.run.smithi190.stdout: Birth: 2024-04-24 23:10:54.139232280 +0000 2024-04-24T23:14:33.245 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-24T23:14:33.311 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records in 2024-04-24T23:14:33.311 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records out 2024-04-24T23:14:33.311 INFO:teuthology.orchestra.run.smithi190.stderr:512 bytes copied, 0.000276387 s, 1.9 MB/s 2024-04-24T23:14:33.312 DEBUG:teuthology.orchestra.run.smithi190:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-24T23:14:33.370 DEBUG:teuthology.orchestra.run.smithi190:> stat /dev/vg_nvme/lv_2 2024-04-24T23:14:33.427 INFO:teuthology.orchestra.run.smithi190.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-24T23:14:33.428 INFO:teuthology.orchestra.run.smithi190.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-24T23:14:33.428 INFO:teuthology.orchestra.run.smithi190.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-04-24T23:14:33.428 INFO:teuthology.orchestra.run.smithi190.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-24T23:14:33.428 INFO:teuthology.orchestra.run.smithi190.stdout:Context: system_u:object_r:device_t:s0 2024-04-24T23:14:33.428 INFO:teuthology.orchestra.run.smithi190.stdout:Access: 2024-04-24 23:14:15.446158679 +0000 2024-04-24T23:14:33.428 INFO:teuthology.orchestra.run.smithi190.stdout:Modify: 2024-04-24 23:10:54.481222525 +0000 2024-04-24T23:14:33.428 INFO:teuthology.orchestra.run.smithi190.stdout:Change: 2024-04-24 23:10:54.481222525 +0000 2024-04-24T23:14:33.428 INFO:teuthology.orchestra.run.smithi190.stdout: Birth: 2024-04-24 23:10:54.481222525 +0000 2024-04-24T23:14:33.428 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-24T23:14:33.492 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records in 2024-04-24T23:14:33.492 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records out 2024-04-24T23:14:33.492 INFO:teuthology.orchestra.run.smithi190.stderr:512 bytes copied, 0.000204598 s, 2.5 MB/s 2024-04-24T23:14:33.494 DEBUG:teuthology.orchestra.run.smithi190:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-24T23:14:33.552 DEBUG:teuthology.orchestra.run.smithi190:> stat /dev/vg_nvme/lv_3 2024-04-24T23:14:33.608 INFO:teuthology.orchestra.run.smithi190.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-24T23:14:33.608 INFO:teuthology.orchestra.run.smithi190.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-24T23:14:33.609 INFO:teuthology.orchestra.run.smithi190.stdout:Device: 5h/5d Inode: 702 Links: 1 2024-04-24T23:14:33.609 INFO:teuthology.orchestra.run.smithi190.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-24T23:14:33.609 INFO:teuthology.orchestra.run.smithi190.stdout:Context: system_u:object_r:device_t:s0 2024-04-24T23:14:33.609 INFO:teuthology.orchestra.run.smithi190.stdout:Access: 2024-04-24 23:14:15.446158679 +0000 2024-04-24T23:14:33.609 INFO:teuthology.orchestra.run.smithi190.stdout:Modify: 2024-04-24 23:10:54.828212627 +0000 2024-04-24T23:14:33.609 INFO:teuthology.orchestra.run.smithi190.stdout:Change: 2024-04-24 23:10:54.828212627 +0000 2024-04-24T23:14:33.609 INFO:teuthology.orchestra.run.smithi190.stdout: Birth: 2024-04-24 23:10:54.828212627 +0000 2024-04-24T23:14:33.609 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-24T23:14:33.672 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records in 2024-04-24T23:14:33.673 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records out 2024-04-24T23:14:33.673 INFO:teuthology.orchestra.run.smithi190.stderr:512 bytes copied, 0.000196413 s, 2.6 MB/s 2024-04-24T23:14:33.674 DEBUG:teuthology.orchestra.run.smithi190:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-24T23:14:33.731 DEBUG:teuthology.orchestra.run.smithi190:> stat /dev/vg_nvme/lv_4 2024-04-24T23:14:33.787 INFO:teuthology.orchestra.run.smithi190.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-24T23:14:33.787 INFO:teuthology.orchestra.run.smithi190.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-24T23:14:33.787 INFO:teuthology.orchestra.run.smithi190.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-04-24T23:14:33.787 INFO:teuthology.orchestra.run.smithi190.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-24T23:14:33.787 INFO:teuthology.orchestra.run.smithi190.stdout:Context: system_u:object_r:device_t:s0 2024-04-24T23:14:33.787 INFO:teuthology.orchestra.run.smithi190.stdout:Access: 2024-04-24 23:14:15.447158651 +0000 2024-04-24T23:14:33.788 INFO:teuthology.orchestra.run.smithi190.stdout:Modify: 2024-04-24 23:10:55.166202986 +0000 2024-04-24T23:14:33.788 INFO:teuthology.orchestra.run.smithi190.stdout:Change: 2024-04-24 23:10:55.166202986 +0000 2024-04-24T23:14:33.788 INFO:teuthology.orchestra.run.smithi190.stdout: Birth: 2024-04-24 23:10:55.166202986 +0000 2024-04-24T23:14:33.788 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-24T23:14:33.850 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records in 2024-04-24T23:14:33.851 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records out 2024-04-24T23:14:33.851 INFO:teuthology.orchestra.run.smithi190.stderr:512 bytes copied, 0.000182469 s, 2.8 MB/s 2024-04-24T23:14:33.852 DEBUG:teuthology.orchestra.run.smithi190:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-24T23:14:33.908 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-24T23:14:33.908 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi006.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi190.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-04-24T23:14:33.908 INFO:tasks.ceph:Generating config... 2024-04-24T23:14:33.910 INFO:tasks.ceph:[client] client mount timeout = 600 2024-04-24T23:14:33.910 INFO:tasks.ceph:[client] client_shutdown_timeout = 120 2024-04-24T23:14:33.910 INFO:tasks.ceph:[client] debug client = 20 2024-04-24T23:14:33.910 INFO:tasks.ceph:[client] debug ms = 1 2024-04-24T23:14:33.910 INFO:tasks.ceph:[client] rados mon op timeout = 900 2024-04-24T23:14:33.910 INFO:tasks.ceph:[client] rados osd op timeout = 900 2024-04-24T23:14:33.911 INFO:tasks.ceph:[global] ms die on skipped message = False 2024-04-24T23:14:33.911 INFO:tasks.ceph:[mds] debug mds = 20 2024-04-24T23:14:33.911 INFO:tasks.ceph:[mds] debug mds balancer = 20 2024-04-24T23:14:33.911 INFO:tasks.ceph:[mds] debug ms = 1 2024-04-24T23:14:33.911 INFO:tasks.ceph:[mds] mds bal fragment size max = 10000 2024-04-24T23:14:33.911 INFO:tasks.ceph:[mds] mds bal merge size = 5 2024-04-24T23:14:33.911 INFO:tasks.ceph:[mds] mds bal split bits = 3 2024-04-24T23:14:33.911 INFO:tasks.ceph:[mds] mds bal split size = 100 2024-04-24T23:14:33.911 INFO:tasks.ceph:[mds] mds debug frag = True 2024-04-24T23:14:33.911 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2024-04-24T23:14:33.911 INFO:tasks.ceph:[mds] mds oft prefetch dirfrags = True 2024-04-24T23:14:33.911 INFO:tasks.ceph:[mds] mds op complaint time = 180 2024-04-24T23:14:33.912 INFO:tasks.ceph:[mds] mds verify scatter = True 2024-04-24T23:14:33.912 INFO:tasks.ceph:[mds] mds_dir_prefetch = True 2024-04-24T23:14:33.912 INFO:tasks.ceph:[mds] mds_sleep_rank_change = 5000000.0 2024-04-24T23:14:33.912 INFO:tasks.ceph:[mds] mon client directed command retry = 5 2024-04-24T23:14:33.912 INFO:tasks.ceph:[mds] ms inject delay max = 1 2024-04-24T23:14:33.912 INFO:tasks.ceph:[mds] ms inject delay probability = 0.005 2024-04-24T23:14:33.912 INFO:tasks.ceph:[mds] ms inject delay type = client mds osd 2024-04-24T23:14:33.912 INFO:tasks.ceph:[mds] ms inject socket failures = 2500 2024-04-24T23:14:33.912 INFO:tasks.ceph:[mds] osd op complaint time = 180 2024-04-24T23:14:33.912 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2024-04-24T23:14:33.912 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2024-04-24T23:14:33.912 INFO:tasks.ceph:[mgr] debug client = 20 2024-04-24T23:14:33.912 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-04-24T23:14:33.912 INFO:tasks.ceph:[mgr] debug ms = 1 2024-04-24T23:14:33.912 INFO:tasks.ceph:[mon] debug mon = 20 2024-04-24T23:14:33.913 INFO:tasks.ceph:[mon] debug ms = 1 2024-04-24T23:14:33.913 INFO:tasks.ceph:[mon] debug paxos = 20 2024-04-24T23:14:33.913 INFO:tasks.ceph:[mon] mon down mkfs grace = 300 2024-04-24T23:14:33.913 INFO:tasks.ceph:[mon] mon op complaint time = 120 2024-04-24T23:14:33.913 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-04-24T23:14:33.913 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-04-24T23:14:33.913 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-04-24T23:14:33.913 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-04-24T23:14:33.913 INFO:tasks.ceph:[osd] debug ms = 1 2024-04-24T23:14:33.913 INFO:tasks.ceph:[osd] debug osd = 20 2024-04-24T23:14:33.913 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-04-24T23:14:33.913 INFO:tasks.ceph:[osd] filestore flush min = 0 2024-04-24T23:14:33.913 INFO:tasks.ceph:[osd] mon client directed command retry = 5 2024-04-24T23:14:33.913 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-04-24T23:14:33.913 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-04-24T23:14:33.914 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-04-24T23:14:33.914 INFO:tasks.ceph:[osd] ms inject delay max = 1 2024-04-24T23:14:33.914 INFO:tasks.ceph:[osd] ms inject delay probability = 0.005 2024-04-24T23:14:33.914 INFO:tasks.ceph:[osd] ms inject delay type = client mds 2024-04-24T23:14:33.914 INFO:tasks.ceph:[osd] ms inject socket failures = 2500 2024-04-24T23:14:33.914 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2024-04-24T23:14:33.914 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-04-24T23:14:33.914 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2024-04-24T23:14:33.914 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-04-24T23:14:33.914 INFO:tasks.ceph:[osd] osd op complaint time = 180 2024-04-24T23:14:33.914 INFO:tasks.ceph:Setting up mon.a... 2024-04-24T23:14:33.914 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-04-24T23:14:34.025 INFO:teuthology.orchestra.run.smithi006.stdout:creating /etc/ceph/ceph.keyring 2024-04-24T23:14:34.032 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-04-24T23:14:34.127 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-24T23:14:34.196 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.6'), ('mon.b', '172.21.15.190'), ('mon.c', '[v2:172.21.15.190:3301,v1:172.21.15.190:6790]')] 2024-04-24T23:14:34.196 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.6,172.21.15.190,[v2:172.21.15.190:3301,v1:172.21.15.190:6790]', 'ms die on skipped message': False}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'filestore flush min': 0, 'mon client directed command retry': 5, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds', 'ms inject socket failures': 2500, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug client': 20}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'client_shutdown_timeout': 120, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.e': {}, 'mds.b': {}, 'mds.d': {}, 'mds.f': {}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds oft prefetch dirfrags': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_dir_prefetch': True, 'mds_sleep_rank_change': 5000000.0, 'mon client directed command retry': 5, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds osd', 'ms inject socket failures': 2500, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2024-04-24T23:14:34.197 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-04-24T23:14:34.197 DEBUG:teuthology.orchestra.run.smithi006:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-04-24T23:14:34.237 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.6 --add b 172.21.15.190 --addv c '[v2:172.21.15.190:3301,v1:172.21.15.190:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2024-04-24T23:14:34.360 INFO:teuthology.orchestra.run.smithi006.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-04-24T23:14:34.360 INFO:teuthology.orchestra.run.smithi006.stdout:monmaptool: generated fsid 18bef68d-5e6d-4ec5-862b-b6f206e2588d 2024-04-24T23:14:34.360 INFO:teuthology.orchestra.run.smithi006.stdout:setting min_mon_release = quincy 2024-04-24T23:14:34.360 INFO:teuthology.orchestra.run.smithi006.stdout:epoch 0 2024-04-24T23:14:34.360 INFO:teuthology.orchestra.run.smithi006.stdout:fsid 18bef68d-5e6d-4ec5-862b-b6f206e2588d 2024-04-24T23:14:34.360 INFO:teuthology.orchestra.run.smithi006.stdout:last_changed 2024-04-24T23:14:34.359164+0000 2024-04-24T23:14:34.360 INFO:teuthology.orchestra.run.smithi006.stdout:created 2024-04-24T23:14:34.359164+0000 2024-04-24T23:14:34.360 INFO:teuthology.orchestra.run.smithi006.stdout:min_mon_release 17 (quincy) 2024-04-24T23:14:34.360 INFO:teuthology.orchestra.run.smithi006.stdout:election_strategy: 1 2024-04-24T23:14:34.360 INFO:teuthology.orchestra.run.smithi006.stdout:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2024-04-24T23:14:34.360 INFO:teuthology.orchestra.run.smithi006.stdout:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-04-24T23:14:34.360 INFO:teuthology.orchestra.run.smithi006.stdout:2: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.c 2024-04-24T23:14:34.361 INFO:teuthology.orchestra.run.smithi006.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-04-24T23:14:34.363 DEBUG:teuthology.orchestra.run.smithi006:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-04-24T23:14:34.404 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 18bef68d-5e6d-4ec5-862b-b6f206e2588d... 2024-04-24T23:14:34.405 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-24T23:14:34.451 DEBUG:teuthology.orchestra.run.smithi190:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-24T23:14:34.497 INFO:teuthology.orchestra.run.smithi190.stdout:[global] 2024-04-24T23:14:34.498 INFO:teuthology.orchestra.run.smithi190.stdout: chdir = "" 2024-04-24T23:14:34.498 INFO:teuthology.orchestra.run.smithi190.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-24T23:14:34.498 INFO:teuthology.orchestra.run.smithi190.stdout: auth supported = cephx 2024-04-24T23:14:34.498 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-04-24T23:14:34.498 INFO:teuthology.orchestra.run.smithi190.stdout: filestore xattr use omap = true 2024-04-24T23:14:34.498 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-04-24T23:14:34.498 INFO:teuthology.orchestra.run.smithi190.stdout: mon clock drift allowed = 1.000 2024-04-24T23:14:34.498 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-04-24T23:14:34.498 INFO:teuthology.orchestra.run.smithi190.stdout: osd crush chooseleaf type = 0 2024-04-24T23:14:34.498 INFO:teuthology.orchestra.run.smithi190.stdout: auth debug = true 2024-04-24T23:14:34.498 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-04-24T23:14:34.498 INFO:teuthology.orchestra.run.smithi190.stdout: ms die on old message = true 2024-04-24T23:14:34.498 INFO:teuthology.orchestra.run.smithi190.stdout: ms die on bug = true 2024-04-24T23:14:34.499 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-04-24T23:14:34.499 INFO:teuthology.orchestra.run.smithi190.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-24T23:14:34.499 INFO:teuthology.orchestra.run.smithi190.stdout: mon pg warn max object skew = 0 2024-04-24T23:14:34.499 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-04-24T23:14:34.499 INFO:teuthology.orchestra.run.smithi190.stdout: # disable pg_autoscaler by default for new pools 2024-04-24T23:14:34.499 INFO:teuthology.orchestra.run.smithi190.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-24T23:14:34.499 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-04-24T23:14:34.499 INFO:teuthology.orchestra.run.smithi190.stdout: osd pool default size = 2 2024-04-24T23:14:34.499 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-04-24T23:14:34.499 INFO:teuthology.orchestra.run.smithi190.stdout: mon osd allow primary affinity = true 2024-04-24T23:14:34.499 INFO:teuthology.orchestra.run.smithi190.stdout: mon osd allow pg remap = true 2024-04-24T23:14:34.499 INFO:teuthology.orchestra.run.smithi190.stdout: mon warn on legacy crush tunables = false 2024-04-24T23:14:34.499 INFO:teuthology.orchestra.run.smithi190.stdout: mon warn on crush straw calc version zero = false 2024-04-24T23:14:34.499 INFO:teuthology.orchestra.run.smithi190.stdout: mon warn on no sortbitwise = false 2024-04-24T23:14:34.499 INFO:teuthology.orchestra.run.smithi190.stdout: mon warn on osd down out interval zero = false 2024-04-24T23:14:34.500 INFO:teuthology.orchestra.run.smithi190.stdout: mon warn on too few osds = false 2024-04-24T23:14:34.500 INFO:teuthology.orchestra.run.smithi190.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-24T23:14:34.500 INFO:teuthology.orchestra.run.smithi190.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-24T23:14:34.500 INFO:teuthology.orchestra.run.smithi190.stdout: mon_allow_pool_size_one = true 2024-04-24T23:14:34.500 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-04-24T23:14:34.500 INFO:teuthology.orchestra.run.smithi190.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-24T23:14:34.500 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-04-24T23:14:34.500 INFO:teuthology.orchestra.run.smithi190.stdout: osd default data pool replay window = 5 2024-04-24T23:14:34.500 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-04-24T23:14:34.500 INFO:teuthology.orchestra.run.smithi190.stdout: mon allow pool delete = true 2024-04-24T23:14:34.500 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-04-24T23:14:34.500 INFO:teuthology.orchestra.run.smithi190.stdout: mon cluster log file level = debug 2024-04-24T23:14:34.500 INFO:teuthology.orchestra.run.smithi190.stdout: debug asserts on shutdown = true 2024-04-24T23:14:34.500 INFO:teuthology.orchestra.run.smithi190.stdout: mon health detail to clog = false 2024-04-24T23:14:34.500 INFO:teuthology.orchestra.run.smithi190.stdout: mon host = "172.21.15.6,172.21.15.190,[v2:172.21.15.190:3301,v1:172.21.15.190:6790]" 2024-04-24T23:14:34.500 INFO:teuthology.orchestra.run.smithi190.stdout: ms die on skipped message = False 2024-04-24T23:14:34.501 INFO:teuthology.orchestra.run.smithi190.stdout: fsid = 18bef68d-5e6d-4ec5-862b-b6f206e2588d 2024-04-24T23:14:34.501 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-04-24T23:14:34.501 INFO:teuthology.orchestra.run.smithi190.stdout:[osd] 2024-04-24T23:14:34.501 INFO:teuthology.orchestra.run.smithi190.stdout: osd journal size = 100 2024-04-24T23:14:34.501 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-04-24T23:14:34.501 INFO:teuthology.orchestra.run.smithi190.stdout: osd scrub load threshold = 5.0 2024-04-24T23:14:34.501 INFO:teuthology.orchestra.run.smithi190.stdout: osd scrub max interval = 600 2024-04-24T23:14:34.501 INFO:teuthology.orchestra.run.smithi190.stdout: osd mclock profile = high_recovery_ops 2024-04-24T23:14:34.501 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-04-24T23:14:34.501 INFO:teuthology.orchestra.run.smithi190.stdout: osd recover clone overlap = true 2024-04-24T23:14:34.501 INFO:teuthology.orchestra.run.smithi190.stdout: osd recovery max chunk = 1048576 2024-04-24T23:14:34.501 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-04-24T23:14:34.501 INFO:teuthology.orchestra.run.smithi190.stdout: osd debug shutdown = true 2024-04-24T23:14:34.501 INFO:teuthology.orchestra.run.smithi190.stdout: osd debug op order = true 2024-04-24T23:14:34.501 INFO:teuthology.orchestra.run.smithi190.stdout: osd debug verify stray on activate = true 2024-04-24T23:14:34.502 INFO:teuthology.orchestra.run.smithi190.stdout: osd debug trim objects = true 2024-04-24T23:14:34.502 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-04-24T23:14:34.502 INFO:teuthology.orchestra.run.smithi190.stdout: osd open classes on start = true 2024-04-24T23:14:34.502 INFO:teuthology.orchestra.run.smithi190.stdout: osd debug pg log writeout = true 2024-04-24T23:14:34.502 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-04-24T23:14:34.502 INFO:teuthology.orchestra.run.smithi190.stdout: osd deep scrub update digest min age = 30 2024-04-24T23:14:34.502 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-04-24T23:14:34.502 INFO:teuthology.orchestra.run.smithi190.stdout: osd map max advance = 10 2024-04-24T23:14:34.502 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-04-24T23:14:34.502 INFO:teuthology.orchestra.run.smithi190.stdout: journal zero on create = true 2024-04-24T23:14:34.502 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-04-24T23:14:34.502 INFO:teuthology.orchestra.run.smithi190.stdout: filestore ondisk finisher threads = 3 2024-04-24T23:14:34.502 INFO:teuthology.orchestra.run.smithi190.stdout: filestore apply finisher threads = 3 2024-04-24T23:14:34.502 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-04-24T23:14:34.502 INFO:teuthology.orchestra.run.smithi190.stdout: bdev debug aio = true 2024-04-24T23:14:34.503 INFO:teuthology.orchestra.run.smithi190.stdout: osd debug misdirected ops = true 2024-04-24T23:14:34.503 INFO:teuthology.orchestra.run.smithi190.stdout: bluestore block size = 96636764160 2024-04-24T23:14:34.503 INFO:teuthology.orchestra.run.smithi190.stdout: bluestore fsck on mount = True 2024-04-24T23:14:34.503 INFO:teuthology.orchestra.run.smithi190.stdout: debug bluefs = 20 2024-04-24T23:14:34.503 INFO:teuthology.orchestra.run.smithi190.stdout: debug bluestore = 20 2024-04-24T23:14:34.503 INFO:teuthology.orchestra.run.smithi190.stdout: debug ms = 1 2024-04-24T23:14:34.503 INFO:teuthology.orchestra.run.smithi190.stdout: debug osd = 20 2024-04-24T23:14:34.503 INFO:teuthology.orchestra.run.smithi190.stdout: debug rocksdb = 10 2024-04-24T23:14:34.503 INFO:teuthology.orchestra.run.smithi190.stdout: filestore flush min = 0 2024-04-24T23:14:34.503 INFO:teuthology.orchestra.run.smithi190.stdout: mon client directed command retry = 5 2024-04-24T23:14:34.503 INFO:teuthology.orchestra.run.smithi190.stdout: mon osd backfillfull_ratio = 0.85 2024-04-24T23:14:34.503 INFO:teuthology.orchestra.run.smithi190.stdout: mon osd full ratio = 0.9 2024-04-24T23:14:34.503 INFO:teuthology.orchestra.run.smithi190.stdout: mon osd nearfull ratio = 0.8 2024-04-24T23:14:34.503 INFO:teuthology.orchestra.run.smithi190.stdout: ms inject delay max = 1 2024-04-24T23:14:34.503 INFO:teuthology.orchestra.run.smithi190.stdout: ms inject delay probability = 0.005 2024-04-24T23:14:34.503 INFO:teuthology.orchestra.run.smithi190.stdout: ms inject delay type = client mds 2024-04-24T23:14:34.504 INFO:teuthology.orchestra.run.smithi190.stdout: ms inject socket failures = 2500 2024-04-24T23:14:34.504 INFO:teuthology.orchestra.run.smithi190.stdout: osd blocked scrub grace period = 3600 2024-04-24T23:14:34.504 INFO:teuthology.orchestra.run.smithi190.stdout: osd failsafe full ratio = 0.95 2024-04-24T23:14:34.504 INFO:teuthology.orchestra.run.smithi190.stdout: osd max markdown count = 1000 2024-04-24T23:14:34.504 INFO:teuthology.orchestra.run.smithi190.stdout: osd objectstore = bluestore 2024-04-24T23:14:34.504 INFO:teuthology.orchestra.run.smithi190.stdout: osd op complaint time = 180 2024-04-24T23:14:34.504 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-04-24T23:14:34.504 INFO:teuthology.orchestra.run.smithi190.stdout:[mgr] 2024-04-24T23:14:34.504 INFO:teuthology.orchestra.run.smithi190.stdout: debug ms = 1 2024-04-24T23:14:34.504 INFO:teuthology.orchestra.run.smithi190.stdout: debug mgr = 20 2024-04-24T23:14:34.504 INFO:teuthology.orchestra.run.smithi190.stdout: debug mon = 20 2024-04-24T23:14:34.504 INFO:teuthology.orchestra.run.smithi190.stdout: debug auth = 20 2024-04-24T23:14:34.504 INFO:teuthology.orchestra.run.smithi190.stdout: mon reweight min pgs per osd = 4 2024-04-24T23:14:34.504 INFO:teuthology.orchestra.run.smithi190.stdout: mon reweight min bytes per osd = 10 2024-04-24T23:14:34.504 INFO:teuthology.orchestra.run.smithi190.stdout: mgr/telemetry/nag = false 2024-04-24T23:14:34.505 INFO:teuthology.orchestra.run.smithi190.stdout: debug client = 20 2024-04-24T23:14:34.505 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-04-24T23:14:34.505 INFO:teuthology.orchestra.run.smithi190.stdout:[mon] 2024-04-24T23:14:34.505 INFO:teuthology.orchestra.run.smithi190.stdout: debug ms = 1 2024-04-24T23:14:34.505 INFO:teuthology.orchestra.run.smithi190.stdout: debug mon = 20 2024-04-24T23:14:34.505 INFO:teuthology.orchestra.run.smithi190.stdout: debug paxos = 20 2024-04-24T23:14:34.505 INFO:teuthology.orchestra.run.smithi190.stdout: debug auth = 20 2024-04-24T23:14:34.505 INFO:teuthology.orchestra.run.smithi190.stdout: mon data avail warn = 5 2024-04-24T23:14:34.505 INFO:teuthology.orchestra.run.smithi190.stdout: mon mgr mkfs grace = 240 2024-04-24T23:14:34.505 INFO:teuthology.orchestra.run.smithi190.stdout: mon reweight min pgs per osd = 4 2024-04-24T23:14:34.505 INFO:teuthology.orchestra.run.smithi190.stdout: mon osd reporter subtree level = osd 2024-04-24T23:14:34.505 INFO:teuthology.orchestra.run.smithi190.stdout: mon osd prime pg temp = true 2024-04-24T23:14:34.505 INFO:teuthology.orchestra.run.smithi190.stdout: mon reweight min bytes per osd = 10 2024-04-24T23:14:34.505 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-04-24T23:14:34.505 INFO:teuthology.orchestra.run.smithi190.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-24T23:14:34.505 INFO:teuthology.orchestra.run.smithi190.stdout: auth mon ticket ttl = 660 # 11m 2024-04-24T23:14:34.506 INFO:teuthology.orchestra.run.smithi190.stdout: auth service ticket ttl = 240 # 4m 2024-04-24T23:14:34.506 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-04-24T23:14:34.506 INFO:teuthology.orchestra.run.smithi190.stdout: # don't complain about insecure global_id in the test suite 2024-04-24T23:14:34.506 INFO:teuthology.orchestra.run.smithi190.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-24T23:14:34.506 INFO:teuthology.orchestra.run.smithi190.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-24T23:14:34.506 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-04-24T23:14:34.506 INFO:teuthology.orchestra.run.smithi190.stdout: # 1m isn't quite enough 2024-04-24T23:14:34.506 INFO:teuthology.orchestra.run.smithi190.stdout: mon_down_mkfs_grace = 2m 2024-04-24T23:14:34.506 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-04-24T23:14:34.506 INFO:teuthology.orchestra.run.smithi190.stdout: mon_warn_on_filestore_osds = false 2024-04-24T23:14:34.506 INFO:teuthology.orchestra.run.smithi190.stdout: mon down mkfs grace = 300 2024-04-24T23:14:34.506 INFO:teuthology.orchestra.run.smithi190.stdout: mon op complaint time = 120 2024-04-24T23:14:34.506 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-04-24T23:14:34.506 INFO:teuthology.orchestra.run.smithi190.stdout:[client] 2024-04-24T23:14:34.506 INFO:teuthology.orchestra.run.smithi190.stdout: rgw cache enabled = true 2024-04-24T23:14:34.507 INFO:teuthology.orchestra.run.smithi190.stdout: rgw enable ops log = true 2024-04-24T23:14:34.507 INFO:teuthology.orchestra.run.smithi190.stdout: rgw enable usage log = true 2024-04-24T23:14:34.507 INFO:teuthology.orchestra.run.smithi190.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-24T23:14:34.507 INFO:teuthology.orchestra.run.smithi190.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-24T23:14:34.507 INFO:teuthology.orchestra.run.smithi190.stdout: client mount timeout = 600 2024-04-24T23:14:34.507 INFO:teuthology.orchestra.run.smithi190.stdout: client_shutdown_timeout = 120 2024-04-24T23:14:34.507 INFO:teuthology.orchestra.run.smithi190.stdout: debug client = 20 2024-04-24T23:14:34.507 INFO:teuthology.orchestra.run.smithi190.stdout: debug ms = 1 2024-04-24T23:14:34.507 INFO:teuthology.orchestra.run.smithi190.stdout: rados mon op timeout = 900 2024-04-24T23:14:34.507 INFO:teuthology.orchestra.run.smithi190.stdout: rados osd op timeout = 900 2024-04-24T23:14:34.507 INFO:teuthology.orchestra.run.smithi190.stdout:[mon.a] 2024-04-24T23:14:34.507 INFO:teuthology.orchestra.run.smithi190.stdout:[mon.b] 2024-04-24T23:14:34.507 INFO:teuthology.orchestra.run.smithi190.stdout:[mon.c] 2024-04-24T23:14:34.507 INFO:teuthology.orchestra.run.smithi190.stdout:[mds.a] 2024-04-24T23:14:34.507 INFO:teuthology.orchestra.run.smithi190.stdout:[mds.c] 2024-04-24T23:14:34.507 INFO:teuthology.orchestra.run.smithi190.stdout:[mds.e] 2024-04-24T23:14:34.508 INFO:teuthology.orchestra.run.smithi190.stdout:[mds.b] 2024-04-24T23:14:34.508 INFO:teuthology.orchestra.run.smithi190.stdout:[mds.d] 2024-04-24T23:14:34.508 INFO:teuthology.orchestra.run.smithi190.stdout:[mds.f] 2024-04-24T23:14:34.508 INFO:teuthology.orchestra.run.smithi190.stdout:[mds] 2024-04-24T23:14:34.508 INFO:teuthology.orchestra.run.smithi190.stdout: debug mds = 20 2024-04-24T23:14:34.508 INFO:teuthology.orchestra.run.smithi190.stdout: debug mds balancer = 20 2024-04-24T23:14:34.508 INFO:teuthology.orchestra.run.smithi190.stdout: debug ms = 1 2024-04-24T23:14:34.508 INFO:teuthology.orchestra.run.smithi190.stdout: mds bal fragment size max = 10000 2024-04-24T23:14:34.508 INFO:teuthology.orchestra.run.smithi190.stdout: mds bal merge size = 5 2024-04-24T23:14:34.508 INFO:teuthology.orchestra.run.smithi190.stdout: mds bal split bits = 3 2024-04-24T23:14:34.508 INFO:teuthology.orchestra.run.smithi190.stdout: mds bal split size = 100 2024-04-24T23:14:34.508 INFO:teuthology.orchestra.run.smithi190.stdout: mds debug frag = True 2024-04-24T23:14:34.508 INFO:teuthology.orchestra.run.smithi190.stdout: mds debug scatterstat = True 2024-04-24T23:14:34.508 INFO:teuthology.orchestra.run.smithi190.stdout: mds oft prefetch dirfrags = True 2024-04-24T23:14:34.508 INFO:teuthology.orchestra.run.smithi190.stdout: mds op complaint time = 180 2024-04-24T23:14:34.508 INFO:teuthology.orchestra.run.smithi190.stdout: mds verify scatter = True 2024-04-24T23:14:34.509 INFO:teuthology.orchestra.run.smithi190.stdout: mds_dir_prefetch = True 2024-04-24T23:14:34.509 INFO:teuthology.orchestra.run.smithi190.stdout: mds_sleep_rank_change = 5000000.0 2024-04-24T23:14:34.509 INFO:teuthology.orchestra.run.smithi190.stdout: mon client directed command retry = 5 2024-04-24T23:14:34.509 INFO:teuthology.orchestra.run.smithi190.stdout: ms inject delay max = 1 2024-04-24T23:14:34.509 INFO:teuthology.orchestra.run.smithi190.stdout: ms inject delay probability = 0.005 2024-04-24T23:14:34.509 INFO:teuthology.orchestra.run.smithi190.stdout: ms inject delay type = client mds osd 2024-04-24T23:14:34.509 INFO:teuthology.orchestra.run.smithi190.stdout: ms inject socket failures = 2500 2024-04-24T23:14:34.509 INFO:teuthology.orchestra.run.smithi190.stdout: osd op complaint time = 180 2024-04-24T23:14:34.509 INFO:teuthology.orchestra.run.smithi190.stdout: rados mon op timeout = 900 2024-04-24T23:14:34.509 INFO:teuthology.orchestra.run.smithi190.stdout: rados osd op timeout = 900 2024-04-24T23:14:34.562 INFO:teuthology.orchestra.run.smithi006.stdout:[global] 2024-04-24T23:14:34.563 INFO:teuthology.orchestra.run.smithi006.stdout: chdir = "" 2024-04-24T23:14:34.563 INFO:teuthology.orchestra.run.smithi006.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-24T23:14:34.563 INFO:teuthology.orchestra.run.smithi006.stdout: auth supported = cephx 2024-04-24T23:14:34.563 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:14:34.563 INFO:teuthology.orchestra.run.smithi006.stdout: filestore xattr use omap = true 2024-04-24T23:14:34.563 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:14:34.563 INFO:teuthology.orchestra.run.smithi006.stdout: mon clock drift allowed = 1.000 2024-04-24T23:14:34.563 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:14:34.563 INFO:teuthology.orchestra.run.smithi006.stdout: osd crush chooseleaf type = 0 2024-04-24T23:14:34.563 INFO:teuthology.orchestra.run.smithi006.stdout: auth debug = true 2024-04-24T23:14:34.563 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:14:34.563 INFO:teuthology.orchestra.run.smithi006.stdout: ms die on old message = true 2024-04-24T23:14:34.564 INFO:teuthology.orchestra.run.smithi006.stdout: ms die on bug = true 2024-04-24T23:14:34.564 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:14:34.564 INFO:teuthology.orchestra.run.smithi006.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-24T23:14:34.564 INFO:teuthology.orchestra.run.smithi006.stdout: mon pg warn max object skew = 0 2024-04-24T23:14:34.564 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:14:34.564 INFO:teuthology.orchestra.run.smithi006.stdout: # disable pg_autoscaler by default for new pools 2024-04-24T23:14:34.564 INFO:teuthology.orchestra.run.smithi006.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-24T23:14:34.564 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:14:34.564 INFO:teuthology.orchestra.run.smithi006.stdout: osd pool default size = 2 2024-04-24T23:14:34.564 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:14:34.564 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd allow primary affinity = true 2024-04-24T23:14:34.564 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd allow pg remap = true 2024-04-24T23:14:34.564 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on legacy crush tunables = false 2024-04-24T23:14:34.564 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on crush straw calc version zero = false 2024-04-24T23:14:34.565 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on no sortbitwise = false 2024-04-24T23:14:34.565 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on osd down out interval zero = false 2024-04-24T23:14:34.565 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on too few osds = false 2024-04-24T23:14:34.565 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-24T23:14:34.565 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-24T23:14:34.565 INFO:teuthology.orchestra.run.smithi006.stdout: mon_allow_pool_size_one = true 2024-04-24T23:14:34.565 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:14:34.565 INFO:teuthology.orchestra.run.smithi006.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-24T23:14:34.565 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:14:34.565 INFO:teuthology.orchestra.run.smithi006.stdout: osd default data pool replay window = 5 2024-04-24T23:14:34.565 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:14:34.565 INFO:teuthology.orchestra.run.smithi006.stdout: mon allow pool delete = true 2024-04-24T23:14:34.565 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:14:34.565 INFO:teuthology.orchestra.run.smithi006.stdout: mon cluster log file level = debug 2024-04-24T23:14:34.565 INFO:teuthology.orchestra.run.smithi006.stdout: debug asserts on shutdown = true 2024-04-24T23:14:34.566 INFO:teuthology.orchestra.run.smithi006.stdout: mon health detail to clog = false 2024-04-24T23:14:34.566 INFO:teuthology.orchestra.run.smithi006.stdout: mon host = "172.21.15.6,172.21.15.190,[v2:172.21.15.190:3301,v1:172.21.15.190:6790]" 2024-04-24T23:14:34.566 INFO:teuthology.orchestra.run.smithi006.stdout: ms die on skipped message = False 2024-04-24T23:14:34.566 INFO:teuthology.orchestra.run.smithi006.stdout: fsid = 18bef68d-5e6d-4ec5-862b-b6f206e2588d 2024-04-24T23:14:34.566 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:14:34.566 INFO:teuthology.orchestra.run.smithi006.stdout:[osd] 2024-04-24T23:14:34.566 INFO:teuthology.orchestra.run.smithi006.stdout: osd journal size = 100 2024-04-24T23:14:34.566 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:14:34.566 INFO:teuthology.orchestra.run.smithi006.stdout: osd scrub load threshold = 5.0 2024-04-24T23:14:34.566 INFO:teuthology.orchestra.run.smithi006.stdout: osd scrub max interval = 600 2024-04-24T23:14:34.566 INFO:teuthology.orchestra.run.smithi006.stdout: osd mclock profile = high_recovery_ops 2024-04-24T23:14:34.566 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:14:34.566 INFO:teuthology.orchestra.run.smithi006.stdout: osd recover clone overlap = true 2024-04-24T23:14:34.566 INFO:teuthology.orchestra.run.smithi006.stdout: osd recovery max chunk = 1048576 2024-04-24T23:14:34.567 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:14:34.567 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug shutdown = true 2024-04-24T23:14:34.567 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug op order = true 2024-04-24T23:14:34.567 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug verify stray on activate = true 2024-04-24T23:14:34.567 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug trim objects = true 2024-04-24T23:14:34.567 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:14:34.567 INFO:teuthology.orchestra.run.smithi006.stdout: osd open classes on start = true 2024-04-24T23:14:34.567 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug pg log writeout = true 2024-04-24T23:14:34.567 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:14:34.567 INFO:teuthology.orchestra.run.smithi006.stdout: osd deep scrub update digest min age = 30 2024-04-24T23:14:34.567 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:14:34.567 INFO:teuthology.orchestra.run.smithi006.stdout: osd map max advance = 10 2024-04-24T23:14:34.567 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:14:34.567 INFO:teuthology.orchestra.run.smithi006.stdout: journal zero on create = true 2024-04-24T23:14:34.568 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:14:34.568 INFO:teuthology.orchestra.run.smithi006.stdout: filestore ondisk finisher threads = 3 2024-04-24T23:14:34.568 INFO:teuthology.orchestra.run.smithi006.stdout: filestore apply finisher threads = 3 2024-04-24T23:14:34.568 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:14:34.568 INFO:teuthology.orchestra.run.smithi006.stdout: bdev debug aio = true 2024-04-24T23:14:34.568 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug misdirected ops = true 2024-04-24T23:14:34.568 INFO:teuthology.orchestra.run.smithi006.stdout: bluestore block size = 96636764160 2024-04-24T23:14:34.568 INFO:teuthology.orchestra.run.smithi006.stdout: bluestore fsck on mount = True 2024-04-24T23:14:34.568 INFO:teuthology.orchestra.run.smithi006.stdout: debug bluefs = 20 2024-04-24T23:14:34.568 INFO:teuthology.orchestra.run.smithi006.stdout: debug bluestore = 20 2024-04-24T23:14:34.568 INFO:teuthology.orchestra.run.smithi006.stdout: debug ms = 1 2024-04-24T23:14:34.568 INFO:teuthology.orchestra.run.smithi006.stdout: debug osd = 20 2024-04-24T23:14:34.568 INFO:teuthology.orchestra.run.smithi006.stdout: debug rocksdb = 10 2024-04-24T23:14:34.568 INFO:teuthology.orchestra.run.smithi006.stdout: filestore flush min = 0 2024-04-24T23:14:34.568 INFO:teuthology.orchestra.run.smithi006.stdout: mon client directed command retry = 5 2024-04-24T23:14:34.569 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd backfillfull_ratio = 0.85 2024-04-24T23:14:34.569 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd full ratio = 0.9 2024-04-24T23:14:34.569 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd nearfull ratio = 0.8 2024-04-24T23:14:34.569 INFO:teuthology.orchestra.run.smithi006.stdout: ms inject delay max = 1 2024-04-24T23:14:34.569 INFO:teuthology.orchestra.run.smithi006.stdout: ms inject delay probability = 0.005 2024-04-24T23:14:34.569 INFO:teuthology.orchestra.run.smithi006.stdout: ms inject delay type = client mds 2024-04-24T23:14:34.569 INFO:teuthology.orchestra.run.smithi006.stdout: ms inject socket failures = 2500 2024-04-24T23:14:34.569 INFO:teuthology.orchestra.run.smithi006.stdout: osd blocked scrub grace period = 3600 2024-04-24T23:14:34.569 INFO:teuthology.orchestra.run.smithi006.stdout: osd failsafe full ratio = 0.95 2024-04-24T23:14:34.569 INFO:teuthology.orchestra.run.smithi006.stdout: osd max markdown count = 1000 2024-04-24T23:14:34.569 INFO:teuthology.orchestra.run.smithi006.stdout: osd objectstore = bluestore 2024-04-24T23:14:34.569 INFO:teuthology.orchestra.run.smithi006.stdout: osd op complaint time = 180 2024-04-24T23:14:34.569 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:14:34.569 INFO:teuthology.orchestra.run.smithi006.stdout:[mgr] 2024-04-24T23:14:34.569 INFO:teuthology.orchestra.run.smithi006.stdout: debug ms = 1 2024-04-24T23:14:34.570 INFO:teuthology.orchestra.run.smithi006.stdout: debug mgr = 20 2024-04-24T23:14:34.570 INFO:teuthology.orchestra.run.smithi006.stdout: debug mon = 20 2024-04-24T23:14:34.570 INFO:teuthology.orchestra.run.smithi006.stdout: debug auth = 20 2024-04-24T23:14:34.570 INFO:teuthology.orchestra.run.smithi006.stdout: mon reweight min pgs per osd = 4 2024-04-24T23:14:34.570 INFO:teuthology.orchestra.run.smithi006.stdout: mon reweight min bytes per osd = 10 2024-04-24T23:14:34.570 INFO:teuthology.orchestra.run.smithi006.stdout: mgr/telemetry/nag = false 2024-04-24T23:14:34.570 INFO:teuthology.orchestra.run.smithi006.stdout: debug client = 20 2024-04-24T23:14:34.570 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:14:34.570 INFO:teuthology.orchestra.run.smithi006.stdout:[mon] 2024-04-24T23:14:34.570 INFO:teuthology.orchestra.run.smithi006.stdout: debug ms = 1 2024-04-24T23:14:34.570 INFO:teuthology.orchestra.run.smithi006.stdout: debug mon = 20 2024-04-24T23:14:34.570 INFO:teuthology.orchestra.run.smithi006.stdout: debug paxos = 20 2024-04-24T23:14:34.570 INFO:teuthology.orchestra.run.smithi006.stdout: debug auth = 20 2024-04-24T23:14:34.570 INFO:teuthology.orchestra.run.smithi006.stdout: mon data avail warn = 5 2024-04-24T23:14:34.571 INFO:teuthology.orchestra.run.smithi006.stdout: mon mgr mkfs grace = 240 2024-04-24T23:14:34.571 INFO:teuthology.orchestra.run.smithi006.stdout: mon reweight min pgs per osd = 4 2024-04-24T23:14:34.571 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd reporter subtree level = osd 2024-04-24T23:14:34.571 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd prime pg temp = true 2024-04-24T23:14:34.571 INFO:teuthology.orchestra.run.smithi006.stdout: mon reweight min bytes per osd = 10 2024-04-24T23:14:34.571 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:14:34.571 INFO:teuthology.orchestra.run.smithi006.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-24T23:14:34.571 INFO:teuthology.orchestra.run.smithi006.stdout: auth mon ticket ttl = 660 # 11m 2024-04-24T23:14:34.571 INFO:teuthology.orchestra.run.smithi006.stdout: auth service ticket ttl = 240 # 4m 2024-04-24T23:14:34.571 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:14:34.571 INFO:teuthology.orchestra.run.smithi006.stdout: # don't complain about insecure global_id in the test suite 2024-04-24T23:14:34.571 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-24T23:14:34.571 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-24T23:14:34.571 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:14:34.572 INFO:teuthology.orchestra.run.smithi006.stdout: # 1m isn't quite enough 2024-04-24T23:14:34.572 INFO:teuthology.orchestra.run.smithi006.stdout: mon_down_mkfs_grace = 2m 2024-04-24T23:14:34.572 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:14:34.572 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_filestore_osds = false 2024-04-24T23:14:34.572 INFO:teuthology.orchestra.run.smithi006.stdout: mon down mkfs grace = 300 2024-04-24T23:14:34.572 INFO:teuthology.orchestra.run.smithi006.stdout: mon op complaint time = 120 2024-04-24T23:14:34.572 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:14:34.572 INFO:teuthology.orchestra.run.smithi006.stdout:[client] 2024-04-24T23:14:34.572 INFO:teuthology.orchestra.run.smithi006.stdout: rgw cache enabled = true 2024-04-24T23:14:34.572 INFO:teuthology.orchestra.run.smithi006.stdout: rgw enable ops log = true 2024-04-24T23:14:34.572 INFO:teuthology.orchestra.run.smithi006.stdout: rgw enable usage log = true 2024-04-24T23:14:34.572 INFO:teuthology.orchestra.run.smithi006.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-24T23:14:34.572 INFO:teuthology.orchestra.run.smithi006.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-24T23:14:34.572 INFO:teuthology.orchestra.run.smithi006.stdout: client mount timeout = 600 2024-04-24T23:14:34.572 INFO:teuthology.orchestra.run.smithi006.stdout: client_shutdown_timeout = 120 2024-04-24T23:14:34.573 INFO:teuthology.orchestra.run.smithi006.stdout: debug client = 20 2024-04-24T23:14:34.573 INFO:teuthology.orchestra.run.smithi006.stdout: debug ms = 1 2024-04-24T23:14:34.573 INFO:teuthology.orchestra.run.smithi006.stdout: rados mon op timeout = 900 2024-04-24T23:14:34.573 INFO:teuthology.orchestra.run.smithi006.stdout: rados osd op timeout = 900 2024-04-24T23:14:34.573 INFO:teuthology.orchestra.run.smithi006.stdout:[mon.a] 2024-04-24T23:14:34.573 INFO:teuthology.orchestra.run.smithi006.stdout:[mon.b] 2024-04-24T23:14:34.573 INFO:teuthology.orchestra.run.smithi006.stdout:[mon.c] 2024-04-24T23:14:34.573 INFO:teuthology.orchestra.run.smithi006.stdout:[mds.a] 2024-04-24T23:14:34.573 INFO:teuthology.orchestra.run.smithi006.stdout:[mds.c] 2024-04-24T23:14:34.573 INFO:teuthology.orchestra.run.smithi006.stdout:[mds.e] 2024-04-24T23:14:34.573 INFO:teuthology.orchestra.run.smithi006.stdout:[mds.b] 2024-04-24T23:14:34.573 INFO:teuthology.orchestra.run.smithi006.stdout:[mds.d] 2024-04-24T23:14:34.573 INFO:teuthology.orchestra.run.smithi006.stdout:[mds.f] 2024-04-24T23:14:34.573 INFO:teuthology.orchestra.run.smithi006.stdout:[mds] 2024-04-24T23:14:34.573 INFO:teuthology.orchestra.run.smithi006.stdout: debug mds = 20 2024-04-24T23:14:34.574 INFO:teuthology.orchestra.run.smithi006.stdout: debug mds balancer = 20 2024-04-24T23:14:34.574 INFO:teuthology.orchestra.run.smithi006.stdout: debug ms = 1 2024-04-24T23:14:34.574 INFO:teuthology.orchestra.run.smithi006.stdout: mds bal fragment size max = 10000 2024-04-24T23:14:34.574 INFO:teuthology.orchestra.run.smithi006.stdout: mds bal merge size = 5 2024-04-24T23:14:34.574 INFO:teuthology.orchestra.run.smithi006.stdout: mds bal split bits = 3 2024-04-24T23:14:34.574 INFO:teuthology.orchestra.run.smithi006.stdout: mds bal split size = 100 2024-04-24T23:14:34.574 INFO:teuthology.orchestra.run.smithi006.stdout: mds debug frag = True 2024-04-24T23:14:34.574 INFO:teuthology.orchestra.run.smithi006.stdout: mds debug scatterstat = True 2024-04-24T23:14:34.574 INFO:teuthology.orchestra.run.smithi006.stdout: mds oft prefetch dirfrags = True 2024-04-24T23:14:34.574 INFO:teuthology.orchestra.run.smithi006.stdout: mds op complaint time = 180 2024-04-24T23:14:34.574 INFO:teuthology.orchestra.run.smithi006.stdout: mds verify scatter = True 2024-04-24T23:14:34.574 INFO:teuthology.orchestra.run.smithi006.stdout: mds_dir_prefetch = True 2024-04-24T23:14:34.574 INFO:teuthology.orchestra.run.smithi006.stdout: mds_sleep_rank_change = 5000000.0 2024-04-24T23:14:34.574 INFO:teuthology.orchestra.run.smithi006.stdout: mon client directed command retry = 5 2024-04-24T23:14:34.574 INFO:teuthology.orchestra.run.smithi006.stdout: ms inject delay max = 1 2024-04-24T23:14:34.575 INFO:teuthology.orchestra.run.smithi006.stdout: ms inject delay probability = 0.005 2024-04-24T23:14:34.575 INFO:teuthology.orchestra.run.smithi006.stdout: ms inject delay type = client mds osd 2024-04-24T23:14:34.575 INFO:teuthology.orchestra.run.smithi006.stdout: ms inject socket failures = 2500 2024-04-24T23:14:34.575 INFO:teuthology.orchestra.run.smithi006.stdout: osd op complaint time = 180 2024-04-24T23:14:34.575 INFO:teuthology.orchestra.run.smithi006.stdout: rados mon op timeout = 900 2024-04-24T23:14:34.575 INFO:teuthology.orchestra.run.smithi006.stdout: rados osd op timeout = 900 2024-04-24T23:14:34.596 INFO:tasks.ceph:Creating admin key on mon.a... 2024-04-24T23:14:34.596 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-04-24T23:14:34.704 INFO:tasks.ceph:Copying monmap to all nodes... 2024-04-24T23:14:34.704 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-04-24T23:14:34.705 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-04-24T23:14:34.754 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-04-24T23:14:34.754 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-04-24T23:14:34.834 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi006.front.sepia.ceph.com 2024-04-24T23:14:34.834 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-04-24T23:14:34.835 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-24T23:14:34.835 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-24T23:14:34.965 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-04-24T23:14:34.965 DEBUG:teuthology.orchestra.run.smithi006:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-24T23:14:35.050 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi190.front.sepia.ceph.com 2024-04-24T23:14:35.050 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-04-24T23:14:35.051 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-24T23:14:35.051 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-24T23:14:35.089 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-04-24T23:14:35.089 DEBUG:teuthology.orchestra.run.smithi190:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-24T23:14:35.149 INFO:tasks.ceph:Setting up mon nodes... 2024-04-24T23:14:35.149 INFO:tasks.ceph:Setting up mgr nodes... 2024-04-24T23:14:35.149 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-04-24T23:14:35.276 INFO:teuthology.orchestra.run.smithi006.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-04-24T23:14:35.283 DEBUG:teuthology.orchestra.run.smithi190:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-04-24T23:14:35.344 INFO:teuthology.orchestra.run.smithi190.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-04-24T23:14:35.347 INFO:tasks.ceph:Setting up mds nodes... 2024-04-24T23:14:35.347 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2024-04-24T23:14:35.471 INFO:teuthology.orchestra.run.smithi006.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2024-04-24T23:14:35.478 DEBUG:teuthology.orchestra.run.smithi006:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2024-04-24T23:14:35.544 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2024-04-24T23:14:35.657 INFO:teuthology.orchestra.run.smithi006.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2024-04-24T23:14:35.663 DEBUG:teuthology.orchestra.run.smithi006:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2024-04-24T23:14:35.724 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /var/lib/ceph/mds/ceph-e && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.e /var/lib/ceph/mds/ceph-e/keyring 2024-04-24T23:14:35.843 INFO:teuthology.orchestra.run.smithi006.stdout:creating /var/lib/ceph/mds/ceph-e/keyring 2024-04-24T23:14:35.848 DEBUG:teuthology.orchestra.run.smithi006:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-e 2024-04-24T23:14:35.911 DEBUG:teuthology.orchestra.run.smithi190:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2024-04-24T23:14:35.968 INFO:teuthology.orchestra.run.smithi190.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2024-04-24T23:14:35.970 DEBUG:teuthology.orchestra.run.smithi190:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2024-04-24T23:14:35.996 DEBUG:teuthology.orchestra.run.smithi190:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2024-04-24T23:14:36.087 INFO:teuthology.orchestra.run.smithi190.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2024-04-24T23:14:36.090 DEBUG:teuthology.orchestra.run.smithi190:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2024-04-24T23:14:36.114 DEBUG:teuthology.orchestra.run.smithi190:> sudo mkdir -p /var/lib/ceph/mds/ceph-f && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.f /var/lib/ceph/mds/ceph-f/keyring 2024-04-24T23:14:36.203 INFO:teuthology.orchestra.run.smithi190.stdout:creating /var/lib/ceph/mds/ceph-f/keyring 2024-04-24T23:14:36.205 DEBUG:teuthology.orchestra.run.smithi190:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-f 2024-04-24T23:14:36.228 INFO:tasks.ceph_client:Setting up client nodes... 2024-04-24T23:14:36.228 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-24T23:14:36.319 INFO:teuthology.orchestra.run.smithi006.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-04-24T23:14:36.349 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-04-24T23:14:36.349 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi006.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi190.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-04-24T23:14:36.349 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-04-24T23:14:36.410 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-24T23:14:36.410 INFO:tasks.ceph:role: osd.0 2024-04-24T23:14:36.410 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi006.front.sepia.ceph.com 2024-04-24T23:14:36.411 DEBUG:teuthology.orchestra.run.smithi006:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-24T23:14:36.478 INFO:teuthology.orchestra.run.smithi006.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-24T23:14:36.478 INFO:teuthology.orchestra.run.smithi006.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-24T23:14:36.478 INFO:teuthology.orchestra.run.smithi006.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-24T23:14:36.478 INFO:teuthology.orchestra.run.smithi006.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-24T23:14:36.478 INFO:teuthology.orchestra.run.smithi006.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-24T23:14:36.479 INFO:teuthology.orchestra.run.smithi006.stdout: = sunit=0 swidth=0 blks 2024-04-24T23:14:36.479 INFO:teuthology.orchestra.run.smithi006.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-24T23:14:36.479 INFO:teuthology.orchestra.run.smithi006.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-24T23:14:36.479 INFO:teuthology.orchestra.run.smithi006.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-24T23:14:36.479 INFO:teuthology.orchestra.run.smithi006.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-24T23:14:36.884 INFO:teuthology.orchestra.run.smithi006.stdout:Discarding blocks...Done. 2024-04-24T23:14:36.890 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi006.front.sepia.ceph.com -o noatime 2024-04-24T23:14:36.890 DEBUG:teuthology.orchestra.run.smithi006:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-04-24T23:14:37.034 DEBUG:teuthology.orchestra.run.smithi006:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-04-24T23:14:37.102 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-04-24T23:14:37.168 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-24T23:14:37.168 INFO:tasks.ceph:role: osd.1 2024-04-24T23:14:37.168 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi006.front.sepia.ceph.com 2024-04-24T23:14:37.168 DEBUG:teuthology.orchestra.run.smithi006:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-24T23:14:37.244 INFO:teuthology.orchestra.run.smithi006.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-24T23:14:37.244 INFO:teuthology.orchestra.run.smithi006.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-24T23:14:37.244 INFO:teuthology.orchestra.run.smithi006.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-24T23:14:37.244 INFO:teuthology.orchestra.run.smithi006.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-24T23:14:37.244 INFO:teuthology.orchestra.run.smithi006.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-24T23:14:37.244 INFO:teuthology.orchestra.run.smithi006.stdout: = sunit=0 swidth=0 blks 2024-04-24T23:14:37.244 INFO:teuthology.orchestra.run.smithi006.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-24T23:14:37.244 INFO:teuthology.orchestra.run.smithi006.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-24T23:14:37.244 INFO:teuthology.orchestra.run.smithi006.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-24T23:14:37.244 INFO:teuthology.orchestra.run.smithi006.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-24T23:14:37.598 INFO:teuthology.orchestra.run.smithi006.stdout:Discarding blocks...Done. 2024-04-24T23:14:37.604 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi006.front.sepia.ceph.com -o noatime 2024-04-24T23:14:37.604 DEBUG:teuthology.orchestra.run.smithi006:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-04-24T23:14:37.697 DEBUG:teuthology.orchestra.run.smithi006:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-04-24T23:14:37.766 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-04-24T23:14:37.826 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-24T23:14:37.826 INFO:tasks.ceph:role: osd.2 2024-04-24T23:14:37.826 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi006.front.sepia.ceph.com 2024-04-24T23:14:37.827 DEBUG:teuthology.orchestra.run.smithi006:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-24T23:14:37.897 INFO:teuthology.orchestra.run.smithi006.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-24T23:14:37.897 INFO:teuthology.orchestra.run.smithi006.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-24T23:14:37.897 INFO:teuthology.orchestra.run.smithi006.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-24T23:14:37.897 INFO:teuthology.orchestra.run.smithi006.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-24T23:14:37.897 INFO:teuthology.orchestra.run.smithi006.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-24T23:14:37.897 INFO:teuthology.orchestra.run.smithi006.stdout: = sunit=0 swidth=0 blks 2024-04-24T23:14:37.897 INFO:teuthology.orchestra.run.smithi006.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-24T23:14:37.897 INFO:teuthology.orchestra.run.smithi006.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-24T23:14:37.897 INFO:teuthology.orchestra.run.smithi006.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-24T23:14:37.897 INFO:teuthology.orchestra.run.smithi006.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-24T23:14:38.280 INFO:teuthology.orchestra.run.smithi006.stdout:Discarding blocks...Done. 2024-04-24T23:14:38.285 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi006.front.sepia.ceph.com -o noatime 2024-04-24T23:14:38.285 DEBUG:teuthology.orchestra.run.smithi006:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-04-24T23:14:38.394 DEBUG:teuthology.orchestra.run.smithi006:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-04-24T23:14:38.465 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-04-24T23:14:38.530 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-24T23:14:38.530 INFO:tasks.ceph:role: osd.3 2024-04-24T23:14:38.530 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi006.front.sepia.ceph.com 2024-04-24T23:14:38.530 DEBUG:teuthology.orchestra.run.smithi006:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-04-24T23:14:38.600 INFO:teuthology.orchestra.run.smithi006.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-04-24T23:14:38.600 INFO:teuthology.orchestra.run.smithi006.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-24T23:14:38.600 INFO:teuthology.orchestra.run.smithi006.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-24T23:14:38.600 INFO:teuthology.orchestra.run.smithi006.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-24T23:14:38.600 INFO:teuthology.orchestra.run.smithi006.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-24T23:14:38.600 INFO:teuthology.orchestra.run.smithi006.stdout: = sunit=0 swidth=0 blks 2024-04-24T23:14:38.600 INFO:teuthology.orchestra.run.smithi006.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-24T23:14:38.600 INFO:teuthology.orchestra.run.smithi006.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-24T23:14:38.600 INFO:teuthology.orchestra.run.smithi006.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-24T23:14:38.600 INFO:teuthology.orchestra.run.smithi006.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-24T23:14:39.024 INFO:teuthology.orchestra.run.smithi006.stdout:Discarding blocks...Done. 2024-04-24T23:14:39.029 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi006.front.sepia.ceph.com -o noatime 2024-04-24T23:14:39.029 DEBUG:teuthology.orchestra.run.smithi006:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-04-24T23:14:39.204 DEBUG:teuthology.orchestra.run.smithi006:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-04-24T23:14:39.290 DEBUG:teuthology.orchestra.run.smithi006:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-24T23:14:39.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:14:39.441+0000 7f16d7916640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-04-24T23:14:39.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:14:39.442+0000 7f16d7916640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-04-24T23:14:39.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:14:39.442+0000 7f16d7916640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-24T23:14:39.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:14:39.442+0000 7f16d7916640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-24T23:14:39.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:14:39.442+0000 7f16d7916640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-24T23:14:39.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:14:39.447+0000 7f16d7916640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-04-24T23:14:42.422 DEBUG:teuthology.orchestra.run.smithi006:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-24T23:14:42.512 DEBUG:teuthology.orchestra.run.smithi006:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-24T23:14:42.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:14:42.647+0000 7f4c4281b640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-04-24T23:14:42.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:14:42.648+0000 7f4c4281b640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-04-24T23:14:42.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:14:42.648+0000 7f4c4281b640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-24T23:14:42.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:14:42.648+0000 7f4c4281b640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-24T23:14:42.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:14:42.649+0000 7f4c4281b640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-24T23:14:42.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:14:42.651+0000 7f4c4281b640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-04-24T23:14:45.846 DEBUG:teuthology.orchestra.run.smithi006:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-24T23:14:45.938 DEBUG:teuthology.orchestra.run.smithi006:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-24T23:14:46.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:14:46.075+0000 7f6258e15640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-04-24T23:14:46.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:14:46.076+0000 7f6258e15640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-04-24T23:14:46.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:14:46.076+0000 7f6258e15640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-24T23:14:46.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:14:46.076+0000 7f6258e15640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-24T23:14:46.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:14:46.076+0000 7f6258e15640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-24T23:14:46.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:14:46.079+0000 7f6258e15640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-04-24T23:14:49.329 DEBUG:teuthology.orchestra.run.smithi006:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-24T23:14:49.402 DEBUG:teuthology.orchestra.run.smithi006:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-24T23:14:49.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:14:49.524+0000 7fb81f521640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-04-24T23:14:49.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:14:49.525+0000 7fb81f521640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-04-24T23:14:49.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:14:49.525+0000 7fb81f521640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-24T23:14:49.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:14:49.525+0000 7fb81f521640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-24T23:14:49.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:14:49.525+0000 7fb81f521640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-24T23:14:49.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:14:49.527+0000 7fb81f521640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-04-24T23:14:52.748 DEBUG:teuthology.orchestra.run.smithi006:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-24T23:14:52.839 DEBUG:teuthology.orchestra.run.smithi190:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-04-24T23:14:52.868 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-24T23:14:52.868 INFO:tasks.ceph:role: osd.4 2024-04-24T23:14:52.868 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi190.front.sepia.ceph.com 2024-04-24T23:14:52.869 DEBUG:teuthology.orchestra.run.smithi190:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-24T23:14:52.944 INFO:teuthology.orchestra.run.smithi190.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-24T23:14:52.944 INFO:teuthology.orchestra.run.smithi190.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-24T23:14:52.944 INFO:teuthology.orchestra.run.smithi190.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-24T23:14:52.944 INFO:teuthology.orchestra.run.smithi190.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-24T23:14:52.944 INFO:teuthology.orchestra.run.smithi190.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-24T23:14:52.945 INFO:teuthology.orchestra.run.smithi190.stdout: = sunit=0 swidth=0 blks 2024-04-24T23:14:52.945 INFO:teuthology.orchestra.run.smithi190.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-24T23:14:52.945 INFO:teuthology.orchestra.run.smithi190.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-24T23:14:52.945 INFO:teuthology.orchestra.run.smithi190.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-24T23:14:52.945 INFO:teuthology.orchestra.run.smithi190.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-24T23:14:53.448 INFO:teuthology.orchestra.run.smithi190.stdout:Discarding blocks...Done. 2024-04-24T23:14:53.450 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi190.front.sepia.ceph.com -o noatime 2024-04-24T23:14:53.451 DEBUG:teuthology.orchestra.run.smithi190:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-04-24T23:14:53.555 DEBUG:teuthology.orchestra.run.smithi190:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-04-24T23:14:53.589 DEBUG:teuthology.orchestra.run.smithi190:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-04-24T23:14:53.655 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-24T23:14:53.655 INFO:tasks.ceph:role: osd.5 2024-04-24T23:14:53.655 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi190.front.sepia.ceph.com 2024-04-24T23:14:53.656 DEBUG:teuthology.orchestra.run.smithi190:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-24T23:14:53.727 INFO:teuthology.orchestra.run.smithi190.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-24T23:14:53.728 INFO:teuthology.orchestra.run.smithi190.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-24T23:14:53.728 INFO:teuthology.orchestra.run.smithi190.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-24T23:14:53.728 INFO:teuthology.orchestra.run.smithi190.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-24T23:14:53.728 INFO:teuthology.orchestra.run.smithi190.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-24T23:14:53.728 INFO:teuthology.orchestra.run.smithi190.stdout: = sunit=0 swidth=0 blks 2024-04-24T23:14:53.728 INFO:teuthology.orchestra.run.smithi190.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-24T23:14:53.728 INFO:teuthology.orchestra.run.smithi190.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-24T23:14:53.728 INFO:teuthology.orchestra.run.smithi190.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-24T23:14:53.728 INFO:teuthology.orchestra.run.smithi190.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-24T23:14:54.187 INFO:teuthology.orchestra.run.smithi190.stdout:Discarding blocks...Done. 2024-04-24T23:14:54.189 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi190.front.sepia.ceph.com -o noatime 2024-04-24T23:14:54.190 DEBUG:teuthology.orchestra.run.smithi190:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-04-24T23:14:54.273 DEBUG:teuthology.orchestra.run.smithi190:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-04-24T23:14:54.344 DEBUG:teuthology.orchestra.run.smithi190:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-04-24T23:14:54.410 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-24T23:14:54.410 INFO:tasks.ceph:role: osd.6 2024-04-24T23:14:54.411 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi190.front.sepia.ceph.com 2024-04-24T23:14:54.411 DEBUG:teuthology.orchestra.run.smithi190:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-24T23:14:54.482 INFO:teuthology.orchestra.run.smithi190.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-24T23:14:54.482 INFO:teuthology.orchestra.run.smithi190.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-24T23:14:54.483 INFO:teuthology.orchestra.run.smithi190.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-24T23:14:54.483 INFO:teuthology.orchestra.run.smithi190.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-24T23:14:54.483 INFO:teuthology.orchestra.run.smithi190.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-24T23:14:54.483 INFO:teuthology.orchestra.run.smithi190.stdout: = sunit=0 swidth=0 blks 2024-04-24T23:14:54.483 INFO:teuthology.orchestra.run.smithi190.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-24T23:14:54.483 INFO:teuthology.orchestra.run.smithi190.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-24T23:14:54.483 INFO:teuthology.orchestra.run.smithi190.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-24T23:14:54.483 INFO:teuthology.orchestra.run.smithi190.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-24T23:14:54.974 INFO:teuthology.orchestra.run.smithi190.stdout:Discarding blocks...Done. 2024-04-24T23:14:54.976 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi190.front.sepia.ceph.com -o noatime 2024-04-24T23:14:54.976 DEBUG:teuthology.orchestra.run.smithi190:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-04-24T23:14:55.074 DEBUG:teuthology.orchestra.run.smithi190:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-04-24T23:14:55.106 DEBUG:teuthology.orchestra.run.smithi190:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-04-24T23:14:55.172 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-24T23:14:55.172 INFO:tasks.ceph:role: osd.7 2024-04-24T23:14:55.172 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi190.front.sepia.ceph.com 2024-04-24T23:14:55.172 DEBUG:teuthology.orchestra.run.smithi190:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-04-24T23:14:55.246 INFO:teuthology.orchestra.run.smithi190.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-04-24T23:14:55.246 INFO:teuthology.orchestra.run.smithi190.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-24T23:14:55.246 INFO:teuthology.orchestra.run.smithi190.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-24T23:14:55.246 INFO:teuthology.orchestra.run.smithi190.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-24T23:14:55.247 INFO:teuthology.orchestra.run.smithi190.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-24T23:14:55.247 INFO:teuthology.orchestra.run.smithi190.stdout: = sunit=0 swidth=0 blks 2024-04-24T23:14:55.247 INFO:teuthology.orchestra.run.smithi190.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-24T23:14:55.247 INFO:teuthology.orchestra.run.smithi190.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-24T23:14:55.247 INFO:teuthology.orchestra.run.smithi190.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-24T23:14:55.247 INFO:teuthology.orchestra.run.smithi190.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-24T23:14:55.768 INFO:teuthology.orchestra.run.smithi190.stdout:Discarding blocks...Done. 2024-04-24T23:14:55.771 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi190.front.sepia.ceph.com -o noatime 2024-04-24T23:14:55.771 DEBUG:teuthology.orchestra.run.smithi190:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-04-24T23:14:55.922 DEBUG:teuthology.orchestra.run.smithi190:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-04-24T23:14:55.955 DEBUG:teuthology.orchestra.run.smithi190:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-24T23:14:56.050 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:14:56.048+0000 7f44fb029640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-04-24T23:14:56.054 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:14:56.053+0000 7f44fb029640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-04-24T23:14:56.054 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:14:56.053+0000 7f44fb029640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-24T23:14:56.054 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:14:56.053+0000 7f44fb029640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-24T23:14:56.055 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:14:56.053+0000 7f44fb029640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-24T23:14:56.058 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:14:56.057+0000 7f44fb029640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-04-24T23:14:58.911 DEBUG:teuthology.orchestra.run.smithi190:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-04-24T23:14:58.940 DEBUG:teuthology.orchestra.run.smithi190:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-24T23:14:59.031 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:14:59.029+0000 7fcf8642c640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-04-24T23:14:59.031 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:14:59.029+0000 7fcf8642c640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-04-24T23:14:59.031 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:14:59.029+0000 7fcf8642c640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-24T23:14:59.031 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:14:59.029+0000 7fcf8642c640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-24T23:14:59.031 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:14:59.029+0000 7fcf8642c640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-24T23:14:59.032 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:14:59.029+0000 7fcf8642c640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-04-24T23:15:01.885 DEBUG:teuthology.orchestra.run.smithi190:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-04-24T23:15:01.913 DEBUG:teuthology.orchestra.run.smithi190:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-24T23:15:02.001 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:01.999+0000 7f59cc40d640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-04-24T23:15:02.002 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:02.000+0000 7f59cc40d640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-04-24T23:15:02.002 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:02.000+0000 7f59cc40d640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-04-24T23:15:02.002 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:02.000+0000 7f59cc40d640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-04-24T23:15:02.002 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:02.000+0000 7f59cc40d640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-04-24T23:15:02.002 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:02.000+0000 7f59cc40d640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-04-24T23:15:02.098 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-24T23:15:02.105 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-24T23:15:04.851 DEBUG:teuthology.orchestra.run.smithi190:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-04-24T23:15:04.879 DEBUG:teuthology.orchestra.run.smithi190:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-24T23:15:04.968 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:04.966+0000 7f41edf55640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-04-24T23:15:04.968 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:04.966+0000 7f41edf55640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-04-24T23:15:04.968 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:04.966+0000 7f41edf55640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-04-24T23:15:04.968 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:04.966+0000 7f41edf55640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-04-24T23:15:04.968 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:04.966+0000 7f41edf55640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-04-24T23:15:04.968 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:04.967+0000 7f41edf55640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-04-24T23:15:07.812 DEBUG:teuthology.orchestra.run.smithi190:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-04-24T23:15:07.840 INFO:tasks.ceph:Reading keys from all nodes... 2024-04-24T23:15:07.840 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-04-24T23:15:07.840 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-04-24T23:15:07.935 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-04-24T23:15:07.935 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2024-04-24T23:15:08.025 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-04-24T23:15:08.025 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2024-04-24T23:15:08.116 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-04-24T23:15:08.116 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/var/lib/ceph/mds/ceph-e/keyring of=/dev/stdout 2024-04-24T23:15:08.209 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-04-24T23:15:08.209 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-04-24T23:15:08.301 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-04-24T23:15:08.301 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-04-24T23:15:08.389 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-04-24T23:15:08.389 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-04-24T23:15:08.478 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-04-24T23:15:08.478 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-04-24T23:15:08.568 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-04-24T23:15:08.568 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-04-24T23:15:08.597 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-04-24T23:15:08.597 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2024-04-24T23:15:08.662 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-04-24T23:15:08.662 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2024-04-24T23:15:08.726 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-04-24T23:15:08.726 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/var/lib/ceph/mds/ceph-f/keyring of=/dev/stdout 2024-04-24T23:15:08.790 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-04-24T23:15:08.790 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-04-24T23:15:08.854 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-04-24T23:15:08.854 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-04-24T23:15:08.918 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-04-24T23:15:08.918 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-04-24T23:15:08.982 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-04-24T23:15:08.982 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-04-24T23:15:09.045 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-04-24T23:15:09.046 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-04-24T23:15:09.107 INFO:tasks.ceph:Adding keys to all mons... 2024-04-24T23:15:09.107 DEBUG:teuthology.orchestra.run.smithi006:> sudo tee -a /etc/ceph/ceph.keyring 2024-04-24T23:15:09.114 DEBUG:teuthology.orchestra.run.smithi190:> sudo tee -a /etc/ceph/ceph.keyring 2024-04-24T23:15:09.136 INFO:teuthology.orchestra.run.smithi190.stdout:[mgr.y] 2024-04-24T23:15:09.136 INFO:teuthology.orchestra.run.smithi190.stdout: key = AQBbkilmGhNzEBAA2MZITxrKwSyGBFPeWbDeJg== 2024-04-24T23:15:09.136 INFO:teuthology.orchestra.run.smithi190.stdout:[mds.a] 2024-04-24T23:15:09.136 INFO:teuthology.orchestra.run.smithi190.stdout: key = AQBbkilmQkUWHBAAihe1I0ifQboPWRn40gKVHQ== 2024-04-24T23:15:09.136 INFO:teuthology.orchestra.run.smithi190.stdout:[mds.c] 2024-04-24T23:15:09.137 INFO:teuthology.orchestra.run.smithi190.stdout: key = AQBbkilmv4AsJxAAm74U6hMV8pDgqMrL4SEw5A== 2024-04-24T23:15:09.137 INFO:teuthology.orchestra.run.smithi190.stdout:[mds.e] 2024-04-24T23:15:09.137 INFO:teuthology.orchestra.run.smithi190.stdout: key = AQBbkilm4us5MhAAjMJ2SDgpiOpwhUsnJZ6F2Q== 2024-04-24T23:15:09.137 INFO:teuthology.orchestra.run.smithi190.stdout:[osd.0] 2024-04-24T23:15:09.137 INFO:teuthology.orchestra.run.smithi190.stdout: key = AQBfkilmk0dpGhAArc3IV9ZnF4sr6HevZZRwJw== 2024-04-24T23:15:09.137 INFO:teuthology.orchestra.run.smithi190.stdout:[osd.1] 2024-04-24T23:15:09.137 INFO:teuthology.orchestra.run.smithi190.stdout: key = AQBikilmrgizJhAANzpuktTf3MikG8IrkYG6MA== 2024-04-24T23:15:09.137 INFO:teuthology.orchestra.run.smithi190.stdout:[osd.2] 2024-04-24T23:15:09.137 INFO:teuthology.orchestra.run.smithi190.stdout: key = AQBmkilmBbOYBBAAd0aruKwBnxYjr9Rb44FYjw== 2024-04-24T23:15:09.137 INFO:teuthology.orchestra.run.smithi190.stdout:[osd.3] 2024-04-24T23:15:09.137 INFO:teuthology.orchestra.run.smithi190.stdout: key = AQBpkilmSNxdHxAA/yqK8++k+keDagXXkuVepg== 2024-04-24T23:15:09.137 INFO:teuthology.orchestra.run.smithi190.stdout:[mgr.x] 2024-04-24T23:15:09.137 INFO:teuthology.orchestra.run.smithi190.stdout: key = AQBbkilms9aCFBAADTfBE8hLSGRTaaxv4EPXQA== 2024-04-24T23:15:09.137 INFO:teuthology.orchestra.run.smithi190.stdout:[mds.b] 2024-04-24T23:15:09.138 INFO:teuthology.orchestra.run.smithi190.stdout: key = AQBbkilmBEmtORAAqOfMrNA3Fb+3kkNAihIxAA== 2024-04-24T23:15:09.138 INFO:teuthology.orchestra.run.smithi190.stdout:[mds.d] 2024-04-24T23:15:09.138 INFO:teuthology.orchestra.run.smithi190.stdout: key = AQBckilmo2I0BRAAeCI+zzd9OPDQkTNPs3rYjw== 2024-04-24T23:15:09.138 INFO:teuthology.orchestra.run.smithi190.stdout:[mds.f] 2024-04-24T23:15:09.138 INFO:teuthology.orchestra.run.smithi190.stdout: key = AQBckilmR50RDBAAVIrEmCHTRggH7dUDsODuJA== 2024-04-24T23:15:09.138 INFO:teuthology.orchestra.run.smithi190.stdout:[osd.4] 2024-04-24T23:15:09.138 INFO:teuthology.orchestra.run.smithi190.stdout: key = AQBwkilmU3/0AhAAS2ja1GenQdh9VeoDRzS6wQ== 2024-04-24T23:15:09.138 INFO:teuthology.orchestra.run.smithi190.stdout:[osd.5] 2024-04-24T23:15:09.138 INFO:teuthology.orchestra.run.smithi190.stdout: key = AQBzkilmhTTNARAABBafGsVx3b5gtm8YcQ0tXA== 2024-04-24T23:15:09.138 INFO:teuthology.orchestra.run.smithi190.stdout:[osd.6] 2024-04-24T23:15:09.138 INFO:teuthology.orchestra.run.smithi190.stdout: key = AQB2kilmt1oKABAAnwhp3Fcs1DX4hERHxfBmYg== 2024-04-24T23:15:09.138 INFO:teuthology.orchestra.run.smithi190.stdout:[osd.7] 2024-04-24T23:15:09.138 INFO:teuthology.orchestra.run.smithi190.stdout: key = AQB4kilmsiimORAA0M2pPP5NUW5erccLomMQ7A== 2024-04-24T23:15:09.138 INFO:teuthology.orchestra.run.smithi190.stdout:[client.0] 2024-04-24T23:15:09.139 INFO:teuthology.orchestra.run.smithi190.stdout: key = AQBckilmk5YDExAAwTMBeio69tb7s9569NTrcA== 2024-04-24T23:15:09.196 INFO:teuthology.orchestra.run.smithi006.stdout:[mgr.y] 2024-04-24T23:15:09.196 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQBbkilmGhNzEBAA2MZITxrKwSyGBFPeWbDeJg== 2024-04-24T23:15:09.196 INFO:teuthology.orchestra.run.smithi006.stdout:[mds.a] 2024-04-24T23:15:09.196 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQBbkilmQkUWHBAAihe1I0ifQboPWRn40gKVHQ== 2024-04-24T23:15:09.196 INFO:teuthology.orchestra.run.smithi006.stdout:[mds.c] 2024-04-24T23:15:09.197 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQBbkilmv4AsJxAAm74U6hMV8pDgqMrL4SEw5A== 2024-04-24T23:15:09.197 INFO:teuthology.orchestra.run.smithi006.stdout:[mds.e] 2024-04-24T23:15:09.197 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQBbkilm4us5MhAAjMJ2SDgpiOpwhUsnJZ6F2Q== 2024-04-24T23:15:09.197 INFO:teuthology.orchestra.run.smithi006.stdout:[osd.0] 2024-04-24T23:15:09.197 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQBfkilmk0dpGhAArc3IV9ZnF4sr6HevZZRwJw== 2024-04-24T23:15:09.197 INFO:teuthology.orchestra.run.smithi006.stdout:[osd.1] 2024-04-24T23:15:09.197 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQBikilmrgizJhAANzpuktTf3MikG8IrkYG6MA== 2024-04-24T23:15:09.197 INFO:teuthology.orchestra.run.smithi006.stdout:[osd.2] 2024-04-24T23:15:09.197 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQBmkilmBbOYBBAAd0aruKwBnxYjr9Rb44FYjw== 2024-04-24T23:15:09.197 INFO:teuthology.orchestra.run.smithi006.stdout:[osd.3] 2024-04-24T23:15:09.197 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQBpkilmSNxdHxAA/yqK8++k+keDagXXkuVepg== 2024-04-24T23:15:09.197 INFO:teuthology.orchestra.run.smithi006.stdout:[mgr.x] 2024-04-24T23:15:09.197 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQBbkilms9aCFBAADTfBE8hLSGRTaaxv4EPXQA== 2024-04-24T23:15:09.197 INFO:teuthology.orchestra.run.smithi006.stdout:[mds.b] 2024-04-24T23:15:09.197 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQBbkilmBEmtORAAqOfMrNA3Fb+3kkNAihIxAA== 2024-04-24T23:15:09.198 INFO:teuthology.orchestra.run.smithi006.stdout:[mds.d] 2024-04-24T23:15:09.198 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQBckilmo2I0BRAAeCI+zzd9OPDQkTNPs3rYjw== 2024-04-24T23:15:09.198 INFO:teuthology.orchestra.run.smithi006.stdout:[mds.f] 2024-04-24T23:15:09.198 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQBckilmR50RDBAAVIrEmCHTRggH7dUDsODuJA== 2024-04-24T23:15:09.198 INFO:teuthology.orchestra.run.smithi006.stdout:[osd.4] 2024-04-24T23:15:09.198 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQBwkilmU3/0AhAAS2ja1GenQdh9VeoDRzS6wQ== 2024-04-24T23:15:09.198 INFO:teuthology.orchestra.run.smithi006.stdout:[osd.5] 2024-04-24T23:15:09.198 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQBzkilmhTTNARAABBafGsVx3b5gtm8YcQ0tXA== 2024-04-24T23:15:09.198 INFO:teuthology.orchestra.run.smithi006.stdout:[osd.6] 2024-04-24T23:15:09.198 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQB2kilmt1oKABAAnwhp3Fcs1DX4hERHxfBmYg== 2024-04-24T23:15:09.198 INFO:teuthology.orchestra.run.smithi006.stdout:[osd.7] 2024-04-24T23:15:09.198 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQB4kilmsiimORAA0M2pPP5NUW5erccLomMQ7A== 2024-04-24T23:15:09.198 INFO:teuthology.orchestra.run.smithi006.stdout:[client.0] 2024-04-24T23:15:09.199 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQBckilmk5YDExAAwTMBeio69tb7s9569NTrcA== 2024-04-24T23:15:09.200 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-24T23:15:09.249 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-24T23:15:09.388 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-24T23:15:09.394 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-24T23:15:09.529 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-24T23:15:09.533 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-24T23:15:09.668 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.e --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-24T23:15:09.675 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.e --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-24T23:15:09.811 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-24T23:15:09.817 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-24T23:15:09.951 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-24T23:15:09.956 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-24T23:15:10.086 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-24T23:15:10.091 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-24T23:15:10.223 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-24T23:15:10.229 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-24T23:15:10.362 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-24T23:15:10.367 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-24T23:15:10.499 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-24T23:15:10.505 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-24T23:15:10.642 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-24T23:15:10.650 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-24T23:15:10.792 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.f --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-24T23:15:10.798 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.f --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-24T23:15:10.940 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-24T23:15:10.947 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-24T23:15:11.081 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-24T23:15:11.087 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-24T23:15:11.217 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-24T23:15:11.222 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-24T23:15:11.358 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-24T23:15:11.364 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-24T23:15:11.497 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-24T23:15:11.504 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-24T23:15:11.640 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-04-24T23:15:11.641 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-04-24T23:15:11.716 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-24T23:15:11.878 DEBUG:teuthology.orchestra.run.smithi006:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-04-24T23:15:11.951 DEBUG:teuthology.orchestra.run.smithi190:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-04-24T23:15:11.975 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-24T23:15:12.069 DEBUG:teuthology.orchestra.run.smithi190:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-04-24T23:15:12.092 DEBUG:teuthology.orchestra.run.smithi190:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-04-24T23:15:12.155 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-24T23:15:12.249 DEBUG:teuthology.orchestra.run.smithi190:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-04-24T23:15:12.272 DEBUG:teuthology.orchestra.run.smithi006:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-04-24T23:15:12.279 DEBUG:teuthology.orchestra.run.smithi190:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-04-24T23:15:12.331 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-04-24T23:15:12.332 INFO:tasks.ceph.mon.a:Restarting daemon 2024-04-24T23:15:12.332 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-04-24T23:15:12.337 INFO:tasks.ceph.mon.a:Started 2024-04-24T23:15:12.337 INFO:tasks.ceph.mon.b:Restarting daemon 2024-04-24T23:15:12.337 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-04-24T23:15:12.369 INFO:tasks.ceph.mon.b:Started 2024-04-24T23:15:12.369 INFO:tasks.ceph.mon.c:Restarting daemon 2024-04-24T23:15:12.369 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-04-24T23:15:12.371 INFO:tasks.ceph.mon.c:Started 2024-04-24T23:15:12.371 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-04-24T23:15:12.371 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-04-24T23:15:12.371 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-04-24T23:15:12.376 INFO:tasks.ceph.mgr.y:Started 2024-04-24T23:15:12.376 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-04-24T23:15:12.376 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-04-24T23:15:12.378 INFO:tasks.ceph.mgr.x:Started 2024-04-24T23:15:12.382 INFO:tasks.ceph:Setting crush tunables to default 2024-04-24T23:15:12.382 DEBUG:teuthology.orchestra.run.smithi006:> sudo ceph --cluster ceph osd crush tunables default 2024-04-24T23:15:12.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:12.562+0000 7f59f6600640 1 Processor -- start 2024-04-24T23:15:12.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:12.562+0000 7f59f6600640 1 -- start start 2024-04-24T23:15:12.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:12.564+0000 7f59f6600640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f59f0154ea0 0x7f59f00b3000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:12.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:12.564+0000 7f59f6600640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f59f00cc7c0 0x7f59f015b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:12.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:12.564+0000 7f59f6600640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f59f00cc200 0x7f59f015b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:12.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:12.564+0000 7f59f6600640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f59f00d0b00 con 0x7f59f00cc7c0 2024-04-24T23:15:12.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:12.564+0000 7f59f6600640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f59f00cab30 con 0x7f59f0154ea0 2024-04-24T23:15:12.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:12.564+0000 7f59f6600640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f59f00cadf0 con 0x7f59f00cc200 2024-04-24T23:15:12.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:12.564+0000 7f59f5200640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f59f0154ea0 0x7f59f00b3000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:12.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:12.564+0000 7f59efe00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f59f00cc7c0 0x7f59f015b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:12.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:12.564+0000 7f59f5c00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f59f00cc200 0x7f59f015b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:12.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:12.564+0000 7f59f5200640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f59f0154ea0 0x7f59f00b3000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:43880/0 (socket says 172.21.15.6:43880) 2024-04-24T23:15:12.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:12.564+0000 7f59f5200640 1 -- 172.21.15.6:0/372847092 learned_addr learned my addr 172.21.15.6:0/372847092 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:15:12.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:12.564+0000 7f59efe00640 1 -- 172.21.15.6:0/372847092 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f59f00cc200 msgr2=0x7f59f015b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:15:12.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:12.564+0000 7f59efe00640 1 --2- 172.21.15.6:0/372847092 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f59f00cc200 0x7f59f015b5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:12.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:12.564+0000 7f59efe00640 1 -- 172.21.15.6:0/372847092 >> v2:172.21.15.190:3300/0 conn(0x7f59f0154ea0 msgr2=0x7f59f00b3000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:15:12.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:12.564+0000 7f59efe00640 1 --2- 172.21.15.6:0/372847092 >> v2:172.21.15.190:3300/0 conn(0x7f59f0154ea0 0x7f59f00b3000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:12.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:12.564+0000 7f59efe00640 1 -- 172.21.15.6:0/372847092 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59f00d0d40 con 0x7f59f00cc7c0 2024-04-24T23:15:12.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:12.565+0000 7f59f5200640 1 --2- 172.21.15.6:0/372847092 >> v2:172.21.15.190:3300/0 conn(0x7f59f0154ea0 0x7f59f00b3000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:15:12.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:12.566+0000 7f59efe00640 1 --2- 172.21.15.6:0/372847092 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f59f00cc7c0 0x7f59f015b0b0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f59f00732e0 tx=0x7f59dc02a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce93c5ebd5025e9b server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:15:12.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:12.566+0000 7f59ef400640 1 -- 172.21.15.6:0/372847092 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59dc037070 con 0x7f59f00cc7c0 2024-04-24T23:15:12.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:12.566+0000 7f59ef400640 1 -- 172.21.15.6:0/372847092 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f59dc032440 con 0x7f59f00cc7c0 2024-04-24T23:15:12.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:12.566+0000 7f59ef400640 1 -- 172.21.15.6:0/372847092 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59dc02ae60 con 0x7f59f00cc7c0 2024-04-24T23:15:12.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:12.567+0000 7f59f6600640 1 -- 172.21.15.6:0/372847092 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f59f00cc7c0 msgr2=0x7f59f015b0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:12.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:12.567+0000 7f59f6600640 1 --2- 172.21.15.6:0/372847092 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f59f00cc7c0 0x7f59f015b0b0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f59f00732e0 tx=0x7f59dc02a110 comp rx=0 tx=0).stop 2024-04-24T23:15:12.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:12.567+0000 7f59f6600640 1 -- 172.21.15.6:0/372847092 shutdown_connections 2024-04-24T23:15:12.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:12.567+0000 7f59f6600640 1 --2- 172.21.15.6:0/372847092 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f59f00cc200 0x7f59f015b5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:12.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:12.567+0000 7f59f6600640 1 --2- 172.21.15.6:0/372847092 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f59f00cc7c0 0x7f59f015b0b0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:12.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:12.567+0000 7f59f6600640 1 --2- 172.21.15.6:0/372847092 >> v2:172.21.15.190:3300/0 conn(0x7f59f0154ea0 0x7f59f00b3000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:12.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:12.567+0000 7f59f6600640 1 -- 172.21.15.6:0/372847092 >> 172.21.15.6:0/372847092 conn(0x7f59f0072c30 msgr2=0x7f59f006bb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:15:12.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:12.568+0000 7f59f6600640 1 -- 172.21.15.6:0/372847092 shutdown_connections 2024-04-24T23:15:12.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:12.568+0000 7f59f6600640 1 -- 172.21.15.6:0/372847092 wait complete. 2024-04-24T23:15:12.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:12.569+0000 7f59f6600640 1 Processor -- start 2024-04-24T23:15:12.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:12.570+0000 7f59f6600640 1 -- start start 2024-04-24T23:15:12.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:12.570+0000 7f59f6600640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f59f00cc200 0x7f59f01577c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:12.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:12.571+0000 7f59f6600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f59f00cc7c0 0x7f59f0155340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:12.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:12.571+0000 7f59f6600640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f59f0154ea0 0x7f59f01f19a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:12.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:12.571+0000 7f59f6600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f59f00cdab0 con 0x7f59f00cc7c0 2024-04-24T23:15:12.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:12.571+0000 7f59f6600640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f59f00cd930 con 0x7f59f00cc200 2024-04-24T23:15:12.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:12.571+0000 7f59f6600640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f59f006d650 con 0x7f59f0154ea0 2024-04-24T23:15:12.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:12.571+0000 7f59f5200640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f59f00cc200 0x7f59f01577c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:12.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:12.571+0000 7f59f5200640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f59f00cc200 0x7f59f01577c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:43888/0 (socket says 172.21.15.6:43888) 2024-04-24T23:15:12.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:12.571+0000 7f59f5200640 1 -- 172.21.15.6:0/3894086283 learned_addr learned my addr 172.21.15.6:0/3894086283 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:15:12.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:12.571+0000 7f59f5c00640 1 --2- 172.21.15.6:0/3894086283 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f59f0154ea0 0x7f59f01f19a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:12.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:12.571+0000 7f59efe00640 1 --2- 172.21.15.6:0/3894086283 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f59f00cc7c0 0x7f59f0155340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:12.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:12.571+0000 7f59f5200640 1 -- 172.21.15.6:0/3894086283 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f59f0154ea0 msgr2=0x7f59f01f19a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:12.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:12.571+0000 7f59f5200640 1 --2- 172.21.15.6:0/3894086283 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f59f0154ea0 0x7f59f01f19a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:12.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:12.571+0000 7f59f5200640 1 -- 172.21.15.6:0/3894086283 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f59f00cc7c0 msgr2=0x7f59f0155340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:12.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:12.571+0000 7f59f5200640 1 --2- 172.21.15.6:0/3894086283 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f59f00cc7c0 0x7f59f0155340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:12.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:12.571+0000 7f59f5200640 1 -- 172.21.15.6:0/3894086283 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59f0157230 con 0x7f59f00cc200 2024-04-24T23:15:12.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:12.571+0000 7f59efe00640 1 --2- 172.21.15.6:0/3894086283 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f59f00cc7c0 0x7f59f0155340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:15:12.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:12.572+0000 7f59f5200640 1 --2- 172.21.15.6:0/3894086283 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f59f00cc200 0x7f59f01577c0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f59e400efc0 tx=0x7f59e400c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:15:12.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:12.572+0000 7f59ecc00640 1 -- 172.21.15.6:0/3894086283 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59e400ce60 con 0x7f59f00cc200 2024-04-24T23:15:12.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:12.572+0000 7f59ecc00640 1 -- 172.21.15.6:0/3894086283 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f59e4019070 con 0x7f59f00cc200 2024-04-24T23:15:12.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:12.572+0000 7f59ecc00640 1 -- 172.21.15.6:0/3894086283 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59e4014680 con 0x7f59f00cc200 2024-04-24T23:15:12.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:12.572+0000 7f59f6600640 1 -- 172.21.15.6:0/3894086283 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59f015bd40 con 0x7f59f00cc200 2024-04-24T23:15:12.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:12.572+0000 7f59f6600640 1 -- 172.21.15.6:0/3894086283 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59f00c96b0 con 0x7f59f00cc200 2024-04-24T23:15:12.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:12.572+0000 7f59ecc00640 1 -- 172.21.15.6:0/3894086283 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f59e4005d10 con 0x7f59f00cc200 2024-04-24T23:15:12.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:12.572+0000 7f59ecc00640 1 -- 172.21.15.6:0/3894086283 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f59e4007530 con 0x7f59f00cc200 2024-04-24T23:15:12.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:12.574+0000 7f59f6600640 1 -- 172.21.15.6:0/3894086283 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f59f0075b30 con 0x7f59f00cc200 2024-04-24T23:15:12.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:12.577+0000 7f59ecc00640 1 -- 172.21.15.6:0/3894086283 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f59e4022020 con 0x7f59f00cc200 2024-04-24T23:15:12.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:12.651+0000 7f59f6600640 1 -- 172.21.15.6:0/3894086283 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7f59f00cb180 con 0x7f59f00cc200 2024-04-24T23:15:12.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:12.658+0000 7f59ecc00640 1 -- 172.21.15.6:0/3894086283 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) ==== 124+0+0 (secure 0 0 0) 0x7f59f0157230 con 0x7f59f00cc200 2024-04-24T23:15:12.660 INFO:teuthology.orchestra.run.smithi006.stderr:adjusted tunables profile to default 2024-04-24T23:15:12.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:12.660+0000 7f59f6600640 1 -- 172.21.15.6:0/3894086283 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f59f00cc200 msgr2=0x7f59f01577c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:12.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:12.660+0000 7f59f6600640 1 --2- 172.21.15.6:0/3894086283 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f59f00cc200 0x7f59f01577c0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f59e400efc0 tx=0x7f59e400c490 comp rx=0 tx=0).stop 2024-04-24T23:15:12.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:12.660+0000 7f59f6600640 1 -- 172.21.15.6:0/3894086283 shutdown_connections 2024-04-24T23:15:12.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:12.660+0000 7f59f6600640 1 --2- 172.21.15.6:0/3894086283 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f59f0154ea0 0x7f59f01f19a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:12.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:12.660+0000 7f59f6600640 1 --2- 172.21.15.6:0/3894086283 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f59f00cc7c0 0x7f59f0155340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:12.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:12.660+0000 7f59f6600640 1 --2- 172.21.15.6:0/3894086283 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f59f00cc200 0x7f59f01577c0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:12.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:12.660+0000 7f59f6600640 1 -- 172.21.15.6:0/3894086283 >> 172.21.15.6:0/3894086283 conn(0x7f59f0072c30 msgr2=0x7f59f0074630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:15:12.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:12.660+0000 7f59f6600640 1 -- 172.21.15.6:0/3894086283 shutdown_connections 2024-04-24T23:15:12.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:12.660+0000 7f59f6600640 1 -- 172.21.15.6:0/3894086283 wait complete. 2024-04-24T23:15:12.663 INFO:tasks.ceph.mgr.x.smithi190.stderr:2024-04-24T23:15:12.661+0000 7f4852779040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-24T23:15:12.678 INFO:tasks.ceph:check_enable_crimson: False 2024-04-24T23:15:12.679 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-04-24T23:15:12.679 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-04-24T23:15:12.679 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-04-24T23:15:12.715 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-04-24T23:15:12.714+0000 7f400607a040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-24T23:15:12.757 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-04-24T23:15:12.757 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-04-24T23:15:12.813 INFO:tasks.ceph.mgr.x.smithi190.stderr:2024-04-24T23:15:12.812+0000 7f4852779040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-24T23:15:12.866 INFO:tasks.ceph.mgr.x.smithi190.stderr:2024-04-24T23:15:12.864+0000 7f4852779040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-24T23:15:12.872 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-04-24T23:15:12.872 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-04-24T23:15:12.920 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-04-24T23:15:12.919+0000 7f400607a040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-24T23:15:12.945 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-04-24T23:15:12.945 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-04-24T23:15:12.985 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-04-24T23:15:12.984+0000 7f400607a040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-24T23:15:13.066 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-04-24T23:15:13.066 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-04-24T23:15:13.089 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-04-24T23:15:13.090 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-04-24T23:15:13.104 INFO:tasks.ceph.mgr.x.smithi190.stderr:2024-04-24T23:15:13.103+0000 7f4852779040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-24T23:15:13.127 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-04-24T23:15:13.127 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-04-24T23:15:13.152 INFO:tasks.ceph.mgr.x.smithi190.stderr:2024-04-24T23:15:13.150+0000 7f4852779040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-24T23:15:13.175 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-04-24T23:15:13.175 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-04-24T23:15:13.238 DEBUG:teuthology.orchestra.run.smithi190:> sudo ceph --cluster ceph osd new 3b7a212d-d498-462e-90bf-ed493c803a09 0 2024-04-24T23:15:13.299 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-04-24T23:15:13.298+0000 7f400607a040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-24T23:15:13.356 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-04-24T23:15:13.355+0000 7f400607a040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-24T23:15:13.374 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.373+0000 7f359b472640 1 Processor -- start 2024-04-24T23:15:13.375 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.373+0000 7f359b472640 1 -- start start 2024-04-24T23:15:13.376 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.373+0000 7f359b472640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f359414f520 0x7f35940701b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:13.376 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.373+0000 7f359b472640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f35941588c0 0x7f35940c40e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:13.376 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.373+0000 7f359b472640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f35940ce910 0x7f35940ced40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:13.377 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.374+0000 7f359b472640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f35940c3c50 con 0x7f35941588c0 2024-04-24T23:15:13.377 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.374+0000 7f359b472640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f35941590e0 con 0x7f35940ce910 2024-04-24T23:15:13.377 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.374+0000 7f359b472640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f35941593c0 con 0x7f359414f520 2024-04-24T23:15:13.377 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.374+0000 7f3599a0f640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f35940ce910 0x7f35940ced40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:13.377 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.374+0000 7f359920e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f359414f520 0x7f35940701b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:13.377 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.374+0000 7f3599a0f640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f35940ce910 0x7f35940ced40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.190:42024/0 (socket says 172.21.15.190:42024) 2024-04-24T23:15:13.377 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.374+0000 7f3599a0f640 1 -- 172.21.15.190:0/292522527 learned_addr learned my addr 172.21.15.190:0/292522527 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-04-24T23:15:13.377 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.374+0000 7f359920e640 1 -- 172.21.15.190:0/292522527 >> v2:172.21.15.190:3300/0 conn(0x7f35940ce910 msgr2=0x7f35940ced40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:15:13.377 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.374+0000 7f359920e640 1 --2- 172.21.15.190:0/292522527 >> v2:172.21.15.190:3300/0 conn(0x7f35940ce910 0x7f35940ced40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:13.377 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.374+0000 7f359920e640 1 -- 172.21.15.190:0/292522527 >> v2:172.21.15.6:3300/0 conn(0x7f35941588c0 msgr2=0x7f35940c40e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:15:13.378 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.374+0000 7f359920e640 1 --2- 172.21.15.190:0/292522527 >> v2:172.21.15.6:3300/0 conn(0x7f35941588c0 0x7f35940c40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:13.378 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.374+0000 7f359920e640 1 -- 172.21.15.190:0/292522527 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35941596a0 con 0x7f359414f520 2024-04-24T23:15:13.378 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.374+0000 7f3599a0f640 1 --2- 172.21.15.190:0/292522527 >> v2:172.21.15.190:3300/0 conn(0x7f35940ce910 0x7f35940ced40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:15:13.378 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.375+0000 7f359920e640 1 --2- 172.21.15.190:0/292522527 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f359414f520 0x7f35940701b0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f35940cc2d0 tx=0x7f358402a110 comp rx=0 tx=0).ready entity=mon.2 client_cookie=91fdf295268cbf5d server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:15:13.378 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.375+0000 7f358bfff640 1 -- 172.21.15.190:0/292522527 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3584037070 con 0x7f359414f520 2024-04-24T23:15:13.378 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.375+0000 7f358bfff640 1 -- 172.21.15.190:0/292522527 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3584032440 con 0x7f359414f520 2024-04-24T23:15:13.378 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.375+0000 7f358bfff640 1 -- 172.21.15.190:0/292522527 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f358402ae60 con 0x7f359414f520 2024-04-24T23:15:13.381 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.375+0000 7f359b472640 1 -- 172.21.15.190:0/292522527 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f359414f520 msgr2=0x7f35940701b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:13.381 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.375+0000 7f359b472640 1 --2- 172.21.15.190:0/292522527 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f359414f520 0x7f35940701b0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f35940cc2d0 tx=0x7f358402a110 comp rx=0 tx=0).stop 2024-04-24T23:15:13.381 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.375+0000 7f359b472640 1 -- 172.21.15.190:0/292522527 shutdown_connections 2024-04-24T23:15:13.381 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.375+0000 7f359b472640 1 --2- 172.21.15.190:0/292522527 >> v2:172.21.15.190:3300/0 conn(0x7f35940ce910 0x7f35940ced40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:13.381 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.375+0000 7f359b472640 1 --2- 172.21.15.190:0/292522527 >> v2:172.21.15.6:3300/0 conn(0x7f35941588c0 0x7f35940c40e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:13.382 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.375+0000 7f359b472640 1 --2- 172.21.15.190:0/292522527 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f359414f520 0x7f35940701b0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:13.382 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.375+0000 7f359b472640 1 -- 172.21.15.190:0/292522527 >> 172.21.15.190:0/292522527 conn(0x7f3594067290 msgr2=0x7f35940676a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:15:13.382 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.376+0000 7f359b472640 1 -- 172.21.15.190:0/292522527 shutdown_connections 2024-04-24T23:15:13.382 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.376+0000 7f359b472640 1 -- 172.21.15.190:0/292522527 wait complete. 2024-04-24T23:15:13.382 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.377+0000 7f359b472640 1 Processor -- start 2024-04-24T23:15:13.382 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.377+0000 7f359b472640 1 -- start start 2024-04-24T23:15:13.382 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.377+0000 7f359b472640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f35940ce910 0x7f3594159cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:13.382 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.377+0000 7f359b472640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f359414f520 0x7f35940b2570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:13.382 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.377+0000 7f359b472640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f35941588c0 0x7f35940b2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:13.382 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.377+0000 7f359b472640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f3594158d20 con 0x7f35941588c0 2024-04-24T23:15:13.382 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.377+0000 7f359b472640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f35940c35d0 con 0x7f359414f520 2024-04-24T23:15:13.382 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.377+0000 7f359b472640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f3594158ea0 con 0x7f35940ce910 2024-04-24T23:15:13.382 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.377+0000 7f359920e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f35940ce910 0x7f3594159cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:13.383 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.377+0000 7f359920e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f35940ce910 0x7f3594159cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.190:40168/0 (socket says 172.21.15.190:40168) 2024-04-24T23:15:13.383 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.377+0000 7f359920e640 1 -- 172.21.15.190:0/1518457953 learned_addr learned my addr 172.21.15.190:0/1518457953 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-04-24T23:15:13.383 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.377+0000 7f3598a0d640 1 --2- 172.21.15.190:0/1518457953 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f359414f520 0x7f35940b2570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:13.383 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.378+0000 7f359920e640 1 -- 172.21.15.190:0/1518457953 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f359414f520 msgr2=0x7f35940b2570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:13.383 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.378+0000 7f359920e640 1 --2- 172.21.15.190:0/1518457953 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f359414f520 0x7f35940b2570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:13.383 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.378+0000 7f359920e640 1 -- 172.21.15.190:0/1518457953 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f35941588c0 msgr2=0x7f35940b2ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:13.383 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.378+0000 7f359920e640 1 --2- 172.21.15.190:0/1518457953 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f35941588c0 0x7f35940b2ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:13.383 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.378+0000 7f359920e640 1 -- 172.21.15.190:0/1518457953 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f359415a760 con 0x7f35940ce910 2024-04-24T23:15:13.383 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.378+0000 7f3598a0d640 1 --2- 172.21.15.190:0/1518457953 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f359414f520 0x7f35940b2570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:15:13.383 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.378+0000 7f359920e640 1 --2- 172.21.15.190:0/1518457953 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f35940ce910 0x7f3594159cf0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f3584009890 tx=0x7f35840046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:15:13.383 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.378+0000 7f3589ffb640 1 -- 172.21.15.190:0/1518457953 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3584037040 con 0x7f35940ce910 2024-04-24T23:15:13.383 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.378+0000 7f3589ffb640 1 -- 172.21.15.190:0/1518457953 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3584042070 con 0x7f35940ce910 2024-04-24T23:15:13.383 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.378+0000 7f3589ffb640 1 -- 172.21.15.190:0/1518457953 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f358403b950 con 0x7f35940ce910 2024-04-24T23:15:13.383 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.378+0000 7f359b472640 1 -- 172.21.15.190:0/1518457953 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f359415b330 con 0x7f35940ce910 2024-04-24T23:15:13.384 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.378+0000 7f359b472640 1 -- 172.21.15.190:0/1518457953 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f359415ace0 con 0x7f35940ce910 2024-04-24T23:15:13.384 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.378+0000 7f3589ffb640 1 -- 172.21.15.190:0/1518457953 <== mon.2 v2:172.21.15.190:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f358403baf0 con 0x7f35940ce910 2024-04-24T23:15:13.384 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.378+0000 7f3589ffb640 1 -- 172.21.15.190:0/1518457953 <== mon.2 v2:172.21.15.190:3301/0 5 ==== osd_map(2..2 src has 1..2) ==== 725+0+0 (secure 0 0 0) 0x7f3584047b90 con 0x7f35940ce910 2024-04-24T23:15:13.384 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.379+0000 7f359b472640 1 -- 172.21.15.190:0/1518457953 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f35940c9050 con 0x7f35940ce910 2024-04-24T23:15:13.384 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.381+0000 7f3589ffb640 1 -- 172.21.15.190:0/1518457953 <== mon.2 v2:172.21.15.190:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f35940c9050 con 0x7f35940ce910 2024-04-24T23:15:13.455 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.454+0000 7f359b472640 1 -- 172.21.15.190:0/1518457953 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "osd new", "uuid": "3b7a212d-d498-462e-90bf-ed493c803a09", "id": 0} v 0) -- 0x7f3594068430 con 0x7f35940ce910 2024-04-24T23:15:13.460 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.458+0000 7f3589ffb640 1 -- 172.21.15.190:0/1518457953 <== mon.2 v2:172.21.15.190:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "3b7a212d-d498-462e-90bf-ed493c803a09", "id": 0}]=0 v3) ==== 112+0+2 (secure 0 0 0) 0x7f3584047460 con 0x7f35940ce910 2024-04-24T23:15:13.460 INFO:teuthology.orchestra.run.smithi190.stdout:0 2024-04-24T23:15:13.461 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.459+0000 7f359b472640 1 -- 172.21.15.190:0/1518457953 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f35940ce910 msgr2=0x7f3594159cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:13.461 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.459+0000 7f359b472640 1 --2- 172.21.15.190:0/1518457953 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f35940ce910 0x7f3594159cf0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f3584009890 tx=0x7f35840046c0 comp rx=0 tx=0).stop 2024-04-24T23:15:13.462 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.459+0000 7f359b472640 1 -- 172.21.15.190:0/1518457953 shutdown_connections 2024-04-24T23:15:13.462 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.459+0000 7f359b472640 1 --2- 172.21.15.190:0/1518457953 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f35941588c0 0x7f35940b2ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:13.462 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.459+0000 7f359b472640 1 --2- 172.21.15.190:0/1518457953 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f359414f520 0x7f35940b2570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:13.462 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.459+0000 7f359b472640 1 --2- 172.21.15.190:0/1518457953 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f35940ce910 0x7f3594159cf0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:13.462 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.459+0000 7f359b472640 1 -- 172.21.15.190:0/1518457953 >> 172.21.15.190:0/1518457953 conn(0x7f3594067290 msgr2=0x7f3594156fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:15:13.462 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.459+0000 7f359b472640 1 -- 172.21.15.190:0/1518457953 shutdown_connections 2024-04-24T23:15:13.462 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.459+0000 7f359b472640 1 -- 172.21.15.190:0/1518457953 wait complete. 2024-04-24T23:15:13.471 DEBUG:teuthology.orchestra.run.smithi190:> sudo ceph --cluster ceph osd new 7c97f5a2-1735-4775-9f15-eb50099b19d0 1 2024-04-24T23:15:13.572 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.570+0000 7f7e3248c640 1 Processor -- start 2024-04-24T23:15:13.572 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.570+0000 7f7e3248c640 1 -- start start 2024-04-24T23:15:13.575 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.570+0000 7f7e3248c640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7e2c06fd50 0x7f7e2c0c7e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:13.575 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.570+0000 7f7e3248c640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f7e2c070130 0x7f7e2c070590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:13.575 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.570+0000 7f7e3248c640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e2c0c8810 con 0x7f7e2c072a20 2024-04-24T23:15:13.575 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.571+0000 7f7e3248c640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f7e2c072ef0 con 0x7f7e2c070130 2024-04-24T23:15:13.575 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.571+0000 7f7e3248c640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f7e2c0c8ab0 con 0x7f7e2c06fd50 2024-04-24T23:15:13.576 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.571+0000 7f7e2bfff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7e2c06fd50 0x7f7e2c0c7e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:13.576 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.571+0000 7f7e2b7fe640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f7e2c072a20 0x7f7e2c15af10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.190:53286/0 (socket says 172.21.15.190:53286) 2024-04-24T23:15:13.576 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.571+0000 7f7e2b7fe640 1 -- 172.21.15.190:0/4114093806 learned_addr learned my addr 172.21.15.190:0/4114093806 (peer_addr_for_me v1:172.21.15.190:0/0) 2024-04-24T23:15:13.576 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.571+0000 7f7e2affd640 1 -- 172.21.15.190:0/4114093806 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2550242292 0 0) 0x7f7e2c0c8810 con 0x7f7e2c072a20 2024-04-24T23:15:13.576 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.571+0000 7f7e2affd640 1 -- 172.21.15.190:0/4114093806 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7e14003520 con 0x7f7e2c072a20 2024-04-24T23:15:13.576 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.571+0000 7f7e2bfff640 1 -- 172.21.15.190:0/4114093806 >> v2:172.21.15.6:3300/0 conn(0x7f7e2c070130 msgr2=0x7f7e2c070590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:15:13.576 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.571+0000 7f7e2bfff640 1 --2- 172.21.15.190:0/4114093806 >> v2:172.21.15.6:3300/0 conn(0x7f7e2c070130 0x7f7e2c070590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:13.576 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.571+0000 7f7e2bfff640 1 -- 172.21.15.190:0/4114093806 >> v1:172.21.15.190:6789/0 conn(0x7f7e2c072a20 legacy=0x7f7e2c15af10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:13.576 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.571+0000 7f7e2bfff640 1 -- 172.21.15.190:0/4114093806 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e2c0c8d90 con 0x7f7e2c06fd50 2024-04-24T23:15:13.577 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.572+0000 7f7e2bfff640 1 --2- 172.21.15.190:0/4114093806 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7e2c06fd50 0x7f7e2c0c7e00 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f7e1c0097a0 tx=0x7f7e1c02a210 comp rx=0 tx=0).ready entity=mon.2 client_cookie=75b69270f3b7e8a7 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:15:13.577 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.572+0000 7f7e2affd640 1 -- 172.21.15.190:0/4114093806 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e1c037070 con 0x7f7e2c06fd50 2024-04-24T23:15:13.577 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.572+0000 7f7e2affd640 1 -- 172.21.15.190:0/4114093806 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e1c032440 con 0x7f7e2c06fd50 2024-04-24T23:15:13.577 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.572+0000 7f7e2affd640 1 -- 172.21.15.190:0/4114093806 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e1c032810 con 0x7f7e2c06fd50 2024-04-24T23:15:13.577 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.572+0000 7f7e3248c640 1 -- 172.21.15.190:0/4114093806 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7e2c06fd50 msgr2=0x7f7e2c0c7e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:13.577 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.572+0000 7f7e3248c640 1 --2- 172.21.15.190:0/4114093806 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7e2c06fd50 0x7f7e2c0c7e00 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f7e1c0097a0 tx=0x7f7e1c02a210 comp rx=0 tx=0).stop 2024-04-24T23:15:13.577 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.573+0000 7f7e3248c640 1 -- 172.21.15.190:0/4114093806 shutdown_connections 2024-04-24T23:15:13.577 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.573+0000 7f7e3248c640 1 --2- 172.21.15.190:0/4114093806 >> v2:172.21.15.6:3300/0 conn(0x7f7e2c070130 0x7f7e2c070590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:13.577 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.573+0000 7f7e3248c640 1 --2- 172.21.15.190:0/4114093806 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7e2c06fd50 0x7f7e2c0c7e00 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:13.577 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.573+0000 7f7e3248c640 1 -- 172.21.15.190:0/4114093806 >> 172.21.15.190:0/4114093806 conn(0x7f7e2c06c1f0 msgr2=0x7f7e2c06c5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:15:13.577 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.573+0000 7f7e3248c640 1 -- 172.21.15.190:0/4114093806 shutdown_connections 2024-04-24T23:15:13.578 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.573+0000 7f7e3248c640 1 -- 172.21.15.190:0/4114093806 wait complete. 2024-04-24T23:15:13.578 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.574+0000 7f7e3248c640 1 Processor -- start 2024-04-24T23:15:13.581 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.574+0000 7f7e3248c640 1 -- start start 2024-04-24T23:15:13.581 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.575+0000 7f7e3248c640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7e2c06fd50 0x7f7e2c1f4cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:13.581 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.575+0000 7f7e3248c640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7e2c070130 0x7f7e2c0c65d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:13.581 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.575+0000 7f7e3248c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7e2c072a20 0x7f7e2c15a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:13.581 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.575+0000 7f7e3248c640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f7e2c15b7a0 con 0x7f7e2c06fd50 2024-04-24T23:15:13.581 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.575+0000 7f7e3248c640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f7e2c15b620 con 0x7f7e2c072a20 2024-04-24T23:15:13.581 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.575+0000 7f7e3248c640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f7e2c1f5210 con 0x7f7e2c070130 2024-04-24T23:15:13.581 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.575+0000 7f7e2b7fe640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7e2c070130 0x7f7e2c0c65d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:13.581 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.575+0000 7f7e30a29640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7e2c072a20 0x7f7e2c15a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:13.581 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.575+0000 7f7e2b7fe640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7e2c070130 0x7f7e2c0c65d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.190:40198/0 (socket says 172.21.15.190:40198) 2024-04-24T23:15:13.582 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.575+0000 7f7e2b7fe640 1 -- 172.21.15.190:0/1194869690 learned_addr learned my addr 172.21.15.190:0/1194869690 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-04-24T23:15:13.582 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.575+0000 7f7e2b7fe640 1 -- 172.21.15.190:0/1194869690 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7e2c072a20 msgr2=0x7f7e2c15a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:13.582 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.575+0000 7f7e2b7fe640 1 --2- 172.21.15.190:0/1194869690 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7e2c072a20 0x7f7e2c15a4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:13.582 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.575+0000 7f7e2b7fe640 1 -- 172.21.15.190:0/1194869690 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7e2c06fd50 msgr2=0x7f7e2c1f4cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:13.582 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.575+0000 7f7e2b7fe640 1 --2- 172.21.15.190:0/1194869690 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7e2c06fd50 0x7f7e2c1f4cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:13.582 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.575+0000 7f7e2b7fe640 1 -- 172.21.15.190:0/1194869690 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e2c0c6e80 con 0x7f7e2c070130 2024-04-24T23:15:13.582 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.575+0000 7f7e30a29640 1 --2- 172.21.15.190:0/1194869690 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7e2c072a20 0x7f7e2c15a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:15:13.582 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.575+0000 7f7e2b7fe640 1 --2- 172.21.15.190:0/1194869690 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7e2c070130 0x7f7e2c0c65d0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f7e2000c390 tx=0x7f7e2000c850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:15:13.582 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.575+0000 7f7e28ff9640 1 -- 172.21.15.190:0/1194869690 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e20017070 con 0x7f7e2c070130 2024-04-24T23:15:13.582 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.575+0000 7f7e28ff9640 1 -- 172.21.15.190:0/1194869690 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e20005d00 con 0x7f7e2c070130 2024-04-24T23:15:13.583 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.575+0000 7f7e3248c640 1 -- 172.21.15.190:0/1194869690 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e2c1f0ea0 con 0x7f7e2c070130 2024-04-24T23:15:13.583 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.575+0000 7f7e28ff9640 1 -- 172.21.15.190:0/1194869690 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e20007310 con 0x7f7e2c070130 2024-04-24T23:15:13.583 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.575+0000 7f7e3248c640 1 -- 172.21.15.190:0/1194869690 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e2c1f1460 con 0x7f7e2c070130 2024-04-24T23:15:13.583 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.575+0000 7f7e28ff9640 1 -- 172.21.15.190:0/1194869690 <== mon.2 v2:172.21.15.190:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f7e2001a070 con 0x7f7e2c070130 2024-04-24T23:15:13.583 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.575+0000 7f7e28ff9640 1 -- 172.21.15.190:0/1194869690 <== mon.2 v2:172.21.15.190:3301/0 5 ==== osd_map(3..3 src has 1..3) ==== 840+0+0 (secure 0 0 0) 0x7f7e200127b0 con 0x7f7e2c070130 2024-04-24T23:15:13.583 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.576+0000 7f7e3248c640 1 -- 172.21.15.190:0/1194869690 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e2c068430 con 0x7f7e2c070130 2024-04-24T23:15:13.583 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.578+0000 7f7e28ff9640 1 -- 172.21.15.190:0/1194869690 <== mon.2 v2:172.21.15.190:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f7e20021070 con 0x7f7e2c070130 2024-04-24T23:15:13.652 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.650+0000 7f7e3248c640 1 -- 172.21.15.190:0/1194869690 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "osd new", "uuid": "7c97f5a2-1735-4775-9f15-eb50099b19d0", "id": 1} v 0) -- 0x7f7e2c152d50 con 0x7f7e2c070130 2024-04-24T23:15:13.658 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.656+0000 7f7e28ff9640 1 -- 172.21.15.190:0/1194869690 <== mon.2 v2:172.21.15.190:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "7c97f5a2-1735-4775-9f15-eb50099b19d0", "id": 1}]=0 v4) ==== 112+0+2 (secure 0 0 0) 0x7f7e20007680 con 0x7f7e2c070130 2024-04-24T23:15:13.658 INFO:teuthology.orchestra.run.smithi190.stdout:1 2024-04-24T23:15:13.659 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.657+0000 7f7e3248c640 1 -- 172.21.15.190:0/1194869690 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7e2c070130 msgr2=0x7f7e2c0c65d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:13.659 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.657+0000 7f7e3248c640 1 --2- 172.21.15.190:0/1194869690 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7e2c070130 0x7f7e2c0c65d0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f7e2000c390 tx=0x7f7e2000c850 comp rx=0 tx=0).stop 2024-04-24T23:15:13.659 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.657+0000 7f7e3248c640 1 -- 172.21.15.190:0/1194869690 shutdown_connections 2024-04-24T23:15:13.659 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.657+0000 7f7e3248c640 1 --2- 172.21.15.190:0/1194869690 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7e2c070130 0x7f7e2c0c65d0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:13.660 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.657+0000 7f7e3248c640 1 --2- 172.21.15.190:0/1194869690 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7e2c072a20 0x7f7e2c15a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:13.660 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.657+0000 7f7e3248c640 1 --2- 172.21.15.190:0/1194869690 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7e2c06fd50 0x7f7e2c1f4cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:13.660 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.657+0000 7f7e3248c640 1 -- 172.21.15.190:0/1194869690 >> 172.21.15.190:0/1194869690 conn(0x7f7e2c06c1f0 msgr2=0x7f7e2c06c5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:15:13.660 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.657+0000 7f7e3248c640 1 -- 172.21.15.190:0/1194869690 shutdown_connections 2024-04-24T23:15:13.660 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.657+0000 7f7e3248c640 1 -- 172.21.15.190:0/1194869690 wait complete. 2024-04-24T23:15:13.669 DEBUG:teuthology.orchestra.run.smithi190:> sudo ceph --cluster ceph osd new d86bbc18-438c-4ec2-8018-ba7dd78f790b 2 2024-04-24T23:15:13.767 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.766+0000 7f291671b640 1 Processor -- start 2024-04-24T23:15:13.768 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.766+0000 7f291671b640 1 -- start start 2024-04-24T23:15:13.769 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.766+0000 7f291671b640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f2910151aa0 0x7f2910073070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:13.769 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.766+0000 7f291671b640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f291015aca0 0x7f29101580c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:13.769 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.767+0000 7f291671b640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2910151e80 con 0x7f29100c73f0 2024-04-24T23:15:13.770 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.767+0000 7f291671b640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f29100c9c90 con 0x7f2910151aa0 2024-04-24T23:15:13.770 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.767+0000 7f291671b640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f2910072cd0 con 0x7f291015aca0 2024-04-24T23:15:13.770 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.767+0000 7f290f7fe640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f291015aca0 0x7f29101580c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:13.770 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.767+0000 7f290f7fe640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f291015aca0 0x7f29101580c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.190:42040/0 (socket says 172.21.15.190:42040) 2024-04-24T23:15:13.770 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.767+0000 7f290f7fe640 1 -- 172.21.15.190:0/2438000179 learned_addr learned my addr 172.21.15.190:0/2438000179 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-04-24T23:15:13.770 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.767+0000 7f290f7fe640 1 -- 172.21.15.190:0/2438000179 >> v2:172.21.15.6:3300/0 conn(0x7f2910151aa0 msgr2=0x7f2910073070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:15:13.770 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.767+0000 7f290f7fe640 1 --2- 172.21.15.190:0/2438000179 >> v2:172.21.15.6:3300/0 conn(0x7f2910151aa0 0x7f2910073070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:13.770 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.767+0000 7f290f7fe640 1 -- 172.21.15.190:0/2438000179 >> v1:172.21.15.6:6789/0 conn(0x7f29100c73f0 legacy=0x7f29100cd390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:15:13.770 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.767+0000 7f290f7fe640 1 -- 172.21.15.190:0/2438000179 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2910157900 con 0x7f291015aca0 2024-04-24T23:15:13.770 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.768+0000 7f290f7fe640 1 --2- 172.21.15.190:0/2438000179 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f291015aca0 0x7f29101580c0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f2910155320 tx=0x7f290402a230 comp rx=0 tx=0).ready entity=mon.1 client_cookie=77e6a2292da7dfbb server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:15:13.770 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.768+0000 7f290effd640 1 -- 172.21.15.190:0/2438000179 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2904037070 con 0x7f291015aca0 2024-04-24T23:15:13.770 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.768+0000 7f290effd640 1 -- 172.21.15.190:0/2438000179 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2904032440 con 0x7f291015aca0 2024-04-24T23:15:13.770 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.768+0000 7f290effd640 1 -- 172.21.15.190:0/2438000179 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2904032810 con 0x7f291015aca0 2024-04-24T23:15:13.771 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.768+0000 7f291671b640 1 -- 172.21.15.190:0/2438000179 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f291015aca0 msgr2=0x7f29101580c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:13.772 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.768+0000 7f291671b640 1 --2- 172.21.15.190:0/2438000179 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f291015aca0 0x7f29101580c0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f2910155320 tx=0x7f290402a230 comp rx=0 tx=0).stop 2024-04-24T23:15:13.772 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.768+0000 7f291671b640 1 -- 172.21.15.190:0/2438000179 shutdown_connections 2024-04-24T23:15:13.772 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.768+0000 7f291671b640 1 --2- 172.21.15.190:0/2438000179 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f291015aca0 0x7f29101580c0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:13.772 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.768+0000 7f291671b640 1 --2- 172.21.15.190:0/2438000179 >> v2:172.21.15.6:3300/0 conn(0x7f2910151aa0 0x7f2910073070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:13.772 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.768+0000 7f291671b640 1 -- 172.21.15.190:0/2438000179 >> 172.21.15.190:0/2438000179 conn(0x7f291006c1f0 msgr2=0x7f291006c600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:15:13.772 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.769+0000 7f291671b640 1 -- 172.21.15.190:0/2438000179 shutdown_connections 2024-04-24T23:15:13.772 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.769+0000 7f291671b640 1 -- 172.21.15.190:0/2438000179 wait complete. 2024-04-24T23:15:13.772 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.770+0000 7f291671b640 1 Processor -- start 2024-04-24T23:15:13.772 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.770+0000 7f291671b640 1 -- start start 2024-04-24T23:15:13.774 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.770+0000 7f291671b640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f29100c73f0 0x7f29101522d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:13.774 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.770+0000 7f291671b640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2910151aa0 0x7f29101f1f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:13.774 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.770+0000 7f291671b640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f291015aca0 0x7f29101f2450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:13.774 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.770+0000 7f291671b640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f29100729f0 con 0x7f291015aca0 2024-04-24T23:15:13.774 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.770+0000 7f291671b640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f29100724c0 con 0x7f29100c73f0 2024-04-24T23:15:13.774 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.770+0000 7f291671b640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f2910157ba0 con 0x7f2910151aa0 2024-04-24T23:15:13.774 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.771+0000 7f290f7fe640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2910151aa0 0x7f29101f1f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:13.775 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.771+0000 7f290ffff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f29100c73f0 0x7f29101522d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:13.775 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.771+0000 7f290f7fe640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2910151aa0 0x7f29101f1f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.190:40214/0 (socket says 172.21.15.190:40214) 2024-04-24T23:15:13.775 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.771+0000 7f290f7fe640 1 -- 172.21.15.190:0/1069506377 learned_addr learned my addr 172.21.15.190:0/1069506377 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-04-24T23:15:13.775 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.771+0000 7f290ffff640 1 -- 172.21.15.190:0/1069506377 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2910151aa0 msgr2=0x7f29101f1f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:13.775 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.771+0000 7f290ffff640 1 --2- 172.21.15.190:0/1069506377 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2910151aa0 0x7f29101f1f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:13.775 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.771+0000 7f290ffff640 1 -- 172.21.15.190:0/1069506377 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f291015aca0 msgr2=0x7f29101f2450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:13.775 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.771+0000 7f290ffff640 1 --2- 172.21.15.190:0/1069506377 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f291015aca0 0x7f29101f2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:13.775 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.771+0000 7f290ffff640 1 -- 172.21.15.190:0/1069506377 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29101f1a30 con 0x7f29100c73f0 2024-04-24T23:15:13.775 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.771+0000 7f290f7fe640 1 --2- 172.21.15.190:0/1069506377 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2910151aa0 0x7f29101f1f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:15:13.775 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.771+0000 7f290ffff640 1 --2- 172.21.15.190:0/1069506377 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f29100c73f0 0x7f29101522d0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f290000b3f0 tx=0x7f290000b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:15:13.775 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.771+0000 7f290cff9640 1 -- 172.21.15.190:0/1069506377 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2900015020 con 0x7f29100c73f0 2024-04-24T23:15:13.776 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.771+0000 7f290cff9640 1 -- 172.21.15.190:0/1069506377 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2900011830 con 0x7f29100c73f0 2024-04-24T23:15:13.776 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.771+0000 7f291671b640 1 -- 172.21.15.190:0/1069506377 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29101563d0 con 0x7f29100c73f0 2024-04-24T23:15:13.776 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.771+0000 7f290cff9640 1 -- 172.21.15.190:0/1069506377 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2900011c20 con 0x7f29100c73f0 2024-04-24T23:15:13.776 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.771+0000 7f291671b640 1 -- 172.21.15.190:0/1069506377 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29101542e0 con 0x7f29100c73f0 2024-04-24T23:15:13.776 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.771+0000 7f290cff9640 1 -- 172.21.15.190:0/1069506377 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f2900018020 con 0x7f29100c73f0 2024-04-24T23:15:13.776 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.771+0000 7f290cff9640 1 -- 172.21.15.190:0/1069506377 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 955+0+0 (secure 0 0 0) 0x7f2900019aa0 con 0x7f29100c73f0 2024-04-24T23:15:13.776 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.772+0000 7f291671b640 1 -- 172.21.15.190:0/1069506377 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f29101f1c00 con 0x7f29100c73f0 2024-04-24T23:15:13.776 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.774+0000 7f290cff9640 1 -- 172.21.15.190:0/1069506377 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f290001f020 con 0x7f29100c73f0 2024-04-24T23:15:13.849 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.847+0000 7f291671b640 1 -- 172.21.15.190:0/1069506377 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "osd new", "uuid": "d86bbc18-438c-4ec2-8018-ba7dd78f790b", "id": 2} v 0) -- 0x7f2910072e50 con 0x7f29100c73f0 2024-04-24T23:15:13.855 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.853+0000 7f290cff9640 1 -- 172.21.15.190:0/1069506377 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "d86bbc18-438c-4ec2-8018-ba7dd78f790b", "id": 2}]=0 v5) ==== 112+0+2 (secure 0 0 0) 0x7f2900019d40 con 0x7f29100c73f0 2024-04-24T23:15:13.855 INFO:teuthology.orchestra.run.smithi190.stdout:2 2024-04-24T23:15:13.856 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.854+0000 7f291671b640 1 -- 172.21.15.190:0/1069506377 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f29100c73f0 msgr2=0x7f29101522d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:13.856 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.854+0000 7f291671b640 1 --2- 172.21.15.190:0/1069506377 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f29100c73f0 0x7f29101522d0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f290000b3f0 tx=0x7f290000b8b0 comp rx=0 tx=0).stop 2024-04-24T23:15:13.856 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.854+0000 7f291671b640 1 -- 172.21.15.190:0/1069506377 shutdown_connections 2024-04-24T23:15:13.857 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.854+0000 7f291671b640 1 --2- 172.21.15.190:0/1069506377 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2910151aa0 0x7f29101f1f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:13.857 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.854+0000 7f291671b640 1 --2- 172.21.15.190:0/1069506377 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f291015aca0 0x7f29101f2450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:13.857 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.854+0000 7f291671b640 1 --2- 172.21.15.190:0/1069506377 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f29100c73f0 0x7f29101522d0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:13.857 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.854+0000 7f291671b640 1 -- 172.21.15.190:0/1069506377 >> 172.21.15.190:0/1069506377 conn(0x7f291006c1f0 msgr2=0x7f29100c7ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:15:13.857 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.854+0000 7f291671b640 1 -- 172.21.15.190:0/1069506377 shutdown_connections 2024-04-24T23:15:13.857 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.854+0000 7f291671b640 1 -- 172.21.15.190:0/1069506377 wait complete. 2024-04-24T23:15:13.867 DEBUG:teuthology.orchestra.run.smithi190:> sudo ceph --cluster ceph osd new 3754ae16-4248-40dd-95f9-0f59c0eb5426 3 2024-04-24T23:15:13.901 INFO:tasks.ceph.mgr.x.smithi190.stderr:2024-04-24T23:15:13.899+0000 7f4852779040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-24T23:15:13.963 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.962+0000 7f8184310640 1 Processor -- start 2024-04-24T23:15:13.964 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.962+0000 7f8184310640 1 -- start start 2024-04-24T23:15:13.966 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.963+0000 7f8184310640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f817c0ca690 0x7f817c15b610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:13.967 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.963+0000 7f8184310640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f817c0ce4c0 0x7f817c0ce930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:13.967 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.963+0000 7f8184310640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f817c0ca280 con 0x7f817c158470 2024-04-24T23:15:13.967 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.963+0000 7f8184310640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f817c15bb50 con 0x7f817c0ca690 2024-04-24T23:15:13.967 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.963+0000 7f8184310640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f817c158850 con 0x7f817c0ce4c0 2024-04-24T23:15:13.967 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.963+0000 7f81818ab640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f817c0ca690 0x7f817c15b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:13.967 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.963+0000 7f81828ad640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f817c0ce4c0 0x7f817c0ce930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:13.967 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.963+0000 7f81818ab640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f817c0ca690 0x7f817c15b610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.190:42066/0 (socket says 172.21.15.190:42066) 2024-04-24T23:15:13.967 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.963+0000 7f81818ab640 1 -- 172.21.15.190:0/2734391149 learned_addr learned my addr 172.21.15.190:0/2734391149 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-04-24T23:15:13.967 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.963+0000 7f81818ab640 1 -- 172.21.15.190:0/2734391149 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f817c0ce4c0 msgr2=0x7f817c0ce930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:15:13.967 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.963+0000 7f81818ab640 1 --2- 172.21.15.190:0/2734391149 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f817c0ce4c0 0x7f817c0ce930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:13.967 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.963+0000 7f81818ab640 1 -- 172.21.15.190:0/2734391149 >> v1:172.21.15.6:6789/0 conn(0x7f817c158470 legacy=0x7f817c0b2b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:15:13.967 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.963+0000 7f81818ab640 1 -- 172.21.15.190:0/2734391149 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f817c0c9ce0 con 0x7f817c0ca690 2024-04-24T23:15:13.968 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.963+0000 7f81828ad640 1 --2- 172.21.15.190:0/2734391149 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f817c0ce4c0 0x7f817c0ce930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:15:13.968 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.964+0000 7f81818ab640 1 --2- 172.21.15.190:0/2734391149 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f817c0ca690 0x7f817c15b610 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f816c009870 tx=0x7f816c02a2a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d0439be84c5670d3 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:15:13.968 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.964+0000 7f81810aa640 1 -- 172.21.15.190:0/2734391149 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f816c037070 con 0x7f817c0ca690 2024-04-24T23:15:13.968 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.964+0000 7f81810aa640 1 -- 172.21.15.190:0/2734391149 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f816c032440 con 0x7f817c0ca690 2024-04-24T23:15:13.968 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.964+0000 7f81810aa640 1 -- 172.21.15.190:0/2734391149 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f816c032810 con 0x7f817c0ca690 2024-04-24T23:15:13.968 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.965+0000 7f8184310640 1 -- 172.21.15.190:0/2734391149 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f817c0ca690 msgr2=0x7f817c15b610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:13.968 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.965+0000 7f8184310640 1 --2- 172.21.15.190:0/2734391149 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f817c0ca690 0x7f817c15b610 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f816c009870 tx=0x7f816c02a2a0 comp rx=0 tx=0).stop 2024-04-24T23:15:13.968 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.965+0000 7f8184310640 1 -- 172.21.15.190:0/2734391149 shutdown_connections 2024-04-24T23:15:13.969 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.965+0000 7f8184310640 1 --2- 172.21.15.190:0/2734391149 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f817c0ce4c0 0x7f817c0ce930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:13.969 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.965+0000 7f8184310640 1 --2- 172.21.15.190:0/2734391149 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f817c0ca690 0x7f817c15b610 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:13.969 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.965+0000 7f8184310640 1 -- 172.21.15.190:0/2734391149 >> 172.21.15.190:0/2734391149 conn(0x7f817c066ea0 msgr2=0x7f817c067280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:15:13.969 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.965+0000 7f8184310640 1 -- 172.21.15.190:0/2734391149 shutdown_connections 2024-04-24T23:15:13.969 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.965+0000 7f8184310640 1 -- 172.21.15.190:0/2734391149 wait complete. 2024-04-24T23:15:13.972 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.966+0000 7f8184310640 1 Processor -- start 2024-04-24T23:15:13.972 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.966+0000 7f8184310640 1 -- start start 2024-04-24T23:15:13.972 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.967+0000 7f8184310640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f817c0ca690 0x7f817c0aff50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:13.972 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.967+0000 7f8184310640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f817c0ce4c0 0x7f817c156bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:13.972 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.967+0000 7f8184310640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f817c158470 0x7f817c157110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:13.972 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.967+0000 7f8184310640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f817c06fbb0 con 0x7f817c158470 2024-04-24T23:15:13.972 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.967+0000 7f8184310640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f817c0c3900 con 0x7f817c0ca690 2024-04-24T23:15:13.972 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.967+0000 7f8184310640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f817c0c3590 con 0x7f817c0ce4c0 2024-04-24T23:15:13.972 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.967+0000 7f81820ac640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f817c0ca690 0x7f817c0aff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:13.972 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.967+0000 7f81818ab640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f817c0ce4c0 0x7f817c156bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:13.972 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.967+0000 7f81818ab640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f817c0ce4c0 0x7f817c156bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.190:40230/0 (socket says 172.21.15.190:40230) 2024-04-24T23:15:13.973 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.967+0000 7f81818ab640 1 -- 172.21.15.190:0/1891022867 learned_addr learned my addr 172.21.15.190:0/1891022867 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-04-24T23:15:13.973 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.967+0000 7f81820ac640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f817c0ca690 0x7f817c0aff50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.190:42072/0 (socket says 172.21.15.190:42072) 2024-04-24T23:15:13.973 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.967+0000 7f81820ac640 1 -- 172.21.15.190:0/1891022867 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f817c0ce4c0 msgr2=0x7f817c156bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:13.973 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.967+0000 7f81820ac640 1 --2- 172.21.15.190:0/1891022867 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f817c0ce4c0 0x7f817c156bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:13.973 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.967+0000 7f81820ac640 1 -- 172.21.15.190:0/1891022867 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f817c158470 msgr2=0x7f817c157110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:13.973 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.967+0000 7f81820ac640 1 --2- 172.21.15.190:0/1891022867 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f817c158470 0x7f817c157110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:13.973 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.967+0000 7f81820ac640 1 -- 172.21.15.190:0/1891022867 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f817c0b0600 con 0x7f817c0ca690 2024-04-24T23:15:13.973 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.967+0000 7f81818ab640 1 --2- 172.21.15.190:0/1891022867 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f817c0ce4c0 0x7f817c156bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:15:13.973 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.967+0000 7f81820ac640 1 --2- 172.21.15.190:0/1891022867 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f817c0ca690 0x7f817c0aff50 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f817800b3e0 tx=0x7f817800b8a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:15:13.973 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.967+0000 7f8172ffd640 1 -- 172.21.15.190:0/1891022867 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8178015020 con 0x7f817c0ca690 2024-04-24T23:15:13.973 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.967+0000 7f8172ffd640 1 -- 172.21.15.190:0/1891022867 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8178011830 con 0x7f817c0ca690 2024-04-24T23:15:13.973 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.967+0000 7f8184310640 1 -- 172.21.15.190:0/1891022867 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f817c0b08e0 con 0x7f817c0ca690 2024-04-24T23:15:13.973 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.967+0000 7f8172ffd640 1 -- 172.21.15.190:0/1891022867 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8178011c00 con 0x7f817c0ca690 2024-04-24T23:15:13.974 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.967+0000 7f8184310640 1 -- 172.21.15.190:0/1891022867 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f817c1560f0 con 0x7f817c0ca690 2024-04-24T23:15:13.974 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.968+0000 7f8172ffd640 1 -- 172.21.15.190:0/1891022867 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f8178018020 con 0x7f817c0ca690 2024-04-24T23:15:13.974 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.968+0000 7f8172ffd640 1 -- 172.21.15.190:0/1891022867 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1070+0+0 (secure 0 0 0) 0x7f8178019c50 con 0x7f817c0ca690 2024-04-24T23:15:13.974 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.968+0000 7f8184310640 1 -- 172.21.15.190:0/1891022867 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f817c074d80 con 0x7f817c0ca690 2024-04-24T23:15:13.974 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:13.970+0000 7f8172ffd640 1 -- 172.21.15.190:0/1891022867 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f817c074d80 con 0x7f817c0ca690 2024-04-24T23:15:13.993 INFO:tasks.ceph.mgr.x.smithi190.stderr:2024-04-24T23:15:13.992+0000 7f4852779040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-24T23:15:14.037 INFO:tasks.ceph.mgr.x.smithi190.stderr:2024-04-24T23:15:14.036+0000 7f4852779040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-24T23:15:14.044 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.043+0000 7f8184310640 1 -- 172.21.15.190:0/1891022867 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "osd new", "uuid": "3754ae16-4248-40dd-95f9-0f59c0eb5426", "id": 3} v 0) -- 0x7f817c068430 con 0x7f817c0ca690 2024-04-24T23:15:14.048 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.046+0000 7f8172ffd640 1 -- 172.21.15.190:0/1891022867 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "3754ae16-4248-40dd-95f9-0f59c0eb5426", "id": 3}]=0 v6) ==== 112+0+2 (secure 0 0 0) 0x7f817801f020 con 0x7f817c0ca690 2024-04-24T23:15:14.048 INFO:teuthology.orchestra.run.smithi190.stdout:3 2024-04-24T23:15:14.050 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.047+0000 7f8184310640 1 -- 172.21.15.190:0/1891022867 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f817c0ca690 msgr2=0x7f817c0aff50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:14.050 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.047+0000 7f8184310640 1 --2- 172.21.15.190:0/1891022867 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f817c0ca690 0x7f817c0aff50 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f817800b3e0 tx=0x7f817800b8a0 comp rx=0 tx=0).stop 2024-04-24T23:15:14.050 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.047+0000 7f8184310640 1 -- 172.21.15.190:0/1891022867 shutdown_connections 2024-04-24T23:15:14.050 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.047+0000 7f8184310640 1 --2- 172.21.15.190:0/1891022867 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f817c0ce4c0 0x7f817c156bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:14.050 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.047+0000 7f8184310640 1 --2- 172.21.15.190:0/1891022867 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f817c158470 0x7f817c157110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:14.050 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.047+0000 7f8184310640 1 --2- 172.21.15.190:0/1891022867 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f817c0ca690 0x7f817c0aff50 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:14.050 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.047+0000 7f8184310640 1 -- 172.21.15.190:0/1891022867 >> 172.21.15.190:0/1891022867 conn(0x7f817c066ea0 msgr2=0x7f817c0cdd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:15:14.050 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.047+0000 7f8184310640 1 -- 172.21.15.190:0/1891022867 shutdown_connections 2024-04-24T23:15:14.050 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.047+0000 7f8184310640 1 -- 172.21.15.190:0/1891022867 wait complete. 2024-04-24T23:15:14.060 DEBUG:teuthology.orchestra.run.smithi190:> sudo ceph --cluster ceph osd new 8b2913a0-d006-479c-84ed-5b21e0e6c392 4 2024-04-24T23:15:14.196 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.195+0000 7f9a25815640 1 Processor -- start 2024-04-24T23:15:14.197 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.195+0000 7f9a25815640 1 -- start start 2024-04-24T23:15:14.198 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.196+0000 7f9a25815640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f9a20152bf0 0x7f9a200c5140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:14.198 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.196+0000 7f9a25815640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9a200c8d20 0x7f9a200c99c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:14.198 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.196+0000 7f9a25815640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9a200ca8e0 con 0x7f9a200c9f00 2024-04-24T23:15:14.198 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.196+0000 7f9a25815640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f9a200cab80 con 0x7f9a20152bf0 2024-04-24T23:15:14.198 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.196+0000 7f9a25815640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f9a200c3340 con 0x7f9a200c8d20 2024-04-24T23:15:14.198 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.196+0000 7f9a1e7fc640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9a200c8d20 0x7f9a200c99c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:14.198 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.196+0000 7f9a1effd640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f9a20152bf0 0x7f9a200c5140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:14.198 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.196+0000 7f9a1effd640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f9a20152bf0 0x7f9a200c5140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.190:42088/0 (socket says 172.21.15.190:42088) 2024-04-24T23:15:14.199 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.196+0000 7f9a1e7fc640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9a200c8d20 0x7f9a200c99c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.190:40240/0 (socket says 172.21.15.190:40240) 2024-04-24T23:15:14.199 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.196+0000 7f9a1effd640 1 -- 172.21.15.190:0/2013115174 learned_addr learned my addr 172.21.15.190:0/2013115174 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-04-24T23:15:14.199 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.196+0000 7f9a1e7fc640 1 -- 172.21.15.190:0/2013115174 >> v2:172.21.15.190:3300/0 conn(0x7f9a20152bf0 msgr2=0x7f9a200c5140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:15:14.199 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.196+0000 7f9a1e7fc640 1 --2- 172.21.15.190:0/2013115174 >> v2:172.21.15.190:3300/0 conn(0x7f9a20152bf0 0x7f9a200c5140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:14.199 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.196+0000 7f9a1e7fc640 1 -- 172.21.15.190:0/2013115174 >> v1:172.21.15.6:6789/0 conn(0x7f9a200c9f00 legacy=0x7f9a200caed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:15:14.199 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.196+0000 7f9a1e7fc640 1 -- 172.21.15.190:0/2013115174 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a200c3620 con 0x7f9a200c8d20 2024-04-24T23:15:14.199 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.196+0000 7f9a1effd640 1 --2- 172.21.15.190:0/2013115174 >> v2:172.21.15.190:3300/0 conn(0x7f9a20152bf0 0x7f9a200c5140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:15:14.201 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.197+0000 7f9a1e7fc640 1 --2- 172.21.15.190:0/2013115174 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9a200c8d20 0x7f9a200c99c0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f9a20071810 tx=0x7f9a1002a130 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8e81991b82d1b22b server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:15:14.201 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.197+0000 7f9a1dffb640 1 -- 172.21.15.190:0/2013115174 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a10037070 con 0x7f9a200c8d20 2024-04-24T23:15:14.201 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.197+0000 7f9a1dffb640 1 -- 172.21.15.190:0/2013115174 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a1002ae40 con 0x7f9a200c8d20 2024-04-24T23:15:14.201 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.197+0000 7f9a1dffb640 1 -- 172.21.15.190:0/2013115174 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a10032660 con 0x7f9a200c8d20 2024-04-24T23:15:14.201 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.198+0000 7f9a25815640 1 -- 172.21.15.190:0/2013115174 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9a200c8d20 msgr2=0x7f9a200c99c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:14.201 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.198+0000 7f9a25815640 1 --2- 172.21.15.190:0/2013115174 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9a200c8d20 0x7f9a200c99c0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f9a20071810 tx=0x7f9a1002a130 comp rx=0 tx=0).stop 2024-04-24T23:15:14.201 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.198+0000 7f9a25815640 1 -- 172.21.15.190:0/2013115174 shutdown_connections 2024-04-24T23:15:14.201 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.198+0000 7f9a25815640 1 --2- 172.21.15.190:0/2013115174 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9a200c8d20 0x7f9a200c99c0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:14.201 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.198+0000 7f9a25815640 1 --2- 172.21.15.190:0/2013115174 >> v2:172.21.15.190:3300/0 conn(0x7f9a20152bf0 0x7f9a200c5140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:14.201 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.198+0000 7f9a25815640 1 -- 172.21.15.190:0/2013115174 >> 172.21.15.190:0/2013115174 conn(0x7f9a20070280 msgr2=0x7f9a2006f320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:15:14.201 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.198+0000 7f9a25815640 1 -- 172.21.15.190:0/2013115174 shutdown_connections 2024-04-24T23:15:14.201 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.198+0000 7f9a25815640 1 -- 172.21.15.190:0/2013115174 wait complete. 2024-04-24T23:15:14.202 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.199+0000 7f9a25815640 1 Processor -- start 2024-04-24T23:15:14.202 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.199+0000 7f9a25815640 1 -- start start 2024-04-24T23:15:14.204 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.200+0000 7f9a25815640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9a200c8d20 0x7f9a201479b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:14.204 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.200+0000 7f9a25815640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9a200c9f00 0x7f9a20150d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:14.204 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.200+0000 7f9a25815640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9a20152bf0 0x7f9a201512c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:14.204 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.200+0000 7f9a25815640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f9a200c9280 con 0x7f9a200c8d20 2024-04-24T23:15:14.204 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.200+0000 7f9a25815640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f9a200c9100 con 0x7f9a20152bf0 2024-04-24T23:15:14.204 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.200+0000 7f9a25815640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f9a20152fd0 con 0x7f9a200c9f00 2024-04-24T23:15:14.204 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.200+0000 7f9a1e7fc640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9a200c9f00 0x7f9a20150d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:14.205 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.200+0000 7f9a1f7fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9a20152bf0 0x7f9a201512c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:14.205 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.200+0000 7f9a1f7fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9a20152bf0 0x7f9a201512c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.190:42104/0 (socket says 172.21.15.190:42104) 2024-04-24T23:15:14.205 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.200+0000 7f9a1f7fe640 1 -- 172.21.15.190:0/1831781633 learned_addr learned my addr 172.21.15.190:0/1831781633 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-04-24T23:15:14.205 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.200+0000 7f9a1e7fc640 1 --2- 172.21.15.190:0/1831781633 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9a200c9f00 0x7f9a20150d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.190:40250/0 (socket says 172.21.15.190:40250) 2024-04-24T23:15:14.205 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.200+0000 7f9a1e7fc640 1 -- 172.21.15.190:0/1831781633 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9a20152bf0 msgr2=0x7f9a201512c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:14.205 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.200+0000 7f9a1e7fc640 1 --2- 172.21.15.190:0/1831781633 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9a20152bf0 0x7f9a201512c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:14.205 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.200+0000 7f9a1e7fc640 1 -- 172.21.15.190:0/1831781633 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9a200c8d20 msgr2=0x7f9a201479b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:14.205 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.200+0000 7f9a1e7fc640 1 --2- 172.21.15.190:0/1831781633 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9a200c8d20 0x7f9a201479b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:14.205 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.200+0000 7f9a1e7fc640 1 -- 172.21.15.190:0/1831781633 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a2006d8b0 con 0x7f9a200c9f00 2024-04-24T23:15:14.205 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.200+0000 7f9a1f7fe640 1 --2- 172.21.15.190:0/1831781633 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9a20152bf0 0x7f9a201512c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:15:14.205 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.200+0000 7f9a1e7fc640 1 --2- 172.21.15.190:0/1831781633 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9a200c9f00 0x7f9a20150d80 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f9a200c34c0 tx=0x7f9a100046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:15:14.205 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.200+0000 7f9a07fff640 1 -- 172.21.15.190:0/1831781633 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a10041020 con 0x7f9a200c9f00 2024-04-24T23:15:14.206 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.200+0000 7f9a07fff640 1 -- 172.21.15.190:0/1831781633 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a1003c670 con 0x7f9a200c9f00 2024-04-24T23:15:14.206 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.200+0000 7f9a07fff640 1 -- 172.21.15.190:0/1831781633 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a10037040 con 0x7f9a200c9f00 2024-04-24T23:15:14.206 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.200+0000 7f9a25815640 1 -- 172.21.15.190:0/1831781633 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a201f0de0 con 0x7f9a200c9f00 2024-04-24T23:15:14.206 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.201+0000 7f9a25815640 1 -- 172.21.15.190:0/1831781633 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a2006db20 con 0x7f9a200c9f00 2024-04-24T23:15:14.206 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.201+0000 7f9a07fff640 1 -- 172.21.15.190:0/1831781633 <== mon.2 v2:172.21.15.190:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f9a10044020 con 0x7f9a200c9f00 2024-04-24T23:15:14.206 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.201+0000 7f9a07fff640 1 -- 172.21.15.190:0/1831781633 <== mon.2 v2:172.21.15.190:3301/0 5 ==== osd_map(6..6 src has 1..6) ==== 1185+0+0 (secure 0 0 0) 0x7f9a1003dd30 con 0x7f9a200c9f00 2024-04-24T23:15:14.206 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.201+0000 7f9a25815640 1 -- 172.21.15.190:0/1831781633 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a200767b0 con 0x7f9a200c9f00 2024-04-24T23:15:14.206 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.203+0000 7f9a07fff640 1 -- 172.21.15.190:0/1831781633 <== mon.2 v2:172.21.15.190:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f9a1004b030 con 0x7f9a200c9f00 2024-04-24T23:15:14.213 INFO:tasks.ceph.mgr.x.smithi190.stderr:2024-04-24T23:15:14.211+0000 7f4852779040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-24T23:15:14.268 INFO:tasks.ceph.mgr.x.smithi190.stderr:2024-04-24T23:15:14.266+0000 7f4852779040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-24T23:15:14.278 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.276+0000 7f9a25815640 1 -- 172.21.15.190:0/1831781633 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "osd new", "uuid": "8b2913a0-d006-479c-84ed-5b21e0e6c392", "id": 4} v 0) -- 0x7f9a20150af0 con 0x7f9a200c9f00 2024-04-24T23:15:14.284 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.283+0000 7f9a07fff640 1 -- 172.21.15.190:0/1831781633 <== mon.2 v2:172.21.15.190:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "8b2913a0-d006-479c-84ed-5b21e0e6c392", "id": 4}]=0 v7) ==== 112+0+2 (secure 0 0 0) 0x7f9a10032e20 con 0x7f9a200c9f00 2024-04-24T23:15:14.284 INFO:teuthology.orchestra.run.smithi190.stdout:4 2024-04-24T23:15:14.286 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.284+0000 7f9a25815640 1 -- 172.21.15.190:0/1831781633 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9a200c9f00 msgr2=0x7f9a20150d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:14.286 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.284+0000 7f9a25815640 1 --2- 172.21.15.190:0/1831781633 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9a200c9f00 0x7f9a20150d80 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f9a200c34c0 tx=0x7f9a100046c0 comp rx=0 tx=0).stop 2024-04-24T23:15:14.286 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.284+0000 7f9a25815640 1 -- 172.21.15.190:0/1831781633 shutdown_connections 2024-04-24T23:15:14.286 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.284+0000 7f9a25815640 1 --2- 172.21.15.190:0/1831781633 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9a200c9f00 0x7f9a20150d80 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:14.286 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.284+0000 7f9a25815640 1 --2- 172.21.15.190:0/1831781633 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9a20152bf0 0x7f9a201512c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:14.286 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.284+0000 7f9a25815640 1 --2- 172.21.15.190:0/1831781633 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9a200c8d20 0x7f9a201479b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:14.286 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.284+0000 7f9a25815640 1 -- 172.21.15.190:0/1831781633 >> 172.21.15.190:0/1831781633 conn(0x7f9a20070280 msgr2=0x7f9a200c66a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:15:14.286 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.284+0000 7f9a25815640 1 -- 172.21.15.190:0/1831781633 shutdown_connections 2024-04-24T23:15:14.286 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.284+0000 7f9a25815640 1 -- 172.21.15.190:0/1831781633 wait complete. 2024-04-24T23:15:14.296 DEBUG:teuthology.orchestra.run.smithi190:> sudo ceph --cluster ceph osd new 499b79d0-0585-4dae-ab69-196a551793da 5 2024-04-24T23:15:14.325 INFO:tasks.ceph.mgr.x.smithi190.stderr:2024-04-24T23:15:14.323+0000 7f4852779040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-24T23:15:14.330 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-04-24T23:15:14.329+0000 7f400607a040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-24T23:15:14.393 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.391+0000 7efea9d4f640 1 Processor -- start 2024-04-24T23:15:14.393 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.391+0000 7efea9d4f640 1 -- start start 2024-04-24T23:15:14.394 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.392+0000 7efea9d4f640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7efea415c1b0 0x7efea40cb810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:14.394 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.392+0000 7efea9d4f640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7efea415e2b0 0x7efea415e740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:14.394 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.392+0000 7efea9d4f640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efea40ca900 con 0x7efea40ca520 2024-04-24T23:15:14.394 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.392+0000 7efea9d4f640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7efea40ccf70 con 0x7efea415e2b0 2024-04-24T23:15:14.395 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.392+0000 7efea9d4f640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7efea40cc360 con 0x7efea415c1b0 2024-04-24T23:15:14.395 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.392+0000 7efe9bfff640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7efea415c1b0 0x7efea40cb810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:14.395 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.392+0000 7efe9bfff640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7efea415c1b0 0x7efea40cb810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.190:42120/0 (socket says 172.21.15.190:42120) 2024-04-24T23:15:14.395 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.392+0000 7efe9bfff640 1 -- 172.21.15.190:0/1915274913 learned_addr learned my addr 172.21.15.190:0/1915274913 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-04-24T23:15:14.395 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.392+0000 7efe9bfff640 1 -- 172.21.15.190:0/1915274913 >> v2:172.21.15.6:3300/0 conn(0x7efea415e2b0 msgr2=0x7efea415e740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:15:14.395 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.392+0000 7efe9bfff640 1 --2- 172.21.15.190:0/1915274913 >> v2:172.21.15.6:3300/0 conn(0x7efea415e2b0 0x7efea415e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:14.395 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.392+0000 7efe9bfff640 1 -- 172.21.15.190:0/1915274913 >> v1:172.21.15.6:6789/0 conn(0x7efea40ca520 legacy=0x7efea40cd400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:15:14.395 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.392+0000 7efe9bfff640 1 -- 172.21.15.190:0/1915274913 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efea40cc640 con 0x7efea415c1b0 2024-04-24T23:15:14.396 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.393+0000 7efe9bfff640 1 --2- 172.21.15.190:0/1915274913 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efea415c1b0 0x7efea40cb810 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7efea4155680 tx=0x7efe8c02a2c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7ee947238ed7c6d5 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:15:14.396 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.394+0000 7efea2ffd640 1 -- 172.21.15.190:0/1915274913 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe8c037070 con 0x7efea415c1b0 2024-04-24T23:15:14.396 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.394+0000 7efea2ffd640 1 -- 172.21.15.190:0/1915274913 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efe8c032440 con 0x7efea415c1b0 2024-04-24T23:15:14.397 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.394+0000 7efea2ffd640 1 -- 172.21.15.190:0/1915274913 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe8c032810 con 0x7efea415c1b0 2024-04-24T23:15:14.397 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.394+0000 7efea9d4f640 1 -- 172.21.15.190:0/1915274913 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efea415c1b0 msgr2=0x7efea40cb810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:14.397 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.394+0000 7efea9d4f640 1 --2- 172.21.15.190:0/1915274913 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efea415c1b0 0x7efea40cb810 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7efea4155680 tx=0x7efe8c02a2c0 comp rx=0 tx=0).stop 2024-04-24T23:15:14.397 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.394+0000 7efea9d4f640 1 -- 172.21.15.190:0/1915274913 shutdown_connections 2024-04-24T23:15:14.397 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.394+0000 7efea9d4f640 1 --2- 172.21.15.190:0/1915274913 >> v2:172.21.15.6:3300/0 conn(0x7efea415e2b0 0x7efea415e740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:14.397 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.394+0000 7efea9d4f640 1 --2- 172.21.15.190:0/1915274913 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efea415c1b0 0x7efea40cb810 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:14.397 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.394+0000 7efea9d4f640 1 -- 172.21.15.190:0/1915274913 >> 172.21.15.190:0/1915274913 conn(0x7efea406c0f0 msgr2=0x7efea4066ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:15:14.397 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.394+0000 7efea9d4f640 1 -- 172.21.15.190:0/1915274913 shutdown_connections 2024-04-24T23:15:14.397 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.394+0000 7efea9d4f640 1 -- 172.21.15.190:0/1915274913 wait complete. 2024-04-24T23:15:14.400 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.395+0000 7efea9d4f640 1 Processor -- start 2024-04-24T23:15:14.400 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.396+0000 7efea9d4f640 1 -- start start 2024-04-24T23:15:14.400 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.396+0000 7efea9d4f640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efea40ca520 0x7efea40c67b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:14.400 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.396+0000 7efea9d4f640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efea415c1b0 0x7efea41f0b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:14.400 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.396+0000 7efea9d4f640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efea415e2b0 0x7efea41f1060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:14.400 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.396+0000 7efea9d4f640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7efea4067890 con 0x7efea415c1b0 2024-04-24T23:15:14.401 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.396+0000 7efea9d4f640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7efea40ccc00 con 0x7efea40ca520 2024-04-24T23:15:14.401 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.396+0000 7efea9d4f640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7efea4067bc0 con 0x7efea415e2b0 2024-04-24T23:15:14.401 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.396+0000 7efea37fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efea40ca520 0x7efea40c67b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:14.401 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.396+0000 7efea3fff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efea415e2b0 0x7efea41f1060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:14.401 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.396+0000 7efea3fff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efea415e2b0 0x7efea41f1060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.190:40266/0 (socket says 172.21.15.190:40266) 2024-04-24T23:15:14.401 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.396+0000 7efea37fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efea40ca520 0x7efea40c67b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.190:42126/0 (socket says 172.21.15.190:42126) 2024-04-24T23:15:14.401 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.396+0000 7efea3fff640 1 -- 172.21.15.190:0/324172799 learned_addr learned my addr 172.21.15.190:0/324172799 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-04-24T23:15:14.401 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.397+0000 7efea3fff640 1 -- 172.21.15.190:0/324172799 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efea40ca520 msgr2=0x7efea40c67b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:14.401 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.397+0000 7efea3fff640 1 --2- 172.21.15.190:0/324172799 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efea40ca520 0x7efea40c67b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:14.401 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.397+0000 7efea3fff640 1 -- 172.21.15.190:0/324172799 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efea415c1b0 msgr2=0x7efea41f0b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:14.401 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.397+0000 7efea3fff640 1 --2- 172.21.15.190:0/324172799 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efea415c1b0 0x7efea41f0b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:14.401 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.397+0000 7efea3fff640 1 -- 172.21.15.190:0/324172799 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efea40c6cf0 con 0x7efea415e2b0 2024-04-24T23:15:14.401 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.397+0000 7efea37fe640 1 --2- 172.21.15.190:0/324172799 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efea40ca520 0x7efea40c67b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:15:14.401 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.397+0000 7efea3fff640 1 --2- 172.21.15.190:0/324172799 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efea415e2b0 0x7efea41f1060 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7efe9c00b970 tx=0x7efe9c00be30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:15:14.402 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.397+0000 7efea0ff9640 1 -- 172.21.15.190:0/324172799 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe9c019070 con 0x7efea415e2b0 2024-04-24T23:15:14.402 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.397+0000 7efea0ff9640 1 -- 172.21.15.190:0/324172799 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efe9c00cb60 con 0x7efea415e2b0 2024-04-24T23:15:14.402 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.397+0000 7efea9d4f640 1 -- 172.21.15.190:0/324172799 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efea41f4ca0 con 0x7efea415e2b0 2024-04-24T23:15:14.402 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.397+0000 7efea0ff9640 1 -- 172.21.15.190:0/324172799 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe9c014440 con 0x7efea415e2b0 2024-04-24T23:15:14.402 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.397+0000 7efea9d4f640 1 -- 172.21.15.190:0/324172799 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7efea414cd40 con 0x7efea415e2b0 2024-04-24T23:15:14.402 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.397+0000 7efea0ff9640 1 -- 172.21.15.190:0/324172799 <== mon.2 v2:172.21.15.190:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7efe9c01c070 con 0x7efea415e2b0 2024-04-24T23:15:14.402 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.397+0000 7efea0ff9640 1 -- 172.21.15.190:0/324172799 <== mon.2 v2:172.21.15.190:3301/0 5 ==== osd_map(7..7 src has 1..7) ==== 1300+0+0 (secure 0 0 0) 0x7efe9c01d930 con 0x7efea415e2b0 2024-04-24T23:15:14.402 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.398+0000 7efea9d4f640 1 -- 172.21.15.190:0/324172799 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efea40c5e00 con 0x7efea415e2b0 2024-04-24T23:15:14.402 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.400+0000 7efea0ff9640 1 -- 172.21.15.190:0/324172799 <== mon.2 v2:172.21.15.190:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7efe9c023070 con 0x7efea415e2b0 2024-04-24T23:15:14.438 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-04-24T23:15:14.437+0000 7f400607a040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-24T23:15:14.444 INFO:tasks.ceph.mgr.x.smithi190.stderr:2024-04-24T23:15:14.442+0000 7f4852779040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-24T23:15:14.473 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.471+0000 7efea9d4f640 1 -- 172.21.15.190:0/324172799 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "osd new", "uuid": "499b79d0-0585-4dae-ab69-196a551793da", "id": 5} v 0) -- 0x7efea40767b0 con 0x7efea415e2b0 2024-04-24T23:15:14.479 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.477+0000 7efea0ff9640 1 -- 172.21.15.190:0/324172799 <== mon.2 v2:172.21.15.190:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "499b79d0-0585-4dae-ab69-196a551793da", "id": 5}]=0 v8) ==== 112+0+2 (secure 0 0 0) 0x7efe9c01db80 con 0x7efea415e2b0 2024-04-24T23:15:14.479 INFO:teuthology.orchestra.run.smithi190.stdout:5 2024-04-24T23:15:14.480 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.478+0000 7efea9d4f640 1 -- 172.21.15.190:0/324172799 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efea415e2b0 msgr2=0x7efea41f1060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:14.480 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.478+0000 7efea9d4f640 1 --2- 172.21.15.190:0/324172799 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efea415e2b0 0x7efea41f1060 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7efe9c00b970 tx=0x7efe9c00be30 comp rx=0 tx=0).stop 2024-04-24T23:15:14.480 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.478+0000 7efea9d4f640 1 -- 172.21.15.190:0/324172799 shutdown_connections 2024-04-24T23:15:14.480 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.478+0000 7efea9d4f640 1 --2- 172.21.15.190:0/324172799 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efea415e2b0 0x7efea41f1060 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:14.481 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.478+0000 7efea9d4f640 1 --2- 172.21.15.190:0/324172799 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efea415c1b0 0x7efea41f0b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:14.481 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.478+0000 7efea9d4f640 1 --2- 172.21.15.190:0/324172799 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efea40ca520 0x7efea40c67b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:14.481 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.478+0000 7efea9d4f640 1 -- 172.21.15.190:0/324172799 >> 172.21.15.190:0/324172799 conn(0x7efea406c0f0 msgr2=0x7efea40c8c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:15:14.481 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.478+0000 7efea9d4f640 1 -- 172.21.15.190:0/324172799 shutdown_connections 2024-04-24T23:15:14.481 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.478+0000 7efea9d4f640 1 -- 172.21.15.190:0/324172799 wait complete. 2024-04-24T23:15:14.491 DEBUG:teuthology.orchestra.run.smithi190:> sudo ceph --cluster ceph osd new 6ba6df3b-4a6b-4a89-a890-0f79f387d4d7 6 2024-04-24T23:15:14.491 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-04-24T23:15:14.490+0000 7f400607a040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-24T23:15:14.628 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.626+0000 7f5c0c488640 1 Processor -- start 2024-04-24T23:15:14.628 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.626+0000 7f5c0c488640 1 -- start start 2024-04-24T23:15:14.631 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.627+0000 7f5c0c488640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f5c04152d50 0x7f5c040c6b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:14.631 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.627+0000 7f5c0c488640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f5c0406d0b0 0x7f5c040c7050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:14.631 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.627+0000 7f5c0c488640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c040c7ea0 con 0x7f5c040c96d0 2024-04-24T23:15:14.631 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.627+0000 7f5c0c488640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f5c040c8420 con 0x7f5c0406d0b0 2024-04-24T23:15:14.632 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.627+0000 7f5c0c488640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f5c040c8140 con 0x7f5c04152d50 2024-04-24T23:15:14.632 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.627+0000 7f5c0a224640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f5c04152d50 0x7f5c040c6b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:14.632 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.627+0000 7f5c0aa25640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f5c040c96d0 0x7f5c0415be70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.190:53294/0 (socket says 172.21.15.190:53294) 2024-04-24T23:15:14.632 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.627+0000 7f5c0aa25640 1 -- 172.21.15.190:0/2487113602 learned_addr learned my addr 172.21.15.190:0/2487113602 (peer_addr_for_me v1:172.21.15.190:0/0) 2024-04-24T23:15:14.632 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.627+0000 7f5c0a224640 1 -- 172.21.15.190:0/2487113602 >> v2:172.21.15.6:3300/0 conn(0x7f5c0406d0b0 msgr2=0x7f5c040c7050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:15:14.632 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.627+0000 7f5c0a224640 1 --2- 172.21.15.190:0/2487113602 >> v2:172.21.15.6:3300/0 conn(0x7f5c0406d0b0 0x7f5c040c7050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:14.632 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.627+0000 7f5c09222640 1 -- 172.21.15.190:0/2487113602 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1617403394 0 0) 0x7f5c040c7ea0 con 0x7f5c040c96d0 2024-04-24T23:15:14.632 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.627+0000 7f5c0a224640 1 -- 172.21.15.190:0/2487113602 >> v1:172.21.15.190:6789/0 conn(0x7f5c040c96d0 legacy=0x7f5c0415be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:14.632 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.627+0000 7f5c0a224640 1 -- 172.21.15.190:0/2487113602 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c0406b180 con 0x7f5c04152d50 2024-04-24T23:15:14.632 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.628+0000 7f5c0a224640 1 --2- 172.21.15.190:0/2487113602 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5c04152d50 0x7f5c040c6b10 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f5c04070a00 tx=0x7f5c0002a1a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=940e6a3f2c8b569f server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:15:14.633 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.628+0000 7f5c09222640 1 -- 172.21.15.190:0/2487113602 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c00037070 con 0x7f5c04152d50 2024-04-24T23:15:14.633 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.628+0000 7f5c09222640 1 -- 172.21.15.190:0/2487113602 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5c00032440 con 0x7f5c04152d50 2024-04-24T23:15:14.633 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.628+0000 7f5c09222640 1 -- 172.21.15.190:0/2487113602 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c00032810 con 0x7f5c04152d50 2024-04-24T23:15:14.633 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.629+0000 7f5c0c488640 1 -- 172.21.15.190:0/2487113602 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5c04152d50 msgr2=0x7f5c040c6b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:14.633 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.629+0000 7f5c0c488640 1 --2- 172.21.15.190:0/2487113602 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5c04152d50 0x7f5c040c6b10 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f5c04070a00 tx=0x7f5c0002a1a0 comp rx=0 tx=0).stop 2024-04-24T23:15:14.633 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.629+0000 7f5c0c488640 1 -- 172.21.15.190:0/2487113602 shutdown_connections 2024-04-24T23:15:14.633 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.629+0000 7f5c0c488640 1 --2- 172.21.15.190:0/2487113602 >> v2:172.21.15.6:3300/0 conn(0x7f5c0406d0b0 0x7f5c040c7050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:14.633 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.629+0000 7f5c0c488640 1 --2- 172.21.15.190:0/2487113602 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5c04152d50 0x7f5c040c6b10 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:14.633 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.629+0000 7f5c0c488640 1 -- 172.21.15.190:0/2487113602 >> 172.21.15.190:0/2487113602 conn(0x7f5c0406e780 msgr2=0x7f5c0406fdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:15:14.634 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.629+0000 7f5c0c488640 1 -- 172.21.15.190:0/2487113602 shutdown_connections 2024-04-24T23:15:14.634 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.629+0000 7f5c0c488640 1 -- 172.21.15.190:0/2487113602 wait complete. 2024-04-24T23:15:14.637 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.630+0000 7f5c0c488640 1 Processor -- start 2024-04-24T23:15:14.637 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.630+0000 7f5c0c488640 1 -- start start 2024-04-24T23:15:14.637 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.631+0000 7f5c0c488640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5c0406d0b0 0x7f5c0415ba90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:14.637 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.631+0000 7f5c0c488640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5c040c96d0 0x7f5c0414f800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:14.638 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.631+0000 7f5c0c488640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5c04152d50 0x7f5c041f1310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:14.638 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.631+0000 7f5c0c488640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f5c040c8740 con 0x7f5c0406d0b0 2024-04-24T23:15:14.638 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.631+0000 7f5c0c488640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f5c040c9290 con 0x7f5c04152d50 2024-04-24T23:15:14.638 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.631+0000 7f5c0c488640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f5c040c88c0 con 0x7f5c040c96d0 2024-04-24T23:15:14.638 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.631+0000 7f5c0aa25640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5c04152d50 0x7f5c041f1310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:14.638 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.631+0000 7f5c09a23640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5c040c96d0 0x7f5c0414f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:14.638 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.631+0000 7f5c0aa25640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5c04152d50 0x7f5c041f1310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.190:42138/0 (socket says 172.21.15.190:42138) 2024-04-24T23:15:14.638 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.631+0000 7f5c0aa25640 1 -- 172.21.15.190:0/3237537266 learned_addr learned my addr 172.21.15.190:0/3237537266 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-04-24T23:15:14.638 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.631+0000 7f5c0aa25640 1 -- 172.21.15.190:0/3237537266 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5c040c96d0 msgr2=0x7f5c0414f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:14.638 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.631+0000 7f5c0aa25640 1 --2- 172.21.15.190:0/3237537266 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5c040c96d0 0x7f5c0414f800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:14.638 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.631+0000 7f5c0aa25640 1 -- 172.21.15.190:0/3237537266 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5c0406d0b0 msgr2=0x7f5c0415ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:14.639 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.631+0000 7f5c0aa25640 1 --2- 172.21.15.190:0/3237537266 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5c0406d0b0 0x7f5c0415ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:14.639 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.631+0000 7f5c0aa25640 1 -- 172.21.15.190:0/3237537266 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c040c5e90 con 0x7f5c04152d50 2024-04-24T23:15:14.639 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.631+0000 7f5c09a23640 1 --2- 172.21.15.190:0/3237537266 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5c040c96d0 0x7f5c0414f800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:15:14.639 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.632+0000 7f5c0aa25640 1 --2- 172.21.15.190:0/3237537266 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5c04152d50 0x7f5c041f1310 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f5bfc00c730 tx=0x7f5bfc00cbf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:15:14.639 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.632+0000 7f5bfaffd640 1 -- 172.21.15.190:0/3237537266 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bfc018070 con 0x7f5c04152d50 2024-04-24T23:15:14.639 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.632+0000 7f5bfaffd640 1 -- 172.21.15.190:0/3237537266 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5bfc00ce40 con 0x7f5c04152d50 2024-04-24T23:15:14.639 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.632+0000 7f5c0c488640 1 -- 172.21.15.190:0/3237537266 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c040c6170 con 0x7f5c04152d50 2024-04-24T23:15:14.639 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.632+0000 7f5bfaffd640 1 -- 172.21.15.190:0/3237537266 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bfc00a860 con 0x7f5c04152d50 2024-04-24T23:15:14.639 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.632+0000 7f5c0c488640 1 -- 172.21.15.190:0/3237537266 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c040c6730 con 0x7f5c04152d50 2024-04-24T23:15:14.639 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.632+0000 7f5bfaffd640 1 -- 172.21.15.190:0/3237537266 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f5bfc01b070 con 0x7f5c04152d50 2024-04-24T23:15:14.639 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.632+0000 7f5bfaffd640 1 -- 172.21.15.190:0/3237537266 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(8..8 src has 1..8) ==== 1415+0+0 (secure 0 0 0) 0x7f5bfc007800 con 0x7f5c04152d50 2024-04-24T23:15:14.639 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.633+0000 7f5c0c488640 1 -- 172.21.15.190:0/3237537266 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5c040c7af0 con 0x7f5c04152d50 2024-04-24T23:15:14.639 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.635+0000 7f5bfaffd640 1 -- 172.21.15.190:0/3237537266 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f5c040c7af0 con 0x7f5c04152d50 2024-04-24T23:15:14.690 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-04-24T23:15:14.688+0000 7f400607a040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-24T23:15:14.707 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.706+0000 7f5c0c488640 1 -- 172.21.15.190:0/3237537266 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "osd new", "uuid": "6ba6df3b-4a6b-4a89-a890-0f79f387d4d7", "id": 6} v 0) -- 0x7f5c041f0be0 con 0x7f5c04152d50 2024-04-24T23:15:14.714 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.713+0000 7f5bfaffd640 1 -- 172.21.15.190:0/3237537266 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "6ba6df3b-4a6b-4a89-a890-0f79f387d4d7", "id": 6}]=0 v9) ==== 112+0+2 (secure 0 0 0) 0x7f5bfc007200 con 0x7f5c04152d50 2024-04-24T23:15:14.714 INFO:teuthology.orchestra.run.smithi190.stdout:6 2024-04-24T23:15:14.715 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.713+0000 7f5c0c488640 1 -- 172.21.15.190:0/3237537266 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5c04152d50 msgr2=0x7f5c041f1310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:14.715 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.713+0000 7f5c0c488640 1 --2- 172.21.15.190:0/3237537266 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5c04152d50 0x7f5c041f1310 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f5bfc00c730 tx=0x7f5bfc00cbf0 comp rx=0 tx=0).stop 2024-04-24T23:15:14.716 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.713+0000 7f5c0c488640 1 -- 172.21.15.190:0/3237537266 shutdown_connections 2024-04-24T23:15:14.716 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.713+0000 7f5c0c488640 1 --2- 172.21.15.190:0/3237537266 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5c040c96d0 0x7f5c0414f800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:14.716 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.713+0000 7f5c0c488640 1 --2- 172.21.15.190:0/3237537266 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5c04152d50 0x7f5c041f1310 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:14.716 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.713+0000 7f5c0c488640 1 --2- 172.21.15.190:0/3237537266 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5c0406d0b0 0x7f5c0415ba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:14.716 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.714+0000 7f5c0c488640 1 -- 172.21.15.190:0/3237537266 >> 172.21.15.190:0/3237537266 conn(0x7f5c0406e780 msgr2=0x7f5c040724b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:15:14.716 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.714+0000 7f5c0c488640 1 -- 172.21.15.190:0/3237537266 shutdown_connections 2024-04-24T23:15:14.716 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.714+0000 7f5c0c488640 1 -- 172.21.15.190:0/3237537266 wait complete. 2024-04-24T23:15:14.725 DEBUG:teuthology.orchestra.run.smithi190:> sudo ceph --cluster ceph osd new e9668b1c-a000-4332-86e8-4efd73b02409 7 2024-04-24T23:15:14.748 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-04-24T23:15:14.746+0000 7f400607a040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-24T23:15:14.811 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-04-24T23:15:14.810+0000 7f400607a040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-24T23:15:14.822 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.821+0000 7ff1eecbd640 1 Processor -- start 2024-04-24T23:15:14.823 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.821+0000 7ff1eecbd640 1 -- start start 2024-04-24T23:15:14.824 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.822+0000 7ff1eecbd640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff1e815b9c0 0x7ff1e80ca010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:14.824 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.822+0000 7ff1eecbd640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff1e8151fb0 con 0x7ff1e8151bd0 2024-04-24T23:15:14.824 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.822+0000 7ff1eecbd640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff1e8073350 con 0x7ff1e8071800 2024-04-24T23:15:14.824 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.822+0000 7ff1eecbd640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7ff1e80c95d0 con 0x7ff1e815b9c0 2024-04-24T23:15:14.824 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.822+0000 7ff1ed25a640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff1e815b9c0 0x7ff1e80ca010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:14.824 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.822+0000 7ff1e7fff640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7ff1e8071800 0x7ff1e815b2b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.190:53302/0 (socket says 172.21.15.190:53302) 2024-04-24T23:15:14.824 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.822+0000 7ff1ed25a640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff1e815b9c0 0x7ff1e80ca010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.190:40288/0 (socket says 172.21.15.190:40288) 2024-04-24T23:15:14.824 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.822+0000 7ff1e7fff640 1 -- 172.21.15.190:0/3427806768 learned_addr learned my addr 172.21.15.190:0/3427806768 (peer_addr_for_me v1:172.21.15.190:0/0) 2024-04-24T23:15:14.825 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.822+0000 7ff1ed25a640 1 -- 172.21.15.190:0/3427806768 >> v1:172.21.15.190:6789/0 conn(0x7ff1e8071800 legacy=0x7ff1e815b2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:14.825 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.822+0000 7ff1ed25a640 1 -- 172.21.15.190:0/3427806768 >> v1:172.21.15.6:6789/0 conn(0x7ff1e8151bd0 legacy=0x7ff1e80c8650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:15:14.825 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.822+0000 7ff1e77fe640 1 -- 172.21.15.190:0/3427806768 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1439822924 0 0) 0x7ff1e8073350 con 0x7ff1e8071800 2024-04-24T23:15:14.825 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.822+0000 7ff1ed25a640 1 -- 172.21.15.190:0/3427806768 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1e80735f0 con 0x7ff1e815b9c0 2024-04-24T23:15:14.826 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.823+0000 7ff1ed25a640 1 --2- 172.21.15.190:0/3427806768 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff1e815b9c0 0x7ff1e80ca010 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7ff1e0009dc0 tx=0x7ff1e002a710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d88ed6c0313203c5 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:15:14.826 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.823+0000 7ff1e77fe640 1 -- 172.21.15.190:0/3427806768 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1e0007740 con 0x7ff1e815b9c0 2024-04-24T23:15:14.826 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.823+0000 7ff1e77fe640 1 -- 172.21.15.190:0/3427806768 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff1e00078e0 con 0x7ff1e815b9c0 2024-04-24T23:15:14.827 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.823+0000 7ff1e77fe640 1 -- 172.21.15.190:0/3427806768 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1e0007cb0 con 0x7ff1e815b9c0 2024-04-24T23:15:14.827 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.824+0000 7ff1eecbd640 1 -- 172.21.15.190:0/3427806768 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff1e815b9c0 msgr2=0x7ff1e80ca010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:14.827 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.824+0000 7ff1eecbd640 1 --2- 172.21.15.190:0/3427806768 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff1e815b9c0 0x7ff1e80ca010 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7ff1e0009dc0 tx=0x7ff1e002a710 comp rx=0 tx=0).stop 2024-04-24T23:15:14.827 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.824+0000 7ff1eecbd640 1 -- 172.21.15.190:0/3427806768 shutdown_connections 2024-04-24T23:15:14.827 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.824+0000 7ff1eecbd640 1 --2- 172.21.15.190:0/3427806768 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff1e815b9c0 0x7ff1e80ca010 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:14.827 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.824+0000 7ff1eecbd640 1 -- 172.21.15.190:0/3427806768 >> 172.21.15.190:0/3427806768 conn(0x7ff1e806dcb0 msgr2=0x7ff1e806f390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:15:14.827 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.824+0000 7ff1eecbd640 1 -- 172.21.15.190:0/3427806768 shutdown_connections 2024-04-24T23:15:14.827 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.824+0000 7ff1eecbd640 1 -- 172.21.15.190:0/3427806768 wait complete. 2024-04-24T23:15:14.827 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.825+0000 7ff1eecbd640 1 Processor -- start 2024-04-24T23:15:14.827 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.825+0000 7ff1eecbd640 1 -- start start 2024-04-24T23:15:14.829 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.826+0000 7ff1eecbd640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff1e8071800 0x7ff1e814fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:14.830 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.826+0000 7ff1eecbd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff1e8151bd0 0x7ff1e81f4570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:14.830 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.826+0000 7ff1eecbd640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff1e815b9c0 0x7ff1e81f4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:14.830 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.826+0000 7ff1eecbd640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7ff1e815ad90 con 0x7ff1e8071800 2024-04-24T23:15:14.830 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.826+0000 7ff1eecbd640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7ff1e80c9b00 con 0x7ff1e8151bd0 2024-04-24T23:15:14.830 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.826+0000 7ff1eecbd640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7ff1e80c8d60 con 0x7ff1e815b9c0 2024-04-24T23:15:14.830 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.826+0000 7ff1e7fff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff1e8151bd0 0x7ff1e81f4570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:14.830 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.826+0000 7ff1ed25a640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff1e815b9c0 0x7ff1e81f4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:14.830 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.826+0000 7ff1e7fff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff1e8151bd0 0x7ff1e81f4570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.190:42142/0 (socket says 172.21.15.190:42142) 2024-04-24T23:15:14.830 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.826+0000 7ff1e7fff640 1 -- 172.21.15.190:0/49125161 learned_addr learned my addr 172.21.15.190:0/49125161 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-04-24T23:15:14.830 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.826+0000 7ff1ed25a640 1 -- 172.21.15.190:0/49125161 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff1e8151bd0 msgr2=0x7ff1e81f4570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:14.830 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.826+0000 7ff1ed25a640 1 --2- 172.21.15.190:0/49125161 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff1e8151bd0 0x7ff1e81f4570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:14.830 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.826+0000 7ff1ed25a640 1 -- 172.21.15.190:0/49125161 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff1e8071800 msgr2=0x7ff1e814fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:14.831 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.826+0000 7ff1ed25a640 1 --2- 172.21.15.190:0/49125161 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff1e8071800 0x7ff1e814fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:14.831 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.826+0000 7ff1ed25a640 1 -- 172.21.15.190:0/49125161 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1e81f0d80 con 0x7ff1e815b9c0 2024-04-24T23:15:14.831 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.826+0000 7ff1e7fff640 1 --2- 172.21.15.190:0/49125161 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff1e8151bd0 0x7ff1e81f4570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:15:14.831 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.826+0000 7ff1ed25a640 1 --2- 172.21.15.190:0/49125161 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff1e815b9c0 0x7ff1e81f4ab0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7ff1e0031000 tx=0x7ff1e0004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:15:14.831 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.827+0000 7ff1e57fa640 1 -- 172.21.15.190:0/49125161 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1e0041030 con 0x7ff1e815b9c0 2024-04-24T23:15:14.831 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.827+0000 7ff1e57fa640 1 -- 172.21.15.190:0/49125161 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff1e00319c0 con 0x7ff1e815b9c0 2024-04-24T23:15:14.831 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.827+0000 7ff1eecbd640 1 -- 172.21.15.190:0/49125161 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1e81f1060 con 0x7ff1e815b9c0 2024-04-24T23:15:14.831 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.827+0000 7ff1e57fa640 1 -- 172.21.15.190:0/49125161 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1e0031d90 con 0x7ff1e815b9c0 2024-04-24T23:15:14.831 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.827+0000 7ff1eecbd640 1 -- 172.21.15.190:0/49125161 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1e81f1620 con 0x7ff1e815b9c0 2024-04-24T23:15:14.831 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.827+0000 7ff1e57fa640 1 -- 172.21.15.190:0/49125161 <== mon.2 v2:172.21.15.190:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7ff1e0044030 con 0x7ff1e815b9c0 2024-04-24T23:15:14.831 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.827+0000 7ff1e57fa640 1 -- 172.21.15.190:0/49125161 <== mon.2 v2:172.21.15.190:3301/0 5 ==== osd_map(9..9 src has 1..9) ==== 1530+0+0 (secure 0 0 0) 0x7ff1e0046b10 con 0x7ff1e815b9c0 2024-04-24T23:15:14.831 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.828+0000 7ff1eecbd640 1 -- 172.21.15.190:0/49125161 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff1e8068b40 con 0x7ff1e815b9c0 2024-04-24T23:15:14.832 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.830+0000 7ff1e57fa640 1 -- 172.21.15.190:0/49125161 <== mon.2 v2:172.21.15.190:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7ff1e8068b40 con 0x7ff1e815b9c0 2024-04-24T23:15:14.843 INFO:tasks.ceph.mgr.x.smithi190.stderr:2024-04-24T23:15:14.841+0000 7f4852779040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-24T23:15:14.904 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.903+0000 7ff1eecbd640 1 -- 172.21.15.190:0/49125161 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "osd new", "uuid": "e9668b1c-a000-4332-86e8-4efd73b02409", "id": 7} v 0) -- 0x7ff1e81510d0 con 0x7ff1e815b9c0 2024-04-24T23:15:14.913 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.912+0000 7ff1e57fa640 1 -- 172.21.15.190:0/49125161 <== mon.2 v2:172.21.15.190:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "e9668b1c-a000-4332-86e8-4efd73b02409", "id": 7}]=0 v10) ==== 112+0+2 (secure 0 0 0) 0x7ff1e81510d0 con 0x7ff1e815b9c0 2024-04-24T23:15:14.914 INFO:teuthology.orchestra.run.smithi190.stdout:7 2024-04-24T23:15:14.914 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.913+0000 7ff1eecbd640 1 -- 172.21.15.190:0/49125161 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff1e815b9c0 msgr2=0x7ff1e81f4ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:14.915 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.913+0000 7ff1eecbd640 1 --2- 172.21.15.190:0/49125161 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff1e815b9c0 0x7ff1e81f4ab0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7ff1e0031000 tx=0x7ff1e0004a40 comp rx=0 tx=0).stop 2024-04-24T23:15:14.915 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.913+0000 7ff1eecbd640 1 -- 172.21.15.190:0/49125161 shutdown_connections 2024-04-24T23:15:14.915 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.913+0000 7ff1eecbd640 1 --2- 172.21.15.190:0/49125161 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff1e815b9c0 0x7ff1e81f4ab0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:14.915 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.913+0000 7ff1eecbd640 1 --2- 172.21.15.190:0/49125161 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff1e8151bd0 0x7ff1e81f4570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:14.915 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.913+0000 7ff1eecbd640 1 --2- 172.21.15.190:0/49125161 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff1e8071800 0x7ff1e814fdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:14.915 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.913+0000 7ff1eecbd640 1 -- 172.21.15.190:0/49125161 >> 172.21.15.190:0/49125161 conn(0x7ff1e806dcb0 msgr2=0x7ff1e80c26a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:15:14.915 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.913+0000 7ff1eecbd640 1 -- 172.21.15.190:0/49125161 shutdown_connections 2024-04-24T23:15:14.915 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-24T23:15:14.913+0000 7ff1eecbd640 1 -- 172.21.15.190:0/49125161 wait complete. 2024-04-24T23:15:14.925 INFO:tasks.ceph.osd.0:Restarting daemon 2024-04-24T23:15:14.925 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-04-24T23:15:14.932 INFO:tasks.ceph.osd.0:Started 2024-04-24T23:15:14.932 INFO:tasks.ceph.osd.1:Restarting daemon 2024-04-24T23:15:14.933 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-04-24T23:15:14.939 INFO:tasks.ceph.osd.1:Started 2024-04-24T23:15:14.939 INFO:tasks.ceph.osd.2:Restarting daemon 2024-04-24T23:15:14.939 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-04-24T23:15:14.944 INFO:tasks.ceph.osd.2:Started 2024-04-24T23:15:14.944 INFO:tasks.ceph.osd.3:Restarting daemon 2024-04-24T23:15:14.944 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-04-24T23:15:14.945 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-04-24T23:15:14.944+0000 7f400607a040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-24T23:15:14.949 INFO:tasks.ceph.osd.3:Started 2024-04-24T23:15:14.949 INFO:tasks.ceph.osd.4:Restarting daemon 2024-04-24T23:15:14.949 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-04-24T23:15:14.951 INFO:tasks.ceph.osd.4:Started 2024-04-24T23:15:14.952 INFO:tasks.ceph.osd.5:Restarting daemon 2024-04-24T23:15:14.952 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-04-24T23:15:14.954 INFO:tasks.ceph.osd.5:Started 2024-04-24T23:15:14.954 INFO:tasks.ceph.osd.6:Restarting daemon 2024-04-24T23:15:14.954 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-04-24T23:15:14.956 INFO:tasks.ceph.osd.6:Started 2024-04-24T23:15:14.956 INFO:tasks.ceph.osd.7:Restarting daemon 2024-04-24T23:15:14.956 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-04-24T23:15:14.958 INFO:tasks.ceph.osd.7:Started 2024-04-24T23:15:14.959 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T23:15:15.122 INFO:tasks.ceph.mgr.x.smithi190.stderr:2024-04-24T23:15:15.120+0000 7f4852779040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-24T23:15:15.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:15.172+0000 7f6e2a800640 1 Processor -- start 2024-04-24T23:15:15.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:15.172+0000 7f6e2a800640 1 -- start start 2024-04-24T23:15:15.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:15.173+0000 7f6e2a800640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6e240c7dc0 0x7f6e241623a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:15.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:15.173+0000 7f6e2a800640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f6e240c81c0 0x7f6e241628e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:15.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:15.173+0000 7f6e2a800640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6e240c7230 con 0x7f6e24159110 2024-04-24T23:15:15.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:15.173+0000 7f6e2a800640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f6e241594f0 con 0x7f6e240c81c0 2024-04-24T23:15:15.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:15.173+0000 7f6e2a800640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f6e240c74d0 con 0x7f6e240c7dc0 2024-04-24T23:15:15.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:15.173+0000 7f6e22a00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6e240c7dc0 0x7f6e241623a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:15.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:15.174+0000 7f6e23e00640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f6e240c81c0 0x7f6e241628e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:15.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:15.174+0000 7f6e23400640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f6e24159110 0x7f6e24072420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.6:51894/0 (socket says 172.21.15.6:51894) 2024-04-24T23:15:15.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:15.174+0000 7f6e23400640 1 -- 172.21.15.6:0/4175747311 learned_addr learned my addr 172.21.15.6:0/4175747311 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:15:15.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:15.174+0000 7f6e23e00640 1 -- 172.21.15.6:0/4175747311 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6e240c7dc0 msgr2=0x7f6e241623a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:15:15.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:15.174+0000 7f6e23e00640 1 --2- 172.21.15.6:0/4175747311 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6e240c7dc0 0x7f6e241623a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:15.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:15.174+0000 7f6e23e00640 1 -- 172.21.15.6:0/4175747311 >> v1:172.21.15.190:6789/0 conn(0x7f6e24159110 legacy=0x7f6e24072420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:15.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:15.174+0000 7f6e23e00640 1 -- 172.21.15.6:0/4175747311 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e2406e440 con 0x7f6e240c81c0 2024-04-24T23:15:15.176 INFO:tasks.ceph.mgr.x.smithi190.stderr:2024-04-24T23:15:15.174+0000 7f4852779040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-24T23:15:15.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:15.175+0000 7f6e23e00640 1 --2- 172.21.15.6:0/4175747311 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6e240c81c0 0x7f6e241628e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f6e1c009d90 tx=0x7f6e1c02a6f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3fb9846d5a6514d1 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:15:15.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:15.175+0000 7f6e22000640 1 -- 172.21.15.6:0/4175747311 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e1c007740 con 0x7f6e240c81c0 2024-04-24T23:15:15.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:15.175+0000 7f6e22000640 1 -- 172.21.15.6:0/4175747311 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6e1c0078e0 con 0x7f6e240c81c0 2024-04-24T23:15:15.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:15.175+0000 7f6e22000640 1 -- 172.21.15.6:0/4175747311 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e1c007cb0 con 0x7f6e240c81c0 2024-04-24T23:15:15.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:15.176+0000 7f6e2a800640 1 -- 172.21.15.6:0/4175747311 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6e240c81c0 msgr2=0x7f6e241628e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:15.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:15.176+0000 7f6e2a800640 1 --2- 172.21.15.6:0/4175747311 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6e240c81c0 0x7f6e241628e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f6e1c009d90 tx=0x7f6e1c02a6f0 comp rx=0 tx=0).stop 2024-04-24T23:15:15.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:15.176+0000 7f6e2a800640 1 -- 172.21.15.6:0/4175747311 shutdown_connections 2024-04-24T23:15:15.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:15.176+0000 7f6e2a800640 1 --2- 172.21.15.6:0/4175747311 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6e240c81c0 0x7f6e241628e0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:15.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:15.176+0000 7f6e2a800640 1 --2- 172.21.15.6:0/4175747311 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6e240c7dc0 0x7f6e241623a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:15.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:15.176+0000 7f6e2a800640 1 -- 172.21.15.6:0/4175747311 >> 172.21.15.6:0/4175747311 conn(0x7f6e2406b0e0 msgr2=0x7f6e2406b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:15:15.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:15.176+0000 7f6e2a800640 1 -- 172.21.15.6:0/4175747311 shutdown_connections 2024-04-24T23:15:15.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:15.176+0000 7f6e2a800640 1 -- 172.21.15.6:0/4175747311 wait complete. 2024-04-24T23:15:15.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:15.178+0000 7f6e2a800640 1 Processor -- start 2024-04-24T23:15:15.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:15.178+0000 7f6e2a800640 1 -- start start 2024-04-24T23:15:15.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:15.179+0000 7f6e2a800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6e240c7dc0 0x7f6e24157620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:15.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:15.179+0000 7f6e2a800640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6e240c81c0 0x7f6e24159ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:15.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:15.179+0000 7f6e2a800640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6e24159110 0x7f6e241f6b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:15.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:15.179+0000 7f6e2a800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f6e2406ece0 con 0x7f6e240c7dc0 2024-04-24T23:15:15.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:15.179+0000 7f6e2a800640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f6e2406ea00 con 0x7f6e240c81c0 2024-04-24T23:15:15.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:15.179+0000 7f6e2a800640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f6e2406f200 con 0x7f6e24159110 2024-04-24T23:15:15.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:15.179+0000 7f6e23400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6e240c7dc0 0x7f6e24157620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:15.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:15.179+0000 7f6e23400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6e240c7dc0 0x7f6e24157620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36120/0 (socket says 172.21.15.6:36120) 2024-04-24T23:15:15.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:15.179+0000 7f6e23400640 1 -- 172.21.15.6:0/2145591258 learned_addr learned my addr 172.21.15.6:0/2145591258 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:15:15.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:15.179+0000 7f6e22a00640 1 --2- 172.21.15.6:0/2145591258 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6e240c81c0 0x7f6e24159ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:15.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:15.179+0000 7f6e23400640 1 -- 172.21.15.6:0/2145591258 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6e24159110 msgr2=0x7f6e241f6b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:15.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:15.179+0000 7f6e23400640 1 --2- 172.21.15.6:0/2145591258 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6e24159110 0x7f6e241f6b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:15.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:15.179+0000 7f6e23400640 1 -- 172.21.15.6:0/2145591258 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6e240c81c0 msgr2=0x7f6e24159ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:15.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:15.179+0000 7f6e23400640 1 --2- 172.21.15.6:0/2145591258 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6e240c81c0 0x7f6e24159ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:15.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:15.179+0000 7f6e23400640 1 -- 172.21.15.6:0/2145591258 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e24157eb0 con 0x7f6e240c7dc0 2024-04-24T23:15:15.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:15.179+0000 7f6e22a00640 1 --2- 172.21.15.6:0/2145591258 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6e240c81c0 0x7f6e24159ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:15:15.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:15.180+0000 7f6e23400640 1 --2- 172.21.15.6:0/2145591258 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6e240c7dc0 0x7f6e24157620 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f6e1400c3c0 tx=0x7f6e1400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:15:15.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:15.180+0000 7f6e1b400640 1 -- 172.21.15.6:0/2145591258 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e14017070 con 0x7f6e240c7dc0 2024-04-24T23:15:15.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:15.180+0000 7f6e1b400640 1 -- 172.21.15.6:0/2145591258 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6e14005ce0 con 0x7f6e240c7dc0 2024-04-24T23:15:15.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:15.180+0000 7f6e1b400640 1 -- 172.21.15.6:0/2145591258 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e14007310 con 0x7f6e240c7dc0 2024-04-24T23:15:15.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:15.180+0000 7f6e2a800640 1 -- 172.21.15.6:0/2145591258 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e2415b080 con 0x7f6e240c7dc0 2024-04-24T23:15:15.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:15.180+0000 7f6e2a800640 1 -- 172.21.15.6:0/2145591258 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e2415b640 con 0x7f6e240c7dc0 2024-04-24T23:15:15.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:15.181+0000 7f6e1b400640 1 -- 172.21.15.6:0/2145591258 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f6e1401a070 con 0x7f6e240c7dc0 2024-04-24T23:15:15.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:15.181+0000 7f6e1b400640 1 -- 172.21.15.6:0/2145591258 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7f6e14012ad0 con 0x7f6e240c7dc0 2024-04-24T23:15:15.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:15.181+0000 7f6e2a800640 1 -- 172.21.15.6:0/2145591258 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6e240767b0 con 0x7f6e240c7dc0 2024-04-24T23:15:15.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:15.184+0000 7f6e1b400640 1 -- 172.21.15.6:0/2145591258 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f6e14021020 con 0x7f6e240c7dc0 2024-04-24T23:15:15.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:15.251+0000 7f6e2a800640 1 -- 172.21.15.6:0/2145591258 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6e240c7970 con 0x7f6e240c7dc0 2024-04-24T23:15:15.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:15.252+0000 7f6e1b400640 1 -- 172.21.15.6:0/2145591258 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7f6e14007560 con 0x7f6e240c7dc0 2024-04-24T23:15:15.253 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:15:15.254 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":10,"fsid":"18bef68d-5e6d-4ec5-862b-b6f206e2588d","created":"2024-04-24T23:15:12.535072+0000","modified":"2024-04-24T23:15:14.905328+0000","last_up_change":"0.000000","last_in_change":"2024-04-24T23:15:14.905328+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3b7a212d-d498-462e-90bf-ed493c803a09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"7c97f5a2-1735-4775-9f15-eb50099b19d0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d86bbc18-438c-4ec2-8018-ba7dd78f790b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3754ae16-4248-40dd-95f9-0f59c0eb5426","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8b2913a0-d006-479c-84ed-5b21e0e6c392","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"499b79d0-0585-4dae-ab69-196a551793da","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6ba6df3b-4a6b-4a89-a890-0f79f387d4d7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e9668b1c-a000-4332-86e8-4efd73b02409","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T23:15:15.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:15.253+0000 7f6e2a800640 1 -- 172.21.15.6:0/2145591258 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6e240c7dc0 msgr2=0x7f6e24157620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:15.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:15.253+0000 7f6e2a800640 1 --2- 172.21.15.6:0/2145591258 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6e240c7dc0 0x7f6e24157620 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f6e1400c3c0 tx=0x7f6e1400c880 comp rx=0 tx=0).stop 2024-04-24T23:15:15.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:15.253+0000 7f6e2a800640 1 -- 172.21.15.6:0/2145591258 shutdown_connections 2024-04-24T23:15:15.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:15.253+0000 7f6e2a800640 1 --2- 172.21.15.6:0/2145591258 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6e24159110 0x7f6e241f6b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:15.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:15.253+0000 7f6e2a800640 1 --2- 172.21.15.6:0/2145591258 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6e240c81c0 0x7f6e24159ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:15.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:15.253+0000 7f6e2a800640 1 --2- 172.21.15.6:0/2145591258 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6e240c7dc0 0x7f6e24157620 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:15.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:15.253+0000 7f6e2a800640 1 -- 172.21.15.6:0/2145591258 >> 172.21.15.6:0/2145591258 conn(0x7f6e2406b0e0 msgr2=0x7f6e240c6070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:15:15.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:15.254+0000 7f6e2a800640 1 -- 172.21.15.6:0/2145591258 shutdown_connections 2024-04-24T23:15:15.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:15.254+0000 7f6e2a800640 1 -- 172.21.15.6:0/2145591258 wait complete. 2024-04-24T23:15:15.271 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-04-24T23:15:15.271 INFO:tasks.ceph:Waiting for OSDs to come up 2024-04-24T23:15:15.291 INFO:tasks.ceph.mgr.x.smithi190.stderr:2024-04-24T23:15:15.290+0000 7f4852779040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-24T23:15:15.485 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-04-24T23:15:15.483+0000 7f400607a040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-24T23:15:15.548 INFO:tasks.ceph.osd.5.smithi190.stderr:2024-04-24T23:15:15.546+0000 7f814b419640 -1 Falling back to public interface 2024-04-24T23:15:15.549 INFO:tasks.ceph.osd.4.smithi190.stderr:2024-04-24T23:15:15.547+0000 7fc8e5c2e640 -1 Falling back to public interface 2024-04-24T23:15:15.566 INFO:tasks.ceph.osd.6.smithi190.stderr:2024-04-24T23:15:15.564+0000 7fefce9ad640 -1 Falling back to public interface 2024-04-24T23:15:15.572 INFO:tasks.ceph.mgr.x.smithi190.stderr:2024-04-24T23:15:15.570+0000 7f4852779040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-24T23:15:15.574 INFO:tasks.ceph.osd.7.smithi190.stderr:2024-04-24T23:15:15.572+0000 7f25dbd2c640 -1 Falling back to public interface 2024-04-24T23:15:15.620 INFO:tasks.ceph.mgr.x.smithi190.stderr:2024-04-24T23:15:15.619+0000 7f4852779040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-24T23:15:15.640 INFO:tasks.ceph.osd.2.smithi006.stderr:2024-04-24T23:15:15.639+0000 7f0517610640 -1 Falling back to public interface 2024-04-24T23:15:15.649 INFO:tasks.ceph.osd.0.smithi006.stderr:2024-04-24T23:15:15.648+0000 7f0e382e1640 -1 Falling back to public interface 2024-04-24T23:15:15.658 INFO:tasks.ceph.osd.1.smithi006.stderr:2024-04-24T23:15:15.657+0000 7fea9a043640 -1 Falling back to public interface 2024-04-24T23:15:15.673 INFO:tasks.ceph.osd.3.smithi006.stderr:2024-04-24T23:15:15.672+0000 7f8a84188640 -1 Falling back to public interface 2024-04-24T23:15:15.774 INFO:tasks.ceph.mgr.x.smithi190.stderr:2024-04-24T23:15:15.772+0000 7f4852779040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-24T23:15:15.796 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-04-24T23:15:15.795+0000 7f400607a040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-24T23:15:15.860 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-04-24T23:15:15.858+0000 7f400607a040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-24T23:15:15.909 INFO:tasks.ceph.mgr.x.smithi190.stderr:2024-04-24T23:15:15.908+0000 7f4852779040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-24T23:15:15.958 INFO:tasks.ceph.mgr.x.smithi190.stderr:2024-04-24T23:15:15.956+0000 7f4852779040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-24T23:15:15.998 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-04-24T23:15:15.997+0000 7f400607a040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-24T23:15:16.026 INFO:tasks.ceph.mgr.x.smithi190.stderr:2024-04-24T23:15:16.024+0000 7f4852779040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-24T23:15:16.070 INFO:tasks.ceph.mgr.x.smithi190.stderr:2024-04-24T23:15:16.069+0000 7f4852779040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-24T23:15:16.077 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-24T23:15:16.246 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:16.245+0000 7f6d11e00640 1 Processor -- start 2024-04-24T23:15:16.246 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:16.245+0000 7f6d11e00640 1 -- start start 2024-04-24T23:15:16.247 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:16.246+0000 7f6d11e00640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f6d0c0cb870 0x7f6d0c0cc990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:16.247 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:16.246+0000 7f6d11e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f6d0c0c7e20 0x7f6d0c0c82d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:16.247 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:16.246+0000 7f6d11e00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6d0c0c8810 con 0x7f6d0c0cb490 2024-04-24T23:15:16.248 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:16.246+0000 7f6d11e00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f6d0c0cced0 con 0x7f6d0c0c7e20 2024-04-24T23:15:16.248 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:16.246+0000 7f6d11e00640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f6d0c0cd0f0 con 0x7f6d0c0cb870 2024-04-24T23:15:16.248 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:16.246+0000 7f6d11400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f6d0c0c7e20 0x7f6d0c0c82d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:16.248 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:16.246+0000 7f6d0be00640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f6d0c0cb870 0x7f6d0c0cc990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:16.248 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:16.246+0000 7f6d11400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f6d0c0c7e20 0x7f6d0c0c82d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36130/0 (socket says 172.21.15.6:36130) 2024-04-24T23:15:16.248 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:16.246+0000 7f6d11400640 1 -- 172.21.15.6:0/3077976139 learned_addr learned my addr 172.21.15.6:0/3077976139 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:15:16.249 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:16.247+0000 7f6d0be00640 1 -- 172.21.15.6:0/3077976139 >> v2:172.21.15.6:3300/0 conn(0x7f6d0c0c7e20 msgr2=0x7f6d0c0c82d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:15:16.249 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:16.247+0000 7f6d0be00640 1 --2- 172.21.15.6:0/3077976139 >> v2:172.21.15.6:3300/0 conn(0x7f6d0c0c7e20 0x7f6d0c0c82d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:16.249 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:16.247+0000 7f6d0be00640 1 -- 172.21.15.6:0/3077976139 >> v1:172.21.15.6:6789/0 conn(0x7f6d0c0cb490 legacy=0x7f6d0c0d05a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:16.249 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:16.247+0000 7f6d0be00640 1 -- 172.21.15.6:0/3077976139 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d0c0cd3d0 con 0x7f6d0c0cb870 2024-04-24T23:15:16.249 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:16.248+0000 7f6d0be00640 1 --2- 172.21.15.6:0/3077976139 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6d0c0cb870 0x7f6d0c0cc990 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f6cf80097a0 tx=0x7f6cf802a270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e032024de5a60b01 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:15:16.249 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:16.248+0000 7f6d0b400640 1 -- 172.21.15.6:0/3077976139 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cf8037070 con 0x7f6d0c0cb870 2024-04-24T23:15:16.249 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:16.248+0000 7f6d0b400640 1 -- 172.21.15.6:0/3077976139 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6cf8032440 con 0x7f6d0c0cb870 2024-04-24T23:15:16.250 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:16.248+0000 7f6d0b400640 1 -- 172.21.15.6:0/3077976139 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cf8032810 con 0x7f6d0c0cb870 2024-04-24T23:15:16.250 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:16.249+0000 7f6d11e00640 1 -- 172.21.15.6:0/3077976139 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6d0c0cb870 msgr2=0x7f6d0c0cc990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:16.250 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:16.249+0000 7f6d11e00640 1 --2- 172.21.15.6:0/3077976139 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6d0c0cb870 0x7f6d0c0cc990 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f6cf80097a0 tx=0x7f6cf802a270 comp rx=0 tx=0).stop 2024-04-24T23:15:16.250 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:16.249+0000 7f6d11e00640 1 -- 172.21.15.6:0/3077976139 shutdown_connections 2024-04-24T23:15:16.251 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:16.249+0000 7f6d11e00640 1 --2- 172.21.15.6:0/3077976139 >> v2:172.21.15.6:3300/0 conn(0x7f6d0c0c7e20 0x7f6d0c0c82d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:16.251 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:16.249+0000 7f6d11e00640 1 --2- 172.21.15.6:0/3077976139 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6d0c0cb870 0x7f6d0c0cc990 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:16.251 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:16.249+0000 7f6d11e00640 1 -- 172.21.15.6:0/3077976139 >> 172.21.15.6:0/3077976139 conn(0x7f6d0c06da80 msgr2=0x7f6d0c072890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:15:16.251 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:16.249+0000 7f6d11e00640 1 -- 172.21.15.6:0/3077976139 shutdown_connections 2024-04-24T23:15:16.251 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:16.249+0000 7f6d11e00640 1 -- 172.21.15.6:0/3077976139 wait complete. 2024-04-24T23:15:16.252 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:16.251+0000 7f6d11e00640 1 Processor -- start 2024-04-24T23:15:16.252 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:16.251+0000 7f6d11e00640 1 -- start start 2024-04-24T23:15:16.254 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:16.252+0000 7f6d11e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6d0c0c7e20 0x7f6d0c1f1f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:16.254 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:16.252+0000 7f6d11e00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6d0c0cb490 0x7f6d0c0c51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:16.254 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:16.252+0000 7f6d11e00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6d0c0cb870 0x7f6d0c0c6ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:16.254 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:16.252+0000 7f6d11e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f6d0c0c7920 con 0x7f6d0c0c7e20 2024-04-24T23:15:16.254 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:16.252+0000 7f6d11e00640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f6d0c0cd990 con 0x7f6d0c0cb870 2024-04-24T23:15:16.254 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:16.252+0000 7f6d11e00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f6d0c0d0cb0 con 0x7f6d0c0cb490 2024-04-24T23:15:16.254 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:16.252+0000 7f6d10a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6d0c0c7e20 0x7f6d0c1f1f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:16.254 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:16.252+0000 7f6d0be00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6d0c0cb490 0x7f6d0c0c51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:16.254 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:16.252+0000 7f6d11400640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6d0c0cb870 0x7f6d0c0c6ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:16.254 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:16.252+0000 7f6d0be00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6d0c0cb490 0x7f6d0c0c51f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.6:41534/0 (socket says 172.21.15.6:41534) 2024-04-24T23:15:16.254 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:16.252+0000 7f6d0be00640 1 -- 172.21.15.6:0/3664699610 learned_addr learned my addr 172.21.15.6:0/3664699610 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:15:16.254 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:16.252+0000 7f6d10a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6d0c0c7e20 0x7f6d0c1f1f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36136/0 (socket says 172.21.15.6:36136) 2024-04-24T23:15:16.255 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:16.253+0000 7f6d11400640 1 -- 172.21.15.6:0/3664699610 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6d0c0cb490 msgr2=0x7f6d0c0c51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:16.255 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:16.253+0000 7f6d11400640 1 --2- 172.21.15.6:0/3664699610 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6d0c0cb490 0x7f6d0c0c51f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:16.255 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:16.253+0000 7f6d11400640 1 -- 172.21.15.6:0/3664699610 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6d0c0c7e20 msgr2=0x7f6d0c1f1f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:16.255 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:16.253+0000 7f6d11400640 1 --2- 172.21.15.6:0/3664699610 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6d0c0c7e20 0x7f6d0c1f1f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:16.255 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:16.253+0000 7f6d11400640 1 -- 172.21.15.6:0/3664699610 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d0c0c3e90 con 0x7f6d0c0cb870 2024-04-24T23:15:16.255 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:16.253+0000 7f6d0be00640 1 --2- 172.21.15.6:0/3664699610 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6d0c0cb490 0x7f6d0c0c51f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:15:16.255 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:16.253+0000 7f6d11400640 1 --2- 172.21.15.6:0/3664699610 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6d0c0cb870 0x7f6d0c0c6ca0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f6d04009d90 tx=0x7f6d0400c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:15:16.256 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:16.253+0000 7f6d08c00640 1 -- 172.21.15.6:0/3664699610 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d0400cd20 con 0x7f6d0c0cb870 2024-04-24T23:15:16.256 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:16.253+0000 7f6d08c00640 1 -- 172.21.15.6:0/3664699610 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6d04003990 con 0x7f6d0c0cb870 2024-04-24T23:15:16.256 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:16.253+0000 7f6d08c00640 1 -- 172.21.15.6:0/3664699610 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d04003d60 con 0x7f6d0c0cb870 2024-04-24T23:15:16.256 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:16.253+0000 7f6d11e00640 1 -- 172.21.15.6:0/3664699610 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d0c0c4150 con 0x7f6d0c0cb870 2024-04-24T23:15:16.256 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:16.253+0000 7f6d11e00640 1 -- 172.21.15.6:0/3664699610 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d0c0c4710 con 0x7f6d0c0cb870 2024-04-24T23:15:16.256 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:16.254+0000 7f6d08c00640 1 -- 172.21.15.6:0/3664699610 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f6d04005d10 con 0x7f6d0c0cb870 2024-04-24T23:15:16.256 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:16.254+0000 7f6d08c00640 1 -- 172.21.15.6:0/3664699610 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7f6d04014530 con 0x7f6d0c0cb870 2024-04-24T23:15:16.256 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:16.254+0000 7f6d11e00640 1 -- 172.21.15.6:0/3664699610 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6d0c068fa0 con 0x7f6d0c0cb870 2024-04-24T23:15:16.258 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:16.257+0000 7f6d08c00640 1 -- 172.21.15.6:0/3664699610 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f6d04022020 con 0x7f6d0c0cb870 2024-04-24T23:15:16.324 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:16.323+0000 7f6d11e00640 1 -- 172.21.15.6:0/3664699610 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6d0c075ba0 con 0x7f6d0c0cb870 2024-04-24T23:15:16.325 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:16.324+0000 7f6d08c00640 1 -- 172.21.15.6:0/3664699610 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7f6d04010040 con 0x7f6d0c0cb870 2024-04-24T23:15:16.325 INFO:teuthology.misc.health.smithi006.stdout: 2024-04-24T23:15:16.326 INFO:teuthology.misc.health.smithi006.stdout:{"epoch":10,"fsid":"18bef68d-5e6d-4ec5-862b-b6f206e2588d","created":"2024-04-24T23:15:12.535072+0000","modified":"2024-04-24T23:15:14.905328+0000","last_up_change":"0.000000","last_in_change":"2024-04-24T23:15:14.905328+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3b7a212d-d498-462e-90bf-ed493c803a09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"7c97f5a2-1735-4775-9f15-eb50099b19d0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d86bbc18-438c-4ec2-8018-ba7dd78f790b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3754ae16-4248-40dd-95f9-0f59c0eb5426","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8b2913a0-d006-479c-84ed-5b21e0e6c392","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"499b79d0-0585-4dae-ab69-196a551793da","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6ba6df3b-4a6b-4a89-a890-0f79f387d4d7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e9668b1c-a000-4332-86e8-4efd73b02409","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T23:15:16.327 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:16.325+0000 7f6d11e00640 1 -- 172.21.15.6:0/3664699610 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6d0c0cb870 msgr2=0x7f6d0c0c6ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:16.327 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:16.325+0000 7f6d11e00640 1 --2- 172.21.15.6:0/3664699610 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6d0c0cb870 0x7f6d0c0c6ca0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f6d04009d90 tx=0x7f6d0400c490 comp rx=0 tx=0).stop 2024-04-24T23:15:16.327 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:16.325+0000 7f6d11e00640 1 -- 172.21.15.6:0/3664699610 shutdown_connections 2024-04-24T23:15:16.327 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:16.325+0000 7f6d11e00640 1 --2- 172.21.15.6:0/3664699610 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6d0c0cb490 0x7f6d0c0c51f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:16.327 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:16.325+0000 7f6d11e00640 1 --2- 172.21.15.6:0/3664699610 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6d0c0cb870 0x7f6d0c0c6ca0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:16.327 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:16.325+0000 7f6d11e00640 1 --2- 172.21.15.6:0/3664699610 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6d0c0c7e20 0x7f6d0c1f1f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:16.327 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:16.325+0000 7f6d11e00640 1 -- 172.21.15.6:0/3664699610 >> 172.21.15.6:0/3664699610 conn(0x7f6d0c06da80 msgr2=0x7f6d0c0b4bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:15:16.327 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:16.325+0000 7f6d11e00640 1 -- 172.21.15.6:0/3664699610 shutdown_connections 2024-04-24T23:15:16.327 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:16.326+0000 7f6d11e00640 1 -- 172.21.15.6:0/3664699610 wait complete. 2024-04-24T23:15:16.340 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-04-24T23:15:16.348 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-04-24T23:15:16.347+0000 7f400607a040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-24T23:15:16.406 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-04-24T23:15:16.405+0000 7f400607a040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-24T23:15:16.546 INFO:tasks.ceph.mgr.x.smithi190.stderr:2024-04-24T23:15:16.545+0000 7f4852779040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-24T23:15:16.598 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-04-24T23:15:16.596+0000 7f400607a040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-24T23:15:16.663 INFO:tasks.ceph.mgr.x.smithi190.stderr:2024-04-24T23:15:16.662+0000 7f4852779040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-24T23:15:16.776 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-04-24T23:15:16.775+0000 7f400607a040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-24T23:15:16.838 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-04-24T23:15:16.837+0000 7f400607a040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-24T23:15:16.923 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-04-24T23:15:16.921+0000 7f400607a040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-24T23:15:16.978 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-04-24T23:15:16.976+0000 7f400607a040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-24T23:15:17.540 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-04-24T23:15:17.539+0000 7f400607a040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-24T23:15:17.679 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-04-24T23:15:17.677+0000 7f400607a040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-24T23:15:17.688 INFO:tasks.ceph.mgr.x.smithi190.stderr:2024-04-24T23:15:17.686+0000 7f47f3bb3640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-04-24T23:15:17.688 INFO:tasks.ceph.mgr.x.smithi190.stderr:2024-04-24T23:15:17.686+0000 7f47f3bb3640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-04-24T23:15:19.032 INFO:tasks.ceph.osd.4.smithi190.stderr:2024-04-24T23:15:19.030+0000 7fc8e5c2e640 -1 osd.4 0 log_to_monitors true 2024-04-24T23:15:19.080 INFO:tasks.ceph.osd.6.smithi190.stderr:2024-04-24T23:15:19.078+0000 7fefce9ad640 -1 osd.6 0 log_to_monitors true 2024-04-24T23:15:19.089 INFO:tasks.ceph.osd.5.smithi190.stderr:2024-04-24T23:15:19.088+0000 7f814b419640 -1 osd.5 0 log_to_monitors true 2024-04-24T23:15:19.093 INFO:tasks.ceph.osd.7.smithi190.stderr:2024-04-24T23:15:19.092+0000 7f25dbd2c640 -1 osd.7 0 log_to_monitors true 2024-04-24T23:15:19.312 INFO:tasks.ceph.osd.0.smithi006.stderr:2024-04-24T23:15:19.311+0000 7f0e382e1640 -1 osd.0 0 log_to_monitors true 2024-04-24T23:15:19.319 INFO:tasks.ceph.osd.1.smithi006.stderr:2024-04-24T23:15:19.318+0000 7fea9a043640 -1 osd.1 0 log_to_monitors true 2024-04-24T23:15:19.494 INFO:tasks.ceph.osd.2.smithi006.stderr:2024-04-24T23:15:19.493+0000 7f0517610640 -1 osd.2 0 log_to_monitors true 2024-04-24T23:15:19.559 INFO:tasks.ceph.osd.3.smithi006.stderr:2024-04-24T23:15:19.558+0000 7f8a84188640 -1 osd.3 0 log_to_monitors true 2024-04-24T23:15:22.758 INFO:tasks.ceph.osd.7.smithi190.stderr:2024-04-24T23:15:22.756+0000 7f25d84d6640 -1 osd.7 0 waiting for initial osdmap 2024-04-24T23:15:22.775 INFO:tasks.ceph.osd.7.smithi190.stderr:2024-04-24T23:15:22.773+0000 7f25cfae2640 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-24T23:15:22.804 INFO:tasks.ceph.osd.5.smithi190.stderr:2024-04-24T23:15:22.802+0000 7f8146d2c640 -1 osd.5 0 waiting for initial osdmap 2024-04-24T23:15:22.811 INFO:tasks.ceph.osd.5.smithi190.stderr:2024-04-24T23:15:22.810+0000 7f813eb4c640 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-24T23:15:22.824 INFO:tasks.ceph.osd.4.smithi190.stderr:2024-04-24T23:15:22.823+0000 7fc8e198c640 -1 osd.4 0 waiting for initial osdmap 2024-04-24T23:15:22.834 INFO:tasks.ceph.osd.6.smithi190.stderr:2024-04-24T23:15:22.832+0000 7fefca943640 -1 osd.6 0 waiting for initial osdmap 2024-04-24T23:15:22.839 INFO:tasks.ceph.osd.4.smithi190.stderr:2024-04-24T23:15:22.838+0000 7fc8d8f98640 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-24T23:15:22.848 INFO:tasks.ceph.osd.6.smithi190.stderr:2024-04-24T23:15:22.847+0000 7fefc2763640 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-24T23:15:23.075 INFO:tasks.ceph.osd.2.smithi006.stderr:2024-04-24T23:15:23.073+0000 7f0511e00640 -1 osd.2 0 waiting for initial osdmap 2024-04-24T23:15:23.083 INFO:tasks.ceph.osd.2.smithi006.stderr:2024-04-24T23:15:23.081+0000 7f0507e00640 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-24T23:15:23.100 INFO:tasks.ceph.osd.3.smithi006.stderr:2024-04-24T23:15:23.098+0000 7f8a7f000640 -1 osd.3 0 waiting for initial osdmap 2024-04-24T23:15:23.105 INFO:tasks.ceph.osd.3.smithi006.stderr:2024-04-24T23:15:23.103+0000 7f8a75000640 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-24T23:15:23.111 INFO:tasks.ceph.osd.1.smithi006.stderr:2024-04-24T23:15:23.109+0000 7fea94e00640 -1 osd.1 0 waiting for initial osdmap 2024-04-24T23:15:23.120 INFO:tasks.ceph.osd.1.smithi006.stderr:2024-04-24T23:15:23.118+0000 7fea8ae00640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-24T23:15:23.133 INFO:tasks.ceph.osd.0.smithi006.stderr:2024-04-24T23:15:23.131+0000 7f0e33200640 -1 osd.0 0 waiting for initial osdmap 2024-04-24T23:15:23.141 INFO:tasks.ceph.osd.0.smithi006.stderr:2024-04-24T23:15:23.139+0000 7f0e29200640 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-24T23:15:23.147 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-24T23:15:23.299 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:23.291+0000 7f3381200640 1 Processor -- start 2024-04-24T23:15:23.299 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:23.291+0000 7f3381200640 1 -- start start 2024-04-24T23:15:23.299 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:23.293+0000 7f3381200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f337c0c5b40 0x7f337c167a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:23.299 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:23.293+0000 7f3381200640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f337c0c6ef0 0x7f337c0c6230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:23.299 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:23.293+0000 7f3381200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f337c0c7380 con 0x7f337c0c8870 2024-04-24T23:15:23.299 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:23.293+0000 7f3381200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f337c16b740 con 0x7f337c0c5b40 2024-04-24T23:15:23.299 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:23.293+0000 7f3381200640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f337c0c7660 con 0x7f337c0c6ef0 2024-04-24T23:15:23.299 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:23.293+0000 7f3379e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f337c0c5b40 0x7f337c167a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:23.299 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:23.293+0000 7f3379e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f337c0c5b40 0x7f337c167a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36172/0 (socket says 172.21.15.6:36172) 2024-04-24T23:15:23.299 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:23.293+0000 7f3379e00640 1 -- 172.21.15.6:0/1235342655 learned_addr learned my addr 172.21.15.6:0/1235342655 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:15:23.299 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:23.293+0000 7f3371400640 1 --2- 172.21.15.6:0/1235342655 >> v2:172.21.15.190:3300/0 conn(0x7f337c0c6ef0 0x7f337c0c6230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:23.299 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:23.294+0000 7f3379400640 1 -- 172.21.15.6:0/1235342655 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 858750576 0 0) 0x7f337c0c7380 con 0x7f337c0c8870 2024-04-24T23:15:23.300 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:23.296+0000 7f3379400640 1 -- 172.21.15.6:0/1235342655 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3360003420 con 0x7f337c0c8870 2024-04-24T23:15:23.300 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:23.296+0000 7f3379e00640 1 -- 172.21.15.6:0/1235342655 >> v2:172.21.15.190:3300/0 conn(0x7f337c0c6ef0 msgr2=0x7f337c0c6230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:15:23.300 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:23.296+0000 7f3379e00640 1 --2- 172.21.15.6:0/1235342655 >> v2:172.21.15.190:3300/0 conn(0x7f337c0c6ef0 0x7f337c0c6230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:23.300 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:23.296+0000 7f3379e00640 1 -- 172.21.15.6:0/1235342655 >> v1:172.21.15.6:6789/0 conn(0x7f337c0c8870 legacy=0x7f337c06b160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:23.300 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:23.296+0000 7f3379e00640 1 -- 172.21.15.6:0/1235342655 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f337c16b0f0 con 0x7f337c0c5b40 2024-04-24T23:15:23.300 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:23.296+0000 7f3379400640 1 -- 172.21.15.6:0/1235342655 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2905141491 0 0) 0x7f3360003420 con 0x7f337c0c8870 2024-04-24T23:15:23.301 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:23.297+0000 7f3379e00640 1 --2- 172.21.15.6:0/1235342655 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f337c0c5b40 0x7f337c167a00 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f337c075010 tx=0x7f336c032b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=332ba843a1b9e5cc server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:15:23.301 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:23.297+0000 7f3379400640 1 -- 172.21.15.6:0/1235342655 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f336c03a070 con 0x7f337c0c5b40 2024-04-24T23:15:23.301 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:23.297+0000 7f3379400640 1 -- 172.21.15.6:0/1235342655 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f336c00ae50 con 0x7f337c0c5b40 2024-04-24T23:15:23.301 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:23.297+0000 7f3379400640 1 -- 172.21.15.6:0/1235342655 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33740019f0 con 0x7f337c0c5b40 2024-04-24T23:15:23.301 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:23.298+0000 7f3381200640 1 -- 172.21.15.6:0/1235342655 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f337c0c5b40 msgr2=0x7f337c167a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:23.301 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:23.298+0000 7f3381200640 1 --2- 172.21.15.6:0/1235342655 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f337c0c5b40 0x7f337c167a00 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f337c075010 tx=0x7f336c032b00 comp rx=0 tx=0).stop 2024-04-24T23:15:23.301 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:23.298+0000 7f3381200640 1 -- 172.21.15.6:0/1235342655 shutdown_connections 2024-04-24T23:15:23.302 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:23.298+0000 7f3381200640 1 --2- 172.21.15.6:0/1235342655 >> v2:172.21.15.190:3300/0 conn(0x7f337c0c6ef0 0x7f337c0c6230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:23.302 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:23.298+0000 7f3381200640 1 --2- 172.21.15.6:0/1235342655 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f337c0c5b40 0x7f337c167a00 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:23.302 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:23.298+0000 7f3381200640 1 -- 172.21.15.6:0/1235342655 >> 172.21.15.6:0/1235342655 conn(0x7f337c0cc2f0 msgr2=0x7f337c0cc700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:15:23.302 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:23.298+0000 7f3381200640 1 -- 172.21.15.6:0/1235342655 shutdown_connections 2024-04-24T23:15:23.302 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:23.298+0000 7f3381200640 1 -- 172.21.15.6:0/1235342655 wait complete. 2024-04-24T23:15:23.302 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:23.300+0000 7f3381200640 1 Processor -- start 2024-04-24T23:15:23.302 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:23.300+0000 7f3381200640 1 -- start start 2024-04-24T23:15:23.304 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:23.301+0000 7f3381200640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f337c0c5b40 0x7f337c06ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:23.304 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:23.301+0000 7f3381200640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f337c0c8870 0x7f337c0c0f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:23.304 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:23.301+0000 7f3381200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f337c0c1460 0x7f337c15df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:23.304 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:23.301+0000 7f3381200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f337c0c6770 con 0x7f337c0c1460 2024-04-24T23:15:23.305 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:23.301+0000 7f3381200640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f337c167f40 con 0x7f337c0c5b40 2024-04-24T23:15:23.305 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:23.301+0000 7f3381200640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f337c06b870 con 0x7f337c0c8870 2024-04-24T23:15:23.305 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:23.301+0000 7f337a800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f337c0c1460 0x7f337c15df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:23.305 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:23.301+0000 7f337a800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f337c0c1460 0x7f337c15df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36184/0 (socket says 172.21.15.6:36184) 2024-04-24T23:15:23.305 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:23.301+0000 7f337a800640 1 -- 172.21.15.6:0/2880488825 learned_addr learned my addr 172.21.15.6:0/2880488825 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:15:23.305 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:23.301+0000 7f3371400640 1 --2- 172.21.15.6:0/2880488825 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f337c0c8870 0x7f337c0c0f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:23.305 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:23.301+0000 7f3379e00640 1 --2- 172.21.15.6:0/2880488825 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f337c0c5b40 0x7f337c06ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:23.305 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:23.301+0000 7f337a800640 1 -- 172.21.15.6:0/2880488825 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f337c0c8870 msgr2=0x7f337c0c0f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:23.305 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:23.301+0000 7f337a800640 1 --2- 172.21.15.6:0/2880488825 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f337c0c8870 0x7f337c0c0f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:23.305 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:23.302+0000 7f337a800640 1 -- 172.21.15.6:0/2880488825 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f337c0c5b40 msgr2=0x7f337c06ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:23.305 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:23.302+0000 7f337a800640 1 --2- 172.21.15.6:0/2880488825 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f337c0c5b40 0x7f337c06ecf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:23.305 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:23.302+0000 7f337a800640 1 -- 172.21.15.6:0/2880488825 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f337c165c80 con 0x7f337c0c1460 2024-04-24T23:15:23.305 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:23.302+0000 7f3379e00640 1 --2- 172.21.15.6:0/2880488825 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f337c0c5b40 0x7f337c06ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:15:23.305 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:23.302+0000 7f337a800640 1 --2- 172.21.15.6:0/2880488825 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f337c0c1460 0x7f337c15df80 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f337400c7d0 tx=0x7f337400cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:15:23.306 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:23.302+0000 7f3372a00640 1 -- 172.21.15.6:0/2880488825 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3374016020 con 0x7f337c0c1460 2024-04-24T23:15:23.306 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:23.302+0000 7f3372a00640 1 -- 172.21.15.6:0/2880488825 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f337400aa70 con 0x7f337c0c1460 2024-04-24T23:15:23.306 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:23.302+0000 7f3372a00640 1 -- 172.21.15.6:0/2880488825 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3374012420 con 0x7f337c0c1460 2024-04-24T23:15:23.306 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:23.302+0000 7f3381200640 1 -- 172.21.15.6:0/2880488825 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f337c200d50 con 0x7f337c0c1460 2024-04-24T23:15:23.306 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:23.302+0000 7f3381200640 1 -- 172.21.15.6:0/2880488825 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f337c0c0440 con 0x7f337c0c1460 2024-04-24T23:15:23.306 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:23.303+0000 7f3381200640 1 -- 172.21.15.6:0/2880488825 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f337c04f500 con 0x7f337c0c1460 2024-04-24T23:15:23.306 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:23.304+0000 7f3372a00640 1 -- 172.21.15.6:0/2880488825 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 4) ==== 90442+0+0 (secure 0 0 0) 0x7f3374019020 con 0x7f337c0c1460 2024-04-24T23:15:23.307 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:23.305+0000 7f3372a00640 1 --2- 172.21.15.6:0/2880488825 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f335806bda0 0x7f335806e260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:23.307 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:23.305+0000 7f3372a00640 1 -- 172.21.15.6:0/2880488825 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(12..12 src has 1..12) ==== 2231+0+0 (secure 0 0 0) 0x7f337408a370 con 0x7f337c0c1460 2024-04-24T23:15:23.307 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:23.305+0000 7f3379e00640 1 --2- 172.21.15.6:0/2880488825 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f335806bda0 0x7f335806e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:23.308 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:23.306+0000 7f3379e00640 1 --2- 172.21.15.6:0/2880488825 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f335806bda0 0x7f335806e260 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f336c03c000 tx=0x7f336c00a000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:15:23.312 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:23.310+0000 7f3372a00640 1 -- 172.21.15.6:0/2880488825 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f3374020020 con 0x7f337c0c1460 2024-04-24T23:15:23.510 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:23.507+0000 7f3381200640 1 -- 172.21.15.6:0/2880488825 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f337c0c7aa0 con 0x7f337c0c1460 2024-04-24T23:15:23.511 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:23.509+0000 7f3372a00640 1 -- 172.21.15.6:0/2880488825 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) ==== 74+0+7058 (secure 0 0 0) 0x7f3374059af0 con 0x7f337c0c1460 2024-04-24T23:15:23.511 INFO:teuthology.misc.health.smithi006.stdout: 2024-04-24T23:15:23.511 INFO:teuthology.misc.health.smithi006.stdout:{"epoch":12,"fsid":"18bef68d-5e6d-4ec5-862b-b6f206e2588d","created":"2024-04-24T23:15:12.535072+0000","modified":"2024-04-24T23:15:20.703153+0000","last_up_change":"0.000000","last_in_change":"2024-04-24T23:15:14.905328+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3b7a212d-d498-462e-90bf-ed493c803a09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"7c97f5a2-1735-4775-9f15-eb50099b19d0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d86bbc18-438c-4ec2-8018-ba7dd78f790b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3754ae16-4248-40dd-95f9-0f59c0eb5426","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8b2913a0-d006-479c-84ed-5b21e0e6c392","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"499b79d0-0585-4dae-ab69-196a551793da","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6ba6df3b-4a6b-4a89-a890-0f79f387d4d7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e9668b1c-a000-4332-86e8-4efd73b02409","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T23:15:23.514 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:23.511+0000 7f3381200640 1 -- 172.21.15.6:0/2880488825 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f335806bda0 msgr2=0x7f335806e260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:23.514 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:23.511+0000 7f3381200640 1 --2- 172.21.15.6:0/2880488825 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f335806bda0 0x7f335806e260 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f336c03c000 tx=0x7f336c00a000 comp rx=0 tx=0).stop 2024-04-24T23:15:23.514 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:23.511+0000 7f3381200640 1 -- 172.21.15.6:0/2880488825 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f337c0c1460 msgr2=0x7f337c15df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:23.514 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:23.511+0000 7f3381200640 1 --2- 172.21.15.6:0/2880488825 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f337c0c1460 0x7f337c15df80 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f337400c7d0 tx=0x7f337400cc90 comp rx=0 tx=0).stop 2024-04-24T23:15:23.514 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:23.512+0000 7f3381200640 1 -- 172.21.15.6:0/2880488825 shutdown_connections 2024-04-24T23:15:23.514 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:23.512+0000 7f3381200640 1 --2- 172.21.15.6:0/2880488825 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f335806bda0 0x7f335806e260 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:23.514 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:23.512+0000 7f3381200640 1 --2- 172.21.15.6:0/2880488825 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f337c0c8870 0x7f337c0c0f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:23.514 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:23.512+0000 7f3381200640 1 --2- 172.21.15.6:0/2880488825 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f337c0c1460 0x7f337c15df80 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:23.514 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:23.512+0000 7f3381200640 1 --2- 172.21.15.6:0/2880488825 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f337c0c5b40 0x7f337c06ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:23.514 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:23.512+0000 7f3381200640 1 -- 172.21.15.6:0/2880488825 >> 172.21.15.6:0/2880488825 conn(0x7f337c0cc2f0 msgr2=0x7f337c0727d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:15:23.515 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:23.512+0000 7f3381200640 1 -- 172.21.15.6:0/2880488825 shutdown_connections 2024-04-24T23:15:23.515 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:23.512+0000 7f3381200640 1 -- 172.21.15.6:0/2880488825 wait complete. 2024-04-24T23:15:23.527 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-04-24T23:15:30.335 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-24T23:15:30.532 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:30.530+0000 7f8701800640 1 Processor -- start 2024-04-24T23:15:30.532 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:30.530+0000 7f8701800640 1 -- start start 2024-04-24T23:15:30.535 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:30.532+0000 7f8701800640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f86fc0c8530 0x7f86fc0cc990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:30.535 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:30.532+0000 7f8701800640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f86fc0cd0f0 con 0x7f86fc0cb490 2024-04-24T23:15:30.535 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:30.532+0000 7f8701800640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f86fc0cd3d0 con 0x7f86fc0cb870 2024-04-24T23:15:30.535 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:30.532+0000 7f8701800640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f86fc0cced0 con 0x7f86fc0c8530 2024-04-24T23:15:30.535 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:30.532+0000 7f86fbe00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f86fc0cb490 0x7f86fc0d05a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:50258/0 (socket says 172.21.15.6:50258) 2024-04-24T23:15:30.535 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:30.532+0000 7f86fbe00640 1 -- 172.21.15.6:0/1102393884 learned_addr learned my addr 172.21.15.6:0/1102393884 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:15:30.535 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:30.532+0000 7f86fb400640 1 --1- 172.21.15.6:0/1102393884 >> v1:172.21.15.190:6789/0 conn(0x7f86fc0cb870 0x7f86fc0c7e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.6:39612/0 (socket says 172.21.15.6:39612) 2024-04-24T23:15:30.535 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:30.532+0000 7f8700e00640 1 --2- 172.21.15.6:0/1102393884 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f86fc0c8530 0x7f86fc0cc990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:30.535 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:30.532+0000 7f86faa00640 1 -- 172.21.15.6:0/1102393884 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 888344642 0 0) 0x7f86fc0cd3d0 con 0x7f86fc0cb870 2024-04-24T23:15:30.536 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:30.532+0000 7f86faa00640 1 -- 172.21.15.6:0/1102393884 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f86e0003520 con 0x7f86fc0cb870 2024-04-24T23:15:30.536 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:30.532+0000 7f86faa00640 1 -- 172.21.15.6:0/1102393884 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3085104830 0 0) 0x7f86fc0cd0f0 con 0x7f86fc0cb490 2024-04-24T23:15:30.536 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:30.533+0000 7f86faa00640 1 -- 172.21.15.6:0/1102393884 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f86fc0cd3d0 con 0x7f86fc0cb490 2024-04-24T23:15:30.536 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:30.533+0000 7f8700e00640 1 -- 172.21.15.6:0/1102393884 >> v1:172.21.15.190:6789/0 conn(0x7f86fc0cb870 legacy=0x7f86fc0c7e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:30.536 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:30.533+0000 7f8700e00640 1 -- 172.21.15.6:0/1102393884 >> v1:172.21.15.6:6789/0 conn(0x7f86fc0cb490 legacy=0x7f86fc0d05a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:30.536 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:30.533+0000 7f8700e00640 1 -- 172.21.15.6:0/1102393884 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86fc0cd6b0 con 0x7f86fc0c8530 2024-04-24T23:15:30.537 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:30.534+0000 7f8700e00640 1 --2- 172.21.15.6:0/1102393884 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f86fc0c8530 0x7f86fc0cc990 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f86f40097d0 tx=0x7f86f402a290 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f2fb66a762372e4b server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:15:30.537 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:30.534+0000 7f86faa00640 1 -- 172.21.15.6:0/1102393884 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86f4037070 con 0x7f86fc0c8530 2024-04-24T23:15:30.537 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:30.534+0000 7f86faa00640 1 -- 172.21.15.6:0/1102393884 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f86f4032440 con 0x7f86fc0c8530 2024-04-24T23:15:30.538 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:30.534+0000 7f86faa00640 1 -- 172.21.15.6:0/1102393884 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86f40327f0 con 0x7f86fc0c8530 2024-04-24T23:15:30.538 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:30.535+0000 7f8701800640 1 -- 172.21.15.6:0/1102393884 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f86fc0c8530 msgr2=0x7f86fc0cc990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:30.538 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:30.535+0000 7f8701800640 1 --2- 172.21.15.6:0/1102393884 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f86fc0c8530 0x7f86fc0cc990 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f86f40097d0 tx=0x7f86f402a290 comp rx=0 tx=0).stop 2024-04-24T23:15:30.538 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:30.535+0000 7f8701800640 1 -- 172.21.15.6:0/1102393884 shutdown_connections 2024-04-24T23:15:30.538 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:30.535+0000 7f8701800640 1 --2- 172.21.15.6:0/1102393884 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f86fc0c8530 0x7f86fc0cc990 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:30.538 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:30.535+0000 7f8701800640 1 -- 172.21.15.6:0/1102393884 >> 172.21.15.6:0/1102393884 conn(0x7f86fc06da80 msgr2=0x7f86fc072890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:15:30.538 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:30.535+0000 7f8701800640 1 -- 172.21.15.6:0/1102393884 shutdown_connections 2024-04-24T23:15:30.538 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:30.535+0000 7f8701800640 1 -- 172.21.15.6:0/1102393884 wait complete. 2024-04-24T23:15:30.539 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:30.537+0000 7f8701800640 1 Processor -- start 2024-04-24T23:15:30.539 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:30.537+0000 7f8701800640 1 -- start start 2024-04-24T23:15:30.541 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:30.538+0000 7f8701800640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f86fc0c8530 0x7f86fc15b4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:30.541 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:30.538+0000 7f8701800640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f86fc0cb490 0x7f86fc1f3bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:30.541 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:30.538+0000 7f8701800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f86fc0cb870 0x7f86fc1f4130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:30.541 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:30.538+0000 7f8701800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f86fc0d0cb0 con 0x7f86fc0cb870 2024-04-24T23:15:30.541 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:30.538+0000 7f8701800640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f86fc0c7920 con 0x7f86fc0c8530 2024-04-24T23:15:30.541 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:30.538+0000 7f8701800640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f86fc0b5940 con 0x7f86fc0cb490 2024-04-24T23:15:30.541 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:30.538+0000 7f8700e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f86fc0cb870 0x7f86fc1f4130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:30.541 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:30.538+0000 7f8700e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f86fc0cb870 0x7f86fc1f4130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:51620/0 (socket says 172.21.15.6:51620) 2024-04-24T23:15:30.541 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:30.538+0000 7f8700e00640 1 -- 172.21.15.6:0/2994544489 learned_addr learned my addr 172.21.15.6:0/2994544489 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:15:30.541 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:30.538+0000 7f86fb400640 1 --2- 172.21.15.6:0/2994544489 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f86fc0cb490 0x7f86fc1f3bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:30.542 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:30.538+0000 7f86fbe00640 1 --2- 172.21.15.6:0/2994544489 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f86fc0c8530 0x7f86fc15b4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:30.542 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:30.538+0000 7f8700e00640 1 -- 172.21.15.6:0/2994544489 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f86fc0cb490 msgr2=0x7f86fc1f3bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:30.542 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:30.538+0000 7f8700e00640 1 --2- 172.21.15.6:0/2994544489 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f86fc0cb490 0x7f86fc1f3bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:30.542 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:30.538+0000 7f8700e00640 1 -- 172.21.15.6:0/2994544489 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f86fc0c8530 msgr2=0x7f86fc15b4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:30.542 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:30.538+0000 7f8700e00640 1 --2- 172.21.15.6:0/2994544489 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f86fc0c8530 0x7f86fc15b4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:30.542 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:30.538+0000 7f8700e00640 1 -- 172.21.15.6:0/2994544489 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86fc1f03d0 con 0x7f86fc0cb870 2024-04-24T23:15:30.542 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:30.539+0000 7f8700e00640 1 --2- 172.21.15.6:0/2994544489 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f86fc0cb870 0x7f86fc1f4130 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f86f4009720 tx=0x7f86f40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:15:30.543 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:30.539+0000 7f86f3e00640 1 -- 172.21.15.6:0/2994544489 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86f4041020 con 0x7f86fc0cb870 2024-04-24T23:15:30.543 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:30.539+0000 7f86f3e00640 1 -- 172.21.15.6:0/2994544489 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f86f403d620 con 0x7f86fc0cb870 2024-04-24T23:15:30.543 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:30.539+0000 7f8701800640 1 -- 172.21.15.6:0/2994544489 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86fc1f06b0 con 0x7f86fc0cb870 2024-04-24T23:15:30.543 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:30.539+0000 7f86f3e00640 1 -- 172.21.15.6:0/2994544489 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86f4037040 con 0x7f86fc0cb870 2024-04-24T23:15:30.543 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:30.539+0000 7f8701800640 1 -- 172.21.15.6:0/2994544489 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86fc1f0c70 con 0x7f86fc0cb870 2024-04-24T23:15:30.543 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:30.541+0000 7f8701800640 1 -- 172.21.15.6:0/2994544489 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f86fc068fa0 con 0x7f86fc0cb870 2024-04-24T23:15:30.544 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:30.542+0000 7f86f3e00640 1 -- 172.21.15.6:0/2994544489 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f86f4044020 con 0x7f86fc0cb870 2024-04-24T23:15:30.545 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:30.543+0000 7f86f3e00640 1 --2- 172.21.15.6:0/2994544489 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f86d806be30 0x7f86d806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:30.545 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:30.543+0000 7f86f3e00640 1 -- 172.21.15.6:0/2994544489 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7f86f40ab150 con 0x7f86fc0cb870 2024-04-24T23:15:30.545 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:30.543+0000 7f86fbe00640 1 --2- 172.21.15.6:0/2994544489 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f86d806be30 0x7f86d806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:30.546 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:30.544+0000 7f86fbe00640 1 --2- 172.21.15.6:0/2994544489 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f86d806be30 0x7f86d806e2f0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f86ec005c00 tx=0x7f86ec009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:15:30.550 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:30.549+0000 7f86f3e00640 1 -- 172.21.15.6:0/2994544489 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f86f404b030 con 0x7f86fc0cb870 2024-04-24T23:15:30.753 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:30.751+0000 7f8701800640 1 -- 172.21.15.6:0/2994544489 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f86fc0cbd60 con 0x7f86fc0cb870 2024-04-24T23:15:30.754 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:30.752+0000 7f86f3e00640 1 -- 172.21.15.6:0/2994544489 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) ==== 74+0+12824 (secure 0 0 0) 0x7f86f4079770 con 0x7f86fc0cb870 2024-04-24T23:15:30.754 INFO:teuthology.misc.health.smithi006.stdout: 2024-04-24T23:15:30.755 INFO:teuthology.misc.health.smithi006.stdout:{"epoch":16,"fsid":"18bef68d-5e6d-4ec5-862b-b6f206e2588d","created":"2024-04-24T23:15:12.535072+0000","modified":"2024-04-24T23:15:26.816031+0000","last_up_change":"2024-04-24T23:15:23.773223+0000","last_in_change":"2024-04-24T23:15:14.905328+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T23:15:24.747441+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3b7a212d-d498-462e-90bf-ed493c803a09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":3022569437},{"type":"v1","addr":"172.21.15.6:6809","nonce":3022569437}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":3022569437},{"type":"v1","addr":"172.21.15.6:6811","nonce":3022569437}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":3022569437},{"type":"v1","addr":"172.21.15.6:6815","nonce":3022569437}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":3022569437},{"type":"v1","addr":"172.21.15.6:6813","nonce":3022569437}]},"public_addr":"172.21.15.6:6809/3022569437","cluster_addr":"172.21.15.6:6811/3022569437","heartbeat_back_addr":"172.21.15.6:6815/3022569437","heartbeat_front_addr":"172.21.15.6:6813/3022569437","state":["exists","up"]},{"osd":1,"uuid":"7c97f5a2-1735-4775-9f15-eb50099b19d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":3687836888},{"type":"v1","addr":"172.21.15.6:6817","nonce":3687836888}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":3687836888},{"type":"v1","addr":"172.21.15.6:6819","nonce":3687836888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":3687836888},{"type":"v1","addr":"172.21.15.6:6823","nonce":3687836888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":3687836888},{"type":"v1","addr":"172.21.15.6:6821","nonce":3687836888}]},"public_addr":"172.21.15.6:6817/3687836888","cluster_addr":"172.21.15.6:6819/3687836888","heartbeat_back_addr":"172.21.15.6:6823/3687836888","heartbeat_front_addr":"172.21.15.6:6821/3687836888","state":["exists","up"]},{"osd":2,"uuid":"d86bbc18-438c-4ec2-8018-ba7dd78f790b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":1020213724},{"type":"v1","addr":"172.21.15.6:6801","nonce":1020213724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":1020213724},{"type":"v1","addr":"172.21.15.6:6803","nonce":1020213724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":1020213724},{"type":"v1","addr":"172.21.15.6:6807","nonce":1020213724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":1020213724},{"type":"v1","addr":"172.21.15.6:6805","nonce":1020213724}]},"public_addr":"172.21.15.6:6801/1020213724","cluster_addr":"172.21.15.6:6803/1020213724","heartbeat_back_addr":"172.21.15.6:6807/1020213724","heartbeat_front_addr":"172.21.15.6:6805/1020213724","state":["exists","up"]},{"osd":3,"uuid":"3754ae16-4248-40dd-95f9-0f59c0eb5426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6824","nonce":3163883468},{"type":"v1","addr":"172.21.15.6:6825","nonce":3163883468}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":3163883468},{"type":"v1","addr":"172.21.15.6:6827","nonce":3163883468}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":3163883468},{"type":"v1","addr":"172.21.15.6:6831","nonce":3163883468}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":3163883468},{"type":"v1","addr":"172.21.15.6:6829","nonce":3163883468}]},"public_addr":"172.21.15.6:6825/3163883468","cluster_addr":"172.21.15.6:6827/3163883468","heartbeat_back_addr":"172.21.15.6:6831/3163883468","heartbeat_front_addr":"172.21.15.6:6829/3163883468","state":["exists","up"]},{"osd":4,"uuid":"8b2913a0-d006-479c-84ed-5b21e0e6c392","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":1989500934},{"type":"v1","addr":"172.21.15.190:6809","nonce":1989500934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":1989500934},{"type":"v1","addr":"172.21.15.190:6811","nonce":1989500934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":1989500934},{"type":"v1","addr":"172.21.15.190:6815","nonce":1989500934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":1989500934},{"type":"v1","addr":"172.21.15.190:6813","nonce":1989500934}]},"public_addr":"172.21.15.190:6809/1989500934","cluster_addr":"172.21.15.190:6811/1989500934","heartbeat_back_addr":"172.21.15.190:6815/1989500934","heartbeat_front_addr":"172.21.15.190:6813/1989500934","state":["exists","up"]},{"osd":5,"uuid":"499b79d0-0585-4dae-ab69-196a551793da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":3687548412},{"type":"v1","addr":"172.21.15.190:6801","nonce":3687548412}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":3687548412},{"type":"v1","addr":"172.21.15.190:6803","nonce":3687548412}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":3687548412},{"type":"v1","addr":"172.21.15.190:6807","nonce":3687548412}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":3687548412},{"type":"v1","addr":"172.21.15.190:6805","nonce":3687548412}]},"public_addr":"172.21.15.190:6801/3687548412","cluster_addr":"172.21.15.190:6803/3687548412","heartbeat_back_addr":"172.21.15.190:6807/3687548412","heartbeat_front_addr":"172.21.15.190:6805/3687548412","state":["exists","up"]},{"osd":6,"uuid":"6ba6df3b-4a6b-4a89-a890-0f79f387d4d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":870571564},{"type":"v1","addr":"172.21.15.190:6817","nonce":870571564}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":870571564},{"type":"v1","addr":"172.21.15.190:6819","nonce":870571564}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":870571564},{"type":"v1","addr":"172.21.15.190:6823","nonce":870571564}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":870571564},{"type":"v1","addr":"172.21.15.190:6821","nonce":870571564}]},"public_addr":"172.21.15.190:6817/870571564","cluster_addr":"172.21.15.190:6819/870571564","heartbeat_back_addr":"172.21.15.190:6823/870571564","heartbeat_front_addr":"172.21.15.190:6821/870571564","state":["exists","up"]},{"osd":7,"uuid":"e9668b1c-a000-4332-86e8-4efd73b02409","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":123639197},{"type":"v1","addr":"172.21.15.190:6825","nonce":123639197}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":123639197},{"type":"v1","addr":"172.21.15.190:6827","nonce":123639197}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":123639197},{"type":"v1","addr":"172.21.15.190:6831","nonce":123639197}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":123639197},{"type":"v1","addr":"172.21.15.190:6829","nonce":123639197}]},"public_addr":"172.21.15.190:6825/123639197","cluster_addr":"172.21.15.190:6827/123639197","heartbeat_back_addr":"172.21.15.190:6831/123639197","heartbeat_front_addr":"172.21.15.190:6829/123639197","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.321646+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.363873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.466963+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.531989+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:19.990707+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.043543+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.032492+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.115004+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T23:15:30.757 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:30.754+0000 7f8701800640 1 -- 172.21.15.6:0/2994544489 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f86d806be30 msgr2=0x7f86d806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:30.757 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:30.754+0000 7f8701800640 1 --2- 172.21.15.6:0/2994544489 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f86d806be30 0x7f86d806e2f0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f86ec005c00 tx=0x7f86ec009040 comp rx=0 tx=0).stop 2024-04-24T23:15:30.757 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:30.754+0000 7f8701800640 1 -- 172.21.15.6:0/2994544489 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f86fc0cb870 msgr2=0x7f86fc1f4130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:30.757 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:30.754+0000 7f8701800640 1 --2- 172.21.15.6:0/2994544489 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f86fc0cb870 0x7f86fc1f4130 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f86f4009720 tx=0x7f86f40046c0 comp rx=0 tx=0).stop 2024-04-24T23:15:30.757 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:30.755+0000 7f8701800640 1 -- 172.21.15.6:0/2994544489 shutdown_connections 2024-04-24T23:15:30.757 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:30.755+0000 7f8701800640 1 --2- 172.21.15.6:0/2994544489 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f86d806be30 0x7f86d806e2f0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:30.757 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:30.755+0000 7f8701800640 1 --2- 172.21.15.6:0/2994544489 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f86fc0cb490 0x7f86fc1f3bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:30.757 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:30.755+0000 7f8701800640 1 --2- 172.21.15.6:0/2994544489 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f86fc0cb870 0x7f86fc1f4130 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:30.757 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:30.755+0000 7f8701800640 1 --2- 172.21.15.6:0/2994544489 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f86fc0c8530 0x7f86fc15b4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:30.758 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:30.755+0000 7f8701800640 1 -- 172.21.15.6:0/2994544489 >> 172.21.15.6:0/2994544489 conn(0x7f86fc06da80 msgr2=0x7f86fc0ceb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:15:30.758 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:30.755+0000 7f8701800640 1 -- 172.21.15.6:0/2994544489 shutdown_connections 2024-04-24T23:15:30.758 INFO:teuthology.misc.health.smithi006.stderr:2024-04-24T23:15:30.755+0000 7f8701800640 1 -- 172.21.15.6:0/2994544489 wait complete. 2024-04-24T23:15:30.772 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-04-24T23:15:30.772 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-04-24T23:15:30.772 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-24T23:15:30.772 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-24T23:15:30.777 INFO:tasks.ceph.mds.a:Started 2024-04-24T23:15:30.777 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-24T23:15:30.777 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-24T23:15:30.781 INFO:tasks.ceph.mds.c:Started 2024-04-24T23:15:30.781 INFO:tasks.ceph.mds.e:Restarting daemon 2024-04-24T23:15:30.782 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i e 2024-04-24T23:15:30.786 INFO:tasks.ceph.mds.e:Started 2024-04-24T23:15:30.786 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-24T23:15:30.786 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-24T23:15:30.789 INFO:tasks.ceph.mds.b:Started 2024-04-24T23:15:30.789 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-24T23:15:30.789 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-24T23:15:30.792 INFO:tasks.ceph.mds.d:Started 2024-04-24T23:15:30.792 INFO:tasks.ceph.mds.f:Restarting daemon 2024-04-24T23:15:30.792 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i f 2024-04-24T23:15:30.794 INFO:tasks.ceph.mds.f:Started 2024-04-24T23:15:30.794 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2024-04-24T23:15:30.794 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T23:15:30.873 INFO:tasks.ceph.mds.b.smithi190.stdout:starting mds.b at 2024-04-24T23:15:30.873 INFO:tasks.ceph.mds.f.smithi190.stdout:starting mds.f at 2024-04-24T23:15:30.873 INFO:tasks.ceph.mds.d.smithi190.stdout:starting mds.d at 2024-04-24T23:15:30.930 INFO:tasks.ceph.mds.a.smithi006.stdout:starting mds.a at 2024-04-24T23:15:30.933 INFO:tasks.ceph.mds.c.smithi006.stdout:starting mds.c at 2024-04-24T23:15:30.936 INFO:tasks.ceph.mds.e.smithi006.stdout:starting mds.e at 2024-04-24T23:15:30.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:30.976+0000 7f3238a00640 1 Processor -- start 2024-04-24T23:15:30.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:30.976+0000 7f3238a00640 1 -- start start 2024-04-24T23:15:30.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:30.977+0000 7f3238a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f32340c8f50 0x7f323406e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:30.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:30.977+0000 7f3238a00640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f32340c9330 0x7f323406e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:30.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:30.977+0000 7f3238a00640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f32340c6e70 con 0x7f3234152bc0 2024-04-24T23:15:30.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:30.977+0000 7f3238a00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f32340c73d0 con 0x7f32340c8f50 2024-04-24T23:15:30.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:30.977+0000 7f3238a00640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f32340c8ac0 con 0x7f32340c9330 2024-04-24T23:15:30.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:30.977+0000 7f322be00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f32340c8f50 0x7f323406e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:30.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:30.977+0000 7f322be00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f32340c8f50 0x7f323406e3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:51680/0 (socket says 172.21.15.6:51680) 2024-04-24T23:15:30.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:30.977+0000 7f3231e00640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f3234152bc0 0x7f3234159390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.6:39624/0 (socket says 172.21.15.6:39624) 2024-04-24T23:15:30.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:30.977+0000 7f322be00640 1 -- 172.21.15.6:0/1410977456 learned_addr learned my addr 172.21.15.6:0/1410977456 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:15:30.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:30.978+0000 7f3232800640 1 --2- 172.21.15.6:0/1410977456 >> v2:172.21.15.190:3300/0 conn(0x7f32340c9330 0x7f323406e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:30.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:30.978+0000 7f322be00640 1 -- 172.21.15.6:0/1410977456 >> v2:172.21.15.190:3300/0 conn(0x7f32340c9330 msgr2=0x7f323406e8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:15:30.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:30.978+0000 7f322be00640 1 --2- 172.21.15.6:0/1410977456 >> v2:172.21.15.190:3300/0 conn(0x7f32340c9330 0x7f323406e8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:30.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:30.978+0000 7f3231400640 1 -- 172.21.15.6:0/1410977456 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2927999941 0 0) 0x7f32340c6e70 con 0x7f3234152bc0 2024-04-24T23:15:30.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:30.978+0000 7f322be00640 1 -- 172.21.15.6:0/1410977456 >> v1:172.21.15.190:6789/0 conn(0x7f3234152bc0 legacy=0x7f3234159390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:30.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:30.978+0000 7f322be00640 1 -- 172.21.15.6:0/1410977456 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32340c7110 con 0x7f32340c8f50 2024-04-24T23:15:30.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:30.979+0000 7f322be00640 1 --2- 172.21.15.6:0/1410977456 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f32340c8f50 0x7f323406e3a0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f32340c7550 tx=0x7f321c02a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f058e264f5a32b2 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:15:30.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:30.979+0000 7f3231400640 1 -- 172.21.15.6:0/1410977456 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f321c037070 con 0x7f32340c8f50 2024-04-24T23:15:30.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:30.979+0000 7f3231400640 1 -- 172.21.15.6:0/1410977456 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f321c032440 con 0x7f32340c8f50 2024-04-24T23:15:30.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:30.979+0000 7f3231400640 1 -- 172.21.15.6:0/1410977456 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f321c032810 con 0x7f32340c8f50 2024-04-24T23:15:30.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:30.980+0000 7f3238a00640 1 -- 172.21.15.6:0/1410977456 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f32340c8f50 msgr2=0x7f323406e3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:30.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:30.980+0000 7f3238a00640 1 --2- 172.21.15.6:0/1410977456 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f32340c8f50 0x7f323406e3a0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f32340c7550 tx=0x7f321c02a1e0 comp rx=0 tx=0).stop 2024-04-24T23:15:30.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:30.980+0000 7f3238a00640 1 -- 172.21.15.6:0/1410977456 shutdown_connections 2024-04-24T23:15:30.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:30.980+0000 7f3238a00640 1 --2- 172.21.15.6:0/1410977456 >> v2:172.21.15.190:3300/0 conn(0x7f32340c9330 0x7f323406e8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:30.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:30.980+0000 7f3238a00640 1 --2- 172.21.15.6:0/1410977456 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f32340c8f50 0x7f323406e3a0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:30.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:30.980+0000 7f3238a00640 1 -- 172.21.15.6:0/1410977456 >> 172.21.15.6:0/1410977456 conn(0x7f323406b0e0 msgr2=0x7f323406b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:15:30.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:30.980+0000 7f3238a00640 1 -- 172.21.15.6:0/1410977456 shutdown_connections 2024-04-24T23:15:30.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:30.980+0000 7f3238a00640 1 -- 172.21.15.6:0/1410977456 wait complete. 2024-04-24T23:15:30.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:30.981+0000 7f3238a00640 1 Processor -- start 2024-04-24T23:15:30.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:30.982+0000 7f3238a00640 1 -- start start 2024-04-24T23:15:30.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:30.982+0000 7f3238a00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f32340c8f50 0x7f323415a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:30.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:30.982+0000 7f3238a00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f32340c9330 0x7f323415ae80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:30.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:30.983+0000 7f3238a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3234152bc0 0x7f3234150990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:30.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:30.983+0000 7f3238a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f32340c8750 con 0x7f3234152bc0 2024-04-24T23:15:30.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:30.983+0000 7f3238a00640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f32340c8470 con 0x7f32340c8f50 2024-04-24T23:15:30.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:30.983+0000 7f3238a00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f3234152fa0 con 0x7f32340c9330 2024-04-24T23:15:30.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:30.983+0000 7f3232800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3234152bc0 0x7f3234150990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:30.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:30.983+0000 7f3231e00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f32340c8f50 0x7f323415a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:30.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:30.983+0000 7f322be00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f32340c9330 0x7f323415ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:30.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:30.983+0000 7f3232800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3234152bc0 0x7f3234150990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:51690/0 (socket says 172.21.15.6:51690) 2024-04-24T23:15:30.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:30.983+0000 7f3232800640 1 -- 172.21.15.6:0/2016808790 learned_addr learned my addr 172.21.15.6:0/2016808790 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:15:30.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:30.983+0000 7f3232800640 1 -- 172.21.15.6:0/2016808790 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f32340c9330 msgr2=0x7f323415ae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:30.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:30.983+0000 7f3232800640 1 --2- 172.21.15.6:0/2016808790 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f32340c9330 0x7f323415ae80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:30.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:30.983+0000 7f3232800640 1 -- 172.21.15.6:0/2016808790 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f32340c8f50 msgr2=0x7f323415a890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:30.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:30.983+0000 7f3232800640 1 --2- 172.21.15.6:0/2016808790 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f32340c8f50 0x7f323415a890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:30.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:30.983+0000 7f3232800640 1 -- 172.21.15.6:0/2016808790 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32341eef40 con 0x7f3234152bc0 2024-04-24T23:15:30.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:30.983+0000 7f3232800640 1 --2- 172.21.15.6:0/2016808790 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3234152bc0 0x7f3234150990 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f322c009dd0 tx=0x7f322c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:15:30.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:30.984+0000 7f322a000640 1 -- 172.21.15.6:0/2016808790 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f322c018020 con 0x7f3234152bc0 2024-04-24T23:15:30.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:30.984+0000 7f322a000640 1 -- 172.21.15.6:0/2016808790 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f322c014480 con 0x7f3234152bc0 2024-04-24T23:15:30.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:30.984+0000 7f322a000640 1 -- 172.21.15.6:0/2016808790 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f322c014870 con 0x7f3234152bc0 2024-04-24T23:15:30.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:30.984+0000 7f3238a00640 1 -- 172.21.15.6:0/2016808790 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32341ef220 con 0x7f3234152bc0 2024-04-24T23:15:30.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:30.984+0000 7f3238a00640 1 -- 172.21.15.6:0/2016808790 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3234159db0 con 0x7f3234152bc0 2024-04-24T23:15:30.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:30.985+0000 7f3238a00640 1 -- 172.21.15.6:0/2016808790 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f32340767b0 con 0x7f3234152bc0 2024-04-24T23:15:30.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:30.987+0000 7f322a000640 1 -- 172.21.15.6:0/2016808790 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f322c005d10 con 0x7f3234152bc0 2024-04-24T23:15:30.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:30.988+0000 7f322a000640 1 --2- 172.21.15.6:0/2016808790 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f321006be60 0x7f321006e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:30.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:30.988+0000 7f322a000640 1 -- 172.21.15.6:0/2016808790 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7f322c08b5f0 con 0x7f3234152bc0 2024-04-24T23:15:30.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:30.988+0000 7f3231e00640 1 --2- 172.21.15.6:0/2016808790 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f321006be60 0x7f321006e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:30.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:30.989+0000 7f3231e00640 1 --2- 172.21.15.6:0/2016808790 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f321006be60 0x7f321006e320 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f3224000f60 tx=0x7f3224007d50 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:15:30.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:30.993+0000 7f322a000640 1 -- 172.21.15.6:0/2016808790 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f322c022070 con 0x7f3234152bc0 2024-04-24T23:15:31.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.190+0000 7f3238a00640 1 -- 172.21.15.6:0/2016808790 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f323414ca10 con 0x7f3234152bc0 2024-04-24T23:15:31.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.192+0000 7f322a000640 1 -- 172.21.15.6:0/2016808790 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) ==== 74+0+12824 (secure 0 0 0) 0x7f322c010040 con 0x7f3234152bc0 2024-04-24T23:15:31.194 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:15:31.195 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":16,"fsid":"18bef68d-5e6d-4ec5-862b-b6f206e2588d","created":"2024-04-24T23:15:12.535072+0000","modified":"2024-04-24T23:15:26.816031+0000","last_up_change":"2024-04-24T23:15:23.773223+0000","last_in_change":"2024-04-24T23:15:14.905328+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T23:15:24.747441+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3b7a212d-d498-462e-90bf-ed493c803a09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":3022569437},{"type":"v1","addr":"172.21.15.6:6809","nonce":3022569437}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":3022569437},{"type":"v1","addr":"172.21.15.6:6811","nonce":3022569437}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":3022569437},{"type":"v1","addr":"172.21.15.6:6815","nonce":3022569437}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":3022569437},{"type":"v1","addr":"172.21.15.6:6813","nonce":3022569437}]},"public_addr":"172.21.15.6:6809/3022569437","cluster_addr":"172.21.15.6:6811/3022569437","heartbeat_back_addr":"172.21.15.6:6815/3022569437","heartbeat_front_addr":"172.21.15.6:6813/3022569437","state":["exists","up"]},{"osd":1,"uuid":"7c97f5a2-1735-4775-9f15-eb50099b19d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":3687836888},{"type":"v1","addr":"172.21.15.6:6817","nonce":3687836888}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":3687836888},{"type":"v1","addr":"172.21.15.6:6819","nonce":3687836888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":3687836888},{"type":"v1","addr":"172.21.15.6:6823","nonce":3687836888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":3687836888},{"type":"v1","addr":"172.21.15.6:6821","nonce":3687836888}]},"public_addr":"172.21.15.6:6817/3687836888","cluster_addr":"172.21.15.6:6819/3687836888","heartbeat_back_addr":"172.21.15.6:6823/3687836888","heartbeat_front_addr":"172.21.15.6:6821/3687836888","state":["exists","up"]},{"osd":2,"uuid":"d86bbc18-438c-4ec2-8018-ba7dd78f790b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":1020213724},{"type":"v1","addr":"172.21.15.6:6801","nonce":1020213724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":1020213724},{"type":"v1","addr":"172.21.15.6:6803","nonce":1020213724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":1020213724},{"type":"v1","addr":"172.21.15.6:6807","nonce":1020213724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":1020213724},{"type":"v1","addr":"172.21.15.6:6805","nonce":1020213724}]},"public_addr":"172.21.15.6:6801/1020213724","cluster_addr":"172.21.15.6:6803/1020213724","heartbeat_back_addr":"172.21.15.6:6807/1020213724","heartbeat_front_addr":"172.21.15.6:6805/1020213724","state":["exists","up"]},{"osd":3,"uuid":"3754ae16-4248-40dd-95f9-0f59c0eb5426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6824","nonce":3163883468},{"type":"v1","addr":"172.21.15.6:6825","nonce":3163883468}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":3163883468},{"type":"v1","addr":"172.21.15.6:6827","nonce":3163883468}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":3163883468},{"type":"v1","addr":"172.21.15.6:6831","nonce":3163883468}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":3163883468},{"type":"v1","addr":"172.21.15.6:6829","nonce":3163883468}]},"public_addr":"172.21.15.6:6825/3163883468","cluster_addr":"172.21.15.6:6827/3163883468","heartbeat_back_addr":"172.21.15.6:6831/3163883468","heartbeat_front_addr":"172.21.15.6:6829/3163883468","state":["exists","up"]},{"osd":4,"uuid":"8b2913a0-d006-479c-84ed-5b21e0e6c392","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":1989500934},{"type":"v1","addr":"172.21.15.190:6809","nonce":1989500934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":1989500934},{"type":"v1","addr":"172.21.15.190:6811","nonce":1989500934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":1989500934},{"type":"v1","addr":"172.21.15.190:6815","nonce":1989500934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":1989500934},{"type":"v1","addr":"172.21.15.190:6813","nonce":1989500934}]},"public_addr":"172.21.15.190:6809/1989500934","cluster_addr":"172.21.15.190:6811/1989500934","heartbeat_back_addr":"172.21.15.190:6815/1989500934","heartbeat_front_addr":"172.21.15.190:6813/1989500934","state":["exists","up"]},{"osd":5,"uuid":"499b79d0-0585-4dae-ab69-196a551793da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":3687548412},{"type":"v1","addr":"172.21.15.190:6801","nonce":3687548412}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":3687548412},{"type":"v1","addr":"172.21.15.190:6803","nonce":3687548412}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":3687548412},{"type":"v1","addr":"172.21.15.190:6807","nonce":3687548412}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":3687548412},{"type":"v1","addr":"172.21.15.190:6805","nonce":3687548412}]},"public_addr":"172.21.15.190:6801/3687548412","cluster_addr":"172.21.15.190:6803/3687548412","heartbeat_back_addr":"172.21.15.190:6807/3687548412","heartbeat_front_addr":"172.21.15.190:6805/3687548412","state":["exists","up"]},{"osd":6,"uuid":"6ba6df3b-4a6b-4a89-a890-0f79f387d4d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":870571564},{"type":"v1","addr":"172.21.15.190:6817","nonce":870571564}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":870571564},{"type":"v1","addr":"172.21.15.190:6819","nonce":870571564}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":870571564},{"type":"v1","addr":"172.21.15.190:6823","nonce":870571564}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":870571564},{"type":"v1","addr":"172.21.15.190:6821","nonce":870571564}]},"public_addr":"172.21.15.190:6817/870571564","cluster_addr":"172.21.15.190:6819/870571564","heartbeat_back_addr":"172.21.15.190:6823/870571564","heartbeat_front_addr":"172.21.15.190:6821/870571564","state":["exists","up"]},{"osd":7,"uuid":"e9668b1c-a000-4332-86e8-4efd73b02409","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":123639197},{"type":"v1","addr":"172.21.15.190:6825","nonce":123639197}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":123639197},{"type":"v1","addr":"172.21.15.190:6827","nonce":123639197}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":123639197},{"type":"v1","addr":"172.21.15.190:6831","nonce":123639197}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":123639197},{"type":"v1","addr":"172.21.15.190:6829","nonce":123639197}]},"public_addr":"172.21.15.190:6825/123639197","cluster_addr":"172.21.15.190:6827/123639197","heartbeat_back_addr":"172.21.15.190:6831/123639197","heartbeat_front_addr":"172.21.15.190:6829/123639197","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.321646+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.363873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.466963+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.531989+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:19.990707+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.043543+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.032492+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.115004+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T23:15:31.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.195+0000 7f3238a00640 1 -- 172.21.15.6:0/2016808790 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f321006be60 msgr2=0x7f321006e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:31.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.195+0000 7f3238a00640 1 --2- 172.21.15.6:0/2016808790 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f321006be60 0x7f321006e320 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f3224000f60 tx=0x7f3224007d50 comp rx=0 tx=0).stop 2024-04-24T23:15:31.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.195+0000 7f3238a00640 1 -- 172.21.15.6:0/2016808790 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3234152bc0 msgr2=0x7f3234150990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:31.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.195+0000 7f3238a00640 1 --2- 172.21.15.6:0/2016808790 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3234152bc0 0x7f3234150990 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f322c009dd0 tx=0x7f322c00c490 comp rx=0 tx=0).stop 2024-04-24T23:15:31.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.195+0000 7f3238a00640 1 -- 172.21.15.6:0/2016808790 shutdown_connections 2024-04-24T23:15:31.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.195+0000 7f3238a00640 1 --2- 172.21.15.6:0/2016808790 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f321006be60 0x7f321006e320 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:31.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.195+0000 7f3238a00640 1 --2- 172.21.15.6:0/2016808790 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f32340c9330 0x7f323415ae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:31.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.195+0000 7f3238a00640 1 --2- 172.21.15.6:0/2016808790 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3234152bc0 0x7f3234150990 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:31.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.195+0000 7f3238a00640 1 --2- 172.21.15.6:0/2016808790 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f32340c8f50 0x7f323415a890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:31.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.195+0000 7f3238a00640 1 -- 172.21.15.6:0/2016808790 >> 172.21.15.6:0/2016808790 conn(0x7f323406b0e0 msgr2=0x7f32340ccc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:15:31.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.195+0000 7f3238a00640 1 -- 172.21.15.6:0/2016808790 shutdown_connections 2024-04-24T23:15:31.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.195+0000 7f3238a00640 1 -- 172.21.15.6:0/2016808790 wait complete. 2024-04-24T23:15:31.213 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T23:15:24.747441+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-24T23:15:31.214 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-24T23:15:31.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.422+0000 7fa8cae00640 1 Processor -- start 2024-04-24T23:15:31.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.422+0000 7fa8cae00640 1 -- start start 2024-04-24T23:15:31.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.423+0000 7fa8cae00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa8c4071530 0x7fa8c40c8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:31.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.424+0000 7fa8cae00640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fa8c40c9230 0x7fa8c4157950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:31.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.424+0000 7fa8cae00640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa8c40c8390 con 0x7fa8c4151bb0 2024-04-24T23:15:31.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.424+0000 7fa8cae00640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7fa8c4071d10 con 0x7fa8c40c9230 2024-04-24T23:15:31.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.424+0000 7fa8cae00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fa8c4071ff0 con 0x7fa8c4071530 2024-04-24T23:15:31.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.424+0000 7fa8c8e00640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fa8c40c9230 0x7fa8c4157950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:31.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.424+0000 7fa8c3400640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa8c4071530 0x7fa8c40c8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:31.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.424+0000 7fa8c8e00640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fa8c40c9230 0x7fa8c4157950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:39518/0 (socket says 172.21.15.6:39518) 2024-04-24T23:15:31.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.424+0000 7fa8c8e00640 1 -- 172.21.15.6:0/2730755165 learned_addr learned my addr 172.21.15.6:0/2730755165 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:15:31.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.424+0000 7fa8c3e00640 1 --1- 172.21.15.6:0/2730755165 >> v1:172.21.15.190:6789/0 conn(0x7fa8c4151bb0 0x7fa8c40c37f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.6:39636/0 (socket says 172.21.15.6:39636) 2024-04-24T23:15:31.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.424+0000 7fa8c8e00640 1 -- 172.21.15.6:0/2730755165 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa8c4071530 msgr2=0x7fa8c40c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:15:31.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.424+0000 7fa8c8e00640 1 --2- 172.21.15.6:0/2730755165 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa8c4071530 0x7fa8c40c8cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:31.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.424+0000 7fa8c8e00640 1 -- 172.21.15.6:0/2730755165 >> v1:172.21.15.190:6789/0 conn(0x7fa8c4151bb0 legacy=0x7fa8c40c37f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:31.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.424+0000 7fa8c8e00640 1 -- 172.21.15.6:0/2730755165 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8c40722d0 con 0x7fa8c40c9230 2024-04-24T23:15:31.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.424+0000 7fa8c3400640 1 --2- 172.21.15.6:0/2730755165 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa8c4071530 0x7fa8c40c8cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:15:31.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.425+0000 7fa8c8e00640 1 --2- 172.21.15.6:0/2730755165 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa8c40c9230 0x7fa8c4157950 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fa8bc00d150 tx=0x7fa8bc02c9c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f956a6f3ea7ef06d server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:15:31.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.426+0000 7fa8c2a00640 1 -- 172.21.15.6:0/2730755165 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8bc035020 con 0x7fa8c40c9230 2024-04-24T23:15:31.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.426+0000 7fa8c2a00640 1 -- 172.21.15.6:0/2730755165 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa8bc00b2d0 con 0x7fa8c40c9230 2024-04-24T23:15:31.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.426+0000 7fa8c2a00640 1 -- 172.21.15.6:0/2730755165 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8bc00b480 con 0x7fa8c40c9230 2024-04-24T23:15:31.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.426+0000 7fa8cae00640 1 -- 172.21.15.6:0/2730755165 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa8c40c9230 msgr2=0x7fa8c4157950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:31.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.426+0000 7fa8cae00640 1 --2- 172.21.15.6:0/2730755165 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa8c40c9230 0x7fa8c4157950 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fa8bc00d150 tx=0x7fa8bc02c9c0 comp rx=0 tx=0).stop 2024-04-24T23:15:31.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.427+0000 7fa8cae00640 1 -- 172.21.15.6:0/2730755165 shutdown_connections 2024-04-24T23:15:31.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.427+0000 7fa8cae00640 1 --2- 172.21.15.6:0/2730755165 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa8c40c9230 0x7fa8c4157950 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:31.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.427+0000 7fa8cae00640 1 --2- 172.21.15.6:0/2730755165 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa8c4071530 0x7fa8c40c8cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:31.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.427+0000 7fa8cae00640 1 -- 172.21.15.6:0/2730755165 >> 172.21.15.6:0/2730755165 conn(0x7fa8c406b0e0 msgr2=0x7fa8c406b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:15:31.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.427+0000 7fa8cae00640 1 -- 172.21.15.6:0/2730755165 shutdown_connections 2024-04-24T23:15:31.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.427+0000 7fa8cae00640 1 -- 172.21.15.6:0/2730755165 wait complete. 2024-04-24T23:15:31.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.429+0000 7fa8cae00640 1 Processor -- start 2024-04-24T23:15:31.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.429+0000 7fa8cae00640 1 -- start start 2024-04-24T23:15:31.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.430+0000 7fa8cae00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa8c4071530 0x7fa8c4156710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:31.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.430+0000 7fa8cae00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa8c40c9230 0x7fa8c4142ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:31.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.430+0000 7fa8cae00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa8c4151bb0 0x7fa8c4150290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:31.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.430+0000 7fa8cae00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fa8c40c80f0 con 0x7fa8c4071530 2024-04-24T23:15:31.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.430+0000 7fa8cae00640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fa8c40c7f70 con 0x7fa8c4151bb0 2024-04-24T23:15:31.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.430+0000 7fa8cae00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fa8c4157e90 con 0x7fa8c40c9230 2024-04-24T23:15:31.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.430+0000 7fa8c3e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa8c4071530 0x7fa8c4156710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:31.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.430+0000 7fa8c3e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa8c4071530 0x7fa8c4156710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:51694/0 (socket says 172.21.15.6:51694) 2024-04-24T23:15:31.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.430+0000 7fa8c3e00640 1 -- 172.21.15.6:0/2187770781 learned_addr learned my addr 172.21.15.6:0/2187770781 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:15:31.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.430+0000 7fa8c3400640 1 --2- 172.21.15.6:0/2187770781 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa8c40c9230 0x7fa8c4142ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:31.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.430+0000 7fa8c8e00640 1 --2- 172.21.15.6:0/2187770781 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa8c4151bb0 0x7fa8c4150290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:31.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.430+0000 7fa8c3e00640 1 -- 172.21.15.6:0/2187770781 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa8c40c9230 msgr2=0x7fa8c4142ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:31.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.430+0000 7fa8c3e00640 1 --2- 172.21.15.6:0/2187770781 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa8c40c9230 0x7fa8c4142ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:31.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.430+0000 7fa8c3e00640 1 -- 172.21.15.6:0/2187770781 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa8c4151bb0 msgr2=0x7fa8c4150290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:31.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.430+0000 7fa8c3e00640 1 --2- 172.21.15.6:0/2187770781 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa8c4151bb0 0x7fa8c4150290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:31.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.430+0000 7fa8c3e00640 1 -- 172.21.15.6:0/2187770781 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8c41f0880 con 0x7fa8c4071530 2024-04-24T23:15:31.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.431+0000 7fa8c3e00640 1 --2- 172.21.15.6:0/2187770781 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa8c4071530 0x7fa8c4156710 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fa8ac00c3c0 tx=0x7fa8ac00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:15:31.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.431+0000 7fa8bbe00640 1 -- 172.21.15.6:0/2187770781 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8ac0074e0 con 0x7fa8c4071530 2024-04-24T23:15:31.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.431+0000 7fa8bbe00640 1 -- 172.21.15.6:0/2187770781 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa8ac007680 con 0x7fa8c4071530 2024-04-24T23:15:31.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.431+0000 7fa8bbe00640 1 -- 172.21.15.6:0/2187770781 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8ac004a00 con 0x7fa8c4071530 2024-04-24T23:15:31.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.431+0000 7fa8cae00640 1 -- 172.21.15.6:0/2187770781 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8c41f0b60 con 0x7fa8c4071530 2024-04-24T23:15:31.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.431+0000 7fa8cae00640 1 -- 172.21.15.6:0/2187770781 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8c41f1120 con 0x7fa8c4071530 2024-04-24T23:15:31.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.433+0000 7fa8cae00640 1 -- 172.21.15.6:0/2187770781 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa8c40767b0 con 0x7fa8c4071530 2024-04-24T23:15:31.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.433+0000 7fa8bbe00640 1 -- 172.21.15.6:0/2187770781 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fa8ac01a020 con 0x7fa8c4071530 2024-04-24T23:15:31.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.434+0000 7fa8bbe00640 1 --2- 172.21.15.6:0/2187770781 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fa8a006be30 0x7fa8a006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:31.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.434+0000 7fa8bbe00640 1 -- 172.21.15.6:0/2187770781 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7fa8ac08a140 con 0x7fa8c4071530 2024-04-24T23:15:31.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.435+0000 7fa8c3400640 1 --2- 172.21.15.6:0/2187770781 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fa8a006be30 0x7fa8a006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:31.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.435+0000 7fa8c3400640 1 --2- 172.21.15.6:0/2187770781 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fa8a006be30 0x7fa8a006e2f0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fa8b4009560 tx=0x7fa8b4009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:15:31.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.439+0000 7fa8bbe00640 1 -- 172.21.15.6:0/2187770781 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fa8ac0587f0 con 0x7fa8c4071530 2024-04-24T23:15:31.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.641+0000 7fa8cae00640 1 -- 172.21.15.6:0/2187770781 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fa8c40c8ad0 con 0x7fa8c4071530 2024-04-24T23:15:31.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.642+0000 7fa8bbe00640 1 -- 172.21.15.6:0/2187770781 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v16) ==== 93+0+10 (secure 0 0 0) 0x7fa8ac05caf0 con 0x7fa8c4071530 2024-04-24T23:15:31.644 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-04-24T23:15:31.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.645+0000 7fa8cae00640 1 -- 172.21.15.6:0/2187770781 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fa8a006be30 msgr2=0x7fa8a006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:31.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.645+0000 7fa8cae00640 1 --2- 172.21.15.6:0/2187770781 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fa8a006be30 0x7fa8a006e2f0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fa8b4009560 tx=0x7fa8b4009190 comp rx=0 tx=0).stop 2024-04-24T23:15:31.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.645+0000 7fa8cae00640 1 -- 172.21.15.6:0/2187770781 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa8c4071530 msgr2=0x7fa8c4156710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:31.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.645+0000 7fa8cae00640 1 --2- 172.21.15.6:0/2187770781 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa8c4071530 0x7fa8c4156710 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fa8ac00c3c0 tx=0x7fa8ac00c880 comp rx=0 tx=0).stop 2024-04-24T23:15:31.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.645+0000 7fa8cae00640 1 -- 172.21.15.6:0/2187770781 shutdown_connections 2024-04-24T23:15:31.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.645+0000 7fa8cae00640 1 --2- 172.21.15.6:0/2187770781 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fa8a006be30 0x7fa8a006e2f0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:31.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.645+0000 7fa8cae00640 1 --2- 172.21.15.6:0/2187770781 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa8c40c9230 0x7fa8c4142ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:31.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.645+0000 7fa8cae00640 1 --2- 172.21.15.6:0/2187770781 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa8c4151bb0 0x7fa8c4150290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:31.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.645+0000 7fa8cae00640 1 --2- 172.21.15.6:0/2187770781 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa8c4071530 0x7fa8c4156710 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:31.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.645+0000 7fa8cae00640 1 -- 172.21.15.6:0/2187770781 >> 172.21.15.6:0/2187770781 conn(0x7fa8c406b0e0 msgr2=0x7fa8c406e230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:15:31.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.645+0000 7fa8cae00640 1 -- 172.21.15.6:0/2187770781 shutdown_connections 2024-04-24T23:15:31.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.645+0000 7fa8cae00640 1 -- 172.21.15.6:0/2187770781 wait complete. 2024-04-24T23:15:31.666 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:15:31.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.877+0000 7f9a90e00640 1 Processor -- start 2024-04-24T23:15:31.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.877+0000 7f9a90e00640 1 -- start start 2024-04-24T23:15:31.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.879+0000 7f9a90e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f9a8c0cc870 0x7f9a8c15b300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:31.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.879+0000 7f9a90e00640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f9a8c15dfe0 0x7f9a8c154840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:31.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.879+0000 7f9a90e00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9a8c15e550 con 0x7f9a8c0ccc50 2024-04-24T23:15:31.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.879+0000 7f9a90e00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f9a8c15e810 con 0x7f9a8c0cc870 2024-04-24T23:15:31.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.879+0000 7f9a90e00640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f9a8c15eaf0 con 0x7f9a8c15dfe0 2024-04-24T23:15:31.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.879+0000 7f9a8aa00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f9a8c0ccc50 0x7f9a8c06db80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:50290/0 (socket says 172.21.15.6:50290) 2024-04-24T23:15:31.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.880+0000 7f9a8aa00640 1 -- 172.21.15.6:0/3238029879 learned_addr learned my addr 172.21.15.6:0/3238029879 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:15:31.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.880+0000 7f9a8b400640 1 --2- 172.21.15.6:0/3238029879 >> v2:172.21.15.6:3300/0 conn(0x7f9a8c0cc870 0x7f9a8c15b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:31.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.880+0000 7f9a8be00640 1 --2- 172.21.15.6:0/3238029879 >> v2:172.21.15.190:3300/0 conn(0x7f9a8c15dfe0 0x7f9a8c154840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:31.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.880+0000 7f9a8b400640 1 -- 172.21.15.6:0/3238029879 >> v2:172.21.15.190:3300/0 conn(0x7f9a8c15dfe0 msgr2=0x7f9a8c154840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:15:31.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.880+0000 7f9a8b400640 1 --2- 172.21.15.6:0/3238029879 >> v2:172.21.15.190:3300/0 conn(0x7f9a8c15dfe0 0x7f9a8c154840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:31.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.880+0000 7f9a8b400640 1 -- 172.21.15.6:0/3238029879 >> v1:172.21.15.6:6789/0 conn(0x7f9a8c0ccc50 legacy=0x7f9a8c06db80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:31.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.880+0000 7f9a8b400640 1 -- 172.21.15.6:0/3238029879 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a8c15ab00 con 0x7f9a8c0cc870 2024-04-24T23:15:31.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.881+0000 7f9a8b400640 1 --2- 172.21.15.6:0/3238029879 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9a8c0cc870 0x7f9a8c15b300 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f9a8c06db10 tx=0x7f9a7c02a250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66bc5c7015c1ba00 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:15:31.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.883+0000 7f9a8a000640 1 -- 172.21.15.6:0/3238029879 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a7c037070 con 0x7f9a8c0cc870 2024-04-24T23:15:31.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.883+0000 7f9a8a000640 1 -- 172.21.15.6:0/3238029879 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a7c032440 con 0x7f9a8c0cc870 2024-04-24T23:15:31.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.883+0000 7f9a8a000640 1 -- 172.21.15.6:0/3238029879 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a7c032810 con 0x7f9a8c0cc870 2024-04-24T23:15:31.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.884+0000 7f9a90e00640 1 -- 172.21.15.6:0/3238029879 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9a8c0cc870 msgr2=0x7f9a8c15b300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:31.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.884+0000 7f9a90e00640 1 --2- 172.21.15.6:0/3238029879 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9a8c0cc870 0x7f9a8c15b300 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f9a8c06db10 tx=0x7f9a7c02a250 comp rx=0 tx=0).stop 2024-04-24T23:15:31.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.884+0000 7f9a90e00640 1 -- 172.21.15.6:0/3238029879 shutdown_connections 2024-04-24T23:15:31.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.884+0000 7f9a90e00640 1 --2- 172.21.15.6:0/3238029879 >> v2:172.21.15.190:3300/0 conn(0x7f9a8c15dfe0 0x7f9a8c154840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:31.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.884+0000 7f9a90e00640 1 --2- 172.21.15.6:0/3238029879 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9a8c0cc870 0x7f9a8c15b300 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:31.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.884+0000 7f9a90e00640 1 -- 172.21.15.6:0/3238029879 >> 172.21.15.6:0/3238029879 conn(0x7f9a8c06ed80 msgr2=0x7f9a8c06f190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:15:31.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.884+0000 7f9a90e00640 1 -- 172.21.15.6:0/3238029879 shutdown_connections 2024-04-24T23:15:31.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.885+0000 7f9a90e00640 1 -- 172.21.15.6:0/3238029879 wait complete. 2024-04-24T23:15:31.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.886+0000 7f9a90e00640 1 Processor -- start 2024-04-24T23:15:31.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.886+0000 7f9a90e00640 1 -- start start 2024-04-24T23:15:31.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.887+0000 7f9a90e00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9a8c0cc870 0x7f9a8c159d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:31.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.888+0000 7f9a90e00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9a8c0ccc50 0x7f9a8c146ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:31.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.888+0000 7f9a90e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9a8c15dfe0 0x7f9a8c147000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:31.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.888+0000 7f9a90e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f9a8c15edd0 con 0x7f9a8c15dfe0 2024-04-24T23:15:31.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.888+0000 7f9a90e00640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f9a8c0c6be0 con 0x7f9a8c0cc870 2024-04-24T23:15:31.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.888+0000 7f9a90e00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f9a8c06e290 con 0x7f9a8c0ccc50 2024-04-24T23:15:31.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.888+0000 7f9a8b400640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9a8c0cc870 0x7f9a8c159d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:31.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.888+0000 7f9a8b400640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9a8c0cc870 0x7f9a8c159d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:39546/0 (socket says 172.21.15.6:39546) 2024-04-24T23:15:31.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.888+0000 7f9a8b400640 1 -- 172.21.15.6:0/347509307 learned_addr learned my addr 172.21.15.6:0/347509307 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:15:31.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.888+0000 7f9a8aa00640 1 --2- 172.21.15.6:0/347509307 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9a8c0ccc50 0x7f9a8c146ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:31.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.888+0000 7f9a8be00640 1 --2- 172.21.15.6:0/347509307 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9a8c15dfe0 0x7f9a8c147000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:31.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.888+0000 7f9a8b400640 1 -- 172.21.15.6:0/347509307 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9a8c0ccc50 msgr2=0x7f9a8c146ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:31.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.888+0000 7f9a8b400640 1 --2- 172.21.15.6:0/347509307 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9a8c0ccc50 0x7f9a8c146ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:31.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.888+0000 7f9a8b400640 1 -- 172.21.15.6:0/347509307 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9a8c15dfe0 msgr2=0x7f9a8c147000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:31.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.888+0000 7f9a8b400640 1 --2- 172.21.15.6:0/347509307 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9a8c15dfe0 0x7f9a8c147000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:31.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.888+0000 7f9a8b400640 1 -- 172.21.15.6:0/347509307 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a8c1599f0 con 0x7f9a8c0cc870 2024-04-24T23:15:31.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.888+0000 7f9a8be00640 1 --2- 172.21.15.6:0/347509307 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9a8c15dfe0 0x7f9a8c147000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:15:31.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.889+0000 7f9a8b400640 1 --2- 172.21.15.6:0/347509307 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9a8c0cc870 0x7f9a8c159d40 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f9a7c000c00 tx=0x7f9a7c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:15:31.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.889+0000 7f9a83400640 1 -- 172.21.15.6:0/347509307 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a7c037040 con 0x7f9a8c0cc870 2024-04-24T23:15:31.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.889+0000 7f9a83400640 1 -- 172.21.15.6:0/347509307 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a7c042070 con 0x7f9a8c0cc870 2024-04-24T23:15:31.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.889+0000 7f9a90e00640 1 -- 172.21.15.6:0/347509307 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a8c159710 con 0x7f9a8c0cc870 2024-04-24T23:15:31.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.889+0000 7f9a83400640 1 -- 172.21.15.6:0/347509307 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a7c030960 con 0x7f9a8c0cc870 2024-04-24T23:15:31.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.889+0000 7f9a90e00640 1 -- 172.21.15.6:0/347509307 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a8c148ac0 con 0x7f9a8c0cc870 2024-04-24T23:15:31.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.891+0000 7f9a90e00640 1 -- 172.21.15.6:0/347509307 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a8c075b30 con 0x7f9a8c0cc870 2024-04-24T23:15:31.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.891+0000 7f9a83400640 1 -- 172.21.15.6:0/347509307 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f9a7c030b00 con 0x7f9a8c0cc870 2024-04-24T23:15:31.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.892+0000 7f9a83400640 1 --2- 172.21.15.6:0/347509307 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f9a6c06be30 0x7f9a6c06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:31.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.893+0000 7f9a83400640 1 -- 172.21.15.6:0/347509307 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7f9a7c0ab540 con 0x7f9a8c0cc870 2024-04-24T23:15:31.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.893+0000 7f9a8aa00640 1 --2- 172.21.15.6:0/347509307 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f9a6c06be30 0x7f9a6c06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:31.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.893+0000 7f9a8aa00640 1 --2- 172.21.15.6:0/347509307 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f9a6c06be30 0x7f9a6c06e2f0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f9a8c159500 tx=0x7f9a78007e80 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:15:31.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:31.898+0000 7f9a83400640 1 -- 172.21.15.6:0/347509307 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f9a7c07a090 con 0x7f9a8c0cc870 2024-04-24T23:15:32.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:32.139+0000 7f9a90e00640 1 -- 172.21.15.6:0/347509307 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9a8c0c8480 con 0x7f9a8c0cc870 2024-04-24T23:15:32.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:32.140+0000 7f9a83400640 1 -- 172.21.15.6:0/347509307 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+5503 (secure 0 0 0) 0x7f9a7c079990 con 0x7f9a8c0cc870 2024-04-24T23:15:32.142 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:15:32.142 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4197,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/105267896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":105267896},{"type":"v1","addr":"172.21.15.6:6834","nonce":105267896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4203,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/2590421094","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":2590421094},{"type":"v1","addr":"172.21.15.6:6837","nonce":2590421094}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4232,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6838/2032280158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2032280158},{"type":"v1","addr":"172.21.15.190:6838","nonce":2032280158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6839/1572872451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":1572872451},{"type":"v1","addr":"172.21.15.190:6839","nonce":1572872451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4282,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6837/580315269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":580315269},{"type":"v1","addr":"172.21.15.190:6837","nonce":580315269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4306,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/489092393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":489092393},{"type":"v1","addr":"172.21.15.6:6835","nonce":489092393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-04-24T23:15:32.142 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 3 2024-04-24T23:15:32.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:32.143+0000 7f9a90e00640 1 -- 172.21.15.6:0/347509307 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f9a6c06be30 msgr2=0x7f9a6c06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:32.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:32.143+0000 7f9a90e00640 1 --2- 172.21.15.6:0/347509307 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f9a6c06be30 0x7f9a6c06e2f0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f9a8c159500 tx=0x7f9a78007e80 comp rx=0 tx=0).stop 2024-04-24T23:15:32.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:32.143+0000 7f9a90e00640 1 -- 172.21.15.6:0/347509307 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9a8c0cc870 msgr2=0x7f9a8c159d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:32.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:32.143+0000 7f9a90e00640 1 --2- 172.21.15.6:0/347509307 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9a8c0cc870 0x7f9a8c159d40 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f9a7c000c00 tx=0x7f9a7c0046c0 comp rx=0 tx=0).stop 2024-04-24T23:15:32.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:32.143+0000 7f9a90e00640 1 -- 172.21.15.6:0/347509307 shutdown_connections 2024-04-24T23:15:32.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:32.143+0000 7f9a90e00640 1 --2- 172.21.15.6:0/347509307 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f9a6c06be30 0x7f9a6c06e2f0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:32.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:32.143+0000 7f9a90e00640 1 --2- 172.21.15.6:0/347509307 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9a8c0ccc50 0x7f9a8c146ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:32.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:32.143+0000 7f9a90e00640 1 --2- 172.21.15.6:0/347509307 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9a8c15dfe0 0x7f9a8c147000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:32.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:32.143+0000 7f9a90e00640 1 --2- 172.21.15.6:0/347509307 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9a8c0cc870 0x7f9a8c159d40 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:32.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:32.143+0000 7f9a90e00640 1 -- 172.21.15.6:0/347509307 >> 172.21.15.6:0/347509307 conn(0x7f9a8c06ed80 msgr2=0x7f9a8c154610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:15:32.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:32.143+0000 7f9a90e00640 1 -- 172.21.15.6:0/347509307 shutdown_connections 2024-04-24T23:15:32.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:32.143+0000 7f9a90e00640 1 -- 172.21.15.6:0/347509307 wait complete. 2024-04-24T23:15:32.163 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T23:15:32.253 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-24T23:15:32.258 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-24T23:15:32.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:32.354+0000 7f7903200640 1 Processor -- start 2024-04-24T23:15:32.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:32.354+0000 7f7903200640 1 -- start start 2024-04-24T23:15:32.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:32.355+0000 7f7903200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f7904072320 0x7f7904157650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:32.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:32.355+0000 7f7903200640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f79040c6340 0x7f790415b610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:32.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:32.355+0000 7f7903200640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7904071680 con 0x7f7904151be0 2024-04-24T23:15:32.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:32.355+0000 7f7903200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f790415abc0 con 0x7f7904072320 2024-04-24T23:15:32.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:32.355+0000 7f7903200640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f7904071960 con 0x7f79040c6340 2024-04-24T23:15:32.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:32.356+0000 7f78fbe00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f7904072320 0x7f7904157650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:32.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:32.356+0000 7f7901e00640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f7904151be0 0x7f79040cc600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.6:39650/0 (socket says 172.21.15.6:39650) 2024-04-24T23:15:32.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:32.356+0000 7f7902800640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f79040c6340 0x7f790415b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:32.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:32.356+0000 7f7901e00640 1 -- 172.21.15.6:0/1638041182 learned_addr learned my addr 172.21.15.6:0/1638041182 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:15:32.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:32.356+0000 7f78fbe00640 1 -- 172.21.15.6:0/1638041182 >> v2:172.21.15.190:3300/0 conn(0x7f79040c6340 msgr2=0x7f790415b610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:15:32.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:32.356+0000 7f78fbe00640 1 --2- 172.21.15.6:0/1638041182 >> v2:172.21.15.190:3300/0 conn(0x7f79040c6340 0x7f790415b610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:32.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:32.356+0000 7f78fbe00640 1 -- 172.21.15.6:0/1638041182 >> v1:172.21.15.190:6789/0 conn(0x7f7904151be0 legacy=0x7f79040cc600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:32.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:32.356+0000 7f78fbe00640 1 -- 172.21.15.6:0/1638041182 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f790415ae10 con 0x7f7904072320 2024-04-24T23:15:32.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:32.357+0000 7f78fbe00640 1 --2- 172.21.15.6:0/1638041182 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7904072320 0x7f7904157650 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f7904070e00 tx=0x7f78ec02a080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d347cb4bcfcc112e server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:15:32.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:32.357+0000 7f7901400640 1 -- 172.21.15.6:0/1638041182 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78ec037070 con 0x7f7904072320 2024-04-24T23:15:32.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:32.358+0000 7f7901400640 1 -- 172.21.15.6:0/1638041182 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f78ec02add0 con 0x7f7904072320 2024-04-24T23:15:32.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:32.358+0000 7f7901400640 1 -- 172.21.15.6:0/1638041182 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78ec032660 con 0x7f7904072320 2024-04-24T23:15:32.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:32.358+0000 7f7903200640 1 -- 172.21.15.6:0/1638041182 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7904072320 msgr2=0x7f7904157650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:32.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:32.358+0000 7f7903200640 1 --2- 172.21.15.6:0/1638041182 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7904072320 0x7f7904157650 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f7904070e00 tx=0x7f78ec02a080 comp rx=0 tx=0).stop 2024-04-24T23:15:32.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:32.358+0000 7f7903200640 1 -- 172.21.15.6:0/1638041182 shutdown_connections 2024-04-24T23:15:32.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:32.358+0000 7f7903200640 1 --2- 172.21.15.6:0/1638041182 >> v2:172.21.15.190:3300/0 conn(0x7f79040c6340 0x7f790415b610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:32.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:32.358+0000 7f7903200640 1 --2- 172.21.15.6:0/1638041182 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7904072320 0x7f7904157650 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:32.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:32.358+0000 7f7903200640 1 -- 172.21.15.6:0/1638041182 >> 172.21.15.6:0/1638041182 conn(0x7f790406ed80 msgr2=0x7f790406f190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:15:32.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:32.358+0000 7f7903200640 1 -- 172.21.15.6:0/1638041182 shutdown_connections 2024-04-24T23:15:32.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:32.358+0000 7f7903200640 1 -- 172.21.15.6:0/1638041182 wait complete. 2024-04-24T23:15:32.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:32.360+0000 7f7903200640 1 Processor -- start 2024-04-24T23:15:32.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:32.360+0000 7f7903200640 1 -- start start 2024-04-24T23:15:32.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:32.361+0000 7f7903200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7904072320 0x7f790414f470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:32.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:32.361+0000 7f7903200640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f79040c6340 0x7f79041587b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:32.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:32.361+0000 7f7903200640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7904151be0 0x7f79041e1920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:32.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:32.361+0000 7f7903200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f79040c36e0 con 0x7f7904072320 2024-04-24T23:15:32.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:32.361+0000 7f7903200640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f79040c3ab0 con 0x7f7904151be0 2024-04-24T23:15:32.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:32.361+0000 7f7903200640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f79040c3860 con 0x7f79040c6340 2024-04-24T23:15:32.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:32.361+0000 7f7901e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7904072320 0x7f790414f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:32.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:32.361+0000 7f7901e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7904072320 0x7f790414f470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:51738/0 (socket says 172.21.15.6:51738) 2024-04-24T23:15:32.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:32.361+0000 7f7901e00640 1 -- 172.21.15.6:0/3875361471 learned_addr learned my addr 172.21.15.6:0/3875361471 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:15:32.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:32.361+0000 7f78fbe00640 1 --2- 172.21.15.6:0/3875361471 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f79040c6340 0x7f79041587b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:32.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:32.361+0000 7f7902800640 1 --2- 172.21.15.6:0/3875361471 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7904151be0 0x7f79041e1920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:32.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:32.362+0000 7f7901e00640 1 -- 172.21.15.6:0/3875361471 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f79040c6340 msgr2=0x7f79041587b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:32.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:32.362+0000 7f7901e00640 1 --2- 172.21.15.6:0/3875361471 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f79040c6340 0x7f79041587b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:32.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:32.362+0000 7f7901e00640 1 -- 172.21.15.6:0/3875361471 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7904151be0 msgr2=0x7f79041e1920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:32.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:32.362+0000 7f7901e00640 1 --2- 172.21.15.6:0/3875361471 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7904151be0 0x7f79041e1920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:32.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:32.362+0000 7f7901e00640 1 -- 172.21.15.6:0/3875361471 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79041ece50 con 0x7f7904072320 2024-04-24T23:15:32.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:32.362+0000 7f7901e00640 1 --2- 172.21.15.6:0/3875361471 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7904072320 0x7f790414f470 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f78f400c3f0 tx=0x7f78f400c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:15:32.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:32.362+0000 7f78fa000640 1 -- 172.21.15.6:0/3875361471 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78f4017070 con 0x7f7904072320 2024-04-24T23:15:32.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:32.362+0000 7f78fa000640 1 -- 172.21.15.6:0/3875361471 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f78f4005ce0 con 0x7f7904072320 2024-04-24T23:15:32.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:32.362+0000 7f7903200640 1 -- 172.21.15.6:0/3875361471 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79041f0710 con 0x7f7904072320 2024-04-24T23:15:32.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:32.362+0000 7f78fa000640 1 -- 172.21.15.6:0/3875361471 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78f40072f0 con 0x7f7904072320 2024-04-24T23:15:32.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:32.363+0000 7f7903200640 1 -- 172.21.15.6:0/3875361471 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79041efc30 con 0x7f7904072320 2024-04-24T23:15:32.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:32.364+0000 7f7903200640 1 -- 172.21.15.6:0/3875361471 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7904159ae0 con 0x7f7904072320 2024-04-24T23:15:32.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:32.365+0000 7f78fa000640 1 -- 172.21.15.6:0/3875361471 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f78f401a070 con 0x7f7904072320 2024-04-24T23:15:32.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:32.366+0000 7f78fa000640 1 --2- 172.21.15.6:0/3875361471 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f78e006be60 0x7f78e006e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:32.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:32.366+0000 7f78fa000640 1 -- 172.21.15.6:0/3875361471 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7f78f4089ef0 con 0x7f7904072320 2024-04-24T23:15:32.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:32.367+0000 7f78fbe00640 1 --2- 172.21.15.6:0/3875361471 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f78e006be60 0x7f78e006e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:32.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:32.367+0000 7f78fbe00640 1 --2- 172.21.15.6:0/3875361471 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f78e006be60 0x7f78e006e320 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f78ec034000 tx=0x7f78ec02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:15:32.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:32.371+0000 7f78fa000640 1 -- 172.21.15.6:0/3875361471 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f78f4058470 con 0x7f7904072320 2024-04-24T23:15:32.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:32.574+0000 7f7903200640 1 -- 172.21.15.6:0/3875361471 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7904075b30 con 0x7f7904072320 2024-04-24T23:15:32.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:32.576+0000 7f78fa000640 1 -- 172.21.15.6:0/3875361471 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) ==== 74+0+12824 (secure 0 0 0) 0x7f78f405c770 con 0x7f7904072320 2024-04-24T23:15:32.578 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:15:32.579 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":16,"fsid":"18bef68d-5e6d-4ec5-862b-b6f206e2588d","created":"2024-04-24T23:15:12.535072+0000","modified":"2024-04-24T23:15:26.816031+0000","last_up_change":"2024-04-24T23:15:23.773223+0000","last_in_change":"2024-04-24T23:15:14.905328+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T23:15:24.747441+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3b7a212d-d498-462e-90bf-ed493c803a09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":3022569437},{"type":"v1","addr":"172.21.15.6:6809","nonce":3022569437}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":3022569437},{"type":"v1","addr":"172.21.15.6:6811","nonce":3022569437}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":3022569437},{"type":"v1","addr":"172.21.15.6:6815","nonce":3022569437}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":3022569437},{"type":"v1","addr":"172.21.15.6:6813","nonce":3022569437}]},"public_addr":"172.21.15.6:6809/3022569437","cluster_addr":"172.21.15.6:6811/3022569437","heartbeat_back_addr":"172.21.15.6:6815/3022569437","heartbeat_front_addr":"172.21.15.6:6813/3022569437","state":["exists","up"]},{"osd":1,"uuid":"7c97f5a2-1735-4775-9f15-eb50099b19d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":3687836888},{"type":"v1","addr":"172.21.15.6:6817","nonce":3687836888}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":3687836888},{"type":"v1","addr":"172.21.15.6:6819","nonce":3687836888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":3687836888},{"type":"v1","addr":"172.21.15.6:6823","nonce":3687836888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":3687836888},{"type":"v1","addr":"172.21.15.6:6821","nonce":3687836888}]},"public_addr":"172.21.15.6:6817/3687836888","cluster_addr":"172.21.15.6:6819/3687836888","heartbeat_back_addr":"172.21.15.6:6823/3687836888","heartbeat_front_addr":"172.21.15.6:6821/3687836888","state":["exists","up"]},{"osd":2,"uuid":"d86bbc18-438c-4ec2-8018-ba7dd78f790b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":1020213724},{"type":"v1","addr":"172.21.15.6:6801","nonce":1020213724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":1020213724},{"type":"v1","addr":"172.21.15.6:6803","nonce":1020213724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":1020213724},{"type":"v1","addr":"172.21.15.6:6807","nonce":1020213724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":1020213724},{"type":"v1","addr":"172.21.15.6:6805","nonce":1020213724}]},"public_addr":"172.21.15.6:6801/1020213724","cluster_addr":"172.21.15.6:6803/1020213724","heartbeat_back_addr":"172.21.15.6:6807/1020213724","heartbeat_front_addr":"172.21.15.6:6805/1020213724","state":["exists","up"]},{"osd":3,"uuid":"3754ae16-4248-40dd-95f9-0f59c0eb5426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6824","nonce":3163883468},{"type":"v1","addr":"172.21.15.6:6825","nonce":3163883468}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":3163883468},{"type":"v1","addr":"172.21.15.6:6827","nonce":3163883468}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":3163883468},{"type":"v1","addr":"172.21.15.6:6831","nonce":3163883468}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":3163883468},{"type":"v1","addr":"172.21.15.6:6829","nonce":3163883468}]},"public_addr":"172.21.15.6:6825/3163883468","cluster_addr":"172.21.15.6:6827/3163883468","heartbeat_back_addr":"172.21.15.6:6831/3163883468","heartbeat_front_addr":"172.21.15.6:6829/3163883468","state":["exists","up"]},{"osd":4,"uuid":"8b2913a0-d006-479c-84ed-5b21e0e6c392","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":1989500934},{"type":"v1","addr":"172.21.15.190:6809","nonce":1989500934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":1989500934},{"type":"v1","addr":"172.21.15.190:6811","nonce":1989500934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":1989500934},{"type":"v1","addr":"172.21.15.190:6815","nonce":1989500934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":1989500934},{"type":"v1","addr":"172.21.15.190:6813","nonce":1989500934}]},"public_addr":"172.21.15.190:6809/1989500934","cluster_addr":"172.21.15.190:6811/1989500934","heartbeat_back_addr":"172.21.15.190:6815/1989500934","heartbeat_front_addr":"172.21.15.190:6813/1989500934","state":["exists","up"]},{"osd":5,"uuid":"499b79d0-0585-4dae-ab69-196a551793da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":3687548412},{"type":"v1","addr":"172.21.15.190:6801","nonce":3687548412}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":3687548412},{"type":"v1","addr":"172.21.15.190:6803","nonce":3687548412}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":3687548412},{"type":"v1","addr":"172.21.15.190:6807","nonce":3687548412}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":3687548412},{"type":"v1","addr":"172.21.15.190:6805","nonce":3687548412}]},"public_addr":"172.21.15.190:6801/3687548412","cluster_addr":"172.21.15.190:6803/3687548412","heartbeat_back_addr":"172.21.15.190:6807/3687548412","heartbeat_front_addr":"172.21.15.190:6805/3687548412","state":["exists","up"]},{"osd":6,"uuid":"6ba6df3b-4a6b-4a89-a890-0f79f387d4d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":870571564},{"type":"v1","addr":"172.21.15.190:6817","nonce":870571564}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":870571564},{"type":"v1","addr":"172.21.15.190:6819","nonce":870571564}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":870571564},{"type":"v1","addr":"172.21.15.190:6823","nonce":870571564}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":870571564},{"type":"v1","addr":"172.21.15.190:6821","nonce":870571564}]},"public_addr":"172.21.15.190:6817/870571564","cluster_addr":"172.21.15.190:6819/870571564","heartbeat_back_addr":"172.21.15.190:6823/870571564","heartbeat_front_addr":"172.21.15.190:6821/870571564","state":["exists","up"]},{"osd":7,"uuid":"e9668b1c-a000-4332-86e8-4efd73b02409","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":123639197},{"type":"v1","addr":"172.21.15.190:6825","nonce":123639197}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":123639197},{"type":"v1","addr":"172.21.15.190:6827","nonce":123639197}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":123639197},{"type":"v1","addr":"172.21.15.190:6831","nonce":123639197}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":123639197},{"type":"v1","addr":"172.21.15.190:6829","nonce":123639197}]},"public_addr":"172.21.15.190:6825/123639197","cluster_addr":"172.21.15.190:6827/123639197","heartbeat_back_addr":"172.21.15.190:6831/123639197","heartbeat_front_addr":"172.21.15.190:6829/123639197","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.321646+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.363873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.466963+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.531989+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:19.990707+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.043543+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.032492+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.115004+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T23:15:32.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:32.579+0000 7f7903200640 1 -- 172.21.15.6:0/3875361471 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f78e006be60 msgr2=0x7f78e006e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:32.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:32.579+0000 7f7903200640 1 --2- 172.21.15.6:0/3875361471 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f78e006be60 0x7f78e006e320 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f78ec034000 tx=0x7f78ec02e000 comp rx=0 tx=0).stop 2024-04-24T23:15:32.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:32.579+0000 7f7903200640 1 -- 172.21.15.6:0/3875361471 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7904072320 msgr2=0x7f790414f470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:32.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:32.579+0000 7f7903200640 1 --2- 172.21.15.6:0/3875361471 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7904072320 0x7f790414f470 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f78f400c3f0 tx=0x7f78f400c8b0 comp rx=0 tx=0).stop 2024-04-24T23:15:32.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:32.579+0000 7f7903200640 1 -- 172.21.15.6:0/3875361471 shutdown_connections 2024-04-24T23:15:32.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:32.579+0000 7f7903200640 1 --2- 172.21.15.6:0/3875361471 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f78e006be60 0x7f78e006e320 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:32.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:32.579+0000 7f7903200640 1 --2- 172.21.15.6:0/3875361471 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f79040c6340 0x7f79041587b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:32.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:32.579+0000 7f7903200640 1 --2- 172.21.15.6:0/3875361471 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7904151be0 0x7f79041e1920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:32.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:32.579+0000 7f7903200640 1 --2- 172.21.15.6:0/3875361471 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7904072320 0x7f790414f470 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:32.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:32.579+0000 7f7903200640 1 -- 172.21.15.6:0/3875361471 >> 172.21.15.6:0/3875361471 conn(0x7f790406ed80 msgr2=0x7f79041490d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:15:32.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:32.579+0000 7f7903200640 1 -- 172.21.15.6:0/3875361471 shutdown_connections 2024-04-24T23:15:32.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:32.579+0000 7f7903200640 1 -- 172.21.15.6:0/3875361471 wait complete. 2024-04-24T23:15:32.601 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T23:15:24.747441+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-24T23:15:32.601 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-24T23:15:32.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:32.856+0000 7f49ff200640 1 Processor -- start 2024-04-24T23:15:32.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:32.856+0000 7f49ff200640 1 -- start start 2024-04-24T23:15:32.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:32.858+0000 7f49ff200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f4a000c5a60 0x7f4a000c6ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:32.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:32.858+0000 7f49ff200640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4a000c5e40 0x7f4a0015bd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:32.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:32.858+0000 7f49ff200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4a000cca80 con 0x7f4a000c8570 2024-04-24T23:15:32.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:32.858+0000 7f49ff200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f4a000ccd60 con 0x7f4a000c5a60 2024-04-24T23:15:32.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:32.858+0000 7f49ff200640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f4a0006eab0 con 0x7f4a000c5e40 2024-04-24T23:15:32.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:32.858+0000 7f49fde00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f4a000c5a60 0x7f4a000c6ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:32.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:32.858+0000 7f49fde00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f4a000c5a60 0x7f4a000c6ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:51744/0 (socket says 172.21.15.6:51744) 2024-04-24T23:15:32.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:32.858+0000 7f49fde00640 1 -- 172.21.15.6:0/1389440292 learned_addr learned my addr 172.21.15.6:0/1389440292 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:15:32.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:32.858+0000 7f49fe800640 1 --1- 172.21.15.6:0/1389440292 >> v1:172.21.15.6:6789/0 conn(0x7f4a000c8570 0x7f4a001524a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:50298/0 (socket says 172.21.15.6:50298) 2024-04-24T23:15:32.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:32.858+0000 7f49fd400640 1 --2- 172.21.15.6:0/1389440292 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4a000c5e40 0x7f4a0015bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:32.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:32.859+0000 7f49fde00640 1 -- 172.21.15.6:0/1389440292 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4a000c5e40 msgr2=0x7f4a0015bd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:15:32.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:32.859+0000 7f49fde00640 1 --2- 172.21.15.6:0/1389440292 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4a000c5e40 0x7f4a0015bd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:32.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:32.859+0000 7f49fde00640 1 -- 172.21.15.6:0/1389440292 >> v1:172.21.15.6:6789/0 conn(0x7f4a000c8570 legacy=0x7f4a001524a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:32.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:32.859+0000 7f49fde00640 1 -- 172.21.15.6:0/1389440292 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a0006ed90 con 0x7f4a000c5a60 2024-04-24T23:15:32.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:32.859+0000 7f49fca00640 1 -- 172.21.15.6:0/1389440292 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3682134815 0 0) 0x7f4a000cca80 con 0x7f4a000c8570 2024-04-24T23:15:32.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:32.860+0000 7f49fde00640 1 --2- 172.21.15.6:0/1389440292 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4a000c5a60 0x7f4a000c6ae0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f49f0009870 tx=0x7f49f002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38e459d4cbfab43a server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:15:32.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:32.860+0000 7f49fca00640 1 -- 172.21.15.6:0/1389440292 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49f0037070 con 0x7f4a000c5a60 2024-04-24T23:15:32.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:32.860+0000 7f49fca00640 1 -- 172.21.15.6:0/1389440292 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f49f0032440 con 0x7f4a000c5a60 2024-04-24T23:15:32.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:32.860+0000 7f49fca00640 1 -- 172.21.15.6:0/1389440292 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49f0032810 con 0x7f4a000c5a60 2024-04-24T23:15:32.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:32.861+0000 7f49ff200640 1 -- 172.21.15.6:0/1389440292 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4a000c5a60 msgr2=0x7f4a000c6ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:32.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:32.861+0000 7f49ff200640 1 --2- 172.21.15.6:0/1389440292 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4a000c5a60 0x7f4a000c6ae0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f49f0009870 tx=0x7f49f002a270 comp rx=0 tx=0).stop 2024-04-24T23:15:32.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:32.861+0000 7f49ff200640 1 -- 172.21.15.6:0/1389440292 shutdown_connections 2024-04-24T23:15:32.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:32.861+0000 7f49ff200640 1 --2- 172.21.15.6:0/1389440292 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4a000c5e40 0x7f4a0015bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:32.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:32.861+0000 7f49ff200640 1 --2- 172.21.15.6:0/1389440292 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4a000c5a60 0x7f4a000c6ae0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:32.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:32.861+0000 7f49ff200640 1 -- 172.21.15.6:0/1389440292 >> 172.21.15.6:0/1389440292 conn(0x7f4a0006cc40 msgr2=0x7f4a0006d050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:15:32.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:32.861+0000 7f49ff200640 1 -- 172.21.15.6:0/1389440292 shutdown_connections 2024-04-24T23:15:32.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:32.861+0000 7f49ff200640 1 -- 172.21.15.6:0/1389440292 wait complete. 2024-04-24T23:15:32.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:32.863+0000 7f49ff200640 1 Processor -- start 2024-04-24T23:15:32.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:32.863+0000 7f49ff200640 1 -- start start 2024-04-24T23:15:32.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:32.864+0000 7f49ff200640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4a000c5a60 0x7f4a001e1330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:32.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:32.864+0000 7f49ff200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4a000c5e40 0x7f4a001eff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:32.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:32.864+0000 7f49ff200640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4a000c8570 0x7f4a001e1f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:32.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:32.864+0000 7f49ff200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f4a0006f590 con 0x7f4a000c5e40 2024-04-24T23:15:32.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:32.864+0000 7f49ff200640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f4a00072730 con 0x7f4a000c5a60 2024-04-24T23:15:32.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:32.864+0000 7f49ff200640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f4a0014e4d0 con 0x7f4a000c8570 2024-04-24T23:15:32.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:32.864+0000 7f49fd400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4a000c5e40 0x7f4a001eff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:32.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:32.864+0000 7f49fd400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4a000c5e40 0x7f4a001eff40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:51748/0 (socket says 172.21.15.6:51748) 2024-04-24T23:15:32.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:32.864+0000 7f49fd400640 1 -- 172.21.15.6:0/3617547586 learned_addr learned my addr 172.21.15.6:0/3617547586 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:15:32.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:32.864+0000 7f49fde00640 1 --2- 172.21.15.6:0/3617547586 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4a000c5a60 0x7f4a001e1330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:32.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:32.864+0000 7f49fe800640 1 --2- 172.21.15.6:0/3617547586 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4a000c8570 0x7f4a001e1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:32.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:32.864+0000 7f49fd400640 1 -- 172.21.15.6:0/3617547586 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4a000c8570 msgr2=0x7f4a001e1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:32.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:32.864+0000 7f49fd400640 1 --2- 172.21.15.6:0/3617547586 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4a000c8570 0x7f4a001e1f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:32.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:32.864+0000 7f49fd400640 1 -- 172.21.15.6:0/3617547586 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4a000c5a60 msgr2=0x7f4a001e1330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:32.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:32.864+0000 7f49fd400640 1 --2- 172.21.15.6:0/3617547586 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4a000c5a60 0x7f4a001e1330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:32.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:32.864+0000 7f49fd400640 1 -- 172.21.15.6:0/3617547586 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a001e19c0 con 0x7f4a000c5e40 2024-04-24T23:15:32.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:32.865+0000 7f49fd400640 1 --2- 172.21.15.6:0/3617547586 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4a000c5e40 0x7f4a001eff40 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f49ec00e9b0 tx=0x7f49ec00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:15:32.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:32.865+0000 7f49f6000640 1 -- 172.21.15.6:0/3617547586 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49ec017070 con 0x7f4a000c5e40 2024-04-24T23:15:32.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:32.865+0000 7f49f6000640 1 -- 172.21.15.6:0/3617547586 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f49ec00c9a0 con 0x7f4a000c5e40 2024-04-24T23:15:32.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:32.865+0000 7f49f6000640 1 -- 172.21.15.6:0/3617547586 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49ec00cd90 con 0x7f4a000c5e40 2024-04-24T23:15:32.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:32.865+0000 7f49ff200640 1 -- 172.21.15.6:0/3617547586 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a001e3050 con 0x7f4a000c5e40 2024-04-24T23:15:32.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:32.865+0000 7f49ff200640 1 -- 172.21.15.6:0/3617547586 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a001e3610 con 0x7f4a000c5e40 2024-04-24T23:15:32.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:32.866+0000 7f49ff200640 1 -- 172.21.15.6:0/3617547586 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a00075b30 con 0x7f4a000c5e40 2024-04-24T23:15:32.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:32.868+0000 7f49f6000640 1 -- 172.21.15.6:0/3617547586 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f49ec01a070 con 0x7f4a000c5e40 2024-04-24T23:15:32.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:32.869+0000 7f49f6000640 1 --2- 172.21.15.6:0/3617547586 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f49e006be30 0x7f49e006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:32.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:32.869+0000 7f49f6000640 1 -- 172.21.15.6:0/3617547586 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7f49ec08adb0 con 0x7f4a000c5e40 2024-04-24T23:15:32.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:32.869+0000 7f49fde00640 1 --2- 172.21.15.6:0/3617547586 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f49e006be30 0x7f49e006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:32.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:32.870+0000 7f49fde00640 1 --2- 172.21.15.6:0/3617547586 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f49e006be30 0x7f49e006e2f0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f49f0034000 tx=0x7f49f002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:15:32.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:32.873+0000 7f49f6000640 1 -- 172.21.15.6:0/3617547586 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f49ec08f070 con 0x7f4a000c5e40 2024-04-24T23:15:33.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.077+0000 7f49ff200640 1 -- 172.21.15.6:0/3617547586 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f4a001f0480 con 0x7f4a000c5e40 2024-04-24T23:15:33.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.078+0000 7f49f6000640 1 -- 172.21.15.6:0/3617547586 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v16) ==== 93+0+10 (secure 0 0 0) 0x7f49ec059450 con 0x7f4a000c5e40 2024-04-24T23:15:33.080 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-04-24T23:15:33.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.080+0000 7f49ff200640 1 -- 172.21.15.6:0/3617547586 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f49e006be30 msgr2=0x7f49e006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:33.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.080+0000 7f49ff200640 1 --2- 172.21.15.6:0/3617547586 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f49e006be30 0x7f49e006e2f0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f49f0034000 tx=0x7f49f002e000 comp rx=0 tx=0).stop 2024-04-24T23:15:33.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.080+0000 7f49ff200640 1 -- 172.21.15.6:0/3617547586 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4a000c5e40 msgr2=0x7f4a001eff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:33.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.080+0000 7f49ff200640 1 --2- 172.21.15.6:0/3617547586 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4a000c5e40 0x7f4a001eff40 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f49ec00e9b0 tx=0x7f49ec00ee70 comp rx=0 tx=0).stop 2024-04-24T23:15:33.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.081+0000 7f49ff200640 1 -- 172.21.15.6:0/3617547586 shutdown_connections 2024-04-24T23:15:33.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.081+0000 7f49ff200640 1 --2- 172.21.15.6:0/3617547586 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f49e006be30 0x7f49e006e2f0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:33.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.081+0000 7f49ff200640 1 --2- 172.21.15.6:0/3617547586 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4a000c8570 0x7f4a001e1f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:33.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.081+0000 7f49ff200640 1 --2- 172.21.15.6:0/3617547586 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4a000c5e40 0x7f4a001eff40 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:33.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.081+0000 7f49ff200640 1 --2- 172.21.15.6:0/3617547586 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4a000c5a60 0x7f4a001e1330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:33.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.081+0000 7f49ff200640 1 -- 172.21.15.6:0/3617547586 >> 172.21.15.6:0/3617547586 conn(0x7f4a0006cc40 msgr2=0x7f4a001566b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:15:33.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.081+0000 7f49ff200640 1 -- 172.21.15.6:0/3617547586 shutdown_connections 2024-04-24T23:15:33.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.081+0000 7f49ff200640 1 -- 172.21.15.6:0/3617547586 wait complete. 2024-04-24T23:15:33.102 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-24T23:15:33.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.308+0000 7f9feb200640 1 Processor -- start 2024-04-24T23:15:33.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.308+0000 7f9feb200640 1 -- start start 2024-04-24T23:15:33.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.310+0000 7f9feb200640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f9fec06ce30 0x7f9fec0cc600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:33.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.310+0000 7f9feb200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f9fec0c6be0 0x7f9fec0ccb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:33.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.310+0000 7f9feb200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9fec06d390 con 0x7f9fec152bd0 2024-04-24T23:15:33.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.310+0000 7f9feb200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f9fec06d970 con 0x7f9fec0c6be0 2024-04-24T23:15:33.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.310+0000 7f9feb200640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f9fec06dbc0 con 0x7f9fec06ce30 2024-04-24T23:15:33.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.310+0000 7f9fe9e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f9fec152bd0 0x7f9fec158e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:50300/0 (socket says 172.21.15.6:50300) 2024-04-24T23:15:33.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.310+0000 7f9fe9e00640 1 -- 172.21.15.6:0/1337585737 learned_addr learned my addr 172.21.15.6:0/1337585737 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:15:33.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.310+0000 7f9fea800640 1 --2- 172.21.15.6:0/1337585737 >> v2:172.21.15.6:3300/0 conn(0x7f9fec0c6be0 0x7f9fec0ccb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:33.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.311+0000 7f9fe3e00640 1 --2- 172.21.15.6:0/1337585737 >> v2:172.21.15.190:3300/0 conn(0x7f9fec06ce30 0x7f9fec0cc600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:33.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.311+0000 7f9fe9400640 1 -- 172.21.15.6:0/1337585737 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 676435614 0 0) 0x7f9fec06d390 con 0x7f9fec152bd0 2024-04-24T23:15:33.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.311+0000 7f9fe9400640 1 -- 172.21.15.6:0/1337585737 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9fd0003520 con 0x7f9fec152bd0 2024-04-24T23:15:33.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.311+0000 7f9fea800640 1 -- 172.21.15.6:0/1337585737 >> v2:172.21.15.190:3300/0 conn(0x7f9fec06ce30 msgr2=0x7f9fec0cc600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:15:33.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.311+0000 7f9fea800640 1 --2- 172.21.15.6:0/1337585737 >> v2:172.21.15.190:3300/0 conn(0x7f9fec06ce30 0x7f9fec0cc600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:33.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.311+0000 7f9fea800640 1 -- 172.21.15.6:0/1337585737 >> v1:172.21.15.6:6789/0 conn(0x7f9fec152bd0 legacy=0x7f9fec158e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:33.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.311+0000 7f9fea800640 1 -- 172.21.15.6:0/1337585737 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9fec06dea0 con 0x7f9fec0c6be0 2024-04-24T23:15:33.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.312+0000 7f9fea800640 1 --2- 172.21.15.6:0/1337585737 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9fec0c6be0 0x7f9fec0ccb40 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f9fec070820 tx=0x7f9fe402a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=816b6fa5e33246a2 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:15:33.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.313+0000 7f9fe9400640 1 -- 172.21.15.6:0/1337585737 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fe4007740 con 0x7f9fec0c6be0 2024-04-24T23:15:33.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.313+0000 7f9fe9400640 1 -- 172.21.15.6:0/1337585737 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9fe40078e0 con 0x7f9fec0c6be0 2024-04-24T23:15:33.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.313+0000 7f9fe9400640 1 -- 172.21.15.6:0/1337585737 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fe4007cb0 con 0x7f9fec0c6be0 2024-04-24T23:15:33.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.313+0000 7f9feb200640 1 -- 172.21.15.6:0/1337585737 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9fec0c6be0 msgr2=0x7f9fec0ccb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:33.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.313+0000 7f9feb200640 1 --2- 172.21.15.6:0/1337585737 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9fec0c6be0 0x7f9fec0ccb40 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f9fec070820 tx=0x7f9fe402a680 comp rx=0 tx=0).stop 2024-04-24T23:15:33.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.314+0000 7f9feb200640 1 -- 172.21.15.6:0/1337585737 shutdown_connections 2024-04-24T23:15:33.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.314+0000 7f9feb200640 1 --2- 172.21.15.6:0/1337585737 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9fec0c6be0 0x7f9fec0ccb40 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:33.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.314+0000 7f9feb200640 1 --2- 172.21.15.6:0/1337585737 >> v2:172.21.15.190:3300/0 conn(0x7f9fec06ce30 0x7f9fec0cc600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:33.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.314+0000 7f9feb200640 1 -- 172.21.15.6:0/1337585737 >> 172.21.15.6:0/1337585737 conn(0x7f9fec06ed80 msgr2=0x7f9fec06f190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:15:33.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.314+0000 7f9feb200640 1 -- 172.21.15.6:0/1337585737 shutdown_connections 2024-04-24T23:15:33.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.314+0000 7f9feb200640 1 -- 172.21.15.6:0/1337585737 wait complete. 2024-04-24T23:15:33.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.315+0000 7f9feb200640 1 Processor -- start 2024-04-24T23:15:33.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.316+0000 7f9feb200640 1 -- start start 2024-04-24T23:15:33.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.317+0000 7f9feb200640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9fec06ce30 0x7f9fec15c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:33.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.317+0000 7f9feb200640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9fec0c6be0 0x7f9fec1f21a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:33.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.317+0000 7f9feb200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9fec152bd0 0x7f9fec1f26e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:33.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.317+0000 7f9feb200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f9fec072650 con 0x7f9fec152bd0 2024-04-24T23:15:33.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.317+0000 7f9feb200640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f9fec071890 con 0x7f9fec06ce30 2024-04-24T23:15:33.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.317+0000 7f9feb200640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f9fec0727d0 con 0x7f9fec0c6be0 2024-04-24T23:15:33.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.317+0000 7f9fe3e00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9fec0c6be0 0x7f9fec1f21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:33.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.317+0000 7f9fe9e00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9fec06ce30 0x7f9fec15c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:33.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.317+0000 7f9fea800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9fec152bd0 0x7f9fec1f26e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:33.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.317+0000 7f9fe3e00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9fec0c6be0 0x7f9fec1f21a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.6:49874/0 (socket says 172.21.15.6:49874) 2024-04-24T23:15:33.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.317+0000 7f9fe3e00640 1 -- 172.21.15.6:0/1031068580 learned_addr learned my addr 172.21.15.6:0/1031068580 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:15:33.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.318+0000 7f9fe3e00640 1 -- 172.21.15.6:0/1031068580 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9fec06ce30 msgr2=0x7f9fec15c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:33.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.318+0000 7f9fe3e00640 1 --2- 172.21.15.6:0/1031068580 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9fec06ce30 0x7f9fec15c7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:33.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.318+0000 7f9fe3e00640 1 -- 172.21.15.6:0/1031068580 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9fec152bd0 msgr2=0x7f9fec1f26e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:33.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.318+0000 7f9fe3e00640 1 --2- 172.21.15.6:0/1031068580 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9fec152bd0 0x7f9fec1f26e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:33.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.318+0000 7f9fe3e00640 1 -- 172.21.15.6:0/1031068580 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9fec0c40f0 con 0x7f9fec0c6be0 2024-04-24T23:15:33.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.318+0000 7f9fe3e00640 1 --2- 172.21.15.6:0/1031068580 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9fec0c6be0 0x7f9fec1f21a0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f9fd400e870 tx=0x7f9fd400ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:15:33.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.318+0000 7f9fe2000640 1 -- 172.21.15.6:0/1031068580 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fd4017070 con 0x7f9fec0c6be0 2024-04-24T23:15:33.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.318+0000 7f9fe2000640 1 -- 172.21.15.6:0/1031068580 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9fd400c940 con 0x7f9fec0c6be0 2024-04-24T23:15:33.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.318+0000 7f9fe2000640 1 -- 172.21.15.6:0/1031068580 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fd400cd30 con 0x7f9fec0c6be0 2024-04-24T23:15:33.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.318+0000 7f9feb200640 1 -- 172.21.15.6:0/1031068580 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9fec0c43b0 con 0x7f9fec0c6be0 2024-04-24T23:15:33.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.319+0000 7f9feb200640 1 -- 172.21.15.6:0/1031068580 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f9fec1f19a0 con 0x7f9fec0c6be0 2024-04-24T23:15:33.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.320+0000 7f9feb200640 1 -- 172.21.15.6:0/1031068580 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9fec074cb0 con 0x7f9fec0c6be0 2024-04-24T23:15:33.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.321+0000 7f9fe2000640 1 -- 172.21.15.6:0/1031068580 <== mon.2 v2:172.21.15.190:3301/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f9fd401a070 con 0x7f9fec0c6be0 2024-04-24T23:15:33.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.321+0000 7f9fe2000640 1 --2- 172.21.15.6:0/1031068580 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f9fc806be60 0x7f9fc806e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:33.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.322+0000 7f9fe2000640 1 -- 172.21.15.6:0/1031068580 <== mon.2 v2:172.21.15.190:3301/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7f9fd408acc0 con 0x7f9fec0c6be0 2024-04-24T23:15:33.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.322+0000 7f9fe9e00640 1 --2- 172.21.15.6:0/1031068580 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f9fc806be60 0x7f9fc806e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:33.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.322+0000 7f9fe9e00640 1 --2- 172.21.15.6:0/1031068580 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f9fc806be60 0x7f9fc806e320 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f9fdc005880 tx=0x7f9fdc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:15:33.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.327+0000 7f9fe2000640 1 -- 172.21.15.6:0/1031068580 <== mon.2 v2:172.21.15.190:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f9fec074cb0 con 0x7f9fec0c6be0 2024-04-24T23:15:33.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.530+0000 7f9feb200640 1 -- 172.21.15.6:0/1031068580 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f9fec075b30 con 0x7f9fec0c6be0 2024-04-24T23:15:33.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.531+0000 7f9fe2000640 1 -- 172.21.15.6:0/1031068580 <== mon.2 v2:172.21.15.190:3301/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v16) ==== 84+0+65 (secure 0 0 0) 0x7f9fd4091080 con 0x7f9fec0c6be0 2024-04-24T23:15:33.533 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:15:33.533 INFO:teuthology.orchestra.run.smithi006.stdout:[ 2024-04-24T23:15:33.533 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-24T23:15:33.533 INFO:teuthology.orchestra.run.smithi006.stdout: "poolnum": 1, 2024-04-24T23:15:33.533 INFO:teuthology.orchestra.run.smithi006.stdout: "poolname": ".mgr" 2024-04-24T23:15:33.533 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-24T23:15:33.533 INFO:teuthology.orchestra.run.smithi006.stdout:] 2024-04-24T23:15:33.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.533+0000 7f9feb200640 1 -- 172.21.15.6:0/1031068580 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f9fc806be60 msgr2=0x7f9fc806e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:33.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.533+0000 7f9feb200640 1 --2- 172.21.15.6:0/1031068580 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f9fc806be60 0x7f9fc806e320 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f9fdc005880 tx=0x7f9fdc009040 comp rx=0 tx=0).stop 2024-04-24T23:15:33.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.533+0000 7f9feb200640 1 -- 172.21.15.6:0/1031068580 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9fec0c6be0 msgr2=0x7f9fec1f21a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:33.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.533+0000 7f9feb200640 1 --2- 172.21.15.6:0/1031068580 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9fec0c6be0 0x7f9fec1f21a0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f9fd400e870 tx=0x7f9fd400ed30 comp rx=0 tx=0).stop 2024-04-24T23:15:33.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.534+0000 7f9feb200640 1 -- 172.21.15.6:0/1031068580 shutdown_connections 2024-04-24T23:15:33.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.534+0000 7f9feb200640 1 --2- 172.21.15.6:0/1031068580 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f9fc806be60 0x7f9fc806e320 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:33.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.534+0000 7f9feb200640 1 --2- 172.21.15.6:0/1031068580 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9fec0c6be0 0x7f9fec1f21a0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:33.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.534+0000 7f9feb200640 1 --2- 172.21.15.6:0/1031068580 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9fec152bd0 0x7f9fec1f26e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:33.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.534+0000 7f9feb200640 1 --2- 172.21.15.6:0/1031068580 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9fec06ce30 0x7f9fec15c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:33.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.534+0000 7f9feb200640 1 -- 172.21.15.6:0/1031068580 >> 172.21.15.6:0/1031068580 conn(0x7f9fec06ed80 msgr2=0x7f9fec0cc5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:15:33.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.534+0000 7f9feb200640 1 -- 172.21.15.6:0/1031068580 shutdown_connections 2024-04-24T23:15:33.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.534+0000 7f9feb200640 1 -- 172.21.15.6:0/1031068580 wait complete. 2024-04-24T23:15:33.555 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-24T23:15:33.556 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-24T23:15:33.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.774+0000 7ff4e1800640 1 Processor -- start 2024-04-24T23:15:33.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.774+0000 7ff4e1800640 1 -- start start 2024-04-24T23:15:33.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.776+0000 7ff4e1800640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7ff4dc0ccaa0 0x7ff4dc067820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:33.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.776+0000 7ff4e1800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ff4dc0cce80 0x7ff4dc0cad00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:33.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.776+0000 7ff4e1800640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff4dc0cb240 con 0x7ff4dc0c58a0 2024-04-24T23:15:33.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.776+0000 7ff4e1800640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7ff4dc0cbb70 con 0x7ff4dc0cce80 2024-04-24T23:15:33.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.776+0000 7ff4e1800640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7ff4dc0cc0a0 con 0x7ff4dc0ccaa0 2024-04-24T23:15:33.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.777+0000 7ff4db400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ff4dc0cce80 0x7ff4dc0cad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:33.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.777+0000 7ff4dbe00640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7ff4dc0ccaa0 0x7ff4dc067820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:33.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.777+0000 7ff4e0e00640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7ff4dc0c58a0 0x7ff4dc154780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.6:39660/0 (socket says 172.21.15.6:39660) 2024-04-24T23:15:33.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.777+0000 7ff4e0e00640 1 -- 172.21.15.6:0/2324190645 learned_addr learned my addr 172.21.15.6:0/2324190645 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:15:33.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.777+0000 7ff4dbe00640 1 -- 172.21.15.6:0/2324190645 >> v2:172.21.15.6:3300/0 conn(0x7ff4dc0cce80 msgr2=0x7ff4dc0cad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:15:33.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.777+0000 7ff4dbe00640 1 --2- 172.21.15.6:0/2324190645 >> v2:172.21.15.6:3300/0 conn(0x7ff4dc0cce80 0x7ff4dc0cad00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:33.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.777+0000 7ff4dbe00640 1 -- 172.21.15.6:0/2324190645 >> v1:172.21.15.190:6789/0 conn(0x7ff4dc0c58a0 legacy=0x7ff4dc154780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:33.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.777+0000 7ff4dbe00640 1 -- 172.21.15.6:0/2324190645 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4dc15b570 con 0x7ff4dc0ccaa0 2024-04-24T23:15:33.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.777+0000 7ff4daa00640 1 -- 172.21.15.6:0/2324190645 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2605445277 0 0) 0x7ff4dc0cb240 con 0x7ff4dc0c58a0 2024-04-24T23:15:33.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.778+0000 7ff4dbe00640 1 --2- 172.21.15.6:0/2324190645 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff4dc0ccaa0 0x7ff4dc067820 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7ff4dc0cda90 tx=0x7ff4cc02a1a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eec1bf3028da9aea server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:15:33.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.779+0000 7ff4daa00640 1 -- 172.21.15.6:0/2324190645 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4cc037070 con 0x7ff4dc0ccaa0 2024-04-24T23:15:33.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.779+0000 7ff4daa00640 1 -- 172.21.15.6:0/2324190645 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4cc032440 con 0x7ff4dc0ccaa0 2024-04-24T23:15:33.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.779+0000 7ff4daa00640 1 -- 172.21.15.6:0/2324190645 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4cc032810 con 0x7ff4dc0ccaa0 2024-04-24T23:15:33.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.779+0000 7ff4e1800640 1 -- 172.21.15.6:0/2324190645 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff4dc0ccaa0 msgr2=0x7ff4dc067820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:33.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.779+0000 7ff4e1800640 1 --2- 172.21.15.6:0/2324190645 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff4dc0ccaa0 0x7ff4dc067820 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7ff4dc0cda90 tx=0x7ff4cc02a1a0 comp rx=0 tx=0).stop 2024-04-24T23:15:33.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.779+0000 7ff4e1800640 1 -- 172.21.15.6:0/2324190645 shutdown_connections 2024-04-24T23:15:33.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.779+0000 7ff4e1800640 1 --2- 172.21.15.6:0/2324190645 >> v2:172.21.15.6:3300/0 conn(0x7ff4dc0cce80 0x7ff4dc0cad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:33.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.779+0000 7ff4e1800640 1 --2- 172.21.15.6:0/2324190645 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff4dc0ccaa0 0x7ff4dc067820 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:33.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.779+0000 7ff4e1800640 1 -- 172.21.15.6:0/2324190645 >> 172.21.15.6:0/2324190645 conn(0x7ff4dc06da30 msgr2=0x7ff4dc066ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:15:33.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.779+0000 7ff4e1800640 1 -- 172.21.15.6:0/2324190645 shutdown_connections 2024-04-24T23:15:33.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.779+0000 7ff4e1800640 1 -- 172.21.15.6:0/2324190645 wait complete. 2024-04-24T23:15:33.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.781+0000 7ff4e1800640 1 Processor -- start 2024-04-24T23:15:33.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.781+0000 7ff4e1800640 1 -- start start 2024-04-24T23:15:33.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.782+0000 7ff4e1800640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff4dc0c58a0 0x7ff4dc14bae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:33.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.782+0000 7ff4e1800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff4dc0ccaa0 0x7ff4dc0c7d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:33.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.782+0000 7ff4e1800640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff4dc0cce80 0x7ff4dc0c82c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:33.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.782+0000 7ff4e1800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7ff4dc0c5380 con 0x7ff4dc0ccaa0 2024-04-24T23:15:33.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.782+0000 7ff4e1800640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7ff4dc0cbdc0 con 0x7ff4dc0c58a0 2024-04-24T23:15:33.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.782+0000 7ff4e1800640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7ff4dc0cc4e0 con 0x7ff4dc0cce80 2024-04-24T23:15:33.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.783+0000 7ff4db400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff4dc0ccaa0 0x7ff4dc0c7d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:33.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.783+0000 7ff4dbe00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff4dc0c58a0 0x7ff4dc14bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:33.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.783+0000 7ff4e0e00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff4dc0cce80 0x7ff4dc0c82c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:33.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.783+0000 7ff4dbe00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff4dc0c58a0 0x7ff4dc14bae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:39616/0 (socket says 172.21.15.6:39616) 2024-04-24T23:15:33.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.783+0000 7ff4dbe00640 1 -- 172.21.15.6:0/3916639221 learned_addr learned my addr 172.21.15.6:0/3916639221 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:15:33.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.783+0000 7ff4dbe00640 1 -- 172.21.15.6:0/3916639221 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff4dc0cce80 msgr2=0x7ff4dc0c82c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:33.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.783+0000 7ff4dbe00640 1 --2- 172.21.15.6:0/3916639221 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff4dc0cce80 0x7ff4dc0c82c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:33.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.783+0000 7ff4dbe00640 1 -- 172.21.15.6:0/3916639221 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff4dc0ccaa0 msgr2=0x7ff4dc0c7d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:33.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.783+0000 7ff4dbe00640 1 --2- 172.21.15.6:0/3916639221 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff4dc0ccaa0 0x7ff4dc0c7d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:33.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.783+0000 7ff4dbe00640 1 -- 172.21.15.6:0/3916639221 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4dc14d4f0 con 0x7ff4dc0c58a0 2024-04-24T23:15:33.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.783+0000 7ff4db400640 1 --2- 172.21.15.6:0/3916639221 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff4dc0ccaa0 0x7ff4dc0c7d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:15:33.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.783+0000 7ff4dbe00640 1 --2- 172.21.15.6:0/3916639221 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff4dc0c58a0 0x7ff4dc14bae0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ff4cc02a790 tx=0x7ff4cc0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:15:33.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.784+0000 7ff4d3e00640 1 -- 172.21.15.6:0/3916639221 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4cc037040 con 0x7ff4dc0c58a0 2024-04-24T23:15:33.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.784+0000 7ff4d3e00640 1 -- 172.21.15.6:0/3916639221 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4cc042070 con 0x7ff4dc0c58a0 2024-04-24T23:15:33.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.784+0000 7ff4d3e00640 1 -- 172.21.15.6:0/3916639221 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4cc030960 con 0x7ff4dc0c58a0 2024-04-24T23:15:33.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.784+0000 7ff4e1800640 1 -- 172.21.15.6:0/3916639221 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4dc14d7d0 con 0x7ff4dc0c58a0 2024-04-24T23:15:33.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.784+0000 7ff4e1800640 1 -- 172.21.15.6:0/3916639221 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4dc14dd90 con 0x7ff4dc0c58a0 2024-04-24T23:15:33.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.785+0000 7ff4e1800640 1 -- 172.21.15.6:0/3916639221 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff4dc075b30 con 0x7ff4dc0c58a0 2024-04-24T23:15:33.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.786+0000 7ff4d3e00640 1 -- 172.21.15.6:0/3916639221 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7ff4cc030b00 con 0x7ff4dc0c58a0 2024-04-24T23:15:33.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.787+0000 7ff4d3e00640 1 --2- 172.21.15.6:0/3916639221 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7ff4bc06be30 0x7ff4bc06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:33.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.787+0000 7ff4d3e00640 1 -- 172.21.15.6:0/3916639221 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7ff4cc0ab3a0 con 0x7ff4dc0c58a0 2024-04-24T23:15:33.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.787+0000 7ff4db400640 1 --2- 172.21.15.6:0/3916639221 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7ff4bc06be30 0x7ff4bc06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:33.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.788+0000 7ff4db400640 1 --2- 172.21.15.6:0/3916639221 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7ff4bc06be30 0x7ff4bc06e2f0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7ff4dc06d160 tx=0x7ff4c8009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:15:33.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.792+0000 7ff4d3e00640 1 -- 172.21.15.6:0/3916639221 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7ff4cc0799b0 con 0x7ff4dc0c58a0 2024-04-24T23:15:33.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:33.994+0000 7ff4e1800640 1 -- 172.21.15.6:0/3916639221 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7ff4dc14df60 con 0x7ff4dc0c58a0 2024-04-24T23:15:34.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:34.926+0000 7ff4d3e00640 1 -- 172.21.15.6:0/3916639221 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v17) ==== 138+0+0 (secure 0 0 0) 0x7ff4cc07dcb0 con 0x7ff4dc0c58a0 2024-04-24T23:15:34.929 INFO:teuthology.orchestra.run.smithi006.stderr:pool 'cephfs_metadata' created 2024-04-24T23:15:34.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:34.935+0000 7ff4e1800640 1 -- 172.21.15.6:0/3916639221 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7ff4bc06be30 msgr2=0x7ff4bc06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:34.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:34.935+0000 7ff4e1800640 1 --2- 172.21.15.6:0/3916639221 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7ff4bc06be30 0x7ff4bc06e2f0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7ff4dc06d160 tx=0x7ff4c8009190 comp rx=0 tx=0).stop 2024-04-24T23:15:34.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:34.936+0000 7ff4e1800640 1 -- 172.21.15.6:0/3916639221 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff4dc0c58a0 msgr2=0x7ff4dc14bae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:34.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:34.936+0000 7ff4e1800640 1 --2- 172.21.15.6:0/3916639221 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff4dc0c58a0 0x7ff4dc14bae0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ff4cc02a790 tx=0x7ff4cc0046c0 comp rx=0 tx=0).stop 2024-04-24T23:15:34.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:34.937+0000 7ff4e1800640 1 -- 172.21.15.6:0/3916639221 shutdown_connections 2024-04-24T23:15:34.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:34.937+0000 7ff4e1800640 1 --2- 172.21.15.6:0/3916639221 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7ff4bc06be30 0x7ff4bc06e2f0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:34.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:34.937+0000 7ff4e1800640 1 --2- 172.21.15.6:0/3916639221 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff4dc0cce80 0x7ff4dc0c82c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:34.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:34.937+0000 7ff4e1800640 1 --2- 172.21.15.6:0/3916639221 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff4dc0ccaa0 0x7ff4dc0c7d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:34.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:34.937+0000 7ff4e1800640 1 --2- 172.21.15.6:0/3916639221 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff4dc0c58a0 0x7ff4dc14bae0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:34.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:34.937+0000 7ff4e1800640 1 -- 172.21.15.6:0/3916639221 >> 172.21.15.6:0/3916639221 conn(0x7ff4dc06da30 msgr2=0x7ff4dc0c5d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:15:34.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:34.937+0000 7ff4e1800640 1 -- 172.21.15.6:0/3916639221 shutdown_connections 2024-04-24T23:15:34.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:34.937+0000 7ff4e1800640 1 -- 172.21.15.6:0/3916639221 wait complete. 2024-04-24T23:15:34.963 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2024-04-24T23:15:35.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:35.154+0000 7fd1df200640 1 Processor -- start 2024-04-24T23:15:35.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:35.154+0000 7fd1df200640 1 -- start start 2024-04-24T23:15:35.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:35.155+0000 7fd1df200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd1d8159110 0x7fd1d8072420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:35.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:35.156+0000 7fd1df200640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fd1d80c7dc0 0x7fd1d81623a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:35.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:35.156+0000 7fd1df200640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd1d80c81c0 0x7fd1d81628e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:35.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:35.156+0000 7fd1df200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7fd1d81594f0 con 0x7fd1d8159110 2024-04-24T23:15:35.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:35.156+0000 7fd1df200640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7fd1d80c7230 con 0x7fd1d80c7dc0 2024-04-24T23:15:35.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:35.156+0000 7fd1df200640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fd1d80c74d0 con 0x7fd1d80c81c0 2024-04-24T23:15:35.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:35.156+0000 7fd1d7e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd1d8159110 0x7fd1d8072420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:35.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:35.156+0000 7fd1d7e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd1d8159110 0x7fd1d8072420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57488/0 (socket says 172.21.15.6:57488) 2024-04-24T23:15:35.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:35.156+0000 7fd1cfe00640 1 --2- 172.21.15.6:0/990424821 >> v2:172.21.15.190:3300/0 conn(0x7fd1d80c7dc0 0x7fd1d81623a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:35.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:35.156+0000 7fd1d7e00640 1 -- 172.21.15.6:0/990424821 learned_addr learned my addr 172.21.15.6:0/990424821 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:15:35.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:35.156+0000 7fd1dd200640 1 --2- 172.21.15.6:0/990424821 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd1d80c81c0 0x7fd1d81628e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:35.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:35.156+0000 7fd1d7e00640 1 -- 172.21.15.6:0/990424821 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd1d80c81c0 msgr2=0x7fd1d81628e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:15:35.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:35.156+0000 7fd1d7e00640 1 --2- 172.21.15.6:0/990424821 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd1d80c81c0 0x7fd1d81628e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:35.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:35.157+0000 7fd1d7e00640 1 -- 172.21.15.6:0/990424821 >> v2:172.21.15.190:3300/0 conn(0x7fd1d80c7dc0 msgr2=0x7fd1d81623a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:15:35.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:35.157+0000 7fd1d7e00640 1 --2- 172.21.15.6:0/990424821 >> v2:172.21.15.190:3300/0 conn(0x7fd1d80c7dc0 0x7fd1d81623a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:35.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:35.157+0000 7fd1d7e00640 1 -- 172.21.15.6:0/990424821 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1d806e440 con 0x7fd1d8159110 2024-04-24T23:15:35.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:35.157+0000 7fd1cfe00640 1 --2- 172.21.15.6:0/990424821 >> v2:172.21.15.190:3300/0 conn(0x7fd1d80c7dc0 0x7fd1d81623a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:15:35.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:35.158+0000 7fd1d7e00640 1 --2- 172.21.15.6:0/990424821 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd1d8159110 0x7fd1d8072420 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fd1d80cd3b0 tx=0x7fd1c802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6377efaa2050f36a server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:15:35.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:35.158+0000 7fd1d7400640 1 -- 172.21.15.6:0/990424821 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1c8037070 con 0x7fd1d8159110 2024-04-24T23:15:35.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:35.158+0000 7fd1d7400640 1 -- 172.21.15.6:0/990424821 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd1c8032440 con 0x7fd1d8159110 2024-04-24T23:15:35.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:35.158+0000 7fd1d7400640 1 -- 172.21.15.6:0/990424821 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1c8032810 con 0x7fd1d8159110 2024-04-24T23:15:35.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:35.159+0000 7fd1df200640 1 -- 172.21.15.6:0/990424821 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd1d8159110 msgr2=0x7fd1d8072420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:35.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:35.159+0000 7fd1df200640 1 --2- 172.21.15.6:0/990424821 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd1d8159110 0x7fd1d8072420 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fd1d80cd3b0 tx=0x7fd1c802a270 comp rx=0 tx=0).stop 2024-04-24T23:15:35.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:35.159+0000 7fd1df200640 1 -- 172.21.15.6:0/990424821 shutdown_connections 2024-04-24T23:15:35.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:35.159+0000 7fd1df200640 1 --2- 172.21.15.6:0/990424821 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd1d80c81c0 0x7fd1d81628e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:35.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:35.159+0000 7fd1df200640 1 --2- 172.21.15.6:0/990424821 >> v2:172.21.15.190:3300/0 conn(0x7fd1d80c7dc0 0x7fd1d81623a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:35.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:35.159+0000 7fd1df200640 1 --2- 172.21.15.6:0/990424821 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd1d8159110 0x7fd1d8072420 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:35.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:35.159+0000 7fd1df200640 1 -- 172.21.15.6:0/990424821 >> 172.21.15.6:0/990424821 conn(0x7fd1d806b0e0 msgr2=0x7fd1d806b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:15:35.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:35.159+0000 7fd1df200640 1 -- 172.21.15.6:0/990424821 shutdown_connections 2024-04-24T23:15:35.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:35.159+0000 7fd1df200640 1 -- 172.21.15.6:0/990424821 wait complete. 2024-04-24T23:15:35.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:35.161+0000 7fd1df200640 1 Processor -- start 2024-04-24T23:15:35.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:35.161+0000 7fd1df200640 1 -- start start 2024-04-24T23:15:35.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:35.162+0000 7fd1df200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd1d80c7dc0 0x7fd1d815ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:35.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:35.162+0000 7fd1df200640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd1d80c81c0 0x7fd1d81522a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:35.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:35.162+0000 7fd1df200640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd1d8159110 0x7fd1d81527e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:35.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:35.162+0000 7fd1df200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fd1d806ece0 con 0x7fd1d80c7dc0 2024-04-24T23:15:35.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:35.162+0000 7fd1df200640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fd1d806ea00 con 0x7fd1d80c81c0 2024-04-24T23:15:35.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:35.162+0000 7fd1df200640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fd1d8072960 con 0x7fd1d8159110 2024-04-24T23:15:35.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:35.162+0000 7fd1d7e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd1d80c7dc0 0x7fd1d815ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:35.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:35.162+0000 7fd1d7e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd1d80c7dc0 0x7fd1d815ded0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57492/0 (socket says 172.21.15.6:57492) 2024-04-24T23:15:35.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:35.162+0000 7fd1d7e00640 1 -- 172.21.15.6:0/2536640360 learned_addr learned my addr 172.21.15.6:0/2536640360 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:15:35.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:35.162+0000 7fd1cfe00640 1 --2- 172.21.15.6:0/2536640360 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd1d80c81c0 0x7fd1d81522a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:35.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:35.162+0000 7fd1dd200640 1 --2- 172.21.15.6:0/2536640360 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd1d8159110 0x7fd1d81527e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:35.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:35.163+0000 7fd1d7e00640 1 -- 172.21.15.6:0/2536640360 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd1d8159110 msgr2=0x7fd1d81527e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:35.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:35.163+0000 7fd1d7e00640 1 --2- 172.21.15.6:0/2536640360 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd1d8159110 0x7fd1d81527e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:35.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:35.163+0000 7fd1d7e00640 1 -- 172.21.15.6:0/2536640360 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd1d80c81c0 msgr2=0x7fd1d81522a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:35.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:35.163+0000 7fd1d7e00640 1 --2- 172.21.15.6:0/2536640360 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd1d80c81c0 0x7fd1d81522a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:35.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:35.163+0000 7fd1d7e00640 1 -- 172.21.15.6:0/2536640360 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1d8160660 con 0x7fd1d80c7dc0 2024-04-24T23:15:35.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:35.163+0000 7fd1dd200640 1 --2- 172.21.15.6:0/2536640360 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd1d8159110 0x7fd1d81527e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:15:35.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:35.163+0000 7fd1d7e00640 1 --2- 172.21.15.6:0/2536640360 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd1d80c7dc0 0x7fd1d815ded0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fd1c802a2b0 tx=0x7fd1c80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:15:35.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:35.163+0000 7fd1d4c00640 1 -- 172.21.15.6:0/2536640360 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1c8041020 con 0x7fd1d80c7dc0 2024-04-24T23:15:35.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:35.163+0000 7fd1d4c00640 1 -- 172.21.15.6:0/2536640360 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd1c803d620 con 0x7fd1d80c7dc0 2024-04-24T23:15:35.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:35.163+0000 7fd1d4c00640 1 -- 172.21.15.6:0/2536640360 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1c8037040 con 0x7fd1d80c7dc0 2024-04-24T23:15:35.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:35.163+0000 7fd1df200640 1 -- 172.21.15.6:0/2536640360 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1d81600a0 con 0x7fd1d80c7dc0 2024-04-24T23:15:35.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:35.163+0000 7fd1df200640 1 -- 172.21.15.6:0/2536640360 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1d815d6d0 con 0x7fd1d80c7dc0 2024-04-24T23:15:35.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:35.165+0000 7fd1df200640 1 -- 172.21.15.6:0/2536640360 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd1d80767b0 con 0x7fd1d80c7dc0 2024-04-24T23:15:35.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:35.166+0000 7fd1d4c00640 1 -- 172.21.15.6:0/2536640360 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fd1c8044020 con 0x7fd1d80c7dc0 2024-04-24T23:15:35.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:35.166+0000 7fd1d4c00640 1 --2- 172.21.15.6:0/2536640360 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fd1b406be30 0x7fd1b406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:35.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:35.167+0000 7fd1d4c00640 1 -- 172.21.15.6:0/2536640360 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 5227+0+0 (secure 0 0 0) 0x7fd1c80ab1c0 con 0x7fd1d80c7dc0 2024-04-24T23:15:35.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:35.167+0000 7fd1cfe00640 1 --2- 172.21.15.6:0/2536640360 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fd1b406be30 0x7fd1b406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:35.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:35.168+0000 7fd1cfe00640 1 --2- 172.21.15.6:0/2536640360 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fd1b406be30 0x7fd1b406e2f0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fd1d815c930 tx=0x7fd1c0009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:15:35.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:35.172+0000 7fd1d4c00640 1 -- 172.21.15.6:0/2536640360 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fd1c804b030 con 0x7fd1d80c7dc0 2024-04-24T23:15:35.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:35.379+0000 7fd1df200640 1 -- 172.21.15.6:0/2536640360 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) -- 0x7fd1d80c8f30 con 0x7fd1d80c7dc0 2024-04-24T23:15:35.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:35.925+0000 7fd1d4c00640 1 -- 172.21.15.6:0/2536640360 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v18) ==== 171+0+0 (secure 0 0 0) 0x7fd1c8079d50 con 0x7fd1d80c7dc0 2024-04-24T23:15:35.928 INFO:teuthology.orchestra.run.smithi006.stderr:pool 'cephfs_data' created 2024-04-24T23:15:35.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:35.929+0000 7fd1df200640 1 -- 172.21.15.6:0/2536640360 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fd1b406be30 msgr2=0x7fd1b406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:35.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:35.929+0000 7fd1df200640 1 --2- 172.21.15.6:0/2536640360 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fd1b406be30 0x7fd1b406e2f0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fd1d815c930 tx=0x7fd1c0009190 comp rx=0 tx=0).stop 2024-04-24T23:15:35.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:35.929+0000 7fd1df200640 1 -- 172.21.15.6:0/2536640360 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd1d80c7dc0 msgr2=0x7fd1d815ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:35.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:35.929+0000 7fd1df200640 1 --2- 172.21.15.6:0/2536640360 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd1d80c7dc0 0x7fd1d815ded0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fd1c802a2b0 tx=0x7fd1c80046c0 comp rx=0 tx=0).stop 2024-04-24T23:15:35.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:35.930+0000 7fd1df200640 1 -- 172.21.15.6:0/2536640360 shutdown_connections 2024-04-24T23:15:35.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:35.930+0000 7fd1df200640 1 --2- 172.21.15.6:0/2536640360 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fd1b406be30 0x7fd1b406e2f0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:35.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:35.930+0000 7fd1df200640 1 --2- 172.21.15.6:0/2536640360 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd1d8159110 0x7fd1d81527e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:35.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:35.930+0000 7fd1df200640 1 --2- 172.21.15.6:0/2536640360 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd1d80c81c0 0x7fd1d81522a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:35.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:35.930+0000 7fd1df200640 1 --2- 172.21.15.6:0/2536640360 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd1d80c7dc0 0x7fd1d815ded0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:35.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:35.930+0000 7fd1df200640 1 -- 172.21.15.6:0/2536640360 >> 172.21.15.6:0/2536640360 conn(0x7fd1d806b0e0 msgr2=0x7fd1d81533b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:15:35.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:35.931+0000 7fd1df200640 1 -- 172.21.15.6:0/2536640360 shutdown_connections 2024-04-24T23:15:35.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:35.931+0000 7fd1df200640 1 -- 172.21.15.6:0/2536640360 wait complete. 2024-04-24T23:15:35.969 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-24T23:15:36.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:36.147+0000 7f1f71400640 1 Processor -- start 2024-04-24T23:15:36.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:36.147+0000 7f1f71400640 1 -- start start 2024-04-24T23:15:36.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:36.149+0000 7f1f71400640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f1f6c154f60 0x7f1f6c06c640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:36.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:36.149+0000 7f1f71400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f1f6c06cb80 0x7f1f6c15b230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:36.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:36.149+0000 7f1f71400640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1f6c0b46f0 con 0x7f1f6c0b35f0 2024-04-24T23:15:36.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:36.149+0000 7f1f71400640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f1f6c0b49d0 con 0x7f1f6c06cb80 2024-04-24T23:15:36.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:36.149+0000 7f1f71400640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f1f6c0b4cb0 con 0x7f1f6c154f60 2024-04-24T23:15:36.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:36.149+0000 7f1f70a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f1f6c06cb80 0x7f1f6c15b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:36.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:36.149+0000 7f1f70a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f1f6c06cb80 0x7f1f6c15b230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57496/0 (socket says 172.21.15.6:57496) 2024-04-24T23:15:36.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:36.149+0000 7f1f6b400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f1f6c0b35f0 0x7f1f6c15ab20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:55094/0 (socket says 172.21.15.6:55094) 2024-04-24T23:15:36.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:36.149+0000 7f1f70a00640 1 -- 172.21.15.6:0/1410982134 learned_addr learned my addr 172.21.15.6:0/1410982134 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:15:36.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:36.149+0000 7f1f6be00640 1 --2- 172.21.15.6:0/1410982134 >> v2:172.21.15.190:3300/0 conn(0x7f1f6c154f60 0x7f1f6c06c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:36.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:36.150+0000 7f1f70a00640 1 -- 172.21.15.6:0/1410982134 >> v2:172.21.15.190:3300/0 conn(0x7f1f6c154f60 msgr2=0x7f1f6c06c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:15:36.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:36.150+0000 7f1f70a00640 1 --2- 172.21.15.6:0/1410982134 >> v2:172.21.15.190:3300/0 conn(0x7f1f6c154f60 0x7f1f6c06c640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:36.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:36.150+0000 7f1f70a00640 1 -- 172.21.15.6:0/1410982134 >> v1:172.21.15.6:6789/0 conn(0x7f1f6c0b35f0 legacy=0x7f1f6c15ab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:36.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:36.150+0000 7f1f70a00640 1 -- 172.21.15.6:0/1410982134 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f6c0b4f90 con 0x7f1f6c06cb80 2024-04-24T23:15:36.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:36.150+0000 7f1f6aa00640 1 -- 172.21.15.6:0/1410982134 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3821602125 0 0) 0x7f1f6c0b46f0 con 0x7f1f6c0b35f0 2024-04-24T23:15:36.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:36.151+0000 7f1f70a00640 1 --2- 172.21.15.6:0/1410982134 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1f6c06cb80 0x7f1f6c15b230 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f1f6c0b3090 tx=0x7f1f6402a7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c69d3f87e1126c8 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:15:36.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:36.151+0000 7f1f6aa00640 1 -- 172.21.15.6:0/1410982134 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f64036070 con 0x7f1f6c06cb80 2024-04-24T23:15:36.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:36.151+0000 7f1f6aa00640 1 -- 172.21.15.6:0/1410982134 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1f64031600 con 0x7f1f6c06cb80 2024-04-24T23:15:36.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:36.151+0000 7f1f6aa00640 1 -- 172.21.15.6:0/1410982134 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f640319f0 con 0x7f1f6c06cb80 2024-04-24T23:15:36.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:36.152+0000 7f1f71400640 1 -- 172.21.15.6:0/1410982134 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1f6c06cb80 msgr2=0x7f1f6c15b230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:36.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:36.152+0000 7f1f71400640 1 --2- 172.21.15.6:0/1410982134 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1f6c06cb80 0x7f1f6c15b230 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f1f6c0b3090 tx=0x7f1f6402a7c0 comp rx=0 tx=0).stop 2024-04-24T23:15:36.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:36.152+0000 7f1f71400640 1 -- 172.21.15.6:0/1410982134 shutdown_connections 2024-04-24T23:15:36.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:36.152+0000 7f1f71400640 1 --2- 172.21.15.6:0/1410982134 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1f6c06cb80 0x7f1f6c15b230 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:36.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:36.152+0000 7f1f71400640 1 --2- 172.21.15.6:0/1410982134 >> v2:172.21.15.190:3300/0 conn(0x7f1f6c154f60 0x7f1f6c06c640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:36.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:36.152+0000 7f1f71400640 1 -- 172.21.15.6:0/1410982134 >> 172.21.15.6:0/1410982134 conn(0x7f1f6c0674d0 msgr2=0x7f1f6c0d0320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:15:36.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:36.152+0000 7f1f71400640 1 -- 172.21.15.6:0/1410982134 shutdown_connections 2024-04-24T23:15:36.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:36.152+0000 7f1f71400640 1 -- 172.21.15.6:0/1410982134 wait complete. 2024-04-24T23:15:36.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:36.154+0000 7f1f71400640 1 Processor -- start 2024-04-24T23:15:36.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:36.154+0000 7f1f71400640 1 -- start start 2024-04-24T23:15:36.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:36.155+0000 7f1f71400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1f6c06cb80 0x7f1f6c14f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:36.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:36.155+0000 7f1f71400640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1f6c0b35f0 0x7f1f6c151460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:36.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:36.155+0000 7f1f71400640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1f6c154f60 0x7f1f6c0c80d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:36.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:36.155+0000 7f1f71400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f1f6c06cf90 con 0x7f1f6c06cb80 2024-04-24T23:15:36.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:36.155+0000 7f1f71400640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f1f6c06c120 con 0x7f1f6c0b35f0 2024-04-24T23:15:36.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:36.155+0000 7f1f71400640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f1f6c0b3240 con 0x7f1f6c154f60 2024-04-24T23:15:36.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:36.155+0000 7f1f6be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1f6c06cb80 0x7f1f6c14f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:36.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:36.155+0000 7f1f6be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1f6c06cb80 0x7f1f6c14f6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57500/0 (socket says 172.21.15.6:57500) 2024-04-24T23:15:36.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:36.155+0000 7f1f6be00640 1 -- 172.21.15.6:0/1371008783 learned_addr learned my addr 172.21.15.6:0/1371008783 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:15:36.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:36.155+0000 7f1f70a00640 1 --2- 172.21.15.6:0/1371008783 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1f6c154f60 0x7f1f6c0c80d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:36.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:36.155+0000 7f1f6b400640 1 --2- 172.21.15.6:0/1371008783 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1f6c0b35f0 0x7f1f6c151460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:36.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:36.155+0000 7f1f6be00640 1 -- 172.21.15.6:0/1371008783 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1f6c154f60 msgr2=0x7f1f6c0c80d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:36.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:36.155+0000 7f1f6be00640 1 --2- 172.21.15.6:0/1371008783 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1f6c154f60 0x7f1f6c0c80d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:36.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:36.155+0000 7f1f6be00640 1 -- 172.21.15.6:0/1371008783 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1f6c0b35f0 msgr2=0x7f1f6c151460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:36.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:36.155+0000 7f1f6be00640 1 --2- 172.21.15.6:0/1371008783 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1f6c0b35f0 0x7f1f6c151460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:36.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:36.155+0000 7f1f6be00640 1 -- 172.21.15.6:0/1371008783 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f6c14d7b0 con 0x7f1f6c06cb80 2024-04-24T23:15:36.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:36.156+0000 7f1f6be00640 1 --2- 172.21.15.6:0/1371008783 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1f6c06cb80 0x7f1f6c14f6b0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f1f5c00e8a0 tx=0x7f1f5c00ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:15:36.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:36.156+0000 7f1f63e00640 1 -- 172.21.15.6:0/1371008783 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f5c017070 con 0x7f1f6c06cb80 2024-04-24T23:15:36.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:36.156+0000 7f1f63e00640 1 -- 172.21.15.6:0/1371008783 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1f5c00c940 con 0x7f1f6c06cb80 2024-04-24T23:15:36.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:36.156+0000 7f1f63e00640 1 -- 172.21.15.6:0/1371008783 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f5c00cd30 con 0x7f1f6c06cb80 2024-04-24T23:15:36.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:36.156+0000 7f1f71400640 1 -- 172.21.15.6:0/1371008783 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f6c14e740 con 0x7f1f6c06cb80 2024-04-24T23:15:36.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:36.156+0000 7f1f71400640 1 -- 172.21.15.6:0/1371008783 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f6c0c78d0 con 0x7f1f6c06cb80 2024-04-24T23:15:36.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:36.157+0000 7f1f71400640 1 -- 172.21.15.6:0/1371008783 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1f6c068740 con 0x7f1f6c06cb80 2024-04-24T23:15:36.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:36.158+0000 7f1f63e00640 1 -- 172.21.15.6:0/1371008783 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f1f5c01a070 con 0x7f1f6c06cb80 2024-04-24T23:15:36.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:36.159+0000 7f1f63e00640 1 --2- 172.21.15.6:0/1371008783 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f1f4806be30 0x7f1f4806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:36.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:36.160+0000 7f1f63e00640 1 -- 172.21.15.6:0/1371008783 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 5609+0+0 (secure 0 0 0) 0x7f1f5c08b090 con 0x7f1f6c06cb80 2024-04-24T23:15:36.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:36.160+0000 7f1f6b400640 1 --2- 172.21.15.6:0/1371008783 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f1f4806be30 0x7f1f4806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:36.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:36.161+0000 7f1f6b400640 1 --2- 172.21.15.6:0/1371008783 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f1f4806be30 0x7f1f4806e2f0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f1f6c06c6f0 tx=0x7f1f54007d50 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:15:36.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:36.165+0000 7f1f63e00640 1 -- 172.21.15.6:0/1371008783 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f1f6c068740 con 0x7f1f6c06cb80 2024-04-24T23:15:36.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:36.398+0000 7f1f71400640 1 -- 172.21.15.6:0/1371008783 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f1f6c075b30 con 0x7f1f6c06cb80 2024-04-24T23:15:36.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:36.964+0000 7f1f63e00640 1 -- 172.21.15.6:0/1371008783 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v4) ==== 172+0+0 (secure 0 0 0) 0x7f1f5c059320 con 0x7f1f6c06cb80 2024-04-24T23:15:36.974 INFO:teuthology.orchestra.run.smithi006.stderr:new fs with metadata pool 2 and data pool 3 2024-04-24T23:15:36.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:36.977+0000 7f1f71400640 1 -- 172.21.15.6:0/1371008783 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f1f4806be30 msgr2=0x7f1f4806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:36.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:36.977+0000 7f1f71400640 1 --2- 172.21.15.6:0/1371008783 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f1f4806be30 0x7f1f4806e2f0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f1f6c06c6f0 tx=0x7f1f54007d50 comp rx=0 tx=0).stop 2024-04-24T23:15:36.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:36.977+0000 7f1f71400640 1 -- 172.21.15.6:0/1371008783 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1f6c06cb80 msgr2=0x7f1f6c14f6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:36.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:36.977+0000 7f1f71400640 1 --2- 172.21.15.6:0/1371008783 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1f6c06cb80 0x7f1f6c14f6b0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f1f5c00e8a0 tx=0x7f1f5c00ed60 comp rx=0 tx=0).stop 2024-04-24T23:15:36.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:36.977+0000 7f1f71400640 1 -- 172.21.15.6:0/1371008783 shutdown_connections 2024-04-24T23:15:36.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:36.977+0000 7f1f71400640 1 --2- 172.21.15.6:0/1371008783 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f1f4806be30 0x7f1f4806e2f0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:36.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:36.977+0000 7f1f71400640 1 --2- 172.21.15.6:0/1371008783 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1f6c154f60 0x7f1f6c0c80d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:36.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:36.977+0000 7f1f71400640 1 --2- 172.21.15.6:0/1371008783 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1f6c0b35f0 0x7f1f6c151460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:36.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:36.977+0000 7f1f71400640 1 --2- 172.21.15.6:0/1371008783 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1f6c06cb80 0x7f1f6c14f6b0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:36.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:36.977+0000 7f1f71400640 1 -- 172.21.15.6:0/1371008783 >> 172.21.15.6:0/1371008783 conn(0x7f1f6c0674d0 msgr2=0x7f1f6c14cb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:15:36.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:36.978+0000 7f1f71400640 1 -- 172.21.15.6:0/1371008783 shutdown_connections 2024-04-24T23:15:36.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:36.979+0000 7f1f71400640 1 -- 172.21.15.6:0/1371008783 wait complete. 2024-04-24T23:15:37.015 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2024-04-24T23:15:37.015 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2024-04-24T23:15:37.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:37.194+0000 7f7e0b200640 1 Processor -- start 2024-04-24T23:15:37.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:37.195+0000 7f7e0b200640 1 -- start start 2024-04-24T23:15:37.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:37.196+0000 7f7e0b200640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f7e04154c80 0x7f7e040c6be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:37.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:37.196+0000 7f7e0b200640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7e04070e50 0x7f7e0415e120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:37.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:37.196+0000 7f7e0b200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e0406f5e0 con 0x7f7e0415e660 2024-04-24T23:15:37.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:37.196+0000 7f7e0b200640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f7e04072320 con 0x7f7e04154c80 2024-04-24T23:15:37.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:37.196+0000 7f7e0b200640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f7e040727b0 con 0x7f7e04070e50 2024-04-24T23:15:37.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:37.196+0000 7f7e09e00640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f7e04154c80 0x7f7e040c6be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:37.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:37.196+0000 7f7e09400640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7e04070e50 0x7f7e0415e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:37.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:37.197+0000 7f7e09e00640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f7e04154c80 0x7f7e040c6be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:50230/0 (socket says 172.21.15.6:50230) 2024-04-24T23:15:37.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:37.197+0000 7f7e09e00640 1 -- 172.21.15.6:0/202140238 learned_addr learned my addr 172.21.15.6:0/202140238 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:15:37.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:37.197+0000 7f7e0a800640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f7e0415e660 0x7f7e040cc130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:55096/0 (socket says 172.21.15.6:55096) 2024-04-24T23:15:37.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:37.197+0000 7f7e09400640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7e04070e50 0x7f7e0415e120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.6:43144/0 (socket says 172.21.15.6:43144) 2024-04-24T23:15:37.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:37.197+0000 7f7e09e00640 1 -- 172.21.15.6:0/202140238 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7e04070e50 msgr2=0x7f7e0415e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:15:37.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:37.197+0000 7f7e09e00640 1 --2- 172.21.15.6:0/202140238 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7e04070e50 0x7f7e0415e120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:37.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:37.197+0000 7f7e09e00640 1 -- 172.21.15.6:0/202140238 >> v1:172.21.15.6:6789/0 conn(0x7f7e0415e660 legacy=0x7f7e040cc130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:37.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:37.197+0000 7f7e09e00640 1 -- 172.21.15.6:0/202140238 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e040c6020 con 0x7f7e04154c80 2024-04-24T23:15:37.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:37.198+0000 7f7e09e00640 1 --2- 172.21.15.6:0/202140238 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7e04154c80 0x7f7e040c6be0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f7df8009870 tx=0x7f7df802a270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a3651f277a2eecb6 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:15:37.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:37.198+0000 7f7e08a00640 1 -- 172.21.15.6:0/202140238 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7df8037070 con 0x7f7e04154c80 2024-04-24T23:15:37.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:37.199+0000 7f7e08a00640 1 -- 172.21.15.6:0/202140238 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7df8032440 con 0x7f7e04154c80 2024-04-24T23:15:37.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:37.199+0000 7f7e08a00640 1 -- 172.21.15.6:0/202140238 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7df8032810 con 0x7f7e04154c80 2024-04-24T23:15:37.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:37.199+0000 7f7e0b200640 1 -- 172.21.15.6:0/202140238 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7e04154c80 msgr2=0x7f7e040c6be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:37.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:37.199+0000 7f7e0b200640 1 --2- 172.21.15.6:0/202140238 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7e04154c80 0x7f7e040c6be0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f7df8009870 tx=0x7f7df802a270 comp rx=0 tx=0).stop 2024-04-24T23:15:37.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:37.199+0000 7f7e0b200640 1 -- 172.21.15.6:0/202140238 shutdown_connections 2024-04-24T23:15:37.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:37.199+0000 7f7e0b200640 1 --2- 172.21.15.6:0/202140238 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7e04070e50 0x7f7e0415e120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:37.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:37.199+0000 7f7e0b200640 1 --2- 172.21.15.6:0/202140238 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7e04154c80 0x7f7e040c6be0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:37.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:37.199+0000 7f7e0b200640 1 -- 172.21.15.6:0/202140238 >> 172.21.15.6:0/202140238 conn(0x7f7e0406cfe0 msgr2=0x7f7e0406ea40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:15:37.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:37.199+0000 7f7e0b200640 1 -- 172.21.15.6:0/202140238 shutdown_connections 2024-04-24T23:15:37.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:37.199+0000 7f7e0b200640 1 -- 172.21.15.6:0/202140238 wait complete. 2024-04-24T23:15:37.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:37.201+0000 7f7e0b200640 1 Processor -- start 2024-04-24T23:15:37.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:37.201+0000 7f7e0b200640 1 -- start start 2024-04-24T23:15:37.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:37.202+0000 7f7e0b200640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7e04070e50 0x7f7e04151fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:37.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:37.202+0000 7f7e0b200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7e04154c80 0x7f7e0414e730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:37.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:37.202+0000 7f7e0b200640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7e0415e660 0x7f7e0414ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:37.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:37.202+0000 7f7e0b200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f7e040c6560 con 0x7f7e04154c80 2024-04-24T23:15:37.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:37.202+0000 7f7e0b200640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f7e040c62c0 con 0x7f7e0415e660 2024-04-24T23:15:37.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:37.202+0000 7f7e0b200640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f7e04071fb0 con 0x7f7e04070e50 2024-04-24T23:15:37.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:37.202+0000 7f7e09400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7e04154c80 0x7f7e0414e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:37.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:37.202+0000 7f7e09e00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7e04070e50 0x7f7e04151fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:37.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:37.202+0000 7f7e0a800640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7e0415e660 0x7f7e0414ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:37.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:37.202+0000 7f7e09400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7e04154c80 0x7f7e0414e730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57514/0 (socket says 172.21.15.6:57514) 2024-04-24T23:15:37.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:37.202+0000 7f7e09400640 1 -- 172.21.15.6:0/2424377552 learned_addr learned my addr 172.21.15.6:0/2424377552 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:15:37.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:37.203+0000 7f7e09e00640 1 -- 172.21.15.6:0/2424377552 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7e0415e660 msgr2=0x7f7e0414ec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:37.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:37.203+0000 7f7e09e00640 1 --2- 172.21.15.6:0/2424377552 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7e0415e660 0x7f7e0414ec70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:37.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:37.203+0000 7f7e09e00640 1 -- 172.21.15.6:0/2424377552 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7e04154c80 msgr2=0x7f7e0414e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:37.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:37.203+0000 7f7e09e00640 1 --2- 172.21.15.6:0/2424377552 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7e04154c80 0x7f7e0414e730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:37.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:37.203+0000 7f7e09e00640 1 -- 172.21.15.6:0/2424377552 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e040c2dd0 con 0x7f7e04070e50 2024-04-24T23:15:37.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:37.203+0000 7f7e09400640 1 --2- 172.21.15.6:0/2424377552 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7e04154c80 0x7f7e0414e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:15:37.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:37.203+0000 7f7e09e00640 1 --2- 172.21.15.6:0/2424377552 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7e04070e50 0x7f7e04151fc0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f7e0406cd90 tx=0x7f7df80046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:15:37.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:37.203+0000 7f7dfe000640 1 -- 172.21.15.6:0/2424377552 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7df8041020 con 0x7f7e04070e50 2024-04-24T23:15:37.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:37.203+0000 7f7dfe000640 1 -- 172.21.15.6:0/2424377552 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7df8030660 con 0x7f7e04070e50 2024-04-24T23:15:37.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:37.203+0000 7f7e0b200640 1 -- 172.21.15.6:0/2424377552 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e041f0980 con 0x7f7e04070e50 2024-04-24T23:15:37.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:37.203+0000 7f7dfe000640 1 -- 172.21.15.6:0/2424377552 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7df8037040 con 0x7f7e04070e50 2024-04-24T23:15:37.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:37.204+0000 7f7e0b200640 1 -- 172.21.15.6:0/2424377552 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e041f0f00 con 0x7f7e04070e50 2024-04-24T23:15:37.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:37.205+0000 7f7e0b200640 1 -- 172.21.15.6:0/2424377552 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e04075b30 con 0x7f7e04070e50 2024-04-24T23:15:37.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:37.206+0000 7f7dfe000640 1 -- 172.21.15.6:0/2424377552 <== mon.2 v2:172.21.15.190:3301/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f7df8044020 con 0x7f7e04070e50 2024-04-24T23:15:37.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:37.207+0000 7f7dfe000640 1 --2- 172.21.15.6:0/2424377552 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f7de806be30 0x7f7de806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:37.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:37.207+0000 7f7dfe000640 1 -- 172.21.15.6:0/2424377552 <== mon.2 v2:172.21.15.190:3301/0 5 ==== osd_map(19..19 src has 1..19) ==== 5709+0+0 (secure 0 0 0) 0x7f7df80ab370 con 0x7f7e04070e50 2024-04-24T23:15:37.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:37.207+0000 7f7e09400640 1 --2- 172.21.15.6:0/2424377552 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f7de806be30 0x7f7de806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:37.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:37.208+0000 7f7e09400640 1 --2- 172.21.15.6:0/2424377552 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f7de806be30 0x7f7de806e2f0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f7e0414ff30 tx=0x7f7df4009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:15:37.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:37.212+0000 7f7dfe000640 1 -- 172.21.15.6:0/2424377552 <== mon.2 v2:172.21.15.190:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f7df804b030 con 0x7f7e04070e50 2024-04-24T23:15:37.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:37.422+0000 7f7e0b200640 1 -- 172.21.15.6:0/2424377552 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) -- 0x7f7e040c3370 con 0x7f7e04070e50 2024-04-24T23:15:37.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:37.970+0000 7f7dfe000640 1 -- 172.21.15.6:0/2424377552 <== mon.2 v2:172.21.15.190:3301/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v20) ==== 157+0+0 (secure 0 0 0) 0x7f7df80796c0 con 0x7f7e04070e50 2024-04-24T23:15:37.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:37.976+0000 7f7e0b200640 1 -- 172.21.15.6:0/2424377552 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f7de806be30 msgr2=0x7f7de806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:37.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:37.976+0000 7f7e0b200640 1 --2- 172.21.15.6:0/2424377552 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f7de806be30 0x7f7de806e2f0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f7e0414ff30 tx=0x7f7df4009190 comp rx=0 tx=0).stop 2024-04-24T23:15:37.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:37.976+0000 7f7e0b200640 1 -- 172.21.15.6:0/2424377552 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7e04070e50 msgr2=0x7f7e04151fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:37.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:37.976+0000 7f7e0b200640 1 --2- 172.21.15.6:0/2424377552 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7e04070e50 0x7f7e04151fc0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f7e0406cd90 tx=0x7f7df80046c0 comp rx=0 tx=0).stop 2024-04-24T23:15:37.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:37.976+0000 7f7e0b200640 1 -- 172.21.15.6:0/2424377552 shutdown_connections 2024-04-24T23:15:37.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:37.976+0000 7f7e0b200640 1 --2- 172.21.15.6:0/2424377552 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f7de806be30 0x7f7de806e2f0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:37.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:37.976+0000 7f7e0b200640 1 --2- 172.21.15.6:0/2424377552 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7e0415e660 0x7f7e0414ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:37.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:37.976+0000 7f7e0b200640 1 --2- 172.21.15.6:0/2424377552 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7e04154c80 0x7f7e0414e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:37.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:37.976+0000 7f7e0b200640 1 --2- 172.21.15.6:0/2424377552 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7e04070e50 0x7f7e04151fc0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:37.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:37.976+0000 7f7e0b200640 1 -- 172.21.15.6:0/2424377552 >> 172.21.15.6:0/2424377552 conn(0x7f7e0406cfe0 msgr2=0x7f7e040c7580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:15:37.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:37.976+0000 7f7e0b200640 1 -- 172.21.15.6:0/2424377552 shutdown_connections 2024-04-24T23:15:37.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:37.976+0000 7f7e0b200640 1 -- 172.21.15.6:0/2424377552 wait complete. 2024-04-24T23:15:37.999 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2024-04-24T23:15:38.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:38.212+0000 7fca3ce00640 1 Processor -- start 2024-04-24T23:15:38.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:38.213+0000 7fca3ce00640 1 -- start start 2024-04-24T23:15:38.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:38.214+0000 7fca3ce00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fca38158290 0x7fca381549f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:38.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:38.214+0000 7fca3ce00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fca3815adf0 con 0x7fca380cc9c0 2024-04-24T23:15:38.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:38.214+0000 7fca3ce00640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fca3815b0d0 con 0x7fca380cc5e0 2024-04-24T23:15:38.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:38.214+0000 7fca3ce00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7fca3815b3b0 con 0x7fca38158290 2024-04-24T23:15:38.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:38.215+0000 7fca36a00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fca380cc9c0 0x7fca38157b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:55098/0 (socket says 172.21.15.6:55098) 2024-04-24T23:15:38.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:38.215+0000 7fca36a00640 1 -- 172.21.15.6:0/3374496943 learned_addr learned my addr 172.21.15.6:0/3374496943 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:15:38.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:38.215+0000 7fca37e00640 1 --2- 172.21.15.6:0/3374496943 >> v2:172.21.15.6:3300/0 conn(0x7fca38158290 0x7fca381549f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:38.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:38.215+0000 7fca37e00640 1 -- 172.21.15.6:0/3374496943 >> v1:172.21.15.190:6789/0 conn(0x7fca380cc5e0 legacy=0x7fca38073170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:38.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:38.215+0000 7fca37e00640 1 -- 172.21.15.6:0/3374496943 >> v1:172.21.15.6:6789/0 conn(0x7fca380cc9c0 legacy=0x7fca38157b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:38.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:38.215+0000 7fca37e00640 1 -- 172.21.15.6:0/3374496943 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca3815b690 con 0x7fca38158290 2024-04-24T23:15:38.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:38.216+0000 7fca37e00640 1 --2- 172.21.15.6:0/3374496943 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fca38158290 0x7fca381549f0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fca30009df0 tx=0x7fca3002c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f93ed4beb30b0e03 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:15:38.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:38.217+0000 7fca36000640 1 -- 172.21.15.6:0/3374496943 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca30036020 con 0x7fca38158290 2024-04-24T23:15:38.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:38.217+0000 7fca36000640 1 -- 172.21.15.6:0/3374496943 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fca30032820 con 0x7fca38158290 2024-04-24T23:15:38.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:38.217+0000 7fca36000640 1 -- 172.21.15.6:0/3374496943 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca30032bf0 con 0x7fca38158290 2024-04-24T23:15:38.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:38.217+0000 7fca3ce00640 1 -- 172.21.15.6:0/3374496943 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fca38158290 msgr2=0x7fca381549f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:38.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:38.217+0000 7fca3ce00640 1 --2- 172.21.15.6:0/3374496943 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fca38158290 0x7fca381549f0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fca30009df0 tx=0x7fca3002c960 comp rx=0 tx=0).stop 2024-04-24T23:15:38.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:38.217+0000 7fca3ce00640 1 -- 172.21.15.6:0/3374496943 shutdown_connections 2024-04-24T23:15:38.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:38.217+0000 7fca3ce00640 1 --2- 172.21.15.6:0/3374496943 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fca38158290 0x7fca381549f0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:38.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:38.217+0000 7fca3ce00640 1 -- 172.21.15.6:0/3374496943 >> 172.21.15.6:0/3374496943 conn(0x7fca3806b2c0 msgr2=0x7fca3806b6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:15:38.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:38.217+0000 7fca3ce00640 1 -- 172.21.15.6:0/3374496943 shutdown_connections 2024-04-24T23:15:38.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:38.217+0000 7fca3ce00640 1 -- 172.21.15.6:0/3374496943 wait complete. 2024-04-24T23:15:38.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:38.219+0000 7fca3ce00640 1 Processor -- start 2024-04-24T23:15:38.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:38.219+0000 7fca3ce00640 1 -- start start 2024-04-24T23:15:38.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:38.220+0000 7fca3ce00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fca380cc5e0 0x7fca3814ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:38.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:38.220+0000 7fca3ce00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fca380cc9c0 0x7fca380cb400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:38.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:38.220+0000 7fca3ce00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fca38158290 0x7fca38156bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:38.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:38.220+0000 7fca3ce00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fca380c8320 con 0x7fca380cc5e0 2024-04-24T23:15:38.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:38.220+0000 7fca3ce00640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fca380c81a0 con 0x7fca380cc9c0 2024-04-24T23:15:38.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:38.220+0000 7fca3ce00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fca380c1150 con 0x7fca38158290 2024-04-24T23:15:38.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:38.220+0000 7fca37400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fca380cc5e0 0x7fca3814ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:38.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:38.220+0000 7fca37400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fca380cc5e0 0x7fca3814ded0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57536/0 (socket says 172.21.15.6:57536) 2024-04-24T23:15:38.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:38.220+0000 7fca36a00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fca380cc9c0 0x7fca380cb400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:38.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:38.220+0000 7fca37400640 1 -- 172.21.15.6:0/2962708294 learned_addr learned my addr 172.21.15.6:0/2962708294 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:15:38.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:38.221+0000 7fca37e00640 1 --2- 172.21.15.6:0/2962708294 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fca38158290 0x7fca38156bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:38.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:38.221+0000 7fca37400640 1 -- 172.21.15.6:0/2962708294 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fca38158290 msgr2=0x7fca38156bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:38.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:38.221+0000 7fca37400640 1 --2- 172.21.15.6:0/2962708294 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fca38158290 0x7fca38156bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:38.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:38.221+0000 7fca37400640 1 -- 172.21.15.6:0/2962708294 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fca380cc9c0 msgr2=0x7fca380cb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:38.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:38.221+0000 7fca37400640 1 --2- 172.21.15.6:0/2962708294 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fca380cc9c0 0x7fca380cb400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:38.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:38.221+0000 7fca37400640 1 -- 172.21.15.6:0/2962708294 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca3815a730 con 0x7fca380cc5e0 2024-04-24T23:15:38.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:38.221+0000 7fca37400640 1 --2- 172.21.15.6:0/2962708294 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fca380cc5e0 0x7fca3814ded0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fca2800c400 tx=0x7fca2800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:15:38.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:38.221+0000 7fca2f400640 1 -- 172.21.15.6:0/2962708294 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca280052d0 con 0x7fca380cc5e0 2024-04-24T23:15:38.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:38.221+0000 7fca2f400640 1 -- 172.21.15.6:0/2962708294 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fca28005490 con 0x7fca380cc5e0 2024-04-24T23:15:38.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:38.221+0000 7fca2f400640 1 -- 172.21.15.6:0/2962708294 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca28007260 con 0x7fca380cc5e0 2024-04-24T23:15:38.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:38.221+0000 7fca3ce00640 1 -- 172.21.15.6:0/2962708294 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca3815aa10 con 0x7fca380cc5e0 2024-04-24T23:15:38.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:38.222+0000 7fca3ce00640 1 -- 172.21.15.6:0/2962708294 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca380c6480 con 0x7fca380cc5e0 2024-04-24T23:15:38.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:38.223+0000 7fca3ce00640 1 -- 172.21.15.6:0/2962708294 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fca38075b30 con 0x7fca380cc5e0 2024-04-24T23:15:38.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:38.224+0000 7fca2f400640 1 -- 172.21.15.6:0/2962708294 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fca28007400 con 0x7fca380cc5e0 2024-04-24T23:15:38.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:38.225+0000 7fca2f400640 1 --2- 172.21.15.6:0/2962708294 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fca1806be60 0x7fca1806e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:38.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:38.225+0000 7fca2f400640 1 -- 172.21.15.6:0/2962708294 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 6007+0+0 (secure 0 0 0) 0x7fca2808a460 con 0x7fca380cc5e0 2024-04-24T23:15:38.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:38.225+0000 7fca36a00640 1 --2- 172.21.15.6:0/2962708294 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fca1806be60 0x7fca1806e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:38.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:38.226+0000 7fca36a00640 1 --2- 172.21.15.6:0/2962708294 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fca1806be60 0x7fca1806e320 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fca381537a0 tx=0x7fca24009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:15:38.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:38.229+0000 7fca2f400640 1 -- 172.21.15.6:0/2962708294 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fca280585f0 con 0x7fca380cc5e0 2024-04-24T23:15:38.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:38.423+0000 7fca3ce00640 1 -- 172.21.15.6:0/2962708294 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7fca380688c0 con 0x7fca380cc5e0 2024-04-24T23:15:39.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:39.976+0000 7fca2f400640 1 -- 172.21.15.6:0/2962708294 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v22) ==== 228+0+0 (secure 0 0 0) 0x7fca2805c8f0 con 0x7fca380cc5e0 2024-04-24T23:15:39.979 INFO:teuthology.orchestra.run.smithi006.stderr:pool 'cephfs_data_ec' created 2024-04-24T23:15:39.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:39.981+0000 7fca3ce00640 1 -- 172.21.15.6:0/2962708294 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fca1806be60 msgr2=0x7fca1806e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:39.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:39.981+0000 7fca3ce00640 1 --2- 172.21.15.6:0/2962708294 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fca1806be60 0x7fca1806e320 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fca381537a0 tx=0x7fca24009040 comp rx=0 tx=0).stop 2024-04-24T23:15:39.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:39.981+0000 7fca3ce00640 1 -- 172.21.15.6:0/2962708294 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fca380cc5e0 msgr2=0x7fca3814ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:39.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:39.982+0000 7fca3ce00640 1 --2- 172.21.15.6:0/2962708294 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fca380cc5e0 0x7fca3814ded0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fca2800c400 tx=0x7fca2800c8c0 comp rx=0 tx=0).stop 2024-04-24T23:15:39.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:39.982+0000 7fca3ce00640 1 -- 172.21.15.6:0/2962708294 shutdown_connections 2024-04-24T23:15:39.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:39.982+0000 7fca3ce00640 1 --2- 172.21.15.6:0/2962708294 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fca1806be60 0x7fca1806e320 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:39.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:39.982+0000 7fca3ce00640 1 --2- 172.21.15.6:0/2962708294 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fca38158290 0x7fca38156bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:39.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:39.982+0000 7fca3ce00640 1 --2- 172.21.15.6:0/2962708294 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fca380cc9c0 0x7fca380cb400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:39.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:39.982+0000 7fca3ce00640 1 --2- 172.21.15.6:0/2962708294 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fca380cc5e0 0x7fca3814ded0 secure :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fca2800c400 tx=0x7fca2800c8c0 comp rx=0 tx=0).stop 2024-04-24T23:15:39.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:39.982+0000 7fca3ce00640 1 -- 172.21.15.6:0/2962708294 >> 172.21.15.6:0/2962708294 conn(0x7fca3806b2c0 msgr2=0x7fca380b0590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:15:39.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:39.984+0000 7fca3ce00640 1 -- 172.21.15.6:0/2962708294 shutdown_connections 2024-04-24T23:15:39.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:39.987+0000 7fca3ce00640 1 -- 172.21.15.6:0/2962708294 wait complete. 2024-04-24T23:15:40.015 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2024-04-24T23:15:40.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:40.196+0000 7fe877200640 1 Processor -- start 2024-04-24T23:15:40.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:40.196+0000 7fe877200640 1 -- start start 2024-04-24T23:15:40.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:40.198+0000 7fe877200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fe8780ccaa0 0x7fe878067820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:40.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:40.198+0000 7fe877200640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe8780cce80 0x7fe8780cad00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:40.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:40.198+0000 7fe877200640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fe8780c58a0 0x7fe8780c5d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:40.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:40.198+0000 7fe877200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7fe8780cbb70 con 0x7fe8780ccaa0 2024-04-24T23:15:40.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:40.198+0000 7fe877200640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7fe8780c6290 con 0x7fe8780c58a0 2024-04-24T23:15:40.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:40.198+0000 7fe877200640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fe8780cb240 con 0x7fe8780cce80 2024-04-24T23:15:40.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:40.198+0000 7fe875e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fe8780ccaa0 0x7fe878067820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:40.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:40.198+0000 7fe875e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fe8780ccaa0 0x7fe878067820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57542/0 (socket says 172.21.15.6:57542) 2024-04-24T23:15:40.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:40.198+0000 7fe875e00640 1 -- 172.21.15.6:0/2032886274 learned_addr learned my addr 172.21.15.6:0/2032886274 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:15:40.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:40.198+0000 7fe876800640 1 --2- 172.21.15.6:0/2032886274 >> v2:172.21.15.190:3300/0 conn(0x7fe8780c58a0 0x7fe8780c5d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:40.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:40.198+0000 7fe875400640 1 --2- 172.21.15.6:0/2032886274 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe8780cce80 0x7fe8780cad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:40.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:40.199+0000 7fe876800640 1 -- 172.21.15.6:0/2032886274 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe8780cce80 msgr2=0x7fe8780cad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:15:40.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:40.199+0000 7fe876800640 1 --2- 172.21.15.6:0/2032886274 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe8780cce80 0x7fe8780cad00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:40.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:40.199+0000 7fe876800640 1 -- 172.21.15.6:0/2032886274 >> v2:172.21.15.6:3300/0 conn(0x7fe8780ccaa0 msgr2=0x7fe878067820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:15:40.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:40.199+0000 7fe876800640 1 --2- 172.21.15.6:0/2032886274 >> v2:172.21.15.6:3300/0 conn(0x7fe8780ccaa0 0x7fe878067820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:40.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:40.199+0000 7fe876800640 1 -- 172.21.15.6:0/2032886274 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8780cc0a0 con 0x7fe8780c58a0 2024-04-24T23:15:40.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:40.200+0000 7fe876800640 1 --2- 172.21.15.6:0/2032886274 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe8780c58a0 0x7fe8780c5d50 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fe8780b3800 tx=0x7fe87002a100 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ea37f38ed5d21513 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:15:40.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:40.200+0000 7fe874a00640 1 -- 172.21.15.6:0/2032886274 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe870037070 con 0x7fe8780c58a0 2024-04-24T23:15:40.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:40.200+0000 7fe874a00640 1 -- 172.21.15.6:0/2032886274 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe87002ae50 con 0x7fe8780c58a0 2024-04-24T23:15:40.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:40.200+0000 7fe874a00640 1 -- 172.21.15.6:0/2032886274 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe870032660 con 0x7fe8780c58a0 2024-04-24T23:15:40.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:40.201+0000 7fe877200640 1 -- 172.21.15.6:0/2032886274 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe8780c58a0 msgr2=0x7fe8780c5d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:40.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:40.201+0000 7fe877200640 1 --2- 172.21.15.6:0/2032886274 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe8780c58a0 0x7fe8780c5d50 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fe8780b3800 tx=0x7fe87002a100 comp rx=0 tx=0).stop 2024-04-24T23:15:40.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:40.201+0000 7fe877200640 1 -- 172.21.15.6:0/2032886274 shutdown_connections 2024-04-24T23:15:40.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:40.201+0000 7fe877200640 1 --2- 172.21.15.6:0/2032886274 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe8780c58a0 0x7fe8780c5d50 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:40.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:40.201+0000 7fe877200640 1 --2- 172.21.15.6:0/2032886274 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe8780cce80 0x7fe8780cad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:40.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:40.201+0000 7fe877200640 1 --2- 172.21.15.6:0/2032886274 >> v2:172.21.15.6:3300/0 conn(0x7fe8780ccaa0 0x7fe878067820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:40.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:40.201+0000 7fe877200640 1 -- 172.21.15.6:0/2032886274 >> 172.21.15.6:0/2032886274 conn(0x7fe87806da30 msgr2=0x7fe878066ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:15:40.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:40.201+0000 7fe877200640 1 -- 172.21.15.6:0/2032886274 shutdown_connections 2024-04-24T23:15:40.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:40.201+0000 7fe877200640 1 -- 172.21.15.6:0/2032886274 wait complete. 2024-04-24T23:15:40.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:40.203+0000 7fe877200640 1 Processor -- start 2024-04-24T23:15:40.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:40.203+0000 7fe877200640 1 -- start start 2024-04-24T23:15:40.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:40.204+0000 7fe877200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe8780c58a0 0x7fe878159350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:40.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:40.204+0000 7fe877200640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe8780ccaa0 0x7fe8780c9c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:40.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:40.204+0000 7fe877200640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe8780cce80 0x7fe87814deb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:40.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:40.204+0000 7fe877200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fe8780cbdc0 con 0x7fe8780c58a0 2024-04-24T23:15:40.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:40.204+0000 7fe877200640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fe87815b850 con 0x7fe8780cce80 2024-04-24T23:15:40.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:40.204+0000 7fe877200640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fe8780c5380 con 0x7fe8780ccaa0 2024-04-24T23:15:40.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:40.204+0000 7fe875e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe8780c58a0 0x7fe878159350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:40.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:40.204+0000 7fe875e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe8780c58a0 0x7fe878159350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57558/0 (socket says 172.21.15.6:57558) 2024-04-24T23:15:40.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:40.204+0000 7fe875e00640 1 -- 172.21.15.6:0/1232218341 learned_addr learned my addr 172.21.15.6:0/1232218341 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:15:40.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:40.204+0000 7fe875400640 1 --2- 172.21.15.6:0/1232218341 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe8780ccaa0 0x7fe8780c9c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:40.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:40.204+0000 7fe876800640 1 --2- 172.21.15.6:0/1232218341 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe8780cce80 0x7fe87814deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:40.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:40.205+0000 7fe875e00640 1 -- 172.21.15.6:0/1232218341 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe8780ccaa0 msgr2=0x7fe8780c9c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:40.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:40.205+0000 7fe875e00640 1 --2- 172.21.15.6:0/1232218341 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe8780ccaa0 0x7fe8780c9c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:40.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:40.205+0000 7fe875e00640 1 -- 172.21.15.6:0/1232218341 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe8780cce80 msgr2=0x7fe87814deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:40.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:40.205+0000 7fe875e00640 1 --2- 172.21.15.6:0/1232218341 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe8780cce80 0x7fe87814deb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:40.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:40.205+0000 7fe875e00640 1 -- 172.21.15.6:0/1232218341 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8780ca590 con 0x7fe8780c58a0 2024-04-24T23:15:40.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:40.205+0000 7fe875400640 1 --2- 172.21.15.6:0/1232218341 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe8780ccaa0 0x7fe8780c9c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:15:40.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:40.205+0000 7fe876800640 1 --2- 172.21.15.6:0/1232218341 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe8780cce80 0x7fe87814deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:15:40.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:40.205+0000 7fe875e00640 1 --2- 172.21.15.6:0/1232218341 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe8780c58a0 0x7fe878159350 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fe86800eed0 tx=0x7fe86800b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:15:40.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:40.205+0000 7fe86e000640 1 -- 172.21.15.6:0/1232218341 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe868009e60 con 0x7fe8780c58a0 2024-04-24T23:15:40.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:40.205+0000 7fe877200640 1 -- 172.21.15.6:0/1232218341 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8780c7860 con 0x7fe8780c58a0 2024-04-24T23:15:40.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:40.205+0000 7fe86e000640 1 -- 172.21.15.6:0/1232218341 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe868019070 con 0x7fe8780c58a0 2024-04-24T23:15:40.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:40.205+0000 7fe86e000640 1 -- 172.21.15.6:0/1232218341 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe868014400 con 0x7fe8780c58a0 2024-04-24T23:15:40.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:40.206+0000 7fe877200640 1 -- 172.21.15.6:0/1232218341 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8780c7e20 con 0x7fe8780c58a0 2024-04-24T23:15:40.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:40.207+0000 7fe877200640 1 -- 172.21.15.6:0/1232218341 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe878068b40 con 0x7fe8780c58a0 2024-04-24T23:15:40.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:40.208+0000 7fe86e000640 1 -- 172.21.15.6:0/1232218341 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fe868005d10 con 0x7fe8780c58a0 2024-04-24T23:15:40.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:40.209+0000 7fe86e000640 1 --2- 172.21.15.6:0/1232218341 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fe85806be30 0x7fe85806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:40.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:40.209+0000 7fe86e000640 1 -- 172.21.15.6:0/1232218341 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 6500+0+0 (secure 0 0 0) 0x7fe86808b180 con 0x7fe8780c58a0 2024-04-24T23:15:40.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:40.209+0000 7fe875400640 1 --2- 172.21.15.6:0/1232218341 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fe85806be30 0x7fe85806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:40.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:40.210+0000 7fe875400640 1 --2- 172.21.15.6:0/1232218341 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fe85806be30 0x7fe85806e2f0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fe87806d370 tx=0x7fe864009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:15:40.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:40.214+0000 7fe86e000640 1 -- 172.21.15.6:0/1232218341 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fe878068b40 con 0x7fe8780c58a0 2024-04-24T23:15:40.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:40.416+0000 7fe877200640 1 -- 172.21.15.6:0/1232218341 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) -- 0x7fe878075b30 con 0x7fe8780c58a0 2024-04-24T23:15:40.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:40.979+0000 7fe86e000640 1 -- 172.21.15.6:0/1232218341 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v23) ==== 169+0+0 (secure 0 0 0) 0x7fe8680591d0 con 0x7fe8780c58a0 2024-04-24T23:15:40.982 INFO:teuthology.orchestra.run.smithi006.stderr:set pool 4 allow_ec_overwrites to true 2024-04-24T23:15:40.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:40.983+0000 7fe877200640 1 -- 172.21.15.6:0/1232218341 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fe85806be30 msgr2=0x7fe85806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:40.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:40.983+0000 7fe877200640 1 --2- 172.21.15.6:0/1232218341 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fe85806be30 0x7fe85806e2f0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fe87806d370 tx=0x7fe864009190 comp rx=0 tx=0).stop 2024-04-24T23:15:40.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:40.984+0000 7fe877200640 1 -- 172.21.15.6:0/1232218341 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe8780c58a0 msgr2=0x7fe878159350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:40.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:40.984+0000 7fe877200640 1 --2- 172.21.15.6:0/1232218341 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe8780c58a0 0x7fe878159350 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fe86800eed0 tx=0x7fe86800b480 comp rx=0 tx=0).stop 2024-04-24T23:15:40.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:40.984+0000 7fe877200640 1 -- 172.21.15.6:0/1232218341 shutdown_connections 2024-04-24T23:15:40.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:40.984+0000 7fe877200640 1 --2- 172.21.15.6:0/1232218341 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fe85806be30 0x7fe85806e2f0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:40.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:40.984+0000 7fe877200640 1 --2- 172.21.15.6:0/1232218341 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe8780ccaa0 0x7fe8780c9c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:40.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:40.984+0000 7fe877200640 1 --2- 172.21.15.6:0/1232218341 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe8780cce80 0x7fe87814deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:40.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:40.984+0000 7fe877200640 1 --2- 172.21.15.6:0/1232218341 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe8780c58a0 0x7fe878159350 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:40.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:40.984+0000 7fe877200640 1 -- 172.21.15.6:0/1232218341 >> 172.21.15.6:0/1232218341 conn(0x7fe87806da30 msgr2=0x7fe878156560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:15:40.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:40.985+0000 7fe877200640 1 -- 172.21.15.6:0/1232218341 shutdown_connections 2024-04-24T23:15:40.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:40.985+0000 7fe877200640 1 -- 172.21.15.6:0/1232218341 wait complete. 2024-04-24T23:15:41.025 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs cephfs_data_ec 2024-04-24T23:15:41.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:41.200+0000 7f345f000640 1 Processor -- start 2024-04-24T23:15:41.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:41.200+0000 7f345f000640 1 -- start start 2024-04-24T23:15:41.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:41.202+0000 7f345f000640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f34580c8f50 0x7f345806e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:41.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:41.202+0000 7f345f000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f34580c9330 0x7f345806e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:41.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:41.202+0000 7f345f000640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f34580c6e70 con 0x7f3458152bc0 2024-04-24T23:15:41.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:41.202+0000 7f345f000640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f34580c73d0 con 0x7f34580c9330 2024-04-24T23:15:41.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:41.202+0000 7f345f000640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f34580c8ac0 con 0x7f34580c8f50 2024-04-24T23:15:41.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:41.202+0000 7f3457400640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f34580c8f50 0x7f345806e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:41.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:41.202+0000 7f345d000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f34580c9330 0x7f345806e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:41.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:41.202+0000 7f3457e00640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f3458152bc0 0x7f3458159390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.6:52754/0 (socket says 172.21.15.6:52754) 2024-04-24T23:15:41.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:41.202+0000 7f3457e00640 1 -- 172.21.15.6:0/2312442828 learned_addr learned my addr 172.21.15.6:0/2312442828 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:15:41.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:41.202+0000 7f3457400640 1 -- 172.21.15.6:0/2312442828 >> v2:172.21.15.6:3300/0 conn(0x7f34580c9330 msgr2=0x7f345806e8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:15:41.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:41.202+0000 7f3457400640 1 --2- 172.21.15.6:0/2312442828 >> v2:172.21.15.6:3300/0 conn(0x7f34580c9330 0x7f345806e8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:41.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:41.202+0000 7f3457400640 1 -- 172.21.15.6:0/2312442828 >> v1:172.21.15.190:6789/0 conn(0x7f3458152bc0 legacy=0x7f3458159390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:41.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:41.202+0000 7f3457400640 1 -- 172.21.15.6:0/2312442828 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34580c7110 con 0x7f34580c8f50 2024-04-24T23:15:41.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:41.203+0000 7f3457400640 1 --2- 172.21.15.6:0/2312442828 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f34580c8f50 0x7f345806e3a0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f34580c7550 tx=0x7f344402a100 comp rx=0 tx=0).ready entity=mon.1 client_cookie=88934607d76a6c4d server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:15:41.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:41.204+0000 7f3456a00640 1 -- 172.21.15.6:0/2312442828 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3444037070 con 0x7f34580c8f50 2024-04-24T23:15:41.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:41.204+0000 7f3456a00640 1 -- 172.21.15.6:0/2312442828 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f344402ae50 con 0x7f34580c8f50 2024-04-24T23:15:41.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:41.204+0000 7f3456a00640 1 -- 172.21.15.6:0/2312442828 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3444032660 con 0x7f34580c8f50 2024-04-24T23:15:41.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:41.205+0000 7f345f000640 1 -- 172.21.15.6:0/2312442828 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f34580c8f50 msgr2=0x7f345806e3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:41.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:41.205+0000 7f345f000640 1 --2- 172.21.15.6:0/2312442828 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f34580c8f50 0x7f345806e3a0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f34580c7550 tx=0x7f344402a100 comp rx=0 tx=0).stop 2024-04-24T23:15:41.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:41.205+0000 7f345f000640 1 -- 172.21.15.6:0/2312442828 shutdown_connections 2024-04-24T23:15:41.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:41.205+0000 7f345f000640 1 --2- 172.21.15.6:0/2312442828 >> v2:172.21.15.6:3300/0 conn(0x7f34580c9330 0x7f345806e8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:41.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:41.205+0000 7f345f000640 1 --2- 172.21.15.6:0/2312442828 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f34580c8f50 0x7f345806e3a0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:41.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:41.205+0000 7f345f000640 1 -- 172.21.15.6:0/2312442828 >> 172.21.15.6:0/2312442828 conn(0x7f345806b0e0 msgr2=0x7f345806b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:15:41.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:41.205+0000 7f345f000640 1 -- 172.21.15.6:0/2312442828 shutdown_connections 2024-04-24T23:15:41.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:41.205+0000 7f345f000640 1 -- 172.21.15.6:0/2312442828 wait complete. 2024-04-24T23:15:41.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:41.207+0000 7f345f000640 1 Processor -- start 2024-04-24T23:15:41.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:41.207+0000 7f345f000640 1 -- start start 2024-04-24T23:15:41.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:41.208+0000 7f345f000640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f34580c8f50 0x7f3458072c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:41.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:41.208+0000 7f345f000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f34580c9330 0x7f34581f5670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:41.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:41.208+0000 7f345f000640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3458152bc0 0x7f34581e3720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:41.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:41.208+0000 7f345f000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f34580c8750 con 0x7f34580c9330 2024-04-24T23:15:41.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:41.208+0000 7f345f000640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f34580c8470 con 0x7f3458152bc0 2024-04-24T23:15:41.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:41.208+0000 7f345f000640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f3458152fa0 con 0x7f34580c8f50 2024-04-24T23:15:41.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:41.209+0000 7f3457400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f34580c9330 0x7f34581f5670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:41.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:41.209+0000 7f345d000640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3458152bc0 0x7f34581e3720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:41.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:41.209+0000 7f3457e00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f34580c8f50 0x7f3458072c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:41.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:41.209+0000 7f345d000640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3458152bc0 0x7f34581e3720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:50274/0 (socket says 172.21.15.6:50274) 2024-04-24T23:15:41.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:41.209+0000 7f345d000640 1 -- 172.21.15.6:0/1711276603 learned_addr learned my addr 172.21.15.6:0/1711276603 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:15:41.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:41.209+0000 7f345d000640 1 -- 172.21.15.6:0/1711276603 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f34580c8f50 msgr2=0x7f3458072c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:41.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:41.209+0000 7f345d000640 1 --2- 172.21.15.6:0/1711276603 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f34580c8f50 0x7f3458072c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:41.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:41.209+0000 7f345d000640 1 -- 172.21.15.6:0/1711276603 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f34580c9330 msgr2=0x7f34581f5670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:41.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:41.209+0000 7f345d000640 1 --2- 172.21.15.6:0/1711276603 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f34580c9330 0x7f34581f5670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:41.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:41.209+0000 7f345d000640 1 -- 172.21.15.6:0/1711276603 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34581e4c30 con 0x7f3458152bc0 2024-04-24T23:15:41.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:41.209+0000 7f345d000640 1 --2- 172.21.15.6:0/1711276603 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3458152bc0 0x7f34581e3720 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f3450009dd0 tx=0x7f345000c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:15:41.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:41.210+0000 7f344fe00640 1 -- 172.21.15.6:0/1711276603 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3450018020 con 0x7f3458152bc0 2024-04-24T23:15:41.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:41.210+0000 7f344fe00640 1 -- 172.21.15.6:0/1711276603 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3450014480 con 0x7f3458152bc0 2024-04-24T23:15:41.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:41.210+0000 7f345f000640 1 -- 172.21.15.6:0/1711276603 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34581f16f0 con 0x7f3458152bc0 2024-04-24T23:15:41.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:41.210+0000 7f344fe00640 1 -- 172.21.15.6:0/1711276603 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3450014870 con 0x7f3458152bc0 2024-04-24T23:15:41.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:41.210+0000 7f345f000640 1 -- 172.21.15.6:0/1711276603 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34581f1c90 con 0x7f3458152bc0 2024-04-24T23:15:41.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:41.211+0000 7f345f000640 1 -- 172.21.15.6:0/1711276603 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f34580cb6d0 con 0x7f3458152bc0 2024-04-24T23:15:41.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:41.212+0000 7f344fe00640 1 -- 172.21.15.6:0/1711276603 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f3450005d10 con 0x7f3458152bc0 2024-04-24T23:15:41.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:41.213+0000 7f344fe00640 1 --2- 172.21.15.6:0/1711276603 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f343806be30 0x7f343806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:41.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:41.213+0000 7f344fe00640 1 -- 172.21.15.6:0/1711276603 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 6500+0+0 (secure 0 0 0) 0x7f345008bc30 con 0x7f3458152bc0 2024-04-24T23:15:41.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:41.213+0000 7f3457e00640 1 --2- 172.21.15.6:0/1711276603 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f343806be30 0x7f343806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:41.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:41.214+0000 7f3457e00640 1 --2- 172.21.15.6:0/1711276603 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f343806be30 0x7f343806e2f0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f3448007fd0 tx=0x7f3448007e80 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:15:41.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:41.218+0000 7f344fe00640 1 -- 172.21.15.6:0/1711276603 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f34580cb6d0 con 0x7f3458152bc0 2024-04-24T23:15:41.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:41.462+0000 7f345f000640 1 -- 172.21.15.6:0/1711276603 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) -- 0x7f34580cb9b0 con 0x7f3458152bc0 2024-04-24T23:15:42.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.055+0000 7f344fe00640 1 -- 172.21.15.6:0/1711276603 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v5) ==== 137+0+0 (secure 0 0 0) 0x7f34580cb9b0 con 0x7f3458152bc0 2024-04-24T23:15:42.057 INFO:teuthology.orchestra.run.smithi006.stderr:added data pool 4 to fsmap 2024-04-24T23:15:42.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.058+0000 7f345f000640 1 -- 172.21.15.6:0/1711276603 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f343806be30 msgr2=0x7f343806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:42.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.058+0000 7f345f000640 1 --2- 172.21.15.6:0/1711276603 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f343806be30 0x7f343806e2f0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f3448007fd0 tx=0x7f3448007e80 comp rx=0 tx=0).stop 2024-04-24T23:15:42.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.058+0000 7f345f000640 1 -- 172.21.15.6:0/1711276603 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3458152bc0 msgr2=0x7f34581e3720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:42.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.058+0000 7f345f000640 1 --2- 172.21.15.6:0/1711276603 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3458152bc0 0x7f34581e3720 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f3450009dd0 tx=0x7f345000c490 comp rx=0 tx=0).stop 2024-04-24T23:15:42.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.059+0000 7f345f000640 1 -- 172.21.15.6:0/1711276603 shutdown_connections 2024-04-24T23:15:42.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.059+0000 7f345f000640 1 --2- 172.21.15.6:0/1711276603 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f343806be30 0x7f343806e2f0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:42.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.059+0000 7f345f000640 1 --2- 172.21.15.6:0/1711276603 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3458152bc0 0x7f34581e3720 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:42.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.059+0000 7f345f000640 1 --2- 172.21.15.6:0/1711276603 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f34580c9330 0x7f34581f5670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:42.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.059+0000 7f345f000640 1 --2- 172.21.15.6:0/1711276603 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f34580c8f50 0x7f3458072c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:42.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.059+0000 7f345f000640 1 -- 172.21.15.6:0/1711276603 >> 172.21.15.6:0/1711276603 conn(0x7f345806b0e0 msgr2=0x7f345806b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:15:42.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.059+0000 7f345f000640 1 -- 172.21.15.6:0/1711276603 shutdown_connections 2024-04-24T23:15:42.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.059+0000 7f345f000640 1 -- 172.21.15.6:0/1711276603 wait complete. 2024-04-24T23:15:42.082 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:15:42.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.283+0000 7f4810200640 1 Processor -- start 2024-04-24T23:15:42.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.283+0000 7f4810200640 1 -- start start 2024-04-24T23:15:42.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.285+0000 7f4810200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f48080c8f50 0x7f480806e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:42.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.285+0000 7f4810200640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f48080c9330 0x7f480806e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:42.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.285+0000 7f4810200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f48080c6e70 con 0x7f4808152bc0 2024-04-24T23:15:42.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.285+0000 7f4810200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f48080c73d0 con 0x7f48080c8f50 2024-04-24T23:15:42.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.285+0000 7f4810200640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f48080c8ac0 con 0x7f48080c9330 2024-04-24T23:15:42.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.285+0000 7f480ce00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f48080c8f50 0x7f480806e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:42.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.286+0000 7f480d800640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f4808152bc0 0x7f4808159390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:55106/0 (socket says 172.21.15.6:55106) 2024-04-24T23:15:42.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.286+0000 7f480d800640 1 -- 172.21.15.6:0/4004141147 learned_addr learned my addr 172.21.15.6:0/4004141147 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:15:42.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.286+0000 7f480e200640 1 --2- 172.21.15.6:0/4004141147 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f48080c9330 0x7f480806e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:42.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.286+0000 7f480e200640 1 -- 172.21.15.6:0/4004141147 >> v2:172.21.15.6:3300/0 conn(0x7f48080c8f50 msgr2=0x7f480806e3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:15:42.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.286+0000 7f480e200640 1 --2- 172.21.15.6:0/4004141147 >> v2:172.21.15.6:3300/0 conn(0x7f48080c8f50 0x7f480806e3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:42.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.286+0000 7f480e200640 1 -- 172.21.15.6:0/4004141147 >> v1:172.21.15.6:6789/0 conn(0x7f4808152bc0 legacy=0x7f4808159390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:42.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.286+0000 7f480e200640 1 -- 172.21.15.6:0/4004141147 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48080c7110 con 0x7f48080c9330 2024-04-24T23:15:42.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.286+0000 7f480ce00640 1 --2- 172.21.15.6:0/4004141147 >> v2:172.21.15.6:3300/0 conn(0x7f48080c8f50 0x7f480806e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:15:42.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.287+0000 7f480e200640 1 --2- 172.21.15.6:0/4004141147 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f48080c9330 0x7f480806e8e0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f4800009d90 tx=0x7f480002a5d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7a365e73a60e6f23 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:15:42.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.288+0000 7f4807e00640 1 -- 172.21.15.6:0/4004141147 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4800031600 con 0x7f48080c9330 2024-04-24T23:15:42.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.288+0000 7f4807e00640 1 -- 172.21.15.6:0/4004141147 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f48000317a0 con 0x7f48080c9330 2024-04-24T23:15:42.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.288+0000 7f4807e00640 1 -- 172.21.15.6:0/4004141147 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4800031b70 con 0x7f48080c9330 2024-04-24T23:15:42.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.288+0000 7f4810200640 1 -- 172.21.15.6:0/4004141147 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f48080c9330 msgr2=0x7f480806e8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:42.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.288+0000 7f4810200640 1 --2- 172.21.15.6:0/4004141147 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f48080c9330 0x7f480806e8e0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f4800009d90 tx=0x7f480002a5d0 comp rx=0 tx=0).stop 2024-04-24T23:15:42.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.288+0000 7f4810200640 1 -- 172.21.15.6:0/4004141147 shutdown_connections 2024-04-24T23:15:42.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.288+0000 7f4810200640 1 --2- 172.21.15.6:0/4004141147 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f48080c9330 0x7f480806e8e0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:42.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.288+0000 7f4810200640 1 --2- 172.21.15.6:0/4004141147 >> v2:172.21.15.6:3300/0 conn(0x7f48080c8f50 0x7f480806e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:42.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.288+0000 7f4810200640 1 -- 172.21.15.6:0/4004141147 >> 172.21.15.6:0/4004141147 conn(0x7f480806b0e0 msgr2=0x7f480806b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:15:42.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.289+0000 7f4810200640 1 -- 172.21.15.6:0/4004141147 shutdown_connections 2024-04-24T23:15:42.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.289+0000 7f4810200640 1 -- 172.21.15.6:0/4004141147 wait complete. 2024-04-24T23:15:42.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.290+0000 7f4810200640 1 Processor -- start 2024-04-24T23:15:42.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.290+0000 7f4810200640 1 -- start start 2024-04-24T23:15:42.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.291+0000 7f4810200640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f48080c8f50 0x7f48081ecef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:42.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.291+0000 7f4810200640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f48080c9330 0x7f48080cb4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:42.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.291+0000 7f4810200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4808152bc0 0x7f480815b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:42.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.291+0000 7f4810200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f48080c8750 con 0x7f4808152bc0 2024-04-24T23:15:42.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.291+0000 7f4810200640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f48080c8470 con 0x7f48080c9330 2024-04-24T23:15:42.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.291+0000 7f4810200640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f4808152fa0 con 0x7f48080c8f50 2024-04-24T23:15:42.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.292+0000 7f480e200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4808152bc0 0x7f480815b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:42.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.292+0000 7f480d800640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f48080c8f50 0x7f48081ecef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:42.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.292+0000 7f480e200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4808152bc0 0x7f480815b9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57602/0 (socket says 172.21.15.6:57602) 2024-04-24T23:15:42.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.292+0000 7f480d800640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f48080c8f50 0x7f48081ecef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.6:43200/0 (socket says 172.21.15.6:43200) 2024-04-24T23:15:42.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.292+0000 7f480e200640 1 -- 172.21.15.6:0/341976196 learned_addr learned my addr 172.21.15.6:0/341976196 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:15:42.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.292+0000 7f480ce00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f48080c9330 0x7f48080cb4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:42.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.292+0000 7f480e200640 1 -- 172.21.15.6:0/341976196 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f48080c8f50 msgr2=0x7f48081ecef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:42.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.292+0000 7f480e200640 1 --2- 172.21.15.6:0/341976196 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f48080c8f50 0x7f48081ecef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:42.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.292+0000 7f480e200640 1 -- 172.21.15.6:0/341976196 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f48080c9330 msgr2=0x7f48080cb4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:42.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.292+0000 7f480e200640 1 --2- 172.21.15.6:0/341976196 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f48080c9330 0x7f48080cb4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:42.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.292+0000 7f480e200640 1 -- 172.21.15.6:0/341976196 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4808070f00 con 0x7f4808152bc0 2024-04-24T23:15:42.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.293+0000 7f480e200640 1 --2- 172.21.15.6:0/341976196 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4808152bc0 0x7f480815b9e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f480002abc0 tx=0x7f4800004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:15:42.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.293+0000 7f4805600640 1 -- 172.21.15.6:0/341976196 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4800040070 con 0x7f4808152bc0 2024-04-24T23:15:42.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.293+0000 7f4805600640 1 -- 172.21.15.6:0/341976196 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4800030b60 con 0x7f4808152bc0 2024-04-24T23:15:42.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.293+0000 7f4810200640 1 -- 172.21.15.6:0/341976196 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4808072d10 con 0x7f4808152bc0 2024-04-24T23:15:42.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.293+0000 7f4805600640 1 -- 172.21.15.6:0/341976196 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4800009450 con 0x7f4808152bc0 2024-04-24T23:15:42.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.293+0000 7f4810200640 1 -- 172.21.15.6:0/341976196 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4808071480 con 0x7f4808152bc0 2024-04-24T23:15:42.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.294+0000 7f4810200640 1 -- 172.21.15.6:0/341976196 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f48080767b0 con 0x7f4808152bc0 2024-04-24T23:15:42.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.295+0000 7f4805600640 1 -- 172.21.15.6:0/341976196 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f4800039040 con 0x7f4808152bc0 2024-04-24T23:15:42.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.296+0000 7f4805600640 1 --2- 172.21.15.6:0/341976196 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f47e806be30 0x7f47e806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:42.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.296+0000 7f4805600640 1 -- 172.21.15.6:0/341976196 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f48000ac710 con 0x7f4808152bc0 2024-04-24T23:15:42.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.296+0000 7f480d800640 1 --2- 172.21.15.6:0/341976196 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f47e806be30 0x7f47e806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:42.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.297+0000 7f480d800640 1 --2- 172.21.15.6:0/341976196 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f47e806be30 0x7f47e806e2f0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f47fc000f60 tx=0x7f47fc007e80 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:15:42.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.300+0000 7f4805600640 1 -- 172.21.15.6:0/341976196 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f480007a690 con 0x7f4808152bc0 2024-04-24T23:15:42.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.537+0000 7f4810200640 1 -- 172.21.15.6:0/341976196 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f48080c75e0 con 0x7f4808152bc0 2024-04-24T23:15:42.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.538+0000 7f4805600640 1 -- 172.21.15.6:0/341976196 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) ==== 93+0+6816 (secure 0 0 0) 0x7f480007e990 con 0x7f4808152bc0 2024-04-24T23:15:42.540 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:15:42.540 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4197,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/105267896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":105267896},{"type":"v1","addr":"172.21.15.6:6834","nonce":105267896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4203,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/2590421094","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":2590421094},{"type":"v1","addr":"172.21.15.6:6837","nonce":2590421094}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4232,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6838/2032280158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2032280158},{"type":"v1","addr":"172.21.15.190:6838","nonce":2032280158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6839/1572872451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":1572872451},{"type":"v1","addr":"172.21.15.190:6839","nonce":1572872451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4282,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6837/580315269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":580315269},{"type":"v1","addr":"172.21.15.190:6837","nonce":580315269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":5,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:15:42.051162+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4306},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/489092393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":489092393},{"type":"v1","addr":"172.21.15.6:6835","nonce":489092393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":4306,"qdb_cluster":[4306]},"id":1}]} 2024-04-24T23:15:42.540 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 6 2024-04-24T23:15:42.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.540+0000 7f4810200640 1 -- 172.21.15.6:0/341976196 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f47e806be30 msgr2=0x7f47e806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:42.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.540+0000 7f4810200640 1 --2- 172.21.15.6:0/341976196 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f47e806be30 0x7f47e806e2f0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f47fc000f60 tx=0x7f47fc007e80 comp rx=0 tx=0).stop 2024-04-24T23:15:42.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.540+0000 7f4810200640 1 -- 172.21.15.6:0/341976196 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4808152bc0 msgr2=0x7f480815b9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:42.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.540+0000 7f4810200640 1 --2- 172.21.15.6:0/341976196 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4808152bc0 0x7f480815b9e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f480002abc0 tx=0x7f4800004a40 comp rx=0 tx=0).stop 2024-04-24T23:15:42.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.540+0000 7f4810200640 1 -- 172.21.15.6:0/341976196 shutdown_connections 2024-04-24T23:15:42.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.540+0000 7f4810200640 1 --2- 172.21.15.6:0/341976196 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f47e806be30 0x7f47e806e2f0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:42.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.540+0000 7f4810200640 1 --2- 172.21.15.6:0/341976196 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4808152bc0 0x7f480815b9e0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:42.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.540+0000 7f4810200640 1 --2- 172.21.15.6:0/341976196 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f48080c9330 0x7f48080cb4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:42.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.540+0000 7f4810200640 1 --2- 172.21.15.6:0/341976196 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f48080c8f50 0x7f48081ecef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:42.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.540+0000 7f4810200640 1 -- 172.21.15.6:0/341976196 >> 172.21.15.6:0/341976196 conn(0x7f480806b0e0 msgr2=0x7f4808150d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:15:42.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.540+0000 7f4810200640 1 -- 172.21.15.6:0/341976196 shutdown_connections 2024-04-24T23:15:42.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.540+0000 7f4810200640 1 -- 172.21.15.6:0/341976196 wait complete. 2024-04-24T23:15:42.561 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T23:15:42.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.768+0000 7f6e22e00640 1 Processor -- start 2024-04-24T23:15:42.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.768+0000 7f6e22e00640 1 -- start start 2024-04-24T23:15:42.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.770+0000 7f6e22e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f6e1c151bb0 0x7f6e1c0c37f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:42.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.770+0000 7f6e22e00640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f6e1c071530 0x7f6e1c0c8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:42.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.770+0000 7f6e22e00640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6e1c0c8390 con 0x7f6e1c0c9230 2024-04-24T23:15:42.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.770+0000 7f6e22e00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f6e1c071d10 con 0x7f6e1c151bb0 2024-04-24T23:15:42.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.770+0000 7f6e22e00640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f6e1c071ff0 con 0x7f6e1c071530 2024-04-24T23:15:42.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.770+0000 7f6e1be00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f6e1c151bb0 0x7f6e1c0c37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:42.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.770+0000 7f6e1be00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f6e1c151bb0 0x7f6e1c0c37f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57610/0 (socket says 172.21.15.6:57610) 2024-04-24T23:15:42.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.770+0000 7f6e1be00640 1 -- 172.21.15.6:0/2887992875 learned_addr learned my addr 172.21.15.6:0/2887992875 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:15:42.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.771+0000 7f6e1b400640 1 --2- 172.21.15.6:0/2887992875 >> v2:172.21.15.190:3300/0 conn(0x7f6e1c071530 0x7f6e1c0c8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:42.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.771+0000 7f6e1be00640 1 -- 172.21.15.6:0/2887992875 >> v2:172.21.15.190:3300/0 conn(0x7f6e1c071530 msgr2=0x7f6e1c0c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:15:42.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.771+0000 7f6e1be00640 1 --2- 172.21.15.6:0/2887992875 >> v2:172.21.15.190:3300/0 conn(0x7f6e1c071530 0x7f6e1c0c8cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:42.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.771+0000 7f6e1be00640 1 -- 172.21.15.6:0/2887992875 >> v1:172.21.15.190:6789/0 conn(0x7f6e1c0c9230 legacy=0x7f6e1c157950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:42.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.771+0000 7f6e1be00640 1 -- 172.21.15.6:0/2887992875 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e1c0722d0 con 0x7f6e1c151bb0 2024-04-24T23:15:42.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.772+0000 7f6e1be00640 1 --2- 172.21.15.6:0/2887992875 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6e1c151bb0 0x7f6e1c0c37f0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f6e1c0ca1a0 tx=0x7f6e0c02a300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb2cd5bb6c587e11 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:15:42.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.772+0000 7f6e1aa00640 1 -- 172.21.15.6:0/2887992875 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e0c037070 con 0x7f6e1c151bb0 2024-04-24T23:15:42.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.772+0000 7f6e1aa00640 1 -- 172.21.15.6:0/2887992875 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6e0c02ae50 con 0x7f6e1c151bb0 2024-04-24T23:15:42.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.772+0000 7f6e1aa00640 1 -- 172.21.15.6:0/2887992875 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e0c032860 con 0x7f6e1c151bb0 2024-04-24T23:15:42.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.773+0000 7f6e22e00640 1 -- 172.21.15.6:0/2887992875 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6e1c151bb0 msgr2=0x7f6e1c0c37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:42.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.773+0000 7f6e22e00640 1 --2- 172.21.15.6:0/2887992875 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6e1c151bb0 0x7f6e1c0c37f0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f6e1c0ca1a0 tx=0x7f6e0c02a300 comp rx=0 tx=0).stop 2024-04-24T23:15:42.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.773+0000 7f6e22e00640 1 -- 172.21.15.6:0/2887992875 shutdown_connections 2024-04-24T23:15:42.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.773+0000 7f6e22e00640 1 --2- 172.21.15.6:0/2887992875 >> v2:172.21.15.190:3300/0 conn(0x7f6e1c071530 0x7f6e1c0c8cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:42.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.773+0000 7f6e22e00640 1 --2- 172.21.15.6:0/2887992875 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6e1c151bb0 0x7f6e1c0c37f0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:42.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.773+0000 7f6e22e00640 1 -- 172.21.15.6:0/2887992875 >> 172.21.15.6:0/2887992875 conn(0x7f6e1c06b0e0 msgr2=0x7f6e1c06b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:15:42.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.773+0000 7f6e22e00640 1 -- 172.21.15.6:0/2887992875 shutdown_connections 2024-04-24T23:15:42.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.773+0000 7f6e22e00640 1 -- 172.21.15.6:0/2887992875 wait complete. 2024-04-24T23:15:42.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.775+0000 7f6e22e00640 1 Processor -- start 2024-04-24T23:15:42.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.775+0000 7f6e22e00640 1 -- start start 2024-04-24T23:15:42.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.776+0000 7f6e22e00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6e1c071530 0x7f6e1c13cc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:42.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.776+0000 7f6e22e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6e1c0c9230 0x7f6e1c15b290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:42.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.776+0000 7f6e22e00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6e1c151bb0 0x7f6e1c1589a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:42.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.776+0000 7f6e22e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f6e1c0c3d30 con 0x7f6e1c0c9230 2024-04-24T23:15:42.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.776+0000 7f6e22e00640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f6e1c0c7f70 con 0x7f6e1c151bb0 2024-04-24T23:15:42.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.776+0000 7f6e22e00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f6e1c13b790 con 0x7f6e1c071530 2024-04-24T23:15:42.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.776+0000 7f6e1b400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6e1c0c9230 0x7f6e1c15b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:42.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.776+0000 7f6e1b400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6e1c0c9230 0x7f6e1c15b290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57616/0 (socket says 172.21.15.6:57616) 2024-04-24T23:15:42.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.776+0000 7f6e1b400640 1 -- 172.21.15.6:0/1597881282 learned_addr learned my addr 172.21.15.6:0/1597881282 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:15:42.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.776+0000 7f6e1be00640 1 --2- 172.21.15.6:0/1597881282 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6e1c071530 0x7f6e1c13cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:42.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.776+0000 7f6e20e00640 1 --2- 172.21.15.6:0/1597881282 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6e1c151bb0 0x7f6e1c1589a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:42.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.776+0000 7f6e1b400640 1 -- 172.21.15.6:0/1597881282 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6e1c071530 msgr2=0x7f6e1c13cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:42.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.776+0000 7f6e1b400640 1 --2- 172.21.15.6:0/1597881282 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6e1c071530 0x7f6e1c13cc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:42.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.776+0000 7f6e1b400640 1 -- 172.21.15.6:0/1597881282 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6e1c151bb0 msgr2=0x7f6e1c1589a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:42.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.776+0000 7f6e1b400640 1 --2- 172.21.15.6:0/1597881282 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6e1c151bb0 0x7f6e1c1589a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:42.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.777+0000 7f6e1b400640 1 -- 172.21.15.6:0/1597881282 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e1c0c37a0 con 0x7f6e1c0c9230 2024-04-24T23:15:42.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.777+0000 7f6e1b400640 1 --2- 172.21.15.6:0/1597881282 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6e1c0c9230 0x7f6e1c15b290 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f6e0400d800 tx=0x7f6e0400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:15:42.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.777+0000 7f6e13e00640 1 -- 172.21.15.6:0/1597881282 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e0400b660 con 0x7f6e1c0c9230 2024-04-24T23:15:42.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.777+0000 7f6e13e00640 1 -- 172.21.15.6:0/1597881282 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6e04019070 con 0x7f6e1c0c9230 2024-04-24T23:15:42.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.777+0000 7f6e13e00640 1 -- 172.21.15.6:0/1597881282 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e0400ba20 con 0x7f6e1c0c9230 2024-04-24T23:15:42.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.777+0000 7f6e22e00640 1 -- 172.21.15.6:0/1597881282 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e1c13e240 con 0x7f6e1c0c9230 2024-04-24T23:15:42.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.777+0000 7f6e22e00640 1 -- 172.21.15.6:0/1597881282 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e1c13d7f0 con 0x7f6e1c0c9230 2024-04-24T23:15:42.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.778+0000 7f6e22e00640 1 -- 172.21.15.6:0/1597881282 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6e1c0767b0 con 0x7f6e1c0c9230 2024-04-24T23:15:42.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.779+0000 7f6e13e00640 1 -- 172.21.15.6:0/1597881282 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f6e0400bbc0 con 0x7f6e1c0c9230 2024-04-24T23:15:42.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.780+0000 7f6e13e00640 1 --2- 172.21.15.6:0/1597881282 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f6df806be30 0x7f6df806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:42.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.781+0000 7f6e13e00640 1 -- 172.21.15.6:0/1597881282 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f6e0408b4c0 con 0x7f6e1c0c9230 2024-04-24T23:15:42.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.781+0000 7f6e1be00640 1 --2- 172.21.15.6:0/1597881282 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f6df806be30 0x7f6df806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:42.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.782+0000 7f6e1be00640 1 --2- 172.21.15.6:0/1597881282 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f6df806be30 0x7f6df806e2f0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f6e0c034000 tx=0x7f6e0c02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:15:42.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.785+0000 7f6e13e00640 1 -- 172.21.15.6:0/1597881282 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f6e1c0c37a0 con 0x7f6e1c0c9230 2024-04-24T23:15:42.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.995+0000 7f6e22e00640 1 -- 172.21.15.6:0/1597881282 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6e1c0c6000 con 0x7f6e1c0c9230 2024-04-24T23:15:42.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.997+0000 7f6e13e00640 1 -- 172.21.15.6:0/1597881282 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+18458 (secure 0 0 0) 0x7f6e040594d0 con 0x7f6e1c0c9230 2024-04-24T23:15:43.000 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:15:43.000 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":24,"fsid":"18bef68d-5e6d-4ec5-862b-b6f206e2588d","created":"2024-04-24T23:15:12.535072+0000","modified":"2024-04-24T23:15:41.465875+0000","last_up_change":"2024-04-24T23:15:23.773223+0000","last_in_change":"2024-04-24T23:15:14.905328+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T23:15:24.747441+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-24T23:15:34.051800+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-24T23:15:35.436870+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-24T23:15:39.034257+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"3b7a212d-d498-462e-90bf-ed493c803a09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":3022569437},{"type":"v1","addr":"172.21.15.6:6809","nonce":3022569437}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":3022569437},{"type":"v1","addr":"172.21.15.6:6811","nonce":3022569437}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":3022569437},{"type":"v1","addr":"172.21.15.6:6815","nonce":3022569437}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":3022569437},{"type":"v1","addr":"172.21.15.6:6813","nonce":3022569437}]},"public_addr":"172.21.15.6:6809/3022569437","cluster_addr":"172.21.15.6:6811/3022569437","heartbeat_back_addr":"172.21.15.6:6815/3022569437","heartbeat_front_addr":"172.21.15.6:6813/3022569437","state":["exists","up"]},{"osd":1,"uuid":"7c97f5a2-1735-4775-9f15-eb50099b19d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":3687836888},{"type":"v1","addr":"172.21.15.6:6817","nonce":3687836888}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":3687836888},{"type":"v1","addr":"172.21.15.6:6819","nonce":3687836888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":3687836888},{"type":"v1","addr":"172.21.15.6:6823","nonce":3687836888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":3687836888},{"type":"v1","addr":"172.21.15.6:6821","nonce":3687836888}]},"public_addr":"172.21.15.6:6817/3687836888","cluster_addr":"172.21.15.6:6819/3687836888","heartbeat_back_addr":"172.21.15.6:6823/3687836888","heartbeat_front_addr":"172.21.15.6:6821/3687836888","state":["exists","up"]},{"osd":2,"uuid":"d86bbc18-438c-4ec2-8018-ba7dd78f790b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":1020213724},{"type":"v1","addr":"172.21.15.6:6801","nonce":1020213724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":1020213724},{"type":"v1","addr":"172.21.15.6:6803","nonce":1020213724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":1020213724},{"type":"v1","addr":"172.21.15.6:6807","nonce":1020213724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":1020213724},{"type":"v1","addr":"172.21.15.6:6805","nonce":1020213724}]},"public_addr":"172.21.15.6:6801/1020213724","cluster_addr":"172.21.15.6:6803/1020213724","heartbeat_back_addr":"172.21.15.6:6807/1020213724","heartbeat_front_addr":"172.21.15.6:6805/1020213724","state":["exists","up"]},{"osd":3,"uuid":"3754ae16-4248-40dd-95f9-0f59c0eb5426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6824","nonce":3163883468},{"type":"v1","addr":"172.21.15.6:6825","nonce":3163883468}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":3163883468},{"type":"v1","addr":"172.21.15.6:6827","nonce":3163883468}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":3163883468},{"type":"v1","addr":"172.21.15.6:6831","nonce":3163883468}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":3163883468},{"type":"v1","addr":"172.21.15.6:6829","nonce":3163883468}]},"public_addr":"172.21.15.6:6825/3163883468","cluster_addr":"172.21.15.6:6827/3163883468","heartbeat_back_addr":"172.21.15.6:6831/3163883468","heartbeat_front_addr":"172.21.15.6:6829/3163883468","state":["exists","up"]},{"osd":4,"uuid":"8b2913a0-d006-479c-84ed-5b21e0e6c392","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":1989500934},{"type":"v1","addr":"172.21.15.190:6809","nonce":1989500934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":1989500934},{"type":"v1","addr":"172.21.15.190:6811","nonce":1989500934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":1989500934},{"type":"v1","addr":"172.21.15.190:6815","nonce":1989500934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":1989500934},{"type":"v1","addr":"172.21.15.190:6813","nonce":1989500934}]},"public_addr":"172.21.15.190:6809/1989500934","cluster_addr":"172.21.15.190:6811/1989500934","heartbeat_back_addr":"172.21.15.190:6815/1989500934","heartbeat_front_addr":"172.21.15.190:6813/1989500934","state":["exists","up"]},{"osd":5,"uuid":"499b79d0-0585-4dae-ab69-196a551793da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":3687548412},{"type":"v1","addr":"172.21.15.190:6801","nonce":3687548412}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":3687548412},{"type":"v1","addr":"172.21.15.190:6803","nonce":3687548412}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":3687548412},{"type":"v1","addr":"172.21.15.190:6807","nonce":3687548412}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":3687548412},{"type":"v1","addr":"172.21.15.190:6805","nonce":3687548412}]},"public_addr":"172.21.15.190:6801/3687548412","cluster_addr":"172.21.15.190:6803/3687548412","heartbeat_back_addr":"172.21.15.190:6807/3687548412","heartbeat_front_addr":"172.21.15.190:6805/3687548412","state":["exists","up"]},{"osd":6,"uuid":"6ba6df3b-4a6b-4a89-a890-0f79f387d4d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":870571564},{"type":"v1","addr":"172.21.15.190:6817","nonce":870571564}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":870571564},{"type":"v1","addr":"172.21.15.190:6819","nonce":870571564}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":870571564},{"type":"v1","addr":"172.21.15.190:6823","nonce":870571564}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":870571564},{"type":"v1","addr":"172.21.15.190:6821","nonce":870571564}]},"public_addr":"172.21.15.190:6817/870571564","cluster_addr":"172.21.15.190:6819/870571564","heartbeat_back_addr":"172.21.15.190:6823/870571564","heartbeat_front_addr":"172.21.15.190:6821/870571564","state":["exists","up"]},{"osd":7,"uuid":"e9668b1c-a000-4332-86e8-4efd73b02409","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":123639197},{"type":"v1","addr":"172.21.15.190:6825","nonce":123639197}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":123639197},{"type":"v1","addr":"172.21.15.190:6827","nonce":123639197}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":123639197},{"type":"v1","addr":"172.21.15.190:6831","nonce":123639197}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":123639197},{"type":"v1","addr":"172.21.15.190:6829","nonce":123639197}]},"public_addr":"172.21.15.190:6825/123639197","cluster_addr":"172.21.15.190:6827/123639197","heartbeat_back_addr":"172.21.15.190:6831/123639197","heartbeat_front_addr":"172.21.15.190:6829/123639197","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.321646+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.363873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.466963+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.531989+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:19.990707+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.043543+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.032492+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.115004+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T23:15:43.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.999+0000 7f6e22e00640 1 -- 172.21.15.6:0/1597881282 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f6df806be30 msgr2=0x7f6df806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:43.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.999+0000 7f6e22e00640 1 --2- 172.21.15.6:0/1597881282 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f6df806be30 0x7f6df806e2f0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f6e0c034000 tx=0x7f6e0c02e000 comp rx=0 tx=0).stop 2024-04-24T23:15:43.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.999+0000 7f6e22e00640 1 -- 172.21.15.6:0/1597881282 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6e1c0c9230 msgr2=0x7f6e1c15b290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:43.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:42.999+0000 7f6e22e00640 1 --2- 172.21.15.6:0/1597881282 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6e1c0c9230 0x7f6e1c15b290 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f6e0400d800 tx=0x7f6e0400dcc0 comp rx=0 tx=0).stop 2024-04-24T23:15:43.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.000+0000 7f6e22e00640 1 -- 172.21.15.6:0/1597881282 shutdown_connections 2024-04-24T23:15:43.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.000+0000 7f6e22e00640 1 --2- 172.21.15.6:0/1597881282 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f6df806be30 0x7f6df806e2f0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:43.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.000+0000 7f6e22e00640 1 --2- 172.21.15.6:0/1597881282 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6e1c151bb0 0x7f6e1c1589a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:43.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.000+0000 7f6e22e00640 1 --2- 172.21.15.6:0/1597881282 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6e1c0c9230 0x7f6e1c15b290 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:43.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.000+0000 7f6e22e00640 1 --2- 172.21.15.6:0/1597881282 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6e1c071530 0x7f6e1c13cc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:43.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.000+0000 7f6e22e00640 1 -- 172.21.15.6:0/1597881282 >> 172.21.15.6:0/1597881282 conn(0x7f6e1c06b0e0 msgr2=0x7f6e1c06b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:15:43.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.000+0000 7f6e22e00640 1 -- 172.21.15.6:0/1597881282 shutdown_connections 2024-04-24T23:15:43.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.000+0000 7f6e22e00640 1 -- 172.21.15.6:0/1597881282 wait complete. 2024-04-24T23:15:43.022 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T23:15:43.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.235+0000 7f4a40800640 1 Processor -- start 2024-04-24T23:15:43.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.236+0000 7f4a40800640 1 -- start start 2024-04-24T23:15:43.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.237+0000 7f4a40800640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4a38154f90 0x7f4a380c7eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:43.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.237+0000 7f4a40800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f4a3815ead0 0x7f4a38072240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:43.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.237+0000 7f4a40800640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4a3815ac20 con 0x7f4a380c84f0 2024-04-24T23:15:43.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.237+0000 7f4a40800640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f4a3815ba80 con 0x7f4a3815ead0 2024-04-24T23:15:43.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.237+0000 7f4a40800640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f4a3815af00 con 0x7f4a38154f90 2024-04-24T23:15:43.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.237+0000 7f4a3e800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f4a3815ead0 0x7f4a38072240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:43.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.237+0000 7f4a3e800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f4a3815ead0 0x7f4a38072240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57626/0 (socket says 172.21.15.6:57626) 2024-04-24T23:15:43.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.237+0000 7f4a3e800640 1 -- 172.21.15.6:0/2832229574 learned_addr learned my addr 172.21.15.6:0/2832229574 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:15:43.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.237+0000 7f4a3de00640 1 --2- 172.21.15.6:0/2832229574 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4a38154f90 0x7f4a380c7eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:43.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.238+0000 7f4a3e800640 1 -- 172.21.15.6:0/2832229574 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4a38154f90 msgr2=0x7f4a380c7eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:15:43.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.238+0000 7f4a3e800640 1 --2- 172.21.15.6:0/2832229574 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4a38154f90 0x7f4a380c7eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:43.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.238+0000 7f4a3e800640 1 -- 172.21.15.6:0/2832229574 >> v1:172.21.15.190:6789/0 conn(0x7f4a380c84f0 legacy=0x7f4a3815e3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:43.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.238+0000 7f4a3e800640 1 -- 172.21.15.6:0/2832229574 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a3815b1e0 con 0x7f4a3815ead0 2024-04-24T23:15:43.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.239+0000 7f4a3e800640 1 --2- 172.21.15.6:0/2832229574 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4a3815ead0 0x7f4a38072240 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f4a34009870 tx=0x7f4a3402a300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0e301ca4b8be6ab server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:15:43.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.239+0000 7f4a3ca00640 1 -- 172.21.15.6:0/2832229574 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a34037070 con 0x7f4a3815ead0 2024-04-24T23:15:43.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.239+0000 7f4a3ca00640 1 -- 172.21.15.6:0/2832229574 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a3402ae50 con 0x7f4a3815ead0 2024-04-24T23:15:43.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.239+0000 7f4a3ca00640 1 -- 172.21.15.6:0/2832229574 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a34032860 con 0x7f4a3815ead0 2024-04-24T23:15:43.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.240+0000 7f4a40800640 1 -- 172.21.15.6:0/2832229574 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4a3815ead0 msgr2=0x7f4a38072240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:43.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.240+0000 7f4a40800640 1 --2- 172.21.15.6:0/2832229574 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4a3815ead0 0x7f4a38072240 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f4a34009870 tx=0x7f4a3402a300 comp rx=0 tx=0).stop 2024-04-24T23:15:43.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.240+0000 7f4a40800640 1 -- 172.21.15.6:0/2832229574 shutdown_connections 2024-04-24T23:15:43.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.240+0000 7f4a40800640 1 --2- 172.21.15.6:0/2832229574 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4a3815ead0 0x7f4a38072240 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:43.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.240+0000 7f4a40800640 1 --2- 172.21.15.6:0/2832229574 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4a38154f90 0x7f4a380c7eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:43.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.240+0000 7f4a40800640 1 -- 172.21.15.6:0/2832229574 >> 172.21.15.6:0/2832229574 conn(0x7f4a3806e780 msgr2=0x7f4a3806fdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:15:43.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.240+0000 7f4a40800640 1 -- 172.21.15.6:0/2832229574 shutdown_connections 2024-04-24T23:15:43.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.240+0000 7f4a40800640 1 -- 172.21.15.6:0/2832229574 wait complete. 2024-04-24T23:15:43.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.242+0000 7f4a40800640 1 Processor -- start 2024-04-24T23:15:43.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.242+0000 7f4a40800640 1 -- start start 2024-04-24T23:15:43.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.243+0000 7f4a40800640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4a380c84f0 0x7f4a381f3b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:43.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.243+0000 7f4a40800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4a38154f90 0x7f4a381502a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:43.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.243+0000 7f4a40800640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4a3815ead0 0x7f4a381507e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:43.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.243+0000 7f4a40800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f4a38072780 con 0x7f4a38154f90 2024-04-24T23:15:43.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.243+0000 7f4a40800640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f4a3815b7a0 con 0x7f4a3815ead0 2024-04-24T23:15:43.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.243+0000 7f4a40800640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f4a38072900 con 0x7f4a380c84f0 2024-04-24T23:15:43.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.243+0000 7f4a3d400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4a38154f90 0x7f4a381502a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:43.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.243+0000 7f4a3d400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4a38154f90 0x7f4a381502a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57634/0 (socket says 172.21.15.6:57634) 2024-04-24T23:15:43.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.243+0000 7f4a3d400640 1 -- 172.21.15.6:0/3753219721 learned_addr learned my addr 172.21.15.6:0/3753219721 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:15:43.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.243+0000 7f4a3e800640 1 --2- 172.21.15.6:0/3753219721 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4a3815ead0 0x7f4a381507e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:43.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.243+0000 7f4a3de00640 1 --2- 172.21.15.6:0/3753219721 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4a380c84f0 0x7f4a381f3b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:43.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.244+0000 7f4a3d400640 1 -- 172.21.15.6:0/3753219721 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4a380c84f0 msgr2=0x7f4a381f3b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:43.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.244+0000 7f4a3d400640 1 --2- 172.21.15.6:0/3753219721 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4a380c84f0 0x7f4a381f3b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:43.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.244+0000 7f4a3d400640 1 -- 172.21.15.6:0/3753219721 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4a3815ead0 msgr2=0x7f4a381507e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:43.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.244+0000 7f4a3d400640 1 --2- 172.21.15.6:0/3753219721 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4a3815ead0 0x7f4a381507e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:43.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.244+0000 7f4a3d400640 1 -- 172.21.15.6:0/3753219721 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a38153670 con 0x7f4a38154f90 2024-04-24T23:15:43.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.244+0000 7f4a3d400640 1 --2- 172.21.15.6:0/3753219721 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4a38154f90 0x7f4a381502a0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f4a2800c400 tx=0x7f4a2800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:15:43.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.244+0000 7f4a2e000640 1 -- 172.21.15.6:0/3753219721 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a280052d0 con 0x7f4a38154f90 2024-04-24T23:15:43.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.244+0000 7f4a2e000640 1 -- 172.21.15.6:0/3753219721 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a28005490 con 0x7f4a38154f90 2024-04-24T23:15:43.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.244+0000 7f4a2e000640 1 -- 172.21.15.6:0/3753219721 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a28007260 con 0x7f4a38154f90 2024-04-24T23:15:43.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.244+0000 7f4a40800640 1 -- 172.21.15.6:0/3753219721 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a381f4530 con 0x7f4a38154f90 2024-04-24T23:15:43.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.244+0000 7f4a40800640 1 -- 172.21.15.6:0/3753219721 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a3814fd80 con 0x7f4a38154f90 2024-04-24T23:15:43.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.245+0000 7f4a40800640 1 -- 172.21.15.6:0/3753219721 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a38068430 con 0x7f4a38154f90 2024-04-24T23:15:43.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.246+0000 7f4a2e000640 1 -- 172.21.15.6:0/3753219721 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f4a28007400 con 0x7f4a38154f90 2024-04-24T23:15:43.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.247+0000 7f4a2e000640 1 --2- 172.21.15.6:0/3753219721 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f4a1c06be30 0x7f4a1c06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:43.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.247+0000 7f4a2e000640 1 -- 172.21.15.6:0/3753219721 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f4a2808a640 con 0x7f4a38154f90 2024-04-24T23:15:43.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.248+0000 7f4a3de00640 1 --2- 172.21.15.6:0/3753219721 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f4a1c06be30 0x7f4a1c06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:43.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.249+0000 7f4a3de00640 1 --2- 172.21.15.6:0/3753219721 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f4a1c06be30 0x7f4a1c06e2f0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f4a300113e0 tx=0x7f4a3000b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:15:43.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.252+0000 7f4a2e000640 1 -- 172.21.15.6:0/3753219721 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f4a28058650 con 0x7f4a38154f90 2024-04-24T23:15:43.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.456+0000 7f4a40800640 1 -- 172.21.15.6:0/3753219721 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4a380c89f0 con 0x7f4a38154f90 2024-04-24T23:15:43.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.458+0000 7f4a2e000640 1 -- 172.21.15.6:0/3753219721 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+18458 (secure 0 0 0) 0x7f4a2805c950 con 0x7f4a38154f90 2024-04-24T23:15:43.461 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:15:43.461 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":24,"fsid":"18bef68d-5e6d-4ec5-862b-b6f206e2588d","created":"2024-04-24T23:15:12.535072+0000","modified":"2024-04-24T23:15:41.465875+0000","last_up_change":"2024-04-24T23:15:23.773223+0000","last_in_change":"2024-04-24T23:15:14.905328+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T23:15:24.747441+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-24T23:15:34.051800+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-24T23:15:35.436870+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-24T23:15:39.034257+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"3b7a212d-d498-462e-90bf-ed493c803a09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":3022569437},{"type":"v1","addr":"172.21.15.6:6809","nonce":3022569437}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":3022569437},{"type":"v1","addr":"172.21.15.6:6811","nonce":3022569437}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":3022569437},{"type":"v1","addr":"172.21.15.6:6815","nonce":3022569437}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":3022569437},{"type":"v1","addr":"172.21.15.6:6813","nonce":3022569437}]},"public_addr":"172.21.15.6:6809/3022569437","cluster_addr":"172.21.15.6:6811/3022569437","heartbeat_back_addr":"172.21.15.6:6815/3022569437","heartbeat_front_addr":"172.21.15.6:6813/3022569437","state":["exists","up"]},{"osd":1,"uuid":"7c97f5a2-1735-4775-9f15-eb50099b19d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":3687836888},{"type":"v1","addr":"172.21.15.6:6817","nonce":3687836888}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":3687836888},{"type":"v1","addr":"172.21.15.6:6819","nonce":3687836888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":3687836888},{"type":"v1","addr":"172.21.15.6:6823","nonce":3687836888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":3687836888},{"type":"v1","addr":"172.21.15.6:6821","nonce":3687836888}]},"public_addr":"172.21.15.6:6817/3687836888","cluster_addr":"172.21.15.6:6819/3687836888","heartbeat_back_addr":"172.21.15.6:6823/3687836888","heartbeat_front_addr":"172.21.15.6:6821/3687836888","state":["exists","up"]},{"osd":2,"uuid":"d86bbc18-438c-4ec2-8018-ba7dd78f790b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":1020213724},{"type":"v1","addr":"172.21.15.6:6801","nonce":1020213724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":1020213724},{"type":"v1","addr":"172.21.15.6:6803","nonce":1020213724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":1020213724},{"type":"v1","addr":"172.21.15.6:6807","nonce":1020213724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":1020213724},{"type":"v1","addr":"172.21.15.6:6805","nonce":1020213724}]},"public_addr":"172.21.15.6:6801/1020213724","cluster_addr":"172.21.15.6:6803/1020213724","heartbeat_back_addr":"172.21.15.6:6807/1020213724","heartbeat_front_addr":"172.21.15.6:6805/1020213724","state":["exists","up"]},{"osd":3,"uuid":"3754ae16-4248-40dd-95f9-0f59c0eb5426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6824","nonce":3163883468},{"type":"v1","addr":"172.21.15.6:6825","nonce":3163883468}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":3163883468},{"type":"v1","addr":"172.21.15.6:6827","nonce":3163883468}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":3163883468},{"type":"v1","addr":"172.21.15.6:6831","nonce":3163883468}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":3163883468},{"type":"v1","addr":"172.21.15.6:6829","nonce":3163883468}]},"public_addr":"172.21.15.6:6825/3163883468","cluster_addr":"172.21.15.6:6827/3163883468","heartbeat_back_addr":"172.21.15.6:6831/3163883468","heartbeat_front_addr":"172.21.15.6:6829/3163883468","state":["exists","up"]},{"osd":4,"uuid":"8b2913a0-d006-479c-84ed-5b21e0e6c392","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":1989500934},{"type":"v1","addr":"172.21.15.190:6809","nonce":1989500934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":1989500934},{"type":"v1","addr":"172.21.15.190:6811","nonce":1989500934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":1989500934},{"type":"v1","addr":"172.21.15.190:6815","nonce":1989500934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":1989500934},{"type":"v1","addr":"172.21.15.190:6813","nonce":1989500934}]},"public_addr":"172.21.15.190:6809/1989500934","cluster_addr":"172.21.15.190:6811/1989500934","heartbeat_back_addr":"172.21.15.190:6815/1989500934","heartbeat_front_addr":"172.21.15.190:6813/1989500934","state":["exists","up"]},{"osd":5,"uuid":"499b79d0-0585-4dae-ab69-196a551793da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":3687548412},{"type":"v1","addr":"172.21.15.190:6801","nonce":3687548412}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":3687548412},{"type":"v1","addr":"172.21.15.190:6803","nonce":3687548412}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":3687548412},{"type":"v1","addr":"172.21.15.190:6807","nonce":3687548412}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":3687548412},{"type":"v1","addr":"172.21.15.190:6805","nonce":3687548412}]},"public_addr":"172.21.15.190:6801/3687548412","cluster_addr":"172.21.15.190:6803/3687548412","heartbeat_back_addr":"172.21.15.190:6807/3687548412","heartbeat_front_addr":"172.21.15.190:6805/3687548412","state":["exists","up"]},{"osd":6,"uuid":"6ba6df3b-4a6b-4a89-a890-0f79f387d4d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":870571564},{"type":"v1","addr":"172.21.15.190:6817","nonce":870571564}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":870571564},{"type":"v1","addr":"172.21.15.190:6819","nonce":870571564}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":870571564},{"type":"v1","addr":"172.21.15.190:6823","nonce":870571564}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":870571564},{"type":"v1","addr":"172.21.15.190:6821","nonce":870571564}]},"public_addr":"172.21.15.190:6817/870571564","cluster_addr":"172.21.15.190:6819/870571564","heartbeat_back_addr":"172.21.15.190:6823/870571564","heartbeat_front_addr":"172.21.15.190:6821/870571564","state":["exists","up"]},{"osd":7,"uuid":"e9668b1c-a000-4332-86e8-4efd73b02409","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":123639197},{"type":"v1","addr":"172.21.15.190:6825","nonce":123639197}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":123639197},{"type":"v1","addr":"172.21.15.190:6827","nonce":123639197}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":123639197},{"type":"v1","addr":"172.21.15.190:6831","nonce":123639197}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":123639197},{"type":"v1","addr":"172.21.15.190:6829","nonce":123639197}]},"public_addr":"172.21.15.190:6825/123639197","cluster_addr":"172.21.15.190:6827/123639197","heartbeat_back_addr":"172.21.15.190:6831/123639197","heartbeat_front_addr":"172.21.15.190:6829/123639197","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.321646+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.363873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.466963+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.531989+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:19.990707+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.043543+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.032492+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.115004+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T23:15:43.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.461+0000 7f4a40800640 1 -- 172.21.15.6:0/3753219721 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f4a1c06be30 msgr2=0x7f4a1c06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:43.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.461+0000 7f4a40800640 1 --2- 172.21.15.6:0/3753219721 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f4a1c06be30 0x7f4a1c06e2f0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f4a300113e0 tx=0x7f4a3000b040 comp rx=0 tx=0).stop 2024-04-24T23:15:43.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.461+0000 7f4a40800640 1 -- 172.21.15.6:0/3753219721 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4a38154f90 msgr2=0x7f4a381502a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:43.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.461+0000 7f4a40800640 1 --2- 172.21.15.6:0/3753219721 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4a38154f90 0x7f4a381502a0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f4a2800c400 tx=0x7f4a2800c8c0 comp rx=0 tx=0).stop 2024-04-24T23:15:43.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.461+0000 7f4a40800640 1 -- 172.21.15.6:0/3753219721 shutdown_connections 2024-04-24T23:15:43.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.461+0000 7f4a40800640 1 --2- 172.21.15.6:0/3753219721 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f4a1c06be30 0x7f4a1c06e2f0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:43.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.461+0000 7f4a40800640 1 --2- 172.21.15.6:0/3753219721 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4a3815ead0 0x7f4a381507e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:43.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.461+0000 7f4a40800640 1 --2- 172.21.15.6:0/3753219721 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4a38154f90 0x7f4a381502a0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:43.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.461+0000 7f4a40800640 1 --2- 172.21.15.6:0/3753219721 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4a380c84f0 0x7f4a381f3b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:43.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.461+0000 7f4a40800640 1 -- 172.21.15.6:0/3753219721 >> 172.21.15.6:0/3753219721 conn(0x7f4a3806e780 msgr2=0x7f4a3814b020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:15:43.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.461+0000 7f4a40800640 1 -- 172.21.15.6:0/3753219721 shutdown_connections 2024-04-24T23:15:43.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.461+0000 7f4a40800640 1 -- 172.21.15.6:0/3753219721 wait complete. 2024-04-24T23:15:43.497 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:15:43.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.709+0000 7fa42b200640 1 Processor -- start 2024-04-24T23:15:43.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.709+0000 7fa42b200640 1 -- start start 2024-04-24T23:15:43.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.710+0000 7fa42b200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa42c152bd0 0x7fa42c158e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:43.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.711+0000 7fa42b200640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fa42c06ce30 0x7fa42c159380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:43.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.711+0000 7fa42b200640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa42c0c6be0 0x7fa42c0cc600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:43.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.711+0000 7fa42b200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7fa42c06d970 con 0x7fa42c152bd0 2024-04-24T23:15:43.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.711+0000 7fa42b200640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7fa42c06d390 con 0x7fa42c06ce30 2024-04-24T23:15:43.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.711+0000 7fa42b200640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fa42c06dbc0 con 0x7fa42c0c6be0 2024-04-24T23:15:43.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.711+0000 7fa429e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa42c152bd0 0x7fa42c158e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:43.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.711+0000 7fa429e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa42c152bd0 0x7fa42c158e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57644/0 (socket says 172.21.15.6:57644) 2024-04-24T23:15:43.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.711+0000 7fa429e00640 1 -- 172.21.15.6:0/3108701481 learned_addr learned my addr 172.21.15.6:0/3108701481 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:15:43.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.711+0000 7fa42a800640 1 --2- 172.21.15.6:0/3108701481 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa42c0c6be0 0x7fa42c0cc600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:43.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.711+0000 7fa423e00640 1 --2- 172.21.15.6:0/3108701481 >> v2:172.21.15.190:3300/0 conn(0x7fa42c06ce30 0x7fa42c159380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:43.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.711+0000 7fa423e00640 1 -- 172.21.15.6:0/3108701481 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa42c0c6be0 msgr2=0x7fa42c0cc600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:15:43.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.711+0000 7fa423e00640 1 --2- 172.21.15.6:0/3108701481 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa42c0c6be0 0x7fa42c0cc600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:43.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.711+0000 7fa423e00640 1 -- 172.21.15.6:0/3108701481 >> v2:172.21.15.6:3300/0 conn(0x7fa42c152bd0 msgr2=0x7fa42c158e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:15:43.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.711+0000 7fa423e00640 1 --2- 172.21.15.6:0/3108701481 >> v2:172.21.15.6:3300/0 conn(0x7fa42c152bd0 0x7fa42c158e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:43.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.712+0000 7fa423e00640 1 -- 172.21.15.6:0/3108701481 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa42c06dea0 con 0x7fa42c06ce30 2024-04-24T23:15:43.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.712+0000 7fa429e00640 1 --2- 172.21.15.6:0/3108701481 >> v2:172.21.15.6:3300/0 conn(0x7fa42c152bd0 0x7fa42c158e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:15:43.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.713+0000 7fa423e00640 1 --2- 172.21.15.6:0/3108701481 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa42c06ce30 0x7fa42c159380 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fa42c070820 tx=0x7fa41402a080 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fef4f9c08ea1cc73 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:15:43.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.713+0000 7fa429400640 1 -- 172.21.15.6:0/3108701481 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa414037070 con 0x7fa42c06ce30 2024-04-24T23:15:43.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.713+0000 7fa429400640 1 -- 172.21.15.6:0/3108701481 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa41402add0 con 0x7fa42c06ce30 2024-04-24T23:15:43.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.713+0000 7fa429400640 1 -- 172.21.15.6:0/3108701481 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa414032660 con 0x7fa42c06ce30 2024-04-24T23:15:43.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.714+0000 7fa42b200640 1 -- 172.21.15.6:0/3108701481 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa42c06ce30 msgr2=0x7fa42c159380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:43.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.714+0000 7fa42b200640 1 --2- 172.21.15.6:0/3108701481 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa42c06ce30 0x7fa42c159380 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fa42c070820 tx=0x7fa41402a080 comp rx=0 tx=0).stop 2024-04-24T23:15:43.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.714+0000 7fa42b200640 1 -- 172.21.15.6:0/3108701481 shutdown_connections 2024-04-24T23:15:43.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.714+0000 7fa42b200640 1 --2- 172.21.15.6:0/3108701481 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa42c0c6be0 0x7fa42c0cc600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:43.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.714+0000 7fa42b200640 1 --2- 172.21.15.6:0/3108701481 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa42c06ce30 0x7fa42c159380 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:43.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.714+0000 7fa42b200640 1 --2- 172.21.15.6:0/3108701481 >> v2:172.21.15.6:3300/0 conn(0x7fa42c152bd0 0x7fa42c158e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:43.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.714+0000 7fa42b200640 1 -- 172.21.15.6:0/3108701481 >> 172.21.15.6:0/3108701481 conn(0x7fa42c06ed80 msgr2=0x7fa42c06f190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:15:43.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.714+0000 7fa42b200640 1 -- 172.21.15.6:0/3108701481 shutdown_connections 2024-04-24T23:15:43.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.714+0000 7fa42b200640 1 -- 172.21.15.6:0/3108701481 wait complete. 2024-04-24T23:15:43.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.716+0000 7fa42b200640 1 Processor -- start 2024-04-24T23:15:43.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.716+0000 7fa42b200640 1 -- start start 2024-04-24T23:15:43.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.717+0000 7fa42b200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa42c06ce30 0x7fa42c14c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:43.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.717+0000 7fa42b200640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa42c0c6be0 0x7fa42c147eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:43.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.717+0000 7fa42b200640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa42c152bd0 0x7fa42c1f1db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:43.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.717+0000 7fa42b200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fa42c072650 con 0x7fa42c06ce30 2024-04-24T23:15:43.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.717+0000 7fa42b200640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fa42c071890 con 0x7fa42c0c6be0 2024-04-24T23:15:43.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.717+0000 7fa42b200640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fa42c0727d0 con 0x7fa42c152bd0 2024-04-24T23:15:43.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.717+0000 7fa429e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa42c06ce30 0x7fa42c14c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:43.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.717+0000 7fa429e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa42c06ce30 0x7fa42c14c3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57646/0 (socket says 172.21.15.6:57646) 2024-04-24T23:15:43.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.717+0000 7fa429e00640 1 -- 172.21.15.6:0/3248601035 learned_addr learned my addr 172.21.15.6:0/3248601035 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:15:43.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.717+0000 7fa42a800640 1 --2- 172.21.15.6:0/3248601035 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa42c152bd0 0x7fa42c1f1db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:43.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.717+0000 7fa423e00640 1 --2- 172.21.15.6:0/3248601035 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa42c0c6be0 0x7fa42c147eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:43.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.718+0000 7fa429e00640 1 -- 172.21.15.6:0/3248601035 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa42c152bd0 msgr2=0x7fa42c1f1db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:43.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.718+0000 7fa429e00640 1 --2- 172.21.15.6:0/3248601035 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa42c152bd0 0x7fa42c1f1db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:43.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.718+0000 7fa429e00640 1 -- 172.21.15.6:0/3248601035 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa42c0c6be0 msgr2=0x7fa42c147eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:43.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.718+0000 7fa429e00640 1 --2- 172.21.15.6:0/3248601035 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa42c0c6be0 0x7fa42c147eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:43.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.718+0000 7fa429e00640 1 -- 172.21.15.6:0/3248601035 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa42c14c900 con 0x7fa42c06ce30 2024-04-24T23:15:43.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.718+0000 7fa429e00640 1 --2- 172.21.15.6:0/3248601035 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa42c06ce30 0x7fa42c14c3c0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa41c00e9d0 tx=0x7fa41c00ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:15:43.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.718+0000 7fa422000640 1 -- 172.21.15.6:0/3248601035 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa41c017070 con 0x7fa42c06ce30 2024-04-24T23:15:43.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.718+0000 7fa422000640 1 -- 172.21.15.6:0/3248601035 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa41c00ca60 con 0x7fa42c06ce30 2024-04-24T23:15:43.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.718+0000 7fa422000640 1 -- 172.21.15.6:0/3248601035 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa41c00ce10 con 0x7fa42c06ce30 2024-04-24T23:15:43.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.718+0000 7fa42b200640 1 -- 172.21.15.6:0/3248601035 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa42c149a70 con 0x7fa42c06ce30 2024-04-24T23:15:43.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.719+0000 7fa42b200640 1 -- 172.21.15.6:0/3248601035 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa42c14b720 con 0x7fa42c06ce30 2024-04-24T23:15:43.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.720+0000 7fa42b200640 1 -- 172.21.15.6:0/3248601035 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa42c075b30 con 0x7fa42c06ce30 2024-04-24T23:15:43.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.721+0000 7fa422000640 1 -- 172.21.15.6:0/3248601035 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fa41c01a070 con 0x7fa42c06ce30 2024-04-24T23:15:43.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.722+0000 7fa422000640 1 --2- 172.21.15.6:0/3248601035 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fa40806be30 0x7fa40806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:43.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.722+0000 7fa422000640 1 -- 172.21.15.6:0/3248601035 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fa41c08b460 con 0x7fa42c06ce30 2024-04-24T23:15:43.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.722+0000 7fa423e00640 1 --2- 172.21.15.6:0/3248601035 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fa40806be30 0x7fa40806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:43.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.723+0000 7fa423e00640 1 --2- 172.21.15.6:0/3248601035 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fa40806be30 0x7fa40806e2f0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fa414034000 tx=0x7fa41402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:15:43.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.726+0000 7fa422000640 1 -- 172.21.15.6:0/3248601035 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fa41c059480 con 0x7fa42c06ce30 2024-04-24T23:15:43.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.973+0000 7fa42b200640 1 -- 172.21.15.6:0/3248601035 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa42c0c8b10 con 0x7fa42c06ce30 2024-04-24T23:15:43.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.974+0000 7fa422000640 1 -- 172.21.15.6:0/3248601035 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) ==== 93+0+6816 (secure 0 0 0) 0x7fa41c05d780 con 0x7fa42c06ce30 2024-04-24T23:15:43.977 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:15:43.977 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4197,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/105267896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":105267896},{"type":"v1","addr":"172.21.15.6:6834","nonce":105267896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4203,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/2590421094","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":2590421094},{"type":"v1","addr":"172.21.15.6:6837","nonce":2590421094}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4232,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6838/2032280158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2032280158},{"type":"v1","addr":"172.21.15.190:6838","nonce":2032280158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6839/1572872451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":1572872451},{"type":"v1","addr":"172.21.15.190:6839","nonce":1572872451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4282,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6837/580315269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":580315269},{"type":"v1","addr":"172.21.15.190:6837","nonce":580315269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":5,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:15:42.051162+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4306},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/489092393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":489092393},{"type":"v1","addr":"172.21.15.6:6835","nonce":489092393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":4306,"qdb_cluster":[4306]},"id":1}]} 2024-04-24T23:15:43.977 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 6 2024-04-24T23:15:43.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.977+0000 7fa42b200640 1 -- 172.21.15.6:0/3248601035 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fa40806be30 msgr2=0x7fa40806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:43.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.977+0000 7fa42b200640 1 --2- 172.21.15.6:0/3248601035 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fa40806be30 0x7fa40806e2f0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fa414034000 tx=0x7fa41402e000 comp rx=0 tx=0).stop 2024-04-24T23:15:43.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.977+0000 7fa42b200640 1 -- 172.21.15.6:0/3248601035 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa42c06ce30 msgr2=0x7fa42c14c3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:43.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.977+0000 7fa42b200640 1 --2- 172.21.15.6:0/3248601035 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa42c06ce30 0x7fa42c14c3c0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa41c00e9d0 tx=0x7fa41c00ee90 comp rx=0 tx=0).stop 2024-04-24T23:15:43.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.977+0000 7fa42b200640 1 -- 172.21.15.6:0/3248601035 shutdown_connections 2024-04-24T23:15:43.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.977+0000 7fa42b200640 1 --2- 172.21.15.6:0/3248601035 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fa40806be30 0x7fa40806e2f0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:43.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.977+0000 7fa42b200640 1 --2- 172.21.15.6:0/3248601035 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa42c152bd0 0x7fa42c1f1db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:43.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.977+0000 7fa42b200640 1 --2- 172.21.15.6:0/3248601035 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa42c0c6be0 0x7fa42c147eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:43.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.977+0000 7fa42b200640 1 --2- 172.21.15.6:0/3248601035 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa42c06ce30 0x7fa42c14c3c0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:43.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.977+0000 7fa42b200640 1 -- 172.21.15.6:0/3248601035 >> 172.21.15.6:0/3248601035 conn(0x7fa42c06ed80 msgr2=0x7fa42c158e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:15:43.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.977+0000 7fa42b200640 1 -- 172.21.15.6:0/3248601035 shutdown_connections 2024-04-24T23:15:43.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:43.977+0000 7fa42b200640 1 -- 172.21.15.6:0/3248601035 wait complete. 2024-04-24T23:15:43.998 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T23:15:44.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:44.201+0000 7f25e4a00640 1 Processor -- start 2024-04-24T23:15:44.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:44.202+0000 7f25e4a00640 1 -- start start 2024-04-24T23:15:44.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:44.203+0000 7f25e4a00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f25e00cb090 0x7f25e00d0820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:44.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:44.203+0000 7f25e4a00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f25e0155170 con 0x7f25e0154d90 2024-04-24T23:15:44.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:44.203+0000 7f25e4a00640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f25e00cc9a0 con 0x7f25e00d0d60 2024-04-24T23:15:44.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:44.203+0000 7f25e4a00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f25e015ad60 con 0x7f25e00cb090 2024-04-24T23:15:44.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:44.204+0000 7f25dfe00640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f25e00d0d60 0x7f25e015e000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.6:52792/0 (socket says 172.21.15.6:52792) 2024-04-24T23:15:44.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:44.204+0000 7f25dea00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f25e00cb090 0x7f25e00d0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:44.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:44.204+0000 7f25dfe00640 1 -- 172.21.15.6:0/4283918969 learned_addr learned my addr 172.21.15.6:0/4283918969 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:15:44.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:44.204+0000 7f25de000640 1 -- 172.21.15.6:0/4283918969 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3612555065 0 0) 0x7f25e00cc9a0 con 0x7f25e00d0d60 2024-04-24T23:15:44.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:44.204+0000 7f25de000640 1 -- 172.21.15.6:0/4283918969 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f25c4003520 con 0x7f25e00d0d60 2024-04-24T23:15:44.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:44.204+0000 7f25dea00640 1 -- 172.21.15.6:0/4283918969 >> v1:172.21.15.190:6789/0 conn(0x7f25e00d0d60 legacy=0x7f25e015e000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:44.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:44.204+0000 7f25dea00640 1 -- 172.21.15.6:0/4283918969 >> v1:172.21.15.6:6789/0 conn(0x7f25e0154d90 legacy=0x7f25e00cbe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:44.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:44.204+0000 7f25dea00640 1 -- 172.21.15.6:0/4283918969 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25e015b040 con 0x7f25e00cb090 2024-04-24T23:15:44.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:44.204+0000 7f25de000640 1 -- 172.21.15.6:0/4283918969 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1779566845 0 0) 0x7f25e0155170 con 0x7f25e0154d90 2024-04-24T23:15:44.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:44.205+0000 7f25dea00640 1 --2- 172.21.15.6:0/4283918969 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f25e00cb090 0x7f25e00d0820 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f25c80097d0 tx=0x7f25c802a290 comp rx=0 tx=0).ready entity=mon.2 client_cookie=59d668932f7f17e3 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:15:44.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:44.205+0000 7f25de000640 1 -- 172.21.15.6:0/4283918969 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25c8037070 con 0x7f25e00cb090 2024-04-24T23:15:44.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:44.206+0000 7f25de000640 1 -- 172.21.15.6:0/4283918969 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f25c8032440 con 0x7f25e00cb090 2024-04-24T23:15:44.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:44.206+0000 7f25de000640 1 -- 172.21.15.6:0/4283918969 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25c80327f0 con 0x7f25e00cb090 2024-04-24T23:15:44.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:44.206+0000 7f25e4a00640 1 -- 172.21.15.6:0/4283918969 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f25e00cb090 msgr2=0x7f25e00d0820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:44.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:44.206+0000 7f25e4a00640 1 --2- 172.21.15.6:0/4283918969 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f25e00cb090 0x7f25e00d0820 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f25c80097d0 tx=0x7f25c802a290 comp rx=0 tx=0).stop 2024-04-24T23:15:44.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:44.206+0000 7f25e4a00640 1 -- 172.21.15.6:0/4283918969 shutdown_connections 2024-04-24T23:15:44.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:44.206+0000 7f25e4a00640 1 --2- 172.21.15.6:0/4283918969 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f25e00cb090 0x7f25e00d0820 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:44.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:44.206+0000 7f25e4a00640 1 -- 172.21.15.6:0/4283918969 >> 172.21.15.6:0/4283918969 conn(0x7f25e006be10 msgr2=0x7f25e006c1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:15:44.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:44.206+0000 7f25e4a00640 1 -- 172.21.15.6:0/4283918969 shutdown_connections 2024-04-24T23:15:44.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:44.206+0000 7f25e4a00640 1 -- 172.21.15.6:0/4283918969 wait complete. 2024-04-24T23:15:44.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:44.208+0000 7f25e4a00640 1 Processor -- start 2024-04-24T23:15:44.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:44.208+0000 7f25e4a00640 1 -- start start 2024-04-24T23:15:44.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:44.209+0000 7f25e4a00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f25e00cb090 0x7f25e0144680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:44.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:44.209+0000 7f25e4a00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f25e00d0d60 0x7f25e0151630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:44.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:44.209+0000 7f25e4a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f25e0154d90 0x7f25e00c5af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:44.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:44.209+0000 7f25e4a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f25e015aa30 con 0x7f25e0154d90 2024-04-24T23:15:44.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:44.209+0000 7f25e4a00640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f25e015b600 con 0x7f25e00cb090 2024-04-24T23:15:44.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:44.209+0000 7f25e4a00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f25e00cc590 con 0x7f25e00d0d60 2024-04-24T23:15:44.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:44.209+0000 7f25dfe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f25e0154d90 0x7f25e00c5af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:44.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:44.209+0000 7f25dfe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f25e0154d90 0x7f25e00c5af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57650/0 (socket says 172.21.15.6:57650) 2024-04-24T23:15:44.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:44.209+0000 7f25dfe00640 1 -- 172.21.15.6:0/3257734884 learned_addr learned my addr 172.21.15.6:0/3257734884 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:15:44.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:44.209+0000 7f25df400640 1 --2- 172.21.15.6:0/3257734884 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f25e00cb090 0x7f25e0144680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:44.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:44.209+0000 7f25dea00640 1 --2- 172.21.15.6:0/3257734884 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f25e00d0d60 0x7f25e0151630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:44.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:44.210+0000 7f25dea00640 1 -- 172.21.15.6:0/3257734884 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f25e00cb090 msgr2=0x7f25e0144680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:44.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:44.210+0000 7f25dea00640 1 --2- 172.21.15.6:0/3257734884 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f25e00cb090 0x7f25e0144680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:44.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:44.210+0000 7f25dea00640 1 -- 172.21.15.6:0/3257734884 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f25e0154d90 msgr2=0x7f25e00c5af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:44.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:44.210+0000 7f25dea00640 1 --2- 172.21.15.6:0/3257734884 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f25e0154d90 0x7f25e00c5af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:44.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:44.210+0000 7f25dea00640 1 -- 172.21.15.6:0/3257734884 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25e014b220 con 0x7f25e00d0d60 2024-04-24T23:15:44.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:44.210+0000 7f25dfe00640 1 --2- 172.21.15.6:0/3257734884 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f25e0154d90 0x7f25e00c5af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:15:44.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:44.210+0000 7f25dea00640 1 --2- 172.21.15.6:0/3257734884 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f25e00d0d60 0x7f25e0151630 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f25e00cde90 tx=0x7f25c80046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:15:44.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:44.210+0000 7f25d7400640 1 -- 172.21.15.6:0/3257734884 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25c8037040 con 0x7f25e00d0d60 2024-04-24T23:15:44.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:44.210+0000 7f25d7400640 1 -- 172.21.15.6:0/3257734884 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f25c8004250 con 0x7f25e00d0d60 2024-04-24T23:15:44.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:44.210+0000 7f25d7400640 1 -- 172.21.15.6:0/3257734884 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25c8003610 con 0x7f25e00d0d60 2024-04-24T23:15:44.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:44.210+0000 7f25e4a00640 1 -- 172.21.15.6:0/3257734884 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25e014b500 con 0x7f25e00d0d60 2024-04-24T23:15:44.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:44.210+0000 7f25e4a00640 1 -- 172.21.15.6:0/3257734884 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f25e014bac0 con 0x7f25e00d0d60 2024-04-24T23:15:44.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:44.211+0000 7f25e4a00640 1 -- 172.21.15.6:0/3257734884 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f25e0075b30 con 0x7f25e00d0d60 2024-04-24T23:15:44.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:44.212+0000 7f25d7400640 1 -- 172.21.15.6:0/3257734884 <== mon.2 v2:172.21.15.190:3301/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f25c80037b0 con 0x7f25e00d0d60 2024-04-24T23:15:44.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:44.213+0000 7f25d7400640 1 --2- 172.21.15.6:0/3257734884 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f25bc06be30 0x7f25bc06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:44.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:44.213+0000 7f25d7400640 1 -- 172.21.15.6:0/3257734884 <== mon.2 v2:172.21.15.190:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f25c80abe00 con 0x7f25e00d0d60 2024-04-24T23:15:44.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:44.214+0000 7f25df400640 1 --2- 172.21.15.6:0/3257734884 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f25bc06be30 0x7f25bc06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:44.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:44.214+0000 7f25df400640 1 --2- 172.21.15.6:0/3257734884 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f25bc06be30 0x7f25bc06e2f0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f25d0000f60 tx=0x7f25d0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:15:44.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:44.218+0000 7f25d7400640 1 -- 172.21.15.6:0/3257734884 <== mon.2 v2:172.21.15.190:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f25c8079d80 con 0x7f25e00d0d60 2024-04-24T23:15:44.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:44.423+0000 7f25e4a00640 1 -- 172.21.15.6:0/3257734884 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f25e014ecb0 con 0x7f25e00d0d60 2024-04-24T23:15:44.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:44.425+0000 7f25d7400640 1 -- 172.21.15.6:0/3257734884 <== mon.2 v2:172.21.15.190:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+18458 (secure 0 0 0) 0x7f25c807e080 con 0x7f25e00d0d60 2024-04-24T23:15:44.427 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:15:44.427 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":24,"fsid":"18bef68d-5e6d-4ec5-862b-b6f206e2588d","created":"2024-04-24T23:15:12.535072+0000","modified":"2024-04-24T23:15:41.465875+0000","last_up_change":"2024-04-24T23:15:23.773223+0000","last_in_change":"2024-04-24T23:15:14.905328+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T23:15:24.747441+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-24T23:15:34.051800+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-24T23:15:35.436870+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-24T23:15:39.034257+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"3b7a212d-d498-462e-90bf-ed493c803a09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":3022569437},{"type":"v1","addr":"172.21.15.6:6809","nonce":3022569437}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":3022569437},{"type":"v1","addr":"172.21.15.6:6811","nonce":3022569437}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":3022569437},{"type":"v1","addr":"172.21.15.6:6815","nonce":3022569437}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":3022569437},{"type":"v1","addr":"172.21.15.6:6813","nonce":3022569437}]},"public_addr":"172.21.15.6:6809/3022569437","cluster_addr":"172.21.15.6:6811/3022569437","heartbeat_back_addr":"172.21.15.6:6815/3022569437","heartbeat_front_addr":"172.21.15.6:6813/3022569437","state":["exists","up"]},{"osd":1,"uuid":"7c97f5a2-1735-4775-9f15-eb50099b19d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":3687836888},{"type":"v1","addr":"172.21.15.6:6817","nonce":3687836888}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":3687836888},{"type":"v1","addr":"172.21.15.6:6819","nonce":3687836888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":3687836888},{"type":"v1","addr":"172.21.15.6:6823","nonce":3687836888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":3687836888},{"type":"v1","addr":"172.21.15.6:6821","nonce":3687836888}]},"public_addr":"172.21.15.6:6817/3687836888","cluster_addr":"172.21.15.6:6819/3687836888","heartbeat_back_addr":"172.21.15.6:6823/3687836888","heartbeat_front_addr":"172.21.15.6:6821/3687836888","state":["exists","up"]},{"osd":2,"uuid":"d86bbc18-438c-4ec2-8018-ba7dd78f790b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":1020213724},{"type":"v1","addr":"172.21.15.6:6801","nonce":1020213724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":1020213724},{"type":"v1","addr":"172.21.15.6:6803","nonce":1020213724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":1020213724},{"type":"v1","addr":"172.21.15.6:6807","nonce":1020213724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":1020213724},{"type":"v1","addr":"172.21.15.6:6805","nonce":1020213724}]},"public_addr":"172.21.15.6:6801/1020213724","cluster_addr":"172.21.15.6:6803/1020213724","heartbeat_back_addr":"172.21.15.6:6807/1020213724","heartbeat_front_addr":"172.21.15.6:6805/1020213724","state":["exists","up"]},{"osd":3,"uuid":"3754ae16-4248-40dd-95f9-0f59c0eb5426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6824","nonce":3163883468},{"type":"v1","addr":"172.21.15.6:6825","nonce":3163883468}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":3163883468},{"type":"v1","addr":"172.21.15.6:6827","nonce":3163883468}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":3163883468},{"type":"v1","addr":"172.21.15.6:6831","nonce":3163883468}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":3163883468},{"type":"v1","addr":"172.21.15.6:6829","nonce":3163883468}]},"public_addr":"172.21.15.6:6825/3163883468","cluster_addr":"172.21.15.6:6827/3163883468","heartbeat_back_addr":"172.21.15.6:6831/3163883468","heartbeat_front_addr":"172.21.15.6:6829/3163883468","state":["exists","up"]},{"osd":4,"uuid":"8b2913a0-d006-479c-84ed-5b21e0e6c392","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":1989500934},{"type":"v1","addr":"172.21.15.190:6809","nonce":1989500934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":1989500934},{"type":"v1","addr":"172.21.15.190:6811","nonce":1989500934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":1989500934},{"type":"v1","addr":"172.21.15.190:6815","nonce":1989500934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":1989500934},{"type":"v1","addr":"172.21.15.190:6813","nonce":1989500934}]},"public_addr":"172.21.15.190:6809/1989500934","cluster_addr":"172.21.15.190:6811/1989500934","heartbeat_back_addr":"172.21.15.190:6815/1989500934","heartbeat_front_addr":"172.21.15.190:6813/1989500934","state":["exists","up"]},{"osd":5,"uuid":"499b79d0-0585-4dae-ab69-196a551793da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":3687548412},{"type":"v1","addr":"172.21.15.190:6801","nonce":3687548412}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":3687548412},{"type":"v1","addr":"172.21.15.190:6803","nonce":3687548412}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":3687548412},{"type":"v1","addr":"172.21.15.190:6807","nonce":3687548412}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":3687548412},{"type":"v1","addr":"172.21.15.190:6805","nonce":3687548412}]},"public_addr":"172.21.15.190:6801/3687548412","cluster_addr":"172.21.15.190:6803/3687548412","heartbeat_back_addr":"172.21.15.190:6807/3687548412","heartbeat_front_addr":"172.21.15.190:6805/3687548412","state":["exists","up"]},{"osd":6,"uuid":"6ba6df3b-4a6b-4a89-a890-0f79f387d4d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":870571564},{"type":"v1","addr":"172.21.15.190:6817","nonce":870571564}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":870571564},{"type":"v1","addr":"172.21.15.190:6819","nonce":870571564}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":870571564},{"type":"v1","addr":"172.21.15.190:6823","nonce":870571564}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":870571564},{"type":"v1","addr":"172.21.15.190:6821","nonce":870571564}]},"public_addr":"172.21.15.190:6817/870571564","cluster_addr":"172.21.15.190:6819/870571564","heartbeat_back_addr":"172.21.15.190:6823/870571564","heartbeat_front_addr":"172.21.15.190:6821/870571564","state":["exists","up"]},{"osd":7,"uuid":"e9668b1c-a000-4332-86e8-4efd73b02409","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":123639197},{"type":"v1","addr":"172.21.15.190:6825","nonce":123639197}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":123639197},{"type":"v1","addr":"172.21.15.190:6827","nonce":123639197}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":123639197},{"type":"v1","addr":"172.21.15.190:6831","nonce":123639197}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":123639197},{"type":"v1","addr":"172.21.15.190:6829","nonce":123639197}]},"public_addr":"172.21.15.190:6825/123639197","cluster_addr":"172.21.15.190:6827/123639197","heartbeat_back_addr":"172.21.15.190:6831/123639197","heartbeat_front_addr":"172.21.15.190:6829/123639197","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.321646+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.363873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.466963+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.531989+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:19.990707+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.043543+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.032492+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.115004+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T23:15:44.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:44.427+0000 7f25e4a00640 1 -- 172.21.15.6:0/3257734884 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f25bc06be30 msgr2=0x7f25bc06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:44.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:44.427+0000 7f25e4a00640 1 --2- 172.21.15.6:0/3257734884 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f25bc06be30 0x7f25bc06e2f0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f25d0000f60 tx=0x7f25d0009040 comp rx=0 tx=0).stop 2024-04-24T23:15:44.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:44.428+0000 7f25e4a00640 1 -- 172.21.15.6:0/3257734884 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f25e00d0d60 msgr2=0x7f25e0151630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:44.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:44.428+0000 7f25e4a00640 1 --2- 172.21.15.6:0/3257734884 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f25e00d0d60 0x7f25e0151630 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f25e00cde90 tx=0x7f25c80046c0 comp rx=0 tx=0).stop 2024-04-24T23:15:44.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:44.428+0000 7f25e4a00640 1 -- 172.21.15.6:0/3257734884 shutdown_connections 2024-04-24T23:15:44.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:44.428+0000 7f25e4a00640 1 --2- 172.21.15.6:0/3257734884 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f25bc06be30 0x7f25bc06e2f0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:44.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:44.428+0000 7f25e4a00640 1 --2- 172.21.15.6:0/3257734884 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f25e00d0d60 0x7f25e0151630 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:44.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:44.428+0000 7f25e4a00640 1 --2- 172.21.15.6:0/3257734884 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f25e0154d90 0x7f25e00c5af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:44.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:44.428+0000 7f25e4a00640 1 --2- 172.21.15.6:0/3257734884 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f25e00cb090 0x7f25e0144680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:44.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:44.428+0000 7f25e4a00640 1 -- 172.21.15.6:0/3257734884 >> 172.21.15.6:0/3257734884 conn(0x7f25e006be10 msgr2=0x7f25e00b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:15:44.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:44.428+0000 7f25e4a00640 1 -- 172.21.15.6:0/3257734884 shutdown_connections 2024-04-24T23:15:44.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:44.428+0000 7f25e4a00640 1 -- 172.21.15.6:0/3257734884 wait complete. 2024-04-24T23:15:45.052 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 5, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-24T23:15:36.400716+0000', 'modified': '2024-04-24T23:15:42.051162+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4306}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4306': {'gid': 4306, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.6:6835/489092393', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6833', 'nonce': 489092393}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 489092393}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1, 'qdb_leader': 4306, 'qdb_cluster': [4306]} 2024-04-24T23:15:45.052 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-24T23:15:45.052 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-04-24T23:15:45.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:45.269+0000 7f9b95800640 1 Processor -- start 2024-04-24T23:15:45.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:45.269+0000 7f9b95800640 1 -- start start 2024-04-24T23:15:45.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:45.271+0000 7f9b95800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f9b90159210 0x7f9b900c74a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:45.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:45.271+0000 7f9b95800640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9b900c8420 0x7f9b90162500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:45.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:45.271+0000 7f9b95800640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f9b90162a40 0x7f9b90162e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:45.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:45.271+0000 7f9b95800640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f9b900c6a50 con 0x7f9b90159210 2024-04-24T23:15:45.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:45.271+0000 7f9b95800640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f9b900c7010 con 0x7f9b90162a40 2024-04-24T23:15:45.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:45.271+0000 7f9b95800640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f9b900c89a0 con 0x7f9b900c8420 2024-04-24T23:15:45.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:45.271+0000 7f9b8ea00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f9b90159210 0x7f9b900c74a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:45.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:45.271+0000 7f9b8ea00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f9b90159210 0x7f9b900c74a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57678/0 (socket says 172.21.15.6:57678) 2024-04-24T23:15:45.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:45.271+0000 7f9b8ea00640 1 -- 172.21.15.6:0/302500472 learned_addr learned my addr 172.21.15.6:0/302500472 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:15:45.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:45.271+0000 7f9b8e000640 1 --2- 172.21.15.6:0/302500472 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9b900c8420 0x7f9b90162500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:45.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:45.271+0000 7f9b8f400640 1 --2- 172.21.15.6:0/302500472 >> v2:172.21.15.190:3300/0 conn(0x7f9b90162a40 0x7f9b90162e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:45.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:45.272+0000 7f9b8ea00640 1 -- 172.21.15.6:0/302500472 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9b900c8420 msgr2=0x7f9b90162500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:15:45.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:45.272+0000 7f9b8ea00640 1 --2- 172.21.15.6:0/302500472 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9b900c8420 0x7f9b90162500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:45.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:45.272+0000 7f9b8ea00640 1 -- 172.21.15.6:0/302500472 >> v2:172.21.15.190:3300/0 conn(0x7f9b90162a40 msgr2=0x7f9b90162e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:15:45.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:45.272+0000 7f9b8ea00640 1 --2- 172.21.15.6:0/302500472 >> v2:172.21.15.190:3300/0 conn(0x7f9b90162a40 0x7f9b90162e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:45.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:45.272+0000 7f9b8ea00640 1 -- 172.21.15.6:0/302500472 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b9006cdb0 con 0x7f9b90159210 2024-04-24T23:15:45.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:45.272+0000 7f9b8ea00640 1 --2- 172.21.15.6:0/302500472 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9b90159210 0x7f9b900c74a0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f9b900c66f0 tx=0x7f9b8002a260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=740e6ee2eeebf5e8 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:15:45.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:45.273+0000 7f9b8d600640 1 -- 172.21.15.6:0/302500472 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b80037070 con 0x7f9b90159210 2024-04-24T23:15:45.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:45.273+0000 7f9b8d600640 1 -- 172.21.15.6:0/302500472 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9b80032440 con 0x7f9b90159210 2024-04-24T23:15:45.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:45.273+0000 7f9b8d600640 1 -- 172.21.15.6:0/302500472 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b80032810 con 0x7f9b90159210 2024-04-24T23:15:45.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:45.273+0000 7f9b95800640 1 -- 172.21.15.6:0/302500472 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9b90159210 msgr2=0x7f9b900c74a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:45.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:45.273+0000 7f9b95800640 1 --2- 172.21.15.6:0/302500472 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9b90159210 0x7f9b900c74a0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f9b900c66f0 tx=0x7f9b8002a260 comp rx=0 tx=0).stop 2024-04-24T23:15:45.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:45.273+0000 7f9b95800640 1 -- 172.21.15.6:0/302500472 shutdown_connections 2024-04-24T23:15:45.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:45.274+0000 7f9b95800640 1 --2- 172.21.15.6:0/302500472 >> v2:172.21.15.190:3300/0 conn(0x7f9b90162a40 0x7f9b90162e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:45.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:45.274+0000 7f9b95800640 1 --2- 172.21.15.6:0/302500472 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9b900c8420 0x7f9b90162500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:45.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:45.274+0000 7f9b95800640 1 --2- 172.21.15.6:0/302500472 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9b90159210 0x7f9b900c74a0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:45.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:45.274+0000 7f9b95800640 1 -- 172.21.15.6:0/302500472 >> 172.21.15.6:0/302500472 conn(0x7f9b900c8c50 msgr2=0x7f9b900c9060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:15:45.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:45.274+0000 7f9b95800640 1 -- 172.21.15.6:0/302500472 shutdown_connections 2024-04-24T23:15:45.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:45.274+0000 7f9b95800640 1 -- 172.21.15.6:0/302500472 wait complete. 2024-04-24T23:15:45.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:45.275+0000 7f9b95800640 1 Processor -- start 2024-04-24T23:15:45.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:45.276+0000 7f9b95800640 1 -- start start 2024-04-24T23:15:45.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:45.276+0000 7f9b95800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9b900c8420 0x7f9b900cb720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:45.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:45.276+0000 7f9b95800640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9b90162a40 0x7f9b901f5750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:45.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:45.277+0000 7f9b95800640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9b901f83f0 0x7f9b901e8360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:45.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:45.277+0000 7f9b95800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f9b900c6ca0 con 0x7f9b900c8420 2024-04-24T23:15:45.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:45.277+0000 7f9b95800640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f9b9006d050 con 0x7f9b90162a40 2024-04-24T23:15:45.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:45.277+0000 7f9b95800640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f9b901595f0 con 0x7f9b901f83f0 2024-04-24T23:15:45.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:45.277+0000 7f9b8ea00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9b900c8420 0x7f9b900cb720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:45.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:45.277+0000 7f9b8ea00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9b900c8420 0x7f9b900cb720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57680/0 (socket says 172.21.15.6:57680) 2024-04-24T23:15:45.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:45.277+0000 7f9b8ea00640 1 -- 172.21.15.6:0/67076903 learned_addr learned my addr 172.21.15.6:0/67076903 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:15:45.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:45.277+0000 7f9b8e000640 1 --2- 172.21.15.6:0/67076903 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9b90162a40 0x7f9b901f5750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:45.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:45.277+0000 7f9b8f400640 1 --2- 172.21.15.6:0/67076903 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9b901f83f0 0x7f9b901e8360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:45.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:45.277+0000 7f9b8ea00640 1 -- 172.21.15.6:0/67076903 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9b901f83f0 msgr2=0x7f9b901e8360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:45.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:45.277+0000 7f9b8ea00640 1 --2- 172.21.15.6:0/67076903 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9b901f83f0 0x7f9b901e8360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:45.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:45.277+0000 7f9b8ea00640 1 -- 172.21.15.6:0/67076903 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9b90162a40 msgr2=0x7f9b901f5750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:45.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:45.277+0000 7f9b8ea00640 1 --2- 172.21.15.6:0/67076903 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9b90162a40 0x7f9b901f5750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:45.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:45.277+0000 7f9b8ea00640 1 -- 172.21.15.6:0/67076903 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b901f4f90 con 0x7f9b900c8420 2024-04-24T23:15:45.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:45.278+0000 7f9b8ea00640 1 --2- 172.21.15.6:0/67076903 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9b900c8420 0x7f9b900cb720 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f9b8002a7d0 tx=0x7f9b800046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:15:45.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:45.278+0000 7f9b86a00640 1 -- 172.21.15.6:0/67076903 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b80037040 con 0x7f9b900c8420 2024-04-24T23:15:45.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:45.278+0000 7f9b86a00640 1 -- 172.21.15.6:0/67076903 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9b80042070 con 0x7f9b900c8420 2024-04-24T23:15:45.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:45.278+0000 7f9b95800640 1 -- 172.21.15.6:0/67076903 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b901f89c0 con 0x7f9b900c8420 2024-04-24T23:15:45.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:45.278+0000 7f9b86a00640 1 -- 172.21.15.6:0/67076903 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b80030920 con 0x7f9b900c8420 2024-04-24T23:15:45.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:45.278+0000 7f9b95800640 1 -- 172.21.15.6:0/67076903 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b901f8f80 con 0x7f9b900c8420 2024-04-24T23:15:45.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:45.280+0000 7f9b86a00640 1 -- 172.21.15.6:0/67076903 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f9b80030ac0 con 0x7f9b900c8420 2024-04-24T23:15:45.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:45.281+0000 7f9b86a00640 1 --2- 172.21.15.6:0/67076903 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f9b7006be30 0x7f9b7006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:45.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:45.281+0000 7f9b86a00640 1 -- 172.21.15.6:0/67076903 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f9b800ab910 con 0x7f9b900c8420 2024-04-24T23:15:45.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:45.281+0000 7f9b8e000640 1 --2- 172.21.15.6:0/67076903 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f9b7006be30 0x7f9b7006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:45.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:45.282+0000 7f9b985eb740 1 Processor -- start 2024-04-24T23:15:45.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:45.282+0000 7f9b985eb740 1 -- start start 2024-04-24T23:15:45.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:45.282+0000 7f9b8e000640 1 --2- 172.21.15.6:0/67076903 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f9b7006be30 0x7f9b7006e2f0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f9b9015ca50 tx=0x7f9b7c009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:15:45.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:45.283+0000 7f9b985eb740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55af2e35f060 0x55af2e37a900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:45.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:45.283+0000 7f9b985eb740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55af2e331bf0 0x55af2e333b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:45.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:45.283+0000 7f9b985eb740 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x55af2e334040 0x55af2e35f440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:45.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:45.283+0000 7f9b985eb740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x55af2e235ed0 con 0x55af2e331bf0 2024-04-24T23:15:45.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:45.283+0000 7f9b985eb740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55af2e298df0 con 0x55af2e35f060 2024-04-24T23:15:45.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:45.283+0000 7f9b985eb740 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x55af2e2418d0 con 0x55af2e334040 2024-04-24T23:15:45.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:45.283+0000 7f9b8e000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55af2e331bf0 0x55af2e333b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:45.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:45.283+0000 7f9b8e000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55af2e331bf0 0x55af2e333b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57692/0 (socket says 172.21.15.6:57692) 2024-04-24T23:15:45.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:45.283+0000 7f9b8e000640 1 -- 172.21.15.6:0/1104377370 learned_addr learned my addr 172.21.15.6:0/1104377370 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:15:45.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:45.283+0000 7f9b8ea00640 1 --2- 172.21.15.6:0/1104377370 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55af2e35f060 0x55af2e37a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:45.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:45.283+0000 7f9b8f400640 1 --2- 172.21.15.6:0/1104377370 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x55af2e334040 0x55af2e35f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:45.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:45.284+0000 7f9b8e000640 1 -- 172.21.15.6:0/1104377370 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x55af2e334040 msgr2=0x55af2e35f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:15:45.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:45.284+0000 7f9b8e000640 1 --2- 172.21.15.6:0/1104377370 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x55af2e334040 0x55af2e35f440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:45.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:45.284+0000 7f9b8e000640 1 -- 172.21.15.6:0/1104377370 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55af2e35f060 msgr2=0x55af2e37a900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:15:45.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:45.284+0000 7f9b8e000640 1 --2- 172.21.15.6:0/1104377370 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55af2e35f060 0x55af2e37a900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:45.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:45.284+0000 7f9b8e000640 1 -- 172.21.15.6:0/1104377370 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55af2dfad2d0 con 0x55af2e331bf0 2024-04-24T23:15:45.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:45.284+0000 7f9b8ea00640 1 --2- 172.21.15.6:0/1104377370 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55af2e35f060 0x55af2e37a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:15:45.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:45.284+0000 7f9b8e000640 1 --2- 172.21.15.6:0/1104377370 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55af2e331bf0 0x55af2e333b00 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f9b7c004430 tx=0x7f9b7c014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed15e868be466d08 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:15:45.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:45.284+0000 7f9b7b400640 1 -- 172.21.15.6:0/1104377370 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b7c01b070 con 0x55af2e331bf0 2024-04-24T23:15:45.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:45.284+0000 7f9b7b400640 1 -- 172.21.15.6:0/1104377370 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9b7c027070 con 0x55af2e331bf0 2024-04-24T23:15:45.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:45.284+0000 7f9b7b400640 1 -- 172.21.15.6:0/1104377370 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b7c003a20 con 0x55af2e331bf0 2024-04-24T23:15:45.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:45.285+0000 7f9b985eb740 1 -- 172.21.15.6:0/1104377370 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55af2e331bf0 msgr2=0x55af2e333b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:45.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:45.285+0000 7f9b985eb740 1 --2- 172.21.15.6:0/1104377370 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55af2e331bf0 0x55af2e333b00 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f9b7c004430 tx=0x7f9b7c014000 comp rx=0 tx=0).stop 2024-04-24T23:15:45.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:45.285+0000 7f9b985eb740 1 -- 172.21.15.6:0/1104377370 shutdown_connections 2024-04-24T23:15:45.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:45.285+0000 7f9b985eb740 1 --2- 172.21.15.6:0/1104377370 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x55af2e334040 0x55af2e35f440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:45.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:45.285+0000 7f9b985eb740 1 --2- 172.21.15.6:0/1104377370 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55af2e331bf0 0x55af2e333b00 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:45.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:45.285+0000 7f9b985eb740 1 --2- 172.21.15.6:0/1104377370 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55af2e35f060 0x55af2e37a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:45.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:45.285+0000 7f9b985eb740 1 -- 172.21.15.6:0/1104377370 >> 172.21.15.6:0/1104377370 conn(0x55af2e332a80 msgr2=0x55af2e379f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:15:45.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:45.285+0000 7f9b985eb740 1 -- 172.21.15.6:0/1104377370 shutdown_connections 2024-04-24T23:15:45.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:45.285+0000 7f9b985eb740 1 -- 172.21.15.6:0/1104377370 wait complete. 2024-04-24T23:15:45.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:45.286+0000 7f9b985eb740 1 -- start start 2024-04-24T23:15:45.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:45.286+0000 7f9b985eb740 1 Processor -- start 2024-04-24T23:15:45.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:45.287+0000 7f9b985eb740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55af2e35f060 0x55af2e35db50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:45.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:45.287+0000 7f9b985eb740 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x55af2e377b20 0x55af2e378a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:45.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:45.287+0000 7f9b985eb740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55af2e35c7d0 0x55af2e35cc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:45.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:45.287+0000 7f9b985eb740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x55af2e254d20 con 0x55af2e35f060 2024-04-24T23:15:45.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:45.287+0000 7f9b985eb740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55af2e294db0 con 0x55af2e35c7d0 2024-04-24T23:15:45.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:45.287+0000 7f9b985eb740 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x55af2e332080 con 0x55af2e377b20 2024-04-24T23:15:45.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:45.287+0000 7f9b985eb740 10 client.0 _ll_register_callbacks cb 0x55af2e28ff10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T23:15:45.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:45.287+0000 7f9b8ea00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55af2e35f060 0x55af2e35db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:45.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:45.287+0000 7f9b8ea00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55af2e35f060 0x55af2e35db50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57696/0 (socket says 172.21.15.6:57696) 2024-04-24T23:15:45.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:45.287+0000 7f9b8ea00640 1 -- 172.21.15.6:0/3711169801 learned_addr learned my addr 172.21.15.6:0/3711169801 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:15:45.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:45.287+0000 7f9b8f400640 1 --2- 172.21.15.6:0/3711169801 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55af2e35c7d0 0x55af2e35cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:45.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:45.287+0000 7f9b8e000640 1 --2- 172.21.15.6:0/3711169801 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x55af2e377b20 0x55af2e378a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:45.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:45.288+0000 7f9b8ea00640 1 -- 172.21.15.6:0/3711169801 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x55af2e377b20 msgr2=0x55af2e378a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:15:45.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:45.288+0000 7f9b8ea00640 1 --2- 172.21.15.6:0/3711169801 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x55af2e377b20 0x55af2e378a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:45.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:45.288+0000 7f9b8ea00640 1 -- 172.21.15.6:0/3711169801 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55af2e35c7d0 msgr2=0x55af2e35cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:15:45.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:45.288+0000 7f9b8ea00640 1 --2- 172.21.15.6:0/3711169801 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55af2e35c7d0 0x55af2e35cc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:45.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:45.288+0000 7f9b8ea00640 1 -- 172.21.15.6:0/3711169801 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55af2e0776b0 con 0x55af2e35f060 2024-04-24T23:15:45.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:45.288+0000 7f9b8ea00640 1 --2- 172.21.15.6:0/3711169801 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55af2e35f060 0x55af2e35db50 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f9b80035040 tx=0x7f9b80079220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=893ab20d34167710 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:15:45.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:45.288+0000 7f9b6fe00640 10 client.0 ms_handle_connect on v2:172.21.15.6:3300/0 2024-04-24T23:15:45.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:45.288+0000 7f9b6fe00640 1 -- 172.21.15.6:0/3711169801 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b800823c0 con 0x55af2e35f060 2024-04-24T23:15:45.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:45.288+0000 7f9b6fe00640 1 -- 172.21.15.6:0/3711169801 <== mon.0 v2:172.21.15.6:3300/0 2 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f9b80081f30 con 0x55af2e35f060 2024-04-24T23:15:45.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:45.288+0000 7f9b985eb740 1 -- 172.21.15.6:0/3711169801 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9b7c027070 con 0x55af2e35f060 2024-04-24T23:15:45.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:45.288+0000 7f9b6fe00640 1 -- 172.21.15.6:0/3711169801 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_get_version_reply(handle=1 version=6) ==== 24+0+0 (secure 0 0 0) 0x7f9b80082150 con 0x55af2e35f060 2024-04-24T23:15:45.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:45.288+0000 7f9b985eb740 10 client.4332 fetch_fsmap learned FSMap version 6 2024-04-24T23:15:45.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:45.288+0000 7f9b985eb740 1 -- 172.21.15.6:0/3711169801 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,fsmap=6,monmap=2+}) -- 0x55af2e378f80 con 0x55af2e35f060 2024-04-24T23:15:45.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:45.289+0000 7f9b6fe00640 1 -- 172.21.15.6:0/3711169801 <== mon.0 v2:172.21.15.6:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9b7c027070 con 0x55af2e35f060 2024-04-24T23:15:45.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:45.289+0000 7f9b6fe00640 1 -- 172.21.15.6:0/3711169801 <== mon.0 v2:172.21.15.6:3300/0 5 ==== fsmap(e 6) ==== 4712+0+0 (secure 0 0 0) 0x7f9b8003a070 con 0x55af2e35f060 2024-04-24T23:15:45.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:45.289+0000 7f9b985eb740 10 client.4332 fetch_fsmap finished waiting for FSMap version 6 2024-04-24T23:15:45.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:45.289+0000 7f9b985eb740 10 client.4332 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-24T23:15:45.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:45.289+0000 7f9b985eb740 20 client.4332 populate_metadata read hostname 'smithi006' 2024-04-24T23:15:45.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:45.289+0000 7f9b985eb740 1 --2- 172.21.15.6:0/3711169801 >> [v2:172.21.15.6:6833/489092393,v1:172.21.15.6:6835/489092393] conn(0x55af2e393ee0 0x55af2e3962d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:45.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:45.289+0000 7f9b985eb740 4 client.4332 mds_command: new command op to 4306 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T23:15:45.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:45.289+0000 7f9b985eb740 1 -- 172.21.15.6:0/3711169801 --> [v2:172.21.15.6:6833/489092393,v1:172.21.15.6:6835/489092393] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55af2e2a6330 con 0x55af2e393ee0 2024-04-24T23:15:45.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:45.289+0000 7f9b8e000640 1 --2- 172.21.15.6:0/3711169801 >> [v2:172.21.15.6:6833/489092393,v1:172.21.15.6:6835/489092393] conn(0x55af2e393ee0 0x55af2e3962d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:45.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:45.290+0000 7f9b8e000640 1 --2- 172.21.15.6:0/3711169801 >> [v2:172.21.15.6:6833/489092393,v1:172.21.15.6:6835/489092393] conn(0x55af2e393ee0 0x55af2e3962d0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=80416efe8eb8a02b server_cookie=a81aebe3de8053e2 in_seq=0 out_seq=0 2024-04-24T23:15:45.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:45.290+0000 7f9b6fe00640 10 client.4332 ms_handle_connect on v2:172.21.15.6:6833/489092393 2024-04-24T23:15:47.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.058+0000 7f9b6fe00640 1 -- 172.21.15.6:0/3711169801 <== mds.0 v2:172.21.15.6:6833/489092393 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55af2e2a6330 con 0x55af2e393ee0 2024-04-24T23:15:47.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.058+0000 7f9b6fe00640 10 client.4332 handle_command_reply: tid=0 2024-04-24T23:15:47.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.058+0000 7f9b985eb740 1 client.4332 shutdown 2024-04-24T23:15:47.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.059+0000 7f9b985eb740 1 -- 172.21.15.6:0/3711169801 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55af2e35f060 msgr2=0x55af2e35db50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:47.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.059+0000 7f9b985eb740 1 --2- 172.21.15.6:0/3711169801 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55af2e35f060 0x55af2e35db50 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f9b80035040 tx=0x7f9b80079220 comp rx=0 tx=0).stop 2024-04-24T23:15:47.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.059+0000 7f9b985eb740 1 -- 172.21.15.6:0/3711169801 shutdown_connections 2024-04-24T23:15:47.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.059+0000 7f9b985eb740 1 --2- 172.21.15.6:0/3711169801 >> [v2:172.21.15.6:6833/489092393,v1:172.21.15.6:6835/489092393] conn(0x55af2e393ee0 0x55af2e3962d0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:47.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.059+0000 7f9b985eb740 1 --2- 172.21.15.6:0/3711169801 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x55af2e377b20 0x55af2e378a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:47.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.059+0000 7f9b6fe00640 0 client.4332 ms_handle_reset on v2:172.21.15.6:6833/489092393 2024-04-24T23:15:47.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.059+0000 7f9b985eb740 1 --2- 172.21.15.6:0/3711169801 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55af2e35c7d0 0x55af2e35cc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:47.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.059+0000 7f9b985eb740 1 --2- 172.21.15.6:0/3711169801 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55af2e35f060 0x55af2e35db50 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:47.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.059+0000 7f9b985eb740 1 -- 172.21.15.6:0/3711169801 >> 172.21.15.6:0/3711169801 conn(0x55af2e334040 msgr2=0x55af2e336850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:15:47.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.059+0000 7f9b985eb740 1 -- 172.21.15.6:0/3711169801 shutdown_connections 2024-04-24T23:15:47.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.059+0000 7f9b985eb740 1 -- 172.21.15.6:0/3711169801 wait complete. 2024-04-24T23:15:47.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.060+0000 7f9b985eb740 20 client.4332 trim_cache size 0 max 16384 2024-04-24T23:15:47.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.086+0000 7f9b985eb740 1 Processor -- start 2024-04-24T23:15:47.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.086+0000 7f9b985eb740 1 -- start start 2024-04-24T23:15:47.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.087+0000 7f9b985eb740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55af2e35c7d0 0x55af2e334470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:47.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.088+0000 7f9b985eb740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55af2e35f060 0x55af2e330580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:47.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.088+0000 7f9b985eb740 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x55af2e377b20 0x55af2e330ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:47.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.088+0000 7f9b985eb740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f9b90201de0 con 0x55af2e35c7d0 2024-04-24T23:15:47.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.088+0000 7f9b985eb740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55af2e294db0 con 0x55af2e35f060 2024-04-24T23:15:47.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.088+0000 7f9b985eb740 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x55af2e254d20 con 0x55af2e377b20 2024-04-24T23:15:47.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.088+0000 7f9b8ea00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55af2e35c7d0 0x55af2e334470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:47.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.088+0000 7f9b8ea00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55af2e35c7d0 0x55af2e334470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57726/0 (socket says 172.21.15.6:57726) 2024-04-24T23:15:47.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.088+0000 7f9b8f400640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x55af2e377b20 0x55af2e330ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:47.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.088+0000 7f9b8e000640 1 --2- 172.21.15.6:0/1979902629 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55af2e35f060 0x55af2e330580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:47.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.088+0000 7f9b8ea00640 1 -- 172.21.15.6:0/1979902629 learned_addr learned my addr 172.21.15.6:0/1979902629 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:15:47.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.088+0000 7f9b8ea00640 1 -- 172.21.15.6:0/1979902629 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x55af2e377b20 msgr2=0x55af2e330ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:15:47.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.088+0000 7f9b8ea00640 1 --2- 172.21.15.6:0/1979902629 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x55af2e377b20 0x55af2e330ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:47.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.088+0000 7f9b8ea00640 1 -- 172.21.15.6:0/1979902629 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55af2e35f060 msgr2=0x55af2e330580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:15:47.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.088+0000 7f9b8ea00640 1 --2- 172.21.15.6:0/1979902629 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55af2e35f060 0x55af2e330580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:47.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.089+0000 7f9b8ea00640 1 -- 172.21.15.6:0/1979902629 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55af2e389f50 con 0x55af2e35c7d0 2024-04-24T23:15:47.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.089+0000 7f9b8ea00640 1 --2- 172.21.15.6:0/1979902629 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55af2e35c7d0 0x55af2e334470 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f9b80079500 tx=0x7f9b8003f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89ec5a225073ccd8 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:15:47.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.089+0000 7f9b7aa00640 1 -- 172.21.15.6:0/1979902629 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b800823c0 con 0x55af2e35c7d0 2024-04-24T23:15:47.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.089+0000 7f9b7aa00640 1 -- 172.21.15.6:0/1979902629 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9b800778a0 con 0x55af2e35c7d0 2024-04-24T23:15:47.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.089+0000 7f9b7aa00640 1 -- 172.21.15.6:0/1979902629 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b80078c50 con 0x55af2e35c7d0 2024-04-24T23:15:47.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.090+0000 7f9b985eb740 1 -- 172.21.15.6:0/1979902629 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55af2e35c7d0 msgr2=0x55af2e334470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:47.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.090+0000 7f9b985eb740 1 --2- 172.21.15.6:0/1979902629 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55af2e35c7d0 0x55af2e334470 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f9b80079500 tx=0x7f9b8003f000 comp rx=0 tx=0).stop 2024-04-24T23:15:47.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.090+0000 7f9b985eb740 1 -- 172.21.15.6:0/1979902629 shutdown_connections 2024-04-24T23:15:47.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.090+0000 7f9b985eb740 1 --2- 172.21.15.6:0/1979902629 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x55af2e377b20 0x55af2e330ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:47.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.090+0000 7f9b985eb740 1 --2- 172.21.15.6:0/1979902629 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55af2e35f060 0x55af2e330580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:47.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.090+0000 7f9b985eb740 1 --2- 172.21.15.6:0/1979902629 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55af2e35c7d0 0x55af2e334470 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:47.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.090+0000 7f9b985eb740 1 -- 172.21.15.6:0/1979902629 >> 172.21.15.6:0/1979902629 conn(0x55af2e334040 msgr2=0x55af2e35cfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:15:47.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.090+0000 7f9b985eb740 1 -- 172.21.15.6:0/1979902629 shutdown_connections 2024-04-24T23:15:47.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.090+0000 7f9b985eb740 1 -- 172.21.15.6:0/1979902629 wait complete. 2024-04-24T23:15:47.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.091+0000 7f9b985eb740 1 -- start start 2024-04-24T23:15:47.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.091+0000 7f9b985eb740 1 Processor -- start 2024-04-24T23:15:47.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.092+0000 7f9b985eb740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55af2e35c7d0 0x55af2e372360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:47.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.092+0000 7f9b985eb740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55af2e35f060 0x55af2e330980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:47.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.092+0000 7f9b985eb740 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x55af2e377b20 0x55af2e3eb610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:47.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.092+0000 7f9b985eb740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x55af2e378520 con 0x55af2e35c7d0 2024-04-24T23:15:47.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.092+0000 7f9b985eb740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55af2e371560 con 0x55af2e35f060 2024-04-24T23:15:47.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.092+0000 7f9b985eb740 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x55af2e3712a0 con 0x55af2e377b20 2024-04-24T23:15:47.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.092+0000 7f9b985eb740 10 client.0 _ll_register_callbacks cb 0x7f9b80075b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T23:15:47.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.092+0000 7f9b8ea00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55af2e35c7d0 0x55af2e372360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:47.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.093+0000 7f9b8e000640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55af2e35f060 0x55af2e330980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:47.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.093+0000 7f9b8f400640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x55af2e377b20 0x55af2e3eb610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:47.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.093+0000 7f9b8ea00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55af2e35c7d0 0x55af2e372360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57736/0 (socket says 172.21.15.6:57736) 2024-04-24T23:15:47.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.093+0000 7f9b8ea00640 1 -- 172.21.15.6:0/1438274224 learned_addr learned my addr 172.21.15.6:0/1438274224 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:15:47.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.093+0000 7f9b8e000640 1 -- 172.21.15.6:0/1438274224 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x55af2e377b20 msgr2=0x55af2e3eb610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:15:47.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.093+0000 7f9b8e000640 1 --2- 172.21.15.6:0/1438274224 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x55af2e377b20 0x55af2e3eb610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:47.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.093+0000 7f9b8e000640 1 -- 172.21.15.6:0/1438274224 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55af2e35c7d0 msgr2=0x55af2e372360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:15:47.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.093+0000 7f9b8e000640 1 --2- 172.21.15.6:0/1438274224 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55af2e35c7d0 0x55af2e372360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:47.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.093+0000 7f9b8e000640 1 -- 172.21.15.6:0/1438274224 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55af2e379850 con 0x55af2e35f060 2024-04-24T23:15:47.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.093+0000 7f9b8ea00640 1 --2- 172.21.15.6:0/1438274224 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55af2e35c7d0 0x55af2e372360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:15:47.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.093+0000 7f9b8f400640 1 --2- 172.21.15.6:0/1438274224 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x55af2e377b20 0x55af2e3eb610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:15:47.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.093+0000 7f9b8e000640 1 --2- 172.21.15.6:0/1438274224 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55af2e35f060 0x55af2e330980 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f9b7c004400 tx=0x7f9b7c004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6b379a9ef233c52f server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:15:47.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.093+0000 7f9b6fe00640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-04-24T23:15:47.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.094+0000 7f9b6fe00640 1 -- 172.21.15.6:0/1438274224 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b7c01b040 con 0x55af2e35f060 2024-04-24T23:15:47.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.094+0000 7f9b6fe00640 1 -- 172.21.15.6:0/1438274224 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f9b7c038600 con 0x55af2e35f060 2024-04-24T23:15:47.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.094+0000 7f9b985eb740 1 -- 172.21.15.6:0/1438274224 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9b800778a0 con 0x55af2e35f060 2024-04-24T23:15:47.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.094+0000 7f9b6fe00640 1 -- 172.21.15.6:0/1438274224 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=6) ==== 24+0+0 (secure 0 0 0) 0x7f9b7c038800 con 0x55af2e35f060 2024-04-24T23:15:47.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.094+0000 7f9b985eb740 10 client.4447 fetch_fsmap learned FSMap version 6 2024-04-24T23:15:47.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.094+0000 7f9b985eb740 1 -- 172.21.15.6:0/1438274224 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=6,monmap=2+}) -- 0x55af2e3f52f0 con 0x55af2e35f060 2024-04-24T23:15:47.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.094+0000 7f9b6fe00640 1 -- 172.21.15.6:0/1438274224 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9b800778a0 con 0x55af2e35f060 2024-04-24T23:15:47.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.094+0000 7f9b6fe00640 1 -- 172.21.15.6:0/1438274224 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 6) ==== 4712+0+0 (secure 0 0 0) 0x7f9b7c0039e0 con 0x55af2e35f060 2024-04-24T23:15:47.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.094+0000 7f9b985eb740 10 client.4447 fetch_fsmap finished waiting for FSMap version 6 2024-04-24T23:15:47.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.094+0000 7f9b985eb740 10 client.4447 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-24T23:15:47.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.094+0000 7f9b985eb740 20 client.4447 populate_metadata read hostname 'smithi006' 2024-04-24T23:15:47.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.095+0000 7f9b985eb740 1 --2- 172.21.15.6:0/1438274224 >> [v2:172.21.15.6:6833/489092393,v1:172.21.15.6:6835/489092393] conn(0x55af2e3f23b0 0x55af2e3fe190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:47.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.095+0000 7f9b985eb740 4 client.4447 mds_command: new command op to 4306 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-24T23:15:47.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.095+0000 7f9b985eb740 1 -- 172.21.15.6:0/1438274224 --> [v2:172.21.15.6:6833/489092393,v1:172.21.15.6:6835/489092393] -- command(tid 0: {"prefix": "status"}) -- 0x55af2e332350 con 0x55af2e3f23b0 2024-04-24T23:15:47.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.095+0000 7f9b8ea00640 1 --2- 172.21.15.6:0/1438274224 >> [v2:172.21.15.6:6833/489092393,v1:172.21.15.6:6835/489092393] conn(0x55af2e3f23b0 0x55af2e3fe190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:47.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.095+0000 7f9b8ea00640 1 --2- 172.21.15.6:0/1438274224 >> [v2:172.21.15.6:6833/489092393,v1:172.21.15.6:6835/489092393] conn(0x55af2e3f23b0 0x55af2e3fe190 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=81d9dae003fc357d server_cookie=86d36ba429345f3f in_seq=0 out_seq=0 2024-04-24T23:15:47.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.095+0000 7f9b6fe00640 10 client.4447 ms_handle_connect on v2:172.21.15.6:6833/489092393 2024-04-24T23:15:47.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.096+0000 7f9b6fe00640 1 -- 172.21.15.6:0/1438274224 <== mds.0 v2:172.21.15.6:6833/489092393 1 ==== command_reply(tid 0: 0 ) ==== 8+0+324 (crc 0 0 0) 0x55af2e332350 con 0x55af2e3f23b0 2024-04-24T23:15:47.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.096+0000 7f9b6fe00640 10 client.4447 handle_command_reply: tid=0 2024-04-24T23:15:47.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.096+0000 7f9b985eb740 1 client.4447 shutdown 2024-04-24T23:15:47.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.096+0000 7f9b985eb740 1 -- 172.21.15.6:0/1438274224 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55af2e35f060 msgr2=0x55af2e330980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:47.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.096+0000 7f9b985eb740 1 --2- 172.21.15.6:0/1438274224 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55af2e35f060 0x55af2e330980 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f9b7c004400 tx=0x7f9b7c004730 comp rx=0 tx=0).stop 2024-04-24T23:15:47.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.096+0000 7f9b985eb740 1 -- 172.21.15.6:0/1438274224 shutdown_connections 2024-04-24T23:15:47.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.096+0000 7f9b985eb740 1 --2- 172.21.15.6:0/1438274224 >> [v2:172.21.15.6:6833/489092393,v1:172.21.15.6:6835/489092393] conn(0x55af2e3f23b0 0x55af2e3fe190 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:47.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.096+0000 7f9b985eb740 1 --2- 172.21.15.6:0/1438274224 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x55af2e377b20 0x55af2e3eb610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:47.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.096+0000 7f9b985eb740 1 --2- 172.21.15.6:0/1438274224 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55af2e35f060 0x55af2e330980 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:47.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.096+0000 7f9b985eb740 1 --2- 172.21.15.6:0/1438274224 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55af2e35c7d0 0x55af2e372360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:47.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.096+0000 7f9b6fe00640 0 client.4447 ms_handle_reset on v2:172.21.15.6:6833/489092393 2024-04-24T23:15:47.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.096+0000 7f9b985eb740 1 -- 172.21.15.6:0/1438274224 >> 172.21.15.6:0/1438274224 conn(0x55af2e334040 msgr2=0x55af2e337190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:15:47.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.096+0000 7f9b985eb740 1 -- 172.21.15.6:0/1438274224 shutdown_connections 2024-04-24T23:15:47.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.097+0000 7f9b985eb740 1 -- 172.21.15.6:0/1438274224 wait complete. 2024-04-24T23:15:47.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.097+0000 7f9b985eb740 20 client.4447 trim_cache size 0 max 16384 2024-04-24T23:15:47.102 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-04-24T23:15:47.102 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_fsid": "18bef68d-5e6d-4ec5-862b-b6f206e2588d", 2024-04-24T23:15:47.102 INFO:teuthology.orchestra.run.smithi006.stdout: "whoami": 0, 2024-04-24T23:15:47.102 INFO:teuthology.orchestra.run.smithi006.stdout: "id": 4306, 2024-04-24T23:15:47.102 INFO:teuthology.orchestra.run.smithi006.stdout: "want_state": "up:active", 2024-04-24T23:15:47.102 INFO:teuthology.orchestra.run.smithi006.stdout: "state": "up:active", 2024-04-24T23:15:47.102 INFO:teuthology.orchestra.run.smithi006.stdout: "fs_name": "cephfs", 2024-04-24T23:15:47.102 INFO:teuthology.orchestra.run.smithi006.stdout: "rank_uptime": 10.125677189999999, 2024-04-24T23:15:47.102 INFO:teuthology.orchestra.run.smithi006.stdout: "mdsmap_epoch": 5, 2024-04-24T23:15:47.102 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap_epoch": 24, 2024-04-24T23:15:47.102 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap_epoch_barrier": 24, 2024-04-24T23:15:47.102 INFO:teuthology.orchestra.run.smithi006.stdout: "uptime": 16.159767785 2024-04-24T23:15:47.103 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-04-24T23:15:47.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.097+0000 7f9b95800640 1 -- 172.21.15.6:0/67076903 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f9b7006be30 msgr2=0x7f9b7006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:47.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.097+0000 7f9b95800640 1 --2- 172.21.15.6:0/67076903 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f9b7006be30 0x7f9b7006e2f0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f9b9015ca50 tx=0x7f9b7c009190 comp rx=0 tx=0).stop 2024-04-24T23:15:47.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.097+0000 7f9b95800640 1 -- 172.21.15.6:0/67076903 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9b900c8420 msgr2=0x7f9b900cb720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:47.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.097+0000 7f9b95800640 1 --2- 172.21.15.6:0/67076903 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9b900c8420 0x7f9b900cb720 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f9b8002a7d0 tx=0x7f9b800046c0 comp rx=0 tx=0).stop 2024-04-24T23:15:47.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.097+0000 7f9b95800640 1 -- 172.21.15.6:0/67076903 shutdown_connections 2024-04-24T23:15:47.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.097+0000 7f9b95800640 1 --2- 172.21.15.6:0/67076903 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f9b7006be30 0x7f9b7006e2f0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:47.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.097+0000 7f9b95800640 1 --2- 172.21.15.6:0/67076903 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9b901f83f0 0x7f9b901e8360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:47.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.097+0000 7f9b95800640 1 --2- 172.21.15.6:0/67076903 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9b90162a40 0x7f9b901f5750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:47.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.098+0000 7f9b95800640 1 --2- 172.21.15.6:0/67076903 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9b900c8420 0x7f9b900cb720 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:47.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.098+0000 7f9b95800640 1 -- 172.21.15.6:0/67076903 >> 172.21.15.6:0/67076903 conn(0x7f9b900c8c50 msgr2=0x7f9b90157b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:15:47.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.098+0000 7f9b95800640 1 -- 172.21.15.6:0/67076903 shutdown_connections 2024-04-24T23:15:47.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.098+0000 7f9b95800640 1 -- 172.21.15.6:0/67076903 wait complete. 2024-04-24T23:15:47.119 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-24T23:15:47.119 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-04-24T23:15:47.119 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-24T23:15:47.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.316+0000 7fe8ac800640 1 Processor -- start 2024-04-24T23:15:47.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.316+0000 7fe8ac800640 1 -- start start 2024-04-24T23:15:47.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.317+0000 7fe8ac800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fe8a4152bc0 0x7fe8a4159390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:47.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.318+0000 7fe8ac800640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fe8a40c8f50 0x7fe8a415bf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:47.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.318+0000 7fe8ac800640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe8a40c6e70 con 0x7fe8a40c9360 2024-04-24T23:15:47.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.318+0000 7fe8ac800640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7fe8a40c73d0 con 0x7fe8a4152bc0 2024-04-24T23:15:47.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.318+0000 7fe8ac800640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7fe8a40c8ac0 con 0x7fe8a40c8f50 2024-04-24T23:15:47.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.318+0000 7fe8a9e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fe8a4152bc0 0x7fe8a4159390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:47.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.318+0000 7fe8a9e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fe8a4152bc0 0x7fe8a4159390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57738/0 (socket says 172.21.15.6:57738) 2024-04-24T23:15:47.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.318+0000 7fe8a9e00640 1 -- 172.21.15.6:0/3980193090 learned_addr learned my addr 172.21.15.6:0/3980193090 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:15:47.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.318+0000 7fe8a9400640 1 --2- 172.21.15.6:0/3980193090 >> v2:172.21.15.190:3300/0 conn(0x7fe8a40c8f50 0x7fe8a415bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:47.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.318+0000 7fe8a9e00640 1 -- 172.21.15.6:0/3980193090 >> v2:172.21.15.190:3300/0 conn(0x7fe8a40c8f50 msgr2=0x7fe8a415bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:15:47.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.318+0000 7fe8a9e00640 1 --2- 172.21.15.6:0/3980193090 >> v2:172.21.15.190:3300/0 conn(0x7fe8a40c8f50 0x7fe8a415bf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:47.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.318+0000 7fe8a9e00640 1 -- 172.21.15.6:0/3980193090 >> v1:172.21.15.190:6789/0 conn(0x7fe8a40c9360 legacy=0x7fe8a415c440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:47.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.318+0000 7fe8a9e00640 1 -- 172.21.15.6:0/3980193090 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8a41598d0 con 0x7fe8a4152bc0 2024-04-24T23:15:47.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.319+0000 7fe8a9e00640 1 --2- 172.21.15.6:0/3980193090 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe8a4152bc0 0x7fe8a4159390 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fe8a40c7550 tx=0x7fe89802a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b242fcd24ebf9064 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:15:47.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.320+0000 7fe8a8a00640 1 -- 172.21.15.6:0/3980193090 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe898037070 con 0x7fe8a4152bc0 2024-04-24T23:15:47.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.320+0000 7fe8a8a00640 1 -- 172.21.15.6:0/3980193090 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe898032440 con 0x7fe8a4152bc0 2024-04-24T23:15:47.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.320+0000 7fe8a8a00640 1 -- 172.21.15.6:0/3980193090 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe898032810 con 0x7fe8a4152bc0 2024-04-24T23:15:47.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.320+0000 7fe8ac800640 1 -- 172.21.15.6:0/3980193090 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe8a4152bc0 msgr2=0x7fe8a4159390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:47.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.321+0000 7fe8ac800640 1 --2- 172.21.15.6:0/3980193090 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe8a4152bc0 0x7fe8a4159390 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fe8a40c7550 tx=0x7fe89802a1e0 comp rx=0 tx=0).stop 2024-04-24T23:15:47.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.321+0000 7fe8ac800640 1 -- 172.21.15.6:0/3980193090 shutdown_connections 2024-04-24T23:15:47.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.321+0000 7fe8ac800640 1 --2- 172.21.15.6:0/3980193090 >> v2:172.21.15.190:3300/0 conn(0x7fe8a40c8f50 0x7fe8a415bf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:47.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.321+0000 7fe8ac800640 1 --2- 172.21.15.6:0/3980193090 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe8a4152bc0 0x7fe8a4159390 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:47.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.321+0000 7fe8ac800640 1 -- 172.21.15.6:0/3980193090 >> 172.21.15.6:0/3980193090 conn(0x7fe8a406b0e0 msgr2=0x7fe8a406b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:15:47.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.321+0000 7fe8ac800640 1 -- 172.21.15.6:0/3980193090 shutdown_connections 2024-04-24T23:15:47.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.321+0000 7fe8ac800640 1 -- 172.21.15.6:0/3980193090 wait complete. 2024-04-24T23:15:47.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.323+0000 7fe8ac800640 1 Processor -- start 2024-04-24T23:15:47.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.323+0000 7fe8ac800640 1 -- start start 2024-04-24T23:15:47.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.324+0000 7fe8ac800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe8a40c8f50 0x7fe8a414e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:47.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.324+0000 7fe8ac800640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe8a40c9360 0x7fe8a41481a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:47.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.324+0000 7fe8ac800640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe8a4152bc0 0x7fe8a413e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:47.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.324+0000 7fe8ac800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fe8a40c8470 con 0x7fe8a40c8f50 2024-04-24T23:15:47.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.324+0000 7fe8ac800640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fe8a40c6700 con 0x7fe8a4152bc0 2024-04-24T23:15:47.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.324+0000 7fe8ac800640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fe8a40c8750 con 0x7fe8a40c9360 2024-04-24T23:15:47.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.325+0000 7fe8a9e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe8a40c8f50 0x7fe8a414e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:47.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.325+0000 7fe8aa800640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe8a4152bc0 0x7fe8a413e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:47.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.325+0000 7fe8a9400640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe8a40c9360 0x7fe8a41481a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:47.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.325+0000 7fe8a9e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe8a40c8f50 0x7fe8a414e910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57740/0 (socket says 172.21.15.6:57740) 2024-04-24T23:15:47.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.325+0000 7fe8a9400640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe8a40c9360 0x7fe8a41481a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.6:46278/0 (socket says 172.21.15.6:46278) 2024-04-24T23:15:47.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.325+0000 7fe8a9e00640 1 -- 172.21.15.6:0/2315437769 learned_addr learned my addr 172.21.15.6:0/2315437769 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:15:47.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.325+0000 7fe8aa800640 1 -- 172.21.15.6:0/2315437769 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe8a40c9360 msgr2=0x7fe8a41481a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:47.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.325+0000 7fe8aa800640 1 --2- 172.21.15.6:0/2315437769 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe8a40c9360 0x7fe8a41481a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:47.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.325+0000 7fe8aa800640 1 -- 172.21.15.6:0/2315437769 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe8a40c8f50 msgr2=0x7fe8a414e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:47.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.325+0000 7fe8aa800640 1 --2- 172.21.15.6:0/2315437769 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe8a40c8f50 0x7fe8a414e910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:47.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.325+0000 7fe8aa800640 1 -- 172.21.15.6:0/2315437769 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8a4072e10 con 0x7fe8a4152bc0 2024-04-24T23:15:47.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.325+0000 7fe8aa800640 1 --2- 172.21.15.6:0/2315437769 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe8a4152bc0 0x7fe8a413e9e0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fe8a000c730 tx=0x7fe8a000cbf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:15:47.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.326+0000 7fe89e000640 1 -- 172.21.15.6:0/2315437769 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8a0003b70 con 0x7fe8a4152bc0 2024-04-24T23:15:47.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.326+0000 7fe89e000640 1 -- 172.21.15.6:0/2315437769 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe8a0003d10 con 0x7fe8a4152bc0 2024-04-24T23:15:47.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.326+0000 7fe89e000640 1 -- 172.21.15.6:0/2315437769 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8a0005310 con 0x7fe8a4152bc0 2024-04-24T23:15:47.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.326+0000 7fe8ac800640 1 -- 172.21.15.6:0/2315437769 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8a414d0b0 con 0x7fe8a4152bc0 2024-04-24T23:15:47.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.326+0000 7fe8ac800640 1 -- 172.21.15.6:0/2315437769 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8a414e110 con 0x7fe8a4152bc0 2024-04-24T23:15:47.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.327+0000 7fe8ac800640 1 -- 172.21.15.6:0/2315437769 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe8a40c6460 con 0x7fe8a4152bc0 2024-04-24T23:15:47.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.328+0000 7fe89e000640 1 -- 172.21.15.6:0/2315437769 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fe8a00056c0 con 0x7fe8a4152bc0 2024-04-24T23:15:47.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.329+0000 7fe89e000640 1 --2- 172.21.15.6:0/2315437769 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fe88806be60 0x7fe88806e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:47.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.329+0000 7fe89e000640 1 -- 172.21.15.6:0/2315437769 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fe8a008b100 con 0x7fe8a4152bc0 2024-04-24T23:15:47.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.329+0000 7fe8a9e00640 1 --2- 172.21.15.6:0/2315437769 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fe88806be60 0x7fe88806e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:47.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.330+0000 7fe8a9e00640 1 --2- 172.21.15.6:0/2315437769 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fe88806be60 0x7fe88806e320 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fe898035040 tx=0x7fe89802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:15:47.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.334+0000 7fe89e000640 1 -- 172.21.15.6:0/2315437769 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fe8a40c6460 con 0x7fe8a4152bc0 2024-04-24T23:15:47.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.579+0000 7fe8ac800640 1 -- 172.21.15.6:0/2315437769 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fe8a41596b0 con 0x7fe8a4152bc0 2024-04-24T23:15:47.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.580+0000 7fe89e000640 1 -- 172.21.15.6:0/2315437769 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v6) ==== 53+0+88 (secure 0 0 0) 0x7fe8a0058ff0 con 0x7fe8a4152bc0 2024-04-24T23:15:47.583 INFO:teuthology.orchestra.run.smithi006.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-04-24T23:15:47.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.583+0000 7fe8ac800640 1 -- 172.21.15.6:0/2315437769 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fe88806be60 msgr2=0x7fe88806e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:47.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.583+0000 7fe8ac800640 1 --2- 172.21.15.6:0/2315437769 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fe88806be60 0x7fe88806e320 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fe898035040 tx=0x7fe89802e000 comp rx=0 tx=0).stop 2024-04-24T23:15:47.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.583+0000 7fe8ac800640 1 -- 172.21.15.6:0/2315437769 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe8a4152bc0 msgr2=0x7fe8a413e9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:47.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.583+0000 7fe8ac800640 1 --2- 172.21.15.6:0/2315437769 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe8a4152bc0 0x7fe8a413e9e0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fe8a000c730 tx=0x7fe8a000cbf0 comp rx=0 tx=0).stop 2024-04-24T23:15:47.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.583+0000 7fe8ac800640 1 -- 172.21.15.6:0/2315437769 shutdown_connections 2024-04-24T23:15:47.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.583+0000 7fe8ac800640 1 --2- 172.21.15.6:0/2315437769 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fe88806be60 0x7fe88806e320 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:47.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.583+0000 7fe8ac800640 1 --2- 172.21.15.6:0/2315437769 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe8a40c9360 0x7fe8a41481a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:47.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.583+0000 7fe8ac800640 1 --2- 172.21.15.6:0/2315437769 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe8a4152bc0 0x7fe8a413e9e0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:47.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.583+0000 7fe8ac800640 1 --2- 172.21.15.6:0/2315437769 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe8a40c8f50 0x7fe8a414e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:47.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.583+0000 7fe8ac800640 1 -- 172.21.15.6:0/2315437769 >> 172.21.15.6:0/2315437769 conn(0x7fe8a406b0e0 msgr2=0x7fe8a40cb3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:15:47.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.583+0000 7fe8ac800640 1 -- 172.21.15.6:0/2315437769 shutdown_connections 2024-04-24T23:15:47.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47.584+0000 7fe8ac800640 1 -- 172.21.15.6:0/2315437769 wait complete. 2024-04-24T23:15:47.606 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-04-24T23:15:47.606 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-24T23:15:47.606 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi006.front.sepia.ceph.com 2024-04-24T23:15:47.606 INFO:tasks.cephfs.mount:self.client.name = client.admin 2024-04-24T23:15:47.606 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-04-24T23:15:47.606 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-04-24T23:15:47.606 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-24T23:15:47.606 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-24T23:15:47.606 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2024-04-24T23:15:47.606 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:15:47.607 DEBUG:teuthology.orchestra.run.smithi006:> ip addr 2024-04-24T23:15:47.660 INFO:teuthology.orchestra.run.smithi006.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-24T23:15:47.660 INFO:teuthology.orchestra.run.smithi006.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-24T23:15:47.660 INFO:teuthology.orchestra.run.smithi006.stdout: inet 127.0.0.1/8 scope host lo 2024-04-24T23:15:47.660 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-04-24T23:15:47.660 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 ::1/128 scope host 2024-04-24T23:15:47.661 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-04-24T23:15:47.661 INFO:teuthology.orchestra.run.smithi006.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-24T23:15:47.661 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:6b:fe:e6 brd ff:ff:ff:ff:ff:ff 2024-04-24T23:15:47.661 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp7s0f0 2024-04-24T23:15:47.661 INFO:teuthology.orchestra.run.smithi006.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-24T23:15:47.661 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:6b:fe:e7 brd ff:ff:ff:ff:ff:ff 2024-04-24T23:15:47.661 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp7s0f1 2024-04-24T23:15:47.661 INFO:teuthology.orchestra.run.smithi006.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-24T23:15:47.661 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:bd:15:e4 brd ff:ff:ff:ff:ff:ff 2024-04-24T23:15:47.661 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp1s0f0 2024-04-24T23:15:47.661 INFO:teuthology.orchestra.run.smithi006.stdout: inet 172.21.15.6/20 brd 172.21.15.255 scope global dynamic ens1f0 2024-04-24T23:15:47.661 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft 42693sec preferred_lft 42693sec 2024-04-24T23:15:47.661 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 fe80::ec4:7aff:febd:15e4/64 scope link noprefixroute 2024-04-24T23:15:47.661 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-04-24T23:15:47.662 INFO:teuthology.orchestra.run.smithi006.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-24T23:15:47.662 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:bd:15:e5 brd ff:ff:ff:ff:ff:ff 2024-04-24T23:15:47.662 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp1s0f1 2024-04-24T23:15:47.663 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-24T23:15:47.663 DEBUG:teuthology.orchestra.run.smithi006:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-24T23:15:47.663 DEBUG:teuthology.orchestra.run.smithi006:> set -e 2024-04-24T23:15:47.663 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip link add name ceph-brx type bridge 2024-04-24T23:15:47.664 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip addr flush dev ceph-brx 2024-04-24T23:15:47.664 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip link set ceph-brx up 2024-04-24T23:15:47.664 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-24T23:15:47.664 DEBUG:teuthology.orchestra.run.smithi006:> ') 2024-04-24T23:15:47.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:47 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-24T23:15:48.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:48 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T23:15:48.174 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:15:48.174 DEBUG:teuthology.orchestra.run.smithi006:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-24T23:15:48.242 INFO:teuthology.orchestra.run.smithi006.stdout:1 2024-04-24T23:15:48.247 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:15:48.247 DEBUG:teuthology.orchestra.run.smithi006:> route 2024-04-24T23:15:48.305 INFO:teuthology.orchestra.run.smithi006.stdout:Kernel IP routing table 2024-04-24T23:15:48.305 INFO:teuthology.orchestra.run.smithi006.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-24T23:15:48.305 INFO:teuthology.orchestra.run.smithi006.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-04-24T23:15:48.305 INFO:teuthology.orchestra.run.smithi006.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-04-24T23:15:48.306 INFO:teuthology.orchestra.run.smithi006.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-24T23:15:48.307 DEBUG:teuthology.orchestra.run.smithi006:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-24T23:15:48.307 DEBUG:teuthology.orchestra.run.smithi006:> set -e 2024-04-24T23:15:48.307 DEBUG:teuthology.orchestra.run.smithi006:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-04-24T23:15:48.307 DEBUG:teuthology.orchestra.run.smithi006:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-04-24T23:15:48.307 DEBUG:teuthology.orchestra.run.smithi006:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-04-24T23:15:48.307 DEBUG:teuthology.orchestra.run.smithi006:> ') 2024-04-24T23:15:48.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:48 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-24T23:15:49.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:49 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T23:15:49.875 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:15:49.875 DEBUG:teuthology.orchestra.run.smithi006:> ip netns list 2024-04-24T23:15:49.925 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:15:49.926 DEBUG:teuthology.orchestra.run.smithi006:> ip netns list-id 2024-04-24T23:15:49.979 DEBUG:teuthology.orchestra.run.smithi006:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-24T23:15:49.979 DEBUG:teuthology.orchestra.run.smithi006:> set -e 2024-04-24T23:15:49.979 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-24T23:15:49.979 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2024-04-24T23:15:49.979 DEBUG:teuthology.orchestra.run.smithi006:> ') 2024-04-24T23:15:50.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:50 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-24T23:15:50.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:50 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T23:15:50.160 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2024-04-24T23:15:50.160 DEBUG:teuthology.orchestra.run.smithi006:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-24T23:15:50.160 DEBUG:teuthology.orchestra.run.smithi006:> set -e 2024-04-24T23:15:50.160 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2024-04-24T23:15:50.160 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-24T23:15:50.160 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2024-04-24T23:15:50.160 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2024-04-24T23:15:50.160 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2024-04-24T23:15:50.160 DEBUG:teuthology.orchestra.run.smithi006:> ') 2024-04-24T23:15:50.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:50 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-24T23:15:50.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:50 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T23:15:50.442 DEBUG:teuthology.orchestra.run.smithi006:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-24T23:15:50.442 DEBUG:teuthology.orchestra.run.smithi006:> set -e 2024-04-24T23:15:50.442 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip link set brx.0 up 2024-04-24T23:15:50.442 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip link set dev brx.0 master ceph-brx 2024-04-24T23:15:50.442 DEBUG:teuthology.orchestra.run.smithi006:> ') 2024-04-24T23:15:50.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:50 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-24T23:15:50.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:50 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T23:15:50.645 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2024-04-24T23:15:50.646 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-24T23:15:50.646 DEBUG:teuthology.orchestra.run.smithi006:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2024-04-24T23:15:50.690 INFO:teuthology.orchestra.run.smithi006.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2024-04-24T23:15:50.690 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-24T23:15:50.690 DEBUG:teuthology.orchestra.run.smithi006:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2024-04-24T23:15:50.785 DEBUG:teuthology.orchestra.run.smithi006:> sudo modprobe fuse 2024-04-24T23:15:50.876 DEBUG:teuthology.orchestra.run.smithi006:> cat /proc/self/mounts | awk '{print $2}' 2024-04-24T23:15:50.927 INFO:teuthology.orchestra.run.smithi006.stdout:/proc 2024-04-24T23:15:50.927 INFO:teuthology.orchestra.run.smithi006.stdout:/sys 2024-04-24T23:15:50.927 INFO:teuthology.orchestra.run.smithi006.stdout:/dev 2024-04-24T23:15:50.927 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/kernel/security 2024-04-24T23:15:50.927 INFO:teuthology.orchestra.run.smithi006.stdout:/dev/shm 2024-04-24T23:15:50.927 INFO:teuthology.orchestra.run.smithi006.stdout:/dev/pts 2024-04-24T23:15:50.928 INFO:teuthology.orchestra.run.smithi006.stdout:/run 2024-04-24T23:15:50.928 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/fs/cgroup 2024-04-24T23:15:50.928 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/fs/pstore 2024-04-24T23:15:50.928 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/fs/bpf 2024-04-24T23:15:50.928 INFO:teuthology.orchestra.run.smithi006.stdout:/ 2024-04-24T23:15:50.928 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/fs/selinux 2024-04-24T23:15:50.928 INFO:teuthology.orchestra.run.smithi006.stdout:/proc/sys/fs/binfmt_misc 2024-04-24T23:15:50.928 INFO:teuthology.orchestra.run.smithi006.stdout:/dev/hugepages 2024-04-24T23:15:50.928 INFO:teuthology.orchestra.run.smithi006.stdout:/dev/mqueue 2024-04-24T23:15:50.928 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/kernel/debug 2024-04-24T23:15:50.928 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/kernel/tracing 2024-04-24T23:15:50.928 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/fs/fuse/connections 2024-04-24T23:15:50.928 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/kernel/config 2024-04-24T23:15:50.928 INFO:teuthology.orchestra.run.smithi006.stdout:/run/credentials/systemd-sysctl.service 2024-04-24T23:15:50.928 INFO:teuthology.orchestra.run.smithi006.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-24T23:15:50.929 INFO:teuthology.orchestra.run.smithi006.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-24T23:15:50.929 INFO:teuthology.orchestra.run.smithi006.stdout:/var/lib/nfs/rpc_pipefs 2024-04-24T23:15:50.929 INFO:teuthology.orchestra.run.smithi006.stdout:/run/user/1000 2024-04-24T23:15:50.929 INFO:teuthology.orchestra.run.smithi006.stdout:/var/lib/ceph 2024-04-24T23:15:50.929 INFO:teuthology.orchestra.run.smithi006.stdout:/var/lib/ceph/osd/ceph-0 2024-04-24T23:15:50.929 INFO:teuthology.orchestra.run.smithi006.stdout:/var/lib/ceph/osd/ceph-1 2024-04-24T23:15:50.929 INFO:teuthology.orchestra.run.smithi006.stdout:/var/lib/ceph/osd/ceph-2 2024-04-24T23:15:50.929 INFO:teuthology.orchestra.run.smithi006.stdout:/var/lib/ceph/osd/ceph-3 2024-04-24T23:15:50.929 INFO:teuthology.orchestra.run.smithi006.stdout:/run/netns 2024-04-24T23:15:50.929 INFO:teuthology.orchestra.run.smithi006.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-24T23:15:50.929 INFO:teuthology.orchestra.run.smithi006.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-24T23:15:50.930 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:15:50.930 DEBUG:teuthology.orchestra.run.smithi006:> ls /sys/fs/fuse/connections 2024-04-24T23:15:50.974 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-24T23:15:50.974 DEBUG:teuthology.orchestra.run.smithi006:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2024-04-24T23:15:51.022 DEBUG:teuthology.orchestra.run.smithi006:> sudo modprobe fuse 2024-04-24T23:15:51.096 DEBUG:teuthology.orchestra.run.smithi006:> cat /proc/self/mounts | awk '{print $2}' 2024-04-24T23:15:51.139 INFO:teuthology.orchestra.run.smithi006.stdout:/proc 2024-04-24T23:15:51.140 INFO:teuthology.orchestra.run.smithi006.stdout:/sys 2024-04-24T23:15:51.140 INFO:teuthology.orchestra.run.smithi006.stdout:/dev 2024-04-24T23:15:51.140 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/kernel/security 2024-04-24T23:15:51.140 INFO:teuthology.orchestra.run.smithi006.stdout:/dev/shm 2024-04-24T23:15:51.140 INFO:teuthology.orchestra.run.smithi006.stdout:/dev/pts 2024-04-24T23:15:51.140 INFO:teuthology.orchestra.run.smithi006.stdout:/run 2024-04-24T23:15:51.140 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/fs/cgroup 2024-04-24T23:15:51.140 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/fs/pstore 2024-04-24T23:15:51.140 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/fs/bpf 2024-04-24T23:15:51.140 INFO:teuthology.orchestra.run.smithi006.stdout:/ 2024-04-24T23:15:51.140 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/fs/selinux 2024-04-24T23:15:51.140 INFO:teuthology.orchestra.run.smithi006.stdout:/proc/sys/fs/binfmt_misc 2024-04-24T23:15:51.140 INFO:teuthology.orchestra.run.smithi006.stdout:/dev/hugepages 2024-04-24T23:15:51.141 INFO:teuthology.orchestra.run.smithi006.stdout:/dev/mqueue 2024-04-24T23:15:51.141 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/kernel/debug 2024-04-24T23:15:51.141 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/kernel/tracing 2024-04-24T23:15:51.141 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/fs/fuse/connections 2024-04-24T23:15:51.141 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/kernel/config 2024-04-24T23:15:51.141 INFO:teuthology.orchestra.run.smithi006.stdout:/run/credentials/systemd-sysctl.service 2024-04-24T23:15:51.141 INFO:teuthology.orchestra.run.smithi006.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-24T23:15:51.141 INFO:teuthology.orchestra.run.smithi006.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-24T23:15:51.141 INFO:teuthology.orchestra.run.smithi006.stdout:/var/lib/nfs/rpc_pipefs 2024-04-24T23:15:51.141 INFO:teuthology.orchestra.run.smithi006.stdout:/run/user/1000 2024-04-24T23:15:51.141 INFO:teuthology.orchestra.run.smithi006.stdout:/var/lib/ceph 2024-04-24T23:15:51.141 INFO:teuthology.orchestra.run.smithi006.stdout:/var/lib/ceph/osd/ceph-0 2024-04-24T23:15:51.141 INFO:teuthology.orchestra.run.smithi006.stdout:/var/lib/ceph/osd/ceph-1 2024-04-24T23:15:51.141 INFO:teuthology.orchestra.run.smithi006.stdout:/var/lib/ceph/osd/ceph-2 2024-04-24T23:15:51.141 INFO:teuthology.orchestra.run.smithi006.stdout:/var/lib/ceph/osd/ceph-3 2024-04-24T23:15:51.142 INFO:teuthology.orchestra.run.smithi006.stdout:/run/netns 2024-04-24T23:15:51.142 INFO:teuthology.orchestra.run.smithi006.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-24T23:15:51.142 INFO:teuthology.orchestra.run.smithi006.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-24T23:15:51.143 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:15:51.143 DEBUG:teuthology.orchestra.run.smithi006:> ls /sys/fs/fuse/connections 2024-04-24T23:15:51.223 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi006.stderr:2024-04-24T23:15:51.221+0000 7f5b5895c480 -1 init, newargv = 0x55ae615dcbe0 newargc=15 2024-04-24T23:15:51.223 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi006.stderr:ceph-fuse[40100]: starting ceph client 2024-04-24T23:15:51.442 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi006.stderr:ceph-fuse[40100]: starting fuse 2024-04-24T23:15:51.892 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-04-24T23:15:51.890+0000 7fef1ce00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-24T23:15:52.330 DEBUG:teuthology.orchestra.run.smithi006:> sudo modprobe fuse 2024-04-24T23:15:52.425 DEBUG:teuthology.orchestra.run.smithi006:> cat /proc/self/mounts | awk '{print $2}' 2024-04-24T23:15:52.483 INFO:teuthology.orchestra.run.smithi006.stdout:/proc 2024-04-24T23:15:52.483 INFO:teuthology.orchestra.run.smithi006.stdout:/sys 2024-04-24T23:15:52.483 INFO:teuthology.orchestra.run.smithi006.stdout:/dev 2024-04-24T23:15:52.483 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/kernel/security 2024-04-24T23:15:52.483 INFO:teuthology.orchestra.run.smithi006.stdout:/dev/shm 2024-04-24T23:15:52.483 INFO:teuthology.orchestra.run.smithi006.stdout:/dev/pts 2024-04-24T23:15:52.483 INFO:teuthology.orchestra.run.smithi006.stdout:/run 2024-04-24T23:15:52.484 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/fs/cgroup 2024-04-24T23:15:52.484 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/fs/pstore 2024-04-24T23:15:52.484 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/fs/bpf 2024-04-24T23:15:52.484 INFO:teuthology.orchestra.run.smithi006.stdout:/ 2024-04-24T23:15:52.484 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/fs/selinux 2024-04-24T23:15:52.484 INFO:teuthology.orchestra.run.smithi006.stdout:/proc/sys/fs/binfmt_misc 2024-04-24T23:15:52.484 INFO:teuthology.orchestra.run.smithi006.stdout:/dev/hugepages 2024-04-24T23:15:52.484 INFO:teuthology.orchestra.run.smithi006.stdout:/dev/mqueue 2024-04-24T23:15:52.484 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/kernel/debug 2024-04-24T23:15:52.484 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/kernel/tracing 2024-04-24T23:15:52.484 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/fs/fuse/connections 2024-04-24T23:15:52.484 INFO:teuthology.orchestra.run.smithi006.stdout:/sys/kernel/config 2024-04-24T23:15:52.484 INFO:teuthology.orchestra.run.smithi006.stdout:/run/credentials/systemd-sysctl.service 2024-04-24T23:15:52.484 INFO:teuthology.orchestra.run.smithi006.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-24T23:15:52.484 INFO:teuthology.orchestra.run.smithi006.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-24T23:15:52.485 INFO:teuthology.orchestra.run.smithi006.stdout:/var/lib/nfs/rpc_pipefs 2024-04-24T23:15:52.485 INFO:teuthology.orchestra.run.smithi006.stdout:/run/user/1000 2024-04-24T23:15:52.485 INFO:teuthology.orchestra.run.smithi006.stdout:/var/lib/ceph 2024-04-24T23:15:52.485 INFO:teuthology.orchestra.run.smithi006.stdout:/var/lib/ceph/osd/ceph-0 2024-04-24T23:15:52.485 INFO:teuthology.orchestra.run.smithi006.stdout:/var/lib/ceph/osd/ceph-1 2024-04-24T23:15:52.485 INFO:teuthology.orchestra.run.smithi006.stdout:/var/lib/ceph/osd/ceph-2 2024-04-24T23:15:52.485 INFO:teuthology.orchestra.run.smithi006.stdout:/var/lib/ceph/osd/ceph-3 2024-04-24T23:15:52.485 INFO:teuthology.orchestra.run.smithi006.stdout:/run/netns 2024-04-24T23:15:52.485 INFO:teuthology.orchestra.run.smithi006.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-24T23:15:52.485 INFO:teuthology.orchestra.run.smithi006.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-24T23:15:52.485 INFO:teuthology.orchestra.run.smithi006.stdout:/home/ubuntu/cephtest/mnt.admin 2024-04-24T23:15:52.486 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:15:52.486 DEBUG:teuthology.orchestra.run.smithi006:> ls /sys/fs/fuse/connections 2024-04-24T23:15:52.544 INFO:teuthology.orchestra.run.smithi006.stdout:40 2024-04-24T23:15:52.544 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [40] 2024-04-24T23:15:52.544 DEBUG:teuthology.orchestra.run.smithi006:> sudo stdin-killer -- python3 -c ' 2024-04-24T23:15:52.544 DEBUG:teuthology.orchestra.run.smithi006:> import glob 2024-04-24T23:15:52.544 DEBUG:teuthology.orchestra.run.smithi006:> import re 2024-04-24T23:15:52.544 DEBUG:teuthology.orchestra.run.smithi006:> import os 2024-04-24T23:15:52.544 DEBUG:teuthology.orchestra.run.smithi006:> import subprocess 2024-04-24T23:15:52.544 DEBUG:teuthology.orchestra.run.smithi006:> 2024-04-24T23:15:52.544 DEBUG:teuthology.orchestra.run.smithi006:> def _find_admin_socket(client_name): 2024-04-24T23:15:52.545 DEBUG:teuthology.orchestra.run.smithi006:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2024-04-24T23:15:52.545 DEBUG:teuthology.orchestra.run.smithi006:> files = glob.glob(asok_path) 2024-04-24T23:15:52.545 DEBUG:teuthology.orchestra.run.smithi006:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2024-04-24T23:15:52.545 DEBUG:teuthology.orchestra.run.smithi006:> 2024-04-24T23:15:52.545 DEBUG:teuthology.orchestra.run.smithi006:> # Given a non-glob path, it better be there 2024-04-24T23:15:52.545 DEBUG:teuthology.orchestra.run.smithi006:> if "*" not in asok_path: 2024-04-24T23:15:52.545 DEBUG:teuthology.orchestra.run.smithi006:> assert(len(files) == 1) 2024-04-24T23:15:52.545 DEBUG:teuthology.orchestra.run.smithi006:> return files[0] 2024-04-24T23:15:52.545 DEBUG:teuthology.orchestra.run.smithi006:> 2024-04-24T23:15:52.545 DEBUG:teuthology.orchestra.run.smithi006:> for f in files: 2024-04-24T23:15:52.545 DEBUG:teuthology.orchestra.run.smithi006:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-24T23:15:52.545 DEBUG:teuthology.orchestra.run.smithi006:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-24T23:15:52.545 DEBUG:teuthology.orchestra.run.smithi006:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-24T23:15:52.545 DEBUG:teuthology.orchestra.run.smithi006:> contents = proc_f.read() 2024-04-24T23:15:52.545 DEBUG:teuthology.orchestra.run.smithi006:> if mountpoint in contents: 2024-04-24T23:15:52.545 DEBUG:teuthology.orchestra.run.smithi006:> return f 2024-04-24T23:15:52.545 DEBUG:teuthology.orchestra.run.smithi006:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-24T23:15:52.545 DEBUG:teuthology.orchestra.run.smithi006:> 2024-04-24T23:15:52.546 DEBUG:teuthology.orchestra.run.smithi006:> print(_find_admin_socket("client.admin")) 2024-04-24T23:15:52.546 DEBUG:teuthology.orchestra.run.smithi006:> ' 2024-04-24T23:15:52.720 INFO:teuthology.orchestra.run.smithi006.stdout:/var/run/ceph/ceph-client.admin.40100.asok 2024-04-24T23:15:52.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:52 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T23:15:52.739 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.40100.asok 2024-04-24T23:15:52.739 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:15:52.739 DEBUG:teuthology.orchestra.run.smithi006:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.40100.asok status 2024-04-24T23:15:52.979 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-04-24T23:15:52.979 INFO:teuthology.orchestra.run.smithi006.stdout: "metadata": { 2024-04-24T23:15:52.979 INFO:teuthology.orchestra.run.smithi006.stdout: "ceph_sha1": "d2c45d977142e024a59b68a8f1dce8f0b9840551", 2024-04-24T23:15:52.980 INFO:teuthology.orchestra.run.smithi006.stdout: "ceph_version": "ceph version 19.0.0-2445-gd2c45d97 (d2c45d977142e024a59b68a8f1dce8f0b9840551) squid (dev)", 2024-04-24T23:15:52.980 INFO:teuthology.orchestra.run.smithi006.stdout: "entity_id": "admin", 2024-04-24T23:15:52.980 INFO:teuthology.orchestra.run.smithi006.stdout: "hostname": "smithi006", 2024-04-24T23:15:52.980 INFO:teuthology.orchestra.run.smithi006.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2024-04-24T23:15:52.980 INFO:teuthology.orchestra.run.smithi006.stdout: "pid": "40100", 2024-04-24T23:15:52.980 INFO:teuthology.orchestra.run.smithi006.stdout: "root": "/" 2024-04-24T23:15:52.980 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-24T23:15:52.980 INFO:teuthology.orchestra.run.smithi006.stdout: "dentry_count": 0, 2024-04-24T23:15:52.980 INFO:teuthology.orchestra.run.smithi006.stdout: "dentry_pinned_count": 0, 2024-04-24T23:15:52.980 INFO:teuthology.orchestra.run.smithi006.stdout: "id": 4347, 2024-04-24T23:15:52.980 INFO:teuthology.orchestra.run.smithi006.stdout: "inst": { 2024-04-24T23:15:52.980 INFO:teuthology.orchestra.run.smithi006.stdout: "name": { 2024-04-24T23:15:52.980 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "client", 2024-04-24T23:15:52.981 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 4347 2024-04-24T23:15:52.981 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-24T23:15:52.981 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": { 2024-04-24T23:15:52.981 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-04-24T23:15:52.981 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "192.168.0.1:0", 2024-04-24T23:15:52.981 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 1840628037 2024-04-24T23:15:52.981 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-24T23:15:52.981 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-24T23:15:52.981 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": { 2024-04-24T23:15:52.981 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-04-24T23:15:52.981 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "192.168.0.1:0", 2024-04-24T23:15:52.981 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 1840628037 2024-04-24T23:15:52.981 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-24T23:15:52.982 INFO:teuthology.orchestra.run.smithi006.stdout: "inst_str": "client.4347 192.168.0.1:0/1840628037", 2024-04-24T23:15:52.982 INFO:teuthology.orchestra.run.smithi006.stdout: "addr_str": "192.168.0.1:0/1840628037", 2024-04-24T23:15:52.982 INFO:teuthology.orchestra.run.smithi006.stdout: "inode_count": 1, 2024-04-24T23:15:52.982 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_epoch": 5, 2024-04-24T23:15:52.982 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_epoch": 24, 2024-04-24T23:15:52.982 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_epoch_barrier": 0, 2024-04-24T23:15:52.982 INFO:teuthology.orchestra.run.smithi006.stdout: "blocklisted": false, 2024-04-24T23:15:52.982 INFO:teuthology.orchestra.run.smithi006.stdout: "fs_name": "cephfs" 2024-04-24T23:15:52.982 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-04-24T23:15:52.997 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:15:52.997 DEBUG:teuthology.orchestra.run.smithi006:> stat --file-system '--printf=%T 2024-04-24T23:15:52.997 DEBUG:teuthology.orchestra.run.smithi006:> ' -- /home/ubuntu/cephtest/mnt.admin 2024-04-24T23:15:53.054 INFO:teuthology.orchestra.run.smithi006.stdout:fuseblk 2024-04-24T23:15:53.054 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2024-04-24T23:15:53.055 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:15:53.055 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2024-04-24T23:15:53.140 DEBUG:teuthology.orchestra.run.smithi006:> (cd /home/ubuntu/cephtest/mnt.admin && exec stdin-killer --timeout=900 -- bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2024-04-24T23:15:53.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:53 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-24T23:15:53.284 INFO:teuthology.orchestra.run.smithi006.stdout:# file: . 2024-04-24T23:15:53.285 INFO:teuthology.orchestra.run.smithi006.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2024-04-24T23:15:53.285 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:15:53.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:53 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T23:15:53.295 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-04-24T23:15:53.295 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-24T23:15:53.347 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T23:15:53.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:53.542+0000 7f4f41600640 1 Processor -- start 2024-04-24T23:15:53.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:53.542+0000 7f4f41600640 1 -- start start 2024-04-24T23:15:53.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:53.544+0000 7f4f41600640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4f3c0cd390 0x7f4f3c0cacf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:53.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:53.544+0000 7f4f41600640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f3c06be70 con 0x7f4f3c0ccfb0 2024-04-24T23:15:53.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:53.544+0000 7f4f41600640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f3c06c150 con 0x7f4f3c1548a0 2024-04-24T23:15:53.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:53.544+0000 7f4f41600640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f4f3c06ccd0 con 0x7f4f3c0cd390 2024-04-24T23:15:53.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:53.544+0000 7f4f3be00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f4f3c0ccfb0 0x7f4f3c15b480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:45034/0 (socket says 172.21.15.6:45034) 2024-04-24T23:15:53.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:53.544+0000 7f4f3be00640 1 -- 172.21.15.6:0/2712708204 learned_addr learned my addr 172.21.15.6:0/2712708204 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:15:53.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:53.545+0000 7f4f3b400640 1 --2- 172.21.15.6:0/2712708204 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4f3c0cd390 0x7f4f3c0cacf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:53.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:53.545+0000 7f4f3aa00640 1 -- 172.21.15.6:0/2712708204 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2710977310 0 0) 0x7f4f3c06be70 con 0x7f4f3c0ccfb0 2024-04-24T23:15:53.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:53.545+0000 7f4f3aa00640 1 -- 172.21.15.6:0/2712708204 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4f20003520 con 0x7f4f3c0ccfb0 2024-04-24T23:15:53.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:53.545+0000 7f4f3aa00640 1 -- 172.21.15.6:0/2712708204 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2488004295 0 0) 0x7f4f3c06c150 con 0x7f4f3c1548a0 2024-04-24T23:15:53.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:53.545+0000 7f4f3aa00640 1 -- 172.21.15.6:0/2712708204 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4f3c06be70 con 0x7f4f3c1548a0 2024-04-24T23:15:53.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:53.545+0000 7f4f3b400640 1 -- 172.21.15.6:0/2712708204 >> v1:172.21.15.190:6789/0 conn(0x7f4f3c1548a0 legacy=0x7f4f3c0cbe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:53.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:53.545+0000 7f4f3b400640 1 -- 172.21.15.6:0/2712708204 >> v1:172.21.15.6:6789/0 conn(0x7f4f3c0ccfb0 legacy=0x7f4f3c15b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:53.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:53.545+0000 7f4f3b400640 1 -- 172.21.15.6:0/2712708204 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f3c06c430 con 0x7f4f3c0cd390 2024-04-24T23:15:53.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:53.546+0000 7f4f3b400640 1 --2- 172.21.15.6:0/2712708204 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4f3c0cd390 0x7f4f3c0cacf0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f4f240097d0 tx=0x7f4f2402a290 comp rx=0 tx=0).ready entity=mon.2 client_cookie=21a44091468582b8 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:15:53.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:53.546+0000 7f4f3aa00640 1 -- 172.21.15.6:0/2712708204 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f24037070 con 0x7f4f3c0cd390 2024-04-24T23:15:53.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:53.546+0000 7f4f3aa00640 1 -- 172.21.15.6:0/2712708204 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f24032440 con 0x7f4f3c0cd390 2024-04-24T23:15:53.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:53.546+0000 7f4f3aa00640 1 -- 172.21.15.6:0/2712708204 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f240327f0 con 0x7f4f3c0cd390 2024-04-24T23:15:53.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:53.547+0000 7f4f41600640 1 -- 172.21.15.6:0/2712708204 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4f3c0cd390 msgr2=0x7f4f3c0cacf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:53.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:53.547+0000 7f4f41600640 1 --2- 172.21.15.6:0/2712708204 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4f3c0cd390 0x7f4f3c0cacf0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f4f240097d0 tx=0x7f4f2402a290 comp rx=0 tx=0).stop 2024-04-24T23:15:53.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:53.547+0000 7f4f41600640 1 -- 172.21.15.6:0/2712708204 shutdown_connections 2024-04-24T23:15:53.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:53.547+0000 7f4f41600640 1 --2- 172.21.15.6:0/2712708204 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4f3c0cd390 0x7f4f3c0cacf0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:53.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:53.547+0000 7f4f41600640 1 -- 172.21.15.6:0/2712708204 >> 172.21.15.6:0/2712708204 conn(0x7f4f3c06d9f0 msgr2=0x7f4f3c066ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:15:53.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:53.547+0000 7f4f41600640 1 -- 172.21.15.6:0/2712708204 shutdown_connections 2024-04-24T23:15:53.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:53.547+0000 7f4f41600640 1 -- 172.21.15.6:0/2712708204 wait complete. 2024-04-24T23:15:53.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:53.549+0000 7f4f41600640 1 Processor -- start 2024-04-24T23:15:53.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:53.549+0000 7f4f41600640 1 -- start start 2024-04-24T23:15:53.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:53.550+0000 7f4f41600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4f3c0ccfb0 0x7f4f3c0ca1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:53.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:53.550+0000 7f4f41600640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4f3c0cd390 0x7f4f3c14c070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:53.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:53.550+0000 7f4f41600640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4f3c1548a0 0x7f4f3c1f0be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:53.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:53.550+0000 7f4f41600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f4f3c06c9f0 con 0x7f4f3c0ccfb0 2024-04-24T23:15:53.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:53.550+0000 7f4f41600640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f4f3c0cb230 con 0x7f4f3c0cd390 2024-04-24T23:15:53.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:53.550+0000 7f4f41600640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f4f3c067bb0 con 0x7f4f3c1548a0 2024-04-24T23:15:53.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:53.550+0000 7f4f3be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4f3c0ccfb0 0x7f4f3c0ca1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:53.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:53.550+0000 7f4f3b400640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4f3c0cd390 0x7f4f3c14c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:53.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:53.550+0000 7f4f40c00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4f3c1548a0 0x7f4f3c1f0be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:53.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:53.551+0000 7f4f3be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4f3c0ccfb0 0x7f4f3c0ca1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57756/0 (socket says 172.21.15.6:57756) 2024-04-24T23:15:53.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:53.551+0000 7f4f3be00640 1 -- 172.21.15.6:0/4184321869 learned_addr learned my addr 172.21.15.6:0/4184321869 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:15:53.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:53.551+0000 7f4f3b400640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4f3c0cd390 0x7f4f3c14c070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:49790/0 (socket says 172.21.15.6:49790) 2024-04-24T23:15:53.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:53.551+0000 7f4f3b400640 1 -- 172.21.15.6:0/4184321869 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4f3c1548a0 msgr2=0x7f4f3c1f0be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:53.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:53.551+0000 7f4f3b400640 1 --2- 172.21.15.6:0/4184321869 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4f3c1548a0 0x7f4f3c1f0be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:53.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:53.551+0000 7f4f3b400640 1 -- 172.21.15.6:0/4184321869 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4f3c0ccfb0 msgr2=0x7f4f3c0ca1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:53.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:53.551+0000 7f4f3b400640 1 --2- 172.21.15.6:0/4184321869 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4f3c0ccfb0 0x7f4f3c0ca1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:53.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:53.551+0000 7f4f3b400640 1 -- 172.21.15.6:0/4184321869 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f3c1f06e0 con 0x7f4f3c0cd390 2024-04-24T23:15:53.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:53.551+0000 7f4f3be00640 1 --2- 172.21.15.6:0/4184321869 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4f3c0ccfb0 0x7f4f3c0ca1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:15:53.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:53.551+0000 7f4f3b400640 1 --2- 172.21.15.6:0/4184321869 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4f3c0cd390 0x7f4f3c14c070 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f4f24031000 tx=0x7f4f240046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:15:53.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:53.552+0000 7f4f33e00640 1 -- 172.21.15.6:0/4184321869 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f24041030 con 0x7f4f3c0cd390 2024-04-24T23:15:53.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:53.552+0000 7f4f33e00640 1 -- 172.21.15.6:0/4184321869 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f2403d420 con 0x7f4f3c0cd390 2024-04-24T23:15:53.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:53.552+0000 7f4f41600640 1 -- 172.21.15.6:0/4184321869 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f3c0cfe30 con 0x7f4f3c0cd390 2024-04-24T23:15:53.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:53.552+0000 7f4f33e00640 1 -- 172.21.15.6:0/4184321869 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f24037040 con 0x7f4f3c0cd390 2024-04-24T23:15:53.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:53.552+0000 7f4f41600640 1 -- 172.21.15.6:0/4184321869 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f3c0d03f0 con 0x7f4f3c0cd390 2024-04-24T23:15:53.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:53.553+0000 7f4f41600640 1 -- 172.21.15.6:0/4184321869 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f3c075b30 con 0x7f4f3c0cd390 2024-04-24T23:15:53.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:53.554+0000 7f4f33e00640 1 -- 172.21.15.6:0/4184321869 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f4f24044070 con 0x7f4f3c0cd390 2024-04-24T23:15:53.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:53.555+0000 7f4f33e00640 1 --2- 172.21.15.6:0/4184321869 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f4f1806be30 0x7f4f1806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:53.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:53.555+0000 7f4f33e00640 1 -- 172.21.15.6:0/4184321869 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f4f240ab290 con 0x7f4f3c0cd390 2024-04-24T23:15:53.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:53.555+0000 7f4f3be00640 1 --2- 172.21.15.6:0/4184321869 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f4f1806be30 0x7f4f1806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:53.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:53.556+0000 7f4f3be00640 1 --2- 172.21.15.6:0/4184321869 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f4f1806be30 0x7f4f1806e2f0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f4f2c000f60 tx=0x7f4f2c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:15:53.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:53.560+0000 7f4f33e00640 1 -- 172.21.15.6:0/4184321869 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f4f24079180 con 0x7f4f3c0cd390 2024-04-24T23:15:53.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:53.762+0000 7f4f41600640 1 -- 172.21.15.6:0/4184321869 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4f3c067990 con 0x7f4f3c0cd390 2024-04-24T23:15:53.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:53.764+0000 7f4f33e00640 1 -- 172.21.15.6:0/4184321869 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+18458 (secure 0 0 0) 0x7f4f2407d480 con 0x7f4f3c0cd390 2024-04-24T23:15:53.766 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:15:53.766 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":24,"fsid":"18bef68d-5e6d-4ec5-862b-b6f206e2588d","created":"2024-04-24T23:15:12.535072+0000","modified":"2024-04-24T23:15:41.465875+0000","last_up_change":"2024-04-24T23:15:23.773223+0000","last_in_change":"2024-04-24T23:15:14.905328+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T23:15:24.747441+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-24T23:15:34.051800+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-24T23:15:35.436870+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-24T23:15:39.034257+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"3b7a212d-d498-462e-90bf-ed493c803a09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":3022569437},{"type":"v1","addr":"172.21.15.6:6809","nonce":3022569437}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":3022569437},{"type":"v1","addr":"172.21.15.6:6811","nonce":3022569437}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":3022569437},{"type":"v1","addr":"172.21.15.6:6815","nonce":3022569437}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":3022569437},{"type":"v1","addr":"172.21.15.6:6813","nonce":3022569437}]},"public_addr":"172.21.15.6:6809/3022569437","cluster_addr":"172.21.15.6:6811/3022569437","heartbeat_back_addr":"172.21.15.6:6815/3022569437","heartbeat_front_addr":"172.21.15.6:6813/3022569437","state":["exists","up"]},{"osd":1,"uuid":"7c97f5a2-1735-4775-9f15-eb50099b19d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":3687836888},{"type":"v1","addr":"172.21.15.6:6817","nonce":3687836888}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":3687836888},{"type":"v1","addr":"172.21.15.6:6819","nonce":3687836888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":3687836888},{"type":"v1","addr":"172.21.15.6:6823","nonce":3687836888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":3687836888},{"type":"v1","addr":"172.21.15.6:6821","nonce":3687836888}]},"public_addr":"172.21.15.6:6817/3687836888","cluster_addr":"172.21.15.6:6819/3687836888","heartbeat_back_addr":"172.21.15.6:6823/3687836888","heartbeat_front_addr":"172.21.15.6:6821/3687836888","state":["exists","up"]},{"osd":2,"uuid":"d86bbc18-438c-4ec2-8018-ba7dd78f790b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":1020213724},{"type":"v1","addr":"172.21.15.6:6801","nonce":1020213724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":1020213724},{"type":"v1","addr":"172.21.15.6:6803","nonce":1020213724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":1020213724},{"type":"v1","addr":"172.21.15.6:6807","nonce":1020213724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":1020213724},{"type":"v1","addr":"172.21.15.6:6805","nonce":1020213724}]},"public_addr":"172.21.15.6:6801/1020213724","cluster_addr":"172.21.15.6:6803/1020213724","heartbeat_back_addr":"172.21.15.6:6807/1020213724","heartbeat_front_addr":"172.21.15.6:6805/1020213724","state":["exists","up"]},{"osd":3,"uuid":"3754ae16-4248-40dd-95f9-0f59c0eb5426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6824","nonce":3163883468},{"type":"v1","addr":"172.21.15.6:6825","nonce":3163883468}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":3163883468},{"type":"v1","addr":"172.21.15.6:6827","nonce":3163883468}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":3163883468},{"type":"v1","addr":"172.21.15.6:6831","nonce":3163883468}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":3163883468},{"type":"v1","addr":"172.21.15.6:6829","nonce":3163883468}]},"public_addr":"172.21.15.6:6825/3163883468","cluster_addr":"172.21.15.6:6827/3163883468","heartbeat_back_addr":"172.21.15.6:6831/3163883468","heartbeat_front_addr":"172.21.15.6:6829/3163883468","state":["exists","up"]},{"osd":4,"uuid":"8b2913a0-d006-479c-84ed-5b21e0e6c392","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":1989500934},{"type":"v1","addr":"172.21.15.190:6809","nonce":1989500934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":1989500934},{"type":"v1","addr":"172.21.15.190:6811","nonce":1989500934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":1989500934},{"type":"v1","addr":"172.21.15.190:6815","nonce":1989500934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":1989500934},{"type":"v1","addr":"172.21.15.190:6813","nonce":1989500934}]},"public_addr":"172.21.15.190:6809/1989500934","cluster_addr":"172.21.15.190:6811/1989500934","heartbeat_back_addr":"172.21.15.190:6815/1989500934","heartbeat_front_addr":"172.21.15.190:6813/1989500934","state":["exists","up"]},{"osd":5,"uuid":"499b79d0-0585-4dae-ab69-196a551793da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":3687548412},{"type":"v1","addr":"172.21.15.190:6801","nonce":3687548412}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":3687548412},{"type":"v1","addr":"172.21.15.190:6803","nonce":3687548412}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":3687548412},{"type":"v1","addr":"172.21.15.190:6807","nonce":3687548412}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":3687548412},{"type":"v1","addr":"172.21.15.190:6805","nonce":3687548412}]},"public_addr":"172.21.15.190:6801/3687548412","cluster_addr":"172.21.15.190:6803/3687548412","heartbeat_back_addr":"172.21.15.190:6807/3687548412","heartbeat_front_addr":"172.21.15.190:6805/3687548412","state":["exists","up"]},{"osd":6,"uuid":"6ba6df3b-4a6b-4a89-a890-0f79f387d4d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":870571564},{"type":"v1","addr":"172.21.15.190:6817","nonce":870571564}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":870571564},{"type":"v1","addr":"172.21.15.190:6819","nonce":870571564}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":870571564},{"type":"v1","addr":"172.21.15.190:6823","nonce":870571564}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":870571564},{"type":"v1","addr":"172.21.15.190:6821","nonce":870571564}]},"public_addr":"172.21.15.190:6817/870571564","cluster_addr":"172.21.15.190:6819/870571564","heartbeat_back_addr":"172.21.15.190:6823/870571564","heartbeat_front_addr":"172.21.15.190:6821/870571564","state":["exists","up"]},{"osd":7,"uuid":"e9668b1c-a000-4332-86e8-4efd73b02409","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":123639197},{"type":"v1","addr":"172.21.15.190:6825","nonce":123639197}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":123639197},{"type":"v1","addr":"172.21.15.190:6827","nonce":123639197}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":123639197},{"type":"v1","addr":"172.21.15.190:6831","nonce":123639197}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":123639197},{"type":"v1","addr":"172.21.15.190:6829","nonce":123639197}]},"public_addr":"172.21.15.190:6825/123639197","cluster_addr":"172.21.15.190:6827/123639197","heartbeat_back_addr":"172.21.15.190:6831/123639197","heartbeat_front_addr":"172.21.15.190:6829/123639197","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.321646+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.363873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.466963+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.531989+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:19.990707+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.043543+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.032492+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.115004+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T23:15:53.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:53.766+0000 7f4f41600640 1 -- 172.21.15.6:0/4184321869 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f4f1806be30 msgr2=0x7f4f1806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:53.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:53.766+0000 7f4f41600640 1 --2- 172.21.15.6:0/4184321869 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f4f1806be30 0x7f4f1806e2f0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f4f2c000f60 tx=0x7f4f2c009040 comp rx=0 tx=0).stop 2024-04-24T23:15:53.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:53.766+0000 7f4f41600640 1 -- 172.21.15.6:0/4184321869 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4f3c0cd390 msgr2=0x7f4f3c14c070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:53.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:53.766+0000 7f4f41600640 1 --2- 172.21.15.6:0/4184321869 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4f3c0cd390 0x7f4f3c14c070 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f4f24031000 tx=0x7f4f240046c0 comp rx=0 tx=0).stop 2024-04-24T23:15:53.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:53.767+0000 7f4f41600640 1 -- 172.21.15.6:0/4184321869 shutdown_connections 2024-04-24T23:15:53.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:53.767+0000 7f4f41600640 1 --2- 172.21.15.6:0/4184321869 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f4f1806be30 0x7f4f1806e2f0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:53.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:53.767+0000 7f4f41600640 1 --2- 172.21.15.6:0/4184321869 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4f3c1548a0 0x7f4f3c1f0be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:53.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:53.767+0000 7f4f41600640 1 --2- 172.21.15.6:0/4184321869 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4f3c0cd390 0x7f4f3c14c070 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:53.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:53.767+0000 7f4f41600640 1 --2- 172.21.15.6:0/4184321869 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4f3c0ccfb0 0x7f4f3c0ca1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:53.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:53.767+0000 7f4f41600640 1 -- 172.21.15.6:0/4184321869 >> 172.21.15.6:0/4184321869 conn(0x7f4f3c06d9f0 msgr2=0x7f4f3c153260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:15:53.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:53.767+0000 7f4f41600640 1 -- 172.21.15.6:0/4184321869 shutdown_connections 2024-04-24T23:15:53.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:53.767+0000 7f4f41600640 1 -- 172.21.15.6:0/4184321869 wait complete. 2024-04-24T23:15:53.788 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T23:15:24.747441+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-24T23:15:34.051800+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-24T23:15:35.436870+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-24T23:15:39.034257+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-24T23:15:53.788 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-24T23:15:54.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.006+0000 7faadee00640 1 Processor -- start 2024-04-24T23:15:54.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.007+0000 7faadee00640 1 -- start start 2024-04-24T23:15:54.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.008+0000 7faadee00640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7faad80c5e40 0x7faad80c6ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:54.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.008+0000 7faadee00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7faad80c8570 0x7faad81524a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:54.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.008+0000 7faadee00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faad80cca80 con 0x7faad80c5a60 2024-04-24T23:15:54.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.008+0000 7faadee00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7faad815c440 con 0x7faad80c8570 2024-04-24T23:15:54.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.008+0000 7faadee00640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7faad80ccd60 con 0x7faad80c5e40 2024-04-24T23:15:54.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.009+0000 7faadda00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7faad80c5a60 0x7faad815bd30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:45050/0 (socket says 172.21.15.6:45050) 2024-04-24T23:15:54.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.009+0000 7faadda00640 1 -- 172.21.15.6:0/1798792133 learned_addr learned my addr 172.21.15.6:0/1798792133 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:15:54.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.009+0000 7faade400640 1 --2- 172.21.15.6:0/1798792133 >> v2:172.21.15.6:3300/0 conn(0x7faad80c8570 0x7faad81524a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:54.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.009+0000 7faadd000640 1 --2- 172.21.15.6:0/1798792133 >> v2:172.21.15.190:3300/0 conn(0x7faad80c5e40 0x7faad80c6ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:54.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.009+0000 7faad7e00640 1 -- 172.21.15.6:0/1798792133 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2010727243 0 0) 0x7faad80cca80 con 0x7faad80c5a60 2024-04-24T23:15:54.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.009+0000 7faad7e00640 1 -- 172.21.15.6:0/1798792133 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faac0003520 con 0x7faad80c5a60 2024-04-24T23:15:54.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.009+0000 7faade400640 1 -- 172.21.15.6:0/1798792133 >> v2:172.21.15.190:3300/0 conn(0x7faad80c5e40 msgr2=0x7faad80c6ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:15:54.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.009+0000 7faade400640 1 --2- 172.21.15.6:0/1798792133 >> v2:172.21.15.190:3300/0 conn(0x7faad80c5e40 0x7faad80c6ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:54.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.009+0000 7faade400640 1 -- 172.21.15.6:0/1798792133 >> v1:172.21.15.6:6789/0 conn(0x7faad80c5a60 legacy=0x7faad815bd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:54.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.009+0000 7faade400640 1 -- 172.21.15.6:0/1798792133 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faad806eab0 con 0x7faad80c8570 2024-04-24T23:15:54.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.011+0000 7faade400640 1 --2- 172.21.15.6:0/1798792133 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faad80c8570 0x7faad81524a0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7faad80cc8e0 tx=0x7faad002a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7944ec95533e42 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:15:54.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.011+0000 7faad7e00640 1 -- 172.21.15.6:0/1798792133 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faad0007740 con 0x7faad80c8570 2024-04-24T23:15:54.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.011+0000 7faad7e00640 1 -- 172.21.15.6:0/1798792133 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faad00078e0 con 0x7faad80c8570 2024-04-24T23:15:54.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.011+0000 7faad7e00640 1 -- 172.21.15.6:0/1798792133 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faad0007cb0 con 0x7faad80c8570 2024-04-24T23:15:54.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.012+0000 7faadee00640 1 -- 172.21.15.6:0/1798792133 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faad80c8570 msgr2=0x7faad81524a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:54.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.012+0000 7faadee00640 1 --2- 172.21.15.6:0/1798792133 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faad80c8570 0x7faad81524a0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7faad80cc8e0 tx=0x7faad002a680 comp rx=0 tx=0).stop 2024-04-24T23:15:54.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.012+0000 7faadee00640 1 -- 172.21.15.6:0/1798792133 shutdown_connections 2024-04-24T23:15:54.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.012+0000 7faadee00640 1 --2- 172.21.15.6:0/1798792133 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faad80c8570 0x7faad81524a0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:54.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.012+0000 7faadee00640 1 --2- 172.21.15.6:0/1798792133 >> v2:172.21.15.190:3300/0 conn(0x7faad80c5e40 0x7faad80c6ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:54.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.012+0000 7faadee00640 1 -- 172.21.15.6:0/1798792133 >> 172.21.15.6:0/1798792133 conn(0x7faad806cc40 msgr2=0x7faad806d050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:15:54.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.012+0000 7faadee00640 1 -- 172.21.15.6:0/1798792133 shutdown_connections 2024-04-24T23:15:54.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.012+0000 7faadee00640 1 -- 172.21.15.6:0/1798792133 wait complete. 2024-04-24T23:15:54.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.014+0000 7faadee00640 1 Processor -- start 2024-04-24T23:15:54.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.014+0000 7faadee00640 1 -- start start 2024-04-24T23:15:54.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.015+0000 7faadee00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faad80c5a60 0x7faad81efea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:54.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.015+0000 7faadee00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faad80c5e40 0x7faad81f0a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:54.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.015+0000 7faadee00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faad80c8570 0x7faad8155780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:54.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.015+0000 7faadee00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7faad806f590 con 0x7faad80c5e40 2024-04-24T23:15:54.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.015+0000 7faadee00640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7faad806f070 con 0x7faad80c8570 2024-04-24T23:15:54.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.015+0000 7faadee00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7faad8153000 con 0x7faad80c5a60 2024-04-24T23:15:54.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.015+0000 7faadd000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faad80c5e40 0x7faad81f0a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:54.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.015+0000 7faadd000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faad80c5e40 0x7faad81f0a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57768/0 (socket says 172.21.15.6:57768) 2024-04-24T23:15:54.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.015+0000 7faadd000640 1 -- 172.21.15.6:0/1420137034 learned_addr learned my addr 172.21.15.6:0/1420137034 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:15:54.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.015+0000 7faadda00640 1 --2- 172.21.15.6:0/1420137034 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faad80c5a60 0x7faad81efea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:54.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.015+0000 7faade400640 1 --2- 172.21.15.6:0/1420137034 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faad80c8570 0x7faad8155780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:54.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.016+0000 7faadd000640 1 -- 172.21.15.6:0/1420137034 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faad80c5a60 msgr2=0x7faad81efea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:54.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.016+0000 7faadd000640 1 --2- 172.21.15.6:0/1420137034 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faad80c5a60 0x7faad81efea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:54.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.016+0000 7faadd000640 1 -- 172.21.15.6:0/1420137034 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faad80c8570 msgr2=0x7faad8155780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:54.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.016+0000 7faadd000640 1 --2- 172.21.15.6:0/1420137034 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faad80c8570 0x7faad8155780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:54.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.016+0000 7faadd000640 1 -- 172.21.15.6:0/1420137034 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faad81532c0 con 0x7faad80c5e40 2024-04-24T23:15:54.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.016+0000 7faadd000640 1 --2- 172.21.15.6:0/1420137034 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faad80c5e40 0x7faad81f0a50 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7faac800e9b0 tx=0x7faac800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:15:54.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.016+0000 7faad5600640 1 -- 172.21.15.6:0/1420137034 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faac8017070 con 0x7faad80c5e40 2024-04-24T23:15:54.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.016+0000 7faad5600640 1 -- 172.21.15.6:0/1420137034 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faac800c9a0 con 0x7faad80c5e40 2024-04-24T23:15:54.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.016+0000 7faadee00640 1 -- 172.21.15.6:0/1420137034 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faad81535a0 con 0x7faad80c5e40 2024-04-24T23:15:54.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.016+0000 7faad5600640 1 -- 172.21.15.6:0/1420137034 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faac800cd70 con 0x7faad80c5e40 2024-04-24T23:15:54.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.016+0000 7faadee00640 1 -- 172.21.15.6:0/1420137034 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faad8154f80 con 0x7faad80c5e40 2024-04-24T23:15:54.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.018+0000 7faadee00640 1 -- 172.21.15.6:0/1420137034 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faad8075b30 con 0x7faad80c5e40 2024-04-24T23:15:54.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.019+0000 7faad5600640 1 -- 172.21.15.6:0/1420137034 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7faac801a070 con 0x7faad80c5e40 2024-04-24T23:15:54.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.020+0000 7faad5600640 1 --2- 172.21.15.6:0/1420137034 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7faabc06be30 0x7faabc06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:54.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.020+0000 7faad5600640 1 -- 172.21.15.6:0/1420137034 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7faac808b460 con 0x7faad80c5e40 2024-04-24T23:15:54.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.021+0000 7faadda00640 1 --2- 172.21.15.6:0/1420137034 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7faabc06be30 0x7faabc06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:54.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.021+0000 7faadda00640 1 --2- 172.21.15.6:0/1420137034 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7faabc06be30 0x7faabc06e2f0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7faacc005880 tx=0x7faacc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:15:54.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.025+0000 7faad5600640 1 -- 172.21.15.6:0/1420137034 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7faac8059470 con 0x7faad80c5e40 2024-04-24T23:15:54.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.225+0000 7faadee00640 1 -- 172.21.15.6:0/1420137034 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7faad8156d10 con 0x7faad80c5e40 2024-04-24T23:15:54.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.226+0000 7faad5600640 1 -- 172.21.15.6:0/1420137034 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) ==== 93+0+10 (secure 0 0 0) 0x7faac805d770 con 0x7faad80c5e40 2024-04-24T23:15:54.228 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-04-24T23:15:54.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.229+0000 7faadee00640 1 -- 172.21.15.6:0/1420137034 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7faabc06be30 msgr2=0x7faabc06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:54.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.229+0000 7faadee00640 1 --2- 172.21.15.6:0/1420137034 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7faabc06be30 0x7faabc06e2f0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7faacc005880 tx=0x7faacc009040 comp rx=0 tx=0).stop 2024-04-24T23:15:54.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.229+0000 7faadee00640 1 -- 172.21.15.6:0/1420137034 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faad80c5e40 msgr2=0x7faad81f0a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:54.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.229+0000 7faadee00640 1 --2- 172.21.15.6:0/1420137034 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faad80c5e40 0x7faad81f0a50 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7faac800e9b0 tx=0x7faac800ee70 comp rx=0 tx=0).stop 2024-04-24T23:15:54.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.229+0000 7faadee00640 1 -- 172.21.15.6:0/1420137034 shutdown_connections 2024-04-24T23:15:54.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.229+0000 7faadee00640 1 --2- 172.21.15.6:0/1420137034 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7faabc06be30 0x7faabc06e2f0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:54.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.229+0000 7faadee00640 1 --2- 172.21.15.6:0/1420137034 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faad80c8570 0x7faad8155780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:54.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.229+0000 7faadee00640 1 --2- 172.21.15.6:0/1420137034 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faad80c5e40 0x7faad81f0a50 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:54.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.229+0000 7faadee00640 1 --2- 172.21.15.6:0/1420137034 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faad80c5a60 0x7faad81efea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:54.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.229+0000 7faadee00640 1 -- 172.21.15.6:0/1420137034 >> 172.21.15.6:0/1420137034 conn(0x7faad806cc40 msgr2=0x7faad81587b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:15:54.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.229+0000 7faadee00640 1 -- 172.21.15.6:0/1420137034 shutdown_connections 2024-04-24T23:15:54.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.229+0000 7faadee00640 1 -- 172.21.15.6:0/1420137034 wait complete. 2024-04-24T23:15:54.251 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-24T23:15:54.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.455+0000 7fd7a8400640 1 Processor -- start 2024-04-24T23:15:54.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.456+0000 7fd7a8400640 1 -- start start 2024-04-24T23:15:54.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.457+0000 7fd7a8400640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd7a0152bc0 0x7fd7a0159390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:54.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.457+0000 7fd7a8400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd7a00c8f50 0x7fd7a015bf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:54.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.457+0000 7fd7a8400640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fd7a00c9330 0x7fd7a015c440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:54.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.457+0000 7fd7a8400640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7fd7a00c73d0 con 0x7fd7a00c8f50 2024-04-24T23:15:54.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.457+0000 7fd7a8400640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7fd7a00c8ac0 con 0x7fd7a00c9330 2024-04-24T23:15:54.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.457+0000 7fd7a8400640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fd7a00c6e70 con 0x7fd7a0152bc0 2024-04-24T23:15:54.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.458+0000 7fd7a6400640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fd7a00c9330 0x7fd7a015c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:54.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.458+0000 7fd7a5a00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd7a0152bc0 0x7fd7a0159390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:54.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.458+0000 7fd7a5000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd7a00c8f50 0x7fd7a015bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:54.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.458+0000 7fd7a6400640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fd7a00c9330 0x7fd7a015c440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:49816/0 (socket says 172.21.15.6:49816) 2024-04-24T23:15:54.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.458+0000 7fd7a6400640 1 -- 172.21.15.6:0/770647199 learned_addr learned my addr 172.21.15.6:0/770647199 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:15:54.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.458+0000 7fd7a6400640 1 -- 172.21.15.6:0/770647199 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd7a0152bc0 msgr2=0x7fd7a0159390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:15:54.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.458+0000 7fd7a6400640 1 --2- 172.21.15.6:0/770647199 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd7a0152bc0 0x7fd7a0159390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:54.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.458+0000 7fd7a6400640 1 -- 172.21.15.6:0/770647199 >> v2:172.21.15.6:3300/0 conn(0x7fd7a00c8f50 msgr2=0x7fd7a015bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:15:54.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.458+0000 7fd7a6400640 1 --2- 172.21.15.6:0/770647199 >> v2:172.21.15.6:3300/0 conn(0x7fd7a00c8f50 0x7fd7a015bf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:54.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.458+0000 7fd7a6400640 1 -- 172.21.15.6:0/770647199 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7a01598d0 con 0x7fd7a00c9330 2024-04-24T23:15:54.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.458+0000 7fd7a5a00640 1 --2- 172.21.15.6:0/770647199 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd7a0152bc0 0x7fd7a0159390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:15:54.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.458+0000 7fd7a5000640 1 --2- 172.21.15.6:0/770647199 >> v2:172.21.15.6:3300/0 conn(0x7fd7a00c8f50 0x7fd7a015bf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:15:54.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.459+0000 7fd7a6400640 1 --2- 172.21.15.6:0/770647199 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd7a00c9330 0x7fd7a015c440 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fd7a0159aa0 tx=0x7fd79802a140 comp rx=0 tx=0).ready entity=mon.1 client_cookie=236ff67e09806212 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:15:54.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.460+0000 7fd79fe00640 1 -- 172.21.15.6:0/770647199 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd798037070 con 0x7fd7a00c9330 2024-04-24T23:15:54.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.460+0000 7fd79fe00640 1 -- 172.21.15.6:0/770647199 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd798032440 con 0x7fd7a00c9330 2024-04-24T23:15:54.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.460+0000 7fd79fe00640 1 -- 172.21.15.6:0/770647199 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd798032810 con 0x7fd7a00c9330 2024-04-24T23:15:54.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.460+0000 7fd7a8400640 1 -- 172.21.15.6:0/770647199 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd7a00c9330 msgr2=0x7fd7a015c440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:54.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.460+0000 7fd7a8400640 1 --2- 172.21.15.6:0/770647199 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd7a00c9330 0x7fd7a015c440 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fd7a0159aa0 tx=0x7fd79802a140 comp rx=0 tx=0).stop 2024-04-24T23:15:54.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.460+0000 7fd7a8400640 1 -- 172.21.15.6:0/770647199 shutdown_connections 2024-04-24T23:15:54.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.460+0000 7fd7a8400640 1 --2- 172.21.15.6:0/770647199 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd7a00c9330 0x7fd7a015c440 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:54.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.460+0000 7fd7a8400640 1 --2- 172.21.15.6:0/770647199 >> v2:172.21.15.6:3300/0 conn(0x7fd7a00c8f50 0x7fd7a015bf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:54.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.460+0000 7fd7a8400640 1 --2- 172.21.15.6:0/770647199 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd7a0152bc0 0x7fd7a0159390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:54.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.460+0000 7fd7a8400640 1 -- 172.21.15.6:0/770647199 >> 172.21.15.6:0/770647199 conn(0x7fd7a006b0e0 msgr2=0x7fd7a006b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:15:54.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.461+0000 7fd7a8400640 1 -- 172.21.15.6:0/770647199 shutdown_connections 2024-04-24T23:15:54.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.461+0000 7fd7a8400640 1 -- 172.21.15.6:0/770647199 wait complete. 2024-04-24T23:15:54.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.462+0000 7fd7a8400640 1 Processor -- start 2024-04-24T23:15:54.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.462+0000 7fd7a8400640 1 -- start start 2024-04-24T23:15:54.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.463+0000 7fd7a8400640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd7a00c8f50 0x7fd7a01e4710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:54.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.463+0000 7fd7a8400640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd7a00c9330 0x7fd7a0159d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:54.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.463+0000 7fd7a8400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd7a0152bc0 0x7fd7a01e29c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:54.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.463+0000 7fd7a8400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fd7a00c8470 con 0x7fd7a0152bc0 2024-04-24T23:15:54.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.463+0000 7fd7a8400640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fd7a00c6700 con 0x7fd7a00c8f50 2024-04-24T23:15:54.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.463+0000 7fd7a8400640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fd7a00c8750 con 0x7fd7a00c9330 2024-04-24T23:15:54.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.464+0000 7fd7a6400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd7a0152bc0 0x7fd7a01e29c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:54.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.464+0000 7fd7a6400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd7a0152bc0 0x7fd7a01e29c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57792/0 (socket says 172.21.15.6:57792) 2024-04-24T23:15:54.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.464+0000 7fd7a6400640 1 -- 172.21.15.6:0/2740673835 learned_addr learned my addr 172.21.15.6:0/2740673835 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:15:54.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.464+0000 7fd7a5a00640 1 --2- 172.21.15.6:0/2740673835 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd7a00c8f50 0x7fd7a01e4710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:54.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.464+0000 7fd7a5000640 1 --2- 172.21.15.6:0/2740673835 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd7a00c9330 0x7fd7a0159d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:54.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.464+0000 7fd7a6400640 1 -- 172.21.15.6:0/2740673835 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd7a00c9330 msgr2=0x7fd7a0159d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:54.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.464+0000 7fd7a6400640 1 --2- 172.21.15.6:0/2740673835 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd7a00c9330 0x7fd7a0159d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:54.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.464+0000 7fd7a6400640 1 -- 172.21.15.6:0/2740673835 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd7a00c8f50 msgr2=0x7fd7a01e4710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:54.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.464+0000 7fd7a6400640 1 --2- 172.21.15.6:0/2740673835 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd7a00c8f50 0x7fd7a01e4710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:54.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.464+0000 7fd7a6400640 1 -- 172.21.15.6:0/2740673835 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7a015c7a0 con 0x7fd7a0152bc0 2024-04-24T23:15:54.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.465+0000 7fd7a6400640 1 --2- 172.21.15.6:0/2740673835 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd7a0152bc0 0x7fd7a01e29c0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fd79802a180 tx=0x7fd7980046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:15:54.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.465+0000 7fd79d600640 1 -- 172.21.15.6:0/2740673835 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd798041020 con 0x7fd7a0152bc0 2024-04-24T23:15:54.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.465+0000 7fd79d600640 1 -- 172.21.15.6:0/2740673835 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd79803d640 con 0x7fd7a0152bc0 2024-04-24T23:15:54.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.465+0000 7fd79d600640 1 -- 172.21.15.6:0/2740673835 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd798037040 con 0x7fd7a0152bc0 2024-04-24T23:15:54.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.465+0000 7fd7a8400640 1 -- 172.21.15.6:0/2740673835 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7a015ca60 con 0x7fd7a0152bc0 2024-04-24T23:15:54.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.465+0000 7fd7a8400640 1 -- 172.21.15.6:0/2740673835 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7a015a8e0 con 0x7fd7a0152bc0 2024-04-24T23:15:54.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.466+0000 7fd7a8400640 1 -- 172.21.15.6:0/2740673835 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd7a006b9f0 con 0x7fd7a0152bc0 2024-04-24T23:15:54.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.468+0000 7fd79d600640 1 -- 172.21.15.6:0/2740673835 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fd798044020 con 0x7fd7a0152bc0 2024-04-24T23:15:54.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.468+0000 7fd79d600640 1 --2- 172.21.15.6:0/2740673835 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fd78006be60 0x7fd78006e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:54.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.469+0000 7fd79d600640 1 -- 172.21.15.6:0/2740673835 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fd7980ab3b0 con 0x7fd7a0152bc0 2024-04-24T23:15:54.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.469+0000 7fd7a5a00640 1 --2- 172.21.15.6:0/2740673835 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fd78006be60 0x7fd78006e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:54.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.470+0000 7fd7a5a00640 1 --2- 172.21.15.6:0/2740673835 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fd78006be60 0x7fd78006e320 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fd794009bd0 tx=0x7fd79400b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:15:54.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.473+0000 7fd79d600640 1 -- 172.21.15.6:0/2740673835 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fd7a006b9f0 con 0x7fd7a0152bc0 2024-04-24T23:15:54.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.678+0000 7fd7a8400640 1 -- 172.21.15.6:0/2740673835 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fd7a0152380 con 0x7fd7a0152bc0 2024-04-24T23:15:54.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.679+0000 7fd79d600640 1 -- 172.21.15.6:0/2740673835 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) ==== 104+0+11 (secure 0 0 0) 0x7fd7a0152380 con 0x7fd7a0152bc0 2024-04-24T23:15:54.681 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 32 2024-04-24T23:15:54.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.681+0000 7fd7a8400640 1 -- 172.21.15.6:0/2740673835 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fd78006be60 msgr2=0x7fd78006e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:54.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.681+0000 7fd7a8400640 1 --2- 172.21.15.6:0/2740673835 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fd78006be60 0x7fd78006e320 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fd794009bd0 tx=0x7fd79400b040 comp rx=0 tx=0).stop 2024-04-24T23:15:54.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.681+0000 7fd7a8400640 1 -- 172.21.15.6:0/2740673835 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd7a0152bc0 msgr2=0x7fd7a01e29c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:54.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.681+0000 7fd7a8400640 1 --2- 172.21.15.6:0/2740673835 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd7a0152bc0 0x7fd7a01e29c0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fd79802a180 tx=0x7fd7980046c0 comp rx=0 tx=0).stop 2024-04-24T23:15:54.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.681+0000 7fd7a8400640 1 -- 172.21.15.6:0/2740673835 shutdown_connections 2024-04-24T23:15:54.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.681+0000 7fd7a8400640 1 --2- 172.21.15.6:0/2740673835 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fd78006be60 0x7fd78006e320 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:54.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.681+0000 7fd7a8400640 1 --2- 172.21.15.6:0/2740673835 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd7a00c9330 0x7fd7a0159d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:54.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.681+0000 7fd7a8400640 1 --2- 172.21.15.6:0/2740673835 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd7a0152bc0 0x7fd7a01e29c0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:54.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.681+0000 7fd7a8400640 1 --2- 172.21.15.6:0/2740673835 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd7a00c8f50 0x7fd7a01e4710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:54.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.681+0000 7fd7a8400640 1 -- 172.21.15.6:0/2740673835 >> 172.21.15.6:0/2740673835 conn(0x7fd7a006b0e0 msgr2=0x7fd7a00c36c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:15:54.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.682+0000 7fd7a8400640 1 -- 172.21.15.6:0/2740673835 shutdown_connections 2024-04-24T23:15:54.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.682+0000 7fd7a8400640 1 -- 172.21.15.6:0/2740673835 wait complete. 2024-04-24T23:15:54.704 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-24T23:15:54.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.921+0000 7f0e91200640 1 Processor -- start 2024-04-24T23:15:54.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.921+0000 7f0e91200640 1 -- start start 2024-04-24T23:15:54.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.923+0000 7f0e91200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f0e8c151bb0 0x7f0e8c0c37f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:54.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.923+0000 7f0e91200640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f0e8c071530 0x7f0e8c0c8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:54.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.923+0000 7f0e91200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0e8c0c8390 con 0x7f0e8c0c9230 2024-04-24T23:15:54.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.923+0000 7f0e91200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f0e8c071d10 con 0x7f0e8c151bb0 2024-04-24T23:15:54.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.923+0000 7f0e91200640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f0e8c071ff0 con 0x7f0e8c071530 2024-04-24T23:15:54.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.923+0000 7f0e89e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f0e8c151bb0 0x7f0e8c0c37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:54.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.923+0000 7f0e89400640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f0e8c071530 0x7f0e8c0c8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:54.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.923+0000 7f0e89e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f0e8c151bb0 0x7f0e8c0c37f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:54140/0 (socket says 172.21.15.6:54140) 2024-04-24T23:15:54.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.923+0000 7f0e89e00640 1 -- 172.21.15.6:0/1026003454 learned_addr learned my addr 172.21.15.6:0/1026003454 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:15:54.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.924+0000 7f0e89e00640 1 -- 172.21.15.6:0/1026003454 >> v2:172.21.15.190:3300/0 conn(0x7f0e8c071530 msgr2=0x7f0e8c0c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:15:54.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.924+0000 7f0e89e00640 1 --2- 172.21.15.6:0/1026003454 >> v2:172.21.15.190:3300/0 conn(0x7f0e8c071530 0x7f0e8c0c8cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:54.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.924+0000 7f0e89e00640 1 -- 172.21.15.6:0/1026003454 >> v1:172.21.15.6:6789/0 conn(0x7f0e8c0c9230 legacy=0x7f0e8c157950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:54.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.924+0000 7f0e89e00640 1 -- 172.21.15.6:0/1026003454 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e8c0722d0 con 0x7f0e8c151bb0 2024-04-24T23:15:54.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.924+0000 7f0e89400640 1 --2- 172.21.15.6:0/1026003454 >> v2:172.21.15.190:3300/0 conn(0x7f0e8c071530 0x7f0e8c0c8cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:15:54.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.925+0000 7f0e89e00640 1 --2- 172.21.15.6:0/1026003454 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0e8c151bb0 0x7f0e8c0c37f0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f0e8c0ca1a0 tx=0x7f0e7c02a310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=326b572c80a3af61 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:15:54.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.925+0000 7f0e88a00640 1 -- 172.21.15.6:0/1026003454 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e7c037070 con 0x7f0e8c151bb0 2024-04-24T23:15:54.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.925+0000 7f0e88a00640 1 -- 172.21.15.6:0/1026003454 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0e7c032640 con 0x7f0e8c151bb0 2024-04-24T23:15:54.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.925+0000 7f0e88a00640 1 -- 172.21.15.6:0/1026003454 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e7c02ae60 con 0x7f0e8c151bb0 2024-04-24T23:15:54.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.925+0000 7f0e91200640 1 -- 172.21.15.6:0/1026003454 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0e8c151bb0 msgr2=0x7f0e8c0c37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:54.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.925+0000 7f0e91200640 1 --2- 172.21.15.6:0/1026003454 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0e8c151bb0 0x7f0e8c0c37f0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f0e8c0ca1a0 tx=0x7f0e7c02a310 comp rx=0 tx=0).stop 2024-04-24T23:15:54.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.926+0000 7f0e91200640 1 -- 172.21.15.6:0/1026003454 shutdown_connections 2024-04-24T23:15:54.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.926+0000 7f0e91200640 1 --2- 172.21.15.6:0/1026003454 >> v2:172.21.15.190:3300/0 conn(0x7f0e8c071530 0x7f0e8c0c8cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:54.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.926+0000 7f0e91200640 1 --2- 172.21.15.6:0/1026003454 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0e8c151bb0 0x7f0e8c0c37f0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:54.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.926+0000 7f0e91200640 1 -- 172.21.15.6:0/1026003454 >> 172.21.15.6:0/1026003454 conn(0x7f0e8c06b0e0 msgr2=0x7f0e8c06b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:15:54.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.926+0000 7f0e91200640 1 -- 172.21.15.6:0/1026003454 shutdown_connections 2024-04-24T23:15:54.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.926+0000 7f0e91200640 1 -- 172.21.15.6:0/1026003454 wait complete. 2024-04-24T23:15:54.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.927+0000 7f0e91200640 1 Processor -- start 2024-04-24T23:15:54.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.928+0000 7f0e91200640 1 -- start start 2024-04-24T23:15:54.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.928+0000 7f0e91200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0e8c071530 0x7f0e8c13bea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:54.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.929+0000 7f0e91200640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0e8c0c9230 0x7f0e8c1567f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:54.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.929+0000 7f0e91200640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0e8c151bb0 0x7f0e8c140fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:54.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.929+0000 7f0e91200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f0e8c0c3d30 con 0x7f0e8c071530 2024-04-24T23:15:54.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.929+0000 7f0e91200640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f0e8c0c7f70 con 0x7f0e8c0c9230 2024-04-24T23:15:54.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.929+0000 7f0e91200640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f0e8c140b20 con 0x7f0e8c151bb0 2024-04-24T23:15:54.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.929+0000 7f0e89e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0e8c071530 0x7f0e8c13bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:54.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.929+0000 7f0e89e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0e8c071530 0x7f0e8c13bea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:54152/0 (socket says 172.21.15.6:54152) 2024-04-24T23:15:54.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.929+0000 7f0e89e00640 1 -- 172.21.15.6:0/4030469760 learned_addr learned my addr 172.21.15.6:0/4030469760 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:15:54.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.929+0000 7f0e8a800640 1 --2- 172.21.15.6:0/4030469760 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0e8c151bb0 0x7f0e8c140fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:54.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.929+0000 7f0e89400640 1 --2- 172.21.15.6:0/4030469760 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0e8c0c9230 0x7f0e8c1567f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:54.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.929+0000 7f0e89e00640 1 -- 172.21.15.6:0/4030469760 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0e8c151bb0 msgr2=0x7f0e8c140fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:54.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.929+0000 7f0e89e00640 1 --2- 172.21.15.6:0/4030469760 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0e8c151bb0 0x7f0e8c140fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:54.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.929+0000 7f0e89e00640 1 -- 172.21.15.6:0/4030469760 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0e8c0c9230 msgr2=0x7f0e8c1567f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:54.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.929+0000 7f0e89e00640 1 --2- 172.21.15.6:0/4030469760 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0e8c0c9230 0x7f0e8c1567f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:54.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.929+0000 7f0e89e00640 1 -- 172.21.15.6:0/4030469760 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e8c157330 con 0x7f0e8c071530 2024-04-24T23:15:54.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.930+0000 7f0e89e00640 1 --2- 172.21.15.6:0/4030469760 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0e8c071530 0x7f0e8c13bea0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f0e7c02a350 tx=0x7f0e7c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:15:54.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.930+0000 7f0e82000640 1 -- 172.21.15.6:0/4030469760 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e7c037040 con 0x7f0e8c071530 2024-04-24T23:15:54.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.930+0000 7f0e82000640 1 -- 172.21.15.6:0/4030469760 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0e7c042070 con 0x7f0e8c071530 2024-04-24T23:15:54.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.930+0000 7f0e91200640 1 -- 172.21.15.6:0/4030469760 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e8c13a320 con 0x7f0e8c071530 2024-04-24T23:15:54.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.930+0000 7f0e82000640 1 -- 172.21.15.6:0/4030469760 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e7c02ae60 con 0x7f0e8c071530 2024-04-24T23:15:54.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.930+0000 7f0e91200640 1 -- 172.21.15.6:0/4030469760 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e8c1407b0 con 0x7f0e8c071530 2024-04-24T23:15:54.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.931+0000 7f0e91200640 1 -- 172.21.15.6:0/4030469760 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0e8c06e990 con 0x7f0e8c071530 2024-04-24T23:15:54.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.932+0000 7f0e82000640 1 -- 172.21.15.6:0/4030469760 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f0e7c030990 con 0x7f0e8c071530 2024-04-24T23:15:54.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.933+0000 7f0e82000640 1 --2- 172.21.15.6:0/4030469760 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f0e6806be80 0x7f0e6806e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:54.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.933+0000 7f0e82000640 1 -- 172.21.15.6:0/4030469760 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f0e7c0ab6e0 con 0x7f0e8c071530 2024-04-24T23:15:54.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.933+0000 7f0e89400640 1 --2- 172.21.15.6:0/4030469760 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f0e6806be80 0x7f0e6806e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:54.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.934+0000 7f0e89400640 1 --2- 172.21.15.6:0/4030469760 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f0e6806be80 0x7f0e6806e340 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f0e74009560 tx=0x7f0e74009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:15:54.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:54.938+0000 7f0e82000640 1 -- 172.21.15.6:0/4030469760 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f0e8c06e990 con 0x7f0e8c071530 2024-04-24T23:15:55.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:55.145+0000 7f0e91200640 1 -- 172.21.15.6:0/4030469760 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f0e8c0767b0 con 0x7f0e8c071530 2024-04-24T23:15:55.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:55.146+0000 7f0e82000640 1 -- 172.21.15.6:0/4030469760 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) ==== 100+0+11 (secure 0 0 0) 0x7f0e7c0796e0 con 0x7f0e8c071530 2024-04-24T23:15:55.149 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 64 2024-04-24T23:15:55.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:55.149+0000 7f0e91200640 1 -- 172.21.15.6:0/4030469760 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f0e6806be80 msgr2=0x7f0e6806e340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:55.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:55.149+0000 7f0e91200640 1 --2- 172.21.15.6:0/4030469760 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f0e6806be80 0x7f0e6806e340 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f0e74009560 tx=0x7f0e74009190 comp rx=0 tx=0).stop 2024-04-24T23:15:55.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:55.149+0000 7f0e91200640 1 -- 172.21.15.6:0/4030469760 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0e8c071530 msgr2=0x7f0e8c13bea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:55.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:55.149+0000 7f0e91200640 1 --2- 172.21.15.6:0/4030469760 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0e8c071530 0x7f0e8c13bea0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f0e7c02a350 tx=0x7f0e7c0046c0 comp rx=0 tx=0).stop 2024-04-24T23:15:55.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:55.149+0000 7f0e91200640 1 -- 172.21.15.6:0/4030469760 shutdown_connections 2024-04-24T23:15:55.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:55.149+0000 7f0e91200640 1 --2- 172.21.15.6:0/4030469760 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f0e6806be80 0x7f0e6806e340 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:55.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:55.149+0000 7f0e91200640 1 --2- 172.21.15.6:0/4030469760 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0e8c151bb0 0x7f0e8c140fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:55.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:55.149+0000 7f0e91200640 1 --2- 172.21.15.6:0/4030469760 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0e8c0c9230 0x7f0e8c1567f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:55.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:55.149+0000 7f0e91200640 1 --2- 172.21.15.6:0/4030469760 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0e8c071530 0x7f0e8c13bea0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:55.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:55.149+0000 7f0e91200640 1 -- 172.21.15.6:0/4030469760 >> 172.21.15.6:0/4030469760 conn(0x7f0e8c06b0e0 msgr2=0x7f0e8c150c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:15:55.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:55.149+0000 7f0e91200640 1 -- 172.21.15.6:0/4030469760 shutdown_connections 2024-04-24T23:15:55.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:55.149+0000 7f0e91200640 1 -- 172.21.15.6:0/4030469760 wait complete. 2024-04-24T23:15:55.172 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-24T23:15:55.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:55.376+0000 7f1787200640 1 Processor -- start 2024-04-24T23:15:55.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:55.376+0000 7f1787200640 1 -- start start 2024-04-24T23:15:55.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:55.377+0000 7f1787200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f1788154f60 0x7f178806c640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:55.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:55.377+0000 7f1787200640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f17880b35f0 0x7f178806cb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:55.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:55.377+0000 7f1787200640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f17880b46f0 con 0x7f178815ab20 2024-04-24T23:15:55.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:55.377+0000 7f1787200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f17880b49d0 con 0x7f1788154f60 2024-04-24T23:15:55.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:55.377+0000 7f1787200640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f17880b4cb0 con 0x7f17880b35f0 2024-04-24T23:15:55.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:55.378+0000 7f177fe00640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f17880b35f0 0x7f178806cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:55.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:55.378+0000 7f177fe00640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f17880b35f0 0x7f178806cb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:38410/0 (socket says 172.21.15.6:38410) 2024-04-24T23:15:55.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:55.378+0000 7f177fe00640 1 -- 172.21.15.6:0/1983697706 learned_addr learned my addr 172.21.15.6:0/1983697706 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:15:55.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:55.378+0000 7f1785e00640 1 --2- 172.21.15.6:0/1983697706 >> v2:172.21.15.6:3300/0 conn(0x7f1788154f60 0x7f178806c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:55.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:55.378+0000 7f177fe00640 1 -- 172.21.15.6:0/1983697706 >> v2:172.21.15.6:3300/0 conn(0x7f1788154f60 msgr2=0x7f178806c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:15:55.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:55.378+0000 7f177fe00640 1 --2- 172.21.15.6:0/1983697706 >> v2:172.21.15.6:3300/0 conn(0x7f1788154f60 0x7f178806c640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:55.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:55.378+0000 7f177fe00640 1 -- 172.21.15.6:0/1983697706 >> v1:172.21.15.190:6789/0 conn(0x7f178815ab20 legacy=0x7f178815af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:55.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:55.378+0000 7f177fe00640 1 -- 172.21.15.6:0/1983697706 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17880b4f90 con 0x7f17880b35f0 2024-04-24T23:15:55.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:55.379+0000 7f177fe00640 1 --2- 172.21.15.6:0/1983697706 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f17880b35f0 0x7f178806cb80 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f17880b62f0 tx=0x7f177002a310 comp rx=0 tx=0).ready entity=mon.1 client_cookie=15832632547539d1 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:15:55.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:55.380+0000 7f1785400640 1 -- 172.21.15.6:0/1983697706 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1770037070 con 0x7f17880b35f0 2024-04-24T23:15:55.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:55.380+0000 7f1785400640 1 -- 172.21.15.6:0/1983697706 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1770032640 con 0x7f17880b35f0 2024-04-24T23:15:55.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:55.380+0000 7f1785400640 1 -- 172.21.15.6:0/1983697706 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f177002ae60 con 0x7f17880b35f0 2024-04-24T23:15:55.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:55.380+0000 7f1787200640 1 -- 172.21.15.6:0/1983697706 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f17880b35f0 msgr2=0x7f178806cb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:55.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:55.380+0000 7f1787200640 1 --2- 172.21.15.6:0/1983697706 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f17880b35f0 0x7f178806cb80 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f17880b62f0 tx=0x7f177002a310 comp rx=0 tx=0).stop 2024-04-24T23:15:55.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:55.380+0000 7f1787200640 1 -- 172.21.15.6:0/1983697706 shutdown_connections 2024-04-24T23:15:55.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:55.380+0000 7f1787200640 1 --2- 172.21.15.6:0/1983697706 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f17880b35f0 0x7f178806cb80 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:55.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:55.380+0000 7f1787200640 1 --2- 172.21.15.6:0/1983697706 >> v2:172.21.15.6:3300/0 conn(0x7f1788154f60 0x7f178806c640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:55.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:55.380+0000 7f1787200640 1 -- 172.21.15.6:0/1983697706 >> 172.21.15.6:0/1983697706 conn(0x7f17880674d0 msgr2=0x7f17880d0320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:15:55.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:55.380+0000 7f1787200640 1 -- 172.21.15.6:0/1983697706 shutdown_connections 2024-04-24T23:15:55.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:55.380+0000 7f1787200640 1 -- 172.21.15.6:0/1983697706 wait complete. 2024-04-24T23:15:55.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:55.382+0000 7f1787200640 1 Processor -- start 2024-04-24T23:15:55.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:55.382+0000 7f1787200640 1 -- start start 2024-04-24T23:15:55.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:55.383+0000 7f1787200640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f17880b35f0 0x7f1788154110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:55.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:55.383+0000 7f1787200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1788154f60 0x7f17881f30e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:55.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:55.383+0000 7f1787200640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f178815ab20 0x7f17881f3620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:55.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:55.383+0000 7f1787200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f17880b3240 con 0x7f1788154f60 2024-04-24T23:15:55.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:55.383+0000 7f1787200640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f178806c120 con 0x7f17880b35f0 2024-04-24T23:15:55.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:55.383+0000 7f1787200640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f17880b33c0 con 0x7f178815ab20 2024-04-24T23:15:55.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:55.384+0000 7f177fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1788154f60 0x7f17881f30e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:55.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:55.384+0000 7f177fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1788154f60 0x7f17881f30e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:54170/0 (socket says 172.21.15.6:54170) 2024-04-24T23:15:55.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:55.384+0000 7f177fe00640 1 -- 172.21.15.6:0/365581824 learned_addr learned my addr 172.21.15.6:0/365581824 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:15:55.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:55.384+0000 7f1785e00640 1 --2- 172.21.15.6:0/365581824 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f17880b35f0 0x7f1788154110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:55.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:55.384+0000 7f1786800640 1 --2- 172.21.15.6:0/365581824 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f178815ab20 0x7f17881f3620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:55.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:55.384+0000 7f177fe00640 1 -- 172.21.15.6:0/365581824 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f178815ab20 msgr2=0x7f17881f3620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:55.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:55.384+0000 7f177fe00640 1 --2- 172.21.15.6:0/365581824 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f178815ab20 0x7f17881f3620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:55.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:55.384+0000 7f177fe00640 1 -- 172.21.15.6:0/365581824 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f17880b35f0 msgr2=0x7f1788154110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:55.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:55.384+0000 7f177fe00640 1 --2- 172.21.15.6:0/365581824 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f17880b35f0 0x7f1788154110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:55.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:55.384+0000 7f177fe00640 1 -- 172.21.15.6:0/365581824 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17880cfb10 con 0x7f1788154f60 2024-04-24T23:15:55.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:55.384+0000 7f177fe00640 1 --2- 172.21.15.6:0/365581824 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1788154f60 0x7f17881f30e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f177002a350 tx=0x7f17700046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:15:55.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:55.385+0000 7f177e000640 1 -- 172.21.15.6:0/365581824 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1770037040 con 0x7f1788154f60 2024-04-24T23:15:55.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:55.385+0000 7f177e000640 1 -- 172.21.15.6:0/365581824 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1770042070 con 0x7f1788154f60 2024-04-24T23:15:55.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:55.385+0000 7f177e000640 1 -- 172.21.15.6:0/365581824 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f177002ae60 con 0x7f1788154f60 2024-04-24T23:15:55.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:55.385+0000 7f1787200640 1 -- 172.21.15.6:0/365581824 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17881f5240 con 0x7f1788154f60 2024-04-24T23:15:55.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:55.385+0000 7f1787200640 1 -- 172.21.15.6:0/365581824 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17881f28e0 con 0x7f1788154f60 2024-04-24T23:15:55.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:55.386+0000 7f1787200640 1 -- 172.21.15.6:0/365581824 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1788158f60 con 0x7f1788154f60 2024-04-24T23:15:55.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:55.387+0000 7f177e000640 1 -- 172.21.15.6:0/365581824 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f1770030990 con 0x7f1788154f60 2024-04-24T23:15:55.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:55.388+0000 7f177e000640 1 --2- 172.21.15.6:0/365581824 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f176406be30 0x7f176406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:55.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:55.389+0000 7f177e000640 1 -- 172.21.15.6:0/365581824 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f17700ab710 con 0x7f1788154f60 2024-04-24T23:15:55.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:55.389+0000 7f1785e00640 1 --2- 172.21.15.6:0/365581824 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f176406be30 0x7f176406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:55.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:55.390+0000 7f1785e00640 1 --2- 172.21.15.6:0/365581824 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f176406be30 0x7f176406e2f0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f1778009b40 tx=0x7f1778009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:15:55.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:55.393+0000 7f177e000640 1 -- 172.21.15.6:0/365581824 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f1788158f60 con 0x7f1788154f60 2024-04-24T23:15:55.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:55.594+0000 7f1787200640 1 -- 172.21.15.6:0/365581824 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f178806d5c0 con 0x7f1788154f60 2024-04-24T23:15:55.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:55.595+0000 7f177e000640 1 -- 172.21.15.6:0/365581824 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v24) ==== 103+0+11 (secure 0 0 0) 0x7f178806d5c0 con 0x7f1788154f60 2024-04-24T23:15:55.598 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 32 2024-04-24T23:15:55.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:55.598+0000 7f1787200640 1 -- 172.21.15.6:0/365581824 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f176406be30 msgr2=0x7f176406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:55.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:55.598+0000 7f1787200640 1 --2- 172.21.15.6:0/365581824 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f176406be30 0x7f176406e2f0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f1778009b40 tx=0x7f1778009190 comp rx=0 tx=0).stop 2024-04-24T23:15:55.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:55.598+0000 7f1787200640 1 -- 172.21.15.6:0/365581824 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1788154f60 msgr2=0x7f17881f30e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:55.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:55.598+0000 7f1787200640 1 --2- 172.21.15.6:0/365581824 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1788154f60 0x7f17881f30e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f177002a350 tx=0x7f17700046c0 comp rx=0 tx=0).stop 2024-04-24T23:15:55.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:55.598+0000 7f1787200640 1 -- 172.21.15.6:0/365581824 shutdown_connections 2024-04-24T23:15:55.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:55.598+0000 7f1787200640 1 --2- 172.21.15.6:0/365581824 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f176406be30 0x7f176406e2f0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:55.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:55.598+0000 7f1787200640 1 --2- 172.21.15.6:0/365581824 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f178815ab20 0x7f17881f3620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:55.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:55.598+0000 7f1787200640 1 --2- 172.21.15.6:0/365581824 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1788154f60 0x7f17881f30e0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:55.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:55.598+0000 7f1787200640 1 --2- 172.21.15.6:0/365581824 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f17880b35f0 0x7f1788154110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:55.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:55.598+0000 7f1787200640 1 -- 172.21.15.6:0/365581824 >> 172.21.15.6:0/365581824 conn(0x7f17880674d0 msgr2=0x7f178806c640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:15:55.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:55.598+0000 7f1787200640 1 -- 172.21.15.6:0/365581824 shutdown_connections 2024-04-24T23:15:55.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:55.598+0000 7f1787200640 1 -- 172.21.15.6:0/365581824 wait complete. 2024-04-24T23:15:55.620 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-24T23:15:55.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:55.840+0000 7f70aaa00640 1 Processor -- start 2024-04-24T23:15:55.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:55.840+0000 7f70aaa00640 1 -- start start 2024-04-24T23:15:55.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:55.842+0000 7f70aaa00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f70a4152d40 0x7f70a41592b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:55.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:55.842+0000 7f70aaa00640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f70a40c4770 0x7f70a4163320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:55.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:55.842+0000 7f70aaa00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f70a4153120 con 0x7f70a406d6d0 2024-04-24T23:15:55.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:55.842+0000 7f70aaa00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f70a40c8aa0 con 0x7f70a4152d40 2024-04-24T23:15:55.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:55.842+0000 7f70aaa00640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f70a406d230 con 0x7f70a40c4770 2024-04-24T23:15:55.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:55.842+0000 7f70a3e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f70a4152d40 0x7f70a41592b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:55.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:55.842+0000 7f70a3e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f70a4152d40 0x7f70a41592b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:54186/0 (socket says 172.21.15.6:54186) 2024-04-24T23:15:55.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:55.842+0000 7f70a3e00640 1 -- 172.21.15.6:0/2349544424 learned_addr learned my addr 172.21.15.6:0/2349544424 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:15:55.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:55.842+0000 7f70a8a00640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f70a40c4770 0x7f70a4163320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:55.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:55.843+0000 7f70a3e00640 1 -- 172.21.15.6:0/2349544424 >> v2:172.21.15.190:3300/0 conn(0x7f70a40c4770 msgr2=0x7f70a4163320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:15:55.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:55.843+0000 7f70a3e00640 1 --2- 172.21.15.6:0/2349544424 >> v2:172.21.15.190:3300/0 conn(0x7f70a40c4770 0x7f70a4163320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:55.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:55.843+0000 7f70a3e00640 1 -- 172.21.15.6:0/2349544424 >> v1:172.21.15.6:6789/0 conn(0x7f70a406d6d0 legacy=0x7f70a40c4060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:55.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:55.843+0000 7f70a3e00640 1 -- 172.21.15.6:0/2349544424 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70a40c63d0 con 0x7f70a4152d40 2024-04-24T23:15:55.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:55.844+0000 7f70a3e00640 1 --2- 172.21.15.6:0/2349544424 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f70a4152d40 0x7f70a41592b0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f70a406ddd0 tx=0x7f709402a310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ad2a16b697f0584 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:15:55.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:55.844+0000 7f70a2a00640 1 -- 172.21.15.6:0/2349544424 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7094037070 con 0x7f70a4152d40 2024-04-24T23:15:55.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:55.844+0000 7f70a2a00640 1 -- 172.21.15.6:0/2349544424 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7094032640 con 0x7f70a4152d40 2024-04-24T23:15:55.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:55.844+0000 7f70a2a00640 1 -- 172.21.15.6:0/2349544424 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f709402ae60 con 0x7f70a4152d40 2024-04-24T23:15:55.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:55.845+0000 7f70aaa00640 1 -- 172.21.15.6:0/2349544424 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f70a4152d40 msgr2=0x7f70a41592b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:55.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:55.845+0000 7f70aaa00640 1 --2- 172.21.15.6:0/2349544424 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f70a4152d40 0x7f70a41592b0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f70a406ddd0 tx=0x7f709402a310 comp rx=0 tx=0).stop 2024-04-24T23:15:55.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:55.845+0000 7f70aaa00640 1 -- 172.21.15.6:0/2349544424 shutdown_connections 2024-04-24T23:15:55.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:55.845+0000 7f70aaa00640 1 --2- 172.21.15.6:0/2349544424 >> v2:172.21.15.190:3300/0 conn(0x7f70a40c4770 0x7f70a4163320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:55.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:55.845+0000 7f70aaa00640 1 --2- 172.21.15.6:0/2349544424 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f70a4152d40 0x7f70a41592b0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:55.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:55.845+0000 7f70aaa00640 1 -- 172.21.15.6:0/2349544424 >> 172.21.15.6:0/2349544424 conn(0x7f70a40c8c50 msgr2=0x7f70a40c9060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:15:55.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:55.845+0000 7f70aaa00640 1 -- 172.21.15.6:0/2349544424 shutdown_connections 2024-04-24T23:15:55.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:55.845+0000 7f70aaa00640 1 -- 172.21.15.6:0/2349544424 wait complete. 2024-04-24T23:15:55.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:55.847+0000 7f70aaa00640 1 Processor -- start 2024-04-24T23:15:55.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:55.847+0000 7f70aaa00640 1 -- start start 2024-04-24T23:15:55.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:55.848+0000 7f70aaa00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f70a406d6d0 0x7f70a40c3460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:55.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:55.848+0000 7f70aaa00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f70a40c4770 0x7f70a41f0d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:55.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:55.848+0000 7f70aaa00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f70a4152d40 0x7f70a41f1270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:55.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:55.848+0000 7f70aaa00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f70a406f8d0 con 0x7f70a4152d40 2024-04-24T23:15:55.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:55.848+0000 7f70aaa00640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f70a4158ab0 con 0x7f70a406d6d0 2024-04-24T23:15:55.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:55.848+0000 7f70aaa00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f70a40c6a50 con 0x7f70a40c4770 2024-04-24T23:15:55.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:55.848+0000 7f70a8a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f70a4152d40 0x7f70a41f1270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:55.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:55.848+0000 7f70a8a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f70a4152d40 0x7f70a41f1270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:54194/0 (socket says 172.21.15.6:54194) 2024-04-24T23:15:55.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:55.848+0000 7f70a8a00640 1 -- 172.21.15.6:0/2092576415 learned_addr learned my addr 172.21.15.6:0/2092576415 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:15:55.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:55.848+0000 7f70a3e00640 1 --2- 172.21.15.6:0/2092576415 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f70a406d6d0 0x7f70a40c3460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:55.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:55.848+0000 7f70a3400640 1 --2- 172.21.15.6:0/2092576415 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f70a40c4770 0x7f70a41f0d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:55.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:55.849+0000 7f70a3e00640 1 -- 172.21.15.6:0/2092576415 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f70a40c4770 msgr2=0x7f70a41f0d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:55.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:55.849+0000 7f70a3e00640 1 --2- 172.21.15.6:0/2092576415 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f70a40c4770 0x7f70a41f0d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:55.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:55.849+0000 7f70a3e00640 1 -- 172.21.15.6:0/2092576415 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f70a4152d40 msgr2=0x7f70a41f1270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:55.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:55.849+0000 7f70a3e00640 1 --2- 172.21.15.6:0/2092576415 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f70a4152d40 0x7f70a41f1270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:55.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:55.849+0000 7f70a3e00640 1 -- 172.21.15.6:0/2092576415 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70a414e7c0 con 0x7f70a406d6d0 2024-04-24T23:15:55.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:55.849+0000 7f70a8a00640 1 --2- 172.21.15.6:0/2092576415 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f70a4152d40 0x7f70a41f1270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:15:55.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:55.849+0000 7f70a3e00640 1 --2- 172.21.15.6:0/2092576415 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f70a406d6d0 0x7f70a40c3460 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f7094031000 tx=0x7f70940046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:15:55.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:55.849+0000 7f709be00640 1 -- 172.21.15.6:0/2092576415 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7094037040 con 0x7f70a406d6d0 2024-04-24T23:15:55.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:55.849+0000 7f709be00640 1 -- 172.21.15.6:0/2092576415 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7094042070 con 0x7f70a406d6d0 2024-04-24T23:15:55.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:55.849+0000 7f70aaa00640 1 -- 172.21.15.6:0/2092576415 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70a414eaa0 con 0x7f70a406d6d0 2024-04-24T23:15:55.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:55.849+0000 7f709be00640 1 -- 172.21.15.6:0/2092576415 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f709402ae60 con 0x7f70a406d6d0 2024-04-24T23:15:55.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:55.849+0000 7f70aaa00640 1 -- 172.21.15.6:0/2092576415 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70a414f940 con 0x7f70a406d6d0 2024-04-24T23:15:55.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:55.850+0000 7f70aaa00640 1 -- 172.21.15.6:0/2092576415 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f70a40767b0 con 0x7f70a406d6d0 2024-04-24T23:15:55.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:55.852+0000 7f709be00640 1 -- 172.21.15.6:0/2092576415 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f70940309b0 con 0x7f70a406d6d0 2024-04-24T23:15:55.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:55.852+0000 7f709be00640 1 --2- 172.21.15.6:0/2092576415 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f708406be30 0x7f708406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:55.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:55.853+0000 7f709be00640 1 -- 172.21.15.6:0/2092576415 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f70940ab8a0 con 0x7f70a406d6d0 2024-04-24T23:15:55.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:55.853+0000 7f70a3400640 1 --2- 172.21.15.6:0/2092576415 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f708406be30 0x7f708406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:55.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:55.853+0000 7f70a3400640 1 --2- 172.21.15.6:0/2092576415 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f708406be30 0x7f708406e2f0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f70a415d020 tx=0x7f7090009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:15:55.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:55.857+0000 7f709be00640 1 -- 172.21.15.6:0/2092576415 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f70940798b0 con 0x7f70a406d6d0 2024-04-24T23:15:56.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:56.078+0000 7f70aaa00640 1 -- 172.21.15.6:0/2092576415 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f70a41f0670 con 0x7f70a406d6d0 2024-04-24T23:15:56.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:56.079+0000 7f709be00640 1 -- 172.21.15.6:0/2092576415 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v24) ==== 80+0+0 (secure 0 0 0) 0x7f709407dbb0 con 0x7f70a406d6d0 2024-04-24T23:15:56.081 INFO:teuthology.orchestra.run.smithi006.stderr:listed 0 entries 2024-04-24T23:15:56.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:56.082+0000 7f70aaa00640 1 -- 172.21.15.6:0/2092576415 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f708406be30 msgr2=0x7f708406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:56.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:56.082+0000 7f70aaa00640 1 --2- 172.21.15.6:0/2092576415 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f708406be30 0x7f708406e2f0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f70a415d020 tx=0x7f7090009040 comp rx=0 tx=0).stop 2024-04-24T23:15:56.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:56.082+0000 7f70aaa00640 1 -- 172.21.15.6:0/2092576415 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f70a406d6d0 msgr2=0x7f70a40c3460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:56.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:56.082+0000 7f70aaa00640 1 --2- 172.21.15.6:0/2092576415 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f70a406d6d0 0x7f70a40c3460 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f7094031000 tx=0x7f70940046c0 comp rx=0 tx=0).stop 2024-04-24T23:15:56.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:56.082+0000 7f70aaa00640 1 -- 172.21.15.6:0/2092576415 shutdown_connections 2024-04-24T23:15:56.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:56.082+0000 7f70aaa00640 1 --2- 172.21.15.6:0/2092576415 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f708406be30 0x7f708406e2f0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:56.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:56.082+0000 7f70aaa00640 1 --2- 172.21.15.6:0/2092576415 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f70a40c4770 0x7f70a41f0d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:56.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:56.082+0000 7f70aaa00640 1 --2- 172.21.15.6:0/2092576415 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f70a4152d40 0x7f70a41f1270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:56.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:56.082+0000 7f70aaa00640 1 --2- 172.21.15.6:0/2092576415 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f70a406d6d0 0x7f70a40c3460 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:56.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:56.082+0000 7f70aaa00640 1 -- 172.21.15.6:0/2092576415 >> 172.21.15.6:0/2092576415 conn(0x7f70a40c8c50 msgr2=0x7f70a40c4060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:15:56.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:56.082+0000 7f70aaa00640 1 -- 172.21.15.6:0/2092576415 shutdown_connections 2024-04-24T23:15:56.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:56.082+0000 7f70aaa00640 1 -- 172.21.15.6:0/2092576415 wait complete. 2024-04-24T23:15:56.103 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-04-24T23:15:56.103 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-24T23:15:56.156 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T23:15:56.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:56.366+0000 7f2d08e00640 1 Processor -- start 2024-04-24T23:15:56.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:56.367+0000 7f2d08e00640 1 -- start start 2024-04-24T23:15:56.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:56.368+0000 7f2d08e00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2d04152bc0 0x7f2d04159390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:56.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:56.368+0000 7f2d08e00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2d0415cd20 con 0x7f2d040c8f50 2024-04-24T23:15:56.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:56.368+0000 7f2d08e00640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2d040c6e70 con 0x7f2d040c9330 2024-04-24T23:15:56.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:56.368+0000 7f2d08e00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f2d040c73d0 con 0x7f2d04152bc0 2024-04-24T23:15:56.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:56.369+0000 7f2d02800640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f2d040c9330 0x7f2d0415c610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.6:56302/0 (socket says 172.21.15.6:56302) 2024-04-24T23:15:56.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:56.369+0000 7f2d01400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f2d040c8f50 0x7f2d0415bf00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:42934/0 (socket says 172.21.15.6:42934) 2024-04-24T23:15:56.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:56.369+0000 7f2d02800640 1 -- 172.21.15.6:0/227466338 learned_addr learned my addr 172.21.15.6:0/227466338 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:15:56.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:56.369+0000 7f2d01e00640 1 --2- 172.21.15.6:0/227466338 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2d04152bc0 0x7f2d04159390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:56.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:56.369+0000 7f2d00a00640 1 -- 172.21.15.6:0/227466338 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3053379602 0 0) 0x7f2d040c6e70 con 0x7f2d040c9330 2024-04-24T23:15:56.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:56.369+0000 7f2d00a00640 1 -- 172.21.15.6:0/227466338 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2ce8003520 con 0x7f2d040c9330 2024-04-24T23:15:56.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:56.369+0000 7f2d01e00640 1 -- 172.21.15.6:0/227466338 >> v1:172.21.15.190:6789/0 conn(0x7f2d040c9330 legacy=0x7f2d0415c610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:56.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:56.369+0000 7f2d01e00640 1 -- 172.21.15.6:0/227466338 >> v1:172.21.15.6:6789/0 conn(0x7f2d040c8f50 legacy=0x7f2d0415bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:56.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:56.369+0000 7f2d00a00640 1 -- 172.21.15.6:0/227466338 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4182526793 0 0) 0x7f2d0415cd20 con 0x7f2d040c8f50 2024-04-24T23:15:56.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:56.369+0000 7f2d01e00640 1 -- 172.21.15.6:0/227466338 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d041598d0 con 0x7f2d04152bc0 2024-04-24T23:15:56.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:56.370+0000 7f2d01e00640 1 --2- 172.21.15.6:0/227466338 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2d04152bc0 0x7f2d04159390 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f2d040c7550 tx=0x7f2cf402a160 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2eba37704c334b40 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:15:56.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:56.371+0000 7f2d00a00640 1 -- 172.21.15.6:0/227466338 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cf4037070 con 0x7f2d04152bc0 2024-04-24T23:15:56.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:56.371+0000 7f2d00a00640 1 -- 172.21.15.6:0/227466338 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2cf4032440 con 0x7f2d04152bc0 2024-04-24T23:15:56.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:56.371+0000 7f2d00a00640 1 -- 172.21.15.6:0/227466338 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cf40327f0 con 0x7f2d04152bc0 2024-04-24T23:15:56.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:56.372+0000 7f2d08e00640 1 -- 172.21.15.6:0/227466338 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2d04152bc0 msgr2=0x7f2d04159390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:56.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:56.372+0000 7f2d08e00640 1 --2- 172.21.15.6:0/227466338 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2d04152bc0 0x7f2d04159390 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f2d040c7550 tx=0x7f2cf402a160 comp rx=0 tx=0).stop 2024-04-24T23:15:56.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:56.372+0000 7f2d08e00640 1 -- 172.21.15.6:0/227466338 shutdown_connections 2024-04-24T23:15:56.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:56.372+0000 7f2d08e00640 1 --2- 172.21.15.6:0/227466338 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2d04152bc0 0x7f2d04159390 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:56.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:56.372+0000 7f2d08e00640 1 -- 172.21.15.6:0/227466338 >> 172.21.15.6:0/227466338 conn(0x7f2d0406b0e0 msgr2=0x7f2d0406b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:15:56.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:56.372+0000 7f2d08e00640 1 -- 172.21.15.6:0/227466338 shutdown_connections 2024-04-24T23:15:56.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:56.372+0000 7f2d08e00640 1 -- 172.21.15.6:0/227466338 wait complete. 2024-04-24T23:15:56.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:56.374+0000 7f2d08e00640 1 Processor -- start 2024-04-24T23:15:56.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:56.374+0000 7f2d08e00640 1 -- start start 2024-04-24T23:15:56.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:56.375+0000 7f2d08e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2d040c8f50 0x7f2d04153200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:56.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:56.375+0000 7f2d08e00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2d040c9330 0x7f2d040caa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:56.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:56.375+0000 7f2d08e00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2d04152bc0 0x7f2d04144be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:56.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:56.375+0000 7f2d08e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f2d040c6700 con 0x7f2d040c8f50 2024-04-24T23:15:56.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:56.375+0000 7f2d08e00640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f2d040c8ac0 con 0x7f2d040c9330 2024-04-24T23:15:56.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:56.375+0000 7f2d08e00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f2d040c8470 con 0x7f2d04152bc0 2024-04-24T23:15:56.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:56.376+0000 7f2d01e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2d040c8f50 0x7f2d04153200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:56.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:56.376+0000 7f2d01e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2d040c8f50 0x7f2d04153200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:54200/0 (socket says 172.21.15.6:54200) 2024-04-24T23:15:56.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:56.376+0000 7f2d01400640 1 --2- 172.21.15.6:0/3844307428 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2d040c9330 0x7f2d040caa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:56.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:56.376+0000 7f2d01e00640 1 -- 172.21.15.6:0/3844307428 learned_addr learned my addr 172.21.15.6:0/3844307428 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:15:56.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:56.376+0000 7f2d02800640 1 --2- 172.21.15.6:0/3844307428 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2d04152bc0 0x7f2d04144be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:56.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:56.376+0000 7f2d01e00640 1 -- 172.21.15.6:0/3844307428 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2d04152bc0 msgr2=0x7f2d04144be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:56.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:56.376+0000 7f2d01e00640 1 --2- 172.21.15.6:0/3844307428 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2d04152bc0 0x7f2d04144be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:56.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:56.376+0000 7f2d01e00640 1 -- 172.21.15.6:0/3844307428 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2d040c9330 msgr2=0x7f2d040caa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:56.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:56.376+0000 7f2d01e00640 1 --2- 172.21.15.6:0/3844307428 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2d040c9330 0x7f2d040caa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:56.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:56.376+0000 7f2d01e00640 1 -- 172.21.15.6:0/3844307428 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d04159650 con 0x7f2d040c8f50 2024-04-24T23:15:56.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:56.376+0000 7f2d01400640 1 --2- 172.21.15.6:0/3844307428 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2d040c9330 0x7f2d040caa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:15:56.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:56.376+0000 7f2d02800640 1 --2- 172.21.15.6:0/3844307428 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2d04152bc0 0x7f2d04144be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:15:56.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:56.377+0000 7f2d01e00640 1 --2- 172.21.15.6:0/3844307428 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2d040c8f50 0x7f2d04153200 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f2cf4000c00 tx=0x7f2cf40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:15:56.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:56.377+0000 7f2cfa000640 1 -- 172.21.15.6:0/3844307428 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cf4040020 con 0x7f2d040c8f50 2024-04-24T23:15:56.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:56.377+0000 7f2cfa000640 1 -- 172.21.15.6:0/3844307428 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2cf4038040 con 0x7f2d040c8f50 2024-04-24T23:15:56.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:56.377+0000 7f2d08e00640 1 -- 172.21.15.6:0/3844307428 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d04158ad0 con 0x7f2d040c8f50 2024-04-24T23:15:56.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:56.377+0000 7f2cfa000640 1 -- 172.21.15.6:0/3844307428 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cf4037070 con 0x7f2d040c8f50 2024-04-24T23:15:56.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:56.377+0000 7f2d08e00640 1 -- 172.21.15.6:0/3844307428 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d04159090 con 0x7f2d040c8f50 2024-04-24T23:15:56.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:56.379+0000 7f2d08e00640 1 -- 172.21.15.6:0/3844307428 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d040767b0 con 0x7f2d040c8f50 2024-04-24T23:15:56.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:56.380+0000 7f2cfa000640 1 -- 172.21.15.6:0/3844307428 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f2cf4043020 con 0x7f2d040c8f50 2024-04-24T23:15:56.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:56.381+0000 7f2cfa000640 1 --2- 172.21.15.6:0/3844307428 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f2ce406be60 0x7f2ce406e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:56.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:56.381+0000 7f2cfa000640 1 -- 172.21.15.6:0/3844307428 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f2cf40aae50 con 0x7f2d040c8f50 2024-04-24T23:15:56.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:56.381+0000 7f2d01400640 1 --2- 172.21.15.6:0/3844307428 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f2ce406be60 0x7f2ce406e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:56.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:56.382+0000 7f2d01400640 1 --2- 172.21.15.6:0/3844307428 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f2ce406be60 0x7f2ce406e320 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f2d0415b380 tx=0x7f2cf0007d50 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:15:56.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:56.386+0000 7f2cfa000640 1 -- 172.21.15.6:0/3844307428 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f2cf404a070 con 0x7f2d040c8f50 2024-04-24T23:15:56.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:56.587+0000 7f2d08e00640 1 -- 172.21.15.6:0/3844307428 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2d0415b3f0 con 0x7f2d040c8f50 2024-04-24T23:15:56.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:56.589+0000 7f2cfa000640 1 -- 172.21.15.6:0/3844307428 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+18458 (secure 0 0 0) 0x7f2cf4078d50 con 0x7f2d040c8f50 2024-04-24T23:15:56.591 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:15:56.591 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":24,"fsid":"18bef68d-5e6d-4ec5-862b-b6f206e2588d","created":"2024-04-24T23:15:12.535072+0000","modified":"2024-04-24T23:15:41.465875+0000","last_up_change":"2024-04-24T23:15:23.773223+0000","last_in_change":"2024-04-24T23:15:14.905328+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T23:15:24.747441+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-24T23:15:34.051800+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-24T23:15:35.436870+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-24T23:15:39.034257+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"3b7a212d-d498-462e-90bf-ed493c803a09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":3022569437},{"type":"v1","addr":"172.21.15.6:6809","nonce":3022569437}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":3022569437},{"type":"v1","addr":"172.21.15.6:6811","nonce":3022569437}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":3022569437},{"type":"v1","addr":"172.21.15.6:6815","nonce":3022569437}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":3022569437},{"type":"v1","addr":"172.21.15.6:6813","nonce":3022569437}]},"public_addr":"172.21.15.6:6809/3022569437","cluster_addr":"172.21.15.6:6811/3022569437","heartbeat_back_addr":"172.21.15.6:6815/3022569437","heartbeat_front_addr":"172.21.15.6:6813/3022569437","state":["exists","up"]},{"osd":1,"uuid":"7c97f5a2-1735-4775-9f15-eb50099b19d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":3687836888},{"type":"v1","addr":"172.21.15.6:6817","nonce":3687836888}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":3687836888},{"type":"v1","addr":"172.21.15.6:6819","nonce":3687836888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":3687836888},{"type":"v1","addr":"172.21.15.6:6823","nonce":3687836888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":3687836888},{"type":"v1","addr":"172.21.15.6:6821","nonce":3687836888}]},"public_addr":"172.21.15.6:6817/3687836888","cluster_addr":"172.21.15.6:6819/3687836888","heartbeat_back_addr":"172.21.15.6:6823/3687836888","heartbeat_front_addr":"172.21.15.6:6821/3687836888","state":["exists","up"]},{"osd":2,"uuid":"d86bbc18-438c-4ec2-8018-ba7dd78f790b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":1020213724},{"type":"v1","addr":"172.21.15.6:6801","nonce":1020213724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":1020213724},{"type":"v1","addr":"172.21.15.6:6803","nonce":1020213724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":1020213724},{"type":"v1","addr":"172.21.15.6:6807","nonce":1020213724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":1020213724},{"type":"v1","addr":"172.21.15.6:6805","nonce":1020213724}]},"public_addr":"172.21.15.6:6801/1020213724","cluster_addr":"172.21.15.6:6803/1020213724","heartbeat_back_addr":"172.21.15.6:6807/1020213724","heartbeat_front_addr":"172.21.15.6:6805/1020213724","state":["exists","up"]},{"osd":3,"uuid":"3754ae16-4248-40dd-95f9-0f59c0eb5426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6824","nonce":3163883468},{"type":"v1","addr":"172.21.15.6:6825","nonce":3163883468}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":3163883468},{"type":"v1","addr":"172.21.15.6:6827","nonce":3163883468}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":3163883468},{"type":"v1","addr":"172.21.15.6:6831","nonce":3163883468}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":3163883468},{"type":"v1","addr":"172.21.15.6:6829","nonce":3163883468}]},"public_addr":"172.21.15.6:6825/3163883468","cluster_addr":"172.21.15.6:6827/3163883468","heartbeat_back_addr":"172.21.15.6:6831/3163883468","heartbeat_front_addr":"172.21.15.6:6829/3163883468","state":["exists","up"]},{"osd":4,"uuid":"8b2913a0-d006-479c-84ed-5b21e0e6c392","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":1989500934},{"type":"v1","addr":"172.21.15.190:6809","nonce":1989500934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":1989500934},{"type":"v1","addr":"172.21.15.190:6811","nonce":1989500934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":1989500934},{"type":"v1","addr":"172.21.15.190:6815","nonce":1989500934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":1989500934},{"type":"v1","addr":"172.21.15.190:6813","nonce":1989500934}]},"public_addr":"172.21.15.190:6809/1989500934","cluster_addr":"172.21.15.190:6811/1989500934","heartbeat_back_addr":"172.21.15.190:6815/1989500934","heartbeat_front_addr":"172.21.15.190:6813/1989500934","state":["exists","up"]},{"osd":5,"uuid":"499b79d0-0585-4dae-ab69-196a551793da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":3687548412},{"type":"v1","addr":"172.21.15.190:6801","nonce":3687548412}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":3687548412},{"type":"v1","addr":"172.21.15.190:6803","nonce":3687548412}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":3687548412},{"type":"v1","addr":"172.21.15.190:6807","nonce":3687548412}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":3687548412},{"type":"v1","addr":"172.21.15.190:6805","nonce":3687548412}]},"public_addr":"172.21.15.190:6801/3687548412","cluster_addr":"172.21.15.190:6803/3687548412","heartbeat_back_addr":"172.21.15.190:6807/3687548412","heartbeat_front_addr":"172.21.15.190:6805/3687548412","state":["exists","up"]},{"osd":6,"uuid":"6ba6df3b-4a6b-4a89-a890-0f79f387d4d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":870571564},{"type":"v1","addr":"172.21.15.190:6817","nonce":870571564}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":870571564},{"type":"v1","addr":"172.21.15.190:6819","nonce":870571564}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":870571564},{"type":"v1","addr":"172.21.15.190:6823","nonce":870571564}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":870571564},{"type":"v1","addr":"172.21.15.190:6821","nonce":870571564}]},"public_addr":"172.21.15.190:6817/870571564","cluster_addr":"172.21.15.190:6819/870571564","heartbeat_back_addr":"172.21.15.190:6823/870571564","heartbeat_front_addr":"172.21.15.190:6821/870571564","state":["exists","up"]},{"osd":7,"uuid":"e9668b1c-a000-4332-86e8-4efd73b02409","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":123639197},{"type":"v1","addr":"172.21.15.190:6825","nonce":123639197}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":123639197},{"type":"v1","addr":"172.21.15.190:6827","nonce":123639197}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":123639197},{"type":"v1","addr":"172.21.15.190:6831","nonce":123639197}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":123639197},{"type":"v1","addr":"172.21.15.190:6829","nonce":123639197}]},"public_addr":"172.21.15.190:6825/123639197","cluster_addr":"172.21.15.190:6827/123639197","heartbeat_back_addr":"172.21.15.190:6831/123639197","heartbeat_front_addr":"172.21.15.190:6829/123639197","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.321646+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.363873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.466963+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.531989+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:19.990707+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.043543+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.032492+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.115004+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T23:15:56.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:56.591+0000 7f2d08e00640 1 -- 172.21.15.6:0/3844307428 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f2ce406be60 msgr2=0x7f2ce406e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:56.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:56.591+0000 7f2d08e00640 1 --2- 172.21.15.6:0/3844307428 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f2ce406be60 0x7f2ce406e320 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f2d0415b380 tx=0x7f2cf0007d50 comp rx=0 tx=0).stop 2024-04-24T23:15:56.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:56.591+0000 7f2d08e00640 1 -- 172.21.15.6:0/3844307428 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2d040c8f50 msgr2=0x7f2d04153200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:56.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:56.591+0000 7f2d08e00640 1 --2- 172.21.15.6:0/3844307428 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2d040c8f50 0x7f2d04153200 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f2cf4000c00 tx=0x7f2cf40046c0 comp rx=0 tx=0).stop 2024-04-24T23:15:56.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:56.591+0000 7f2d08e00640 1 -- 172.21.15.6:0/3844307428 shutdown_connections 2024-04-24T23:15:56.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:56.591+0000 7f2d08e00640 1 --2- 172.21.15.6:0/3844307428 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f2ce406be60 0x7f2ce406e320 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:56.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:56.591+0000 7f2d08e00640 1 --2- 172.21.15.6:0/3844307428 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2d04152bc0 0x7f2d04144be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:56.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:56.591+0000 7f2d08e00640 1 --2- 172.21.15.6:0/3844307428 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2d040c9330 0x7f2d040caa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:56.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:56.591+0000 7f2d08e00640 1 --2- 172.21.15.6:0/3844307428 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2d040c8f50 0x7f2d04153200 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:56.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:56.591+0000 7f2d08e00640 1 -- 172.21.15.6:0/3844307428 >> 172.21.15.6:0/3844307428 conn(0x7f2d0406b0e0 msgr2=0x7f2d0406e130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:15:56.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:56.592+0000 7f2d08e00640 1 -- 172.21.15.6:0/3844307428 shutdown_connections 2024-04-24T23:15:56.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:56.592+0000 7f2d08e00640 1 -- 172.21.15.6:0/3844307428 wait complete. 2024-04-24T23:15:56.614 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T23:15:24.747441+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-24T23:15:34.051800+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-24T23:15:35.436870+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-24T23:15:39.034257+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-24T23:15:56.615 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-24T23:15:56.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:56.827+0000 7f68f7a00640 1 Processor -- start 2024-04-24T23:15:56.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:56.828+0000 7f68f7a00640 1 -- start start 2024-04-24T23:15:56.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:56.829+0000 7f68f7a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f68f0152bc0 0x7f68f0159390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:56.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:56.829+0000 7f68f7a00640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f68f00c8f50 0x7f68f015bf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:56.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:56.829+0000 7f68f7a00640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f68f00c6e70 con 0x7f68f00c9360 2024-04-24T23:15:56.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:56.829+0000 7f68f7a00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f68f00c73d0 con 0x7f68f0152bc0 2024-04-24T23:15:56.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:56.829+0000 7f68f7a00640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f68f00c8ac0 con 0x7f68f00c8f50 2024-04-24T23:15:56.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:56.829+0000 7f68f5000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f68f0152bc0 0x7f68f0159390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:56.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:56.829+0000 7f68f5000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f68f0152bc0 0x7f68f0159390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:54212/0 (socket says 172.21.15.6:54212) 2024-04-24T23:15:56.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:56.829+0000 7f68f5000640 1 -- 172.21.15.6:0/3257235522 learned_addr learned my addr 172.21.15.6:0/3257235522 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:15:56.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:56.829+0000 7f68efe00640 1 --2- 172.21.15.6:0/3257235522 >> v2:172.21.15.190:3300/0 conn(0x7f68f00c8f50 0x7f68f015bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:56.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:56.830+0000 7f68f5000640 1 -- 172.21.15.6:0/3257235522 >> v2:172.21.15.190:3300/0 conn(0x7f68f00c8f50 msgr2=0x7f68f015bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:15:56.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:56.830+0000 7f68f5000640 1 --2- 172.21.15.6:0/3257235522 >> v2:172.21.15.190:3300/0 conn(0x7f68f00c8f50 0x7f68f015bf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:56.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:56.830+0000 7f68f5000640 1 -- 172.21.15.6:0/3257235522 >> v1:172.21.15.190:6789/0 conn(0x7f68f00c9360 legacy=0x7f68f015c440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:56.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:56.830+0000 7f68f5000640 1 -- 172.21.15.6:0/3257235522 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68f01598d0 con 0x7f68f0152bc0 2024-04-24T23:15:56.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:56.831+0000 7f68f5000640 1 --2- 172.21.15.6:0/3257235522 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f68f0152bc0 0x7f68f0159390 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f68f00c7550 tx=0x7f68e002a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5d547c133618a24 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:15:56.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:56.831+0000 7f68ef400640 1 -- 172.21.15.6:0/3257235522 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68e0036070 con 0x7f68f0152bc0 2024-04-24T23:15:56.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:56.831+0000 7f68ef400640 1 -- 172.21.15.6:0/3257235522 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f68e0031600 con 0x7f68f0152bc0 2024-04-24T23:15:56.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:56.831+0000 7f68ef400640 1 -- 172.21.15.6:0/3257235522 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68e00319d0 con 0x7f68f0152bc0 2024-04-24T23:15:56.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:56.832+0000 7f68f7a00640 1 -- 172.21.15.6:0/3257235522 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f68f0152bc0 msgr2=0x7f68f0159390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:56.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:56.832+0000 7f68f7a00640 1 --2- 172.21.15.6:0/3257235522 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f68f0152bc0 0x7f68f0159390 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f68f00c7550 tx=0x7f68e002a6a0 comp rx=0 tx=0).stop 2024-04-24T23:15:56.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:56.832+0000 7f68f7a00640 1 -- 172.21.15.6:0/3257235522 shutdown_connections 2024-04-24T23:15:56.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:56.832+0000 7f68f7a00640 1 --2- 172.21.15.6:0/3257235522 >> v2:172.21.15.190:3300/0 conn(0x7f68f00c8f50 0x7f68f015bf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:56.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:56.832+0000 7f68f7a00640 1 --2- 172.21.15.6:0/3257235522 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f68f0152bc0 0x7f68f0159390 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:56.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:56.832+0000 7f68f7a00640 1 -- 172.21.15.6:0/3257235522 >> 172.21.15.6:0/3257235522 conn(0x7f68f006b0e0 msgr2=0x7f68f006b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:15:56.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:56.832+0000 7f68f7a00640 1 -- 172.21.15.6:0/3257235522 shutdown_connections 2024-04-24T23:15:56.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:56.832+0000 7f68f7a00640 1 -- 172.21.15.6:0/3257235522 wait complete. 2024-04-24T23:15:56.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:56.834+0000 7f68f7a00640 1 Processor -- start 2024-04-24T23:15:56.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:56.834+0000 7f68f7a00640 1 -- start start 2024-04-24T23:15:56.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:56.835+0000 7f68f7a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f68f00c8f50 0x7f68f0072910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:56.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:56.835+0000 7f68f7a00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f68f00c9360 0x7f68f0145220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:56.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:56.835+0000 7f68f7a00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f68f0152bc0 0x7f68f013f9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:56.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:56.835+0000 7f68f7a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f68f00c8470 con 0x7f68f00c8f50 2024-04-24T23:15:56.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:56.835+0000 7f68f7a00640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f68f00c6700 con 0x7f68f00c9360 2024-04-24T23:15:56.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:56.835+0000 7f68f7a00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f68f00c8750 con 0x7f68f0152bc0 2024-04-24T23:15:56.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:56.835+0000 7f68f5000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f68f00c8f50 0x7f68f0072910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:56.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:56.835+0000 7f68f5000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f68f00c8f50 0x7f68f0072910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:54220/0 (socket says 172.21.15.6:54220) 2024-04-24T23:15:56.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:56.835+0000 7f68f5000640 1 -- 172.21.15.6:0/3815391227 learned_addr learned my addr 172.21.15.6:0/3815391227 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:15:56.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:56.835+0000 7f68efe00640 1 --2- 172.21.15.6:0/3815391227 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f68f00c9360 0x7f68f0145220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:56.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:56.835+0000 7f68f5a00640 1 --2- 172.21.15.6:0/3815391227 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f68f0152bc0 0x7f68f013f9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:56.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:56.836+0000 7f68f5000640 1 -- 172.21.15.6:0/3815391227 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f68f0152bc0 msgr2=0x7f68f013f9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:56.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:56.836+0000 7f68f5000640 1 --2- 172.21.15.6:0/3815391227 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f68f0152bc0 0x7f68f013f9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:56.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:56.836+0000 7f68f5000640 1 -- 172.21.15.6:0/3815391227 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f68f00c9360 msgr2=0x7f68f0145220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:56.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:56.836+0000 7f68f5000640 1 --2- 172.21.15.6:0/3815391227 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f68f00c9360 0x7f68f0145220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:56.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:56.836+0000 7f68f5000640 1 -- 172.21.15.6:0/3815391227 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68f0159c50 con 0x7f68f00c8f50 2024-04-24T23:15:56.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:56.836+0000 7f68f5000640 1 --2- 172.21.15.6:0/3815391227 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f68f00c8f50 0x7f68f0072910 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f68e0009f10 tx=0x7f68e0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:15:56.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:56.836+0000 7f68ecc00640 1 -- 172.21.15.6:0/3815391227 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68e003f020 con 0x7f68f00c8f50 2024-04-24T23:15:56.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:56.837+0000 7f68ecc00640 1 -- 172.21.15.6:0/3815391227 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f68e0037040 con 0x7f68f00c8f50 2024-04-24T23:15:56.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:56.837+0000 7f68ecc00640 1 -- 172.21.15.6:0/3815391227 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68e0036070 con 0x7f68f00c8f50 2024-04-24T23:15:56.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:56.837+0000 7f68f7a00640 1 -- 172.21.15.6:0/3815391227 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68f0159f30 con 0x7f68f00c8f50 2024-04-24T23:15:56.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:56.837+0000 7f68f7a00640 1 -- 172.21.15.6:0/3815391227 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68f015a4f0 con 0x7f68f00c8f50 2024-04-24T23:15:56.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:56.838+0000 7f68f7a00640 1 -- 172.21.15.6:0/3815391227 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f68f00767b0 con 0x7f68f00c8f50 2024-04-24T23:15:56.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:56.839+0000 7f68ecc00640 1 -- 172.21.15.6:0/3815391227 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f68e0042020 con 0x7f68f00c8f50 2024-04-24T23:15:56.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:56.840+0000 7f68ecc00640 1 --2- 172.21.15.6:0/3815391227 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f68cc06be30 0x7f68cc06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:56.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:56.840+0000 7f68ecc00640 1 -- 172.21.15.6:0/3815391227 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f68e00ab530 con 0x7f68f00c8f50 2024-04-24T23:15:56.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:56.841+0000 7f68efe00640 1 --2- 172.21.15.6:0/3815391227 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f68cc06be30 0x7f68cc06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:56.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:56.841+0000 7f68efe00640 1 --2- 172.21.15.6:0/3815391227 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f68cc06be30 0x7f68cc06e2f0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f68f0159580 tx=0x7f68d8009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:15:56.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:56.845+0000 7f68ecc00640 1 -- 172.21.15.6:0/3815391227 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f68e0049030 con 0x7f68f00c8f50 2024-04-24T23:15:57.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.051+0000 7f68f7a00640 1 -- 172.21.15.6:0/3815391227 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f68f00c75e0 con 0x7f68f00c8f50 2024-04-24T23:15:57.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.052+0000 7f68ecc00640 1 -- 172.21.15.6:0/3815391227 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) ==== 93+0+10 (secure 0 0 0) 0x7f68e00794b0 con 0x7f68f00c8f50 2024-04-24T23:15:57.054 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-04-24T23:15:57.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.054+0000 7f68f7a00640 1 -- 172.21.15.6:0/3815391227 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f68cc06be30 msgr2=0x7f68cc06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:57.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.054+0000 7f68f7a00640 1 --2- 172.21.15.6:0/3815391227 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f68cc06be30 0x7f68cc06e2f0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f68f0159580 tx=0x7f68d8009190 comp rx=0 tx=0).stop 2024-04-24T23:15:57.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.054+0000 7f68f7a00640 1 -- 172.21.15.6:0/3815391227 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f68f00c8f50 msgr2=0x7f68f0072910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:57.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.054+0000 7f68f7a00640 1 --2- 172.21.15.6:0/3815391227 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f68f00c8f50 0x7f68f0072910 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f68e0009f10 tx=0x7f68e0004a40 comp rx=0 tx=0).stop 2024-04-24T23:15:57.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.054+0000 7f68f7a00640 1 -- 172.21.15.6:0/3815391227 shutdown_connections 2024-04-24T23:15:57.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.054+0000 7f68f7a00640 1 --2- 172.21.15.6:0/3815391227 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f68cc06be30 0x7f68cc06e2f0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:57.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.054+0000 7f68f7a00640 1 --2- 172.21.15.6:0/3815391227 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f68f0152bc0 0x7f68f013f9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:57.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.054+0000 7f68f7a00640 1 --2- 172.21.15.6:0/3815391227 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f68f00c9360 0x7f68f0145220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:57.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.054+0000 7f68f7a00640 1 --2- 172.21.15.6:0/3815391227 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f68f00c8f50 0x7f68f0072910 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:57.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.054+0000 7f68f7a00640 1 -- 172.21.15.6:0/3815391227 >> 172.21.15.6:0/3815391227 conn(0x7f68f006b0e0 msgr2=0x7f68f01e6fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:15:57.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.055+0000 7f68f7a00640 1 -- 172.21.15.6:0/3815391227 shutdown_connections 2024-04-24T23:15:57.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.055+0000 7f68f7a00640 1 -- 172.21.15.6:0/3815391227 wait complete. 2024-04-24T23:15:57.075 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-24T23:15:57.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.287+0000 7f7eaa200640 1 Processor -- start 2024-04-24T23:15:57.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.288+0000 7f7eaa200640 1 -- start start 2024-04-24T23:15:57.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.289+0000 7f7eaa200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f7ea4152bc0 0x7f7ea4159390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:57.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.289+0000 7f7eaa200640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f7ea40c9360 0x7f7ea415c610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:57.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.289+0000 7f7eaa200640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7ea40c6e70 con 0x7f7ea40c8f50 2024-04-24T23:15:57.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.290+0000 7f7eaa200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f7ea40c73d0 con 0x7f7ea4152bc0 2024-04-24T23:15:57.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.290+0000 7f7eaa200640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f7ea40c8ac0 con 0x7f7ea40c9360 2024-04-24T23:15:57.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.290+0000 7f7ea3400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f7ea4152bc0 0x7f7ea4159390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:57.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.290+0000 7f7ea3400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f7ea4152bc0 0x7f7ea4159390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:54230/0 (socket says 172.21.15.6:54230) 2024-04-24T23:15:57.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.290+0000 7f7ea3e00640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f7ea40c9360 0x7f7ea415c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:57.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.290+0000 7f7ea2a00640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f7ea40c8f50 0x7f7ea415bf00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.6:56314/0 (socket says 172.21.15.6:56314) 2024-04-24T23:15:57.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.290+0000 7f7ea3400640 1 -- 172.21.15.6:0/1084274593 learned_addr learned my addr 172.21.15.6:0/1084274593 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:15:57.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.291+0000 7f7ea3e00640 1 -- 172.21.15.6:0/1084274593 >> v2:172.21.15.6:3300/0 conn(0x7f7ea4152bc0 msgr2=0x7f7ea4159390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:15:57.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.291+0000 7f7ea3e00640 1 --2- 172.21.15.6:0/1084274593 >> v2:172.21.15.6:3300/0 conn(0x7f7ea4152bc0 0x7f7ea4159390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:57.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.291+0000 7f7ea3e00640 1 -- 172.21.15.6:0/1084274593 >> v1:172.21.15.190:6789/0 conn(0x7f7ea40c8f50 legacy=0x7f7ea415bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:57.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.291+0000 7f7ea3e00640 1 -- 172.21.15.6:0/1084274593 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ea41598d0 con 0x7f7ea40c9360 2024-04-24T23:15:57.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.291+0000 7f7ea2000640 1 -- 172.21.15.6:0/1084274593 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2433334688 0 0) 0x7f7ea40c6e70 con 0x7f7ea40c8f50 2024-04-24T23:15:57.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.292+0000 7f7ea3e00640 1 --2- 172.21.15.6:0/1084274593 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7ea40c9360 0x7f7ea415c610 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f7ea40c7550 tx=0x7f7e9c02a5d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6951cbfe4d09a5b2 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:15:57.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.292+0000 7f7ea2000640 1 -- 172.21.15.6:0/1084274593 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e9c031600 con 0x7f7ea40c9360 2024-04-24T23:15:57.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.292+0000 7f7ea2000640 1 -- 172.21.15.6:0/1084274593 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e9c0317a0 con 0x7f7ea40c9360 2024-04-24T23:15:57.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.292+0000 7f7ea2000640 1 -- 172.21.15.6:0/1084274593 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e9c031b70 con 0x7f7ea40c9360 2024-04-24T23:15:57.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.293+0000 7f7eaa200640 1 -- 172.21.15.6:0/1084274593 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7ea40c9360 msgr2=0x7f7ea415c610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:57.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.293+0000 7f7eaa200640 1 --2- 172.21.15.6:0/1084274593 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7ea40c9360 0x7f7ea415c610 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f7ea40c7550 tx=0x7f7e9c02a5d0 comp rx=0 tx=0).stop 2024-04-24T23:15:57.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.293+0000 7f7eaa200640 1 -- 172.21.15.6:0/1084274593 shutdown_connections 2024-04-24T23:15:57.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.293+0000 7f7eaa200640 1 --2- 172.21.15.6:0/1084274593 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7ea40c9360 0x7f7ea415c610 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:57.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.293+0000 7f7eaa200640 1 --2- 172.21.15.6:0/1084274593 >> v2:172.21.15.6:3300/0 conn(0x7f7ea4152bc0 0x7f7ea4159390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:57.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.293+0000 7f7eaa200640 1 -- 172.21.15.6:0/1084274593 >> 172.21.15.6:0/1084274593 conn(0x7f7ea406b0e0 msgr2=0x7f7ea406b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:15:57.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.293+0000 7f7eaa200640 1 -- 172.21.15.6:0/1084274593 shutdown_connections 2024-04-24T23:15:57.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.293+0000 7f7eaa200640 1 -- 172.21.15.6:0/1084274593 wait complete. 2024-04-24T23:15:57.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.295+0000 7f7eaa200640 1 Processor -- start 2024-04-24T23:15:57.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.295+0000 7f7eaa200640 1 -- start start 2024-04-24T23:15:57.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.296+0000 7f7eaa200640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7ea40c8f50 0x7f7ea4157360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:57.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.296+0000 7f7eaa200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7ea40c9360 0x7f7ea41578a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:57.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.296+0000 7f7eaa200640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7ea4152bc0 0x7f7ea4140f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:57.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.296+0000 7f7eaa200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f7ea40c8470 con 0x7f7ea40c9360 2024-04-24T23:15:57.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.296+0000 7f7eaa200640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f7ea40c6700 con 0x7f7ea40c8f50 2024-04-24T23:15:57.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.296+0000 7f7eaa200640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f7ea40c8750 con 0x7f7ea4152bc0 2024-04-24T23:15:57.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.296+0000 7f7ea2a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7ea40c9360 0x7f7ea41578a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:57.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.296+0000 7f7ea2a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7ea40c9360 0x7f7ea41578a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:54244/0 (socket says 172.21.15.6:54244) 2024-04-24T23:15:57.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.296+0000 7f7ea2a00640 1 -- 172.21.15.6:0/3608723604 learned_addr learned my addr 172.21.15.6:0/3608723604 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:15:57.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.296+0000 7f7ea3400640 1 --2- 172.21.15.6:0/3608723604 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7ea40c8f50 0x7f7ea4157360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:57.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.296+0000 7f7ea3e00640 1 --2- 172.21.15.6:0/3608723604 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7ea4152bc0 0x7f7ea4140f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:57.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.296+0000 7f7ea2a00640 1 -- 172.21.15.6:0/3608723604 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7ea4152bc0 msgr2=0x7f7ea4140f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:57.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.297+0000 7f7ea2a00640 1 --2- 172.21.15.6:0/3608723604 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7ea4152bc0 0x7f7ea4140f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:57.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.297+0000 7f7ea2a00640 1 -- 172.21.15.6:0/3608723604 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7ea40c8f50 msgr2=0x7f7ea4157360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:57.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.297+0000 7f7ea2a00640 1 --2- 172.21.15.6:0/3608723604 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7ea40c8f50 0x7f7ea4157360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:57.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.297+0000 7f7ea2a00640 1 -- 172.21.15.6:0/3608723604 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ea4159270 con 0x7f7ea40c9360 2024-04-24T23:15:57.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.297+0000 7f7ea2a00640 1 --2- 172.21.15.6:0/3608723604 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7ea40c9360 0x7f7ea41578a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f7e9000c460 tx=0x7f7e9000c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:15:57.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.297+0000 7f7e9b400640 1 -- 172.21.15.6:0/3608723604 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e900074e0 con 0x7f7ea40c9360 2024-04-24T23:15:57.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.297+0000 7f7e9b400640 1 -- 172.21.15.6:0/3608723604 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e900047e0 con 0x7f7ea40c9360 2024-04-24T23:15:57.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.297+0000 7f7e9b400640 1 -- 172.21.15.6:0/3608723604 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e90004bd0 con 0x7f7ea40c9360 2024-04-24T23:15:57.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.297+0000 7f7eaa200640 1 -- 172.21.15.6:0/3608723604 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ea4141bd0 con 0x7f7ea40c9360 2024-04-24T23:15:57.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.297+0000 7f7eaa200640 1 -- 172.21.15.6:0/3608723604 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ea4072a50 con 0x7f7ea40c9360 2024-04-24T23:15:57.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.299+0000 7f7eaa200640 1 -- 172.21.15.6:0/3608723604 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ea40767b0 con 0x7f7ea40c9360 2024-04-24T23:15:57.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.300+0000 7f7e9b400640 1 -- 172.21.15.6:0/3608723604 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f7e90011070 con 0x7f7ea40c9360 2024-04-24T23:15:57.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.301+0000 7f7e9b400640 1 --2- 172.21.15.6:0/3608723604 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f7e8406be60 0x7f7e8406e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:57.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.301+0000 7f7e9b400640 1 -- 172.21.15.6:0/3608723604 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f7e9008abf0 con 0x7f7ea40c9360 2024-04-24T23:15:57.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.301+0000 7f7ea3400640 1 --2- 172.21.15.6:0/3608723604 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f7e8406be60 0x7f7e8406e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:57.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.302+0000 7f7ea3400640 1 --2- 172.21.15.6:0/3608723604 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f7e8406be60 0x7f7e8406e320 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f7e94009bd0 tx=0x7f7e94009240 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:15:57.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.305+0000 7f7e9b400640 1 -- 172.21.15.6:0/3608723604 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f7ea4159270 con 0x7f7ea40c9360 2024-04-24T23:15:57.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.509+0000 7f7eaa200640 1 -- 172.21.15.6:0/3608723604 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f7ea40c75e0 con 0x7f7ea40c9360 2024-04-24T23:15:57.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.510+0000 7f7e9b400640 1 -- 172.21.15.6:0/3608723604 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) ==== 104+0+11 (secure 0 0 0) 0x7f7e90058c00 con 0x7f7ea40c9360 2024-04-24T23:15:57.512 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 32 2024-04-24T23:15:57.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.513+0000 7f7eaa200640 1 -- 172.21.15.6:0/3608723604 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f7e8406be60 msgr2=0x7f7e8406e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:57.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.513+0000 7f7eaa200640 1 --2- 172.21.15.6:0/3608723604 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f7e8406be60 0x7f7e8406e320 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f7e94009bd0 tx=0x7f7e94009240 comp rx=0 tx=0).stop 2024-04-24T23:15:57.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.513+0000 7f7eaa200640 1 -- 172.21.15.6:0/3608723604 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7ea40c9360 msgr2=0x7f7ea41578a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:57.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.513+0000 7f7eaa200640 1 --2- 172.21.15.6:0/3608723604 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7ea40c9360 0x7f7ea41578a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f7e9000c460 tx=0x7f7e9000c920 comp rx=0 tx=0).stop 2024-04-24T23:15:57.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.513+0000 7f7eaa200640 1 -- 172.21.15.6:0/3608723604 shutdown_connections 2024-04-24T23:15:57.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.513+0000 7f7eaa200640 1 --2- 172.21.15.6:0/3608723604 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f7e8406be60 0x7f7e8406e320 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:57.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.513+0000 7f7eaa200640 1 --2- 172.21.15.6:0/3608723604 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7ea4152bc0 0x7f7ea4140f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:57.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.513+0000 7f7eaa200640 1 --2- 172.21.15.6:0/3608723604 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7ea40c9360 0x7f7ea41578a0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:57.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.513+0000 7f7eaa200640 1 --2- 172.21.15.6:0/3608723604 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7ea40c8f50 0x7f7ea4157360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:57.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.513+0000 7f7eaa200640 1 -- 172.21.15.6:0/3608723604 >> 172.21.15.6:0/3608723604 conn(0x7f7ea406b0e0 msgr2=0x7f7ea406b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:15:57.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.513+0000 7f7eaa200640 1 -- 172.21.15.6:0/3608723604 shutdown_connections 2024-04-24T23:15:57.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.513+0000 7f7eaa200640 1 -- 172.21.15.6:0/3608723604 wait complete. 2024-04-24T23:15:57.535 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-24T23:15:57.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.739+0000 7f30e5c00640 1 Processor -- start 2024-04-24T23:15:57.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.739+0000 7f30e5c00640 1 -- start start 2024-04-24T23:15:57.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.741+0000 7f30e5c00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f30e006eb80 0x7f30e00c9330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:57.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.741+0000 7f30e5c00640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f30e015ed90 0x7f30e0162a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:57.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.741+0000 7f30e5c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f30e006ef60 0x7f30e0162f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:57.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.741+0000 7f30e5c00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f30e00c8bd0 con 0x7f30e006ef60 2024-04-24T23:15:57.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.741+0000 7f30e5c00640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f30e015f240 con 0x7f30e015ed90 2024-04-24T23:15:57.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.741+0000 7f30e5c00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f30e015f520 con 0x7f30e006eb80 2024-04-24T23:15:57.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.741+0000 7f30df400640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f30e015ed90 0x7f30e0162a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:57.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.741+0000 7f30dfe00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f30e006eb80 0x7f30e00c9330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:57.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.742+0000 7f30df400640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f30e015ed90 0x7f30e0162a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:38482/0 (socket says 172.21.15.6:38482) 2024-04-24T23:15:57.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.742+0000 7f30df400640 1 -- 172.21.15.6:0/2739832996 learned_addr learned my addr 172.21.15.6:0/2739832996 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:15:57.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.742+0000 7f30e5200640 1 --2- 172.21.15.6:0/2739832996 >> v2:172.21.15.6:3300/0 conn(0x7f30e006ef60 0x7f30e0162f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:57.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.742+0000 7f30df400640 1 -- 172.21.15.6:0/2739832996 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f30e006eb80 msgr2=0x7f30e00c9330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:15:57.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.742+0000 7f30df400640 1 --2- 172.21.15.6:0/2739832996 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f30e006eb80 0x7f30e00c9330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:57.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.742+0000 7f30df400640 1 -- 172.21.15.6:0/2739832996 >> v2:172.21.15.6:3300/0 conn(0x7f30e006ef60 msgr2=0x7f30e0162f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:15:57.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.742+0000 7f30df400640 1 --2- 172.21.15.6:0/2739832996 >> v2:172.21.15.6:3300/0 conn(0x7f30e006ef60 0x7f30e0162f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:57.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.742+0000 7f30df400640 1 -- 172.21.15.6:0/2739832996 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30e015f800 con 0x7f30e015ed90 2024-04-24T23:15:57.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.743+0000 7f30df400640 1 --2- 172.21.15.6:0/2739832996 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f30e015ed90 0x7f30e0162a40 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f30e00cd170 tx=0x7f30cc02a260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2605343aa73bb2b5 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:15:57.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.744+0000 7f30dea00640 1 -- 172.21.15.6:0/2739832996 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30cc037070 con 0x7f30e015ed90 2024-04-24T23:15:57.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.744+0000 7f30dea00640 1 -- 172.21.15.6:0/2739832996 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f30cc032440 con 0x7f30e015ed90 2024-04-24T23:15:57.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.744+0000 7f30dea00640 1 -- 172.21.15.6:0/2739832996 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30cc032810 con 0x7f30e015ed90 2024-04-24T23:15:57.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.744+0000 7f30e5c00640 1 -- 172.21.15.6:0/2739832996 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f30e015ed90 msgr2=0x7f30e0162a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:57.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.744+0000 7f30e5c00640 1 --2- 172.21.15.6:0/2739832996 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f30e015ed90 0x7f30e0162a40 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f30e00cd170 tx=0x7f30cc02a260 comp rx=0 tx=0).stop 2024-04-24T23:15:57.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.744+0000 7f30e5c00640 1 -- 172.21.15.6:0/2739832996 shutdown_connections 2024-04-24T23:15:57.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.744+0000 7f30e5c00640 1 --2- 172.21.15.6:0/2739832996 >> v2:172.21.15.6:3300/0 conn(0x7f30e006ef60 0x7f30e0162f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:57.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.744+0000 7f30e5c00640 1 --2- 172.21.15.6:0/2739832996 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f30e015ed90 0x7f30e0162a40 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:57.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.744+0000 7f30e5c00640 1 --2- 172.21.15.6:0/2739832996 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f30e006eb80 0x7f30e00c9330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:57.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.744+0000 7f30e5c00640 1 -- 172.21.15.6:0/2739832996 >> 172.21.15.6:0/2739832996 conn(0x7f30e006cc40 msgr2=0x7f30e006d050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:15:57.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.744+0000 7f30e5c00640 1 -- 172.21.15.6:0/2739832996 shutdown_connections 2024-04-24T23:15:57.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.744+0000 7f30e5c00640 1 -- 172.21.15.6:0/2739832996 wait complete. 2024-04-24T23:15:57.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.746+0000 7f30e5c00640 1 Processor -- start 2024-04-24T23:15:57.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.746+0000 7f30e5c00640 1 -- start start 2024-04-24T23:15:57.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.747+0000 7f30e5c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f30e006eb80 0x7f30e00ca440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:57.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.747+0000 7f30e5c00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f30e006ef60 0x7f30e00cc7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:57.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.747+0000 7f30e5c00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f30e015ed90 0x7f30e01f51d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:57.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.747+0000 7f30e5c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f30e0070e00 con 0x7f30e006eb80 2024-04-24T23:15:57.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.747+0000 7f30e5c00640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f30e0162520 con 0x7f30e015ed90 2024-04-24T23:15:57.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.747+0000 7f30e5c00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f30e0070f80 con 0x7f30e006ef60 2024-04-24T23:15:57.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.747+0000 7f30dfe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f30e006eb80 0x7f30e00ca440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:57.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.747+0000 7f30dfe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f30e006eb80 0x7f30e00ca440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:54272/0 (socket says 172.21.15.6:54272) 2024-04-24T23:15:57.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.748+0000 7f30dfe00640 1 -- 172.21.15.6:0/1117241732 learned_addr learned my addr 172.21.15.6:0/1117241732 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:15:57.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.748+0000 7f30df400640 1 --2- 172.21.15.6:0/1117241732 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f30e006ef60 0x7f30e00cc7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:57.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.748+0000 7f30e5200640 1 --2- 172.21.15.6:0/1117241732 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f30e015ed90 0x7f30e01f51d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:57.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.748+0000 7f30dfe00640 1 -- 172.21.15.6:0/1117241732 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f30e006ef60 msgr2=0x7f30e00cc7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:57.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.748+0000 7f30dfe00640 1 --2- 172.21.15.6:0/1117241732 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f30e006ef60 0x7f30e00cc7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:57.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.748+0000 7f30dfe00640 1 -- 172.21.15.6:0/1117241732 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f30e015ed90 msgr2=0x7f30e01f51d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:57.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.748+0000 7f30dfe00640 1 --2- 172.21.15.6:0/1117241732 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f30e015ed90 0x7f30e01f51d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:57.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.748+0000 7f30dfe00640 1 -- 172.21.15.6:0/1117241732 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30e00c9ee0 con 0x7f30e006eb80 2024-04-24T23:15:57.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.748+0000 7f30df400640 1 --2- 172.21.15.6:0/1117241732 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f30e006ef60 0x7f30e00cc7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:15:57.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.748+0000 7f30dfe00640 1 --2- 172.21.15.6:0/1117241732 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f30e006eb80 0x7f30e00ca440 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f30d000e870 tx=0x7f30d000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:15:57.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.749+0000 7f30d7e00640 1 -- 172.21.15.6:0/1117241732 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30d0017070 con 0x7f30e006eb80 2024-04-24T23:15:57.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.749+0000 7f30d7e00640 1 -- 172.21.15.6:0/1117241732 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f30d000ee60 con 0x7f30e006eb80 2024-04-24T23:15:57.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.749+0000 7f30d7e00640 1 -- 172.21.15.6:0/1117241732 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30d000cb80 con 0x7f30e006eb80 2024-04-24T23:15:57.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.749+0000 7f30e5c00640 1 -- 172.21.15.6:0/1117241732 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30e00ca160 con 0x7f30e006eb80 2024-04-24T23:15:57.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.749+0000 7f30e5c00640 1 -- 172.21.15.6:0/1117241732 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30e01e7d50 con 0x7f30e006eb80 2024-04-24T23:15:57.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.750+0000 7f30e5c00640 1 -- 172.21.15.6:0/1117241732 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f30e006e730 con 0x7f30e006eb80 2024-04-24T23:15:57.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.751+0000 7f30d7e00640 1 -- 172.21.15.6:0/1117241732 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f30d001a070 con 0x7f30e006eb80 2024-04-24T23:15:57.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.752+0000 7f30d7e00640 1 --2- 172.21.15.6:0/1117241732 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f30c006be30 0x7f30c006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:57.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.753+0000 7f30d7e00640 1 -- 172.21.15.6:0/1117241732 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f30d008aff0 con 0x7f30e006eb80 2024-04-24T23:15:57.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.753+0000 7f30df400640 1 --2- 172.21.15.6:0/1117241732 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f30c006be30 0x7f30c006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:57.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.754+0000 7f30df400640 1 --2- 172.21.15.6:0/1117241732 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f30c006be30 0x7f30c006e2f0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f30cc034000 tx=0x7f30cc02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:15:57.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.757+0000 7f30d7e00640 1 -- 172.21.15.6:0/1117241732 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f30e006e730 con 0x7f30e006eb80 2024-04-24T23:15:57.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.959+0000 7f30e5c00640 1 -- 172.21.15.6:0/1117241732 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f30e0075b30 con 0x7f30e006eb80 2024-04-24T23:15:57.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.960+0000 7f30d7e00640 1 -- 172.21.15.6:0/1117241732 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) ==== 100+0+11 (secure 0 0 0) 0x7f30e00c9ee0 con 0x7f30e006eb80 2024-04-24T23:15:57.962 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 64 2024-04-24T23:15:57.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.963+0000 7f30e5c00640 1 -- 172.21.15.6:0/1117241732 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f30c006be30 msgr2=0x7f30c006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:57.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.963+0000 7f30e5c00640 1 --2- 172.21.15.6:0/1117241732 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f30c006be30 0x7f30c006e2f0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f30cc034000 tx=0x7f30cc02e000 comp rx=0 tx=0).stop 2024-04-24T23:15:57.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.963+0000 7f30e5c00640 1 -- 172.21.15.6:0/1117241732 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f30e006eb80 msgr2=0x7f30e00ca440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:57.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.963+0000 7f30e5c00640 1 --2- 172.21.15.6:0/1117241732 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f30e006eb80 0x7f30e00ca440 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f30d000e870 tx=0x7f30d000ed30 comp rx=0 tx=0).stop 2024-04-24T23:15:57.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.963+0000 7f30e5c00640 1 -- 172.21.15.6:0/1117241732 shutdown_connections 2024-04-24T23:15:57.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.963+0000 7f30e5c00640 1 --2- 172.21.15.6:0/1117241732 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f30c006be30 0x7f30c006e2f0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:57.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.963+0000 7f30e5c00640 1 --2- 172.21.15.6:0/1117241732 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f30e006ef60 0x7f30e00cc7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:57.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.963+0000 7f30e5c00640 1 --2- 172.21.15.6:0/1117241732 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f30e015ed90 0x7f30e01f51d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:57.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.963+0000 7f30e5c00640 1 --2- 172.21.15.6:0/1117241732 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f30e006eb80 0x7f30e00ca440 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:57.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.963+0000 7f30e5c00640 1 -- 172.21.15.6:0/1117241732 >> 172.21.15.6:0/1117241732 conn(0x7f30e006cc40 msgr2=0x7f30e006d020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:15:57.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.963+0000 7f30e5c00640 1 -- 172.21.15.6:0/1117241732 shutdown_connections 2024-04-24T23:15:57.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:57.963+0000 7f30e5c00640 1 -- 172.21.15.6:0/1117241732 wait complete. 2024-04-24T23:15:57.984 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-24T23:15:58.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:58.200+0000 7f9061c00640 1 Processor -- start 2024-04-24T23:15:58.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:58.201+0000 7f9061c00640 1 -- start start 2024-04-24T23:15:58.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:58.202+0000 7f9061c00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f905c0c5a60 0x7f905c0c6ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:58.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:58.202+0000 7f9061c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f905c0c5e40 0x7f905c15bd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:58.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:58.202+0000 7f9061c00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f905c0cca80 con 0x7f905c0c8570 2024-04-24T23:15:58.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:58.202+0000 7f9061c00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f905c0ccd60 con 0x7f905c0c5e40 2024-04-24T23:15:58.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:58.202+0000 7f9061c00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f905c06eab0 con 0x7f905c0c5a60 2024-04-24T23:15:58.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:58.203+0000 7f9053e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f905c0c5e40 0x7f905c15bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:58.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:58.203+0000 7f905be00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f905c0c5a60 0x7f905c0c6ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:58.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:58.203+0000 7f9061200640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f905c0c8570 0x7f905c1524a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:42940/0 (socket says 172.21.15.6:42940) 2024-04-24T23:15:58.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:58.203+0000 7f9061200640 1 -- 172.21.15.6:0/3860187213 learned_addr learned my addr 172.21.15.6:0/3860187213 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:15:58.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:58.203+0000 7f905be00640 1 -- 172.21.15.6:0/3860187213 >> v2:172.21.15.6:3300/0 conn(0x7f905c0c5e40 msgr2=0x7f905c15bd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:15:58.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:58.203+0000 7f905be00640 1 --2- 172.21.15.6:0/3860187213 >> v2:172.21.15.6:3300/0 conn(0x7f905c0c5e40 0x7f905c15bd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:58.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:58.203+0000 7f905be00640 1 -- 172.21.15.6:0/3860187213 >> v1:172.21.15.6:6789/0 conn(0x7f905c0c8570 legacy=0x7f905c1524a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:58.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:58.203+0000 7f905be00640 1 -- 172.21.15.6:0/3860187213 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f905c06ed90 con 0x7f905c0c5a60 2024-04-24T23:15:58.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:58.203+0000 7f9053e00640 1 --2- 172.21.15.6:0/3860187213 >> v2:172.21.15.6:3300/0 conn(0x7f905c0c5e40 0x7f905c15bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:15:58.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:58.204+0000 7f905be00640 1 --2- 172.21.15.6:0/3860187213 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f905c0c5a60 0x7f905c0c6ae0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f904c0097a0 tx=0x7f904c02a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2192985a3856bdc0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:15:58.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:58.205+0000 7f905b400640 1 -- 172.21.15.6:0/3860187213 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f904c037070 con 0x7f905c0c5a60 2024-04-24T23:15:58.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:58.205+0000 7f905b400640 1 -- 172.21.15.6:0/3860187213 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f904c032440 con 0x7f905c0c5a60 2024-04-24T23:15:58.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:58.205+0000 7f905b400640 1 -- 172.21.15.6:0/3860187213 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f904c032810 con 0x7f905c0c5a60 2024-04-24T23:15:58.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:58.205+0000 7f9061c00640 1 -- 172.21.15.6:0/3860187213 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f905c0c5a60 msgr2=0x7f905c0c6ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:58.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:58.205+0000 7f9061c00640 1 --2- 172.21.15.6:0/3860187213 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f905c0c5a60 0x7f905c0c6ae0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f904c0097a0 tx=0x7f904c02a180 comp rx=0 tx=0).stop 2024-04-24T23:15:58.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:58.206+0000 7f9061c00640 1 -- 172.21.15.6:0/3860187213 shutdown_connections 2024-04-24T23:15:58.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:58.206+0000 7f9061c00640 1 --2- 172.21.15.6:0/3860187213 >> v2:172.21.15.6:3300/0 conn(0x7f905c0c5e40 0x7f905c15bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:58.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:58.206+0000 7f9061c00640 1 --2- 172.21.15.6:0/3860187213 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f905c0c5a60 0x7f905c0c6ae0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:58.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:58.206+0000 7f9061c00640 1 -- 172.21.15.6:0/3860187213 >> 172.21.15.6:0/3860187213 conn(0x7f905c06cc40 msgr2=0x7f905c06d050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:15:58.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:58.206+0000 7f9061c00640 1 -- 172.21.15.6:0/3860187213 shutdown_connections 2024-04-24T23:15:58.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:58.206+0000 7f9061c00640 1 -- 172.21.15.6:0/3860187213 wait complete. 2024-04-24T23:15:58.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:58.207+0000 7f9061c00640 1 Processor -- start 2024-04-24T23:15:58.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:58.208+0000 7f9061c00640 1 -- start start 2024-04-24T23:15:58.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:58.209+0000 7f9061c00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f905c0c5a60 0x7f905c1f44f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:58.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:58.209+0000 7f9061c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f905c0c5e40 0x7f905c0716c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:58.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:58.209+0000 7f9061c00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f905c0c8570 0x7f905c154130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:58.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:58.209+0000 7f9061c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f905c06f590 con 0x7f905c0c5e40 2024-04-24T23:15:58.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:58.209+0000 7f9061c00640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f905c072730 con 0x7f905c0c8570 2024-04-24T23:15:58.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:58.209+0000 7f9061c00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f905c156560 con 0x7f905c0c5a60 2024-04-24T23:15:58.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:58.209+0000 7f9053e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f905c0c5e40 0x7f905c0716c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:58.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:58.209+0000 7f9053e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f905c0c5e40 0x7f905c0716c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:54292/0 (socket says 172.21.15.6:54292) 2024-04-24T23:15:58.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:58.209+0000 7f9053e00640 1 -- 172.21.15.6:0/3843117562 learned_addr learned my addr 172.21.15.6:0/3843117562 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:15:58.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:58.209+0000 7f9061200640 1 --2- 172.21.15.6:0/3843117562 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f905c0c8570 0x7f905c154130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:58.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:58.209+0000 7f905be00640 1 --2- 172.21.15.6:0/3843117562 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f905c0c5a60 0x7f905c1f44f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:58.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:58.210+0000 7f9053e00640 1 -- 172.21.15.6:0/3843117562 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f905c0c5a60 msgr2=0x7f905c1f44f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:58.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:58.210+0000 7f9053e00640 1 --2- 172.21.15.6:0/3843117562 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f905c0c5a60 0x7f905c1f44f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:58.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:58.210+0000 7f9053e00640 1 -- 172.21.15.6:0/3843117562 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f905c0c8570 msgr2=0x7f905c154130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:58.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:58.210+0000 7f9053e00640 1 --2- 172.21.15.6:0/3843117562 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f905c0c8570 0x7f905c154130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:58.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:58.210+0000 7f9053e00640 1 -- 172.21.15.6:0/3843117562 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f905c14e9b0 con 0x7f905c0c5e40 2024-04-24T23:15:58.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:58.210+0000 7f9053e00640 1 --2- 172.21.15.6:0/3843117562 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f905c0c5e40 0x7f905c0716c0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f904400ebe0 tx=0x7f904400c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:15:58.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:58.210+0000 7f9058c00640 1 -- 172.21.15.6:0/3843117562 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9044017070 con 0x7f905c0c5e40 2024-04-24T23:15:58.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:58.210+0000 7f9058c00640 1 -- 172.21.15.6:0/3843117562 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f904400cc30 con 0x7f905c0c5e40 2024-04-24T23:15:58.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:58.210+0000 7f9058c00640 1 -- 172.21.15.6:0/3843117562 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9044013470 con 0x7f905c0c5e40 2024-04-24T23:15:58.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:58.210+0000 7f9061c00640 1 -- 172.21.15.6:0/3843117562 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f905c14ec90 con 0x7f905c0c5e40 2024-04-24T23:15:58.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:58.210+0000 7f9061c00640 1 -- 172.21.15.6:0/3843117562 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f905c153650 con 0x7f905c0c5e40 2024-04-24T23:15:58.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:58.211+0000 7f9061c00640 1 -- 172.21.15.6:0/3843117562 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f905c075b30 con 0x7f905c0c5e40 2024-04-24T23:15:58.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:58.213+0000 7f9058c00640 1 -- 172.21.15.6:0/3843117562 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f904401a070 con 0x7f905c0c5e40 2024-04-24T23:15:58.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:58.213+0000 7f9058c00640 1 --2- 172.21.15.6:0/3843117562 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f903806be80 0x7f903806e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:58.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:58.214+0000 7f9058c00640 1 -- 172.21.15.6:0/3843117562 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f904408afe0 con 0x7f905c0c5e40 2024-04-24T23:15:58.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:58.214+0000 7f905be00640 1 --2- 172.21.15.6:0/3843117562 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f903806be80 0x7f903806e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:58.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:58.215+0000 7f905be00640 1 --2- 172.21.15.6:0/3843117562 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f903806be80 0x7f903806e340 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f904c034000 tx=0x7f904c02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:15:58.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:58.218+0000 7f9058c00640 1 -- 172.21.15.6:0/3843117562 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f9044059090 con 0x7f905c0c5e40 2024-04-24T23:15:58.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:58.425+0000 7f9061c00640 1 -- 172.21.15.6:0/3843117562 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f905c151ec0 con 0x7f905c0c5e40 2024-04-24T23:15:58.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:58.426+0000 7f9058c00640 1 -- 172.21.15.6:0/3843117562 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v24) ==== 103+0+11 (secure 0 0 0) 0x7f904405d390 con 0x7f905c0c5e40 2024-04-24T23:15:58.428 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 32 2024-04-24T23:15:58.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:58.428+0000 7f9061c00640 1 -- 172.21.15.6:0/3843117562 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f903806be80 msgr2=0x7f903806e340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:58.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:58.428+0000 7f9061c00640 1 --2- 172.21.15.6:0/3843117562 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f903806be80 0x7f903806e340 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f904c034000 tx=0x7f904c02e000 comp rx=0 tx=0).stop 2024-04-24T23:15:58.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:58.428+0000 7f9061c00640 1 -- 172.21.15.6:0/3843117562 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f905c0c5e40 msgr2=0x7f905c0716c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:58.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:58.428+0000 7f9061c00640 1 --2- 172.21.15.6:0/3843117562 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f905c0c5e40 0x7f905c0716c0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f904400ebe0 tx=0x7f904400c520 comp rx=0 tx=0).stop 2024-04-24T23:15:58.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:58.429+0000 7f9061c00640 1 -- 172.21.15.6:0/3843117562 shutdown_connections 2024-04-24T23:15:58.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:58.429+0000 7f9061c00640 1 --2- 172.21.15.6:0/3843117562 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f903806be80 0x7f903806e340 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:58.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:58.429+0000 7f9061c00640 1 --2- 172.21.15.6:0/3843117562 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f905c0c8570 0x7f905c154130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:58.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:58.429+0000 7f9061c00640 1 --2- 172.21.15.6:0/3843117562 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f905c0c5e40 0x7f905c0716c0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:58.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:58.429+0000 7f9061c00640 1 --2- 172.21.15.6:0/3843117562 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f905c0c5a60 0x7f905c1f44f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:58.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:58.429+0000 7f9061c00640 1 -- 172.21.15.6:0/3843117562 >> 172.21.15.6:0/3843117562 conn(0x7f905c06cc40 msgr2=0x7f905c0c74f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:15:58.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:58.429+0000 7f9061c00640 1 -- 172.21.15.6:0/3843117562 shutdown_connections 2024-04-24T23:15:58.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:58.429+0000 7f9061c00640 1 -- 172.21.15.6:0/3843117562 wait complete. 2024-04-24T23:15:58.451 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-24T23:15:58.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:58.662+0000 7f8936200640 1 Processor -- start 2024-04-24T23:15:58.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:58.662+0000 7f8936200640 1 -- start start 2024-04-24T23:15:58.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:58.664+0000 7f8936200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f8930151a10 0x7f89300c46f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:58.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:58.664+0000 7f8936200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f893015bbc0 con 0x7f893006f9e0 2024-04-24T23:15:58.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:58.664+0000 7f8936200640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8930070760 con 0x7f893006ff30 2024-04-24T23:15:58.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:58.664+0000 7f8936200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f893015b730 con 0x7f8930151a10 2024-04-24T23:15:58.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:58.664+0000 7f8934e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f8930151a10 0x7f89300c46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:58.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:58.664+0000 7f8934e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f8930151a10 0x7f89300c46f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:54302/0 (socket says 172.21.15.6:54302) 2024-04-24T23:15:58.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:58.664+0000 7f8934e00640 1 -- 172.21.15.6:0/875036561 learned_addr learned my addr 172.21.15.6:0/875036561 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:15:58.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:58.664+0000 7f892fe00640 1 --1- 172.21.15.6:0/875036561 >> v1:172.21.15.190:6789/0 conn(0x7f893006ff30 0x7f8930157760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.6:56318/0 (socket says 172.21.15.6:56318) 2024-04-24T23:15:58.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:58.664+0000 7f8935800640 1 --1- 172.21.15.6:0/875036561 >> v1:172.21.15.6:6789/0 conn(0x7f893006f9e0 0x7f89300c6180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:42954/0 (socket says 172.21.15.6:42954) 2024-04-24T23:15:58.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:58.665+0000 7f8934e00640 1 -- 172.21.15.6:0/875036561 >> v1:172.21.15.190:6789/0 conn(0x7f893006ff30 legacy=0x7f8930157760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:58.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:58.665+0000 7f8934e00640 1 -- 172.21.15.6:0/875036561 >> v1:172.21.15.6:6789/0 conn(0x7f893006f9e0 legacy=0x7f89300c6180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:58.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:58.665+0000 7f892f400640 1 -- 172.21.15.6:0/875036561 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3096533476 0 0) 0x7f8930070760 con 0x7f893006ff30 2024-04-24T23:15:58.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:58.665+0000 7f8934e00640 1 -- 172.21.15.6:0/875036561 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8930070480 con 0x7f8930151a10 2024-04-24T23:15:58.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:58.666+0000 7f8934e00640 1 --2- 172.21.15.6:0/875036561 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8930151a10 0x7f89300c46f0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f893015b8b0 tx=0x7f892002c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1bf70bd10075bbc0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:15:58.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:58.666+0000 7f892f400640 1 -- 172.21.15.6:0/875036561 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8920009600 con 0x7f8930151a10 2024-04-24T23:15:58.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:58.667+0000 7f892f400640 1 -- 172.21.15.6:0/875036561 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8920037070 con 0x7f8930151a10 2024-04-24T23:15:58.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:58.667+0000 7f892f400640 1 -- 172.21.15.6:0/875036561 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89200099c0 con 0x7f8930151a10 2024-04-24T23:15:58.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:58.667+0000 7f8936200640 1 -- 172.21.15.6:0/875036561 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8930151a10 msgr2=0x7f89300c46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:58.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:58.667+0000 7f8936200640 1 --2- 172.21.15.6:0/875036561 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8930151a10 0x7f89300c46f0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f893015b8b0 tx=0x7f892002c920 comp rx=0 tx=0).stop 2024-04-24T23:15:58.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:58.667+0000 7f8936200640 1 -- 172.21.15.6:0/875036561 shutdown_connections 2024-04-24T23:15:58.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:58.667+0000 7f8936200640 1 --2- 172.21.15.6:0/875036561 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8930151a10 0x7f89300c46f0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:58.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:58.667+0000 7f8936200640 1 -- 172.21.15.6:0/875036561 >> 172.21.15.6:0/875036561 conn(0x7f893006c1f0 msgr2=0x7f893006c5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:15:58.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:58.667+0000 7f8936200640 1 -- 172.21.15.6:0/875036561 shutdown_connections 2024-04-24T23:15:58.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:58.668+0000 7f8936200640 1 -- 172.21.15.6:0/875036561 wait complete. 2024-04-24T23:15:58.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:58.669+0000 7f8936200640 1 Processor -- start 2024-04-24T23:15:58.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:58.669+0000 7f8936200640 1 -- start start 2024-04-24T23:15:58.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:58.670+0000 7f8936200640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f893006f9e0 0x7f8930152280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:58.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:58.670+0000 7f8936200640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f893006ff30 0x7f89301527c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:58.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:58.670+0000 7f8936200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8930151a10 0x7f89300cb360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:58.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:58.670+0000 7f8936200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f893015b3c0 con 0x7f8930151a10 2024-04-24T23:15:58.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:58.670+0000 7f8936200640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f893015b0e0 con 0x7f893006f9e0 2024-04-24T23:15:58.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:58.670+0000 7f8936200640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f8930157e70 con 0x7f893006ff30 2024-04-24T23:15:58.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:58.671+0000 7f8935800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8930151a10 0x7f89300cb360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:58.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:58.671+0000 7f8934e00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f893006f9e0 0x7f8930152280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:58.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:58.671+0000 7f892fe00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f893006ff30 0x7f89301527c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:58.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:58.671+0000 7f8934e00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f893006f9e0 0x7f8930152280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:38506/0 (socket says 172.21.15.6:38506) 2024-04-24T23:15:58.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:58.671+0000 7f8935800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8930151a10 0x7f89300cb360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:54316/0 (socket says 172.21.15.6:54316) 2024-04-24T23:15:58.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:58.671+0000 7f8934e00640 1 -- 172.21.15.6:0/60911952 learned_addr learned my addr 172.21.15.6:0/60911952 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:15:58.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:58.671+0000 7f8934e00640 1 -- 172.21.15.6:0/60911952 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f893006ff30 msgr2=0x7f89301527c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:58.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:58.671+0000 7f8934e00640 1 --2- 172.21.15.6:0/60911952 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f893006ff30 0x7f89301527c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:58.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:58.671+0000 7f8934e00640 1 -- 172.21.15.6:0/60911952 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8930151a10 msgr2=0x7f89300cb360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:58.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:58.671+0000 7f8934e00640 1 --2- 172.21.15.6:0/60911952 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8930151a10 0x7f89300cb360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:58.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:58.671+0000 7f8934e00640 1 -- 172.21.15.6:0/60911952 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89300cc8b0 con 0x7f893006f9e0 2024-04-24T23:15:58.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:58.672+0000 7f8934e00640 1 --2- 172.21.15.6:0/60911952 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f893006f9e0 0x7f8930152280 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f8920034040 tx=0x7f8920004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:15:58.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:58.672+0000 7f892cc00640 1 -- 172.21.15.6:0/60911952 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f892000b720 con 0x7f893006f9e0 2024-04-24T23:15:58.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:58.672+0000 7f892cc00640 1 -- 172.21.15.6:0/60911952 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8920032820 con 0x7f893006f9e0 2024-04-24T23:15:58.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:58.672+0000 7f892cc00640 1 -- 172.21.15.6:0/60911952 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8920032c10 con 0x7f893006f9e0 2024-04-24T23:15:58.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:58.672+0000 7f8936200640 1 -- 172.21.15.6:0/60911952 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89300ccb30 con 0x7f893006f9e0 2024-04-24T23:15:58.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:58.672+0000 7f8936200640 1 -- 172.21.15.6:0/60911952 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89300cd0f0 con 0x7f893006f9e0 2024-04-24T23:15:58.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:58.673+0000 7f8936200640 1 -- 172.21.15.6:0/60911952 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8930075ba0 con 0x7f893006f9e0 2024-04-24T23:15:58.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:58.674+0000 7f892cc00640 1 -- 172.21.15.6:0/60911952 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f8920043020 con 0x7f893006f9e0 2024-04-24T23:15:58.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:58.675+0000 7f892cc00640 1 --2- 172.21.15.6:0/60911952 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f890c06be60 0x7f890c06e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:15:58.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:58.675+0000 7f892cc00640 1 -- 172.21.15.6:0/60911952 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f89200ac2d0 con 0x7f893006f9e0 2024-04-24T23:15:58.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:58.675+0000 7f892fe00640 1 --2- 172.21.15.6:0/60911952 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f890c06be60 0x7f890c06e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:15:58.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:58.676+0000 7f892fe00640 1 --2- 172.21.15.6:0/60911952 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f890c06be60 0x7f890c06e320 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f8918007f70 tx=0x7f8918007f00 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:15:58.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:58.680+0000 7f892cc00640 1 -- 172.21.15.6:0/60911952 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f89300cc8b0 con 0x7f893006f9e0 2024-04-24T23:15:58.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:58.896+0000 7f8936200640 1 -- 172.21.15.6:0/60911952 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f89300c7470 con 0x7f893006f9e0 2024-04-24T23:15:58.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:58.897+0000 7f892cc00640 1 -- 172.21.15.6:0/60911952 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v24) ==== 80+0+0 (secure 0 0 0) 0x7f892002f070 con 0x7f893006f9e0 2024-04-24T23:15:58.900 INFO:teuthology.orchestra.run.smithi006.stderr:listed 0 entries 2024-04-24T23:15:58.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:58.900+0000 7f8936200640 1 -- 172.21.15.6:0/60911952 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f890c06be60 msgr2=0x7f890c06e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:58.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:58.901+0000 7f8936200640 1 --2- 172.21.15.6:0/60911952 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f890c06be60 0x7f890c06e320 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f8918007f70 tx=0x7f8918007f00 comp rx=0 tx=0).stop 2024-04-24T23:15:58.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:58.901+0000 7f8936200640 1 -- 172.21.15.6:0/60911952 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f893006f9e0 msgr2=0x7f8930152280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:15:58.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:58.901+0000 7f8936200640 1 --2- 172.21.15.6:0/60911952 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f893006f9e0 0x7f8930152280 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f8920034040 tx=0x7f8920004a40 comp rx=0 tx=0).stop 2024-04-24T23:15:58.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:58.901+0000 7f8936200640 1 -- 172.21.15.6:0/60911952 shutdown_connections 2024-04-24T23:15:58.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:58.901+0000 7f8936200640 1 --2- 172.21.15.6:0/60911952 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f890c06be60 0x7f890c06e320 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:58.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:58.901+0000 7f8936200640 1 --2- 172.21.15.6:0/60911952 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f893006ff30 0x7f89301527c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:58.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:58.901+0000 7f8936200640 1 --2- 172.21.15.6:0/60911952 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8930151a10 0x7f89300cb360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:58.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:58.901+0000 7f8936200640 1 --2- 172.21.15.6:0/60911952 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f893006f9e0 0x7f8930152280 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:15:58.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:58.901+0000 7f8936200640 1 -- 172.21.15.6:0/60911952 >> 172.21.15.6:0/60911952 conn(0x7f893006c1f0 msgr2=0x7f8930158ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:15:58.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:58.901+0000 7f8936200640 1 -- 172.21.15.6:0/60911952 shutdown_connections 2024-04-24T23:15:58.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:15:58.901+0000 7f8936200640 1 -- 172.21.15.6:0/60911952 wait complete. 2024-04-24T23:15:58.922 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi006.front.sepia.ceph.com... 2024-04-24T23:15:58.922 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:15:58.923 DEBUG:teuthology.orchestra.run.smithi006:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2024-04-24T23:15:59.008 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi006.stderr:ceph-fuse[40100]: fuse finished with error 0 and tester_r 0 2024-04-24T23:15:59.016 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T23:16:02.387 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-24T23:16:02.395 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-24T23:16:05.117 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi006.front.sepia.ceph.com 2024-04-24T23:16:05.118 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:16:05.118 DEBUG:teuthology.orchestra.run.smithi006:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2024-04-24T23:16:05.186 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T23:16:05.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:05.402+0000 7fc3d2400640 1 Processor -- start 2024-04-24T23:16:05.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:05.403+0000 7fc3d2400640 1 -- start start 2024-04-24T23:16:05.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:05.404+0000 7fc3d2400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc3cc152bd0 0x7fc3cc158e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:05.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:05.404+0000 7fc3d2400640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fc3cc06ce30 0x7fc3cc159380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:05.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:05.404+0000 7fc3d2400640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3cc06d390 con 0x7fc3cc0c6be0 2024-04-24T23:16:05.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:05.404+0000 7fc3d2400640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7fc3cc06d970 con 0x7fc3cc152bd0 2024-04-24T23:16:05.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:05.404+0000 7fc3d2400640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7fc3cc06dbc0 con 0x7fc3cc06ce30 2024-04-24T23:16:05.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:05.404+0000 7fc3d1000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc3cc152bd0 0x7fc3cc158e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:05.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:05.404+0000 7fc3cbe00640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fc3cc06ce30 0x7fc3cc159380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:05.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:05.404+0000 7fc3d1a00640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7fc3cc0c6be0 0x7fc3cc0cc600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.6:50530/0 (socket says 172.21.15.6:50530) 2024-04-24T23:16:05.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:05.404+0000 7fc3d1a00640 1 -- 172.21.15.6:0/367664975 learned_addr learned my addr 172.21.15.6:0/367664975 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:16:05.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:05.404+0000 7fc3d1000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc3cc152bd0 0x7fc3cc158e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:59824/0 (socket says 172.21.15.6:59824) 2024-04-24T23:16:05.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:05.405+0000 7fc3cbe00640 1 -- 172.21.15.6:0/367664975 >> v2:172.21.15.6:3300/0 conn(0x7fc3cc152bd0 msgr2=0x7fc3cc158e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:16:05.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:05.405+0000 7fc3cbe00640 1 --2- 172.21.15.6:0/367664975 >> v2:172.21.15.6:3300/0 conn(0x7fc3cc152bd0 0x7fc3cc158e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:05.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:05.405+0000 7fc3cbe00640 1 -- 172.21.15.6:0/367664975 >> v1:172.21.15.190:6789/0 conn(0x7fc3cc0c6be0 legacy=0x7fc3cc0cc600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:05.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:05.405+0000 7fc3cbe00640 1 -- 172.21.15.6:0/367664975 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3cc06dea0 con 0x7fc3cc06ce30 2024-04-24T23:16:05.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:05.406+0000 7fc3cbe00640 1 --2- 172.21.15.6:0/367664975 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc3cc06ce30 0x7fc3cc159380 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fc3cc070820 tx=0x7fc3b802a1a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a71997fc662edd15 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:05.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:05.406+0000 7fc3cb400640 1 -- 172.21.15.6:0/367664975 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3b8037070 con 0x7fc3cc06ce30 2024-04-24T23:16:05.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:05.406+0000 7fc3cb400640 1 -- 172.21.15.6:0/367664975 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc3b8032440 con 0x7fc3cc06ce30 2024-04-24T23:16:05.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:05.406+0000 7fc3cb400640 1 -- 172.21.15.6:0/367664975 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3b8032810 con 0x7fc3cc06ce30 2024-04-24T23:16:05.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:05.407+0000 7fc3d2400640 1 -- 172.21.15.6:0/367664975 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc3cc06ce30 msgr2=0x7fc3cc159380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:05.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:05.407+0000 7fc3d2400640 1 --2- 172.21.15.6:0/367664975 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc3cc06ce30 0x7fc3cc159380 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fc3cc070820 tx=0x7fc3b802a1a0 comp rx=0 tx=0).stop 2024-04-24T23:16:05.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:05.407+0000 7fc3d2400640 1 -- 172.21.15.6:0/367664975 shutdown_connections 2024-04-24T23:16:05.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:05.407+0000 7fc3d2400640 1 --2- 172.21.15.6:0/367664975 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc3cc06ce30 0x7fc3cc159380 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:05.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:05.407+0000 7fc3d2400640 1 --2- 172.21.15.6:0/367664975 >> v2:172.21.15.6:3300/0 conn(0x7fc3cc152bd0 0x7fc3cc158e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:05.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:05.407+0000 7fc3d2400640 1 -- 172.21.15.6:0/367664975 >> 172.21.15.6:0/367664975 conn(0x7fc3cc06ed80 msgr2=0x7fc3cc06f190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:05.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:05.407+0000 7fc3d2400640 1 -- 172.21.15.6:0/367664975 shutdown_connections 2024-04-24T23:16:05.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:05.407+0000 7fc3d2400640 1 -- 172.21.15.6:0/367664975 wait complete. 2024-04-24T23:16:05.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:05.409+0000 7fc3d2400640 1 Processor -- start 2024-04-24T23:16:05.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:05.409+0000 7fc3d2400640 1 -- start start 2024-04-24T23:16:05.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:05.410+0000 7fc3d2400640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc3cc06ce30 0x7fc3cc0cb4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:05.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:05.410+0000 7fc3d2400640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc3cc0c6be0 0x7fc3cc1e1b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:05.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:05.410+0000 7fc3d2400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc3cc152bd0 0x7fc3cc1f1d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:05.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:05.410+0000 7fc3d2400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fc3cc072650 con 0x7fc3cc152bd0 2024-04-24T23:16:05.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:05.410+0000 7fc3d2400640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fc3cc071890 con 0x7fc3cc0c6be0 2024-04-24T23:16:05.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:05.410+0000 7fc3d2400640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fc3cc0727d0 con 0x7fc3cc06ce30 2024-04-24T23:16:05.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:05.410+0000 7fc3d1a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc3cc152bd0 0x7fc3cc1f1d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:05.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:05.410+0000 7fc3d1a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc3cc152bd0 0x7fc3cc1f1d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:59826/0 (socket says 172.21.15.6:59826) 2024-04-24T23:16:05.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:05.410+0000 7fc3d1a00640 1 -- 172.21.15.6:0/3361290666 learned_addr learned my addr 172.21.15.6:0/3361290666 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:16:05.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:05.410+0000 7fc3cbe00640 1 --2- 172.21.15.6:0/3361290666 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc3cc0c6be0 0x7fc3cc1e1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:05.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:05.410+0000 7fc3d1000640 1 --2- 172.21.15.6:0/3361290666 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc3cc06ce30 0x7fc3cc0cb4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:05.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:05.411+0000 7fc3d1a00640 1 -- 172.21.15.6:0/3361290666 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc3cc06ce30 msgr2=0x7fc3cc0cb4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:05.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:05.411+0000 7fc3d1a00640 1 --2- 172.21.15.6:0/3361290666 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc3cc06ce30 0x7fc3cc0cb4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:05.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:05.411+0000 7fc3d1a00640 1 -- 172.21.15.6:0/3361290666 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc3cc0c6be0 msgr2=0x7fc3cc1e1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:05.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:05.411+0000 7fc3d1a00640 1 --2- 172.21.15.6:0/3361290666 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc3cc0c6be0 0x7fc3cc1e1b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:05.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:05.411+0000 7fc3d1a00640 1 -- 172.21.15.6:0/3361290666 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3cc0c36e0 con 0x7fc3cc152bd0 2024-04-24T23:16:05.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:05.411+0000 7fc3cbe00640 1 --2- 172.21.15.6:0/3361290666 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc3cc0c6be0 0x7fc3cc1e1b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:16:05.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:05.411+0000 7fc3d1000640 1 --2- 172.21.15.6:0/3361290666 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc3cc06ce30 0x7fc3cc0cb4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:16:05.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:05.411+0000 7fc3d1a00640 1 --2- 172.21.15.6:0/3361290666 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc3cc152bd0 0x7fc3cc1f1d80 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fc3c400cbf0 tx=0x7fc3c400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:05.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:05.411+0000 7fc3c8c00640 1 -- 172.21.15.6:0/3361290666 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3c4017070 con 0x7fc3cc152bd0 2024-04-24T23:16:05.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:05.411+0000 7fc3c8c00640 1 -- 172.21.15.6:0/3361290666 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc3c400ad40 con 0x7fc3cc152bd0 2024-04-24T23:16:05.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:05.411+0000 7fc3c8c00640 1 -- 172.21.15.6:0/3361290666 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3c4012600 con 0x7fc3cc152bd0 2024-04-24T23:16:05.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:05.411+0000 7fc3d2400640 1 -- 172.21.15.6:0/3361290666 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3cc0c39c0 con 0x7fc3cc152bd0 2024-04-24T23:16:05.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:05.411+0000 7fc3d2400640 1 -- 172.21.15.6:0/3361290666 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3cc0c3f80 con 0x7fc3cc152bd0 2024-04-24T23:16:05.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:05.412+0000 7fc3d2400640 1 -- 172.21.15.6:0/3361290666 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc3cc075b30 con 0x7fc3cc152bd0 2024-04-24T23:16:05.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:05.413+0000 7fc3c8c00640 1 -- 172.21.15.6:0/3361290666 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fc3c401a070 con 0x7fc3cc152bd0 2024-04-24T23:16:05.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:05.414+0000 7fc3c8c00640 1 --2- 172.21.15.6:0/3361290666 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fc3ac06be60 0x7fc3ac06e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:05.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:05.414+0000 7fc3c8c00640 1 -- 172.21.15.6:0/3361290666 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fc3c408af90 con 0x7fc3cc152bd0 2024-04-24T23:16:05.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:05.415+0000 7fc3d1000640 1 --2- 172.21.15.6:0/3361290666 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fc3ac06be60 0x7fc3ac06e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:05.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:05.415+0000 7fc3d1000640 1 --2- 172.21.15.6:0/3361290666 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fc3ac06be60 0x7fc3ac06e320 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fc3c0009af0 tx=0x7fc3c0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:05.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:05.419+0000 7fc3c8c00640 1 -- 172.21.15.6:0/3361290666 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fc3c4058e80 con 0x7fc3cc152bd0 2024-04-24T23:16:05.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:05.624+0000 7fc3d2400640 1 -- 172.21.15.6:0/3361290666 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc3cc06d750 con 0x7fc3cc152bd0 2024-04-24T23:16:05.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:05.627+0000 7fc3c8c00640 1 -- 172.21.15.6:0/3361290666 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+18458 (secure 0 0 0) 0x7fc3c405d180 con 0x7fc3cc152bd0 2024-04-24T23:16:05.630 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:16:05.631 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":24,"fsid":"18bef68d-5e6d-4ec5-862b-b6f206e2588d","created":"2024-04-24T23:15:12.535072+0000","modified":"2024-04-24T23:15:41.465875+0000","last_up_change":"2024-04-24T23:15:23.773223+0000","last_in_change":"2024-04-24T23:15:14.905328+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T23:15:24.747441+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-24T23:15:34.051800+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-24T23:15:35.436870+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-24T23:15:39.034257+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"3b7a212d-d498-462e-90bf-ed493c803a09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":3022569437},{"type":"v1","addr":"172.21.15.6:6809","nonce":3022569437}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":3022569437},{"type":"v1","addr":"172.21.15.6:6811","nonce":3022569437}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":3022569437},{"type":"v1","addr":"172.21.15.6:6815","nonce":3022569437}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":3022569437},{"type":"v1","addr":"172.21.15.6:6813","nonce":3022569437}]},"public_addr":"172.21.15.6:6809/3022569437","cluster_addr":"172.21.15.6:6811/3022569437","heartbeat_back_addr":"172.21.15.6:6815/3022569437","heartbeat_front_addr":"172.21.15.6:6813/3022569437","state":["exists","up"]},{"osd":1,"uuid":"7c97f5a2-1735-4775-9f15-eb50099b19d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":3687836888},{"type":"v1","addr":"172.21.15.6:6817","nonce":3687836888}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":3687836888},{"type":"v1","addr":"172.21.15.6:6819","nonce":3687836888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":3687836888},{"type":"v1","addr":"172.21.15.6:6823","nonce":3687836888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":3687836888},{"type":"v1","addr":"172.21.15.6:6821","nonce":3687836888}]},"public_addr":"172.21.15.6:6817/3687836888","cluster_addr":"172.21.15.6:6819/3687836888","heartbeat_back_addr":"172.21.15.6:6823/3687836888","heartbeat_front_addr":"172.21.15.6:6821/3687836888","state":["exists","up"]},{"osd":2,"uuid":"d86bbc18-438c-4ec2-8018-ba7dd78f790b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":1020213724},{"type":"v1","addr":"172.21.15.6:6801","nonce":1020213724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":1020213724},{"type":"v1","addr":"172.21.15.6:6803","nonce":1020213724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":1020213724},{"type":"v1","addr":"172.21.15.6:6807","nonce":1020213724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":1020213724},{"type":"v1","addr":"172.21.15.6:6805","nonce":1020213724}]},"public_addr":"172.21.15.6:6801/1020213724","cluster_addr":"172.21.15.6:6803/1020213724","heartbeat_back_addr":"172.21.15.6:6807/1020213724","heartbeat_front_addr":"172.21.15.6:6805/1020213724","state":["exists","up"]},{"osd":3,"uuid":"3754ae16-4248-40dd-95f9-0f59c0eb5426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6824","nonce":3163883468},{"type":"v1","addr":"172.21.15.6:6825","nonce":3163883468}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":3163883468},{"type":"v1","addr":"172.21.15.6:6827","nonce":3163883468}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":3163883468},{"type":"v1","addr":"172.21.15.6:6831","nonce":3163883468}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":3163883468},{"type":"v1","addr":"172.21.15.6:6829","nonce":3163883468}]},"public_addr":"172.21.15.6:6825/3163883468","cluster_addr":"172.21.15.6:6827/3163883468","heartbeat_back_addr":"172.21.15.6:6831/3163883468","heartbeat_front_addr":"172.21.15.6:6829/3163883468","state":["exists","up"]},{"osd":4,"uuid":"8b2913a0-d006-479c-84ed-5b21e0e6c392","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":1989500934},{"type":"v1","addr":"172.21.15.190:6809","nonce":1989500934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":1989500934},{"type":"v1","addr":"172.21.15.190:6811","nonce":1989500934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":1989500934},{"type":"v1","addr":"172.21.15.190:6815","nonce":1989500934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":1989500934},{"type":"v1","addr":"172.21.15.190:6813","nonce":1989500934}]},"public_addr":"172.21.15.190:6809/1989500934","cluster_addr":"172.21.15.190:6811/1989500934","heartbeat_back_addr":"172.21.15.190:6815/1989500934","heartbeat_front_addr":"172.21.15.190:6813/1989500934","state":["exists","up"]},{"osd":5,"uuid":"499b79d0-0585-4dae-ab69-196a551793da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":3687548412},{"type":"v1","addr":"172.21.15.190:6801","nonce":3687548412}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":3687548412},{"type":"v1","addr":"172.21.15.190:6803","nonce":3687548412}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":3687548412},{"type":"v1","addr":"172.21.15.190:6807","nonce":3687548412}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":3687548412},{"type":"v1","addr":"172.21.15.190:6805","nonce":3687548412}]},"public_addr":"172.21.15.190:6801/3687548412","cluster_addr":"172.21.15.190:6803/3687548412","heartbeat_back_addr":"172.21.15.190:6807/3687548412","heartbeat_front_addr":"172.21.15.190:6805/3687548412","state":["exists","up"]},{"osd":6,"uuid":"6ba6df3b-4a6b-4a89-a890-0f79f387d4d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":870571564},{"type":"v1","addr":"172.21.15.190:6817","nonce":870571564}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":870571564},{"type":"v1","addr":"172.21.15.190:6819","nonce":870571564}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":870571564},{"type":"v1","addr":"172.21.15.190:6823","nonce":870571564}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":870571564},{"type":"v1","addr":"172.21.15.190:6821","nonce":870571564}]},"public_addr":"172.21.15.190:6817/870571564","cluster_addr":"172.21.15.190:6819/870571564","heartbeat_back_addr":"172.21.15.190:6823/870571564","heartbeat_front_addr":"172.21.15.190:6821/870571564","state":["exists","up"]},{"osd":7,"uuid":"e9668b1c-a000-4332-86e8-4efd73b02409","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":123639197},{"type":"v1","addr":"172.21.15.190:6825","nonce":123639197}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":123639197},{"type":"v1","addr":"172.21.15.190:6827","nonce":123639197}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":123639197},{"type":"v1","addr":"172.21.15.190:6831","nonce":123639197}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":123639197},{"type":"v1","addr":"172.21.15.190:6829","nonce":123639197}]},"public_addr":"172.21.15.190:6825/123639197","cluster_addr":"172.21.15.190:6827/123639197","heartbeat_back_addr":"172.21.15.190:6831/123639197","heartbeat_front_addr":"172.21.15.190:6829/123639197","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.321646+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.363873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.466963+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.531989+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:19.990707+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.043543+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.032492+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.115004+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T23:16:05.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:05.630+0000 7fc3d2400640 1 -- 172.21.15.6:0/3361290666 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fc3ac06be60 msgr2=0x7fc3ac06e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:05.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:05.630+0000 7fc3d2400640 1 --2- 172.21.15.6:0/3361290666 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fc3ac06be60 0x7fc3ac06e320 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fc3c0009af0 tx=0x7fc3c0008040 comp rx=0 tx=0).stop 2024-04-24T23:16:05.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:05.630+0000 7fc3d2400640 1 -- 172.21.15.6:0/3361290666 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc3cc152bd0 msgr2=0x7fc3cc1f1d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:05.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:05.630+0000 7fc3d2400640 1 --2- 172.21.15.6:0/3361290666 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc3cc152bd0 0x7fc3cc1f1d80 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fc3c400cbf0 tx=0x7fc3c400a430 comp rx=0 tx=0).stop 2024-04-24T23:16:05.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:05.630+0000 7fc3d2400640 1 -- 172.21.15.6:0/3361290666 shutdown_connections 2024-04-24T23:16:05.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:05.630+0000 7fc3d2400640 1 --2- 172.21.15.6:0/3361290666 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fc3ac06be60 0x7fc3ac06e320 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:05.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:05.630+0000 7fc3d2400640 1 --2- 172.21.15.6:0/3361290666 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc3cc152bd0 0x7fc3cc1f1d80 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:05.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:05.630+0000 7fc3d2400640 1 --2- 172.21.15.6:0/3361290666 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc3cc0c6be0 0x7fc3cc1e1b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:05.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:05.630+0000 7fc3d2400640 1 --2- 172.21.15.6:0/3361290666 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc3cc06ce30 0x7fc3cc0cb4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:05.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:05.631+0000 7fc3d2400640 1 -- 172.21.15.6:0/3361290666 >> 172.21.15.6:0/3361290666 conn(0x7fc3cc06ed80 msgr2=0x7fc3cc0c7230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:05.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:05.631+0000 7fc3d2400640 1 -- 172.21.15.6:0/3361290666 shutdown_connections 2024-04-24T23:16:05.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:05.631+0000 7fc3d2400640 1 -- 172.21.15.6:0/3361290666 wait complete. 2024-04-24T23:16:05.654 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T23:16:05.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:05.865+0000 7f1e3fe00640 1 Processor -- start 2024-04-24T23:16:05.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:05.866+0000 7f1e3fe00640 1 -- start start 2024-04-24T23:16:05.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:05.867+0000 7f1e3fe00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1e400cb090 0x7f1e400d0820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:05.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:05.867+0000 7f1e3fe00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1e40155170 con 0x7f1e40154d90 2024-04-24T23:16:05.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:05.867+0000 7f1e3fe00640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1e400cc9a0 con 0x7f1e400d0d60 2024-04-24T23:16:05.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:05.867+0000 7f1e3fe00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f1e4015ad60 con 0x7f1e400cb090 2024-04-24T23:16:05.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:05.867+0000 7f1e3ea00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f1e40154d90 0x7f1e400cbe80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:43362/0 (socket says 172.21.15.6:43362) 2024-04-24T23:16:05.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:05.868+0000 7f1e3ea00640 1 -- 172.21.15.6:0/2465108029 learned_addr learned my addr 172.21.15.6:0/2465108029 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:16:05.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:05.868+0000 7f1e3e000640 1 --2- 172.21.15.6:0/2465108029 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1e400cb090 0x7f1e400d0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:05.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:05.868+0000 7f1e3d600640 1 -- 172.21.15.6:0/2465108029 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2254329681 0 0) 0x7f1e400cc9a0 con 0x7f1e400d0d60 2024-04-24T23:16:05.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:05.868+0000 7f1e3d600640 1 -- 172.21.15.6:0/2465108029 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1e20003520 con 0x7f1e400d0d60 2024-04-24T23:16:05.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:05.868+0000 7f1e3d600640 1 -- 172.21.15.6:0/2465108029 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2394441097 0 0) 0x7f1e40155170 con 0x7f1e40154d90 2024-04-24T23:16:05.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:05.868+0000 7f1e3d600640 1 -- 172.21.15.6:0/2465108029 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1e400cc9a0 con 0x7f1e40154d90 2024-04-24T23:16:05.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:05.868+0000 7f1e3e000640 1 -- 172.21.15.6:0/2465108029 >> v1:172.21.15.190:6789/0 conn(0x7f1e400d0d60 legacy=0x7f1e4015e000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:05.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:05.868+0000 7f1e3e000640 1 -- 172.21.15.6:0/2465108029 >> v1:172.21.15.6:6789/0 conn(0x7f1e40154d90 legacy=0x7f1e400cbe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:05.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:05.868+0000 7f1e3e000640 1 -- 172.21.15.6:0/2465108029 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e4015b040 con 0x7f1e400cb090 2024-04-24T23:16:05.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:05.869+0000 7f1e3e000640 1 --2- 172.21.15.6:0/2465108029 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1e400cb090 0x7f1e400d0820 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f1e28009dc0 tx=0x7f1e2802a760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=116861b2f22b2a40 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:05.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:05.870+0000 7f1e3d600640 1 -- 172.21.15.6:0/2465108029 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e28007740 con 0x7f1e400cb090 2024-04-24T23:16:05.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:05.870+0000 7f1e3d600640 1 -- 172.21.15.6:0/2465108029 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e280078e0 con 0x7f1e400cb090 2024-04-24T23:16:05.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:05.870+0000 7f1e3d600640 1 -- 172.21.15.6:0/2465108029 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e28007c90 con 0x7f1e400cb090 2024-04-24T23:16:05.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:05.870+0000 7f1e3fe00640 1 -- 172.21.15.6:0/2465108029 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1e400cb090 msgr2=0x7f1e400d0820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:05.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:05.870+0000 7f1e3fe00640 1 --2- 172.21.15.6:0/2465108029 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1e400cb090 0x7f1e400d0820 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f1e28009dc0 tx=0x7f1e2802a760 comp rx=0 tx=0).stop 2024-04-24T23:16:05.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:05.871+0000 7f1e3fe00640 1 -- 172.21.15.6:0/2465108029 shutdown_connections 2024-04-24T23:16:05.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:05.871+0000 7f1e3fe00640 1 --2- 172.21.15.6:0/2465108029 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1e400cb090 0x7f1e400d0820 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:05.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:05.871+0000 7f1e3fe00640 1 -- 172.21.15.6:0/2465108029 >> 172.21.15.6:0/2465108029 conn(0x7f1e4006be10 msgr2=0x7f1e4006c1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:05.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:05.871+0000 7f1e3fe00640 1 -- 172.21.15.6:0/2465108029 shutdown_connections 2024-04-24T23:16:05.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:05.871+0000 7f1e3fe00640 1 -- 172.21.15.6:0/2465108029 wait complete. 2024-04-24T23:16:05.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:05.872+0000 7f1e3fe00640 1 Processor -- start 2024-04-24T23:16:05.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:05.872+0000 7f1e3fe00640 1 -- start start 2024-04-24T23:16:05.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:05.873+0000 7f1e3fe00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1e400cb090 0x7f1e40157ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:05.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:05.873+0000 7f1e3fe00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1e400d0d60 0x7f1e400cf4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:05.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:05.873+0000 7f1e3fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1e40154d90 0x7f1e401f3c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:05.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:05.873+0000 7f1e3fe00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f1e4015aa30 con 0x7f1e40154d90 2024-04-24T23:16:05.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:05.873+0000 7f1e3fe00640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f1e4015b600 con 0x7f1e400cb090 2024-04-24T23:16:05.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:05.873+0000 7f1e3fe00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f1e400cc590 con 0x7f1e400d0d60 2024-04-24T23:16:05.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:05.874+0000 7f1e3f400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1e40154d90 0x7f1e401f3c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:05.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:05.874+0000 7f1e3ea00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1e400cb090 0x7f1e40157ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:05.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:05.874+0000 7f1e3f400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1e40154d90 0x7f1e401f3c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:59832/0 (socket says 172.21.15.6:59832) 2024-04-24T23:16:05.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:05.874+0000 7f1e3f400640 1 -- 172.21.15.6:0/2728914057 learned_addr learned my addr 172.21.15.6:0/2728914057 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:16:05.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:05.874+0000 7f1e3e000640 1 --2- 172.21.15.6:0/2728914057 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1e400d0d60 0x7f1e400cf4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:05.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:05.874+0000 7f1e3f400640 1 -- 172.21.15.6:0/2728914057 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1e400d0d60 msgr2=0x7f1e400cf4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:05.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:05.874+0000 7f1e3f400640 1 --2- 172.21.15.6:0/2728914057 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1e400d0d60 0x7f1e400cf4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:05.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:05.874+0000 7f1e3f400640 1 -- 172.21.15.6:0/2728914057 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1e400cb090 msgr2=0x7f1e40157ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:05.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:05.874+0000 7f1e3f400640 1 --2- 172.21.15.6:0/2728914057 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1e400cb090 0x7f1e40157ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:05.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:05.874+0000 7f1e3f400640 1 -- 172.21.15.6:0/2728914057 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e40067700 con 0x7f1e40154d90 2024-04-24T23:16:05.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:05.874+0000 7f1e3ea00640 1 --2- 172.21.15.6:0/2728914057 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1e400cb090 0x7f1e40157ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:16:05.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:05.874+0000 7f1e3f400640 1 --2- 172.21.15.6:0/2728914057 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1e40154d90 0x7f1e401f3c00 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f1e3800c460 tx=0x7f1e3800c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:05.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:05.875+0000 7f1e37400640 1 -- 172.21.15.6:0/2728914057 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e38017070 con 0x7f1e40154d90 2024-04-24T23:16:05.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:05.875+0000 7f1e37400640 1 -- 172.21.15.6:0/2728914057 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e380052d0 con 0x7f1e40154d90 2024-04-24T23:16:05.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:05.875+0000 7f1e37400640 1 -- 172.21.15.6:0/2728914057 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e38003810 con 0x7f1e40154d90 2024-04-24T23:16:05.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:05.875+0000 7f1e3fe00640 1 -- 172.21.15.6:0/2728914057 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e4015a300 con 0x7f1e40154d90 2024-04-24T23:16:05.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:05.875+0000 7f1e3fe00640 1 -- 172.21.15.6:0/2728914057 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e4014ca60 con 0x7f1e40154d90 2024-04-24T23:16:05.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:05.876+0000 7f1e3fe00640 1 -- 172.21.15.6:0/2728914057 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1e40075b30 con 0x7f1e40154d90 2024-04-24T23:16:05.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:05.877+0000 7f1e37400640 1 -- 172.21.15.6:0/2728914057 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f1e3801a070 con 0x7f1e40154d90 2024-04-24T23:16:05.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:05.878+0000 7f1e37400640 1 --2- 172.21.15.6:0/2728914057 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f1e1c06be60 0x7f1e1c06e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:05.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:05.878+0000 7f1e37400640 1 -- 172.21.15.6:0/2728914057 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f1e3808a480 con 0x7f1e40154d90 2024-04-24T23:16:05.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:05.879+0000 7f1e3ea00640 1 --2- 172.21.15.6:0/2728914057 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f1e1c06be60 0x7f1e1c06e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:05.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:05.879+0000 7f1e3ea00640 1 --2- 172.21.15.6:0/2728914057 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f1e1c06be60 0x7f1e1c06e320 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f1e300011e0 tx=0x7f1e30007d30 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:05.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:05.883+0000 7f1e37400640 1 -- 172.21.15.6:0/2728914057 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f1e38058490 con 0x7f1e40154d90 2024-04-24T23:16:06.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:06.086+0000 7f1e3fe00640 1 -- 172.21.15.6:0/2728914057 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1e400cbc60 con 0x7f1e40154d90 2024-04-24T23:16:06.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:06.089+0000 7f1e37400640 1 -- 172.21.15.6:0/2728914057 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+18458 (secure 0 0 0) 0x7f1e3805c790 con 0x7f1e40154d90 2024-04-24T23:16:06.091 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:16:06.091 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":24,"fsid":"18bef68d-5e6d-4ec5-862b-b6f206e2588d","created":"2024-04-24T23:15:12.535072+0000","modified":"2024-04-24T23:15:41.465875+0000","last_up_change":"2024-04-24T23:15:23.773223+0000","last_in_change":"2024-04-24T23:15:14.905328+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T23:15:24.747441+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-24T23:15:34.051800+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-24T23:15:35.436870+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-24T23:15:39.034257+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"3b7a212d-d498-462e-90bf-ed493c803a09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":3022569437},{"type":"v1","addr":"172.21.15.6:6809","nonce":3022569437}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":3022569437},{"type":"v1","addr":"172.21.15.6:6811","nonce":3022569437}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":3022569437},{"type":"v1","addr":"172.21.15.6:6815","nonce":3022569437}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":3022569437},{"type":"v1","addr":"172.21.15.6:6813","nonce":3022569437}]},"public_addr":"172.21.15.6:6809/3022569437","cluster_addr":"172.21.15.6:6811/3022569437","heartbeat_back_addr":"172.21.15.6:6815/3022569437","heartbeat_front_addr":"172.21.15.6:6813/3022569437","state":["exists","up"]},{"osd":1,"uuid":"7c97f5a2-1735-4775-9f15-eb50099b19d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":3687836888},{"type":"v1","addr":"172.21.15.6:6817","nonce":3687836888}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":3687836888},{"type":"v1","addr":"172.21.15.6:6819","nonce":3687836888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":3687836888},{"type":"v1","addr":"172.21.15.6:6823","nonce":3687836888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":3687836888},{"type":"v1","addr":"172.21.15.6:6821","nonce":3687836888}]},"public_addr":"172.21.15.6:6817/3687836888","cluster_addr":"172.21.15.6:6819/3687836888","heartbeat_back_addr":"172.21.15.6:6823/3687836888","heartbeat_front_addr":"172.21.15.6:6821/3687836888","state":["exists","up"]},{"osd":2,"uuid":"d86bbc18-438c-4ec2-8018-ba7dd78f790b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":1020213724},{"type":"v1","addr":"172.21.15.6:6801","nonce":1020213724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":1020213724},{"type":"v1","addr":"172.21.15.6:6803","nonce":1020213724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":1020213724},{"type":"v1","addr":"172.21.15.6:6807","nonce":1020213724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":1020213724},{"type":"v1","addr":"172.21.15.6:6805","nonce":1020213724}]},"public_addr":"172.21.15.6:6801/1020213724","cluster_addr":"172.21.15.6:6803/1020213724","heartbeat_back_addr":"172.21.15.6:6807/1020213724","heartbeat_front_addr":"172.21.15.6:6805/1020213724","state":["exists","up"]},{"osd":3,"uuid":"3754ae16-4248-40dd-95f9-0f59c0eb5426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6824","nonce":3163883468},{"type":"v1","addr":"172.21.15.6:6825","nonce":3163883468}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":3163883468},{"type":"v1","addr":"172.21.15.6:6827","nonce":3163883468}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":3163883468},{"type":"v1","addr":"172.21.15.6:6831","nonce":3163883468}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":3163883468},{"type":"v1","addr":"172.21.15.6:6829","nonce":3163883468}]},"public_addr":"172.21.15.6:6825/3163883468","cluster_addr":"172.21.15.6:6827/3163883468","heartbeat_back_addr":"172.21.15.6:6831/3163883468","heartbeat_front_addr":"172.21.15.6:6829/3163883468","state":["exists","up"]},{"osd":4,"uuid":"8b2913a0-d006-479c-84ed-5b21e0e6c392","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":1989500934},{"type":"v1","addr":"172.21.15.190:6809","nonce":1989500934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":1989500934},{"type":"v1","addr":"172.21.15.190:6811","nonce":1989500934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":1989500934},{"type":"v1","addr":"172.21.15.190:6815","nonce":1989500934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":1989500934},{"type":"v1","addr":"172.21.15.190:6813","nonce":1989500934}]},"public_addr":"172.21.15.190:6809/1989500934","cluster_addr":"172.21.15.190:6811/1989500934","heartbeat_back_addr":"172.21.15.190:6815/1989500934","heartbeat_front_addr":"172.21.15.190:6813/1989500934","state":["exists","up"]},{"osd":5,"uuid":"499b79d0-0585-4dae-ab69-196a551793da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":3687548412},{"type":"v1","addr":"172.21.15.190:6801","nonce":3687548412}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":3687548412},{"type":"v1","addr":"172.21.15.190:6803","nonce":3687548412}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":3687548412},{"type":"v1","addr":"172.21.15.190:6807","nonce":3687548412}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":3687548412},{"type":"v1","addr":"172.21.15.190:6805","nonce":3687548412}]},"public_addr":"172.21.15.190:6801/3687548412","cluster_addr":"172.21.15.190:6803/3687548412","heartbeat_back_addr":"172.21.15.190:6807/3687548412","heartbeat_front_addr":"172.21.15.190:6805/3687548412","state":["exists","up"]},{"osd":6,"uuid":"6ba6df3b-4a6b-4a89-a890-0f79f387d4d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":870571564},{"type":"v1","addr":"172.21.15.190:6817","nonce":870571564}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":870571564},{"type":"v1","addr":"172.21.15.190:6819","nonce":870571564}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":870571564},{"type":"v1","addr":"172.21.15.190:6823","nonce":870571564}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":870571564},{"type":"v1","addr":"172.21.15.190:6821","nonce":870571564}]},"public_addr":"172.21.15.190:6817/870571564","cluster_addr":"172.21.15.190:6819/870571564","heartbeat_back_addr":"172.21.15.190:6823/870571564","heartbeat_front_addr":"172.21.15.190:6821/870571564","state":["exists","up"]},{"osd":7,"uuid":"e9668b1c-a000-4332-86e8-4efd73b02409","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":123639197},{"type":"v1","addr":"172.21.15.190:6825","nonce":123639197}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":123639197},{"type":"v1","addr":"172.21.15.190:6827","nonce":123639197}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":123639197},{"type":"v1","addr":"172.21.15.190:6831","nonce":123639197}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":123639197},{"type":"v1","addr":"172.21.15.190:6829","nonce":123639197}]},"public_addr":"172.21.15.190:6825/123639197","cluster_addr":"172.21.15.190:6827/123639197","heartbeat_back_addr":"172.21.15.190:6831/123639197","heartbeat_front_addr":"172.21.15.190:6829/123639197","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.321646+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.363873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.466963+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.531989+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:19.990707+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.043543+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.032492+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.115004+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T23:16:06.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:06.091+0000 7f1e3fe00640 1 -- 172.21.15.6:0/2728914057 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f1e1c06be60 msgr2=0x7f1e1c06e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:06.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:06.091+0000 7f1e3fe00640 1 --2- 172.21.15.6:0/2728914057 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f1e1c06be60 0x7f1e1c06e320 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f1e300011e0 tx=0x7f1e30007d30 comp rx=0 tx=0).stop 2024-04-24T23:16:06.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:06.091+0000 7f1e3fe00640 1 -- 172.21.15.6:0/2728914057 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1e40154d90 msgr2=0x7f1e401f3c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:06.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:06.091+0000 7f1e3fe00640 1 --2- 172.21.15.6:0/2728914057 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1e40154d90 0x7f1e401f3c00 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f1e3800c460 tx=0x7f1e3800c920 comp rx=0 tx=0).stop 2024-04-24T23:16:06.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:06.092+0000 7f1e3fe00640 1 -- 172.21.15.6:0/2728914057 shutdown_connections 2024-04-24T23:16:06.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:06.092+0000 7f1e3fe00640 1 --2- 172.21.15.6:0/2728914057 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f1e1c06be60 0x7f1e1c06e320 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:06.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:06.092+0000 7f1e3fe00640 1 --2- 172.21.15.6:0/2728914057 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1e400d0d60 0x7f1e400cf4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:06.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:06.092+0000 7f1e3fe00640 1 --2- 172.21.15.6:0/2728914057 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1e40154d90 0x7f1e401f3c00 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:06.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:06.092+0000 7f1e3fe00640 1 --2- 172.21.15.6:0/2728914057 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1e400cb090 0x7f1e40157ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:06.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:06.092+0000 7f1e3fe00640 1 -- 172.21.15.6:0/2728914057 >> 172.21.15.6:0/2728914057 conn(0x7f1e4006be10 msgr2=0x7f1e4014d440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:06.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:06.092+0000 7f1e3fe00640 1 -- 172.21.15.6:0/2728914057 shutdown_connections 2024-04-24T23:16:06.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:06.092+0000 7f1e3fe00640 1 -- 172.21.15.6:0/2728914057 wait complete. 2024-04-24T23:16:06.114 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-24T23:16:06.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:06.325+0000 7f1610a00640 1 Processor -- start 2024-04-24T23:16:06.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:06.325+0000 7f1610a00640 1 -- start start 2024-04-24T23:16:06.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:06.327+0000 7f1610a00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f160c0c5a60 0x7f160c0c6ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:06.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:06.327+0000 7f1610a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f160c0c8570 0x7f160c1524a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:06.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:06.327+0000 7f1610a00640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f160c0cca80 con 0x7f160c0c5e40 2024-04-24T23:16:06.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:06.327+0000 7f1610a00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f160c15c440 con 0x7f160c0c8570 2024-04-24T23:16:06.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:06.327+0000 7f1610a00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f160c0ccd60 con 0x7f160c0c5a60 2024-04-24T23:16:06.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:06.327+0000 7f160be00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f160c0c8570 0x7f160c1524a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:06.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:06.327+0000 7f160b400640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f160c0c5a60 0x7f160c0c6ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:06.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:06.327+0000 7f160be00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f160c0c8570 0x7f160c1524a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:59846/0 (socket says 172.21.15.6:59846) 2024-04-24T23:16:06.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:06.327+0000 7f160be00640 1 -- 172.21.15.6:0/2008770450 learned_addr learned my addr 172.21.15.6:0/2008770450 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:16:06.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:06.328+0000 7f160be00640 1 -- 172.21.15.6:0/2008770450 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f160c0c5a60 msgr2=0x7f160c0c6ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:16:06.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:06.328+0000 7f160be00640 1 --2- 172.21.15.6:0/2008770450 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f160c0c5a60 0x7f160c0c6ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:06.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:06.328+0000 7f160be00640 1 -- 172.21.15.6:0/2008770450 >> v1:172.21.15.190:6789/0 conn(0x7f160c0c5e40 legacy=0x7f160c15bd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:06.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:06.328+0000 7f160be00640 1 -- 172.21.15.6:0/2008770450 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f160c06eab0 con 0x7f160c0c8570 2024-04-24T23:16:06.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:06.328+0000 7f160aa00640 1 -- 172.21.15.6:0/2008770450 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 827107392 0 0) 0x7f160c0cca80 con 0x7f160c0c5e40 2024-04-24T23:16:06.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:06.329+0000 7f160be00640 1 --2- 172.21.15.6:0/2008770450 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f160c0c8570 0x7f160c1524a0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f1604009e50 tx=0x7f160402a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb286a6448240250 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:06.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:06.329+0000 7f160aa00640 1 -- 172.21.15.6:0/2008770450 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1604036070 con 0x7f160c0c8570 2024-04-24T23:16:06.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:06.329+0000 7f160aa00640 1 -- 172.21.15.6:0/2008770450 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1604031600 con 0x7f160c0c8570 2024-04-24T23:16:06.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:06.329+0000 7f160aa00640 1 -- 172.21.15.6:0/2008770450 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16040319d0 con 0x7f160c0c8570 2024-04-24T23:16:06.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:06.330+0000 7f1610a00640 1 -- 172.21.15.6:0/2008770450 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f160c0c8570 msgr2=0x7f160c1524a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:06.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:06.330+0000 7f1610a00640 1 --2- 172.21.15.6:0/2008770450 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f160c0c8570 0x7f160c1524a0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f1604009e50 tx=0x7f160402a710 comp rx=0 tx=0).stop 2024-04-24T23:16:06.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:06.330+0000 7f1610a00640 1 -- 172.21.15.6:0/2008770450 shutdown_connections 2024-04-24T23:16:06.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:06.330+0000 7f1610a00640 1 --2- 172.21.15.6:0/2008770450 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f160c0c8570 0x7f160c1524a0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:06.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:06.330+0000 7f1610a00640 1 --2- 172.21.15.6:0/2008770450 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f160c0c5a60 0x7f160c0c6ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:06.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:06.330+0000 7f1610a00640 1 -- 172.21.15.6:0/2008770450 >> 172.21.15.6:0/2008770450 conn(0x7f160c06cc40 msgr2=0x7f160c06d050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:06.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:06.330+0000 7f1610a00640 1 -- 172.21.15.6:0/2008770450 shutdown_connections 2024-04-24T23:16:06.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:06.330+0000 7f1610a00640 1 -- 172.21.15.6:0/2008770450 wait complete. 2024-04-24T23:16:06.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:06.332+0000 7f1610a00640 1 Processor -- start 2024-04-24T23:16:06.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:06.332+0000 7f1610a00640 1 -- start start 2024-04-24T23:16:06.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:06.333+0000 7f1610a00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f160c0c5a60 0x7f160c0cb1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:06.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:06.333+0000 7f1610a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f160c0c5e40 0x7f160c1f27b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:06.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:06.333+0000 7f1610a00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f160c0c8570 0x7f160c1467c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:06.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:06.333+0000 7f1610a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f160c06f590 con 0x7f160c0c5e40 2024-04-24T23:16:06.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:06.333+0000 7f1610a00640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f160c06f070 con 0x7f160c0c5a60 2024-04-24T23:16:06.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:06.333+0000 7f1610a00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f160c1f1a80 con 0x7f160c0c8570 2024-04-24T23:16:06.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:06.333+0000 7f1603e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f160c0c5e40 0x7f160c1f27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:06.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:06.333+0000 7f160b400640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f160c0c5a60 0x7f160c0cb1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:06.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:06.333+0000 7f1603e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f160c0c5e40 0x7f160c1f27b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:59848/0 (socket says 172.21.15.6:59848) 2024-04-24T23:16:06.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:06.333+0000 7f1603e00640 1 -- 172.21.15.6:0/537282026 learned_addr learned my addr 172.21.15.6:0/537282026 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:16:06.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:06.333+0000 7f160b400640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f160c0c5a60 0x7f160c0cb1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:59618/0 (socket says 172.21.15.6:59618) 2024-04-24T23:16:06.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:06.333+0000 7f160be00640 1 --2- 172.21.15.6:0/537282026 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f160c0c8570 0x7f160c1467c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:06.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:06.334+0000 7f160b400640 1 -- 172.21.15.6:0/537282026 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f160c0c8570 msgr2=0x7f160c1467c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:06.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:06.334+0000 7f160b400640 1 --2- 172.21.15.6:0/537282026 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f160c0c8570 0x7f160c1467c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:06.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:06.334+0000 7f160b400640 1 -- 172.21.15.6:0/537282026 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f160c0c5e40 msgr2=0x7f160c1f27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:06.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:06.334+0000 7f160b400640 1 --2- 172.21.15.6:0/537282026 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f160c0c5e40 0x7f160c1f27b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:06.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:06.334+0000 7f160b400640 1 -- 172.21.15.6:0/537282026 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f160c0cbfd0 con 0x7f160c0c5a60 2024-04-24T23:16:06.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:06.334+0000 7f1603e00640 1 --2- 172.21.15.6:0/537282026 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f160c0c5e40 0x7f160c1f27b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:16:06.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:06.334+0000 7f160b400640 1 --2- 172.21.15.6:0/537282026 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f160c0c5a60 0x7f160c0cb1a0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f15fc00e880 tx=0x7f15fc00ed40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:06.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:06.334+0000 7f1603400640 1 -- 172.21.15.6:0/537282026 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15fc017070 con 0x7f160c0c5a60 2024-04-24T23:16:06.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:06.334+0000 7f1603400640 1 -- 172.21.15.6:0/537282026 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f15fc00c940 con 0x7f160c0c5a60 2024-04-24T23:16:06.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:06.334+0000 7f1603400640 1 -- 172.21.15.6:0/537282026 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15fc00cd30 con 0x7f160c0c5a60 2024-04-24T23:16:06.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:06.334+0000 7f1610a00640 1 -- 172.21.15.6:0/537282026 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f160c0cc270 con 0x7f160c0c5a60 2024-04-24T23:16:06.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:06.334+0000 7f1610a00640 1 -- 172.21.15.6:0/537282026 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f160c1f1fb0 con 0x7f160c0c5a60 2024-04-24T23:16:06.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:06.335+0000 7f1610a00640 1 -- 172.21.15.6:0/537282026 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f160c075b30 con 0x7f160c0c5a60 2024-04-24T23:16:06.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:06.336+0000 7f1603400640 1 -- 172.21.15.6:0/537282026 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f15fc01a070 con 0x7f160c0c5a60 2024-04-24T23:16:06.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:06.337+0000 7f1603400640 1 --2- 172.21.15.6:0/537282026 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f15e806be30 0x7f15e806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:06.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:06.337+0000 7f1603400640 1 -- 172.21.15.6:0/537282026 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f15fc08b0a0 con 0x7f160c0c5a60 2024-04-24T23:16:06.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:06.338+0000 7f1603e00640 1 --2- 172.21.15.6:0/537282026 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f15e806be30 0x7f15e806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:06.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:06.338+0000 7f1603e00640 1 --2- 172.21.15.6:0/537282026 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f15e806be30 0x7f15e806e2f0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f160c159470 tx=0x7f15f4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:06.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:06.342+0000 7f1603400640 1 -- 172.21.15.6:0/537282026 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f15fc0590b0 con 0x7f160c0c5a60 2024-04-24T23:16:06.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:06.587+0000 7f1610a00640 1 -- 172.21.15.6:0/537282026 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f160c06d150 con 0x7f160c0c5a60 2024-04-24T23:16:07.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:07.136+0000 7f1603400640 1 -- 172.21.15.6:0/537282026 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v7) ==== 118+0+0 (secure 0 0 0) 0x7f15fc05d3b0 con 0x7f160c0c5a60 2024-04-24T23:16:07.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:07.139+0000 7f1610a00640 1 -- 172.21.15.6:0/537282026 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f15e806be30 msgr2=0x7f15e806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:07.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:07.139+0000 7f1610a00640 1 --2- 172.21.15.6:0/537282026 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f15e806be30 0x7f15e806e2f0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f160c159470 tx=0x7f15f4009040 comp rx=0 tx=0).stop 2024-04-24T23:16:07.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:07.139+0000 7f1610a00640 1 -- 172.21.15.6:0/537282026 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f160c0c5a60 msgr2=0x7f160c0cb1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:07.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:07.139+0000 7f1610a00640 1 --2- 172.21.15.6:0/537282026 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f160c0c5a60 0x7f160c0cb1a0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f15fc00e880 tx=0x7f15fc00ed40 comp rx=0 tx=0).stop 2024-04-24T23:16:07.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:07.140+0000 7f1610a00640 1 -- 172.21.15.6:0/537282026 shutdown_connections 2024-04-24T23:16:07.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:07.140+0000 7f1610a00640 1 --2- 172.21.15.6:0/537282026 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f15e806be30 0x7f15e806e2f0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:07.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:07.140+0000 7f1610a00640 1 --2- 172.21.15.6:0/537282026 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f160c0c8570 0x7f160c1467c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:07.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:07.140+0000 7f1610a00640 1 --2- 172.21.15.6:0/537282026 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f160c0c5e40 0x7f160c1f27b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:07.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:07.140+0000 7f1610a00640 1 --2- 172.21.15.6:0/537282026 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f160c0c5a60 0x7f160c0cb1a0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:07.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:07.140+0000 7f1610a00640 1 -- 172.21.15.6:0/537282026 >> 172.21.15.6:0/537282026 conn(0x7f160c06cc40 msgr2=0x7f160c14fee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:07.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:07.140+0000 7f1610a00640 1 -- 172.21.15.6:0/537282026 shutdown_connections 2024-04-24T23:16:07.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:07.140+0000 7f1610a00640 1 -- 172.21.15.6:0/537282026 wait complete. 2024-04-24T23:16:07.163 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 5, 'session_timeout': 300} 2024-04-24T23:16:07.163 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 5 2024-04-24T23:16:07.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:07.376+0000 7fac71400640 1 Processor -- start 2024-04-24T23:16:07.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:07.377+0000 7fac71400640 1 -- start start 2024-04-24T23:16:07.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:07.378+0000 7fac71400640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fac6c06cd50 0x7fac6c15e090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:07.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:07.378+0000 7fac71400640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fac6c0b35f0 0x7fac6c15e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:07.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:07.378+0000 7fac71400640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fac6c0b46f0 con 0x7fac6c154f60 2024-04-24T23:16:07.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:07.378+0000 7fac71400640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7fac6c0b49d0 con 0x7fac6c06cd50 2024-04-24T23:16:07.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:07.378+0000 7fac71400640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fac6c0b4cb0 con 0x7fac6c0b35f0 2024-04-24T23:16:07.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:07.378+0000 7fac6be00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fac6c154f60 0x7fac6c06c640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:43376/0 (socket says 172.21.15.6:43376) 2024-04-24T23:16:07.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:07.378+0000 7fac6be00640 1 -- 172.21.15.6:0/2272228870 learned_addr learned my addr 172.21.15.6:0/2272228870 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:16:07.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:07.379+0000 7fac6b400640 1 --2- 172.21.15.6:0/2272228870 >> v2:172.21.15.190:3300/0 conn(0x7fac6c06cd50 0x7fac6c15e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:07.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:07.379+0000 7fac70a00640 1 --2- 172.21.15.6:0/2272228870 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fac6c0b35f0 0x7fac6c15e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:07.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:07.379+0000 7fac6aa00640 1 -- 172.21.15.6:0/2272228870 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3832498122 0 0) 0x7fac6c0b46f0 con 0x7fac6c154f60 2024-04-24T23:16:07.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:07.379+0000 7fac6aa00640 1 -- 172.21.15.6:0/2272228870 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fac50003520 con 0x7fac6c154f60 2024-04-24T23:16:07.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:07.379+0000 7fac6b400640 1 -- 172.21.15.6:0/2272228870 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fac6c0b35f0 msgr2=0x7fac6c15e5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:16:07.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:07.379+0000 7fac6b400640 1 --2- 172.21.15.6:0/2272228870 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fac6c0b35f0 0x7fac6c15e5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:07.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:07.379+0000 7fac6b400640 1 -- 172.21.15.6:0/2272228870 >> v1:172.21.15.6:6789/0 conn(0x7fac6c154f60 legacy=0x7fac6c06c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:07.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:07.379+0000 7fac6b400640 1 -- 172.21.15.6:0/2272228870 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac6c0b4f90 con 0x7fac6c06cd50 2024-04-24T23:16:07.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:07.380+0000 7fac6b400640 1 --2- 172.21.15.6:0/2272228870 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fac6c06cd50 0x7fac6c15e090 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fac540097a0 tx=0x7fac5402a210 comp rx=0 tx=0).ready entity=mon.1 client_cookie=84be9c00e8efad59 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:07.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:07.380+0000 7fac6aa00640 1 -- 172.21.15.6:0/2272228870 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac54037070 con 0x7fac6c06cd50 2024-04-24T23:16:07.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:07.381+0000 7fac6aa00640 1 -- 172.21.15.6:0/2272228870 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac54032440 con 0x7fac6c06cd50 2024-04-24T23:16:07.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:07.381+0000 7fac6aa00640 1 -- 172.21.15.6:0/2272228870 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac540327f0 con 0x7fac6c06cd50 2024-04-24T23:16:07.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:07.381+0000 7fac71400640 1 -- 172.21.15.6:0/2272228870 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fac6c06cd50 msgr2=0x7fac6c15e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:07.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:07.381+0000 7fac71400640 1 --2- 172.21.15.6:0/2272228870 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fac6c06cd50 0x7fac6c15e090 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fac540097a0 tx=0x7fac5402a210 comp rx=0 tx=0).stop 2024-04-24T23:16:07.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:07.381+0000 7fac71400640 1 -- 172.21.15.6:0/2272228870 shutdown_connections 2024-04-24T23:16:07.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:07.381+0000 7fac71400640 1 --2- 172.21.15.6:0/2272228870 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fac6c0b35f0 0x7fac6c15e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:07.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:07.381+0000 7fac71400640 1 --2- 172.21.15.6:0/2272228870 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fac6c06cd50 0x7fac6c15e090 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:07.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:07.381+0000 7fac71400640 1 -- 172.21.15.6:0/2272228870 >> 172.21.15.6:0/2272228870 conn(0x7fac6c0674d0 msgr2=0x7fac6c0d0320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:07.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:07.381+0000 7fac71400640 1 -- 172.21.15.6:0/2272228870 shutdown_connections 2024-04-24T23:16:07.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:07.382+0000 7fac71400640 1 -- 172.21.15.6:0/2272228870 wait complete. 2024-04-24T23:16:07.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:07.384+0000 7fac71400640 1 Processor -- start 2024-04-24T23:16:07.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:07.384+0000 7fac71400640 1 -- start start 2024-04-24T23:16:07.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:07.385+0000 7fac71400640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fac6c06cd50 0x7fac6c14ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:07.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:07.385+0000 7fac71400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fac6c0b35f0 0x7fac6c0c2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:07.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:07.385+0000 7fac71400640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fac6c154f60 0x7fac6c0c2ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:07.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:07.385+0000 7fac71400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fac6c0b3240 con 0x7fac6c0b35f0 2024-04-24T23:16:07.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:07.385+0000 7fac71400640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fac6c06c120 con 0x7fac6c06cd50 2024-04-24T23:16:07.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:07.385+0000 7fac71400640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fac6c0b33c0 con 0x7fac6c154f60 2024-04-24T23:16:07.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:07.386+0000 7fac6be00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fac6c06cd50 0x7fac6c14ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:07.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:07.386+0000 7fac70a00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fac6c154f60 0x7fac6c0c2ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:07.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:07.386+0000 7fac6be00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fac6c06cd50 0x7fac6c14ffb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:59626/0 (socket says 172.21.15.6:59626) 2024-04-24T23:16:07.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:07.386+0000 7fac6be00640 1 -- 172.21.15.6:0/1945610799 learned_addr learned my addr 172.21.15.6:0/1945610799 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:16:07.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:07.386+0000 7fac6b400640 1 --2- 172.21.15.6:0/1945610799 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fac6c0b35f0 0x7fac6c0c2ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:07.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:07.386+0000 7fac6be00640 1 -- 172.21.15.6:0/1945610799 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fac6c154f60 msgr2=0x7fac6c0c2ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:07.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:07.386+0000 7fac6be00640 1 --2- 172.21.15.6:0/1945610799 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fac6c154f60 0x7fac6c0c2ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:07.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:07.386+0000 7fac6be00640 1 -- 172.21.15.6:0/1945610799 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fac6c0b35f0 msgr2=0x7fac6c0c2ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:07.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:07.386+0000 7fac6be00640 1 --2- 172.21.15.6:0/1945610799 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fac6c0b35f0 0x7fac6c0c2ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:07.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:07.386+0000 7fac6be00640 1 -- 172.21.15.6:0/1945610799 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac6c1f1420 con 0x7fac6c06cd50 2024-04-24T23:16:07.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:07.387+0000 7fac6be00640 1 --2- 172.21.15.6:0/1945610799 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fac6c06cd50 0x7fac6c14ffb0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fac5c00caa0 tx=0x7fac5c00cf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:07.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:07.387+0000 7fac63e00640 1 -- 172.21.15.6:0/1945610799 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac5c017070 con 0x7fac6c06cd50 2024-04-24T23:16:07.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:07.387+0000 7fac63e00640 1 -- 172.21.15.6:0/1945610799 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac5c00ac00 con 0x7fac6c06cd50 2024-04-24T23:16:07.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:07.387+0000 7fac63e00640 1 -- 172.21.15.6:0/1945610799 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac5c012400 con 0x7fac6c06cd50 2024-04-24T23:16:07.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:07.387+0000 7fac71400640 1 -- 172.21.15.6:0/1945610799 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac6c14b940 con 0x7fac6c06cd50 2024-04-24T23:16:07.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:07.387+0000 7fac71400640 1 -- 172.21.15.6:0/1945610799 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac6c14bec0 con 0x7fac6c06cd50 2024-04-24T23:16:07.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:07.388+0000 7fac71400640 1 -- 172.21.15.6:0/1945610799 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fac6c075b30 con 0x7fac6c06cd50 2024-04-24T23:16:07.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:07.389+0000 7fac63e00640 1 -- 172.21.15.6:0/1945610799 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fac5c01a070 con 0x7fac6c06cd50 2024-04-24T23:16:07.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:07.390+0000 7fac63e00640 1 --2- 172.21.15.6:0/1945610799 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fac4806be60 0x7fac4806e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:07.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:07.390+0000 7fac63e00640 1 -- 172.21.15.6:0/1945610799 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fac5c08af50 con 0x7fac6c06cd50 2024-04-24T23:16:07.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:07.391+0000 7fac6b400640 1 --2- 172.21.15.6:0/1945610799 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fac4806be60 0x7fac4806e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:07.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:07.391+0000 7fac6b400640 1 --2- 172.21.15.6:0/1945610799 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fac4806be60 0x7fac4806e320 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fac54034000 tx=0x7fac5402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:07.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:07.395+0000 7fac63e00640 1 -- 172.21.15.6:0/1945610799 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fac5c058ed0 con 0x7fac6c06cd50 2024-04-24T23:16:07.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:07.629+0000 7fac71400640 1 -- 172.21.15.6:0/1945610799 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"} v 0) -- 0x7fac6c0b2a60 con 0x7fac6c06cd50 2024-04-24T23:16:08.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:08.138+0000 7fac63e00640 1 -- 172.21.15.6:0/1945610799 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]=0 v8) ==== 105+0+0 (secure 0 0 0) 0x7fac5c05d1d0 con 0x7fac6c06cd50 2024-04-24T23:16:08.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:08.141+0000 7fac71400640 1 -- 172.21.15.6:0/1945610799 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fac4806be60 msgr2=0x7fac4806e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:08.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:08.141+0000 7fac71400640 1 --2- 172.21.15.6:0/1945610799 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fac4806be60 0x7fac4806e320 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fac54034000 tx=0x7fac5402e000 comp rx=0 tx=0).stop 2024-04-24T23:16:08.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:08.141+0000 7fac71400640 1 -- 172.21.15.6:0/1945610799 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fac6c06cd50 msgr2=0x7fac6c14ffb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:08.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:08.141+0000 7fac71400640 1 --2- 172.21.15.6:0/1945610799 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fac6c06cd50 0x7fac6c14ffb0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fac5c00caa0 tx=0x7fac5c00cf60 comp rx=0 tx=0).stop 2024-04-24T23:16:08.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:08.142+0000 7fac71400640 1 -- 172.21.15.6:0/1945610799 shutdown_connections 2024-04-24T23:16:08.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:08.142+0000 7fac71400640 1 --2- 172.21.15.6:0/1945610799 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fac4806be60 0x7fac4806e320 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:08.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:08.142+0000 7fac71400640 1 --2- 172.21.15.6:0/1945610799 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fac6c154f60 0x7fac6c0c2ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:08.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:08.142+0000 7fac71400640 1 --2- 172.21.15.6:0/1945610799 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fac6c0b35f0 0x7fac6c0c2ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:08.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:08.142+0000 7fac71400640 1 --2- 172.21.15.6:0/1945610799 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fac6c06cd50 0x7fac6c14ffb0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:08.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:08.142+0000 7fac71400640 1 -- 172.21.15.6:0/1945610799 >> 172.21.15.6:0/1945610799 conn(0x7fac6c0674d0 msgr2=0x7fac6c0caba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:08.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:08.142+0000 7fac71400640 1 -- 172.21.15.6:0/1945610799 shutdown_connections 2024-04-24T23:16:08.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:08.142+0000 7fac71400640 1 -- 172.21.15.6:0/1945610799 wait complete. 2024-04-24T23:16:08.166 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-04-24T23:16:08.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:08.375+0000 7f7947000640 1 Processor -- start 2024-04-24T23:16:08.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:08.376+0000 7f7947000640 1 -- start start 2024-04-24T23:16:08.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:08.377+0000 7f7947000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f794006eb80 0x7f79400c9330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:08.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:08.377+0000 7f7947000640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f794015f240 con 0x7f794015ed90 2024-04-24T23:16:08.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:08.377+0000 7f7947000640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f794015f520 con 0x7f794006ef60 2024-04-24T23:16:08.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:08.377+0000 7f7947000640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f79400c8bd0 con 0x7f794006eb80 2024-04-24T23:16:08.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:08.377+0000 7f7945c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f794006eb80 0x7f79400c9330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:08.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:08.378+0000 7f7945c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f794006eb80 0x7f79400c9330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:59866/0 (socket says 172.21.15.6:59866) 2024-04-24T23:16:08.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:08.378+0000 7f7945c00640 1 -- 172.21.15.6:0/2633617723 learned_addr learned my addr 172.21.15.6:0/2633617723 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:16:08.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:08.378+0000 7f7945200640 1 --1- 172.21.15.6:0/2633617723 >> v1:172.21.15.6:6789/0 conn(0x7f794015ed90 0x7f7940162a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:43392/0 (socket says 172.21.15.6:43392) 2024-04-24T23:16:08.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:08.378+0000 7f7945c00640 1 -- 172.21.15.6:0/2633617723 >> v1:172.21.15.190:6789/0 conn(0x7f794006ef60 legacy=0x7f7940158b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:08.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:08.378+0000 7f7945c00640 1 -- 172.21.15.6:0/2633617723 >> v1:172.21.15.6:6789/0 conn(0x7f794015ed90 legacy=0x7f7940162a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:08.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:08.378+0000 7f793fe00640 1 -- 172.21.15.6:0/2633617723 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 461592255 0 0) 0x7f794015f240 con 0x7f794015ed90 2024-04-24T23:16:08.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:08.378+0000 7f7945c00640 1 -- 172.21.15.6:0/2633617723 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f794015f800 con 0x7f794006eb80 2024-04-24T23:16:08.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:08.379+0000 7f7945c00640 1 --2- 172.21.15.6:0/2633617723 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f794006eb80 0x7f79400c9330 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f7934009870 tx=0x7f793402a3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c83683ffc5514436 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:08.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:08.380+0000 7f793fe00640 1 -- 172.21.15.6:0/2633617723 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7934037070 con 0x7f794006eb80 2024-04-24T23:16:08.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:08.380+0000 7f793fe00640 1 -- 172.21.15.6:0/2633617723 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7934032640 con 0x7f794006eb80 2024-04-24T23:16:08.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:08.380+0000 7f793fe00640 1 -- 172.21.15.6:0/2633617723 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7934032a10 con 0x7f794006eb80 2024-04-24T23:16:08.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:08.380+0000 7f7947000640 1 -- 172.21.15.6:0/2633617723 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f794006eb80 msgr2=0x7f79400c9330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:08.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:08.380+0000 7f7947000640 1 --2- 172.21.15.6:0/2633617723 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f794006eb80 0x7f79400c9330 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f7934009870 tx=0x7f793402a3c0 comp rx=0 tx=0).stop 2024-04-24T23:16:08.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:08.380+0000 7f7947000640 1 -- 172.21.15.6:0/2633617723 shutdown_connections 2024-04-24T23:16:08.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:08.380+0000 7f7947000640 1 --2- 172.21.15.6:0/2633617723 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f794006eb80 0x7f79400c9330 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:08.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:08.380+0000 7f7947000640 1 -- 172.21.15.6:0/2633617723 >> 172.21.15.6:0/2633617723 conn(0x7f794006cc40 msgr2=0x7f794006d050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:08.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:08.380+0000 7f7947000640 1 -- 172.21.15.6:0/2633617723 shutdown_connections 2024-04-24T23:16:08.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:08.381+0000 7f7947000640 1 -- 172.21.15.6:0/2633617723 wait complete. 2024-04-24T23:16:08.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:08.382+0000 7f7947000640 1 Processor -- start 2024-04-24T23:16:08.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:08.382+0000 7f7947000640 1 -- start start 2024-04-24T23:16:08.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:08.383+0000 7f7947000640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f794006eb80 0x7f79400cba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:08.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:08.383+0000 7f7947000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f794006ef60 0x7f7940156670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:08.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:08.383+0000 7f7947000640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f794015ed90 0x7f79401f44d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:08.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:08.383+0000 7f7947000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f7940163150 con 0x7f794006ef60 2024-04-24T23:16:08.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:08.383+0000 7f7947000640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f7940162520 con 0x7f794006eb80 2024-04-24T23:16:08.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:08.383+0000 7f7947000640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f79401632d0 con 0x7f794015ed90 2024-04-24T23:16:08.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:08.383+0000 7f7945200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f794006ef60 0x7f7940156670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:08.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:08.384+0000 7f7945c00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f794006eb80 0x7f79400cba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:08.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:08.384+0000 7f7946600640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f794015ed90 0x7f79401f44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:08.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:08.384+0000 7f7945200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f794006ef60 0x7f7940156670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:59870/0 (socket says 172.21.15.6:59870) 2024-04-24T23:16:08.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:08.384+0000 7f7945200640 1 -- 172.21.15.6:0/1845008011 learned_addr learned my addr 172.21.15.6:0/1845008011 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:16:08.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:08.384+0000 7f7945c00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f794006eb80 0x7f79400cba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:59628/0 (socket says 172.21.15.6:59628) 2024-04-24T23:16:08.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:08.384+0000 7f7945200640 1 -- 172.21.15.6:0/1845008011 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f794015ed90 msgr2=0x7f79401f44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:08.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:08.384+0000 7f7945200640 1 --2- 172.21.15.6:0/1845008011 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f794015ed90 0x7f79401f44d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:08.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:08.384+0000 7f7945200640 1 -- 172.21.15.6:0/1845008011 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f794006eb80 msgr2=0x7f79400cba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:08.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:08.384+0000 7f7945200640 1 --2- 172.21.15.6:0/1845008011 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f794006eb80 0x7f79400cba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:08.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:08.384+0000 7f7945200640 1 -- 172.21.15.6:0/1845008011 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79401534a0 con 0x7f794006ef60 2024-04-24T23:16:08.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:08.384+0000 7f7945c00640 1 --2- 172.21.15.6:0/1845008011 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f794006eb80 0x7f79400cba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:16:08.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:08.384+0000 7f7946600640 1 --2- 172.21.15.6:0/1845008011 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f794015ed90 0x7f79401f44d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:16:08.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:08.384+0000 7f7945200640 1 --2- 172.21.15.6:0/1845008011 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f794006ef60 0x7f7940156670 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f792c00c3f0 tx=0x7f792c00c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:08.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:08.385+0000 7f793d600640 1 -- 172.21.15.6:0/1845008011 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f792c0074e0 con 0x7f794006ef60 2024-04-24T23:16:08.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:08.385+0000 7f793d600640 1 -- 172.21.15.6:0/1845008011 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f792c0047e0 con 0x7f794006ef60 2024-04-24T23:16:08.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:08.385+0000 7f793d600640 1 -- 172.21.15.6:0/1845008011 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f792c004bd0 con 0x7f794006ef60 2024-04-24T23:16:08.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:08.385+0000 7f7947000640 1 -- 172.21.15.6:0/1845008011 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79401f7be0 con 0x7f794006ef60 2024-04-24T23:16:08.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:08.385+0000 7f7947000640 1 -- 172.21.15.6:0/1845008011 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79401f81a0 con 0x7f794006ef60 2024-04-24T23:16:08.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:08.386+0000 7f7947000640 1 -- 172.21.15.6:0/1845008011 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7940075b30 con 0x7f794006ef60 2024-04-24T23:16:08.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:08.387+0000 7f793d600640 1 -- 172.21.15.6:0/1845008011 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f792c011070 con 0x7f794006ef60 2024-04-24T23:16:08.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:08.388+0000 7f793d600640 1 --2- 172.21.15.6:0/1845008011 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f792006be30 0x7f792006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:08.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:08.388+0000 7f793d600640 1 -- 172.21.15.6:0/1845008011 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f792c08ad00 con 0x7f794006ef60 2024-04-24T23:16:08.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:08.388+0000 7f7945c00640 1 --2- 172.21.15.6:0/1845008011 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f792006be30 0x7f792006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:08.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:08.389+0000 7f7945c00640 1 --2- 172.21.15.6:0/1845008011 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f792006be30 0x7f792006e2f0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f7934034000 tx=0x7f793402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:08.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:08.393+0000 7f793d600640 1 -- 172.21.15.6:0/1845008011 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f792c058c80 con 0x7f794006ef60 2024-04-24T23:16:08.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:08.632+0000 7f7947000640 1 -- 172.21.15.6:0/1845008011 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f79400c66e0 con 0x7f794006ef60 2024-04-24T23:16:09.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:09.154+0000 7f793d600640 1 -- 172.21.15.6:0/1845008011 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v10) ==== 122+0+0 (secure 0 0 0) 0x7f792c05cf80 con 0x7f794006ef60 2024-04-24T23:16:09.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:09.158+0000 7f7947000640 1 -- 172.21.15.6:0/1845008011 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f792006be30 msgr2=0x7f792006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:09.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:09.158+0000 7f7947000640 1 --2- 172.21.15.6:0/1845008011 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f792006be30 0x7f792006e2f0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f7934034000 tx=0x7f793402e000 comp rx=0 tx=0).stop 2024-04-24T23:16:09.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:09.158+0000 7f7947000640 1 -- 172.21.15.6:0/1845008011 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f794006ef60 msgr2=0x7f7940156670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:09.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:09.158+0000 7f7947000640 1 --2- 172.21.15.6:0/1845008011 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f794006ef60 0x7f7940156670 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f792c00c3f0 tx=0x7f792c00c8b0 comp rx=0 tx=0).stop 2024-04-24T23:16:09.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:09.158+0000 7f7947000640 1 -- 172.21.15.6:0/1845008011 shutdown_connections 2024-04-24T23:16:09.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:09.158+0000 7f7947000640 1 --2- 172.21.15.6:0/1845008011 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f792006be30 0x7f792006e2f0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:09.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:09.158+0000 7f7947000640 1 --2- 172.21.15.6:0/1845008011 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f794015ed90 0x7f79401f44d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:09.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:09.158+0000 7f7947000640 1 --2- 172.21.15.6:0/1845008011 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f794006ef60 0x7f7940156670 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:09.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:09.158+0000 7f7947000640 1 --2- 172.21.15.6:0/1845008011 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f794006eb80 0x7f79400cba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:09.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:09.159+0000 7f7947000640 1 -- 172.21.15.6:0/1845008011 >> 172.21.15.6:0/1845008011 conn(0x7f794006cc40 msgr2=0x7f7940162a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:09.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:09.159+0000 7f7947000640 1 -- 172.21.15.6:0/1845008011 shutdown_connections 2024-04-24T23:16:09.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:09.159+0000 7f7947000640 1 -- 172.21.15.6:0/1845008011 wait complete. 2024-04-24T23:16:09.180 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2024-04-24T23:16:09.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:09.389+0000 7fb60b200640 1 Processor -- start 2024-04-24T23:16:09.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:09.390+0000 7fb60b200640 1 -- start start 2024-04-24T23:16:09.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:09.391+0000 7fb60b200640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb60c154f60 0x7fb60c06c640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:09.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:09.391+0000 7fb60b200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fb60c06cb80 0x7fb60c15b230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:09.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:09.391+0000 7fb60b200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb60c06cfd0 con 0x7fb60c0b35f0 2024-04-24T23:16:09.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:09.391+0000 7fb60b200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7fb60c0b46f0 con 0x7fb60c06cb80 2024-04-24T23:16:09.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:09.391+0000 7fb60b200640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fb60c0b49d0 con 0x7fb60c154f60 2024-04-24T23:16:09.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:09.392+0000 7fb609e00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb60c154f60 0x7fb60c06c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:09.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:09.392+0000 7fb60a800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fb60c06cb80 0x7fb60c15b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:09.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:09.392+0000 7fb609e00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb60c154f60 0x7fb60c06c640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.6:53074/0 (socket says 172.21.15.6:53074) 2024-04-24T23:16:09.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:09.392+0000 7fb60a800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fb60c06cb80 0x7fb60c15b230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:59886/0 (socket says 172.21.15.6:59886) 2024-04-24T23:16:09.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:09.392+0000 7fb609e00640 1 -- 172.21.15.6:0/2825539049 learned_addr learned my addr 172.21.15.6:0/2825539049 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:16:09.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:09.392+0000 7fb60a800640 1 -- 172.21.15.6:0/2825539049 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb60c154f60 msgr2=0x7fb60c06c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:16:09.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:09.392+0000 7fb60a800640 1 --2- 172.21.15.6:0/2825539049 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb60c154f60 0x7fb60c06c640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:09.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:09.392+0000 7fb60a800640 1 -- 172.21.15.6:0/2825539049 >> v1:172.21.15.6:6789/0 conn(0x7fb60c0b35f0 legacy=0x7fb60c15ab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:09.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:09.392+0000 7fb60a800640 1 -- 172.21.15.6:0/2825539049 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb60c0b4cb0 con 0x7fb60c06cb80 2024-04-24T23:16:09.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:09.392+0000 7fb608a00640 1 -- 172.21.15.6:0/2825539049 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4170592218 0 0) 0x7fb60c06cfd0 con 0x7fb60c0b35f0 2024-04-24T23:16:09.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:09.394+0000 7fb60a800640 1 --2- 172.21.15.6:0/2825539049 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb60c06cb80 0x7fb60c15b230 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fb604009d90 tx=0x7fb60402a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d437e6dff353577 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:09.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:09.394+0000 7fb608a00640 1 -- 172.21.15.6:0/2825539049 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb604007740 con 0x7fb60c06cb80 2024-04-24T23:16:09.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:09.394+0000 7fb608a00640 1 -- 172.21.15.6:0/2825539049 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb6040078e0 con 0x7fb60c06cb80 2024-04-24T23:16:09.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:09.394+0000 7fb608a00640 1 -- 172.21.15.6:0/2825539049 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb604007cb0 con 0x7fb60c06cb80 2024-04-24T23:16:09.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:09.394+0000 7fb60b200640 1 -- 172.21.15.6:0/2825539049 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb60c06cb80 msgr2=0x7fb60c15b230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:09.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:09.394+0000 7fb60b200640 1 --2- 172.21.15.6:0/2825539049 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb60c06cb80 0x7fb60c15b230 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fb604009d90 tx=0x7fb60402a6f0 comp rx=0 tx=0).stop 2024-04-24T23:16:09.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:09.394+0000 7fb60b200640 1 -- 172.21.15.6:0/2825539049 shutdown_connections 2024-04-24T23:16:09.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:09.394+0000 7fb60b200640 1 --2- 172.21.15.6:0/2825539049 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb60c06cb80 0x7fb60c15b230 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:09.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:09.394+0000 7fb60b200640 1 --2- 172.21.15.6:0/2825539049 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb60c154f60 0x7fb60c06c640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:09.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:09.394+0000 7fb60b200640 1 -- 172.21.15.6:0/2825539049 >> 172.21.15.6:0/2825539049 conn(0x7fb60c0674d0 msgr2=0x7fb60c0d0320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:09.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:09.395+0000 7fb60b200640 1 -- 172.21.15.6:0/2825539049 shutdown_connections 2024-04-24T23:16:09.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:09.395+0000 7fb60b200640 1 -- 172.21.15.6:0/2825539049 wait complete. 2024-04-24T23:16:09.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:09.396+0000 7fb60b200640 1 Processor -- start 2024-04-24T23:16:09.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:09.396+0000 7fb60b200640 1 -- start start 2024-04-24T23:16:09.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:09.397+0000 7fb60b200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb60c06cb80 0x7fb60c0cae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:09.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:09.397+0000 7fb60b200640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb60c0b35f0 0x7fb60c147e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:09.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:09.397+0000 7fb60b200640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb60c154f60 0x7fb60c148380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:09.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:09.397+0000 7fb60b200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fb60c06c120 con 0x7fb60c06cb80 2024-04-24T23:16:09.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:09.397+0000 7fb60b200640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fb60c06be40 con 0x7fb60c0b35f0 2024-04-24T23:16:09.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:09.397+0000 7fb60b200640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fb60c0b3240 con 0x7fb60c154f60 2024-04-24T23:16:09.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:09.397+0000 7fb609e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb60c06cb80 0x7fb60c0cae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:09.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:09.398+0000 7fb609e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb60c06cb80 0x7fb60c0cae10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:59892/0 (socket says 172.21.15.6:59892) 2024-04-24T23:16:09.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:09.398+0000 7fb609e00640 1 -- 172.21.15.6:0/3064521258 learned_addr learned my addr 172.21.15.6:0/3064521258 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:16:09.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:09.398+0000 7fb609400640 1 --2- 172.21.15.6:0/3064521258 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb60c0b35f0 0x7fb60c147e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:09.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:09.398+0000 7fb60a800640 1 --2- 172.21.15.6:0/3064521258 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb60c154f60 0x7fb60c148380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:09.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:09.398+0000 7fb609e00640 1 -- 172.21.15.6:0/3064521258 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb60c154f60 msgr2=0x7fb60c148380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:09.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:09.398+0000 7fb609e00640 1 --2- 172.21.15.6:0/3064521258 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb60c154f60 0x7fb60c148380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:09.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:09.398+0000 7fb609e00640 1 -- 172.21.15.6:0/3064521258 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb60c0b35f0 msgr2=0x7fb60c147e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:09.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:09.398+0000 7fb609e00640 1 --2- 172.21.15.6:0/3064521258 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb60c0b35f0 0x7fb60c147e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:09.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:09.398+0000 7fb609e00640 1 -- 172.21.15.6:0/3064521258 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb60c14e860 con 0x7fb60c06cb80 2024-04-24T23:16:09.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:09.398+0000 7fb609e00640 1 --2- 172.21.15.6:0/3064521258 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb60c06cb80 0x7fb60c0cae10 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fb5fc00e8b0 tx=0x7fb5fc00ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:09.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:09.399+0000 7fb602000640 1 -- 172.21.15.6:0/3064521258 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5fc017070 con 0x7fb60c06cb80 2024-04-24T23:16:09.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:09.399+0000 7fb602000640 1 -- 172.21.15.6:0/3064521258 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb5fc00c940 con 0x7fb60c06cb80 2024-04-24T23:16:09.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:09.399+0000 7fb602000640 1 -- 172.21.15.6:0/3064521258 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5fc00cd10 con 0x7fb60c06cb80 2024-04-24T23:16:09.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:09.399+0000 7fb60b200640 1 -- 172.21.15.6:0/3064521258 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb60c14eb40 con 0x7fb60c06cb80 2024-04-24T23:16:09.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:09.399+0000 7fb60b200640 1 -- 172.21.15.6:0/3064521258 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb60c14f100 con 0x7fb60c06cb80 2024-04-24T23:16:09.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:09.400+0000 7fb60b200640 1 -- 172.21.15.6:0/3064521258 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb60c075b30 con 0x7fb60c06cb80 2024-04-24T23:16:09.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:09.401+0000 7fb602000640 1 -- 172.21.15.6:0/3064521258 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fb5fc01a070 con 0x7fb60c06cb80 2024-04-24T23:16:09.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:09.402+0000 7fb602000640 1 --2- 172.21.15.6:0/3064521258 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fb5e806be30 0x7fb5e806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:09.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:09.402+0000 7fb602000640 1 -- 172.21.15.6:0/3064521258 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fb5fc08b0e0 con 0x7fb60c06cb80 2024-04-24T23:16:09.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:09.402+0000 7fb609400640 1 --2- 172.21.15.6:0/3064521258 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fb5e806be30 0x7fb5e806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:09.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:09.403+0000 7fb609400640 1 --2- 172.21.15.6:0/3064521258 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fb5e806be30 0x7fb5e806e2f0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fb60c147ae0 tx=0x7fb5f4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:09.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:09.406+0000 7fb602000640 1 -- 172.21.15.6:0/3064521258 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fb5fc059060 con 0x7fb60c06cb80 2024-04-24T23:16:09.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:09.651+0000 7fb60b200640 1 -- 172.21.15.6:0/3064521258 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) -- 0x7fb60c0b2a60 con 0x7fb60c06cb80 2024-04-24T23:16:10.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.165+0000 7fb602000640 1 -- 172.21.15.6:0/3064521258 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v11) ==== 115+0+0 (secure 0 0 0) 0x7fb5fc05d360 con 0x7fb60c06cb80 2024-04-24T23:16:10.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.169+0000 7fb60b200640 1 -- 172.21.15.6:0/3064521258 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fb5e806be30 msgr2=0x7fb5e806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:10.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.169+0000 7fb60b200640 1 --2- 172.21.15.6:0/3064521258 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fb5e806be30 0x7fb5e806e2f0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fb60c147ae0 tx=0x7fb5f4009040 comp rx=0 tx=0).stop 2024-04-24T23:16:10.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.169+0000 7fb60b200640 1 -- 172.21.15.6:0/3064521258 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb60c06cb80 msgr2=0x7fb60c0cae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:10.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.169+0000 7fb60b200640 1 --2- 172.21.15.6:0/3064521258 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb60c06cb80 0x7fb60c0cae10 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fb5fc00e8b0 tx=0x7fb5fc00ed70 comp rx=0 tx=0).stop 2024-04-24T23:16:10.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.169+0000 7fb60b200640 1 -- 172.21.15.6:0/3064521258 shutdown_connections 2024-04-24T23:16:10.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.169+0000 7fb60b200640 1 --2- 172.21.15.6:0/3064521258 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fb5e806be30 0x7fb5e806e2f0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:10.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.169+0000 7fb60b200640 1 --2- 172.21.15.6:0/3064521258 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb60c154f60 0x7fb60c148380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:10.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.169+0000 7fb60b200640 1 --2- 172.21.15.6:0/3064521258 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb60c0b35f0 0x7fb60c147e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:10.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.169+0000 7fb60b200640 1 --2- 172.21.15.6:0/3064521258 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb60c06cb80 0x7fb60c0cae10 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:10.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.169+0000 7fb60b200640 1 -- 172.21.15.6:0/3064521258 >> 172.21.15.6:0/3064521258 conn(0x7fb60c0674d0 msgr2=0x7fb60c0ced60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:10.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.169+0000 7fb60b200640 1 -- 172.21.15.6:0/3064521258 shutdown_connections 2024-04-24T23:16:10.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.169+0000 7fb60b200640 1 -- 172.21.15.6:0/3064521258 wait complete. 2024-04-24T23:16:10.193 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-04-24T23:16:10.193 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:16:10.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.402+0000 7f1388c00640 1 Processor -- start 2024-04-24T23:16:10.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.402+0000 7f1388c00640 1 -- start start 2024-04-24T23:16:10.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.403+0000 7f1388c00640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f1384152990 0x7f1384071fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:10.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.403+0000 7f1388c00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1384070590 con 0x7f13840c6a10 2024-04-24T23:16:10.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.403+0000 7f1388c00640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13841587f0 con 0x7f1384159400 2024-04-24T23:16:10.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.403+0000 7f1388c00640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f13840724e0 con 0x7f1384152990 2024-04-24T23:16:10.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.404+0000 7f1382a00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f13840c6a10 0x7f1384158cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:43402/0 (socket says 172.21.15.6:43402) 2024-04-24T23:16:10.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.404+0000 7f1382a00640 1 -- 172.21.15.6:0/1938342275 learned_addr learned my addr 172.21.15.6:0/1938342275 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:16:10.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.404+0000 7f1383400640 1 --2- 172.21.15.6:0/1938342275 >> v2:172.21.15.190:3300/0 conn(0x7f1384152990 0x7f1384071fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:10.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.404+0000 7f1382000640 1 -- 172.21.15.6:0/1938342275 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2193048495 0 0) 0x7f13841587f0 con 0x7f1384159400 2024-04-24T23:16:10.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.404+0000 7f1382000640 1 -- 172.21.15.6:0/1938342275 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1368003520 con 0x7f1384159400 2024-04-24T23:16:10.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.404+0000 7f1382000640 1 -- 172.21.15.6:0/1938342275 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1503914622 0 0) 0x7f1384070590 con 0x7f13840c6a10 2024-04-24T23:16:10.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.404+0000 7f1382000640 1 -- 172.21.15.6:0/1938342275 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f13841587f0 con 0x7f13840c6a10 2024-04-24T23:16:10.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.404+0000 7f1383400640 1 -- 172.21.15.6:0/1938342275 >> v1:172.21.15.190:6789/0 conn(0x7f1384159400 legacy=0x7f1384163170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:10.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.404+0000 7f1383400640 1 -- 172.21.15.6:0/1938342275 >> v1:172.21.15.6:6789/0 conn(0x7f13840c6a10 legacy=0x7f1384158cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:10.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.404+0000 7f1383400640 1 -- 172.21.15.6:0/1938342275 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1384070830 con 0x7f1384152990 2024-04-24T23:16:10.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.406+0000 7f1383400640 1 --2- 172.21.15.6:0/1938342275 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1384152990 0x7f1384071fa0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f1374009d90 tx=0x7f137402a780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ab492e0df5bb36ef server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:10.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.406+0000 7f1382000640 1 -- 172.21.15.6:0/1938342275 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1374037070 con 0x7f1384152990 2024-04-24T23:16:10.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.406+0000 7f1382000640 1 -- 172.21.15.6:0/1938342275 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1374007740 con 0x7f1384152990 2024-04-24T23:16:10.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.406+0000 7f1382000640 1 -- 172.21.15.6:0/1938342275 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1374007af0 con 0x7f1384152990 2024-04-24T23:16:10.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.406+0000 7f1388c00640 1 -- 172.21.15.6:0/1938342275 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1384152990 msgr2=0x7f1384071fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:10.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.406+0000 7f1388c00640 1 --2- 172.21.15.6:0/1938342275 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1384152990 0x7f1384071fa0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f1374009d90 tx=0x7f137402a780 comp rx=0 tx=0).stop 2024-04-24T23:16:10.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.407+0000 7f1388c00640 1 -- 172.21.15.6:0/1938342275 shutdown_connections 2024-04-24T23:16:10.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.407+0000 7f1388c00640 1 --2- 172.21.15.6:0/1938342275 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1384152990 0x7f1384071fa0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:10.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.407+0000 7f1388c00640 1 -- 172.21.15.6:0/1938342275 >> 172.21.15.6:0/1938342275 conn(0x7f138406e590 msgr2=0x7f13840cd300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:10.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.407+0000 7f1388c00640 1 -- 172.21.15.6:0/1938342275 shutdown_connections 2024-04-24T23:16:10.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.407+0000 7f1388c00640 1 -- 172.21.15.6:0/1938342275 wait complete. 2024-04-24T23:16:10.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.408+0000 7f1388c00640 1 Processor -- start 2024-04-24T23:16:10.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.409+0000 7f1388c00640 1 -- start start 2024-04-24T23:16:10.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.409+0000 7f1388c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f13840c6a10 0x7f1384154cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:10.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.409+0000 7f1388c00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1384152990 0x7f1384155a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:10.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.409+0000 7f1388c00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1384159400 0x7f13840cca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:10.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.409+0000 7f1388c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f13840731c0 con 0x7f13840c6a10 2024-04-24T23:16:10.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.409+0000 7f1388c00640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f1384072ee0 con 0x7f1384152990 2024-04-24T23:16:10.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.409+0000 7f1388c00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f13840728b0 con 0x7f1384159400 2024-04-24T23:16:10.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.410+0000 7f1383400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f13840c6a10 0x7f1384154cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:10.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.410+0000 7f1383400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f13840c6a10 0x7f1384154cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:59894/0 (socket says 172.21.15.6:59894) 2024-04-24T23:16:10.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.410+0000 7f1383400640 1 -- 172.21.15.6:0/1486051742 learned_addr learned my addr 172.21.15.6:0/1486051742 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:16:10.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.410+0000 7f1382a00640 1 --2- 172.21.15.6:0/1486051742 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1384152990 0x7f1384155a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:10.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.410+0000 7f1383e00640 1 --2- 172.21.15.6:0/1486051742 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1384159400 0x7f13840cca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:10.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.410+0000 7f1383400640 1 -- 172.21.15.6:0/1486051742 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1384159400 msgr2=0x7f13840cca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:10.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.410+0000 7f1383400640 1 --2- 172.21.15.6:0/1486051742 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1384159400 0x7f13840cca90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:10.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.410+0000 7f1383400640 1 -- 172.21.15.6:0/1486051742 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1384152990 msgr2=0x7f1384155a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:10.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.410+0000 7f1383400640 1 --2- 172.21.15.6:0/1486051742 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1384152990 0x7f1384155a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:10.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.410+0000 7f1383400640 1 -- 172.21.15.6:0/1486051742 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13841e37f0 con 0x7f13840c6a10 2024-04-24T23:16:10.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.410+0000 7f1383e00640 1 --2- 172.21.15.6:0/1486051742 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1384159400 0x7f13840cca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:16:10.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.410+0000 7f1382a00640 1 --2- 172.21.15.6:0/1486051742 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1384152990 0x7f1384155a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:16:10.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.411+0000 7f1383400640 1 --2- 172.21.15.6:0/1486051742 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f13840c6a10 0x7f1384154cb0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f1374000f80 tx=0x7f1374004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:10.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.411+0000 7f137b400640 1 -- 172.21.15.6:0/1486051742 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1374042020 con 0x7f13840c6a10 2024-04-24T23:16:10.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.411+0000 7f137b400640 1 -- 172.21.15.6:0/1486051742 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1374032870 con 0x7f13840c6a10 2024-04-24T23:16:10.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.411+0000 7f137b400640 1 -- 172.21.15.6:0/1486051742 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1374037040 con 0x7f13840c6a10 2024-04-24T23:16:10.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.411+0000 7f1388c00640 1 -- 172.21.15.6:0/1486051742 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13841552b0 con 0x7f13840c6a10 2024-04-24T23:16:10.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.411+0000 7f1388c00640 1 -- 172.21.15.6:0/1486051742 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13841e3d70 con 0x7f13840c6a10 2024-04-24T23:16:10.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.413+0000 7f1388c00640 1 -- 172.21.15.6:0/1486051742 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1384075b30 con 0x7f13840c6a10 2024-04-24T23:16:10.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.413+0000 7f137b400640 1 -- 172.21.15.6:0/1486051742 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f1374005d10 con 0x7f13840c6a10 2024-04-24T23:16:10.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.414+0000 7f137b400640 1 --2- 172.21.15.6:0/1486051742 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f136006be30 0x7f136006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:10.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.414+0000 7f137b400640 1 -- 172.21.15.6:0/1486051742 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f13740ac040 con 0x7f13840c6a10 2024-04-24T23:16:10.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.415+0000 7f1382a00640 1 --2- 172.21.15.6:0/1486051742 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f136006be30 0x7f136006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:10.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.415+0000 7f1382a00640 1 --2- 172.21.15.6:0/1486051742 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f136006be30 0x7f136006e2f0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f13841567d0 tx=0x7f136c000bc0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:10.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.419+0000 7f137b400640 1 -- 172.21.15.6:0/1486051742 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f137404c070 con 0x7f13840c6a10 2024-04-24T23:16:10.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.657+0000 7f1388c00640 1 -- 172.21.15.6:0/1486051742 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f13840c65b0 con 0x7f13840c6a10 2024-04-24T23:16:10.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.658+0000 7f137b400640 1 -- 172.21.15.6:0/1486051742 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+6835 (secure 0 0 0) 0x7f137407a0f0 con 0x7f13840c6a10 2024-04-24T23:16:10.661 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:16:10.661 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":11,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4197,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/105267896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":105267896},{"type":"v1","addr":"172.21.15.6:6834","nonce":105267896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4203,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/2590421094","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":2590421094},{"type":"v1","addr":"172.21.15.6:6837","nonce":2590421094}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4232,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6838/2032280158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2032280158},{"type":"v1","addr":"172.21.15.190:6838","nonce":2032280158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6839/1572872451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":1572872451},{"type":"v1","addr":"172.21.15.190:6839","nonce":1572872451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":11,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:16:09.653660+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1],"up":{"mds_0":4306,"mds_1":4282},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4282":{"gid":4282,"name":"f","rank":1,"incarnation":9,"state":"up:creating","state_seq":1,"addr":"172.21.15.190:6837/580315269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":580315269},{"type":"v1","addr":"172.21.15.190:6837","nonce":580315269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/489092393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":489092393},{"type":"v1","addr":"172.21.15.6:6835","nonce":489092393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4306,"qdb_cluster":[4306]},"id":1}]} 2024-04-24T23:16:10.661 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 11 2024-04-24T23:16:10.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.661+0000 7f1388c00640 1 -- 172.21.15.6:0/1486051742 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f136006be30 msgr2=0x7f136006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:10.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.661+0000 7f1388c00640 1 --2- 172.21.15.6:0/1486051742 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f136006be30 0x7f136006e2f0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f13841567d0 tx=0x7f136c000bc0 comp rx=0 tx=0).stop 2024-04-24T23:16:10.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.661+0000 7f1388c00640 1 -- 172.21.15.6:0/1486051742 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f13840c6a10 msgr2=0x7f1384154cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:10.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.661+0000 7f1388c00640 1 --2- 172.21.15.6:0/1486051742 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f13840c6a10 0x7f1384154cb0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f1374000f80 tx=0x7f1374004a40 comp rx=0 tx=0).stop 2024-04-24T23:16:10.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.661+0000 7f1388c00640 1 -- 172.21.15.6:0/1486051742 shutdown_connections 2024-04-24T23:16:10.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.661+0000 7f1388c00640 1 --2- 172.21.15.6:0/1486051742 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f136006be30 0x7f136006e2f0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:10.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.661+0000 7f1388c00640 1 --2- 172.21.15.6:0/1486051742 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1384159400 0x7f13840cca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:10.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.661+0000 7f1388c00640 1 --2- 172.21.15.6:0/1486051742 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1384152990 0x7f1384155a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:10.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.661+0000 7f1388c00640 1 --2- 172.21.15.6:0/1486051742 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f13840c6a10 0x7f1384154cb0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:10.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.661+0000 7f1388c00640 1 -- 172.21.15.6:0/1486051742 >> 172.21.15.6:0/1486051742 conn(0x7f138406e590 msgr2=0x7f1384156d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:10.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.661+0000 7f1388c00640 1 -- 172.21.15.6:0/1486051742 shutdown_connections 2024-04-24T23:16:10.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.662+0000 7f1388c00640 1 -- 172.21.15.6:0/1486051742 wait complete. 2024-04-24T23:16:10.683 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T23:16:10.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.888+0000 7fd7eaa00640 1 Processor -- start 2024-04-24T23:16:10.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.889+0000 7fd7eaa00640 1 -- start start 2024-04-24T23:16:10.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.890+0000 7fd7eaa00640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fd7e4152bc0 0x7fd7e4159390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:10.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.890+0000 7fd7eaa00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd7e40c8f50 0x7fd7e415bf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:10.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.890+0000 7fd7eaa00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7e40c6e70 con 0x7fd7e40c9330 2024-04-24T23:16:10.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.890+0000 7fd7eaa00640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7fd7e40c73d0 con 0x7fd7e4152bc0 2024-04-24T23:16:10.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.890+0000 7fd7eaa00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fd7e40c8ac0 con 0x7fd7e40c8f50 2024-04-24T23:16:10.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.891+0000 7fd7e8a00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fd7e40c9330 0x7fd7e415c440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:43410/0 (socket says 172.21.15.6:43410) 2024-04-24T23:16:10.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.891+0000 7fd7e8a00640 1 -- 172.21.15.6:0/1399009804 learned_addr learned my addr 172.21.15.6:0/1399009804 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:16:10.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.891+0000 7fd7e3e00640 1 --2- 172.21.15.6:0/1399009804 >> v2:172.21.15.190:3300/0 conn(0x7fd7e4152bc0 0x7fd7e4159390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:10.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.891+0000 7fd7dbe00640 1 --2- 172.21.15.6:0/1399009804 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd7e40c8f50 0x7fd7e415bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:10.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.891+0000 7fd7e3e00640 1 -- 172.21.15.6:0/1399009804 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd7e40c8f50 msgr2=0x7fd7e415bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:16:10.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.891+0000 7fd7e3e00640 1 --2- 172.21.15.6:0/1399009804 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd7e40c8f50 0x7fd7e415bf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:10.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.891+0000 7fd7e3400640 1 -- 172.21.15.6:0/1399009804 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3019531347 0 0) 0x7fd7e40c6e70 con 0x7fd7e40c9330 2024-04-24T23:16:10.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.891+0000 7fd7e3e00640 1 -- 172.21.15.6:0/1399009804 >> v1:172.21.15.6:6789/0 conn(0x7fd7e40c9330 legacy=0x7fd7e415c440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:10.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.891+0000 7fd7e3e00640 1 -- 172.21.15.6:0/1399009804 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7e41598d0 con 0x7fd7e4152bc0 2024-04-24T23:16:10.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.891+0000 7fd7dbe00640 1 --2- 172.21.15.6:0/1399009804 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd7e40c8f50 0x7fd7e415bf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:16:10.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.892+0000 7fd7e3e00640 1 --2- 172.21.15.6:0/1399009804 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd7e4152bc0 0x7fd7e4159390 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fd7d40097a0 tx=0x7fd7d402a100 comp rx=0 tx=0).ready entity=mon.1 client_cookie=584c45a90db2ff05 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:10.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.893+0000 7fd7e3400640 1 -- 172.21.15.6:0/1399009804 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7d4037070 con 0x7fd7e4152bc0 2024-04-24T23:16:10.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.893+0000 7fd7e3400640 1 -- 172.21.15.6:0/1399009804 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7d402ae50 con 0x7fd7e4152bc0 2024-04-24T23:16:10.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.893+0000 7fd7e3400640 1 -- 172.21.15.6:0/1399009804 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7d4032660 con 0x7fd7e4152bc0 2024-04-24T23:16:10.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.893+0000 7fd7eaa00640 1 -- 172.21.15.6:0/1399009804 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd7e4152bc0 msgr2=0x7fd7e4159390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:10.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.893+0000 7fd7eaa00640 1 --2- 172.21.15.6:0/1399009804 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd7e4152bc0 0x7fd7e4159390 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fd7d40097a0 tx=0x7fd7d402a100 comp rx=0 tx=0).stop 2024-04-24T23:16:10.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.893+0000 7fd7eaa00640 1 -- 172.21.15.6:0/1399009804 shutdown_connections 2024-04-24T23:16:10.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.893+0000 7fd7eaa00640 1 --2- 172.21.15.6:0/1399009804 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd7e40c8f50 0x7fd7e415bf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:10.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.893+0000 7fd7eaa00640 1 --2- 172.21.15.6:0/1399009804 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd7e4152bc0 0x7fd7e4159390 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:10.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.893+0000 7fd7eaa00640 1 -- 172.21.15.6:0/1399009804 >> 172.21.15.6:0/1399009804 conn(0x7fd7e406b0e0 msgr2=0x7fd7e406b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:10.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.894+0000 7fd7eaa00640 1 -- 172.21.15.6:0/1399009804 shutdown_connections 2024-04-24T23:16:10.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.894+0000 7fd7eaa00640 1 -- 172.21.15.6:0/1399009804 wait complete. 2024-04-24T23:16:10.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.895+0000 7fd7eaa00640 1 Processor -- start 2024-04-24T23:16:10.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.896+0000 7fd7eaa00640 1 -- start start 2024-04-24T23:16:10.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.896+0000 7fd7eaa00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd7e40c8f50 0x7fd7e41f3da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:10.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.896+0000 7fd7eaa00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd7e40c9330 0x7fd7e413fe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:10.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.896+0000 7fd7eaa00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd7e4152bc0 0x7fd7e4147090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:10.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.896+0000 7fd7eaa00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fd7e40c8470 con 0x7fd7e4152bc0 2024-04-24T23:16:10.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.896+0000 7fd7eaa00640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fd7e40c6700 con 0x7fd7e40c8f50 2024-04-24T23:16:10.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.896+0000 7fd7eaa00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fd7e40c8750 con 0x7fd7e40c9330 2024-04-24T23:16:10.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.897+0000 7fd7dbe00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd7e40c9330 0x7fd7e413fe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:10.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.897+0000 7fd7e8a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd7e4152bc0 0x7fd7e4147090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:10.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.897+0000 7fd7e3e00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd7e40c8f50 0x7fd7e41f3da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:10.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.897+0000 7fd7e3e00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd7e40c8f50 0x7fd7e41f3da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:59682/0 (socket says 172.21.15.6:59682) 2024-04-24T23:16:10.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.897+0000 7fd7dbe00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd7e40c9330 0x7fd7e413fe70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.6:53102/0 (socket says 172.21.15.6:53102) 2024-04-24T23:16:10.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.897+0000 7fd7e3e00640 1 -- 172.21.15.6:0/2468268449 learned_addr learned my addr 172.21.15.6:0/2468268449 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:16:10.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.897+0000 7fd7e3e00640 1 -- 172.21.15.6:0/2468268449 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd7e40c9330 msgr2=0x7fd7e413fe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:10.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.897+0000 7fd7e3e00640 1 --2- 172.21.15.6:0/2468268449 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd7e40c9330 0x7fd7e413fe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:10.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.897+0000 7fd7e3e00640 1 -- 172.21.15.6:0/2468268449 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd7e4152bc0 msgr2=0x7fd7e4147090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:10.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.897+0000 7fd7e3e00640 1 --2- 172.21.15.6:0/2468268449 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd7e4152bc0 0x7fd7e4147090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:10.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.897+0000 7fd7e3e00640 1 -- 172.21.15.6:0/2468268449 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7e40724e0 con 0x7fd7e40c8f50 2024-04-24T23:16:10.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.897+0000 7fd7dbe00640 1 --2- 172.21.15.6:0/2468268449 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd7e40c9330 0x7fd7e413fe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:16:10.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.898+0000 7fd7e3e00640 1 --2- 172.21.15.6:0/2468268449 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd7e40c8f50 0x7fd7e41f3da0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fd7d40097a0 tx=0x7fd7d40046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:10.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.898+0000 7fd7e0c00640 1 -- 172.21.15.6:0/2468268449 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7d4037040 con 0x7fd7e40c8f50 2024-04-24T23:16:10.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.898+0000 7fd7e0c00640 1 -- 172.21.15.6:0/2468268449 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7d4004030 con 0x7fd7e40c8f50 2024-04-24T23:16:10.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.898+0000 7fd7e0c00640 1 -- 172.21.15.6:0/2468268449 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7d4004420 con 0x7fd7e40c8f50 2024-04-24T23:16:10.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.898+0000 7fd7eaa00640 1 -- 172.21.15.6:0/2468268449 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7e40727c0 con 0x7fd7e40c8f50 2024-04-24T23:16:10.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.898+0000 7fd7eaa00640 1 -- 172.21.15.6:0/2468268449 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7e4146420 con 0x7fd7e40c8f50 2024-04-24T23:16:10.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.899+0000 7fd7eaa00640 1 -- 172.21.15.6:0/2468268449 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd7e40767b0 con 0x7fd7e40c8f50 2024-04-24T23:16:10.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.901+0000 7fd7e0c00640 1 -- 172.21.15.6:0/2468268449 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fd7d4003610 con 0x7fd7e40c8f50 2024-04-24T23:16:10.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.901+0000 7fd7e0c00640 1 --2- 172.21.15.6:0/2468268449 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fd7c006be80 0x7fd7c006e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:10.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.901+0000 7fd7e0c00640 1 -- 172.21.15.6:0/2468268449 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fd7d40ac240 con 0x7fd7e40c8f50 2024-04-24T23:16:10.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.902+0000 7fd7dbe00640 1 --2- 172.21.15.6:0/2468268449 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fd7c006be80 0x7fd7c006e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:10.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.902+0000 7fd7dbe00640 1 --2- 172.21.15.6:0/2468268449 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fd7c006be80 0x7fd7c006e340 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fd7e4151c60 tx=0x7fd7cc009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:10.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:10.905+0000 7fd7e0c00640 1 -- 172.21.15.6:0/2468268449 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fd7d407a2d0 con 0x7fd7e40c8f50 2024-04-24T23:16:11.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:11.108+0000 7fd7eaa00640 1 -- 172.21.15.6:0/2468268449 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd7e40c75e0 con 0x7fd7e40c8f50 2024-04-24T23:16:11.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:11.110+0000 7fd7e0c00640 1 -- 172.21.15.6:0/2468268449 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+18458 (secure 0 0 0) 0x7fd7d407e5d0 con 0x7fd7e40c8f50 2024-04-24T23:16:11.114 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:16:11.114 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":24,"fsid":"18bef68d-5e6d-4ec5-862b-b6f206e2588d","created":"2024-04-24T23:15:12.535072+0000","modified":"2024-04-24T23:15:41.465875+0000","last_up_change":"2024-04-24T23:15:23.773223+0000","last_in_change":"2024-04-24T23:15:14.905328+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T23:15:24.747441+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-24T23:15:34.051800+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-24T23:15:35.436870+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-24T23:15:39.034257+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"3b7a212d-d498-462e-90bf-ed493c803a09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":3022569437},{"type":"v1","addr":"172.21.15.6:6809","nonce":3022569437}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":3022569437},{"type":"v1","addr":"172.21.15.6:6811","nonce":3022569437}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":3022569437},{"type":"v1","addr":"172.21.15.6:6815","nonce":3022569437}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":3022569437},{"type":"v1","addr":"172.21.15.6:6813","nonce":3022569437}]},"public_addr":"172.21.15.6:6809/3022569437","cluster_addr":"172.21.15.6:6811/3022569437","heartbeat_back_addr":"172.21.15.6:6815/3022569437","heartbeat_front_addr":"172.21.15.6:6813/3022569437","state":["exists","up"]},{"osd":1,"uuid":"7c97f5a2-1735-4775-9f15-eb50099b19d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":3687836888},{"type":"v1","addr":"172.21.15.6:6817","nonce":3687836888}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":3687836888},{"type":"v1","addr":"172.21.15.6:6819","nonce":3687836888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":3687836888},{"type":"v1","addr":"172.21.15.6:6823","nonce":3687836888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":3687836888},{"type":"v1","addr":"172.21.15.6:6821","nonce":3687836888}]},"public_addr":"172.21.15.6:6817/3687836888","cluster_addr":"172.21.15.6:6819/3687836888","heartbeat_back_addr":"172.21.15.6:6823/3687836888","heartbeat_front_addr":"172.21.15.6:6821/3687836888","state":["exists","up"]},{"osd":2,"uuid":"d86bbc18-438c-4ec2-8018-ba7dd78f790b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":1020213724},{"type":"v1","addr":"172.21.15.6:6801","nonce":1020213724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":1020213724},{"type":"v1","addr":"172.21.15.6:6803","nonce":1020213724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":1020213724},{"type":"v1","addr":"172.21.15.6:6807","nonce":1020213724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":1020213724},{"type":"v1","addr":"172.21.15.6:6805","nonce":1020213724}]},"public_addr":"172.21.15.6:6801/1020213724","cluster_addr":"172.21.15.6:6803/1020213724","heartbeat_back_addr":"172.21.15.6:6807/1020213724","heartbeat_front_addr":"172.21.15.6:6805/1020213724","state":["exists","up"]},{"osd":3,"uuid":"3754ae16-4248-40dd-95f9-0f59c0eb5426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6824","nonce":3163883468},{"type":"v1","addr":"172.21.15.6:6825","nonce":3163883468}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":3163883468},{"type":"v1","addr":"172.21.15.6:6827","nonce":3163883468}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":3163883468},{"type":"v1","addr":"172.21.15.6:6831","nonce":3163883468}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":3163883468},{"type":"v1","addr":"172.21.15.6:6829","nonce":3163883468}]},"public_addr":"172.21.15.6:6825/3163883468","cluster_addr":"172.21.15.6:6827/3163883468","heartbeat_back_addr":"172.21.15.6:6831/3163883468","heartbeat_front_addr":"172.21.15.6:6829/3163883468","state":["exists","up"]},{"osd":4,"uuid":"8b2913a0-d006-479c-84ed-5b21e0e6c392","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":1989500934},{"type":"v1","addr":"172.21.15.190:6809","nonce":1989500934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":1989500934},{"type":"v1","addr":"172.21.15.190:6811","nonce":1989500934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":1989500934},{"type":"v1","addr":"172.21.15.190:6815","nonce":1989500934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":1989500934},{"type":"v1","addr":"172.21.15.190:6813","nonce":1989500934}]},"public_addr":"172.21.15.190:6809/1989500934","cluster_addr":"172.21.15.190:6811/1989500934","heartbeat_back_addr":"172.21.15.190:6815/1989500934","heartbeat_front_addr":"172.21.15.190:6813/1989500934","state":["exists","up"]},{"osd":5,"uuid":"499b79d0-0585-4dae-ab69-196a551793da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":3687548412},{"type":"v1","addr":"172.21.15.190:6801","nonce":3687548412}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":3687548412},{"type":"v1","addr":"172.21.15.190:6803","nonce":3687548412}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":3687548412},{"type":"v1","addr":"172.21.15.190:6807","nonce":3687548412}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":3687548412},{"type":"v1","addr":"172.21.15.190:6805","nonce":3687548412}]},"public_addr":"172.21.15.190:6801/3687548412","cluster_addr":"172.21.15.190:6803/3687548412","heartbeat_back_addr":"172.21.15.190:6807/3687548412","heartbeat_front_addr":"172.21.15.190:6805/3687548412","state":["exists","up"]},{"osd":6,"uuid":"6ba6df3b-4a6b-4a89-a890-0f79f387d4d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":870571564},{"type":"v1","addr":"172.21.15.190:6817","nonce":870571564}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":870571564},{"type":"v1","addr":"172.21.15.190:6819","nonce":870571564}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":870571564},{"type":"v1","addr":"172.21.15.190:6823","nonce":870571564}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":870571564},{"type":"v1","addr":"172.21.15.190:6821","nonce":870571564}]},"public_addr":"172.21.15.190:6817/870571564","cluster_addr":"172.21.15.190:6819/870571564","heartbeat_back_addr":"172.21.15.190:6823/870571564","heartbeat_front_addr":"172.21.15.190:6821/870571564","state":["exists","up"]},{"osd":7,"uuid":"e9668b1c-a000-4332-86e8-4efd73b02409","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":123639197},{"type":"v1","addr":"172.21.15.190:6825","nonce":123639197}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":123639197},{"type":"v1","addr":"172.21.15.190:6827","nonce":123639197}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":123639197},{"type":"v1","addr":"172.21.15.190:6831","nonce":123639197}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":123639197},{"type":"v1","addr":"172.21.15.190:6829","nonce":123639197}]},"public_addr":"172.21.15.190:6825/123639197","cluster_addr":"172.21.15.190:6827/123639197","heartbeat_back_addr":"172.21.15.190:6831/123639197","heartbeat_front_addr":"172.21.15.190:6829/123639197","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.321646+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.363873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.466963+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.531989+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:19.990707+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.043543+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.032492+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.115004+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T23:16:11.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:11.113+0000 7fd7eaa00640 1 -- 172.21.15.6:0/2468268449 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fd7c006be80 msgr2=0x7fd7c006e340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:11.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:11.113+0000 7fd7eaa00640 1 --2- 172.21.15.6:0/2468268449 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fd7c006be80 0x7fd7c006e340 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fd7e4151c60 tx=0x7fd7cc009190 comp rx=0 tx=0).stop 2024-04-24T23:16:11.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:11.113+0000 7fd7eaa00640 1 -- 172.21.15.6:0/2468268449 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd7e40c8f50 msgr2=0x7fd7e41f3da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:11.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:11.113+0000 7fd7eaa00640 1 --2- 172.21.15.6:0/2468268449 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd7e40c8f50 0x7fd7e41f3da0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fd7d40097a0 tx=0x7fd7d40046c0 comp rx=0 tx=0).stop 2024-04-24T23:16:11.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:11.113+0000 7fd7eaa00640 1 -- 172.21.15.6:0/2468268449 shutdown_connections 2024-04-24T23:16:11.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:11.113+0000 7fd7eaa00640 1 --2- 172.21.15.6:0/2468268449 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fd7c006be80 0x7fd7c006e340 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:11.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:11.113+0000 7fd7eaa00640 1 --2- 172.21.15.6:0/2468268449 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd7e40c9330 0x7fd7e413fe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:11.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:11.113+0000 7fd7eaa00640 1 --2- 172.21.15.6:0/2468268449 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd7e4152bc0 0x7fd7e4147090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:11.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:11.113+0000 7fd7eaa00640 1 --2- 172.21.15.6:0/2468268449 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd7e40c8f50 0x7fd7e41f3da0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:11.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:11.113+0000 7fd7eaa00640 1 -- 172.21.15.6:0/2468268449 >> 172.21.15.6:0/2468268449 conn(0x7fd7e406b0e0 msgr2=0x7fd7e41485f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:11.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:11.113+0000 7fd7eaa00640 1 -- 172.21.15.6:0/2468268449 shutdown_connections 2024-04-24T23:16:11.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:11.114+0000 7fd7eaa00640 1 -- 172.21.15.6:0/2468268449 wait complete. 2024-04-24T23:16:11.137 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-24T23:16:11.137 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T23:16:11.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:11.332+0000 7fb4f7200640 1 Processor -- start 2024-04-24T23:16:11.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:11.333+0000 7fb4f7200640 1 -- start start 2024-04-24T23:16:11.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:11.334+0000 7fb4f7200640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb4f806ce30 0x7fb4f80cc600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:11.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:11.334+0000 7fb4f7200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb4f806d390 con 0x7fb4f8152bd0 2024-04-24T23:16:11.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:11.334+0000 7fb4f7200640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb4f806dbc0 con 0x7fb4f80c6be0 2024-04-24T23:16:11.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:11.334+0000 7fb4f7200640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fb4f806d970 con 0x7fb4f806ce30 2024-04-24T23:16:11.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:11.334+0000 7fb4f5e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fb4f8152bd0 0x7fb4f8158e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:43412/0 (socket says 172.21.15.6:43412) 2024-04-24T23:16:11.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:11.334+0000 7fb4f5e00640 1 -- 172.21.15.6:0/1557741858 learned_addr learned my addr 172.21.15.6:0/1557741858 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:16:11.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:11.334+0000 7fb4f5400640 1 --2- 172.21.15.6:0/1557741858 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb4f806ce30 0x7fb4f80cc600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:11.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:11.335+0000 7fb4f4a00640 1 -- 172.21.15.6:0/1557741858 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 235065252 0 0) 0x7fb4f806d390 con 0x7fb4f8152bd0 2024-04-24T23:16:11.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:11.335+0000 7fb4f4a00640 1 -- 172.21.15.6:0/1557741858 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb4dc003520 con 0x7fb4f8152bd0 2024-04-24T23:16:11.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:11.335+0000 7fb4f4a00640 1 -- 172.21.15.6:0/1557741858 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2248162753 0 0) 0x7fb4f806dbc0 con 0x7fb4f80c6be0 2024-04-24T23:16:11.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:11.335+0000 7fb4f4a00640 1 -- 172.21.15.6:0/1557741858 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb4f806d390 con 0x7fb4f80c6be0 2024-04-24T23:16:11.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:11.335+0000 7fb4f5400640 1 -- 172.21.15.6:0/1557741858 >> v1:172.21.15.190:6789/0 conn(0x7fb4f80c6be0 legacy=0x7fb4f8163090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:11.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:11.335+0000 7fb4f5400640 1 -- 172.21.15.6:0/1557741858 >> v1:172.21.15.6:6789/0 conn(0x7fb4f8152bd0 legacy=0x7fb4f8158e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:11.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:11.335+0000 7fb4f5400640 1 -- 172.21.15.6:0/1557741858 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4f806dea0 con 0x7fb4f806ce30 2024-04-24T23:16:11.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:11.336+0000 7fb4f5400640 1 --2- 172.21.15.6:0/1557741858 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb4f806ce30 0x7fb4f80cc600 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fb4e40097d0 tx=0x7fb4e402a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=84d13eb47794d175 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:11.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:11.336+0000 7fb4f4a00640 1 -- 172.21.15.6:0/1557741858 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4e4037070 con 0x7fb4f806ce30 2024-04-24T23:16:11.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:11.336+0000 7fb4f4a00640 1 -- 172.21.15.6:0/1557741858 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb4e4032440 con 0x7fb4f806ce30 2024-04-24T23:16:11.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:11.336+0000 7fb4f4a00640 1 -- 172.21.15.6:0/1557741858 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4e4032810 con 0x7fb4f806ce30 2024-04-24T23:16:11.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:11.337+0000 7fb4f7200640 1 -- 172.21.15.6:0/1557741858 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb4f806ce30 msgr2=0x7fb4f80cc600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:11.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:11.337+0000 7fb4f7200640 1 --2- 172.21.15.6:0/1557741858 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb4f806ce30 0x7fb4f80cc600 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fb4e40097d0 tx=0x7fb4e402a180 comp rx=0 tx=0).stop 2024-04-24T23:16:11.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:11.337+0000 7fb4f7200640 1 -- 172.21.15.6:0/1557741858 shutdown_connections 2024-04-24T23:16:11.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:11.337+0000 7fb4f7200640 1 --2- 172.21.15.6:0/1557741858 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb4f806ce30 0x7fb4f80cc600 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:11.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:11.337+0000 7fb4f7200640 1 -- 172.21.15.6:0/1557741858 >> 172.21.15.6:0/1557741858 conn(0x7fb4f806ed80 msgr2=0x7fb4f806f190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:11.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:11.337+0000 7fb4f7200640 1 -- 172.21.15.6:0/1557741858 shutdown_connections 2024-04-24T23:16:11.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:11.337+0000 7fb4f7200640 1 -- 172.21.15.6:0/1557741858 wait complete. 2024-04-24T23:16:11.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:11.339+0000 7fb4f7200640 1 Processor -- start 2024-04-24T23:16:11.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:11.339+0000 7fb4f7200640 1 -- start start 2024-04-24T23:16:11.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:11.340+0000 7fb4f7200640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb4f806ce30 0x7fb4f814f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:11.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:11.340+0000 7fb4f7200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb4f80c6be0 0x7fb4f81f2260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:11.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:11.340+0000 7fb4f7200640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb4f8152bd0 0x7fb4f81eecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:11.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:11.340+0000 7fb4f7200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fb4f8072650 con 0x7fb4f80c6be0 2024-04-24T23:16:11.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:11.340+0000 7fb4f7200640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fb4f8071890 con 0x7fb4f8152bd0 2024-04-24T23:16:11.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:11.340+0000 7fb4f7200640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fb4f80727d0 con 0x7fb4f806ce30 2024-04-24T23:16:11.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:11.340+0000 7fb4f5400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb4f80c6be0 0x7fb4f81f2260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:11.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:11.340+0000 7fb4f5400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb4f80c6be0 0x7fb4f81f2260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:59914/0 (socket says 172.21.15.6:59914) 2024-04-24T23:16:11.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:11.340+0000 7fb4f5400640 1 -- 172.21.15.6:0/475758777 learned_addr learned my addr 172.21.15.6:0/475758777 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:16:11.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:11.340+0000 7fb4f6800640 1 --2- 172.21.15.6:0/475758777 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb4f8152bd0 0x7fb4f81eecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:11.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:11.340+0000 7fb4f5e00640 1 --2- 172.21.15.6:0/475758777 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb4f806ce30 0x7fb4f814f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:11.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:11.341+0000 7fb4f5400640 1 -- 172.21.15.6:0/475758777 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb4f806ce30 msgr2=0x7fb4f814f820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:11.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:11.341+0000 7fb4f5400640 1 --2- 172.21.15.6:0/475758777 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb4f806ce30 0x7fb4f814f820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:11.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:11.341+0000 7fb4f5400640 1 -- 172.21.15.6:0/475758777 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb4f8152bd0 msgr2=0x7fb4f81eecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:11.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:11.341+0000 7fb4f5400640 1 --2- 172.21.15.6:0/475758777 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb4f8152bd0 0x7fb4f81eecf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:11.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:11.341+0000 7fb4f5400640 1 -- 172.21.15.6:0/475758777 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4f81ef230 con 0x7fb4f80c6be0 2024-04-24T23:16:11.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:11.341+0000 7fb4f5400640 1 --2- 172.21.15.6:0/475758777 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb4f80c6be0 0x7fb4f81f2260 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fb4e4035040 tx=0x7fb4e40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:11.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:11.341+0000 7fb4ee000640 1 -- 172.21.15.6:0/475758777 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4e4041020 con 0x7fb4f80c6be0 2024-04-24T23:16:11.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:11.341+0000 7fb4ee000640 1 -- 172.21.15.6:0/475758777 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb4e403d720 con 0x7fb4f80c6be0 2024-04-24T23:16:11.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:11.342+0000 7fb4ee000640 1 -- 172.21.15.6:0/475758777 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4e4037040 con 0x7fb4f80c6be0 2024-04-24T23:16:11.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:11.342+0000 7fb4f7200640 1 -- 172.21.15.6:0/475758777 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4f8155af0 con 0x7fb4f80c6be0 2024-04-24T23:16:11.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:11.342+0000 7fb4f7200640 1 -- 172.21.15.6:0/475758777 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4f81ee4f0 con 0x7fb4f80c6be0 2024-04-24T23:16:11.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:11.343+0000 7fb4f7200640 1 -- 172.21.15.6:0/475758777 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb4f8156010 con 0x7fb4f80c6be0 2024-04-24T23:16:11.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:11.344+0000 7fb4ee000640 1 -- 172.21.15.6:0/475758777 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fb4e4044020 con 0x7fb4f80c6be0 2024-04-24T23:16:11.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:11.345+0000 7fb4ee000640 1 --2- 172.21.15.6:0/475758777 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fb4d806be30 0x7fb4d806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:11.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:11.345+0000 7fb4ee000640 1 -- 172.21.15.6:0/475758777 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fb4e40ab820 con 0x7fb4f80c6be0 2024-04-24T23:16:11.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:11.345+0000 7fb4f5e00640 1 --2- 172.21.15.6:0/475758777 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fb4d806be30 0x7fb4d806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:11.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:11.346+0000 7fb4f5e00640 1 --2- 172.21.15.6:0/475758777 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fb4d806be30 0x7fb4d806e2f0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fb4e8000f60 tx=0x7fb4e8007d50 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:11.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:11.349+0000 7fb4ee000640 1 -- 172.21.15.6:0/475758777 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fb4f8156010 con 0x7fb4f80c6be0 2024-04-24T23:16:11.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:11.546+0000 7fb4f7200640 1 -- 172.21.15.6:0/475758777 --> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fb4f81ee2e0 con 0x7fb4d806be30 2024-04-24T23:16:11.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:11.557+0000 7fb4ee000640 1 -- 172.21.15.6:0/475758777 <== mgr.4105 v2:172.21.15.190:6832/806769663 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+332533 (secure 0 0 0) 0x7fb4f81ee2e0 con 0x7fb4d806be30 2024-04-24T23:16:11.565 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:16:11.567 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-04-24T23:16:11.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:11.563+0000 7fb4f7200640 1 -- 172.21.15.6:0/475758777 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fb4d806be30 msgr2=0x7fb4d806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:11.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:11.563+0000 7fb4f7200640 1 --2- 172.21.15.6:0/475758777 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fb4d806be30 0x7fb4d806e2f0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fb4e8000f60 tx=0x7fb4e8007d50 comp rx=0 tx=0).stop 2024-04-24T23:16:11.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:11.563+0000 7fb4f7200640 1 -- 172.21.15.6:0/475758777 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb4f80c6be0 msgr2=0x7fb4f81f2260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:11.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:11.563+0000 7fb4f7200640 1 --2- 172.21.15.6:0/475758777 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb4f80c6be0 0x7fb4f81f2260 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fb4e4035040 tx=0x7fb4e40046c0 comp rx=0 tx=0).stop 2024-04-24T23:16:11.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:11.563+0000 7fb4f7200640 1 -- 172.21.15.6:0/475758777 shutdown_connections 2024-04-24T23:16:11.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:11.563+0000 7fb4f7200640 1 --2- 172.21.15.6:0/475758777 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fb4d806be30 0x7fb4d806e2f0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:11.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:11.563+0000 7fb4f7200640 1 --2- 172.21.15.6:0/475758777 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb4f8152bd0 0x7fb4f81eecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:11.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:11.563+0000 7fb4f7200640 1 --2- 172.21.15.6:0/475758777 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb4f80c6be0 0x7fb4f81f2260 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:11.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:11.563+0000 7fb4f7200640 1 --2- 172.21.15.6:0/475758777 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb4f806ce30 0x7fb4f814f820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:11.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:11.563+0000 7fb4f7200640 1 -- 172.21.15.6:0/475758777 >> 172.21.15.6:0/475758777 conn(0x7fb4f806ed80 msgr2=0x7fb4f80c8f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:11.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:11.563+0000 7fb4f7200640 1 -- 172.21.15.6:0/475758777 shutdown_connections 2024-04-24T23:16:11.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:11.563+0000 7fb4f7200640 1 -- 172.21.15.6:0/475758777 wait complete. 2024-04-24T23:16:11.584 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":40,"stamp":"2024-04-24T23:16:10.688215+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595008,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":208,"num_write_kb":2435,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160,"ondisk_log_size":160,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219368,"kb_used_data":3208,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755352,"statfs":{"total":773094113280,"available":772869480448,"internally_reserved":0,"allocated":3284992,"data_stored":1992128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":141,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003445"},"pg_stats":[{"pgid":"2.1f","version":"23'4","reported_seq":26,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.051292+0000","last_change":"2024-04-24T23:15:35.964646+0000","last_active":"2024-04-24T23:15:42.051292+0000","last_peered":"2024-04-24T23:15:42.051292+0000","last_clean":"2024-04-24T23:15:42.051292+0000","last_became_active":"2024-04-24T23:15:35.964357+0000","last_became_peered":"2024-04-24T23:15:35.964357+0000","last_unstale":"2024-04-24T23:15:42.051292+0000","last_undegraded":"2024-04-24T23:15:42.051292+0000","last_fullsized":"2024-04-24T23:15:42.051292+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:01:17.912993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.295716+0000","last_change":"2024-04-24T23:15:36.989772+0000","last_active":"2024-04-24T23:15:42.295716+0000","last_peered":"2024-04-24T23:15:42.295716+0000","last_clean":"2024-04-24T23:15:42.295716+0000","last_became_active":"2024-04-24T23:15:36.989543+0000","last_became_peered":"2024-04-24T23:15:36.989543+0000","last_unstale":"2024-04-24T23:15:42.295716+0000","last_undegraded":"2024-04-24T23:15:42.295716+0000","last_fullsized":"2024-04-24T23:15:42.295716+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:37:16.737354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.187704+0000","last_change":"2024-04-24T23:15:41.017182+0000","last_active":"2024-04-24T23:15:43.187704+0000","last_peered":"2024-04-24T23:15:43.187704+0000","last_clean":"2024-04-24T23:15:43.187704+0000","last_became_active":"2024-04-24T23:15:41.016719+0000","last_became_peered":"2024-04-24T23:15:41.016719+0000","last_unstale":"2024-04-24T23:15:43.187704+0000","last_undegraded":"2024-04-24T23:15:43.187704+0000","last_fullsized":"2024-04-24T23:15:43.187704+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:42:11.186486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4,1],"acting":[7,6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1e","version":"23'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.295817+0000","last_change":"2024-04-24T23:15:35.946083+0000","last_active":"2024-04-24T23:15:42.295817+0000","last_peered":"2024-04-24T23:15:42.295817+0000","last_clean":"2024-04-24T23:15:42.295817+0000","last_became_active":"2024-04-24T23:15:35.945227+0000","last_became_peered":"2024-04-24T23:15:35.945227+0000","last_unstale":"2024-04-24T23:15:42.295817+0000","last_undegraded":"2024-04-24T23:15:42.295817+0000","last_fullsized":"2024-04-24T23:15:42.295817+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:05:17.727725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.295775+0000","last_change":"2024-04-24T23:15:36.998245+0000","last_active":"2024-04-24T23:15:42.295775+0000","last_peered":"2024-04-24T23:15:42.295775+0000","last_clean":"2024-04-24T23:15:42.295775+0000","last_became_active":"2024-04-24T23:15:36.998021+0000","last_became_peered":"2024-04-24T23:15:36.998021+0000","last_unstale":"2024-04-24T23:15:42.295775+0000","last_undegraded":"2024-04-24T23:15:42.295775+0000","last_fullsized":"2024-04-24T23:15:42.295775+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:03:42.359117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.106527+0000","last_change":"2024-04-24T23:15:41.008313+0000","last_active":"2024-04-24T23:15:42.106527+0000","last_peered":"2024-04-24T23:15:42.106527+0000","last_clean":"2024-04-24T23:15:42.106527+0000","last_became_active":"2024-04-24T23:15:41.007908+0000","last_became_peered":"2024-04-24T23:15:41.007908+0000","last_unstale":"2024-04-24T23:15:42.106527+0000","last_undegraded":"2024-04-24T23:15:42.106527+0000","last_fullsized":"2024-04-24T23:15:42.106527+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:02:11.902575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4,6],"acting":[2,7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1d","version":"23'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.193739+0000","last_change":"2024-04-24T23:15:35.968312+0000","last_active":"2024-04-24T23:15:43.193739+0000","last_peered":"2024-04-24T23:15:43.193739+0000","last_clean":"2024-04-24T23:15:43.193739+0000","last_became_active":"2024-04-24T23:15:35.967873+0000","last_became_peered":"2024-04-24T23:15:35.967873+0000","last_unstale":"2024-04-24T23:15:43.193739+0000","last_undegraded":"2024-04-24T23:15:43.193739+0000","last_fullsized":"2024-04-24T23:15:43.193739+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:21:40.116977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.733492+0000","last_change":"2024-04-24T23:15:36.997361+0000","last_active":"2024-04-24T23:15:42.733492+0000","last_peered":"2024-04-24T23:15:42.733492+0000","last_clean":"2024-04-24T23:15:42.733492+0000","last_became_active":"2024-04-24T23:15:36.997083+0000","last_became_peered":"2024-04-24T23:15:36.997083+0000","last_unstale":"2024-04-24T23:15:42.733492+0000","last_undegraded":"2024-04-24T23:15:42.733492+0000","last_fullsized":"2024-04-24T23:15:42.733492+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:47:59.334093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.107355+0000","last_change":"2024-04-24T23:15:41.025153+0000","last_active":"2024-04-24T23:15:42.107355+0000","last_peered":"2024-04-24T23:15:42.107355+0000","last_clean":"2024-04-24T23:15:42.107355+0000","last_became_active":"2024-04-24T23:15:41.024739+0000","last_became_peered":"2024-04-24T23:15:41.024739+0000","last_unstale":"2024-04-24T23:15:42.107355+0000","last_undegraded":"2024-04-24T23:15:42.107355+0000","last_fullsized":"2024-04-24T23:15:42.107355+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:31:00.160213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3,6],"acting":[2,5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1c","version":"23'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.193705+0000","last_change":"2024-04-24T23:15:35.967338+0000","last_active":"2024-04-24T23:15:43.193705+0000","last_peered":"2024-04-24T23:15:43.193705+0000","last_clean":"2024-04-24T23:15:43.193705+0000","last_became_active":"2024-04-24T23:15:35.966951+0000","last_became_peered":"2024-04-24T23:15:35.966951+0000","last_unstale":"2024-04-24T23:15:43.193705+0000","last_undegraded":"2024-04-24T23:15:43.193705+0000","last_fullsized":"2024-04-24T23:15:43.193705+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:08:05.248330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.191745+0000","last_change":"2024-04-24T23:15:36.998072+0000","last_active":"2024-04-24T23:15:43.191745+0000","last_peered":"2024-04-24T23:15:43.191745+0000","last_clean":"2024-04-24T23:15:43.191745+0000","last_became_active":"2024-04-24T23:15:36.997735+0000","last_became_peered":"2024-04-24T23:15:36.997735+0000","last_unstale":"2024-04-24T23:15:43.191745+0000","last_undegraded":"2024-04-24T23:15:43.191745+0000","last_fullsized":"2024-04-24T23:15:43.191745+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:22:06.869644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.106640+0000","last_change":"2024-04-24T23:15:41.008895+0000","last_active":"2024-04-24T23:15:42.106640+0000","last_peered":"2024-04-24T23:15:42.106640+0000","last_clean":"2024-04-24T23:15:42.106640+0000","last_became_active":"2024-04-24T23:15:41.006199+0000","last_became_peered":"2024-04-24T23:15:41.006199+0000","last_unstale":"2024-04-24T23:15:42.106640+0000","last_undegraded":"2024-04-24T23:15:42.106640+0000","last_fullsized":"2024-04-24T23:15:42.106640+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:13:34.583602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5,7],"acting":[2,6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.191106+0000","last_change":"2024-04-24T23:15:37.004074+0000","last_active":"2024-04-24T23:15:43.191106+0000","last_peered":"2024-04-24T23:15:43.191106+0000","last_clean":"2024-04-24T23:15:43.191106+0000","last_became_active":"2024-04-24T23:15:37.003544+0000","last_became_peered":"2024-04-24T23:15:37.003544+0000","last_unstale":"2024-04-24T23:15:43.191106+0000","last_undegraded":"2024-04-24T23:15:43.191106+0000","last_fullsized":"2024-04-24T23:15:43.191106+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:38:28.024702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"23'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.191747+0000","last_change":"2024-04-24T23:15:35.963000+0000","last_active":"2024-04-24T23:15:43.191747+0000","last_peered":"2024-04-24T23:15:43.191747+0000","last_clean":"2024-04-24T23:15:43.191747+0000","last_became_active":"2024-04-24T23:15:35.961960+0000","last_became_peered":"2024-04-24T23:15:35.961960+0000","last_unstale":"2024-04-24T23:15:43.191747+0000","last_undegraded":"2024-04-24T23:15:43.191747+0000","last_fullsized":"2024-04-24T23:15:43.191747+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:56:26.651193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.191430+0000","last_change":"2024-04-24T23:15:41.022481+0000","last_active":"2024-04-24T23:15:43.191430+0000","last_peered":"2024-04-24T23:15:43.191430+0000","last_clean":"2024-04-24T23:15:43.191430+0000","last_became_active":"2024-04-24T23:15:41.020034+0000","last_became_peered":"2024-04-24T23:15:41.020034+0000","last_unstale":"2024-04-24T23:15:43.191430+0000","last_undegraded":"2024-04-24T23:15:43.191430+0000","last_fullsized":"2024-04-24T23:15:43.191430+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:27:25.337503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,5],"acting":[3,1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.187801+0000","last_change":"2024-04-24T23:15:36.972020+0000","last_active":"2024-04-24T23:15:43.187801+0000","last_peered":"2024-04-24T23:15:43.187801+0000","last_clean":"2024-04-24T23:15:43.187801+0000","last_became_active":"2024-04-24T23:15:36.971583+0000","last_became_peered":"2024-04-24T23:15:36.971583+0000","last_unstale":"2024-04-24T23:15:43.187801+0000","last_undegraded":"2024-04-24T23:15:43.187801+0000","last_fullsized":"2024-04-24T23:15:43.187801+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:29:05.568608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.050091+0000","last_change":"2024-04-24T23:15:35.945916+0000","last_active":"2024-04-24T23:15:42.050091+0000","last_peered":"2024-04-24T23:15:42.050091+0000","last_clean":"2024-04-24T23:15:42.050091+0000","last_became_active":"2024-04-24T23:15:35.945610+0000","last_became_peered":"2024-04-24T23:15:35.945610+0000","last_unstale":"2024-04-24T23:15:42.050091+0000","last_undegraded":"2024-04-24T23:15:42.050091+0000","last_fullsized":"2024-04-24T23:15:42.050091+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:57:05.976074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.192627+0000","last_change":"2024-04-24T23:15:41.030165+0000","last_active":"2024-04-24T23:15:43.192627+0000","last_peered":"2024-04-24T23:15:43.192627+0000","last_clean":"2024-04-24T23:15:43.192627+0000","last_became_active":"2024-04-24T23:15:41.029685+0000","last_became_peered":"2024-04-24T23:15:41.029685+0000","last_unstale":"2024-04-24T23:15:43.192627+0000","last_undegraded":"2024-04-24T23:15:43.192627+0000","last_fullsized":"2024-04-24T23:15:43.192627+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:20:43.498190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6,1],"acting":[0,4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.049168+0000","last_change":"2024-04-24T23:15:36.965949+0000","last_active":"2024-04-24T23:15:42.049168+0000","last_peered":"2024-04-24T23:15:42.049168+0000","last_clean":"2024-04-24T23:15:42.049168+0000","last_became_active":"2024-04-24T23:15:36.965658+0000","last_became_peered":"2024-04-24T23:15:36.965658+0000","last_unstale":"2024-04-24T23:15:42.049168+0000","last_undegraded":"2024-04-24T23:15:42.049168+0000","last_fullsized":"2024-04-24T23:15:42.049168+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:38:28.024702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.049695+0000","last_change":"2024-04-24T23:15:35.972165+0000","last_active":"2024-04-24T23:15:42.049695+0000","last_peered":"2024-04-24T23:15:42.049695+0000","last_clean":"2024-04-24T23:15:42.049695+0000","last_became_active":"2024-04-24T23:15:35.971972+0000","last_became_peered":"2024-04-24T23:15:35.971972+0000","last_unstale":"2024-04-24T23:15:42.049695+0000","last_undegraded":"2024-04-24T23:15:42.049695+0000","last_fullsized":"2024-04-24T23:15:42.049695+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:11:12.088698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.188224+0000","last_change":"2024-04-24T23:15:41.019799+0000","last_active":"2024-04-24T23:15:43.188224+0000","last_peered":"2024-04-24T23:15:43.188224+0000","last_clean":"2024-04-24T23:15:43.188224+0000","last_became_active":"2024-04-24T23:15:41.019339+0000","last_became_peered":"2024-04-24T23:15:41.019339+0000","last_unstale":"2024-04-24T23:15:43.188224+0000","last_undegraded":"2024-04-24T23:15:43.188224+0000","last_fullsized":"2024-04-24T23:15:43.188224+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:59:34.601004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2,3],"acting":[7,4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.190629+0000","last_change":"2024-04-24T23:15:36.991991+0000","last_active":"2024-04-24T23:15:43.190629+0000","last_peered":"2024-04-24T23:15:43.190629+0000","last_clean":"2024-04-24T23:15:43.190629+0000","last_became_active":"2024-04-24T23:15:36.989336+0000","last_became_peered":"2024-04-24T23:15:36.989336+0000","last_unstale":"2024-04-24T23:15:43.190629+0000","last_undegraded":"2024-04-24T23:15:43.190629+0000","last_fullsized":"2024-04-24T23:15:43.190629+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:25:56.038903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.190149+0000","last_change":"2024-04-24T23:15:35.965443+0000","last_active":"2024-04-24T23:15:43.190149+0000","last_peered":"2024-04-24T23:15:43.190149+0000","last_clean":"2024-04-24T23:15:43.190149+0000","last_became_active":"2024-04-24T23:15:35.965043+0000","last_became_peered":"2024-04-24T23:15:35.965043+0000","last_unstale":"2024-04-24T23:15:43.190149+0000","last_undegraded":"2024-04-24T23:15:43.190149+0000","last_fullsized":"2024-04-24T23:15:43.190149+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:07:23.320042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.107146+0000","last_change":"2024-04-24T23:15:41.025131+0000","last_active":"2024-04-24T23:15:42.107146+0000","last_peered":"2024-04-24T23:15:42.107146+0000","last_clean":"2024-04-24T23:15:42.107146+0000","last_became_active":"2024-04-24T23:15:41.024731+0000","last_became_peered":"2024-04-24T23:15:41.024731+0000","last_unstale":"2024-04-24T23:15:42.107146+0000","last_undegraded":"2024-04-24T23:15:42.107146+0000","last_fullsized":"2024-04-24T23:15:42.107146+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:28:07.952854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,7],"acting":[2,1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.049954+0000","last_change":"2024-04-24T23:15:36.965850+0000","last_active":"2024-04-24T23:15:42.049954+0000","last_peered":"2024-04-24T23:15:42.049954+0000","last_clean":"2024-04-24T23:15:42.049954+0000","last_became_active":"2024-04-24T23:15:36.965547+0000","last_became_peered":"2024-04-24T23:15:36.965547+0000","last_unstale":"2024-04-24T23:15:42.049954+0000","last_undegraded":"2024-04-24T23:15:42.049954+0000","last_fullsized":"2024-04-24T23:15:42.049954+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:06:37.125573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"23'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.192302+0000","last_change":"2024-04-24T23:15:35.971040+0000","last_active":"2024-04-24T23:15:43.192302+0000","last_peered":"2024-04-24T23:15:43.192302+0000","last_clean":"2024-04-24T23:15:43.192302+0000","last_became_active":"2024-04-24T23:15:35.970184+0000","last_became_peered":"2024-04-24T23:15:35.970184+0000","last_unstale":"2024-04-24T23:15:43.192302+0000","last_undegraded":"2024-04-24T23:15:43.192302+0000","last_fullsized":"2024-04-24T23:15:43.192302+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:45:20.913759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.106686+0000","last_change":"2024-04-24T23:15:41.025813+0000","last_active":"2024-04-24T23:15:42.106686+0000","last_peered":"2024-04-24T23:15:42.106686+0000","last_clean":"2024-04-24T23:15:42.106686+0000","last_became_active":"2024-04-24T23:15:41.025137+0000","last_became_peered":"2024-04-24T23:15:41.025137+0000","last_unstale":"2024-04-24T23:15:42.106686+0000","last_undegraded":"2024-04-24T23:15:42.106686+0000","last_fullsized":"2024-04-24T23:15:42.106686+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:35:17.784854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,3,1],"acting":[2,6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.051163+0000","last_change":"2024-04-24T23:15:36.967476+0000","last_active":"2024-04-24T23:15:42.051163+0000","last_peered":"2024-04-24T23:15:42.051163+0000","last_clean":"2024-04-24T23:15:42.051163+0000","last_became_active":"2024-04-24T23:15:36.967098+0000","last_became_peered":"2024-04-24T23:15:36.967098+0000","last_unstale":"2024-04-24T23:15:42.051163+0000","last_undegraded":"2024-04-24T23:15:42.051163+0000","last_fullsized":"2024-04-24T23:15:42.051163+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:25:56.038903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"24'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.050091+0000","last_change":"2024-04-24T23:15:35.970630+0000","last_active":"2024-04-24T23:15:42.050091+0000","last_peered":"2024-04-24T23:15:42.050091+0000","last_clean":"2024-04-24T23:15:42.050091+0000","last_became_active":"2024-04-24T23:15:35.970430+0000","last_became_peered":"2024-04-24T23:15:35.970430+0000","last_unstale":"2024-04-24T23:15:42.050091+0000","last_undegraded":"2024-04-24T23:15:42.050091+0000","last_fullsized":"2024-04-24T23:15:42.050091+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:15:59.219084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.049399+0000","last_change":"2024-04-24T23:15:41.005022+0000","last_active":"2024-04-24T23:15:42.049399+0000","last_peered":"2024-04-24T23:15:42.049399+0000","last_clean":"2024-04-24T23:15:42.049399+0000","last_became_active":"2024-04-24T23:15:41.004648+0000","last_became_peered":"2024-04-24T23:15:41.004648+0000","last_unstale":"2024-04-24T23:15:42.049399+0000","last_undegraded":"2024-04-24T23:15:42.049399+0000","last_fullsized":"2024-04-24T23:15:42.049399+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:06:01.559760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6,3],"acting":[4,1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.051008+0000","last_change":"2024-04-24T23:15:36.967053+0000","last_active":"2024-04-24T23:15:42.051008+0000","last_peered":"2024-04-24T23:15:42.051008+0000","last_clean":"2024-04-24T23:15:42.051008+0000","last_became_active":"2024-04-24T23:15:36.966746+0000","last_became_peered":"2024-04-24T23:15:36.966746+0000","last_unstale":"2024-04-24T23:15:42.051008+0000","last_undegraded":"2024-04-24T23:15:42.051008+0000","last_fullsized":"2024-04-24T23:15:42.051008+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:16:08.779860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.192257+0000","last_change":"2024-04-24T23:15:35.965890+0000","last_active":"2024-04-24T23:15:43.192257+0000","last_peered":"2024-04-24T23:15:43.192257+0000","last_clean":"2024-04-24T23:15:43.192257+0000","last_became_active":"2024-04-24T23:15:35.965631+0000","last_became_peered":"2024-04-24T23:15:35.965631+0000","last_unstale":"2024-04-24T23:15:43.192257+0000","last_undegraded":"2024-04-24T23:15:43.192257+0000","last_fullsized":"2024-04-24T23:15:43.192257+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:17:35.650731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.050382+0000","last_change":"2024-04-24T23:15:41.011405+0000","last_active":"2024-04-24T23:15:42.050382+0000","last_peered":"2024-04-24T23:15:42.050382+0000","last_clean":"2024-04-24T23:15:42.050382+0000","last_became_active":"2024-04-24T23:15:41.010982+0000","last_became_peered":"2024-04-24T23:15:41.010982+0000","last_unstale":"2024-04-24T23:15:42.050382+0000","last_undegraded":"2024-04-24T23:15:42.050382+0000","last_fullsized":"2024-04-24T23:15:42.050382+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:08:00.914296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2,7],"acting":[6,0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.106945+0000","last_change":"2024-04-24T23:15:36.975244+0000","last_active":"2024-04-24T23:15:42.106945+0000","last_peered":"2024-04-24T23:15:42.106945+0000","last_clean":"2024-04-24T23:15:42.106945+0000","last_became_active":"2024-04-24T23:15:36.974872+0000","last_became_peered":"2024-04-24T23:15:36.974872+0000","last_unstale":"2024-04-24T23:15:42.106945+0000","last_undegraded":"2024-04-24T23:15:42.106945+0000","last_fullsized":"2024-04-24T23:15:42.106945+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:45:57.018205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.295386+0000","last_change":"2024-04-24T23:15:35.945792+0000","last_active":"2024-04-24T23:15:42.295386+0000","last_peered":"2024-04-24T23:15:42.295386+0000","last_clean":"2024-04-24T23:15:42.295386+0000","last_became_active":"2024-04-24T23:15:35.945094+0000","last_became_peered":"2024-04-24T23:15:35.945094+0000","last_unstale":"2024-04-24T23:15:42.295386+0000","last_undegraded":"2024-04-24T23:15:42.295386+0000","last_fullsized":"2024-04-24T23:15:42.295386+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:49:12.719127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.049802+0000","last_change":"2024-04-24T23:15:41.022105+0000","last_active":"2024-04-24T23:15:42.049802+0000","last_peered":"2024-04-24T23:15:42.049802+0000","last_clean":"2024-04-24T23:15:42.049802+0000","last_became_active":"2024-04-24T23:15:41.021645+0000","last_became_peered":"2024-04-24T23:15:41.021645+0000","last_unstale":"2024-04-24T23:15:42.049802+0000","last_undegraded":"2024-04-24T23:15:42.049802+0000","last_fullsized":"2024-04-24T23:15:42.049802+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:03:52.974835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5,7],"acting":[6,3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.192545+0000","last_change":"2024-04-24T23:15:36.991239+0000","last_active":"2024-04-24T23:15:43.192545+0000","last_peered":"2024-04-24T23:15:43.192545+0000","last_clean":"2024-04-24T23:15:43.192545+0000","last_became_active":"2024-04-24T23:15:36.990963+0000","last_became_peered":"2024-04-24T23:15:36.990963+0000","last_unstale":"2024-04-24T23:15:43.192545+0000","last_undegraded":"2024-04-24T23:15:43.192545+0000","last_fullsized":"2024-04-24T23:15:43.192545+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:53:28.101334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.191521+0000","last_change":"2024-04-24T23:15:35.957931+0000","last_active":"2024-04-24T23:15:43.191521+0000","last_peered":"2024-04-24T23:15:43.191521+0000","last_clean":"2024-04-24T23:15:43.191521+0000","last_became_active":"2024-04-24T23:15:35.957432+0000","last_became_peered":"2024-04-24T23:15:35.957432+0000","last_unstale":"2024-04-24T23:15:43.191521+0000","last_undegraded":"2024-04-24T23:15:43.191521+0000","last_fullsized":"2024-04-24T23:15:43.191521+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:11:38.773425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.050839+0000","last_change":"2024-04-24T23:15:41.007857+0000","last_active":"2024-04-24T23:15:42.050839+0000","last_peered":"2024-04-24T23:15:42.050839+0000","last_clean":"2024-04-24T23:15:42.050839+0000","last_became_active":"2024-04-24T23:15:41.007449+0000","last_became_peered":"2024-04-24T23:15:41.007449+0000","last_unstale":"2024-04-24T23:15:42.050839+0000","last_undegraded":"2024-04-24T23:15:42.050839+0000","last_fullsized":"2024-04-24T23:15:42.050839+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:05:06.145440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0,3],"acting":[6,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.295077+0000","last_change":"2024-04-24T23:15:36.970444+0000","last_active":"2024-04-24T23:15:42.295077+0000","last_peered":"2024-04-24T23:15:42.295077+0000","last_clean":"2024-04-24T23:15:42.295077+0000","last_became_active":"2024-04-24T23:15:36.970159+0000","last_became_peered":"2024-04-24T23:15:36.970159+0000","last_unstale":"2024-04-24T23:15:42.295077+0000","last_undegraded":"2024-04-24T23:15:42.295077+0000","last_fullsized":"2024-04-24T23:15:42.295077+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:37:02.956224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.187767+0000","last_change":"2024-04-24T23:15:35.945461+0000","last_active":"2024-04-24T23:15:43.187767+0000","last_peered":"2024-04-24T23:15:43.187767+0000","last_clean":"2024-04-24T23:15:43.187767+0000","last_became_active":"2024-04-24T23:15:35.945185+0000","last_became_peered":"2024-04-24T23:15:35.945185+0000","last_unstale":"2024-04-24T23:15:43.187767+0000","last_undegraded":"2024-04-24T23:15:43.187767+0000","last_fullsized":"2024-04-24T23:15:43.187767+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:37:34.175539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.050260+0000","last_change":"2024-04-24T23:15:41.014584+0000","last_active":"2024-04-24T23:15:42.050260+0000","last_peered":"2024-04-24T23:15:42.050260+0000","last_clean":"2024-04-24T23:15:42.050260+0000","last_became_active":"2024-04-24T23:15:41.014137+0000","last_became_peered":"2024-04-24T23:15:41.014137+0000","last_unstale":"2024-04-24T23:15:42.050260+0000","last_undegraded":"2024-04-24T23:15:42.050260+0000","last_fullsized":"2024-04-24T23:15:42.050260+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:23:50.387635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,7],"acting":[6,0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.049485+0000","last_change":"2024-04-24T23:15:36.981783+0000","last_active":"2024-04-24T23:15:42.049485+0000","last_peered":"2024-04-24T23:15:42.049485+0000","last_clean":"2024-04-24T23:15:42.049485+0000","last_became_active":"2024-04-24T23:15:36.981502+0000","last_became_peered":"2024-04-24T23:15:36.981502+0000","last_unstale":"2024-04-24T23:15:42.049485+0000","last_undegraded":"2024-04-24T23:15:42.049485+0000","last_fullsized":"2024-04-24T23:15:42.049485+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:56:08.483410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.190765+0000","last_change":"2024-04-24T23:15:35.960519+0000","last_active":"2024-04-24T23:15:43.190765+0000","last_peered":"2024-04-24T23:15:43.190765+0000","last_clean":"2024-04-24T23:15:43.190765+0000","last_became_active":"2024-04-24T23:15:35.960180+0000","last_became_peered":"2024-04-24T23:15:35.960180+0000","last_unstale":"2024-04-24T23:15:43.190765+0000","last_undegraded":"2024-04-24T23:15:43.190765+0000","last_fullsized":"2024-04-24T23:15:43.190765+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:50:00.199120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.295661+0000","last_change":"2024-04-24T23:15:41.012876+0000","last_active":"2024-04-24T23:15:42.295661+0000","last_peered":"2024-04-24T23:15:42.295661+0000","last_clean":"2024-04-24T23:15:42.295661+0000","last_became_active":"2024-04-24T23:15:41.012493+0000","last_became_peered":"2024-04-24T23:15:41.012493+0000","last_unstale":"2024-04-24T23:15:42.295661+0000","last_undegraded":"2024-04-24T23:15:42.295661+0000","last_fullsized":"2024-04-24T23:15:42.295661+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:21:28.287059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6,1],"acting":[5,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.050057+0000","last_change":"2024-04-24T23:15:36.981449+0000","last_active":"2024-04-24T23:15:42.050057+0000","last_peered":"2024-04-24T23:15:42.050057+0000","last_clean":"2024-04-24T23:15:42.050057+0000","last_became_active":"2024-04-24T23:15:36.980837+0000","last_became_peered":"2024-04-24T23:15:36.980837+0000","last_unstale":"2024-04-24T23:15:42.050057+0000","last_undegraded":"2024-04-24T23:15:42.050057+0000","last_fullsized":"2024-04-24T23:15:42.050057+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:15:13.513341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.192038+0000","last_change":"2024-04-24T23:15:35.952947+0000","last_active":"2024-04-24T23:15:43.192038+0000","last_peered":"2024-04-24T23:15:43.192038+0000","last_clean":"2024-04-24T23:15:43.192038+0000","last_became_active":"2024-04-24T23:15:35.952655+0000","last_became_peered":"2024-04-24T23:15:35.952655+0000","last_unstale":"2024-04-24T23:15:43.192038+0000","last_undegraded":"2024-04-24T23:15:43.192038+0000","last_fullsized":"2024-04-24T23:15:43.192038+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:50:10.939396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":178,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.732636+0000","last_change":"2024-04-24T23:15:25.826673+0000","last_active":"2024-04-24T23:15:42.732636+0000","last_peered":"2024-04-24T23:15:42.732636+0000","last_clean":"2024-04-24T23:15:42.732636+0000","last_became_active":"2024-04-24T23:15:25.820030+0000","last_became_peered":"2024-04-24T23:15:25.820030+0000","last_unstale":"2024-04-24T23:15:42.732636+0000","last_undegraded":"2024-04-24T23:15:42.732636+0000","last_fullsized":"2024-04-24T23:15:42.732636+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:24.790928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:24.790928+0000","last_clean_scrub_stamp":"2024-04-24T23:15:24.790928+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:37:24.054332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.192066+0000","last_change":"2024-04-24T23:15:41.024712+0000","last_active":"2024-04-24T23:15:43.192066+0000","last_peered":"2024-04-24T23:15:43.192066+0000","last_clean":"2024-04-24T23:15:43.192066+0000","last_became_active":"2024-04-24T23:15:41.022697+0000","last_became_peered":"2024-04-24T23:15:41.022697+0000","last_unstale":"2024-04-24T23:15:43.192066+0000","last_undegraded":"2024-04-24T23:15:43.192066+0000","last_fullsized":"2024-04-24T23:15:43.192066+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:37:08.231105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2,0],"acting":[3,1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.191847+0000","last_change":"2024-04-24T23:15:37.000489+0000","last_active":"2024-04-24T23:15:43.191847+0000","last_peered":"2024-04-24T23:15:43.191847+0000","last_clean":"2024-04-24T23:15:43.191847+0000","last_became_active":"2024-04-24T23:15:37.000179+0000","last_became_peered":"2024-04-24T23:15:37.000179+0000","last_unstale":"2024-04-24T23:15:43.191847+0000","last_undegraded":"2024-04-24T23:15:43.191847+0000","last_fullsized":"2024-04-24T23:15:43.191847+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:06:37.125573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.051120+0000","last_change":"2024-04-24T23:15:35.966020+0000","last_active":"2024-04-24T23:15:42.051120+0000","last_peered":"2024-04-24T23:15:42.051120+0000","last_clean":"2024-04-24T23:15:42.051120+0000","last_became_active":"2024-04-24T23:15:35.965758+0000","last_became_peered":"2024-04-24T23:15:35.965758+0000","last_unstale":"2024-04-24T23:15:42.051120+0000","last_undegraded":"2024-04-24T23:15:42.051120+0000","last_fullsized":"2024-04-24T23:15:42.051120+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:39:21.233461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.106667+0000","last_change":"2024-04-24T23:15:41.024489+0000","last_active":"2024-04-24T23:15:42.106667+0000","last_peered":"2024-04-24T23:15:42.106667+0000","last_clean":"2024-04-24T23:15:42.106667+0000","last_became_active":"2024-04-24T23:15:41.024041+0000","last_became_peered":"2024-04-24T23:15:41.024041+0000","last_unstale":"2024-04-24T23:15:42.106667+0000","last_undegraded":"2024-04-24T23:15:42.106667+0000","last_fullsized":"2024-04-24T23:15:42.106667+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:27:18.197003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7,4],"acting":[2,3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.193516+0000","last_change":"2024-04-24T23:15:36.979887+0000","last_active":"2024-04-24T23:15:43.193516+0000","last_peered":"2024-04-24T23:15:43.193516+0000","last_clean":"2024-04-24T23:15:43.193516+0000","last_became_active":"2024-04-24T23:15:36.979611+0000","last_became_peered":"2024-04-24T23:15:36.979611+0000","last_unstale":"2024-04-24T23:15:43.193516+0000","last_undegraded":"2024-04-24T23:15:43.193516+0000","last_fullsized":"2024-04-24T23:15:43.193516+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:07:18.261077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"23'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.295532+0000","last_change":"2024-04-24T23:15:35.965192+0000","last_active":"2024-04-24T23:15:42.295532+0000","last_peered":"2024-04-24T23:15:42.295532+0000","last_clean":"2024-04-24T23:15:42.295532+0000","last_became_active":"2024-04-24T23:15:35.964873+0000","last_became_peered":"2024-04-24T23:15:35.964873+0000","last_unstale":"2024-04-24T23:15:42.295532+0000","last_undegraded":"2024-04-24T23:15:42.295532+0000","last_fullsized":"2024-04-24T23:15:42.295532+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:14:27.884944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.295549+0000","last_change":"2024-04-24T23:15:41.004002+0000","last_active":"2024-04-24T23:15:42.295549+0000","last_peered":"2024-04-24T23:15:42.295549+0000","last_clean":"2024-04-24T23:15:42.295549+0000","last_became_active":"2024-04-24T23:15:41.003335+0000","last_became_peered":"2024-04-24T23:15:41.003335+0000","last_unstale":"2024-04-24T23:15:42.295549+0000","last_undegraded":"2024-04-24T23:15:42.295549+0000","last_fullsized":"2024-04-24T23:15:42.295549+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:49:36.206903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1,3],"acting":[5,4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.193583+0000","last_change":"2024-04-24T23:15:37.002854+0000","last_active":"2024-04-24T23:15:43.193583+0000","last_peered":"2024-04-24T23:15:43.193583+0000","last_clean":"2024-04-24T23:15:43.193583+0000","last_became_active":"2024-04-24T23:15:37.002583+0000","last_became_peered":"2024-04-24T23:15:37.002583+0000","last_unstale":"2024-04-24T23:15:43.193583+0000","last_undegraded":"2024-04-24T23:15:43.193583+0000","last_fullsized":"2024-04-24T23:15:43.193583+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:27:04.251547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"23'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.050875+0000","last_change":"2024-04-24T23:15:35.948752+0000","last_active":"2024-04-24T23:15:42.050875+0000","last_peered":"2024-04-24T23:15:42.050875+0000","last_clean":"2024-04-24T23:15:42.050875+0000","last_became_active":"2024-04-24T23:15:35.948438+0000","last_became_peered":"2024-04-24T23:15:35.948438+0000","last_unstale":"2024-04-24T23:15:42.050875+0000","last_undegraded":"2024-04-24T23:15:42.050875+0000","last_fullsized":"2024-04-24T23:15:42.050875+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:45:56.006278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.190120+0000","last_change":"2024-04-24T23:15:41.017524+0000","last_active":"2024-04-24T23:15:43.190120+0000","last_peered":"2024-04-24T23:15:43.190120+0000","last_clean":"2024-04-24T23:15:43.190120+0000","last_became_active":"2024-04-24T23:15:41.017092+0000","last_became_peered":"2024-04-24T23:15:41.017092+0000","last_unstale":"2024-04-24T23:15:43.190120+0000","last_undegraded":"2024-04-24T23:15:43.190120+0000","last_fullsized":"2024-04-24T23:15:43.190120+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:04:53.235038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.107289+0000","last_change":"2024-04-24T23:15:36.990441+0000","last_active":"2024-04-24T23:15:42.107289+0000","last_peered":"2024-04-24T23:15:42.107289+0000","last_clean":"2024-04-24T23:15:42.107289+0000","last_became_active":"2024-04-24T23:15:36.989863+0000","last_became_peered":"2024-04-24T23:15:36.989863+0000","last_unstale":"2024-04-24T23:15:42.107289+0000","last_undegraded":"2024-04-24T23:15:42.107289+0000","last_fullsized":"2024-04-24T23:15:42.107289+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:50:07.975347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.f","version":"23'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.193960+0000","last_change":"2024-04-24T23:15:35.969317+0000","last_active":"2024-04-24T23:15:43.193960+0000","last_peered":"2024-04-24T23:15:43.193960+0000","last_clean":"2024-04-24T23:15:43.193960+0000","last_became_active":"2024-04-24T23:15:35.968994+0000","last_became_peered":"2024-04-24T23:15:35.968994+0000","last_unstale":"2024-04-24T23:15:43.193960+0000","last_undegraded":"2024-04-24T23:15:43.193960+0000","last_fullsized":"2024-04-24T23:15:43.193960+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:51:57.100772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.733165+0000","last_change":"2024-04-24T23:15:41.010103+0000","last_active":"2024-04-24T23:15:42.733165+0000","last_peered":"2024-04-24T23:15:42.733165+0000","last_clean":"2024-04-24T23:15:42.733165+0000","last_became_active":"2024-04-24T23:15:41.009328+0000","last_became_peered":"2024-04-24T23:15:41.009328+0000","last_unstale":"2024-04-24T23:15:42.733165+0000","last_undegraded":"2024-04-24T23:15:42.733165+0000","last_fullsized":"2024-04-24T23:15:42.733165+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:46:49.579501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,2],"acting":[1,5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.190520+0000","last_change":"2024-04-24T23:15:36.996765+0000","last_active":"2024-04-24T23:15:43.190520+0000","last_peered":"2024-04-24T23:15:43.190520+0000","last_clean":"2024-04-24T23:15:43.190520+0000","last_became_active":"2024-04-24T23:15:36.995801+0000","last_became_peered":"2024-04-24T23:15:36.995801+0000","last_unstale":"2024-04-24T23:15:43.190520+0000","last_undegraded":"2024-04-24T23:15:43.190520+0000","last_fullsized":"2024-04-24T23:15:43.190520+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:25:19.421667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"23'8","reported_seq":30,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.049288+0000","last_change":"2024-04-24T23:15:35.944023+0000","last_active":"2024-04-24T23:15:42.049288+0000","last_peered":"2024-04-24T23:15:42.049288+0000","last_clean":"2024-04-24T23:15:42.049288+0000","last_became_active":"2024-04-24T23:15:35.943710+0000","last_became_peered":"2024-04-24T23:15:35.943710+0000","last_unstale":"2024-04-24T23:15:42.049288+0000","last_undegraded":"2024-04-24T23:15:42.049288+0000","last_fullsized":"2024-04-24T23:15:42.049288+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:46:48.090958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.295767+0000","last_change":"2024-04-24T23:15:41.013027+0000","last_active":"2024-04-24T23:15:42.295767+0000","last_peered":"2024-04-24T23:15:42.295767+0000","last_clean":"2024-04-24T23:15:42.295767+0000","last_became_active":"2024-04-24T23:15:41.012587+0000","last_became_peered":"2024-04-24T23:15:41.012587+0000","last_unstale":"2024-04-24T23:15:42.295767+0000","last_undegraded":"2024-04-24T23:15:42.295767+0000","last_fullsized":"2024-04-24T23:15:42.295767+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:27:04.276246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1,6],"acting":[5,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.732647+0000","last_change":"2024-04-24T23:15:36.979702+0000","last_active":"2024-04-24T23:15:42.732647+0000","last_peered":"2024-04-24T23:15:42.732647+0000","last_clean":"2024-04-24T23:15:42.732647+0000","last_became_active":"2024-04-24T23:15:36.979356+0000","last_became_peered":"2024-04-24T23:15:36.979356+0000","last_unstale":"2024-04-24T23:15:42.732647+0000","last_undegraded":"2024-04-24T23:15:42.732647+0000","last_fullsized":"2024-04-24T23:15:42.732647+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:14:24.309192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.193698+0000","last_change":"2024-04-24T23:15:35.973026+0000","last_active":"2024-04-24T23:15:43.193698+0000","last_peered":"2024-04-24T23:15:43.193698+0000","last_clean":"2024-04-24T23:15:43.193698+0000","last_became_active":"2024-04-24T23:15:35.972655+0000","last_became_peered":"2024-04-24T23:15:35.972655+0000","last_unstale":"2024-04-24T23:15:43.193698+0000","last_undegraded":"2024-04-24T23:15:43.193698+0000","last_fullsized":"2024-04-24T23:15:43.193698+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:25:34.352908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.295901+0000","last_change":"2024-04-24T23:15:41.004221+0000","last_active":"2024-04-24T23:15:42.295901+0000","last_peered":"2024-04-24T23:15:42.295901+0000","last_clean":"2024-04-24T23:15:42.295901+0000","last_became_active":"2024-04-24T23:15:41.003790+0000","last_became_peered":"2024-04-24T23:15:41.003790+0000","last_unstale":"2024-04-24T23:15:42.295901+0000","last_undegraded":"2024-04-24T23:15:42.295901+0000","last_fullsized":"2024-04-24T23:15:42.295901+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:28:58.866024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6,4],"acting":[5,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.191414+0000","last_change":"2024-04-24T23:15:36.997389+0000","last_active":"2024-04-24T23:15:43.191414+0000","last_peered":"2024-04-24T23:15:43.191414+0000","last_clean":"2024-04-24T23:15:43.191414+0000","last_became_active":"2024-04-24T23:15:36.995923+0000","last_became_peered":"2024-04-24T23:15:36.995923+0000","last_unstale":"2024-04-24T23:15:43.191414+0000","last_undegraded":"2024-04-24T23:15:43.191414+0000","last_fullsized":"2024-04-24T23:15:43.191414+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:55:47.134320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.191422+0000","last_change":"2024-04-24T23:15:35.957715+0000","last_active":"2024-04-24T23:15:43.191422+0000","last_peered":"2024-04-24T23:15:43.191422+0000","last_clean":"2024-04-24T23:15:43.191422+0000","last_became_active":"2024-04-24T23:15:35.957088+0000","last_became_peered":"2024-04-24T23:15:35.957088+0000","last_unstale":"2024-04-24T23:15:43.191422+0000","last_undegraded":"2024-04-24T23:15:43.191422+0000","last_fullsized":"2024-04-24T23:15:43.191422+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:30:34.483669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.050833+0000","last_change":"2024-04-24T23:15:41.008165+0000","last_active":"2024-04-24T23:15:42.050833+0000","last_peered":"2024-04-24T23:15:42.050833+0000","last_clean":"2024-04-24T23:15:42.050833+0000","last_became_active":"2024-04-24T23:15:41.007555+0000","last_became_peered":"2024-04-24T23:15:41.007555+0000","last_unstale":"2024-04-24T23:15:42.050833+0000","last_undegraded":"2024-04-24T23:15:42.050833+0000","last_fullsized":"2024-04-24T23:15:42.050833+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:18:08.195747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0,7],"acting":[6,4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.050006+0000","last_change":"2024-04-24T23:15:36.991382+0000","last_active":"2024-04-24T23:15:42.050006+0000","last_peered":"2024-04-24T23:15:42.050006+0000","last_clean":"2024-04-24T23:15:42.050006+0000","last_became_active":"2024-04-24T23:15:36.991115+0000","last_became_peered":"2024-04-24T23:15:36.991115+0000","last_unstale":"2024-04-24T23:15:42.050006+0000","last_undegraded":"2024-04-24T23:15:42.050006+0000","last_fullsized":"2024-04-24T23:15:42.050006+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:21:51.863947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"23'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.188008+0000","last_change":"2024-04-24T23:15:35.964245+0000","last_active":"2024-04-24T23:15:43.188008+0000","last_peered":"2024-04-24T23:15:43.188008+0000","last_clean":"2024-04-24T23:15:43.188008+0000","last_became_active":"2024-04-24T23:15:35.963907+0000","last_became_peered":"2024-04-24T23:15:35.963907+0000","last_unstale":"2024-04-24T23:15:43.188008+0000","last_undegraded":"2024-04-24T23:15:43.188008+0000","last_fullsized":"2024-04-24T23:15:43.188008+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:15:00.913899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.192087+0000","last_change":"2024-04-24T23:15:41.021981+0000","last_active":"2024-04-24T23:15:43.192087+0000","last_peered":"2024-04-24T23:15:43.192087+0000","last_clean":"2024-04-24T23:15:43.192087+0000","last_became_active":"2024-04-24T23:15:41.019193+0000","last_became_peered":"2024-04-24T23:15:41.019193+0000","last_unstale":"2024-04-24T23:15:43.192087+0000","last_undegraded":"2024-04-24T23:15:43.192087+0000","last_fullsized":"2024-04-24T23:15:43.192087+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:04:39.828502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5,6],"acting":[3,0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.193203+0000","last_change":"2024-04-24T23:15:36.986833+0000","last_active":"2024-04-24T23:15:43.193203+0000","last_peered":"2024-04-24T23:15:43.193203+0000","last_clean":"2024-04-24T23:15:43.193203+0000","last_became_active":"2024-04-24T23:15:36.986484+0000","last_became_peered":"2024-04-24T23:15:36.986484+0000","last_unstale":"2024-04-24T23:15:43.193203+0000","last_undegraded":"2024-04-24T23:15:43.193203+0000","last_fullsized":"2024-04-24T23:15:43.193203+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:12:32.245267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.14","version":"24'10","reported_seq":32,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:59.011711+0000","last_change":"2024-04-24T23:15:35.957124+0000","last_active":"2024-04-24T23:15:59.011711+0000","last_peered":"2024-04-24T23:15:59.011711+0000","last_clean":"2024-04-24T23:15:59.011711+0000","last_became_active":"2024-04-24T23:15:35.951923+0000","last_became_peered":"2024-04-24T23:15:35.951923+0000","last_unstale":"2024-04-24T23:15:59.011711+0000","last_undegraded":"2024-04-24T23:15:59.011711+0000","last_fullsized":"2024-04-24T23:15:59.011711+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:44:16.820640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.049936+0000","last_change":"2024-04-24T23:15:41.015494+0000","last_active":"2024-04-24T23:15:42.049936+0000","last_peered":"2024-04-24T23:15:42.049936+0000","last_clean":"2024-04-24T23:15:42.049936+0000","last_became_active":"2024-04-24T23:15:41.015179+0000","last_became_peered":"2024-04-24T23:15:41.015179+0000","last_unstale":"2024-04-24T23:15:42.049936+0000","last_undegraded":"2024-04-24T23:15:42.049936+0000","last_fullsized":"2024-04-24T23:15:42.049936+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:18:32.208506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5,0],"acting":[4,3,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.190450+0000","last_change":"2024-04-24T23:15:36.999334+0000","last_active":"2024-04-24T23:15:43.190450+0000","last_peered":"2024-04-24T23:15:43.190450+0000","last_clean":"2024-04-24T23:15:43.190450+0000","last_became_active":"2024-04-24T23:15:36.995470+0000","last_became_peered":"2024-04-24T23:15:36.995470+0000","last_unstale":"2024-04-24T23:15:43.190450+0000","last_undegraded":"2024-04-24T23:15:43.190450+0000","last_fullsized":"2024-04-24T23:15:43.190450+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:19:57.973402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.15","version":"23'3","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.187978+0000","last_change":"2024-04-24T23:15:35.946041+0000","last_active":"2024-04-24T23:15:43.187978+0000","last_peered":"2024-04-24T23:15:43.187978+0000","last_clean":"2024-04-24T23:15:43.187978+0000","last_became_active":"2024-04-24T23:15:35.944805+0000","last_became_peered":"2024-04-24T23:15:35.944805+0000","last_unstale":"2024-04-24T23:15:43.187978+0000","last_undegraded":"2024-04-24T23:15:43.187978+0000","last_fullsized":"2024-04-24T23:15:43.187978+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:52:19.261306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.107105+0000","last_change":"2024-04-24T23:15:41.024392+0000","last_active":"2024-04-24T23:15:42.107105+0000","last_peered":"2024-04-24T23:15:42.107105+0000","last_clean":"2024-04-24T23:15:42.107105+0000","last_became_active":"2024-04-24T23:15:41.023951+0000","last_became_peered":"2024-04-24T23:15:41.023951+0000","last_unstale":"2024-04-24T23:15:42.107105+0000","last_undegraded":"2024-04-24T23:15:42.107105+0000","last_fullsized":"2024-04-24T23:15:42.107105+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:24:17.810117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1,7],"acting":[2,3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.295845+0000","last_change":"2024-04-24T23:15:36.992243+0000","last_active":"2024-04-24T23:15:42.295845+0000","last_peered":"2024-04-24T23:15:42.295845+0000","last_clean":"2024-04-24T23:15:42.295845+0000","last_became_active":"2024-04-24T23:15:36.992002+0000","last_became_peered":"2024-04-24T23:15:36.992002+0000","last_unstale":"2024-04-24T23:15:42.295845+0000","last_undegraded":"2024-04-24T23:15:42.295845+0000","last_fullsized":"2024-04-24T23:15:42.295845+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:52:15.911009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.193044+0000","last_change":"2024-04-24T23:15:35.967408+0000","last_active":"2024-04-24T23:15:43.193044+0000","last_peered":"2024-04-24T23:15:43.193044+0000","last_clean":"2024-04-24T23:15:43.193044+0000","last_became_active":"2024-04-24T23:15:35.966646+0000","last_became_peered":"2024-04-24T23:15:35.966646+0000","last_unstale":"2024-04-24T23:15:43.193044+0000","last_undegraded":"2024-04-24T23:15:43.193044+0000","last_fullsized":"2024-04-24T23:15:43.193044+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:07:07.931773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.187622+0000","last_change":"2024-04-24T23:15:41.021697+0000","last_active":"2024-04-24T23:15:43.187622+0000","last_peered":"2024-04-24T23:15:43.187622+0000","last_clean":"2024-04-24T23:15:43.187622+0000","last_became_active":"2024-04-24T23:15:41.021281+0000","last_became_peered":"2024-04-24T23:15:41.021281+0000","last_unstale":"2024-04-24T23:15:43.187622+0000","last_undegraded":"2024-04-24T23:15:43.187622+0000","last_fullsized":"2024-04-24T23:15:43.187622+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:59:37.306593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3,2],"acting":[7,1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.107218+0000","last_change":"2024-04-24T23:15:36.990988+0000","last_active":"2024-04-24T23:15:42.107218+0000","last_peered":"2024-04-24T23:15:42.107218+0000","last_clean":"2024-04-24T23:15:42.107218+0000","last_became_active":"2024-04-24T23:15:36.990711+0000","last_became_peered":"2024-04-24T23:15:36.990711+0000","last_unstale":"2024-04-24T23:15:42.107218+0000","last_undegraded":"2024-04-24T23:15:42.107218+0000","last_fullsized":"2024-04-24T23:15:42.107218+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:33:20.663776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.734116+0000","last_change":"2024-04-24T23:15:35.957838+0000","last_active":"2024-04-24T23:15:42.734116+0000","last_peered":"2024-04-24T23:15:42.734116+0000","last_clean":"2024-04-24T23:15:42.734116+0000","last_became_active":"2024-04-24T23:15:35.956870+0000","last_became_peered":"2024-04-24T23:15:35.956870+0000","last_unstale":"2024-04-24T23:15:42.734116+0000","last_undegraded":"2024-04-24T23:15:42.734116+0000","last_fullsized":"2024-04-24T23:15:42.734116+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:56:03.889693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.049546+0000","last_change":"2024-04-24T23:15:41.006098+0000","last_active":"2024-04-24T23:15:42.049546+0000","last_peered":"2024-04-24T23:15:42.049546+0000","last_clean":"2024-04-24T23:15:42.049546+0000","last_became_active":"2024-04-24T23:15:41.005578+0000","last_became_peered":"2024-04-24T23:15:41.005578+0000","last_unstale":"2024-04-24T23:15:42.049546+0000","last_undegraded":"2024-04-24T23:15:42.049546+0000","last_fullsized":"2024-04-24T23:15:42.049546+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:21:12.802112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3,7],"acting":[4,0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.049896+0000","last_change":"2024-04-24T23:15:41.022122+0000","last_active":"2024-04-24T23:15:42.049896+0000","last_peered":"2024-04-24T23:15:42.049896+0000","last_clean":"2024-04-24T23:15:42.049896+0000","last_became_active":"2024-04-24T23:15:41.021698+0000","last_became_peered":"2024-04-24T23:15:41.021698+0000","last_unstale":"2024-04-24T23:15:42.049896+0000","last_undegraded":"2024-04-24T23:15:42.049896+0000","last_fullsized":"2024-04-24T23:15:42.049896+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:13:56.730050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5,3],"acting":[4,1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.049600+0000","last_change":"2024-04-24T23:15:36.979816+0000","last_active":"2024-04-24T23:15:42.049600+0000","last_peered":"2024-04-24T23:15:42.049600+0000","last_clean":"2024-04-24T23:15:42.049600+0000","last_became_active":"2024-04-24T23:15:36.979553+0000","last_became_peered":"2024-04-24T23:15:36.979553+0000","last_unstale":"2024-04-24T23:15:42.049600+0000","last_undegraded":"2024-04-24T23:15:42.049600+0000","last_fullsized":"2024-04-24T23:15:42.049600+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:55:47.134320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.190578+0000","last_change":"2024-04-24T23:15:35.964744+0000","last_active":"2024-04-24T23:15:43.190578+0000","last_peered":"2024-04-24T23:15:43.190578+0000","last_clean":"2024-04-24T23:15:43.190578+0000","last_became_active":"2024-04-24T23:15:35.964434+0000","last_became_peered":"2024-04-24T23:15:35.964434+0000","last_unstale":"2024-04-24T23:15:43.190578+0000","last_undegraded":"2024-04-24T23:15:43.190578+0000","last_fullsized":"2024-04-24T23:15:43.190578+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:26:32.587197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.191463+0000","last_change":"2024-04-24T23:15:41.022032+0000","last_active":"2024-04-24T23:15:43.191463+0000","last_peered":"2024-04-24T23:15:43.191463+0000","last_clean":"2024-04-24T23:15:43.191463+0000","last_became_active":"2024-04-24T23:15:41.021616+0000","last_became_peered":"2024-04-24T23:15:41.021616+0000","last_unstale":"2024-04-24T23:15:43.191463+0000","last_undegraded":"2024-04-24T23:15:43.191463+0000","last_fullsized":"2024-04-24T23:15:43.191463+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:08:59.242392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5,1],"acting":[3,2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.295164+0000","last_change":"2024-04-24T23:15:36.967198+0000","last_active":"2024-04-24T23:15:42.295164+0000","last_peered":"2024-04-24T23:15:42.295164+0000","last_clean":"2024-04-24T23:15:42.295164+0000","last_became_active":"2024-04-24T23:15:36.966910+0000","last_became_peered":"2024-04-24T23:15:36.966910+0000","last_unstale":"2024-04-24T23:15:42.295164+0000","last_undegraded":"2024-04-24T23:15:42.295164+0000","last_fullsized":"2024-04-24T23:15:42.295164+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:51:19.788437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.049504+0000","last_change":"2024-04-24T23:15:35.967169+0000","last_active":"2024-04-24T23:15:42.049504+0000","last_peered":"2024-04-24T23:15:42.049504+0000","last_clean":"2024-04-24T23:15:42.049504+0000","last_became_active":"2024-04-24T23:15:35.966939+0000","last_became_peered":"2024-04-24T23:15:35.966939+0000","last_unstale":"2024-04-24T23:15:42.049504+0000","last_undegraded":"2024-04-24T23:15:42.049504+0000","last_fullsized":"2024-04-24T23:15:42.049504+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:38:55.460569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"23'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.191308+0000","last_change":"2024-04-24T23:15:35.971236+0000","last_active":"2024-04-24T23:15:43.191308+0000","last_peered":"2024-04-24T23:15:43.191308+0000","last_clean":"2024-04-24T23:15:43.191308+0000","last_became_active":"2024-04-24T23:15:35.970908+0000","last_became_peered":"2024-04-24T23:15:35.970908+0000","last_unstale":"2024-04-24T23:15:43.191308+0000","last_undegraded":"2024-04-24T23:15:43.191308+0000","last_fullsized":"2024-04-24T23:15:43.191308+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:14:15.548024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.050320+0000","last_change":"2024-04-24T23:15:36.988156+0000","last_active":"2024-04-24T23:15:42.050320+0000","last_peered":"2024-04-24T23:15:42.050320+0000","last_clean":"2024-04-24T23:15:42.050320+0000","last_became_active":"2024-04-24T23:15:36.987896+0000","last_became_peered":"2024-04-24T23:15:36.987896+0000","last_unstale":"2024-04-24T23:15:42.050320+0000","last_undegraded":"2024-04-24T23:15:42.050320+0000","last_fullsized":"2024-04-24T23:15:42.050320+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:22:06.869644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.050441+0000","last_change":"2024-04-24T23:15:40.997559+0000","last_active":"2024-04-24T23:15:42.050441+0000","last_peered":"2024-04-24T23:15:42.050441+0000","last_clean":"2024-04-24T23:15:42.050441+0000","last_became_active":"2024-04-24T23:15:40.997102+0000","last_became_peered":"2024-04-24T23:15:40.997102+0000","last_unstale":"2024-04-24T23:15:42.050441+0000","last_undegraded":"2024-04-24T23:15:42.050441+0000","last_fullsized":"2024-04-24T23:15:42.050441+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:25:02.046761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,7],"acting":[4,6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"23'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.295440+0000","last_change":"2024-04-24T23:15:35.944842+0000","last_active":"2024-04-24T23:15:42.295440+0000","last_peered":"2024-04-24T23:15:42.295440+0000","last_clean":"2024-04-24T23:15:42.295440+0000","last_became_active":"2024-04-24T23:15:35.944563+0000","last_became_peered":"2024-04-24T23:15:35.944563+0000","last_unstale":"2024-04-24T23:15:42.295440+0000","last_undegraded":"2024-04-24T23:15:42.295440+0000","last_fullsized":"2024-04-24T23:15:42.295440+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:40:58.933384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.732992+0000","last_change":"2024-04-24T23:15:36.981103+0000","last_active":"2024-04-24T23:15:42.732992+0000","last_peered":"2024-04-24T23:15:42.732992+0000","last_clean":"2024-04-24T23:15:42.732992+0000","last_became_active":"2024-04-24T23:15:36.979637+0000","last_became_peered":"2024-04-24T23:15:36.979637+0000","last_unstale":"2024-04-24T23:15:42.732992+0000","last_undegraded":"2024-04-24T23:15:42.732992+0000","last_fullsized":"2024-04-24T23:15:42.732992+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:09:39.750272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.188153+0000","last_change":"2024-04-24T23:15:41.017087+0000","last_active":"2024-04-24T23:15:43.188153+0000","last_peered":"2024-04-24T23:15:43.188153+0000","last_clean":"2024-04-24T23:15:43.188153+0000","last_became_active":"2024-04-24T23:15:41.016696+0000","last_became_peered":"2024-04-24T23:15:41.016696+0000","last_unstale":"2024-04-24T23:15:43.188153+0000","last_undegraded":"2024-04-24T23:15:43.188153+0000","last_fullsized":"2024-04-24T23:15:43.188153+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:05:57.823447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1,0],"acting":[7,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.106609+0000","last_change":"2024-04-24T23:15:36.970211+0000","last_active":"2024-04-24T23:15:42.106609+0000","last_peered":"2024-04-24T23:15:42.106609+0000","last_clean":"2024-04-24T23:15:42.106609+0000","last_became_active":"2024-04-24T23:15:36.969090+0000","last_became_peered":"2024-04-24T23:15:36.969090+0000","last_unstale":"2024-04-24T23:15:42.106609+0000","last_undegraded":"2024-04-24T23:15:42.106609+0000","last_fullsized":"2024-04-24T23:15:42.106609+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:16:00.231012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.187632+0000","last_change":"2024-04-24T23:15:36.982528+0000","last_active":"2024-04-24T23:15:43.187632+0000","last_peered":"2024-04-24T23:15:43.187632+0000","last_clean":"2024-04-24T23:15:43.187632+0000","last_became_active":"2024-04-24T23:15:36.982299+0000","last_became_peered":"2024-04-24T23:15:36.982299+0000","last_unstale":"2024-04-24T23:15:43.187632+0000","last_undegraded":"2024-04-24T23:15:43.187632+0000","last_fullsized":"2024-04-24T23:15:43.187632+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:02:17.732324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.191005+0000","last_change":"2024-04-24T23:15:36.989899+0000","last_active":"2024-04-24T23:15:43.191005+0000","last_peered":"2024-04-24T23:15:43.191005+0000","last_clean":"2024-04-24T23:15:43.191005+0000","last_became_active":"2024-04-24T23:15:36.989557+0000","last_became_peered":"2024-04-24T23:15:36.989557+0000","last_unstale":"2024-04-24T23:15:43.191005+0000","last_undegraded":"2024-04-24T23:15:43.191005+0000","last_fullsized":"2024-04-24T23:15:43.191005+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:15:13.513341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.295325+0000","last_change":"2024-04-24T23:15:36.969628+0000","last_active":"2024-04-24T23:15:42.295325+0000","last_peered":"2024-04-24T23:15:42.295325+0000","last_clean":"2024-04-24T23:15:42.295325+0000","last_became_active":"2024-04-24T23:15:36.969327+0000","last_became_peered":"2024-04-24T23:15:36.969327+0000","last_unstale":"2024-04-24T23:15:42.295325+0000","last_undegraded":"2024-04-24T23:15:42.295325+0000","last_fullsized":"2024-04-24T23:15:42.295325+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:08:47.021999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.050888+0000","last_change":"2024-04-24T23:15:36.986343+0000","last_active":"2024-04-24T23:15:42.050888+0000","last_peered":"2024-04-24T23:15:42.050888+0000","last_clean":"2024-04-24T23:15:42.050888+0000","last_became_active":"2024-04-24T23:15:36.986026+0000","last_became_peered":"2024-04-24T23:15:36.986026+0000","last_unstale":"2024-04-24T23:15:42.050888+0000","last_undegraded":"2024-04-24T23:15:42.050888+0000","last_fullsized":"2024-04-24T23:15:42.050888+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:10:00.347826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.049809+0000","last_change":"2024-04-24T23:15:36.968538+0000","last_active":"2024-04-24T23:15:42.049809+0000","last_peered":"2024-04-24T23:15:42.049809+0000","last_clean":"2024-04-24T23:15:42.049809+0000","last_became_active":"2024-04-24T23:15:36.968263+0000","last_became_peered":"2024-04-24T23:15:36.968263+0000","last_unstale":"2024-04-24T23:15:42.049809+0000","last_undegraded":"2024-04-24T23:15:42.049809+0000","last_fullsized":"2024-04-24T23:15:42.049809+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:07:43.731738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.107186+0000","last_change":"2024-04-24T23:15:36.991008+0000","last_active":"2024-04-24T23:15:42.107186+0000","last_peered":"2024-04-24T23:15:42.107186+0000","last_clean":"2024-04-24T23:15:42.107186+0000","last_became_active":"2024-04-24T23:15:36.990764+0000","last_became_peered":"2024-04-24T23:15:36.990764+0000","last_unstale":"2024-04-24T23:15:42.107186+0000","last_undegraded":"2024-04-24T23:15:42.107186+0000","last_fullsized":"2024-04-24T23:15:42.107186+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:50:21.865233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.295685+0000","last_change":"2024-04-24T23:15:36.993550+0000","last_active":"2024-04-24T23:15:42.295685+0000","last_peered":"2024-04-24T23:15:42.295685+0000","last_clean":"2024-04-24T23:15:42.295685+0000","last_became_active":"2024-04-24T23:15:36.993310+0000","last_became_peered":"2024-04-24T23:15:36.993310+0000","last_unstale":"2024-04-24T23:15:42.295685+0000","last_undegraded":"2024-04-24T23:15:42.295685+0000","last_fullsized":"2024-04-24T23:15:42.295685+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:21:20.617940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.190455+0000","last_change":"2024-04-24T23:15:36.987810+0000","last_active":"2024-04-24T23:15:43.190455+0000","last_peered":"2024-04-24T23:15:43.190455+0000","last_clean":"2024-04-24T23:15:43.190455+0000","last_became_active":"2024-04-24T23:15:36.987540+0000","last_became_peered":"2024-04-24T23:15:36.987540+0000","last_unstale":"2024-04-24T23:15:43.190455+0000","last_undegraded":"2024-04-24T23:15:43.190455+0000","last_fullsized":"2024-04-24T23:15:43.190455+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:49:56.176021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.295070+0000","last_change":"2024-04-24T23:15:36.981187+0000","last_active":"2024-04-24T23:15:42.295070+0000","last_peered":"2024-04-24T23:15:42.295070+0000","last_clean":"2024-04-24T23:15:42.295070+0000","last_became_active":"2024-04-24T23:15:36.980838+0000","last_became_peered":"2024-04-24T23:15:36.980838+0000","last_unstale":"2024-04-24T23:15:42.295070+0000","last_undegraded":"2024-04-24T23:15:42.295070+0000","last_fullsized":"2024-04-24T23:15:42.295070+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:57:41.178367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.732913+0000","last_change":"2024-04-24T23:15:36.999238+0000","last_active":"2024-04-24T23:15:42.732913+0000","last_peered":"2024-04-24T23:15:42.732913+0000","last_clean":"2024-04-24T23:15:42.732913+0000","last_became_active":"2024-04-24T23:15:36.996245+0000","last_became_peered":"2024-04-24T23:15:36.996245+0000","last_unstale":"2024-04-24T23:15:42.732913+0000","last_undegraded":"2024-04-24T23:15:42.732913+0000","last_fullsized":"2024-04-24T23:15:42.732913+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:21:20.443640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.050214+0000","last_change":"2024-04-24T23:15:36.981219+0000","last_active":"2024-04-24T23:15:42.050214+0000","last_peered":"2024-04-24T23:15:42.050214+0000","last_clean":"2024-04-24T23:15:42.050214+0000","last_became_active":"2024-04-24T23:15:36.980911+0000","last_became_peered":"2024-04-24T23:15:36.980911+0000","last_unstale":"2024-04-24T23:15:42.050214+0000","last_undegraded":"2024-04-24T23:15:42.050214+0000","last_fullsized":"2024-04-24T23:15:42.050214+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:19:57.973402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.193222+0000","last_change":"2024-04-24T23:15:36.991418+0000","last_active":"2024-04-24T23:15:43.193222+0000","last_peered":"2024-04-24T23:15:43.193222+0000","last_clean":"2024-04-24T23:15:43.193222+0000","last_became_active":"2024-04-24T23:15:36.989792+0000","last_became_peered":"2024-04-24T23:15:36.989792+0000","last_unstale":"2024-04-24T23:15:43.193222+0000","last_undegraded":"2024-04-24T23:15:43.193222+0000","last_fullsized":"2024-04-24T23:15:43.193222+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:02:53.601311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.192125+0000","last_change":"2024-04-24T23:15:36.987810+0000","last_active":"2024-04-24T23:15:43.192125+0000","last_peered":"2024-04-24T23:15:43.192125+0000","last_clean":"2024-04-24T23:15:43.192125+0000","last_became_active":"2024-04-24T23:15:36.987494+0000","last_became_peered":"2024-04-24T23:15:36.987494+0000","last_unstale":"2024-04-24T23:15:43.192125+0000","last_undegraded":"2024-04-24T23:15:43.192125+0000","last_fullsized":"2024-04-24T23:15:43.192125+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:43:21.141172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.188217+0000","last_change":"2024-04-24T23:15:36.986576+0000","last_active":"2024-04-24T23:15:43.188217+0000","last_peered":"2024-04-24T23:15:43.188217+0000","last_clean":"2024-04-24T23:15:43.188217+0000","last_became_active":"2024-04-24T23:15:36.986272+0000","last_became_peered":"2024-04-24T23:15:36.986272+0000","last_unstale":"2024-04-24T23:15:43.188217+0000","last_undegraded":"2024-04-24T23:15:43.188217+0000","last_fullsized":"2024-04-24T23:15:43.188217+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:55:08.894945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.192001+0000","last_change":"2024-04-24T23:15:36.998812+0000","last_active":"2024-04-24T23:15:43.192001+0000","last_peered":"2024-04-24T23:15:43.192001+0000","last_clean":"2024-04-24T23:15:43.192001+0000","last_became_active":"2024-04-24T23:15:36.998456+0000","last_became_peered":"2024-04-24T23:15:36.998456+0000","last_unstale":"2024-04-24T23:15:43.192001+0000","last_undegraded":"2024-04-24T23:15:43.192001+0000","last_fullsized":"2024-04-24T23:15:43.192001+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:33:11.424218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.190262+0000","last_change":"2024-04-24T23:15:36.996521+0000","last_active":"2024-04-24T23:15:43.190262+0000","last_peered":"2024-04-24T23:15:43.190262+0000","last_clean":"2024-04-24T23:15:43.190262+0000","last_became_active":"2024-04-24T23:15:36.995507+0000","last_became_peered":"2024-04-24T23:15:36.995507+0000","last_unstale":"2024-04-24T23:15:43.190262+0000","last_undegraded":"2024-04-24T23:15:43.190262+0000","last_fullsized":"2024-04-24T23:15:43.190262+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:58:50.153471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.294996+0000","last_change":"2024-04-24T23:15:36.984994+0000","last_active":"2024-04-24T23:15:42.294996+0000","last_peered":"2024-04-24T23:15:42.294996+0000","last_clean":"2024-04-24T23:15:42.294996+0000","last_became_active":"2024-04-24T23:15:36.984741+0000","last_became_peered":"2024-04-24T23:15:36.984741+0000","last_unstale":"2024-04-24T23:15:42.294996+0000","last_undegraded":"2024-04-24T23:15:42.294996+0000","last_fullsized":"2024-04-24T23:15:42.294996+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:45:23.429836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.049917+0000","last_change":"2024-04-24T23:15:36.966165+0000","last_active":"2024-04-24T23:15:42.049917+0000","last_peered":"2024-04-24T23:15:42.049917+0000","last_clean":"2024-04-24T23:15:42.049917+0000","last_became_active":"2024-04-24T23:15:36.965877+0000","last_became_peered":"2024-04-24T23:15:36.965877+0000","last_unstale":"2024-04-24T23:15:42.049917+0000","last_undegraded":"2024-04-24T23:15:42.049917+0000","last_fullsized":"2024-04-24T23:15:42.049917+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:30:12.390978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.050537+0000","last_change":"2024-04-24T23:15:36.990667+0000","last_active":"2024-04-24T23:15:42.050537+0000","last_peered":"2024-04-24T23:15:42.050537+0000","last_clean":"2024-04-24T23:15:42.050537+0000","last_became_active":"2024-04-24T23:15:36.990383+0000","last_became_peered":"2024-04-24T23:15:36.990383+0000","last_unstale":"2024-04-24T23:15:42.050537+0000","last_undegraded":"2024-04-24T23:15:42.050537+0000","last_fullsized":"2024-04-24T23:15:42.050537+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:36:43.850046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.193024+0000","last_change":"2024-04-24T23:15:36.991686+0000","last_active":"2024-04-24T23:15:43.193024+0000","last_peered":"2024-04-24T23:15:43.193024+0000","last_clean":"2024-04-24T23:15:43.193024+0000","last_became_active":"2024-04-24T23:15:36.990155+0000","last_became_peered":"2024-04-24T23:15:36.990155+0000","last_unstale":"2024-04-24T23:15:43.193024+0000","last_undegraded":"2024-04-24T23:15:43.193024+0000","last_fullsized":"2024-04-24T23:15:43.193024+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:00:52.424019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.192909+0000","last_change":"2024-04-24T23:15:36.992958+0000","last_active":"2024-04-24T23:15:43.192909+0000","last_peered":"2024-04-24T23:15:43.192909+0000","last_clean":"2024-04-24T23:15:43.192909+0000","last_became_active":"2024-04-24T23:15:36.992608+0000","last_became_peered":"2024-04-24T23:15:36.992608+0000","last_unstale":"2024-04-24T23:15:43.192909+0000","last_undegraded":"2024-04-24T23:15:43.192909+0000","last_fullsized":"2024-04-24T23:15:43.192909+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:15:30.416763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.192849+0000","last_change":"2024-04-24T23:15:36.992943+0000","last_active":"2024-04-24T23:15:43.192849+0000","last_peered":"2024-04-24T23:15:43.192849+0000","last_clean":"2024-04-24T23:15:43.192849+0000","last_became_active":"2024-04-24T23:15:36.992567+0000","last_became_peered":"2024-04-24T23:15:36.992567+0000","last_unstale":"2024-04-24T23:15:43.192849+0000","last_undegraded":"2024-04-24T23:15:43.192849+0000","last_fullsized":"2024-04-24T23:15:43.192849+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:50:43.209232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.734006+0000","last_change":"2024-04-24T23:15:36.999697+0000","last_active":"2024-04-24T23:15:42.734006+0000","last_peered":"2024-04-24T23:15:42.734006+0000","last_clean":"2024-04-24T23:15:42.734006+0000","last_became_active":"2024-04-24T23:15:36.999413+0000","last_became_peered":"2024-04-24T23:15:36.999413+0000","last_unstale":"2024-04-24T23:15:42.734006+0000","last_undegraded":"2024-04-24T23:15:42.734006+0000","last_fullsized":"2024-04-24T23:15:42.734006+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:25:42.856471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.295006+0000","last_change":"2024-04-24T23:15:36.986742+0000","last_active":"2024-04-24T23:15:42.295006+0000","last_peered":"2024-04-24T23:15:42.295006+0000","last_clean":"2024-04-24T23:15:42.295006+0000","last_became_active":"2024-04-24T23:15:36.986013+0000","last_became_peered":"2024-04-24T23:15:36.986013+0000","last_unstale":"2024-04-24T23:15:42.295006+0000","last_undegraded":"2024-04-24T23:15:42.295006+0000","last_fullsized":"2024-04-24T23:15:42.295006+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:16:15.203151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.193480+0000","last_change":"2024-04-24T23:15:36.984336+0000","last_active":"2024-04-24T23:15:43.193480+0000","last_peered":"2024-04-24T23:15:43.193480+0000","last_clean":"2024-04-24T23:15:43.193480+0000","last_became_active":"2024-04-24T23:15:36.983966+0000","last_became_peered":"2024-04-24T23:15:36.983966+0000","last_unstale":"2024-04-24T23:15:43.193480+0000","last_undegraded":"2024-04-24T23:15:43.193480+0000","last_fullsized":"2024-04-24T23:15:43.193480+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:54:16.319608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.106762+0000","last_change":"2024-04-24T23:15:36.979079+0000","last_active":"2024-04-24T23:15:42.106762+0000","last_peered":"2024-04-24T23:15:42.106762+0000","last_clean":"2024-04-24T23:15:42.106762+0000","last_became_active":"2024-04-24T23:15:36.978706+0000","last_became_peered":"2024-04-24T23:15:36.978706+0000","last_unstale":"2024-04-24T23:15:42.106762+0000","last_undegraded":"2024-04-24T23:15:42.106762+0000","last_fullsized":"2024-04-24T23:15:42.106762+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:53:39.888150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.193929+0000","last_change":"2024-04-24T23:15:36.986008+0000","last_active":"2024-04-24T23:15:43.193929+0000","last_peered":"2024-04-24T23:15:43.193929+0000","last_clean":"2024-04-24T23:15:43.193929+0000","last_became_active":"2024-04-24T23:15:36.985552+0000","last_became_peered":"2024-04-24T23:15:36.985552+0000","last_unstale":"2024-04-24T23:15:43.193929+0000","last_undegraded":"2024-04-24T23:15:43.193929+0000","last_fullsized":"2024-04-24T23:15:43.193929+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:36:50.827503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.106908+0000","last_change":"2024-04-24T23:15:36.978673+0000","last_active":"2024-04-24T23:15:42.106908+0000","last_peered":"2024-04-24T23:15:42.106908+0000","last_clean":"2024-04-24T23:15:42.106908+0000","last_became_active":"2024-04-24T23:15:36.972744+0000","last_became_peered":"2024-04-24T23:15:36.972744+0000","last_unstale":"2024-04-24T23:15:42.106908+0000","last_undegraded":"2024-04-24T23:15:42.106908+0000","last_fullsized":"2024-04-24T23:15:42.106908+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:28:57.028825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.295395+0000","last_change":"2024-04-24T23:15:36.970238+0000","last_active":"2024-04-24T23:15:42.295395+0000","last_peered":"2024-04-24T23:15:42.295395+0000","last_clean":"2024-04-24T23:15:42.295395+0000","last_became_active":"2024-04-24T23:15:36.969751+0000","last_became_peered":"2024-04-24T23:15:36.969751+0000","last_unstale":"2024-04-24T23:15:42.295395+0000","last_undegraded":"2024-04-24T23:15:42.295395+0000","last_fullsized":"2024-04-24T23:15:42.295395+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:19:56.456801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.295592+0000","last_change":"2024-04-24T23:15:36.968825+0000","last_active":"2024-04-24T23:15:42.295592+0000","last_peered":"2024-04-24T23:15:42.295592+0000","last_clean":"2024-04-24T23:15:42.295592+0000","last_became_active":"2024-04-24T23:15:36.968543+0000","last_became_peered":"2024-04-24T23:15:36.968543+0000","last_unstale":"2024-04-24T23:15:42.295592+0000","last_undegraded":"2024-04-24T23:15:42.295592+0000","last_fullsized":"2024-04-24T23:15:42.295592+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:36:48.795062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.193226+0000","last_change":"2024-04-24T23:15:36.992800+0000","last_active":"2024-04-24T23:15:43.193226+0000","last_peered":"2024-04-24T23:15:43.193226+0000","last_clean":"2024-04-24T23:15:43.193226+0000","last_became_active":"2024-04-24T23:15:36.992476+0000","last_became_peered":"2024-04-24T23:15:36.992476+0000","last_unstale":"2024-04-24T23:15:43.193226+0000","last_undegraded":"2024-04-24T23:15:43.193226+0000","last_fullsized":"2024-04-24T23:15:43.193226+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:01:21.103559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4640,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":49,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574860,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":262144,"data_stored":100388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574860,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27272,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":253952,"data_stored":100264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574860,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27856,"kb_used_data":836,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343984,"statfs":{"total":96636764160,"available":96608239616,"internally_reserved":0,"allocated":856064,"data_stored":690724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574861,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":274432,"data_stored":104778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574861,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27272,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":258048,"data_stored":100846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574860,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":266240,"data_stored":104106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574859,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27852,"kb_used_data":832,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343988,"statfs":{"total":96636764160,"available":96608243712,"internally_reserved":0,"allocated":851968,"data_stored":690702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574860,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":262144,"data_stored":100320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T23:16:11.591 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T23:16:11.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:11.837+0000 7f7fa6200640 1 Processor -- start 2024-04-24T23:16:11.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:11.837+0000 7f7fa6200640 1 -- start start 2024-04-24T23:16:11.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:11.839+0000 7f7fa6200640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f7fa0154b70 0x7f7fa006f020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:11.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:11.839+0000 7f7fa6200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7fa0072530 con 0x7f7fa0071000 2024-04-24T23:16:11.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:11.839+0000 7f7fa6200640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7fa00727d0 con 0x7f7fa00c7080 2024-04-24T23:16:11.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:11.839+0000 7f7fa6200640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f7fa0072a70 con 0x7f7fa0154b70 2024-04-24T23:16:11.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:11.839+0000 7f7f9fe00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f7fa0071000 0x7f7fa015b0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:43414/0 (socket says 172.21.15.6:43414) 2024-04-24T23:16:11.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:11.839+0000 7f7f9fe00640 1 -- 172.21.15.6:0/899454784 learned_addr learned my addr 172.21.15.6:0/899454784 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:16:11.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:11.839+0000 7f7f9f400640 1 --2- 172.21.15.6:0/899454784 >> v2:172.21.15.190:3300/0 conn(0x7f7fa0154b70 0x7f7fa006f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:11.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:11.840+0000 7f7f9e000640 1 -- 172.21.15.6:0/899454784 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3922442459 0 0) 0x7f7fa00727d0 con 0x7f7fa00c7080 2024-04-24T23:16:11.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:11.840+0000 7f7f9e000640 1 -- 172.21.15.6:0/899454784 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7f84003520 con 0x7f7fa00c7080 2024-04-24T23:16:11.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:11.840+0000 7f7f9e000640 1 -- 172.21.15.6:0/899454784 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 959385084 0 0) 0x7f7fa0072530 con 0x7f7fa0071000 2024-04-24T23:16:11.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:11.840+0000 7f7f9f400640 1 -- 172.21.15.6:0/899454784 >> v1:172.21.15.190:6789/0 conn(0x7f7fa00c7080 legacy=0x7f7fa015a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:11.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:11.840+0000 7f7f9f400640 1 -- 172.21.15.6:0/899454784 >> v1:172.21.15.6:6789/0 conn(0x7f7fa0071000 legacy=0x7f7fa015b0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:11.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:11.840+0000 7f7f9f400640 1 -- 172.21.15.6:0/899454784 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fa006e540 con 0x7f7fa0154b70 2024-04-24T23:16:11.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:11.841+0000 7f7f9f400640 1 --2- 172.21.15.6:0/899454784 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7fa0154b70 0x7f7fa006f020 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f7f900097d0 tx=0x7f7f9002a270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9271e6c79e072ded server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:11.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:11.841+0000 7f7f9e000640 1 -- 172.21.15.6:0/899454784 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f90037070 con 0x7f7fa0154b70 2024-04-24T23:16:11.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:11.841+0000 7f7f9e000640 1 -- 172.21.15.6:0/899454784 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f90032440 con 0x7f7fa0154b70 2024-04-24T23:16:11.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:11.841+0000 7f7f9e000640 1 -- 172.21.15.6:0/899454784 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f900327f0 con 0x7f7fa0154b70 2024-04-24T23:16:11.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:11.842+0000 7f7fa6200640 1 -- 172.21.15.6:0/899454784 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7fa0154b70 msgr2=0x7f7fa006f020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:11.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:11.842+0000 7f7fa6200640 1 --2- 172.21.15.6:0/899454784 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7fa0154b70 0x7f7fa006f020 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f7f900097d0 tx=0x7f7f9002a270 comp rx=0 tx=0).stop 2024-04-24T23:16:11.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:11.842+0000 7f7fa6200640 1 -- 172.21.15.6:0/899454784 shutdown_connections 2024-04-24T23:16:11.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:11.842+0000 7f7fa6200640 1 --2- 172.21.15.6:0/899454784 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7fa0154b70 0x7f7fa006f020 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:11.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:11.842+0000 7f7fa6200640 1 -- 172.21.15.6:0/899454784 >> 172.21.15.6:0/899454784 conn(0x7f7fa006b0e0 msgr2=0x7f7fa006b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:11.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:11.842+0000 7f7fa6200640 1 -- 172.21.15.6:0/899454784 shutdown_connections 2024-04-24T23:16:11.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:11.842+0000 7f7fa6200640 1 -- 172.21.15.6:0/899454784 wait complete. 2024-04-24T23:16:11.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:11.844+0000 7f7fa6200640 1 Processor -- start 2024-04-24T23:16:11.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:11.844+0000 7f7fa6200640 1 -- start start 2024-04-24T23:16:11.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:11.845+0000 7f7fa6200640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7fa0071000 0x7f7fa01e4480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:11.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:11.845+0000 7f7fa6200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7fa00c7080 0x7f7fa01f3890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:11.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:11.845+0000 7f7fa6200640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7fa0154b70 0x7f7fa015dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:11.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:11.845+0000 7f7fa6200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f7fa015b7f0 con 0x7f7fa00c7080 2024-04-24T23:16:11.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:11.845+0000 7f7fa6200640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f7fa006eb00 con 0x7f7fa0071000 2024-04-24T23:16:11.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:11.845+0000 7f7fa6200640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f7fa0154f50 con 0x7f7fa0154b70 2024-04-24T23:16:11.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:11.845+0000 7f7f9ea00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7fa00c7080 0x7f7fa01f3890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:11.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:11.845+0000 7f7f9f400640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7fa0071000 0x7f7fa01e4480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:11.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:11.845+0000 7f7f9ea00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7fa00c7080 0x7f7fa01f3890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:59920/0 (socket says 172.21.15.6:59920) 2024-04-24T23:16:11.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:11.845+0000 7f7f9ea00640 1 -- 172.21.15.6:0/2809189497 learned_addr learned my addr 172.21.15.6:0/2809189497 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:16:11.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:11.845+0000 7f7f9fe00640 1 --2- 172.21.15.6:0/2809189497 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7fa0154b70 0x7f7fa015dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:11.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:11.846+0000 7f7f9ea00640 1 -- 172.21.15.6:0/2809189497 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7fa0154b70 msgr2=0x7f7fa015dd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:11.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:11.846+0000 7f7f9ea00640 1 --2- 172.21.15.6:0/2809189497 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7fa0154b70 0x7f7fa015dd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:11.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:11.846+0000 7f7f9ea00640 1 -- 172.21.15.6:0/2809189497 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7fa0071000 msgr2=0x7f7fa01e4480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:11.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:11.846+0000 7f7f9ea00640 1 --2- 172.21.15.6:0/2809189497 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7fa0071000 0x7f7fa01e4480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:11.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:11.846+0000 7f7f9ea00640 1 -- 172.21.15.6:0/2809189497 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fa01f41c0 con 0x7f7fa00c7080 2024-04-24T23:16:11.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:11.846+0000 7f7f9f400640 1 --2- 172.21.15.6:0/2809189497 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7fa0071000 0x7f7fa01e4480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:16:11.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:11.846+0000 7f7f9fe00640 1 --2- 172.21.15.6:0/2809189497 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7fa0154b70 0x7f7fa015dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:16:11.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:11.846+0000 7f7f9ea00640 1 --2- 172.21.15.6:0/2809189497 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7fa00c7080 0x7f7fa01f3890 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f7f8c00c4c0 tx=0x7f7f8c00c980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:11.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:11.846+0000 7f7f97400640 1 -- 172.21.15.6:0/2809189497 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f8c0052d0 con 0x7f7fa00c7080 2024-04-24T23:16:11.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:11.846+0000 7f7f97400640 1 -- 172.21.15.6:0/2809189497 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f8c005470 con 0x7f7fa00c7080 2024-04-24T23:16:11.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:11.846+0000 7f7f97400640 1 -- 172.21.15.6:0/2809189497 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f8c012870 con 0x7f7fa00c7080 2024-04-24T23:16:11.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:11.846+0000 7f7fa6200640 1 -- 172.21.15.6:0/2809189497 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7fa01f44a0 con 0x7f7fa00c7080 2024-04-24T23:16:11.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:11.846+0000 7f7fa6200640 1 -- 172.21.15.6:0/2809189497 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7fa0153540 con 0x7f7fa00c7080 2024-04-24T23:16:11.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:11.848+0000 7f7fa6200640 1 -- 172.21.15.6:0/2809189497 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7fa00767b0 con 0x7f7fa00c7080 2024-04-24T23:16:11.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:11.849+0000 7f7f97400640 1 -- 172.21.15.6:0/2809189497 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f7f8c012a10 con 0x7f7fa00c7080 2024-04-24T23:16:11.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:11.849+0000 7f7f97400640 1 --2- 172.21.15.6:0/2809189497 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f7f8006be30 0x7f7f8006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:11.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:11.849+0000 7f7f97400640 1 -- 172.21.15.6:0/2809189497 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f7f8c08ae20 con 0x7f7fa00c7080 2024-04-24T23:16:11.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:11.850+0000 7f7f9f400640 1 --2- 172.21.15.6:0/2809189497 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f7f8006be30 0x7f7f8006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:11.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:11.851+0000 7f7f9f400640 1 --2- 172.21.15.6:0/2809189497 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f7f8006be30 0x7f7f8006e2f0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f7f90034000 tx=0x7f7f9002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:11.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:11.854+0000 7f7f97400640 1 -- 172.21.15.6:0/2809189497 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f7f8c058d10 con 0x7f7fa00c7080 2024-04-24T23:16:12.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:12.056+0000 7f7fa6200640 1 -- 172.21.15.6:0/2809189497 --> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f7fa00c82e0 con 0x7f7f8006be30 2024-04-24T23:16:12.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:12.066+0000 7f7f97400640 1 -- 172.21.15.6:0/2809189497 <== mgr.4105 v2:172.21.15.190:6832/806769663 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+332533 (secure 0 0 0) 0x7f7fa00c82e0 con 0x7f7f8006be30 2024-04-24T23:16:12.070 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:16:12.074 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-04-24T23:16:12.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:12.073+0000 7f7fa6200640 1 -- 172.21.15.6:0/2809189497 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f7f8006be30 msgr2=0x7f7f8006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:12.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:12.073+0000 7f7fa6200640 1 --2- 172.21.15.6:0/2809189497 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f7f8006be30 0x7f7f8006e2f0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f7f90034000 tx=0x7f7f9002e000 comp rx=0 tx=0).stop 2024-04-24T23:16:12.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:12.073+0000 7f7fa6200640 1 -- 172.21.15.6:0/2809189497 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7fa00c7080 msgr2=0x7f7fa01f3890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:12.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:12.073+0000 7f7fa6200640 1 --2- 172.21.15.6:0/2809189497 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7fa00c7080 0x7f7fa01f3890 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f7f8c00c4c0 tx=0x7f7f8c00c980 comp rx=0 tx=0).stop 2024-04-24T23:16:12.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:12.073+0000 7f7fa6200640 1 -- 172.21.15.6:0/2809189497 shutdown_connections 2024-04-24T23:16:12.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:12.073+0000 7f7fa6200640 1 --2- 172.21.15.6:0/2809189497 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f7f8006be30 0x7f7f8006e2f0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:12.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:12.073+0000 7f7fa6200640 1 --2- 172.21.15.6:0/2809189497 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7fa0154b70 0x7f7fa015dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:12.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:12.073+0000 7f7fa6200640 1 --2- 172.21.15.6:0/2809189497 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7fa00c7080 0x7f7fa01f3890 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:12.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:12.073+0000 7f7fa6200640 1 --2- 172.21.15.6:0/2809189497 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7fa0071000 0x7f7fa01e4480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:12.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:12.073+0000 7f7fa6200640 1 -- 172.21.15.6:0/2809189497 >> 172.21.15.6:0/2809189497 conn(0x7f7fa006b0e0 msgr2=0x7f7fa014ecc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:12.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:12.074+0000 7f7fa6200640 1 -- 172.21.15.6:0/2809189497 shutdown_connections 2024-04-24T23:16:12.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:12.074+0000 7f7fa6200640 1 -- 172.21.15.6:0/2809189497 wait complete. 2024-04-24T23:16:12.096 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":40,"stamp":"2024-04-24T23:16:10.688215+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595008,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":208,"num_write_kb":2435,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160,"ondisk_log_size":160,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219368,"kb_used_data":3208,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755352,"statfs":{"total":773094113280,"available":772869480448,"internally_reserved":0,"allocated":3284992,"data_stored":1992128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":141,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003445"},"pg_stats":[{"pgid":"2.1f","version":"23'4","reported_seq":26,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.051292+0000","last_change":"2024-04-24T23:15:35.964646+0000","last_active":"2024-04-24T23:15:42.051292+0000","last_peered":"2024-04-24T23:15:42.051292+0000","last_clean":"2024-04-24T23:15:42.051292+0000","last_became_active":"2024-04-24T23:15:35.964357+0000","last_became_peered":"2024-04-24T23:15:35.964357+0000","last_unstale":"2024-04-24T23:15:42.051292+0000","last_undegraded":"2024-04-24T23:15:42.051292+0000","last_fullsized":"2024-04-24T23:15:42.051292+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:01:17.912993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.295716+0000","last_change":"2024-04-24T23:15:36.989772+0000","last_active":"2024-04-24T23:15:42.295716+0000","last_peered":"2024-04-24T23:15:42.295716+0000","last_clean":"2024-04-24T23:15:42.295716+0000","last_became_active":"2024-04-24T23:15:36.989543+0000","last_became_peered":"2024-04-24T23:15:36.989543+0000","last_unstale":"2024-04-24T23:15:42.295716+0000","last_undegraded":"2024-04-24T23:15:42.295716+0000","last_fullsized":"2024-04-24T23:15:42.295716+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:37:16.737354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.187704+0000","last_change":"2024-04-24T23:15:41.017182+0000","last_active":"2024-04-24T23:15:43.187704+0000","last_peered":"2024-04-24T23:15:43.187704+0000","last_clean":"2024-04-24T23:15:43.187704+0000","last_became_active":"2024-04-24T23:15:41.016719+0000","last_became_peered":"2024-04-24T23:15:41.016719+0000","last_unstale":"2024-04-24T23:15:43.187704+0000","last_undegraded":"2024-04-24T23:15:43.187704+0000","last_fullsized":"2024-04-24T23:15:43.187704+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:42:11.186486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4,1],"acting":[7,6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1e","version":"23'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.295817+0000","last_change":"2024-04-24T23:15:35.946083+0000","last_active":"2024-04-24T23:15:42.295817+0000","last_peered":"2024-04-24T23:15:42.295817+0000","last_clean":"2024-04-24T23:15:42.295817+0000","last_became_active":"2024-04-24T23:15:35.945227+0000","last_became_peered":"2024-04-24T23:15:35.945227+0000","last_unstale":"2024-04-24T23:15:42.295817+0000","last_undegraded":"2024-04-24T23:15:42.295817+0000","last_fullsized":"2024-04-24T23:15:42.295817+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:05:17.727725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.295775+0000","last_change":"2024-04-24T23:15:36.998245+0000","last_active":"2024-04-24T23:15:42.295775+0000","last_peered":"2024-04-24T23:15:42.295775+0000","last_clean":"2024-04-24T23:15:42.295775+0000","last_became_active":"2024-04-24T23:15:36.998021+0000","last_became_peered":"2024-04-24T23:15:36.998021+0000","last_unstale":"2024-04-24T23:15:42.295775+0000","last_undegraded":"2024-04-24T23:15:42.295775+0000","last_fullsized":"2024-04-24T23:15:42.295775+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:03:42.359117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.106527+0000","last_change":"2024-04-24T23:15:41.008313+0000","last_active":"2024-04-24T23:15:42.106527+0000","last_peered":"2024-04-24T23:15:42.106527+0000","last_clean":"2024-04-24T23:15:42.106527+0000","last_became_active":"2024-04-24T23:15:41.007908+0000","last_became_peered":"2024-04-24T23:15:41.007908+0000","last_unstale":"2024-04-24T23:15:42.106527+0000","last_undegraded":"2024-04-24T23:15:42.106527+0000","last_fullsized":"2024-04-24T23:15:42.106527+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:02:11.902575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4,6],"acting":[2,7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1d","version":"23'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.193739+0000","last_change":"2024-04-24T23:15:35.968312+0000","last_active":"2024-04-24T23:15:43.193739+0000","last_peered":"2024-04-24T23:15:43.193739+0000","last_clean":"2024-04-24T23:15:43.193739+0000","last_became_active":"2024-04-24T23:15:35.967873+0000","last_became_peered":"2024-04-24T23:15:35.967873+0000","last_unstale":"2024-04-24T23:15:43.193739+0000","last_undegraded":"2024-04-24T23:15:43.193739+0000","last_fullsized":"2024-04-24T23:15:43.193739+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:21:40.116977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.733492+0000","last_change":"2024-04-24T23:15:36.997361+0000","last_active":"2024-04-24T23:15:42.733492+0000","last_peered":"2024-04-24T23:15:42.733492+0000","last_clean":"2024-04-24T23:15:42.733492+0000","last_became_active":"2024-04-24T23:15:36.997083+0000","last_became_peered":"2024-04-24T23:15:36.997083+0000","last_unstale":"2024-04-24T23:15:42.733492+0000","last_undegraded":"2024-04-24T23:15:42.733492+0000","last_fullsized":"2024-04-24T23:15:42.733492+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:47:59.334093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.107355+0000","last_change":"2024-04-24T23:15:41.025153+0000","last_active":"2024-04-24T23:15:42.107355+0000","last_peered":"2024-04-24T23:15:42.107355+0000","last_clean":"2024-04-24T23:15:42.107355+0000","last_became_active":"2024-04-24T23:15:41.024739+0000","last_became_peered":"2024-04-24T23:15:41.024739+0000","last_unstale":"2024-04-24T23:15:42.107355+0000","last_undegraded":"2024-04-24T23:15:42.107355+0000","last_fullsized":"2024-04-24T23:15:42.107355+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:31:00.160213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3,6],"acting":[2,5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1c","version":"23'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.193705+0000","last_change":"2024-04-24T23:15:35.967338+0000","last_active":"2024-04-24T23:15:43.193705+0000","last_peered":"2024-04-24T23:15:43.193705+0000","last_clean":"2024-04-24T23:15:43.193705+0000","last_became_active":"2024-04-24T23:15:35.966951+0000","last_became_peered":"2024-04-24T23:15:35.966951+0000","last_unstale":"2024-04-24T23:15:43.193705+0000","last_undegraded":"2024-04-24T23:15:43.193705+0000","last_fullsized":"2024-04-24T23:15:43.193705+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:08:05.248330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.191745+0000","last_change":"2024-04-24T23:15:36.998072+0000","last_active":"2024-04-24T23:15:43.191745+0000","last_peered":"2024-04-24T23:15:43.191745+0000","last_clean":"2024-04-24T23:15:43.191745+0000","last_became_active":"2024-04-24T23:15:36.997735+0000","last_became_peered":"2024-04-24T23:15:36.997735+0000","last_unstale":"2024-04-24T23:15:43.191745+0000","last_undegraded":"2024-04-24T23:15:43.191745+0000","last_fullsized":"2024-04-24T23:15:43.191745+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:22:06.869644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.106640+0000","last_change":"2024-04-24T23:15:41.008895+0000","last_active":"2024-04-24T23:15:42.106640+0000","last_peered":"2024-04-24T23:15:42.106640+0000","last_clean":"2024-04-24T23:15:42.106640+0000","last_became_active":"2024-04-24T23:15:41.006199+0000","last_became_peered":"2024-04-24T23:15:41.006199+0000","last_unstale":"2024-04-24T23:15:42.106640+0000","last_undegraded":"2024-04-24T23:15:42.106640+0000","last_fullsized":"2024-04-24T23:15:42.106640+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:13:34.583602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5,7],"acting":[2,6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.191106+0000","last_change":"2024-04-24T23:15:37.004074+0000","last_active":"2024-04-24T23:15:43.191106+0000","last_peered":"2024-04-24T23:15:43.191106+0000","last_clean":"2024-04-24T23:15:43.191106+0000","last_became_active":"2024-04-24T23:15:37.003544+0000","last_became_peered":"2024-04-24T23:15:37.003544+0000","last_unstale":"2024-04-24T23:15:43.191106+0000","last_undegraded":"2024-04-24T23:15:43.191106+0000","last_fullsized":"2024-04-24T23:15:43.191106+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:38:28.024702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"23'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.191747+0000","last_change":"2024-04-24T23:15:35.963000+0000","last_active":"2024-04-24T23:15:43.191747+0000","last_peered":"2024-04-24T23:15:43.191747+0000","last_clean":"2024-04-24T23:15:43.191747+0000","last_became_active":"2024-04-24T23:15:35.961960+0000","last_became_peered":"2024-04-24T23:15:35.961960+0000","last_unstale":"2024-04-24T23:15:43.191747+0000","last_undegraded":"2024-04-24T23:15:43.191747+0000","last_fullsized":"2024-04-24T23:15:43.191747+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:56:26.651193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.191430+0000","last_change":"2024-04-24T23:15:41.022481+0000","last_active":"2024-04-24T23:15:43.191430+0000","last_peered":"2024-04-24T23:15:43.191430+0000","last_clean":"2024-04-24T23:15:43.191430+0000","last_became_active":"2024-04-24T23:15:41.020034+0000","last_became_peered":"2024-04-24T23:15:41.020034+0000","last_unstale":"2024-04-24T23:15:43.191430+0000","last_undegraded":"2024-04-24T23:15:43.191430+0000","last_fullsized":"2024-04-24T23:15:43.191430+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:27:25.337503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,5],"acting":[3,1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.187801+0000","last_change":"2024-04-24T23:15:36.972020+0000","last_active":"2024-04-24T23:15:43.187801+0000","last_peered":"2024-04-24T23:15:43.187801+0000","last_clean":"2024-04-24T23:15:43.187801+0000","last_became_active":"2024-04-24T23:15:36.971583+0000","last_became_peered":"2024-04-24T23:15:36.971583+0000","last_unstale":"2024-04-24T23:15:43.187801+0000","last_undegraded":"2024-04-24T23:15:43.187801+0000","last_fullsized":"2024-04-24T23:15:43.187801+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:29:05.568608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.050091+0000","last_change":"2024-04-24T23:15:35.945916+0000","last_active":"2024-04-24T23:15:42.050091+0000","last_peered":"2024-04-24T23:15:42.050091+0000","last_clean":"2024-04-24T23:15:42.050091+0000","last_became_active":"2024-04-24T23:15:35.945610+0000","last_became_peered":"2024-04-24T23:15:35.945610+0000","last_unstale":"2024-04-24T23:15:42.050091+0000","last_undegraded":"2024-04-24T23:15:42.050091+0000","last_fullsized":"2024-04-24T23:15:42.050091+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:57:05.976074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.192627+0000","last_change":"2024-04-24T23:15:41.030165+0000","last_active":"2024-04-24T23:15:43.192627+0000","last_peered":"2024-04-24T23:15:43.192627+0000","last_clean":"2024-04-24T23:15:43.192627+0000","last_became_active":"2024-04-24T23:15:41.029685+0000","last_became_peered":"2024-04-24T23:15:41.029685+0000","last_unstale":"2024-04-24T23:15:43.192627+0000","last_undegraded":"2024-04-24T23:15:43.192627+0000","last_fullsized":"2024-04-24T23:15:43.192627+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:20:43.498190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6,1],"acting":[0,4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.049168+0000","last_change":"2024-04-24T23:15:36.965949+0000","last_active":"2024-04-24T23:15:42.049168+0000","last_peered":"2024-04-24T23:15:42.049168+0000","last_clean":"2024-04-24T23:15:42.049168+0000","last_became_active":"2024-04-24T23:15:36.965658+0000","last_became_peered":"2024-04-24T23:15:36.965658+0000","last_unstale":"2024-04-24T23:15:42.049168+0000","last_undegraded":"2024-04-24T23:15:42.049168+0000","last_fullsized":"2024-04-24T23:15:42.049168+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:38:28.024702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.049695+0000","last_change":"2024-04-24T23:15:35.972165+0000","last_active":"2024-04-24T23:15:42.049695+0000","last_peered":"2024-04-24T23:15:42.049695+0000","last_clean":"2024-04-24T23:15:42.049695+0000","last_became_active":"2024-04-24T23:15:35.971972+0000","last_became_peered":"2024-04-24T23:15:35.971972+0000","last_unstale":"2024-04-24T23:15:42.049695+0000","last_undegraded":"2024-04-24T23:15:42.049695+0000","last_fullsized":"2024-04-24T23:15:42.049695+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:11:12.088698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.188224+0000","last_change":"2024-04-24T23:15:41.019799+0000","last_active":"2024-04-24T23:15:43.188224+0000","last_peered":"2024-04-24T23:15:43.188224+0000","last_clean":"2024-04-24T23:15:43.188224+0000","last_became_active":"2024-04-24T23:15:41.019339+0000","last_became_peered":"2024-04-24T23:15:41.019339+0000","last_unstale":"2024-04-24T23:15:43.188224+0000","last_undegraded":"2024-04-24T23:15:43.188224+0000","last_fullsized":"2024-04-24T23:15:43.188224+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:59:34.601004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2,3],"acting":[7,4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.190629+0000","last_change":"2024-04-24T23:15:36.991991+0000","last_active":"2024-04-24T23:15:43.190629+0000","last_peered":"2024-04-24T23:15:43.190629+0000","last_clean":"2024-04-24T23:15:43.190629+0000","last_became_active":"2024-04-24T23:15:36.989336+0000","last_became_peered":"2024-04-24T23:15:36.989336+0000","last_unstale":"2024-04-24T23:15:43.190629+0000","last_undegraded":"2024-04-24T23:15:43.190629+0000","last_fullsized":"2024-04-24T23:15:43.190629+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:25:56.038903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.190149+0000","last_change":"2024-04-24T23:15:35.965443+0000","last_active":"2024-04-24T23:15:43.190149+0000","last_peered":"2024-04-24T23:15:43.190149+0000","last_clean":"2024-04-24T23:15:43.190149+0000","last_became_active":"2024-04-24T23:15:35.965043+0000","last_became_peered":"2024-04-24T23:15:35.965043+0000","last_unstale":"2024-04-24T23:15:43.190149+0000","last_undegraded":"2024-04-24T23:15:43.190149+0000","last_fullsized":"2024-04-24T23:15:43.190149+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:07:23.320042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.107146+0000","last_change":"2024-04-24T23:15:41.025131+0000","last_active":"2024-04-24T23:15:42.107146+0000","last_peered":"2024-04-24T23:15:42.107146+0000","last_clean":"2024-04-24T23:15:42.107146+0000","last_became_active":"2024-04-24T23:15:41.024731+0000","last_became_peered":"2024-04-24T23:15:41.024731+0000","last_unstale":"2024-04-24T23:15:42.107146+0000","last_undegraded":"2024-04-24T23:15:42.107146+0000","last_fullsized":"2024-04-24T23:15:42.107146+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:28:07.952854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,7],"acting":[2,1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.049954+0000","last_change":"2024-04-24T23:15:36.965850+0000","last_active":"2024-04-24T23:15:42.049954+0000","last_peered":"2024-04-24T23:15:42.049954+0000","last_clean":"2024-04-24T23:15:42.049954+0000","last_became_active":"2024-04-24T23:15:36.965547+0000","last_became_peered":"2024-04-24T23:15:36.965547+0000","last_unstale":"2024-04-24T23:15:42.049954+0000","last_undegraded":"2024-04-24T23:15:42.049954+0000","last_fullsized":"2024-04-24T23:15:42.049954+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:06:37.125573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"23'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.192302+0000","last_change":"2024-04-24T23:15:35.971040+0000","last_active":"2024-04-24T23:15:43.192302+0000","last_peered":"2024-04-24T23:15:43.192302+0000","last_clean":"2024-04-24T23:15:43.192302+0000","last_became_active":"2024-04-24T23:15:35.970184+0000","last_became_peered":"2024-04-24T23:15:35.970184+0000","last_unstale":"2024-04-24T23:15:43.192302+0000","last_undegraded":"2024-04-24T23:15:43.192302+0000","last_fullsized":"2024-04-24T23:15:43.192302+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:45:20.913759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.106686+0000","last_change":"2024-04-24T23:15:41.025813+0000","last_active":"2024-04-24T23:15:42.106686+0000","last_peered":"2024-04-24T23:15:42.106686+0000","last_clean":"2024-04-24T23:15:42.106686+0000","last_became_active":"2024-04-24T23:15:41.025137+0000","last_became_peered":"2024-04-24T23:15:41.025137+0000","last_unstale":"2024-04-24T23:15:42.106686+0000","last_undegraded":"2024-04-24T23:15:42.106686+0000","last_fullsized":"2024-04-24T23:15:42.106686+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:35:17.784854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,3,1],"acting":[2,6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.051163+0000","last_change":"2024-04-24T23:15:36.967476+0000","last_active":"2024-04-24T23:15:42.051163+0000","last_peered":"2024-04-24T23:15:42.051163+0000","last_clean":"2024-04-24T23:15:42.051163+0000","last_became_active":"2024-04-24T23:15:36.967098+0000","last_became_peered":"2024-04-24T23:15:36.967098+0000","last_unstale":"2024-04-24T23:15:42.051163+0000","last_undegraded":"2024-04-24T23:15:42.051163+0000","last_fullsized":"2024-04-24T23:15:42.051163+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:25:56.038903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"24'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.050091+0000","last_change":"2024-04-24T23:15:35.970630+0000","last_active":"2024-04-24T23:15:42.050091+0000","last_peered":"2024-04-24T23:15:42.050091+0000","last_clean":"2024-04-24T23:15:42.050091+0000","last_became_active":"2024-04-24T23:15:35.970430+0000","last_became_peered":"2024-04-24T23:15:35.970430+0000","last_unstale":"2024-04-24T23:15:42.050091+0000","last_undegraded":"2024-04-24T23:15:42.050091+0000","last_fullsized":"2024-04-24T23:15:42.050091+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:15:59.219084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.049399+0000","last_change":"2024-04-24T23:15:41.005022+0000","last_active":"2024-04-24T23:15:42.049399+0000","last_peered":"2024-04-24T23:15:42.049399+0000","last_clean":"2024-04-24T23:15:42.049399+0000","last_became_active":"2024-04-24T23:15:41.004648+0000","last_became_peered":"2024-04-24T23:15:41.004648+0000","last_unstale":"2024-04-24T23:15:42.049399+0000","last_undegraded":"2024-04-24T23:15:42.049399+0000","last_fullsized":"2024-04-24T23:15:42.049399+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:06:01.559760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6,3],"acting":[4,1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.051008+0000","last_change":"2024-04-24T23:15:36.967053+0000","last_active":"2024-04-24T23:15:42.051008+0000","last_peered":"2024-04-24T23:15:42.051008+0000","last_clean":"2024-04-24T23:15:42.051008+0000","last_became_active":"2024-04-24T23:15:36.966746+0000","last_became_peered":"2024-04-24T23:15:36.966746+0000","last_unstale":"2024-04-24T23:15:42.051008+0000","last_undegraded":"2024-04-24T23:15:42.051008+0000","last_fullsized":"2024-04-24T23:15:42.051008+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:16:08.779860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.192257+0000","last_change":"2024-04-24T23:15:35.965890+0000","last_active":"2024-04-24T23:15:43.192257+0000","last_peered":"2024-04-24T23:15:43.192257+0000","last_clean":"2024-04-24T23:15:43.192257+0000","last_became_active":"2024-04-24T23:15:35.965631+0000","last_became_peered":"2024-04-24T23:15:35.965631+0000","last_unstale":"2024-04-24T23:15:43.192257+0000","last_undegraded":"2024-04-24T23:15:43.192257+0000","last_fullsized":"2024-04-24T23:15:43.192257+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:17:35.650731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.050382+0000","last_change":"2024-04-24T23:15:41.011405+0000","last_active":"2024-04-24T23:15:42.050382+0000","last_peered":"2024-04-24T23:15:42.050382+0000","last_clean":"2024-04-24T23:15:42.050382+0000","last_became_active":"2024-04-24T23:15:41.010982+0000","last_became_peered":"2024-04-24T23:15:41.010982+0000","last_unstale":"2024-04-24T23:15:42.050382+0000","last_undegraded":"2024-04-24T23:15:42.050382+0000","last_fullsized":"2024-04-24T23:15:42.050382+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:08:00.914296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2,7],"acting":[6,0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.106945+0000","last_change":"2024-04-24T23:15:36.975244+0000","last_active":"2024-04-24T23:15:42.106945+0000","last_peered":"2024-04-24T23:15:42.106945+0000","last_clean":"2024-04-24T23:15:42.106945+0000","last_became_active":"2024-04-24T23:15:36.974872+0000","last_became_peered":"2024-04-24T23:15:36.974872+0000","last_unstale":"2024-04-24T23:15:42.106945+0000","last_undegraded":"2024-04-24T23:15:42.106945+0000","last_fullsized":"2024-04-24T23:15:42.106945+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:45:57.018205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.295386+0000","last_change":"2024-04-24T23:15:35.945792+0000","last_active":"2024-04-24T23:15:42.295386+0000","last_peered":"2024-04-24T23:15:42.295386+0000","last_clean":"2024-04-24T23:15:42.295386+0000","last_became_active":"2024-04-24T23:15:35.945094+0000","last_became_peered":"2024-04-24T23:15:35.945094+0000","last_unstale":"2024-04-24T23:15:42.295386+0000","last_undegraded":"2024-04-24T23:15:42.295386+0000","last_fullsized":"2024-04-24T23:15:42.295386+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:49:12.719127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.049802+0000","last_change":"2024-04-24T23:15:41.022105+0000","last_active":"2024-04-24T23:15:42.049802+0000","last_peered":"2024-04-24T23:15:42.049802+0000","last_clean":"2024-04-24T23:15:42.049802+0000","last_became_active":"2024-04-24T23:15:41.021645+0000","last_became_peered":"2024-04-24T23:15:41.021645+0000","last_unstale":"2024-04-24T23:15:42.049802+0000","last_undegraded":"2024-04-24T23:15:42.049802+0000","last_fullsized":"2024-04-24T23:15:42.049802+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:03:52.974835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5,7],"acting":[6,3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.192545+0000","last_change":"2024-04-24T23:15:36.991239+0000","last_active":"2024-04-24T23:15:43.192545+0000","last_peered":"2024-04-24T23:15:43.192545+0000","last_clean":"2024-04-24T23:15:43.192545+0000","last_became_active":"2024-04-24T23:15:36.990963+0000","last_became_peered":"2024-04-24T23:15:36.990963+0000","last_unstale":"2024-04-24T23:15:43.192545+0000","last_undegraded":"2024-04-24T23:15:43.192545+0000","last_fullsized":"2024-04-24T23:15:43.192545+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:53:28.101334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.191521+0000","last_change":"2024-04-24T23:15:35.957931+0000","last_active":"2024-04-24T23:15:43.191521+0000","last_peered":"2024-04-24T23:15:43.191521+0000","last_clean":"2024-04-24T23:15:43.191521+0000","last_became_active":"2024-04-24T23:15:35.957432+0000","last_became_peered":"2024-04-24T23:15:35.957432+0000","last_unstale":"2024-04-24T23:15:43.191521+0000","last_undegraded":"2024-04-24T23:15:43.191521+0000","last_fullsized":"2024-04-24T23:15:43.191521+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:11:38.773425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.050839+0000","last_change":"2024-04-24T23:15:41.007857+0000","last_active":"2024-04-24T23:15:42.050839+0000","last_peered":"2024-04-24T23:15:42.050839+0000","last_clean":"2024-04-24T23:15:42.050839+0000","last_became_active":"2024-04-24T23:15:41.007449+0000","last_became_peered":"2024-04-24T23:15:41.007449+0000","last_unstale":"2024-04-24T23:15:42.050839+0000","last_undegraded":"2024-04-24T23:15:42.050839+0000","last_fullsized":"2024-04-24T23:15:42.050839+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:05:06.145440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0,3],"acting":[6,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.295077+0000","last_change":"2024-04-24T23:15:36.970444+0000","last_active":"2024-04-24T23:15:42.295077+0000","last_peered":"2024-04-24T23:15:42.295077+0000","last_clean":"2024-04-24T23:15:42.295077+0000","last_became_active":"2024-04-24T23:15:36.970159+0000","last_became_peered":"2024-04-24T23:15:36.970159+0000","last_unstale":"2024-04-24T23:15:42.295077+0000","last_undegraded":"2024-04-24T23:15:42.295077+0000","last_fullsized":"2024-04-24T23:15:42.295077+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:37:02.956224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.187767+0000","last_change":"2024-04-24T23:15:35.945461+0000","last_active":"2024-04-24T23:15:43.187767+0000","last_peered":"2024-04-24T23:15:43.187767+0000","last_clean":"2024-04-24T23:15:43.187767+0000","last_became_active":"2024-04-24T23:15:35.945185+0000","last_became_peered":"2024-04-24T23:15:35.945185+0000","last_unstale":"2024-04-24T23:15:43.187767+0000","last_undegraded":"2024-04-24T23:15:43.187767+0000","last_fullsized":"2024-04-24T23:15:43.187767+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:37:34.175539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.050260+0000","last_change":"2024-04-24T23:15:41.014584+0000","last_active":"2024-04-24T23:15:42.050260+0000","last_peered":"2024-04-24T23:15:42.050260+0000","last_clean":"2024-04-24T23:15:42.050260+0000","last_became_active":"2024-04-24T23:15:41.014137+0000","last_became_peered":"2024-04-24T23:15:41.014137+0000","last_unstale":"2024-04-24T23:15:42.050260+0000","last_undegraded":"2024-04-24T23:15:42.050260+0000","last_fullsized":"2024-04-24T23:15:42.050260+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:23:50.387635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,7],"acting":[6,0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.049485+0000","last_change":"2024-04-24T23:15:36.981783+0000","last_active":"2024-04-24T23:15:42.049485+0000","last_peered":"2024-04-24T23:15:42.049485+0000","last_clean":"2024-04-24T23:15:42.049485+0000","last_became_active":"2024-04-24T23:15:36.981502+0000","last_became_peered":"2024-04-24T23:15:36.981502+0000","last_unstale":"2024-04-24T23:15:42.049485+0000","last_undegraded":"2024-04-24T23:15:42.049485+0000","last_fullsized":"2024-04-24T23:15:42.049485+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:56:08.483410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.190765+0000","last_change":"2024-04-24T23:15:35.960519+0000","last_active":"2024-04-24T23:15:43.190765+0000","last_peered":"2024-04-24T23:15:43.190765+0000","last_clean":"2024-04-24T23:15:43.190765+0000","last_became_active":"2024-04-24T23:15:35.960180+0000","last_became_peered":"2024-04-24T23:15:35.960180+0000","last_unstale":"2024-04-24T23:15:43.190765+0000","last_undegraded":"2024-04-24T23:15:43.190765+0000","last_fullsized":"2024-04-24T23:15:43.190765+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:50:00.199120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.295661+0000","last_change":"2024-04-24T23:15:41.012876+0000","last_active":"2024-04-24T23:15:42.295661+0000","last_peered":"2024-04-24T23:15:42.295661+0000","last_clean":"2024-04-24T23:15:42.295661+0000","last_became_active":"2024-04-24T23:15:41.012493+0000","last_became_peered":"2024-04-24T23:15:41.012493+0000","last_unstale":"2024-04-24T23:15:42.295661+0000","last_undegraded":"2024-04-24T23:15:42.295661+0000","last_fullsized":"2024-04-24T23:15:42.295661+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:21:28.287059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6,1],"acting":[5,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.050057+0000","last_change":"2024-04-24T23:15:36.981449+0000","last_active":"2024-04-24T23:15:42.050057+0000","last_peered":"2024-04-24T23:15:42.050057+0000","last_clean":"2024-04-24T23:15:42.050057+0000","last_became_active":"2024-04-24T23:15:36.980837+0000","last_became_peered":"2024-04-24T23:15:36.980837+0000","last_unstale":"2024-04-24T23:15:42.050057+0000","last_undegraded":"2024-04-24T23:15:42.050057+0000","last_fullsized":"2024-04-24T23:15:42.050057+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:15:13.513341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.192038+0000","last_change":"2024-04-24T23:15:35.952947+0000","last_active":"2024-04-24T23:15:43.192038+0000","last_peered":"2024-04-24T23:15:43.192038+0000","last_clean":"2024-04-24T23:15:43.192038+0000","last_became_active":"2024-04-24T23:15:35.952655+0000","last_became_peered":"2024-04-24T23:15:35.952655+0000","last_unstale":"2024-04-24T23:15:43.192038+0000","last_undegraded":"2024-04-24T23:15:43.192038+0000","last_fullsized":"2024-04-24T23:15:43.192038+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:50:10.939396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":178,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.732636+0000","last_change":"2024-04-24T23:15:25.826673+0000","last_active":"2024-04-24T23:15:42.732636+0000","last_peered":"2024-04-24T23:15:42.732636+0000","last_clean":"2024-04-24T23:15:42.732636+0000","last_became_active":"2024-04-24T23:15:25.820030+0000","last_became_peered":"2024-04-24T23:15:25.820030+0000","last_unstale":"2024-04-24T23:15:42.732636+0000","last_undegraded":"2024-04-24T23:15:42.732636+0000","last_fullsized":"2024-04-24T23:15:42.732636+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:24.790928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:24.790928+0000","last_clean_scrub_stamp":"2024-04-24T23:15:24.790928+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:37:24.054332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.192066+0000","last_change":"2024-04-24T23:15:41.024712+0000","last_active":"2024-04-24T23:15:43.192066+0000","last_peered":"2024-04-24T23:15:43.192066+0000","last_clean":"2024-04-24T23:15:43.192066+0000","last_became_active":"2024-04-24T23:15:41.022697+0000","last_became_peered":"2024-04-24T23:15:41.022697+0000","last_unstale":"2024-04-24T23:15:43.192066+0000","last_undegraded":"2024-04-24T23:15:43.192066+0000","last_fullsized":"2024-04-24T23:15:43.192066+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:37:08.231105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2,0],"acting":[3,1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.191847+0000","last_change":"2024-04-24T23:15:37.000489+0000","last_active":"2024-04-24T23:15:43.191847+0000","last_peered":"2024-04-24T23:15:43.191847+0000","last_clean":"2024-04-24T23:15:43.191847+0000","last_became_active":"2024-04-24T23:15:37.000179+0000","last_became_peered":"2024-04-24T23:15:37.000179+0000","last_unstale":"2024-04-24T23:15:43.191847+0000","last_undegraded":"2024-04-24T23:15:43.191847+0000","last_fullsized":"2024-04-24T23:15:43.191847+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:06:37.125573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.051120+0000","last_change":"2024-04-24T23:15:35.966020+0000","last_active":"2024-04-24T23:15:42.051120+0000","last_peered":"2024-04-24T23:15:42.051120+0000","last_clean":"2024-04-24T23:15:42.051120+0000","last_became_active":"2024-04-24T23:15:35.965758+0000","last_became_peered":"2024-04-24T23:15:35.965758+0000","last_unstale":"2024-04-24T23:15:42.051120+0000","last_undegraded":"2024-04-24T23:15:42.051120+0000","last_fullsized":"2024-04-24T23:15:42.051120+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:39:21.233461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.106667+0000","last_change":"2024-04-24T23:15:41.024489+0000","last_active":"2024-04-24T23:15:42.106667+0000","last_peered":"2024-04-24T23:15:42.106667+0000","last_clean":"2024-04-24T23:15:42.106667+0000","last_became_active":"2024-04-24T23:15:41.024041+0000","last_became_peered":"2024-04-24T23:15:41.024041+0000","last_unstale":"2024-04-24T23:15:42.106667+0000","last_undegraded":"2024-04-24T23:15:42.106667+0000","last_fullsized":"2024-04-24T23:15:42.106667+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:27:18.197003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7,4],"acting":[2,3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.193516+0000","last_change":"2024-04-24T23:15:36.979887+0000","last_active":"2024-04-24T23:15:43.193516+0000","last_peered":"2024-04-24T23:15:43.193516+0000","last_clean":"2024-04-24T23:15:43.193516+0000","last_became_active":"2024-04-24T23:15:36.979611+0000","last_became_peered":"2024-04-24T23:15:36.979611+0000","last_unstale":"2024-04-24T23:15:43.193516+0000","last_undegraded":"2024-04-24T23:15:43.193516+0000","last_fullsized":"2024-04-24T23:15:43.193516+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:07:18.261077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"23'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.295532+0000","last_change":"2024-04-24T23:15:35.965192+0000","last_active":"2024-04-24T23:15:42.295532+0000","last_peered":"2024-04-24T23:15:42.295532+0000","last_clean":"2024-04-24T23:15:42.295532+0000","last_became_active":"2024-04-24T23:15:35.964873+0000","last_became_peered":"2024-04-24T23:15:35.964873+0000","last_unstale":"2024-04-24T23:15:42.295532+0000","last_undegraded":"2024-04-24T23:15:42.295532+0000","last_fullsized":"2024-04-24T23:15:42.295532+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:14:27.884944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.295549+0000","last_change":"2024-04-24T23:15:41.004002+0000","last_active":"2024-04-24T23:15:42.295549+0000","last_peered":"2024-04-24T23:15:42.295549+0000","last_clean":"2024-04-24T23:15:42.295549+0000","last_became_active":"2024-04-24T23:15:41.003335+0000","last_became_peered":"2024-04-24T23:15:41.003335+0000","last_unstale":"2024-04-24T23:15:42.295549+0000","last_undegraded":"2024-04-24T23:15:42.295549+0000","last_fullsized":"2024-04-24T23:15:42.295549+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:49:36.206903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1,3],"acting":[5,4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.193583+0000","last_change":"2024-04-24T23:15:37.002854+0000","last_active":"2024-04-24T23:15:43.193583+0000","last_peered":"2024-04-24T23:15:43.193583+0000","last_clean":"2024-04-24T23:15:43.193583+0000","last_became_active":"2024-04-24T23:15:37.002583+0000","last_became_peered":"2024-04-24T23:15:37.002583+0000","last_unstale":"2024-04-24T23:15:43.193583+0000","last_undegraded":"2024-04-24T23:15:43.193583+0000","last_fullsized":"2024-04-24T23:15:43.193583+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:27:04.251547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"23'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.050875+0000","last_change":"2024-04-24T23:15:35.948752+0000","last_active":"2024-04-24T23:15:42.050875+0000","last_peered":"2024-04-24T23:15:42.050875+0000","last_clean":"2024-04-24T23:15:42.050875+0000","last_became_active":"2024-04-24T23:15:35.948438+0000","last_became_peered":"2024-04-24T23:15:35.948438+0000","last_unstale":"2024-04-24T23:15:42.050875+0000","last_undegraded":"2024-04-24T23:15:42.050875+0000","last_fullsized":"2024-04-24T23:15:42.050875+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:45:56.006278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.190120+0000","last_change":"2024-04-24T23:15:41.017524+0000","last_active":"2024-04-24T23:15:43.190120+0000","last_peered":"2024-04-24T23:15:43.190120+0000","last_clean":"2024-04-24T23:15:43.190120+0000","last_became_active":"2024-04-24T23:15:41.017092+0000","last_became_peered":"2024-04-24T23:15:41.017092+0000","last_unstale":"2024-04-24T23:15:43.190120+0000","last_undegraded":"2024-04-24T23:15:43.190120+0000","last_fullsized":"2024-04-24T23:15:43.190120+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:04:53.235038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.107289+0000","last_change":"2024-04-24T23:15:36.990441+0000","last_active":"2024-04-24T23:15:42.107289+0000","last_peered":"2024-04-24T23:15:42.107289+0000","last_clean":"2024-04-24T23:15:42.107289+0000","last_became_active":"2024-04-24T23:15:36.989863+0000","last_became_peered":"2024-04-24T23:15:36.989863+0000","last_unstale":"2024-04-24T23:15:42.107289+0000","last_undegraded":"2024-04-24T23:15:42.107289+0000","last_fullsized":"2024-04-24T23:15:42.107289+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:50:07.975347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.f","version":"23'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.193960+0000","last_change":"2024-04-24T23:15:35.969317+0000","last_active":"2024-04-24T23:15:43.193960+0000","last_peered":"2024-04-24T23:15:43.193960+0000","last_clean":"2024-04-24T23:15:43.193960+0000","last_became_active":"2024-04-24T23:15:35.968994+0000","last_became_peered":"2024-04-24T23:15:35.968994+0000","last_unstale":"2024-04-24T23:15:43.193960+0000","last_undegraded":"2024-04-24T23:15:43.193960+0000","last_fullsized":"2024-04-24T23:15:43.193960+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:51:57.100772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.733165+0000","last_change":"2024-04-24T23:15:41.010103+0000","last_active":"2024-04-24T23:15:42.733165+0000","last_peered":"2024-04-24T23:15:42.733165+0000","last_clean":"2024-04-24T23:15:42.733165+0000","last_became_active":"2024-04-24T23:15:41.009328+0000","last_became_peered":"2024-04-24T23:15:41.009328+0000","last_unstale":"2024-04-24T23:15:42.733165+0000","last_undegraded":"2024-04-24T23:15:42.733165+0000","last_fullsized":"2024-04-24T23:15:42.733165+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:46:49.579501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,2],"acting":[1,5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.190520+0000","last_change":"2024-04-24T23:15:36.996765+0000","last_active":"2024-04-24T23:15:43.190520+0000","last_peered":"2024-04-24T23:15:43.190520+0000","last_clean":"2024-04-24T23:15:43.190520+0000","last_became_active":"2024-04-24T23:15:36.995801+0000","last_became_peered":"2024-04-24T23:15:36.995801+0000","last_unstale":"2024-04-24T23:15:43.190520+0000","last_undegraded":"2024-04-24T23:15:43.190520+0000","last_fullsized":"2024-04-24T23:15:43.190520+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:25:19.421667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"23'8","reported_seq":30,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.049288+0000","last_change":"2024-04-24T23:15:35.944023+0000","last_active":"2024-04-24T23:15:42.049288+0000","last_peered":"2024-04-24T23:15:42.049288+0000","last_clean":"2024-04-24T23:15:42.049288+0000","last_became_active":"2024-04-24T23:15:35.943710+0000","last_became_peered":"2024-04-24T23:15:35.943710+0000","last_unstale":"2024-04-24T23:15:42.049288+0000","last_undegraded":"2024-04-24T23:15:42.049288+0000","last_fullsized":"2024-04-24T23:15:42.049288+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:46:48.090958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.295767+0000","last_change":"2024-04-24T23:15:41.013027+0000","last_active":"2024-04-24T23:15:42.295767+0000","last_peered":"2024-04-24T23:15:42.295767+0000","last_clean":"2024-04-24T23:15:42.295767+0000","last_became_active":"2024-04-24T23:15:41.012587+0000","last_became_peered":"2024-04-24T23:15:41.012587+0000","last_unstale":"2024-04-24T23:15:42.295767+0000","last_undegraded":"2024-04-24T23:15:42.295767+0000","last_fullsized":"2024-04-24T23:15:42.295767+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:27:04.276246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1,6],"acting":[5,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.732647+0000","last_change":"2024-04-24T23:15:36.979702+0000","last_active":"2024-04-24T23:15:42.732647+0000","last_peered":"2024-04-24T23:15:42.732647+0000","last_clean":"2024-04-24T23:15:42.732647+0000","last_became_active":"2024-04-24T23:15:36.979356+0000","last_became_peered":"2024-04-24T23:15:36.979356+0000","last_unstale":"2024-04-24T23:15:42.732647+0000","last_undegraded":"2024-04-24T23:15:42.732647+0000","last_fullsized":"2024-04-24T23:15:42.732647+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:14:24.309192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.193698+0000","last_change":"2024-04-24T23:15:35.973026+0000","last_active":"2024-04-24T23:15:43.193698+0000","last_peered":"2024-04-24T23:15:43.193698+0000","last_clean":"2024-04-24T23:15:43.193698+0000","last_became_active":"2024-04-24T23:15:35.972655+0000","last_became_peered":"2024-04-24T23:15:35.972655+0000","last_unstale":"2024-04-24T23:15:43.193698+0000","last_undegraded":"2024-04-24T23:15:43.193698+0000","last_fullsized":"2024-04-24T23:15:43.193698+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:25:34.352908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.295901+0000","last_change":"2024-04-24T23:15:41.004221+0000","last_active":"2024-04-24T23:15:42.295901+0000","last_peered":"2024-04-24T23:15:42.295901+0000","last_clean":"2024-04-24T23:15:42.295901+0000","last_became_active":"2024-04-24T23:15:41.003790+0000","last_became_peered":"2024-04-24T23:15:41.003790+0000","last_unstale":"2024-04-24T23:15:42.295901+0000","last_undegraded":"2024-04-24T23:15:42.295901+0000","last_fullsized":"2024-04-24T23:15:42.295901+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:28:58.866024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6,4],"acting":[5,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.191414+0000","last_change":"2024-04-24T23:15:36.997389+0000","last_active":"2024-04-24T23:15:43.191414+0000","last_peered":"2024-04-24T23:15:43.191414+0000","last_clean":"2024-04-24T23:15:43.191414+0000","last_became_active":"2024-04-24T23:15:36.995923+0000","last_became_peered":"2024-04-24T23:15:36.995923+0000","last_unstale":"2024-04-24T23:15:43.191414+0000","last_undegraded":"2024-04-24T23:15:43.191414+0000","last_fullsized":"2024-04-24T23:15:43.191414+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:55:47.134320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.191422+0000","last_change":"2024-04-24T23:15:35.957715+0000","last_active":"2024-04-24T23:15:43.191422+0000","last_peered":"2024-04-24T23:15:43.191422+0000","last_clean":"2024-04-24T23:15:43.191422+0000","last_became_active":"2024-04-24T23:15:35.957088+0000","last_became_peered":"2024-04-24T23:15:35.957088+0000","last_unstale":"2024-04-24T23:15:43.191422+0000","last_undegraded":"2024-04-24T23:15:43.191422+0000","last_fullsized":"2024-04-24T23:15:43.191422+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:30:34.483669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.050833+0000","last_change":"2024-04-24T23:15:41.008165+0000","last_active":"2024-04-24T23:15:42.050833+0000","last_peered":"2024-04-24T23:15:42.050833+0000","last_clean":"2024-04-24T23:15:42.050833+0000","last_became_active":"2024-04-24T23:15:41.007555+0000","last_became_peered":"2024-04-24T23:15:41.007555+0000","last_unstale":"2024-04-24T23:15:42.050833+0000","last_undegraded":"2024-04-24T23:15:42.050833+0000","last_fullsized":"2024-04-24T23:15:42.050833+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:18:08.195747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0,7],"acting":[6,4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.050006+0000","last_change":"2024-04-24T23:15:36.991382+0000","last_active":"2024-04-24T23:15:42.050006+0000","last_peered":"2024-04-24T23:15:42.050006+0000","last_clean":"2024-04-24T23:15:42.050006+0000","last_became_active":"2024-04-24T23:15:36.991115+0000","last_became_peered":"2024-04-24T23:15:36.991115+0000","last_unstale":"2024-04-24T23:15:42.050006+0000","last_undegraded":"2024-04-24T23:15:42.050006+0000","last_fullsized":"2024-04-24T23:15:42.050006+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:21:51.863947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"23'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.188008+0000","last_change":"2024-04-24T23:15:35.964245+0000","last_active":"2024-04-24T23:15:43.188008+0000","last_peered":"2024-04-24T23:15:43.188008+0000","last_clean":"2024-04-24T23:15:43.188008+0000","last_became_active":"2024-04-24T23:15:35.963907+0000","last_became_peered":"2024-04-24T23:15:35.963907+0000","last_unstale":"2024-04-24T23:15:43.188008+0000","last_undegraded":"2024-04-24T23:15:43.188008+0000","last_fullsized":"2024-04-24T23:15:43.188008+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:15:00.913899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.192087+0000","last_change":"2024-04-24T23:15:41.021981+0000","last_active":"2024-04-24T23:15:43.192087+0000","last_peered":"2024-04-24T23:15:43.192087+0000","last_clean":"2024-04-24T23:15:43.192087+0000","last_became_active":"2024-04-24T23:15:41.019193+0000","last_became_peered":"2024-04-24T23:15:41.019193+0000","last_unstale":"2024-04-24T23:15:43.192087+0000","last_undegraded":"2024-04-24T23:15:43.192087+0000","last_fullsized":"2024-04-24T23:15:43.192087+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:04:39.828502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5,6],"acting":[3,0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.193203+0000","last_change":"2024-04-24T23:15:36.986833+0000","last_active":"2024-04-24T23:15:43.193203+0000","last_peered":"2024-04-24T23:15:43.193203+0000","last_clean":"2024-04-24T23:15:43.193203+0000","last_became_active":"2024-04-24T23:15:36.986484+0000","last_became_peered":"2024-04-24T23:15:36.986484+0000","last_unstale":"2024-04-24T23:15:43.193203+0000","last_undegraded":"2024-04-24T23:15:43.193203+0000","last_fullsized":"2024-04-24T23:15:43.193203+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:12:32.245267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.14","version":"24'10","reported_seq":32,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:59.011711+0000","last_change":"2024-04-24T23:15:35.957124+0000","last_active":"2024-04-24T23:15:59.011711+0000","last_peered":"2024-04-24T23:15:59.011711+0000","last_clean":"2024-04-24T23:15:59.011711+0000","last_became_active":"2024-04-24T23:15:35.951923+0000","last_became_peered":"2024-04-24T23:15:35.951923+0000","last_unstale":"2024-04-24T23:15:59.011711+0000","last_undegraded":"2024-04-24T23:15:59.011711+0000","last_fullsized":"2024-04-24T23:15:59.011711+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:44:16.820640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.049936+0000","last_change":"2024-04-24T23:15:41.015494+0000","last_active":"2024-04-24T23:15:42.049936+0000","last_peered":"2024-04-24T23:15:42.049936+0000","last_clean":"2024-04-24T23:15:42.049936+0000","last_became_active":"2024-04-24T23:15:41.015179+0000","last_became_peered":"2024-04-24T23:15:41.015179+0000","last_unstale":"2024-04-24T23:15:42.049936+0000","last_undegraded":"2024-04-24T23:15:42.049936+0000","last_fullsized":"2024-04-24T23:15:42.049936+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:18:32.208506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5,0],"acting":[4,3,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.190450+0000","last_change":"2024-04-24T23:15:36.999334+0000","last_active":"2024-04-24T23:15:43.190450+0000","last_peered":"2024-04-24T23:15:43.190450+0000","last_clean":"2024-04-24T23:15:43.190450+0000","last_became_active":"2024-04-24T23:15:36.995470+0000","last_became_peered":"2024-04-24T23:15:36.995470+0000","last_unstale":"2024-04-24T23:15:43.190450+0000","last_undegraded":"2024-04-24T23:15:43.190450+0000","last_fullsized":"2024-04-24T23:15:43.190450+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:19:57.973402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.15","version":"23'3","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.187978+0000","last_change":"2024-04-24T23:15:35.946041+0000","last_active":"2024-04-24T23:15:43.187978+0000","last_peered":"2024-04-24T23:15:43.187978+0000","last_clean":"2024-04-24T23:15:43.187978+0000","last_became_active":"2024-04-24T23:15:35.944805+0000","last_became_peered":"2024-04-24T23:15:35.944805+0000","last_unstale":"2024-04-24T23:15:43.187978+0000","last_undegraded":"2024-04-24T23:15:43.187978+0000","last_fullsized":"2024-04-24T23:15:43.187978+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:52:19.261306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.107105+0000","last_change":"2024-04-24T23:15:41.024392+0000","last_active":"2024-04-24T23:15:42.107105+0000","last_peered":"2024-04-24T23:15:42.107105+0000","last_clean":"2024-04-24T23:15:42.107105+0000","last_became_active":"2024-04-24T23:15:41.023951+0000","last_became_peered":"2024-04-24T23:15:41.023951+0000","last_unstale":"2024-04-24T23:15:42.107105+0000","last_undegraded":"2024-04-24T23:15:42.107105+0000","last_fullsized":"2024-04-24T23:15:42.107105+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:24:17.810117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1,7],"acting":[2,3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.295845+0000","last_change":"2024-04-24T23:15:36.992243+0000","last_active":"2024-04-24T23:15:42.295845+0000","last_peered":"2024-04-24T23:15:42.295845+0000","last_clean":"2024-04-24T23:15:42.295845+0000","last_became_active":"2024-04-24T23:15:36.992002+0000","last_became_peered":"2024-04-24T23:15:36.992002+0000","last_unstale":"2024-04-24T23:15:42.295845+0000","last_undegraded":"2024-04-24T23:15:42.295845+0000","last_fullsized":"2024-04-24T23:15:42.295845+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:52:15.911009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.193044+0000","last_change":"2024-04-24T23:15:35.967408+0000","last_active":"2024-04-24T23:15:43.193044+0000","last_peered":"2024-04-24T23:15:43.193044+0000","last_clean":"2024-04-24T23:15:43.193044+0000","last_became_active":"2024-04-24T23:15:35.966646+0000","last_became_peered":"2024-04-24T23:15:35.966646+0000","last_unstale":"2024-04-24T23:15:43.193044+0000","last_undegraded":"2024-04-24T23:15:43.193044+0000","last_fullsized":"2024-04-24T23:15:43.193044+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:07:07.931773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.187622+0000","last_change":"2024-04-24T23:15:41.021697+0000","last_active":"2024-04-24T23:15:43.187622+0000","last_peered":"2024-04-24T23:15:43.187622+0000","last_clean":"2024-04-24T23:15:43.187622+0000","last_became_active":"2024-04-24T23:15:41.021281+0000","last_became_peered":"2024-04-24T23:15:41.021281+0000","last_unstale":"2024-04-24T23:15:43.187622+0000","last_undegraded":"2024-04-24T23:15:43.187622+0000","last_fullsized":"2024-04-24T23:15:43.187622+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:59:37.306593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3,2],"acting":[7,1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.107218+0000","last_change":"2024-04-24T23:15:36.990988+0000","last_active":"2024-04-24T23:15:42.107218+0000","last_peered":"2024-04-24T23:15:42.107218+0000","last_clean":"2024-04-24T23:15:42.107218+0000","last_became_active":"2024-04-24T23:15:36.990711+0000","last_became_peered":"2024-04-24T23:15:36.990711+0000","last_unstale":"2024-04-24T23:15:42.107218+0000","last_undegraded":"2024-04-24T23:15:42.107218+0000","last_fullsized":"2024-04-24T23:15:42.107218+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:33:20.663776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.734116+0000","last_change":"2024-04-24T23:15:35.957838+0000","last_active":"2024-04-24T23:15:42.734116+0000","last_peered":"2024-04-24T23:15:42.734116+0000","last_clean":"2024-04-24T23:15:42.734116+0000","last_became_active":"2024-04-24T23:15:35.956870+0000","last_became_peered":"2024-04-24T23:15:35.956870+0000","last_unstale":"2024-04-24T23:15:42.734116+0000","last_undegraded":"2024-04-24T23:15:42.734116+0000","last_fullsized":"2024-04-24T23:15:42.734116+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:56:03.889693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.049546+0000","last_change":"2024-04-24T23:15:41.006098+0000","last_active":"2024-04-24T23:15:42.049546+0000","last_peered":"2024-04-24T23:15:42.049546+0000","last_clean":"2024-04-24T23:15:42.049546+0000","last_became_active":"2024-04-24T23:15:41.005578+0000","last_became_peered":"2024-04-24T23:15:41.005578+0000","last_unstale":"2024-04-24T23:15:42.049546+0000","last_undegraded":"2024-04-24T23:15:42.049546+0000","last_fullsized":"2024-04-24T23:15:42.049546+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:21:12.802112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3,7],"acting":[4,0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.049896+0000","last_change":"2024-04-24T23:15:41.022122+0000","last_active":"2024-04-24T23:15:42.049896+0000","last_peered":"2024-04-24T23:15:42.049896+0000","last_clean":"2024-04-24T23:15:42.049896+0000","last_became_active":"2024-04-24T23:15:41.021698+0000","last_became_peered":"2024-04-24T23:15:41.021698+0000","last_unstale":"2024-04-24T23:15:42.049896+0000","last_undegraded":"2024-04-24T23:15:42.049896+0000","last_fullsized":"2024-04-24T23:15:42.049896+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:13:56.730050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5,3],"acting":[4,1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.049600+0000","last_change":"2024-04-24T23:15:36.979816+0000","last_active":"2024-04-24T23:15:42.049600+0000","last_peered":"2024-04-24T23:15:42.049600+0000","last_clean":"2024-04-24T23:15:42.049600+0000","last_became_active":"2024-04-24T23:15:36.979553+0000","last_became_peered":"2024-04-24T23:15:36.979553+0000","last_unstale":"2024-04-24T23:15:42.049600+0000","last_undegraded":"2024-04-24T23:15:42.049600+0000","last_fullsized":"2024-04-24T23:15:42.049600+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:55:47.134320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.190578+0000","last_change":"2024-04-24T23:15:35.964744+0000","last_active":"2024-04-24T23:15:43.190578+0000","last_peered":"2024-04-24T23:15:43.190578+0000","last_clean":"2024-04-24T23:15:43.190578+0000","last_became_active":"2024-04-24T23:15:35.964434+0000","last_became_peered":"2024-04-24T23:15:35.964434+0000","last_unstale":"2024-04-24T23:15:43.190578+0000","last_undegraded":"2024-04-24T23:15:43.190578+0000","last_fullsized":"2024-04-24T23:15:43.190578+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:26:32.587197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.191463+0000","last_change":"2024-04-24T23:15:41.022032+0000","last_active":"2024-04-24T23:15:43.191463+0000","last_peered":"2024-04-24T23:15:43.191463+0000","last_clean":"2024-04-24T23:15:43.191463+0000","last_became_active":"2024-04-24T23:15:41.021616+0000","last_became_peered":"2024-04-24T23:15:41.021616+0000","last_unstale":"2024-04-24T23:15:43.191463+0000","last_undegraded":"2024-04-24T23:15:43.191463+0000","last_fullsized":"2024-04-24T23:15:43.191463+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:08:59.242392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5,1],"acting":[3,2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.295164+0000","last_change":"2024-04-24T23:15:36.967198+0000","last_active":"2024-04-24T23:15:42.295164+0000","last_peered":"2024-04-24T23:15:42.295164+0000","last_clean":"2024-04-24T23:15:42.295164+0000","last_became_active":"2024-04-24T23:15:36.966910+0000","last_became_peered":"2024-04-24T23:15:36.966910+0000","last_unstale":"2024-04-24T23:15:42.295164+0000","last_undegraded":"2024-04-24T23:15:42.295164+0000","last_fullsized":"2024-04-24T23:15:42.295164+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:51:19.788437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.049504+0000","last_change":"2024-04-24T23:15:35.967169+0000","last_active":"2024-04-24T23:15:42.049504+0000","last_peered":"2024-04-24T23:15:42.049504+0000","last_clean":"2024-04-24T23:15:42.049504+0000","last_became_active":"2024-04-24T23:15:35.966939+0000","last_became_peered":"2024-04-24T23:15:35.966939+0000","last_unstale":"2024-04-24T23:15:42.049504+0000","last_undegraded":"2024-04-24T23:15:42.049504+0000","last_fullsized":"2024-04-24T23:15:42.049504+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:38:55.460569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"23'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.191308+0000","last_change":"2024-04-24T23:15:35.971236+0000","last_active":"2024-04-24T23:15:43.191308+0000","last_peered":"2024-04-24T23:15:43.191308+0000","last_clean":"2024-04-24T23:15:43.191308+0000","last_became_active":"2024-04-24T23:15:35.970908+0000","last_became_peered":"2024-04-24T23:15:35.970908+0000","last_unstale":"2024-04-24T23:15:43.191308+0000","last_undegraded":"2024-04-24T23:15:43.191308+0000","last_fullsized":"2024-04-24T23:15:43.191308+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:14:15.548024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.050320+0000","last_change":"2024-04-24T23:15:36.988156+0000","last_active":"2024-04-24T23:15:42.050320+0000","last_peered":"2024-04-24T23:15:42.050320+0000","last_clean":"2024-04-24T23:15:42.050320+0000","last_became_active":"2024-04-24T23:15:36.987896+0000","last_became_peered":"2024-04-24T23:15:36.987896+0000","last_unstale":"2024-04-24T23:15:42.050320+0000","last_undegraded":"2024-04-24T23:15:42.050320+0000","last_fullsized":"2024-04-24T23:15:42.050320+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:22:06.869644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.050441+0000","last_change":"2024-04-24T23:15:40.997559+0000","last_active":"2024-04-24T23:15:42.050441+0000","last_peered":"2024-04-24T23:15:42.050441+0000","last_clean":"2024-04-24T23:15:42.050441+0000","last_became_active":"2024-04-24T23:15:40.997102+0000","last_became_peered":"2024-04-24T23:15:40.997102+0000","last_unstale":"2024-04-24T23:15:42.050441+0000","last_undegraded":"2024-04-24T23:15:42.050441+0000","last_fullsized":"2024-04-24T23:15:42.050441+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:25:02.046761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,7],"acting":[4,6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"23'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.295440+0000","last_change":"2024-04-24T23:15:35.944842+0000","last_active":"2024-04-24T23:15:42.295440+0000","last_peered":"2024-04-24T23:15:42.295440+0000","last_clean":"2024-04-24T23:15:42.295440+0000","last_became_active":"2024-04-24T23:15:35.944563+0000","last_became_peered":"2024-04-24T23:15:35.944563+0000","last_unstale":"2024-04-24T23:15:42.295440+0000","last_undegraded":"2024-04-24T23:15:42.295440+0000","last_fullsized":"2024-04-24T23:15:42.295440+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:40:58.933384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.732992+0000","last_change":"2024-04-24T23:15:36.981103+0000","last_active":"2024-04-24T23:15:42.732992+0000","last_peered":"2024-04-24T23:15:42.732992+0000","last_clean":"2024-04-24T23:15:42.732992+0000","last_became_active":"2024-04-24T23:15:36.979637+0000","last_became_peered":"2024-04-24T23:15:36.979637+0000","last_unstale":"2024-04-24T23:15:42.732992+0000","last_undegraded":"2024-04-24T23:15:42.732992+0000","last_fullsized":"2024-04-24T23:15:42.732992+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:09:39.750272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.188153+0000","last_change":"2024-04-24T23:15:41.017087+0000","last_active":"2024-04-24T23:15:43.188153+0000","last_peered":"2024-04-24T23:15:43.188153+0000","last_clean":"2024-04-24T23:15:43.188153+0000","last_became_active":"2024-04-24T23:15:41.016696+0000","last_became_peered":"2024-04-24T23:15:41.016696+0000","last_unstale":"2024-04-24T23:15:43.188153+0000","last_undegraded":"2024-04-24T23:15:43.188153+0000","last_fullsized":"2024-04-24T23:15:43.188153+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:05:57.823447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1,0],"acting":[7,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.106609+0000","last_change":"2024-04-24T23:15:36.970211+0000","last_active":"2024-04-24T23:15:42.106609+0000","last_peered":"2024-04-24T23:15:42.106609+0000","last_clean":"2024-04-24T23:15:42.106609+0000","last_became_active":"2024-04-24T23:15:36.969090+0000","last_became_peered":"2024-04-24T23:15:36.969090+0000","last_unstale":"2024-04-24T23:15:42.106609+0000","last_undegraded":"2024-04-24T23:15:42.106609+0000","last_fullsized":"2024-04-24T23:15:42.106609+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:16:00.231012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.187632+0000","last_change":"2024-04-24T23:15:36.982528+0000","last_active":"2024-04-24T23:15:43.187632+0000","last_peered":"2024-04-24T23:15:43.187632+0000","last_clean":"2024-04-24T23:15:43.187632+0000","last_became_active":"2024-04-24T23:15:36.982299+0000","last_became_peered":"2024-04-24T23:15:36.982299+0000","last_unstale":"2024-04-24T23:15:43.187632+0000","last_undegraded":"2024-04-24T23:15:43.187632+0000","last_fullsized":"2024-04-24T23:15:43.187632+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:02:17.732324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.191005+0000","last_change":"2024-04-24T23:15:36.989899+0000","last_active":"2024-04-24T23:15:43.191005+0000","last_peered":"2024-04-24T23:15:43.191005+0000","last_clean":"2024-04-24T23:15:43.191005+0000","last_became_active":"2024-04-24T23:15:36.989557+0000","last_became_peered":"2024-04-24T23:15:36.989557+0000","last_unstale":"2024-04-24T23:15:43.191005+0000","last_undegraded":"2024-04-24T23:15:43.191005+0000","last_fullsized":"2024-04-24T23:15:43.191005+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:15:13.513341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.295325+0000","last_change":"2024-04-24T23:15:36.969628+0000","last_active":"2024-04-24T23:15:42.295325+0000","last_peered":"2024-04-24T23:15:42.295325+0000","last_clean":"2024-04-24T23:15:42.295325+0000","last_became_active":"2024-04-24T23:15:36.969327+0000","last_became_peered":"2024-04-24T23:15:36.969327+0000","last_unstale":"2024-04-24T23:15:42.295325+0000","last_undegraded":"2024-04-24T23:15:42.295325+0000","last_fullsized":"2024-04-24T23:15:42.295325+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:08:47.021999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.050888+0000","last_change":"2024-04-24T23:15:36.986343+0000","last_active":"2024-04-24T23:15:42.050888+0000","last_peered":"2024-04-24T23:15:42.050888+0000","last_clean":"2024-04-24T23:15:42.050888+0000","last_became_active":"2024-04-24T23:15:36.986026+0000","last_became_peered":"2024-04-24T23:15:36.986026+0000","last_unstale":"2024-04-24T23:15:42.050888+0000","last_undegraded":"2024-04-24T23:15:42.050888+0000","last_fullsized":"2024-04-24T23:15:42.050888+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:10:00.347826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.049809+0000","last_change":"2024-04-24T23:15:36.968538+0000","last_active":"2024-04-24T23:15:42.049809+0000","last_peered":"2024-04-24T23:15:42.049809+0000","last_clean":"2024-04-24T23:15:42.049809+0000","last_became_active":"2024-04-24T23:15:36.968263+0000","last_became_peered":"2024-04-24T23:15:36.968263+0000","last_unstale":"2024-04-24T23:15:42.049809+0000","last_undegraded":"2024-04-24T23:15:42.049809+0000","last_fullsized":"2024-04-24T23:15:42.049809+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:07:43.731738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.107186+0000","last_change":"2024-04-24T23:15:36.991008+0000","last_active":"2024-04-24T23:15:42.107186+0000","last_peered":"2024-04-24T23:15:42.107186+0000","last_clean":"2024-04-24T23:15:42.107186+0000","last_became_active":"2024-04-24T23:15:36.990764+0000","last_became_peered":"2024-04-24T23:15:36.990764+0000","last_unstale":"2024-04-24T23:15:42.107186+0000","last_undegraded":"2024-04-24T23:15:42.107186+0000","last_fullsized":"2024-04-24T23:15:42.107186+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:50:21.865233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.295685+0000","last_change":"2024-04-24T23:15:36.993550+0000","last_active":"2024-04-24T23:15:42.295685+0000","last_peered":"2024-04-24T23:15:42.295685+0000","last_clean":"2024-04-24T23:15:42.295685+0000","last_became_active":"2024-04-24T23:15:36.993310+0000","last_became_peered":"2024-04-24T23:15:36.993310+0000","last_unstale":"2024-04-24T23:15:42.295685+0000","last_undegraded":"2024-04-24T23:15:42.295685+0000","last_fullsized":"2024-04-24T23:15:42.295685+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:21:20.617940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.190455+0000","last_change":"2024-04-24T23:15:36.987810+0000","last_active":"2024-04-24T23:15:43.190455+0000","last_peered":"2024-04-24T23:15:43.190455+0000","last_clean":"2024-04-24T23:15:43.190455+0000","last_became_active":"2024-04-24T23:15:36.987540+0000","last_became_peered":"2024-04-24T23:15:36.987540+0000","last_unstale":"2024-04-24T23:15:43.190455+0000","last_undegraded":"2024-04-24T23:15:43.190455+0000","last_fullsized":"2024-04-24T23:15:43.190455+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:49:56.176021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.295070+0000","last_change":"2024-04-24T23:15:36.981187+0000","last_active":"2024-04-24T23:15:42.295070+0000","last_peered":"2024-04-24T23:15:42.295070+0000","last_clean":"2024-04-24T23:15:42.295070+0000","last_became_active":"2024-04-24T23:15:36.980838+0000","last_became_peered":"2024-04-24T23:15:36.980838+0000","last_unstale":"2024-04-24T23:15:42.295070+0000","last_undegraded":"2024-04-24T23:15:42.295070+0000","last_fullsized":"2024-04-24T23:15:42.295070+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:57:41.178367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.732913+0000","last_change":"2024-04-24T23:15:36.999238+0000","last_active":"2024-04-24T23:15:42.732913+0000","last_peered":"2024-04-24T23:15:42.732913+0000","last_clean":"2024-04-24T23:15:42.732913+0000","last_became_active":"2024-04-24T23:15:36.996245+0000","last_became_peered":"2024-04-24T23:15:36.996245+0000","last_unstale":"2024-04-24T23:15:42.732913+0000","last_undegraded":"2024-04-24T23:15:42.732913+0000","last_fullsized":"2024-04-24T23:15:42.732913+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:21:20.443640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.050214+0000","last_change":"2024-04-24T23:15:36.981219+0000","last_active":"2024-04-24T23:15:42.050214+0000","last_peered":"2024-04-24T23:15:42.050214+0000","last_clean":"2024-04-24T23:15:42.050214+0000","last_became_active":"2024-04-24T23:15:36.980911+0000","last_became_peered":"2024-04-24T23:15:36.980911+0000","last_unstale":"2024-04-24T23:15:42.050214+0000","last_undegraded":"2024-04-24T23:15:42.050214+0000","last_fullsized":"2024-04-24T23:15:42.050214+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:19:57.973402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.193222+0000","last_change":"2024-04-24T23:15:36.991418+0000","last_active":"2024-04-24T23:15:43.193222+0000","last_peered":"2024-04-24T23:15:43.193222+0000","last_clean":"2024-04-24T23:15:43.193222+0000","last_became_active":"2024-04-24T23:15:36.989792+0000","last_became_peered":"2024-04-24T23:15:36.989792+0000","last_unstale":"2024-04-24T23:15:43.193222+0000","last_undegraded":"2024-04-24T23:15:43.193222+0000","last_fullsized":"2024-04-24T23:15:43.193222+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:02:53.601311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.192125+0000","last_change":"2024-04-24T23:15:36.987810+0000","last_active":"2024-04-24T23:15:43.192125+0000","last_peered":"2024-04-24T23:15:43.192125+0000","last_clean":"2024-04-24T23:15:43.192125+0000","last_became_active":"2024-04-24T23:15:36.987494+0000","last_became_peered":"2024-04-24T23:15:36.987494+0000","last_unstale":"2024-04-24T23:15:43.192125+0000","last_undegraded":"2024-04-24T23:15:43.192125+0000","last_fullsized":"2024-04-24T23:15:43.192125+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:43:21.141172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.188217+0000","last_change":"2024-04-24T23:15:36.986576+0000","last_active":"2024-04-24T23:15:43.188217+0000","last_peered":"2024-04-24T23:15:43.188217+0000","last_clean":"2024-04-24T23:15:43.188217+0000","last_became_active":"2024-04-24T23:15:36.986272+0000","last_became_peered":"2024-04-24T23:15:36.986272+0000","last_unstale":"2024-04-24T23:15:43.188217+0000","last_undegraded":"2024-04-24T23:15:43.188217+0000","last_fullsized":"2024-04-24T23:15:43.188217+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:55:08.894945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.192001+0000","last_change":"2024-04-24T23:15:36.998812+0000","last_active":"2024-04-24T23:15:43.192001+0000","last_peered":"2024-04-24T23:15:43.192001+0000","last_clean":"2024-04-24T23:15:43.192001+0000","last_became_active":"2024-04-24T23:15:36.998456+0000","last_became_peered":"2024-04-24T23:15:36.998456+0000","last_unstale":"2024-04-24T23:15:43.192001+0000","last_undegraded":"2024-04-24T23:15:43.192001+0000","last_fullsized":"2024-04-24T23:15:43.192001+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:33:11.424218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.190262+0000","last_change":"2024-04-24T23:15:36.996521+0000","last_active":"2024-04-24T23:15:43.190262+0000","last_peered":"2024-04-24T23:15:43.190262+0000","last_clean":"2024-04-24T23:15:43.190262+0000","last_became_active":"2024-04-24T23:15:36.995507+0000","last_became_peered":"2024-04-24T23:15:36.995507+0000","last_unstale":"2024-04-24T23:15:43.190262+0000","last_undegraded":"2024-04-24T23:15:43.190262+0000","last_fullsized":"2024-04-24T23:15:43.190262+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:58:50.153471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.294996+0000","last_change":"2024-04-24T23:15:36.984994+0000","last_active":"2024-04-24T23:15:42.294996+0000","last_peered":"2024-04-24T23:15:42.294996+0000","last_clean":"2024-04-24T23:15:42.294996+0000","last_became_active":"2024-04-24T23:15:36.984741+0000","last_became_peered":"2024-04-24T23:15:36.984741+0000","last_unstale":"2024-04-24T23:15:42.294996+0000","last_undegraded":"2024-04-24T23:15:42.294996+0000","last_fullsized":"2024-04-24T23:15:42.294996+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:45:23.429836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.049917+0000","last_change":"2024-04-24T23:15:36.966165+0000","last_active":"2024-04-24T23:15:42.049917+0000","last_peered":"2024-04-24T23:15:42.049917+0000","last_clean":"2024-04-24T23:15:42.049917+0000","last_became_active":"2024-04-24T23:15:36.965877+0000","last_became_peered":"2024-04-24T23:15:36.965877+0000","last_unstale":"2024-04-24T23:15:42.049917+0000","last_undegraded":"2024-04-24T23:15:42.049917+0000","last_fullsized":"2024-04-24T23:15:42.049917+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:30:12.390978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.050537+0000","last_change":"2024-04-24T23:15:36.990667+0000","last_active":"2024-04-24T23:15:42.050537+0000","last_peered":"2024-04-24T23:15:42.050537+0000","last_clean":"2024-04-24T23:15:42.050537+0000","last_became_active":"2024-04-24T23:15:36.990383+0000","last_became_peered":"2024-04-24T23:15:36.990383+0000","last_unstale":"2024-04-24T23:15:42.050537+0000","last_undegraded":"2024-04-24T23:15:42.050537+0000","last_fullsized":"2024-04-24T23:15:42.050537+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:36:43.850046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.193024+0000","last_change":"2024-04-24T23:15:36.991686+0000","last_active":"2024-04-24T23:15:43.193024+0000","last_peered":"2024-04-24T23:15:43.193024+0000","last_clean":"2024-04-24T23:15:43.193024+0000","last_became_active":"2024-04-24T23:15:36.990155+0000","last_became_peered":"2024-04-24T23:15:36.990155+0000","last_unstale":"2024-04-24T23:15:43.193024+0000","last_undegraded":"2024-04-24T23:15:43.193024+0000","last_fullsized":"2024-04-24T23:15:43.193024+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:00:52.424019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.192909+0000","last_change":"2024-04-24T23:15:36.992958+0000","last_active":"2024-04-24T23:15:43.192909+0000","last_peered":"2024-04-24T23:15:43.192909+0000","last_clean":"2024-04-24T23:15:43.192909+0000","last_became_active":"2024-04-24T23:15:36.992608+0000","last_became_peered":"2024-04-24T23:15:36.992608+0000","last_unstale":"2024-04-24T23:15:43.192909+0000","last_undegraded":"2024-04-24T23:15:43.192909+0000","last_fullsized":"2024-04-24T23:15:43.192909+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:15:30.416763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.192849+0000","last_change":"2024-04-24T23:15:36.992943+0000","last_active":"2024-04-24T23:15:43.192849+0000","last_peered":"2024-04-24T23:15:43.192849+0000","last_clean":"2024-04-24T23:15:43.192849+0000","last_became_active":"2024-04-24T23:15:36.992567+0000","last_became_peered":"2024-04-24T23:15:36.992567+0000","last_unstale":"2024-04-24T23:15:43.192849+0000","last_undegraded":"2024-04-24T23:15:43.192849+0000","last_fullsized":"2024-04-24T23:15:43.192849+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:50:43.209232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.734006+0000","last_change":"2024-04-24T23:15:36.999697+0000","last_active":"2024-04-24T23:15:42.734006+0000","last_peered":"2024-04-24T23:15:42.734006+0000","last_clean":"2024-04-24T23:15:42.734006+0000","last_became_active":"2024-04-24T23:15:36.999413+0000","last_became_peered":"2024-04-24T23:15:36.999413+0000","last_unstale":"2024-04-24T23:15:42.734006+0000","last_undegraded":"2024-04-24T23:15:42.734006+0000","last_fullsized":"2024-04-24T23:15:42.734006+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:25:42.856471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.295006+0000","last_change":"2024-04-24T23:15:36.986742+0000","last_active":"2024-04-24T23:15:42.295006+0000","last_peered":"2024-04-24T23:15:42.295006+0000","last_clean":"2024-04-24T23:15:42.295006+0000","last_became_active":"2024-04-24T23:15:36.986013+0000","last_became_peered":"2024-04-24T23:15:36.986013+0000","last_unstale":"2024-04-24T23:15:42.295006+0000","last_undegraded":"2024-04-24T23:15:42.295006+0000","last_fullsized":"2024-04-24T23:15:42.295006+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:16:15.203151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.193480+0000","last_change":"2024-04-24T23:15:36.984336+0000","last_active":"2024-04-24T23:15:43.193480+0000","last_peered":"2024-04-24T23:15:43.193480+0000","last_clean":"2024-04-24T23:15:43.193480+0000","last_became_active":"2024-04-24T23:15:36.983966+0000","last_became_peered":"2024-04-24T23:15:36.983966+0000","last_unstale":"2024-04-24T23:15:43.193480+0000","last_undegraded":"2024-04-24T23:15:43.193480+0000","last_fullsized":"2024-04-24T23:15:43.193480+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:54:16.319608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.106762+0000","last_change":"2024-04-24T23:15:36.979079+0000","last_active":"2024-04-24T23:15:42.106762+0000","last_peered":"2024-04-24T23:15:42.106762+0000","last_clean":"2024-04-24T23:15:42.106762+0000","last_became_active":"2024-04-24T23:15:36.978706+0000","last_became_peered":"2024-04-24T23:15:36.978706+0000","last_unstale":"2024-04-24T23:15:42.106762+0000","last_undegraded":"2024-04-24T23:15:42.106762+0000","last_fullsized":"2024-04-24T23:15:42.106762+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:53:39.888150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.193929+0000","last_change":"2024-04-24T23:15:36.986008+0000","last_active":"2024-04-24T23:15:43.193929+0000","last_peered":"2024-04-24T23:15:43.193929+0000","last_clean":"2024-04-24T23:15:43.193929+0000","last_became_active":"2024-04-24T23:15:36.985552+0000","last_became_peered":"2024-04-24T23:15:36.985552+0000","last_unstale":"2024-04-24T23:15:43.193929+0000","last_undegraded":"2024-04-24T23:15:43.193929+0000","last_fullsized":"2024-04-24T23:15:43.193929+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:36:50.827503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.106908+0000","last_change":"2024-04-24T23:15:36.978673+0000","last_active":"2024-04-24T23:15:42.106908+0000","last_peered":"2024-04-24T23:15:42.106908+0000","last_clean":"2024-04-24T23:15:42.106908+0000","last_became_active":"2024-04-24T23:15:36.972744+0000","last_became_peered":"2024-04-24T23:15:36.972744+0000","last_unstale":"2024-04-24T23:15:42.106908+0000","last_undegraded":"2024-04-24T23:15:42.106908+0000","last_fullsized":"2024-04-24T23:15:42.106908+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:28:57.028825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.295395+0000","last_change":"2024-04-24T23:15:36.970238+0000","last_active":"2024-04-24T23:15:42.295395+0000","last_peered":"2024-04-24T23:15:42.295395+0000","last_clean":"2024-04-24T23:15:42.295395+0000","last_became_active":"2024-04-24T23:15:36.969751+0000","last_became_peered":"2024-04-24T23:15:36.969751+0000","last_unstale":"2024-04-24T23:15:42.295395+0000","last_undegraded":"2024-04-24T23:15:42.295395+0000","last_fullsized":"2024-04-24T23:15:42.295395+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:19:56.456801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.295592+0000","last_change":"2024-04-24T23:15:36.968825+0000","last_active":"2024-04-24T23:15:42.295592+0000","last_peered":"2024-04-24T23:15:42.295592+0000","last_clean":"2024-04-24T23:15:42.295592+0000","last_became_active":"2024-04-24T23:15:36.968543+0000","last_became_peered":"2024-04-24T23:15:36.968543+0000","last_unstale":"2024-04-24T23:15:42.295592+0000","last_undegraded":"2024-04-24T23:15:42.295592+0000","last_fullsized":"2024-04-24T23:15:42.295592+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:36:48.795062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.193226+0000","last_change":"2024-04-24T23:15:36.992800+0000","last_active":"2024-04-24T23:15:43.193226+0000","last_peered":"2024-04-24T23:15:43.193226+0000","last_clean":"2024-04-24T23:15:43.193226+0000","last_became_active":"2024-04-24T23:15:36.992476+0000","last_became_peered":"2024-04-24T23:15:36.992476+0000","last_unstale":"2024-04-24T23:15:43.193226+0000","last_undegraded":"2024-04-24T23:15:43.193226+0000","last_fullsized":"2024-04-24T23:15:43.193226+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:01:21.103559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4640,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":49,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574860,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":262144,"data_stored":100388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574860,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27272,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":253952,"data_stored":100264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574860,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27856,"kb_used_data":836,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343984,"statfs":{"total":96636764160,"available":96608239616,"internally_reserved":0,"allocated":856064,"data_stored":690724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574861,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":274432,"data_stored":104778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574861,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27272,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":258048,"data_stored":100846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574860,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":266240,"data_stored":104106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574859,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27852,"kb_used_data":832,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343988,"statfs":{"total":96636764160,"available":96608243712,"internally_reserved":0,"allocated":851968,"data_stored":690702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574860,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":262144,"data_stored":100320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T23:16:12.101 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-24T23:16:12.102 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-24T23:16:12.102 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-04-24T23:16:12.102 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-04-24T23:16:12.102 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-04-24T23:16:12.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:12.348+0000 7f764fe00640 1 Processor -- start 2024-04-24T23:16:12.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:12.348+0000 7f764fe00640 1 -- start start 2024-04-24T23:16:12.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:12.350+0000 7f764fe00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f76500c6be0 0x7f76501640a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:12.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:12.350+0000 7f764fe00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f765006d390 con 0x7f7650152bd0 2024-04-24T23:16:12.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:12.350+0000 7f764fe00640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f765006dbc0 con 0x7f765006ce30 2024-04-24T23:16:12.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:12.350+0000 7f764fe00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f765006d970 con 0x7f76500c6be0 2024-04-24T23:16:12.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:12.350+0000 7f764ea00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f7650152bd0 0x7f7650158e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:43416/0 (socket says 172.21.15.6:43416) 2024-04-24T23:16:12.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:12.350+0000 7f764ea00640 1 -- 172.21.15.6:0/3556935392 learned_addr learned my addr 172.21.15.6:0/3556935392 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:16:12.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:12.350+0000 7f764f400640 1 --2- 172.21.15.6:0/3556935392 >> v2:172.21.15.6:3300/0 conn(0x7f76500c6be0 0x7f76501640a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:12.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:12.351+0000 7f764d600640 1 -- 172.21.15.6:0/3556935392 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3916666863 0 0) 0x7f765006dbc0 con 0x7f765006ce30 2024-04-24T23:16:12.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:12.351+0000 7f764d600640 1 -- 172.21.15.6:0/3556935392 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7634003520 con 0x7f765006ce30 2024-04-24T23:16:12.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:12.351+0000 7f764d600640 1 -- 172.21.15.6:0/3556935392 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 719032122 0 0) 0x7f765006d390 con 0x7f7650152bd0 2024-04-24T23:16:12.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:12.351+0000 7f764d600640 1 -- 172.21.15.6:0/3556935392 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f765006dbc0 con 0x7f7650152bd0 2024-04-24T23:16:12.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:12.351+0000 7f764f400640 1 -- 172.21.15.6:0/3556935392 >> v1:172.21.15.190:6789/0 conn(0x7f765006ce30 legacy=0x7f76500cc600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:12.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:12.351+0000 7f764f400640 1 -- 172.21.15.6:0/3556935392 >> v1:172.21.15.6:6789/0 conn(0x7f7650152bd0 legacy=0x7f7650158e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:12.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:12.351+0000 7f764f400640 1 -- 172.21.15.6:0/3556935392 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f765006dea0 con 0x7f76500c6be0 2024-04-24T23:16:12.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:12.352+0000 7f764f400640 1 --2- 172.21.15.6:0/3556935392 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f76500c6be0 0x7f76501640a0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f7648009d90 tx=0x7f764802a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34300b982bd2dac4 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:12.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:12.352+0000 7f764d600640 1 -- 172.21.15.6:0/3556935392 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7648007740 con 0x7f76500c6be0 2024-04-24T23:16:12.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:12.352+0000 7f764d600640 1 -- 172.21.15.6:0/3556935392 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f76480078e0 con 0x7f76500c6be0 2024-04-24T23:16:12.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:12.352+0000 7f764d600640 1 -- 172.21.15.6:0/3556935392 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7648007cb0 con 0x7f76500c6be0 2024-04-24T23:16:12.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:12.353+0000 7f764fe00640 1 -- 172.21.15.6:0/3556935392 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f76500c6be0 msgr2=0x7f76501640a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:12.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:12.353+0000 7f764fe00640 1 --2- 172.21.15.6:0/3556935392 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f76500c6be0 0x7f76501640a0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f7648009d90 tx=0x7f764802a6f0 comp rx=0 tx=0).stop 2024-04-24T23:16:12.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:12.353+0000 7f764fe00640 1 -- 172.21.15.6:0/3556935392 shutdown_connections 2024-04-24T23:16:12.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:12.353+0000 7f764fe00640 1 --2- 172.21.15.6:0/3556935392 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f76500c6be0 0x7f76501640a0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:12.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:12.353+0000 7f764fe00640 1 -- 172.21.15.6:0/3556935392 >> 172.21.15.6:0/3556935392 conn(0x7f765006ed80 msgr2=0x7f765006f190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:12.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:12.353+0000 7f764fe00640 1 -- 172.21.15.6:0/3556935392 shutdown_connections 2024-04-24T23:16:12.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:12.353+0000 7f764fe00640 1 -- 172.21.15.6:0/3556935392 wait complete. 2024-04-24T23:16:12.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:12.355+0000 7f764fe00640 1 Processor -- start 2024-04-24T23:16:12.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:12.355+0000 7f764fe00640 1 -- start start 2024-04-24T23:16:12.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:12.356+0000 7f764fe00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f765006ce30 0x7f76501432e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:12.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:12.356+0000 7f764fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f76500c6be0 0x7f765015a490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:12.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:12.356+0000 7f764fe00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7650152bd0 0x7f76501f3c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:12.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:12.356+0000 7f764fe00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f7650072650 con 0x7f76500c6be0 2024-04-24T23:16:12.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:12.356+0000 7f764fe00640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f7650071890 con 0x7f7650152bd0 2024-04-24T23:16:12.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:12.356+0000 7f764fe00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f76500727d0 con 0x7f765006ce30 2024-04-24T23:16:12.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:12.356+0000 7f764f400640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7650152bd0 0x7f76501f3c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:12.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:12.357+0000 7f764ea00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f765006ce30 0x7f76501432e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:12.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:12.357+0000 7f764e000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f76500c6be0 0x7f765015a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:12.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:12.357+0000 7f764f400640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7650152bd0 0x7f76501f3c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:59728/0 (socket says 172.21.15.6:59728) 2024-04-24T23:16:12.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:12.357+0000 7f764f400640 1 -- 172.21.15.6:0/1882603408 learned_addr learned my addr 172.21.15.6:0/1882603408 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:16:12.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:12.357+0000 7f764ea00640 1 -- 172.21.15.6:0/1882603408 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7650152bd0 msgr2=0x7f76501f3c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:12.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:12.357+0000 7f764ea00640 1 --2- 172.21.15.6:0/1882603408 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7650152bd0 0x7f76501f3c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:12.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:12.357+0000 7f764ea00640 1 -- 172.21.15.6:0/1882603408 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f76500c6be0 msgr2=0x7f765015a490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:12.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:12.357+0000 7f764ea00640 1 --2- 172.21.15.6:0/1882603408 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f76500c6be0 0x7f765015a490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:12.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:12.357+0000 7f764ea00640 1 -- 172.21.15.6:0/1882603408 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7650159dc0 con 0x7f765006ce30 2024-04-24T23:16:12.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:12.357+0000 7f764e000640 1 --2- 172.21.15.6:0/1882603408 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f76500c6be0 0x7f765015a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:16:12.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:12.357+0000 7f764ea00640 1 --2- 172.21.15.6:0/1882603408 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f765006ce30 0x7f76501432e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f764000c3c0 tx=0x7f7640001e70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:12.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:12.358+0000 7f7646a00640 1 -- 172.21.15.6:0/1882603408 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7640017070 con 0x7f765006ce30 2024-04-24T23:16:12.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:12.358+0000 7f7646a00640 1 -- 172.21.15.6:0/1882603408 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7640005ce0 con 0x7f765006ce30 2024-04-24T23:16:12.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:12.358+0000 7f7646a00640 1 -- 172.21.15.6:0/1882603408 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76400072d0 con 0x7f765006ce30 2024-04-24T23:16:12.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:12.358+0000 7f764fe00640 1 -- 172.21.15.6:0/1882603408 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76501f4160 con 0x7f765006ce30 2024-04-24T23:16:12.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:12.358+0000 7f764fe00640 1 -- 172.21.15.6:0/1882603408 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f765015c5a0 con 0x7f765006ce30 2024-04-24T23:16:12.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:12.359+0000 7f764fe00640 1 -- 172.21.15.6:0/1882603408 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7650075b30 con 0x7f765006ce30 2024-04-24T23:16:12.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:12.360+0000 7f7646a00640 1 -- 172.21.15.6:0/1882603408 <== mon.2 v2:172.21.15.190:3301/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f764001a070 con 0x7f765006ce30 2024-04-24T23:16:12.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:12.361+0000 7f7646a00640 1 --2- 172.21.15.6:0/1882603408 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f763006be60 0x7f763006e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:12.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:12.361+0000 7f7646a00640 1 -- 172.21.15.6:0/1882603408 <== mon.2 v2:172.21.15.190:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f764008a480 con 0x7f765006ce30 2024-04-24T23:16:12.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:12.361+0000 7f764e000640 1 --2- 172.21.15.6:0/1882603408 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f763006be60 0x7f763006e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:12.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:12.362+0000 7f764e000640 1 --2- 172.21.15.6:0/1882603408 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f763006be60 0x7f763006e320 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f763c007fd0 tx=0x7f763c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:12.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:12.366+0000 7f7646a00640 1 -- 172.21.15.6:0/1882603408 <== mon.2 v2:172.21.15.190:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f7640058400 con 0x7f765006ce30 2024-04-24T23:16:12.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:12.625+0000 7f764fe00640 1 -- 172.21.15.6:0/1882603408 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7f7650146a30 con 0x7f765006ce30 2024-04-24T23:16:12.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:12.629+0000 7f7646a00640 1 -- 172.21.15.6:0/1882603408 <== mon.2 v2:172.21.15.190:3301/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) ==== 74+0+172943 (secure 0 0 0) 0x7f764005c700 con 0x7f765006ce30 2024-04-24T23:16:12.633 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:16:12.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:12.632+0000 7f764fe00640 1 -- 172.21.15.6:0/1882603408 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f763006be60 msgr2=0x7f763006e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:12.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:12.632+0000 7f764fe00640 1 --2- 172.21.15.6:0/1882603408 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f763006be60 0x7f763006e320 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f763c007fd0 tx=0x7f763c009040 comp rx=0 tx=0).stop 2024-04-24T23:16:12.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:12.632+0000 7f764fe00640 1 -- 172.21.15.6:0/1882603408 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f765006ce30 msgr2=0x7f76501432e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:12.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:12.632+0000 7f764fe00640 1 --2- 172.21.15.6:0/1882603408 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f765006ce30 0x7f76501432e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f764000c3c0 tx=0x7f7640001e70 comp rx=0 tx=0).stop 2024-04-24T23:16:12.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:12.632+0000 7f764fe00640 1 -- 172.21.15.6:0/1882603408 shutdown_connections 2024-04-24T23:16:12.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:12.632+0000 7f764fe00640 1 --2- 172.21.15.6:0/1882603408 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f763006be60 0x7f763006e320 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:12.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:12.632+0000 7f764fe00640 1 --2- 172.21.15.6:0/1882603408 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7650152bd0 0x7f76501f3c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:12.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:12.632+0000 7f764fe00640 1 --2- 172.21.15.6:0/1882603408 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f76500c6be0 0x7f765015a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:12.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:12.632+0000 7f764fe00640 1 --2- 172.21.15.6:0/1882603408 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f765006ce30 0x7f76501432e0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:12.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:12.632+0000 7f764fe00640 1 -- 172.21.15.6:0/1882603408 >> 172.21.15.6:0/1882603408 conn(0x7f765006ed80 msgr2=0x7f7650071c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:12.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:12.632+0000 7f764fe00640 1 -- 172.21.15.6:0/1882603408 shutdown_connections 2024-04-24T23:16:12.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:12.633+0000 7f764fe00640 1 -- 172.21.15.6:0/1882603408 wait complete. 2024-04-24T23:16:12.653 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":5,"flags":0,"active_gid":4105,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":806769663},{"type":"v1","addr":"172.21.15.190:6833","nonce":806769663}]},"active_addr":"172.21.15.190:6833/806769663","active_change":"2024-04-24T23:15:16.665445+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":4103,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.190:0","nonce":1880087557}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.190:0","nonce":2134673868}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.190:0","nonce":3539675466}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.190:0","nonce":3941567574}]}]} 2024-04-24T23:16:12.657 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-04-24T23:16:12.657 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-04-24T23:16:12.657 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T23:16:12.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:12.869+0000 7f8771a00640 1 Processor -- start 2024-04-24T23:16:12.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:12.869+0000 7f8771a00640 1 -- start start 2024-04-24T23:16:12.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:12.871+0000 7f8771a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f876c152bc0 0x7f876c159390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:12.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:12.871+0000 7f8771a00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f876c0c9360 0x7f876c15c610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:12.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:12.871+0000 7f8771a00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f876c0c6e70 con 0x7f876c0c8f50 2024-04-24T23:16:12.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:12.871+0000 7f8771a00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f876c0c73d0 con 0x7f876c152bc0 2024-04-24T23:16:12.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:12.871+0000 7f8771a00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f876c0c8ac0 con 0x7f876c0c9360 2024-04-24T23:16:12.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:12.871+0000 7f876aa00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f876c152bc0 0x7f876c159390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:12.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:12.871+0000 7f876b400640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f876c0c9360 0x7f876c15c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:12.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:12.871+0000 7f876a000640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f876c0c8f50 0x7f876c15bf00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:43426/0 (socket says 172.21.15.6:43426) 2024-04-24T23:16:12.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:12.871+0000 7f876a000640 1 -- 172.21.15.6:0/2453383943 learned_addr learned my addr 172.21.15.6:0/2453383943 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:16:12.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:12.872+0000 7f876b400640 1 -- 172.21.15.6:0/2453383943 >> v2:172.21.15.6:3300/0 conn(0x7f876c152bc0 msgr2=0x7f876c159390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:16:12.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:12.872+0000 7f876b400640 1 --2- 172.21.15.6:0/2453383943 >> v2:172.21.15.6:3300/0 conn(0x7f876c152bc0 0x7f876c159390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:12.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:12.872+0000 7f876b400640 1 -- 172.21.15.6:0/2453383943 >> v1:172.21.15.6:6789/0 conn(0x7f876c0c8f50 legacy=0x7f876c15bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:12.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:12.872+0000 7f876b400640 1 -- 172.21.15.6:0/2453383943 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f876c1598d0 con 0x7f876c0c9360 2024-04-24T23:16:12.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:12.873+0000 7f876b400640 1 --2- 172.21.15.6:0/2453383943 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f876c0c9360 0x7f876c15c610 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f8764009d90 tx=0x7f876402a5d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=88712b15d9c4cc36 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:12.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:12.873+0000 7f8769600640 1 -- 172.21.15.6:0/2453383943 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8764031600 con 0x7f876c0c9360 2024-04-24T23:16:12.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:12.873+0000 7f8769600640 1 -- 172.21.15.6:0/2453383943 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f87640317a0 con 0x7f876c0c9360 2024-04-24T23:16:12.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:12.873+0000 7f8769600640 1 -- 172.21.15.6:0/2453383943 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8764031b70 con 0x7f876c0c9360 2024-04-24T23:16:12.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:12.874+0000 7f8771a00640 1 -- 172.21.15.6:0/2453383943 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f876c0c9360 msgr2=0x7f876c15c610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:12.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:12.874+0000 7f8771a00640 1 --2- 172.21.15.6:0/2453383943 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f876c0c9360 0x7f876c15c610 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f8764009d90 tx=0x7f876402a5d0 comp rx=0 tx=0).stop 2024-04-24T23:16:12.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:12.874+0000 7f8771a00640 1 -- 172.21.15.6:0/2453383943 shutdown_connections 2024-04-24T23:16:12.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:12.874+0000 7f8771a00640 1 --2- 172.21.15.6:0/2453383943 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f876c0c9360 0x7f876c15c610 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:12.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:12.874+0000 7f8771a00640 1 --2- 172.21.15.6:0/2453383943 >> v2:172.21.15.6:3300/0 conn(0x7f876c152bc0 0x7f876c159390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:12.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:12.874+0000 7f8771a00640 1 -- 172.21.15.6:0/2453383943 >> 172.21.15.6:0/2453383943 conn(0x7f876c06b0e0 msgr2=0x7f876c06b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:12.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:12.874+0000 7f8771a00640 1 -- 172.21.15.6:0/2453383943 shutdown_connections 2024-04-24T23:16:12.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:12.874+0000 7f8771a00640 1 -- 172.21.15.6:0/2453383943 wait complete. 2024-04-24T23:16:12.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:12.876+0000 7f8771a00640 1 Processor -- start 2024-04-24T23:16:12.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:12.876+0000 7f8771a00640 1 -- start start 2024-04-24T23:16:12.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:12.877+0000 7f8771a00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f876c0c8f50 0x7f876c141a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:12.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:12.877+0000 7f8771a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f876c0c9360 0x7f876c0ca3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:12.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:12.877+0000 7f8771a00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f876c152bc0 0x7f876c0ca910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:12.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:12.877+0000 7f8771a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f876c0c8470 con 0x7f876c0c9360 2024-04-24T23:16:12.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:12.877+0000 7f8771a00640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f876c0c6700 con 0x7f876c152bc0 2024-04-24T23:16:12.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:12.877+0000 7f8771a00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f876c0c8750 con 0x7f876c0c8f50 2024-04-24T23:16:12.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:12.877+0000 7f876a000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f876c0c9360 0x7f876c0ca3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:12.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:12.877+0000 7f876aa00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f876c0c8f50 0x7f876c141a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:12.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:12.877+0000 7f876b400640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f876c152bc0 0x7f876c0ca910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:12.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:12.877+0000 7f876a000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f876c0c9360 0x7f876c0ca3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:59948/0 (socket says 172.21.15.6:59948) 2024-04-24T23:16:12.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:12.877+0000 7f876a000640 1 -- 172.21.15.6:0/970276220 learned_addr learned my addr 172.21.15.6:0/970276220 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:16:12.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:12.878+0000 7f876aa00640 1 -- 172.21.15.6:0/970276220 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f876c152bc0 msgr2=0x7f876c0ca910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:12.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:12.878+0000 7f876aa00640 1 --2- 172.21.15.6:0/970276220 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f876c152bc0 0x7f876c0ca910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:12.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:12.878+0000 7f876aa00640 1 -- 172.21.15.6:0/970276220 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f876c0c9360 msgr2=0x7f876c0ca3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:12.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:12.878+0000 7f876aa00640 1 --2- 172.21.15.6:0/970276220 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f876c0c9360 0x7f876c0ca3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:12.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:12.878+0000 7f876aa00640 1 -- 172.21.15.6:0/970276220 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f876c140670 con 0x7f876c0c8f50 2024-04-24T23:16:12.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:12.878+0000 7f876a000640 1 --2- 172.21.15.6:0/970276220 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f876c0c9360 0x7f876c0ca3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:16:12.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:12.878+0000 7f876aa00640 1 --2- 172.21.15.6:0/970276220 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f876c0c8f50 0x7f876c141a10 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f875c00e870 tx=0x7f875c00ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:12.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:12.878+0000 7f8762a00640 1 -- 172.21.15.6:0/970276220 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f875c017070 con 0x7f876c0c8f50 2024-04-24T23:16:12.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:12.878+0000 7f8762a00640 1 -- 172.21.15.6:0/970276220 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f875c00ee60 con 0x7f876c0c8f50 2024-04-24T23:16:12.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:12.878+0000 7f8762a00640 1 -- 172.21.15.6:0/970276220 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f875c00cb60 con 0x7f876c0c8f50 2024-04-24T23:16:12.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:12.878+0000 7f8771a00640 1 -- 172.21.15.6:0/970276220 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f876c140950 con 0x7f876c0c8f50 2024-04-24T23:16:12.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:12.879+0000 7f8771a00640 1 -- 172.21.15.6:0/970276220 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f876c140f10 con 0x7f876c0c8f50 2024-04-24T23:16:12.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:12.880+0000 7f8771a00640 1 -- 172.21.15.6:0/970276220 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f876c0c56a0 con 0x7f876c0c8f50 2024-04-24T23:16:12.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:12.881+0000 7f8762a00640 1 -- 172.21.15.6:0/970276220 <== mon.2 v2:172.21.15.190:3301/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f875c01a070 con 0x7f876c0c8f50 2024-04-24T23:16:12.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:12.882+0000 7f8762a00640 1 --2- 172.21.15.6:0/970276220 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f874c06be60 0x7f874c06e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:12.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:12.882+0000 7f8762a00640 1 -- 172.21.15.6:0/970276220 <== mon.2 v2:172.21.15.190:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f875c08b3b0 con 0x7f876c0c8f50 2024-04-24T23:16:12.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:12.882+0000 7f876a000640 1 --2- 172.21.15.6:0/970276220 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f874c06be60 0x7f874c06e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:12.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:12.883+0000 7f876a000640 1 --2- 172.21.15.6:0/970276220 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f874c06be60 0x7f874c06e320 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f876c1e6030 tx=0x7f8758007f10 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:12.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:12.886+0000 7f8762a00640 1 -- 172.21.15.6:0/970276220 <== mon.2 v2:172.21.15.190:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f876c0c56a0 con 0x7f876c0c8f50 2024-04-24T23:16:13.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.096+0000 7f8771a00640 1 -- 172.21.15.6:0/970276220 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f876c0767b0 con 0x7f876c0c8f50 2024-04-24T23:16:13.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.099+0000 7f8762a00640 1 -- 172.21.15.6:0/970276220 <== mon.2 v2:172.21.15.190:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+18458 (secure 0 0 0) 0x7f875c0592a0 con 0x7f876c0c8f50 2024-04-24T23:16:13.101 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:16:13.101 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":24,"fsid":"18bef68d-5e6d-4ec5-862b-b6f206e2588d","created":"2024-04-24T23:15:12.535072+0000","modified":"2024-04-24T23:15:41.465875+0000","last_up_change":"2024-04-24T23:15:23.773223+0000","last_in_change":"2024-04-24T23:15:14.905328+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T23:15:24.747441+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-24T23:15:34.051800+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-24T23:15:35.436870+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-24T23:15:39.034257+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"3b7a212d-d498-462e-90bf-ed493c803a09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":3022569437},{"type":"v1","addr":"172.21.15.6:6809","nonce":3022569437}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":3022569437},{"type":"v1","addr":"172.21.15.6:6811","nonce":3022569437}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":3022569437},{"type":"v1","addr":"172.21.15.6:6815","nonce":3022569437}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":3022569437},{"type":"v1","addr":"172.21.15.6:6813","nonce":3022569437}]},"public_addr":"172.21.15.6:6809/3022569437","cluster_addr":"172.21.15.6:6811/3022569437","heartbeat_back_addr":"172.21.15.6:6815/3022569437","heartbeat_front_addr":"172.21.15.6:6813/3022569437","state":["exists","up"]},{"osd":1,"uuid":"7c97f5a2-1735-4775-9f15-eb50099b19d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":3687836888},{"type":"v1","addr":"172.21.15.6:6817","nonce":3687836888}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":3687836888},{"type":"v1","addr":"172.21.15.6:6819","nonce":3687836888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":3687836888},{"type":"v1","addr":"172.21.15.6:6823","nonce":3687836888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":3687836888},{"type":"v1","addr":"172.21.15.6:6821","nonce":3687836888}]},"public_addr":"172.21.15.6:6817/3687836888","cluster_addr":"172.21.15.6:6819/3687836888","heartbeat_back_addr":"172.21.15.6:6823/3687836888","heartbeat_front_addr":"172.21.15.6:6821/3687836888","state":["exists","up"]},{"osd":2,"uuid":"d86bbc18-438c-4ec2-8018-ba7dd78f790b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":1020213724},{"type":"v1","addr":"172.21.15.6:6801","nonce":1020213724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":1020213724},{"type":"v1","addr":"172.21.15.6:6803","nonce":1020213724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":1020213724},{"type":"v1","addr":"172.21.15.6:6807","nonce":1020213724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":1020213724},{"type":"v1","addr":"172.21.15.6:6805","nonce":1020213724}]},"public_addr":"172.21.15.6:6801/1020213724","cluster_addr":"172.21.15.6:6803/1020213724","heartbeat_back_addr":"172.21.15.6:6807/1020213724","heartbeat_front_addr":"172.21.15.6:6805/1020213724","state":["exists","up"]},{"osd":3,"uuid":"3754ae16-4248-40dd-95f9-0f59c0eb5426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6824","nonce":3163883468},{"type":"v1","addr":"172.21.15.6:6825","nonce":3163883468}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":3163883468},{"type":"v1","addr":"172.21.15.6:6827","nonce":3163883468}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":3163883468},{"type":"v1","addr":"172.21.15.6:6831","nonce":3163883468}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":3163883468},{"type":"v1","addr":"172.21.15.6:6829","nonce":3163883468}]},"public_addr":"172.21.15.6:6825/3163883468","cluster_addr":"172.21.15.6:6827/3163883468","heartbeat_back_addr":"172.21.15.6:6831/3163883468","heartbeat_front_addr":"172.21.15.6:6829/3163883468","state":["exists","up"]},{"osd":4,"uuid":"8b2913a0-d006-479c-84ed-5b21e0e6c392","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":1989500934},{"type":"v1","addr":"172.21.15.190:6809","nonce":1989500934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":1989500934},{"type":"v1","addr":"172.21.15.190:6811","nonce":1989500934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":1989500934},{"type":"v1","addr":"172.21.15.190:6815","nonce":1989500934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":1989500934},{"type":"v1","addr":"172.21.15.190:6813","nonce":1989500934}]},"public_addr":"172.21.15.190:6809/1989500934","cluster_addr":"172.21.15.190:6811/1989500934","heartbeat_back_addr":"172.21.15.190:6815/1989500934","heartbeat_front_addr":"172.21.15.190:6813/1989500934","state":["exists","up"]},{"osd":5,"uuid":"499b79d0-0585-4dae-ab69-196a551793da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":3687548412},{"type":"v1","addr":"172.21.15.190:6801","nonce":3687548412}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":3687548412},{"type":"v1","addr":"172.21.15.190:6803","nonce":3687548412}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":3687548412},{"type":"v1","addr":"172.21.15.190:6807","nonce":3687548412}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":3687548412},{"type":"v1","addr":"172.21.15.190:6805","nonce":3687548412}]},"public_addr":"172.21.15.190:6801/3687548412","cluster_addr":"172.21.15.190:6803/3687548412","heartbeat_back_addr":"172.21.15.190:6807/3687548412","heartbeat_front_addr":"172.21.15.190:6805/3687548412","state":["exists","up"]},{"osd":6,"uuid":"6ba6df3b-4a6b-4a89-a890-0f79f387d4d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":870571564},{"type":"v1","addr":"172.21.15.190:6817","nonce":870571564}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":870571564},{"type":"v1","addr":"172.21.15.190:6819","nonce":870571564}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":870571564},{"type":"v1","addr":"172.21.15.190:6823","nonce":870571564}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":870571564},{"type":"v1","addr":"172.21.15.190:6821","nonce":870571564}]},"public_addr":"172.21.15.190:6817/870571564","cluster_addr":"172.21.15.190:6819/870571564","heartbeat_back_addr":"172.21.15.190:6823/870571564","heartbeat_front_addr":"172.21.15.190:6821/870571564","state":["exists","up"]},{"osd":7,"uuid":"e9668b1c-a000-4332-86e8-4efd73b02409","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":123639197},{"type":"v1","addr":"172.21.15.190:6825","nonce":123639197}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":123639197},{"type":"v1","addr":"172.21.15.190:6827","nonce":123639197}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":123639197},{"type":"v1","addr":"172.21.15.190:6831","nonce":123639197}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":123639197},{"type":"v1","addr":"172.21.15.190:6829","nonce":123639197}]},"public_addr":"172.21.15.190:6825/123639197","cluster_addr":"172.21.15.190:6827/123639197","heartbeat_back_addr":"172.21.15.190:6831/123639197","heartbeat_front_addr":"172.21.15.190:6829/123639197","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.321646+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.363873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.466963+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.531989+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:19.990707+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.043543+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.032492+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.115004+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T23:16:13.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.101+0000 7f8771a00640 1 -- 172.21.15.6:0/970276220 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f874c06be60 msgr2=0x7f874c06e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:13.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.101+0000 7f8771a00640 1 --2- 172.21.15.6:0/970276220 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f874c06be60 0x7f874c06e320 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f876c1e6030 tx=0x7f8758007f10 comp rx=0 tx=0).stop 2024-04-24T23:16:13.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.101+0000 7f8771a00640 1 -- 172.21.15.6:0/970276220 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f876c0c8f50 msgr2=0x7f876c141a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:13.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.101+0000 7f8771a00640 1 --2- 172.21.15.6:0/970276220 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f876c0c8f50 0x7f876c141a10 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f875c00e870 tx=0x7f875c00ed30 comp rx=0 tx=0).stop 2024-04-24T23:16:13.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.102+0000 7f8771a00640 1 -- 172.21.15.6:0/970276220 shutdown_connections 2024-04-24T23:16:13.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.102+0000 7f8771a00640 1 --2- 172.21.15.6:0/970276220 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f874c06be60 0x7f874c06e320 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:13.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.102+0000 7f8771a00640 1 --2- 172.21.15.6:0/970276220 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f876c152bc0 0x7f876c0ca910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:13.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.102+0000 7f8771a00640 1 --2- 172.21.15.6:0/970276220 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f876c0c9360 0x7f876c0ca3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:13.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.102+0000 7f8771a00640 1 --2- 172.21.15.6:0/970276220 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f876c0c8f50 0x7f876c141a10 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:13.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.102+0000 7f8771a00640 1 -- 172.21.15.6:0/970276220 >> 172.21.15.6:0/970276220 conn(0x7f876c06b0e0 msgr2=0x7f876c0c36c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:13.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.102+0000 7f8771a00640 1 -- 172.21.15.6:0/970276220 shutdown_connections 2024-04-24T23:16:13.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.102+0000 7f8771a00640 1 -- 172.21.15.6:0/970276220 wait complete. 2024-04-24T23:16:13.122 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-04-24T23:16:13.123 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T23:16:13.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.318+0000 7fa1ad000640 1 Processor -- start 2024-04-24T23:16:13.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.318+0000 7fa1ad000640 1 -- start start 2024-04-24T23:16:13.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.320+0000 7fa1ad000640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fa1a806ce30 0x7fa1a80cc600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:13.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.320+0000 7fa1ad000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa1a80c6be0 0x7fa1a80ccb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:13.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.320+0000 7fa1ad000640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa1a806d390 con 0x7fa1a8152bd0 2024-04-24T23:16:13.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.320+0000 7fa1ad000640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7fa1a806d970 con 0x7fa1a80c6be0 2024-04-24T23:16:13.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.320+0000 7fa1ad000640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7fa1a806dbc0 con 0x7fa1a806ce30 2024-04-24T23:16:13.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.320+0000 7fa1a7400640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7fa1a8152bd0 0x7fa1a8158e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.6:50584/0 (socket says 172.21.15.6:50584) 2024-04-24T23:16:13.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.320+0000 7fa1a7400640 1 -- 172.21.15.6:0/2766835763 learned_addr learned my addr 172.21.15.6:0/2766835763 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:16:13.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.320+0000 7fa1a7e00640 1 --2- 172.21.15.6:0/2766835763 >> v2:172.21.15.6:3300/0 conn(0x7fa1a80c6be0 0x7fa1a80ccb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:13.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.320+0000 7fa1a6a00640 1 --2- 172.21.15.6:0/2766835763 >> v2:172.21.15.190:3300/0 conn(0x7fa1a806ce30 0x7fa1a80cc600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:13.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.321+0000 7fa1a6000640 1 -- 172.21.15.6:0/2766835763 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1972407332 0 0) 0x7fa1a806d390 con 0x7fa1a8152bd0 2024-04-24T23:16:13.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.321+0000 7fa1a6000640 1 -- 172.21.15.6:0/2766835763 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa18c003520 con 0x7fa1a8152bd0 2024-04-24T23:16:13.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.321+0000 7fa1a7e00640 1 -- 172.21.15.6:0/2766835763 >> v2:172.21.15.190:3300/0 conn(0x7fa1a806ce30 msgr2=0x7fa1a80cc600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:16:13.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.321+0000 7fa1a7e00640 1 --2- 172.21.15.6:0/2766835763 >> v2:172.21.15.190:3300/0 conn(0x7fa1a806ce30 0x7fa1a80cc600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:13.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.321+0000 7fa1a7e00640 1 -- 172.21.15.6:0/2766835763 >> v1:172.21.15.190:6789/0 conn(0x7fa1a8152bd0 legacy=0x7fa1a8158e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:13.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.321+0000 7fa1a7e00640 1 -- 172.21.15.6:0/2766835763 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1a806dea0 con 0x7fa1a80c6be0 2024-04-24T23:16:13.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.322+0000 7fa1a7e00640 1 --2- 172.21.15.6:0/2766835763 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa1a80c6be0 0x7fa1a80ccb40 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fa1a8070820 tx=0x7fa1a002a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de8804d86dcfa4d5 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:13.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.322+0000 7fa1a6000640 1 -- 172.21.15.6:0/2766835763 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1a0007740 con 0x7fa1a80c6be0 2024-04-24T23:16:13.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.322+0000 7fa1a6000640 1 -- 172.21.15.6:0/2766835763 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa1a00078e0 con 0x7fa1a80c6be0 2024-04-24T23:16:13.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.322+0000 7fa1a6000640 1 -- 172.21.15.6:0/2766835763 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1a0007c90 con 0x7fa1a80c6be0 2024-04-24T23:16:13.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.323+0000 7fa1ad000640 1 -- 172.21.15.6:0/2766835763 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa1a80c6be0 msgr2=0x7fa1a80ccb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:13.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.323+0000 7fa1ad000640 1 --2- 172.21.15.6:0/2766835763 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa1a80c6be0 0x7fa1a80ccb40 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fa1a8070820 tx=0x7fa1a002a680 comp rx=0 tx=0).stop 2024-04-24T23:16:13.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.323+0000 7fa1ad000640 1 -- 172.21.15.6:0/2766835763 shutdown_connections 2024-04-24T23:16:13.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.323+0000 7fa1ad000640 1 --2- 172.21.15.6:0/2766835763 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa1a80c6be0 0x7fa1a80ccb40 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:13.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.323+0000 7fa1ad000640 1 --2- 172.21.15.6:0/2766835763 >> v2:172.21.15.190:3300/0 conn(0x7fa1a806ce30 0x7fa1a80cc600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:13.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.323+0000 7fa1ad000640 1 -- 172.21.15.6:0/2766835763 >> 172.21.15.6:0/2766835763 conn(0x7fa1a806ed80 msgr2=0x7fa1a806f190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:13.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.323+0000 7fa1ad000640 1 -- 172.21.15.6:0/2766835763 shutdown_connections 2024-04-24T23:16:13.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.323+0000 7fa1ad000640 1 -- 172.21.15.6:0/2766835763 wait complete. 2024-04-24T23:16:13.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.325+0000 7fa1ad000640 1 Processor -- start 2024-04-24T23:16:13.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.325+0000 7fa1ad000640 1 -- start start 2024-04-24T23:16:13.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.326+0000 7fa1ad000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa1a806ce30 0x7fa1a8157ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:13.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.326+0000 7fa1ad000640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa1a80c6be0 0x7fa1a80c39c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:13.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.326+0000 7fa1ad000640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa1a8152bd0 0x7fa1a80c3f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:13.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.326+0000 7fa1ad000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fa1a8072650 con 0x7fa1a806ce30 2024-04-24T23:16:13.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.326+0000 7fa1ad000640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fa1a8071890 con 0x7fa1a80c6be0 2024-04-24T23:16:13.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.326+0000 7fa1ad000640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fa1a80727d0 con 0x7fa1a8152bd0 2024-04-24T23:16:13.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.326+0000 7fa1a7400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa1a806ce30 0x7fa1a8157ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:13.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.326+0000 7fa1a7400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa1a806ce30 0x7fa1a8157ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:59974/0 (socket says 172.21.15.6:59974) 2024-04-24T23:16:13.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.326+0000 7fa1a7400640 1 -- 172.21.15.6:0/204207939 learned_addr learned my addr 172.21.15.6:0/204207939 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:16:13.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.326+0000 7fa1a7e00640 1 --2- 172.21.15.6:0/204207939 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa1a8152bd0 0x7fa1a80c3f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:13.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.326+0000 7fa1a6a00640 1 --2- 172.21.15.6:0/204207939 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa1a80c6be0 0x7fa1a80c39c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:13.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.326+0000 7fa1a7400640 1 -- 172.21.15.6:0/204207939 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa1a8152bd0 msgr2=0x7fa1a80c3f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:13.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.326+0000 7fa1a7400640 1 --2- 172.21.15.6:0/204207939 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa1a8152bd0 0x7fa1a80c3f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:13.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.326+0000 7fa1a7400640 1 -- 172.21.15.6:0/204207939 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa1a80c6be0 msgr2=0x7fa1a80c39c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:13.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.326+0000 7fa1a7400640 1 --2- 172.21.15.6:0/204207939 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa1a80c6be0 0x7fa1a80c39c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:13.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.326+0000 7fa1a7400640 1 -- 172.21.15.6:0/204207939 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1a81553a0 con 0x7fa1a806ce30 2024-04-24T23:16:13.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.327+0000 7fa1a7400640 1 --2- 172.21.15.6:0/204207939 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa1a806ce30 0x7fa1a8157ca0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fa19800c530 tx=0x7fa19800c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:13.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.327+0000 7fa19f400640 1 -- 172.21.15.6:0/204207939 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa198017070 con 0x7fa1a806ce30 2024-04-24T23:16:13.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.327+0000 7fa19f400640 1 -- 172.21.15.6:0/204207939 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa198005ce0 con 0x7fa1a806ce30 2024-04-24T23:16:13.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.327+0000 7fa1ad000640 1 -- 172.21.15.6:0/204207939 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1a81544b0 con 0x7fa1a806ce30 2024-04-24T23:16:13.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.327+0000 7fa19f400640 1 -- 172.21.15.6:0/204207939 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1980072b0 con 0x7fa1a806ce30 2024-04-24T23:16:13.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.327+0000 7fa1ad000640 1 -- 172.21.15.6:0/204207939 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1a8154a70 con 0x7fa1a806ce30 2024-04-24T23:16:13.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.328+0000 7fa1ad000640 1 -- 172.21.15.6:0/204207939 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa1a8075b30 con 0x7fa1a806ce30 2024-04-24T23:16:13.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.329+0000 7fa19f400640 1 -- 172.21.15.6:0/204207939 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fa19801a070 con 0x7fa1a806ce30 2024-04-24T23:16:13.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.330+0000 7fa19f400640 1 --2- 172.21.15.6:0/204207939 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fa18806be30 0x7fa18806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:13.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.330+0000 7fa19f400640 1 -- 172.21.15.6:0/204207939 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fa19808a520 con 0x7fa1a806ce30 2024-04-24T23:16:13.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.331+0000 7fa1a6a00640 1 --2- 172.21.15.6:0/204207939 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fa18806be30 0x7fa18806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:13.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.331+0000 7fa1a6a00640 1 --2- 172.21.15.6:0/204207939 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fa18806be30 0x7fa18806e2f0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fa1a80c5170 tx=0x7fa194009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:13.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.335+0000 7fa19f400640 1 -- 172.21.15.6:0/204207939 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fa1980585f0 con 0x7fa1a806ce30 2024-04-24T23:16:13.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.538+0000 7fa1ad000640 1 -- 172.21.15.6:0/204207939 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa1a806d750 con 0x7fa1a806ce30 2024-04-24T23:16:13.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.541+0000 7fa19f400640 1 -- 172.21.15.6:0/204207939 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+18458 (secure 0 0 0) 0x7fa19805c8f0 con 0x7fa1a806ce30 2024-04-24T23:16:13.543 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:16:13.543 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":24,"fsid":"18bef68d-5e6d-4ec5-862b-b6f206e2588d","created":"2024-04-24T23:15:12.535072+0000","modified":"2024-04-24T23:15:41.465875+0000","last_up_change":"2024-04-24T23:15:23.773223+0000","last_in_change":"2024-04-24T23:15:14.905328+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T23:15:24.747441+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-24T23:15:34.051800+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-24T23:15:35.436870+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-24T23:15:39.034257+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"3b7a212d-d498-462e-90bf-ed493c803a09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":3022569437},{"type":"v1","addr":"172.21.15.6:6809","nonce":3022569437}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":3022569437},{"type":"v1","addr":"172.21.15.6:6811","nonce":3022569437}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":3022569437},{"type":"v1","addr":"172.21.15.6:6815","nonce":3022569437}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":3022569437},{"type":"v1","addr":"172.21.15.6:6813","nonce":3022569437}]},"public_addr":"172.21.15.6:6809/3022569437","cluster_addr":"172.21.15.6:6811/3022569437","heartbeat_back_addr":"172.21.15.6:6815/3022569437","heartbeat_front_addr":"172.21.15.6:6813/3022569437","state":["exists","up"]},{"osd":1,"uuid":"7c97f5a2-1735-4775-9f15-eb50099b19d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":3687836888},{"type":"v1","addr":"172.21.15.6:6817","nonce":3687836888}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":3687836888},{"type":"v1","addr":"172.21.15.6:6819","nonce":3687836888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":3687836888},{"type":"v1","addr":"172.21.15.6:6823","nonce":3687836888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":3687836888},{"type":"v1","addr":"172.21.15.6:6821","nonce":3687836888}]},"public_addr":"172.21.15.6:6817/3687836888","cluster_addr":"172.21.15.6:6819/3687836888","heartbeat_back_addr":"172.21.15.6:6823/3687836888","heartbeat_front_addr":"172.21.15.6:6821/3687836888","state":["exists","up"]},{"osd":2,"uuid":"d86bbc18-438c-4ec2-8018-ba7dd78f790b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":1020213724},{"type":"v1","addr":"172.21.15.6:6801","nonce":1020213724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":1020213724},{"type":"v1","addr":"172.21.15.6:6803","nonce":1020213724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":1020213724},{"type":"v1","addr":"172.21.15.6:6807","nonce":1020213724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":1020213724},{"type":"v1","addr":"172.21.15.6:6805","nonce":1020213724}]},"public_addr":"172.21.15.6:6801/1020213724","cluster_addr":"172.21.15.6:6803/1020213724","heartbeat_back_addr":"172.21.15.6:6807/1020213724","heartbeat_front_addr":"172.21.15.6:6805/1020213724","state":["exists","up"]},{"osd":3,"uuid":"3754ae16-4248-40dd-95f9-0f59c0eb5426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6824","nonce":3163883468},{"type":"v1","addr":"172.21.15.6:6825","nonce":3163883468}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":3163883468},{"type":"v1","addr":"172.21.15.6:6827","nonce":3163883468}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":3163883468},{"type":"v1","addr":"172.21.15.6:6831","nonce":3163883468}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":3163883468},{"type":"v1","addr":"172.21.15.6:6829","nonce":3163883468}]},"public_addr":"172.21.15.6:6825/3163883468","cluster_addr":"172.21.15.6:6827/3163883468","heartbeat_back_addr":"172.21.15.6:6831/3163883468","heartbeat_front_addr":"172.21.15.6:6829/3163883468","state":["exists","up"]},{"osd":4,"uuid":"8b2913a0-d006-479c-84ed-5b21e0e6c392","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":1989500934},{"type":"v1","addr":"172.21.15.190:6809","nonce":1989500934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":1989500934},{"type":"v1","addr":"172.21.15.190:6811","nonce":1989500934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":1989500934},{"type":"v1","addr":"172.21.15.190:6815","nonce":1989500934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":1989500934},{"type":"v1","addr":"172.21.15.190:6813","nonce":1989500934}]},"public_addr":"172.21.15.190:6809/1989500934","cluster_addr":"172.21.15.190:6811/1989500934","heartbeat_back_addr":"172.21.15.190:6815/1989500934","heartbeat_front_addr":"172.21.15.190:6813/1989500934","state":["exists","up"]},{"osd":5,"uuid":"499b79d0-0585-4dae-ab69-196a551793da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":3687548412},{"type":"v1","addr":"172.21.15.190:6801","nonce":3687548412}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":3687548412},{"type":"v1","addr":"172.21.15.190:6803","nonce":3687548412}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":3687548412},{"type":"v1","addr":"172.21.15.190:6807","nonce":3687548412}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":3687548412},{"type":"v1","addr":"172.21.15.190:6805","nonce":3687548412}]},"public_addr":"172.21.15.190:6801/3687548412","cluster_addr":"172.21.15.190:6803/3687548412","heartbeat_back_addr":"172.21.15.190:6807/3687548412","heartbeat_front_addr":"172.21.15.190:6805/3687548412","state":["exists","up"]},{"osd":6,"uuid":"6ba6df3b-4a6b-4a89-a890-0f79f387d4d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":870571564},{"type":"v1","addr":"172.21.15.190:6817","nonce":870571564}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":870571564},{"type":"v1","addr":"172.21.15.190:6819","nonce":870571564}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":870571564},{"type":"v1","addr":"172.21.15.190:6823","nonce":870571564}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":870571564},{"type":"v1","addr":"172.21.15.190:6821","nonce":870571564}]},"public_addr":"172.21.15.190:6817/870571564","cluster_addr":"172.21.15.190:6819/870571564","heartbeat_back_addr":"172.21.15.190:6823/870571564","heartbeat_front_addr":"172.21.15.190:6821/870571564","state":["exists","up"]},{"osd":7,"uuid":"e9668b1c-a000-4332-86e8-4efd73b02409","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":123639197},{"type":"v1","addr":"172.21.15.190:6825","nonce":123639197}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":123639197},{"type":"v1","addr":"172.21.15.190:6827","nonce":123639197}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":123639197},{"type":"v1","addr":"172.21.15.190:6831","nonce":123639197}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":123639197},{"type":"v1","addr":"172.21.15.190:6829","nonce":123639197}]},"public_addr":"172.21.15.190:6825/123639197","cluster_addr":"172.21.15.190:6827/123639197","heartbeat_back_addr":"172.21.15.190:6831/123639197","heartbeat_front_addr":"172.21.15.190:6829/123639197","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.321646+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.363873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.466963+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.531989+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:19.990707+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.043543+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.032492+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.115004+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T23:16:13.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.543+0000 7fa1ad000640 1 -- 172.21.15.6:0/204207939 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fa18806be30 msgr2=0x7fa18806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:13.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.543+0000 7fa1ad000640 1 --2- 172.21.15.6:0/204207939 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fa18806be30 0x7fa18806e2f0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fa1a80c5170 tx=0x7fa194009190 comp rx=0 tx=0).stop 2024-04-24T23:16:13.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.543+0000 7fa1ad000640 1 -- 172.21.15.6:0/204207939 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa1a806ce30 msgr2=0x7fa1a8157ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:13.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.543+0000 7fa1ad000640 1 --2- 172.21.15.6:0/204207939 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa1a806ce30 0x7fa1a8157ca0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fa19800c530 tx=0x7fa19800c9f0 comp rx=0 tx=0).stop 2024-04-24T23:16:13.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.544+0000 7fa1ad000640 1 -- 172.21.15.6:0/204207939 shutdown_connections 2024-04-24T23:16:13.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.544+0000 7fa1ad000640 1 --2- 172.21.15.6:0/204207939 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fa18806be30 0x7fa18806e2f0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:13.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.544+0000 7fa1ad000640 1 --2- 172.21.15.6:0/204207939 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa1a8152bd0 0x7fa1a80c3f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:13.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.544+0000 7fa1ad000640 1 --2- 172.21.15.6:0/204207939 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa1a80c6be0 0x7fa1a80c39c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:13.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.544+0000 7fa1ad000640 1 --2- 172.21.15.6:0/204207939 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa1a806ce30 0x7fa1a8157ca0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:13.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.544+0000 7fa1ad000640 1 -- 172.21.15.6:0/204207939 >> 172.21.15.6:0/204207939 conn(0x7fa1a806ed80 msgr2=0x7fa1a80c8cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:13.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.544+0000 7fa1ad000640 1 -- 172.21.15.6:0/204207939 shutdown_connections 2024-04-24T23:16:13.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.544+0000 7fa1ad000640 1 -- 172.21.15.6:0/204207939 wait complete. 2024-04-24T23:16:13.566 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-04-24T23:16:13.566 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-04-24T23:16:13.567 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-04-24T23:16:13.567 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-04-24T23:16:13.568 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-04-24T23:16:13.568 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-04-24T23:16:13.568 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-04-24T23:16:13.569 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-04-24T23:16:13.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.864+0000 7fa436600640 1 Processor -- start 2024-04-24T23:16:13.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.864+0000 7fa436600640 1 -- start start 2024-04-24T23:16:13.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.866+0000 7f6bfbc00640 1 Processor -- start 2024-04-24T23:16:13.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.866+0000 7f6bfbc00640 1 -- start start 2024-04-24T23:16:13.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.867+0000 7fa436600640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fa4300c9770 0x7fa430150bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:13.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.868+0000 7fa436600640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa43006d930 con 0x7fa4300c8670 2024-04-24T23:16:13.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.868+0000 7fa436600640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa4301576d0 con 0x7fa4300c7810 2024-04-24T23:16:13.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.868+0000 7fa436600640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7fa4300c7380 con 0x7fa4300c9770 2024-04-24T23:16:13.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.869+0000 7f6bfbc00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6bec0d9040 0x7f6bec0061f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:13.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.869+0000 7f6bfbc00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f6bec049420 0x7f6bec049850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:13.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.869+0000 7f6bfbc00640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6bec006860 con 0x7f6bec0508e0 2024-04-24T23:16:13.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.869+0000 7f6bfbc00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f6bec049d90 con 0x7f6bec049420 2024-04-24T23:16:13.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.869+0000 7f6bfbc00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f6bec006b40 con 0x7f6bec0d9040 2024-04-24T23:16:13.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.870+0000 7f6bf9c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f6bec049420 0x7f6bec049850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:13.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.870+0000 7f6bf9c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f6bec049420 0x7f6bec049850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:59990/0 (socket says 172.21.15.6:59990) 2024-04-24T23:16:13.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.870+0000 7f6bf9c00640 1 -- 172.21.15.6:0/545794384 learned_addr learned my addr 172.21.15.6:0/545794384 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:16:13.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.871+0000 7f6bf9c00640 1 -- 172.21.15.6:0/545794384 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6bec0d9040 msgr2=0x7f6bec0061f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-24T23:16:13.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.868+0000 7fa42f400640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fa4300c9770 0x7fa430150bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:13.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.868+0000 7fa42ea00640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7fa4300c7810 0x7fa430071000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.6:50592/0 (socket says 172.21.15.6:50592) 2024-04-24T23:16:13.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.868+0000 7fa42ea00640 1 -- 172.21.15.6:0/2030432410 learned_addr learned my addr 172.21.15.6:0/2030432410 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:16:13.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.870+0000 7fa42f400640 1 -- 172.21.15.6:0/2030432410 >> v1:172.21.15.190:6789/0 conn(0x7fa4300c7810 legacy=0x7fa430071000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:13.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.870+0000 7fa42f400640 1 -- 172.21.15.6:0/2030432410 >> v1:172.21.15.6:6789/0 conn(0x7fa4300c8670 legacy=0x7fa430156b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:16:13.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.870+0000 7fa42f400640 1 -- 172.21.15.6:0/2030432410 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4301579b0 con 0x7fa4300c9770 2024-04-24T23:16:13.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.871+0000 7f6bf9c00640 1 --2- 172.21.15.6:0/545794384 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6bec0d9040 0x7f6bec0061f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:13.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.872+0000 7fa42f400640 1 --2- 172.21.15.6:0/2030432410 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa4300c9770 0x7fa430150bb0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fa418010b90 tx=0x7fa418032c40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5984d77edaadaa22 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:13.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.871+0000 7f6bf9c00640 1 -- 172.21.15.6:0/545794384 >> v1:172.21.15.190:6789/0 conn(0x7f6bec0508e0 legacy=0x7f6bec0e2310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:16:13.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.871+0000 7f6bf9c00640 1 -- 172.21.15.6:0/545794384 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6bec006e20 con 0x7f6bec049420 2024-04-24T23:16:13.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.872+0000 7fa42e000640 1 -- 172.21.15.6:0/2030432410 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa418040070 con 0x7fa4300c9770 2024-04-24T23:16:13.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.873+0000 7fa42e000640 1 -- 172.21.15.6:0/2030432410 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa418033950 con 0x7fa4300c9770 2024-04-24T23:16:13.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.873+0000 7fa42e000640 1 -- 172.21.15.6:0/2030432410 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa418040070 con 0x7fa4300c9770 2024-04-24T23:16:13.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.873+0000 7f6bf9c00640 1 --2- 172.21.15.6:0/545794384 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6bec049420 0x7f6bec049850 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f6be8009870 tx=0x7f6be802a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58d737c84fc31b2f server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:13.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.873+0000 7f6bf3400640 1 -- 172.21.15.6:0/545794384 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6be8037070 con 0x7f6bec049420 2024-04-24T23:16:13.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.873+0000 7f6bf3400640 1 -- 172.21.15.6:0/545794384 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6be8032440 con 0x7f6bec049420 2024-04-24T23:16:13.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.874+0000 7f6bf3400640 1 -- 172.21.15.6:0/545794384 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6be8037070 con 0x7f6bec049420 2024-04-24T23:16:13.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.874+0000 7fa436600640 1 -- 172.21.15.6:0/2030432410 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa4300c9770 msgr2=0x7fa430150bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:13.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.874+0000 7fa436600640 1 --2- 172.21.15.6:0/2030432410 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa4300c9770 0x7fa430150bb0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fa418010b90 tx=0x7fa418032c40 comp rx=0 tx=0).stop 2024-04-24T23:16:13.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.874+0000 7fa436600640 1 -- 172.21.15.6:0/2030432410 shutdown_connections 2024-04-24T23:16:13.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.874+0000 7fa436600640 1 --2- 172.21.15.6:0/2030432410 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa4300c9770 0x7fa430150bb0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:13.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.874+0000 7fa436600640 1 -- 172.21.15.6:0/2030432410 >> 172.21.15.6:0/2030432410 conn(0x7fa4300c8c50 msgr2=0x7fa4300c9060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:13.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.874+0000 7fa436600640 1 -- 172.21.15.6:0/2030432410 shutdown_connections 2024-04-24T23:16:13.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.874+0000 7f6bfbc00640 1 -- 172.21.15.6:0/545794384 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6bec049420 msgr2=0x7f6bec049850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:13.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.874+0000 7f6bfbc00640 1 --2- 172.21.15.6:0/545794384 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6bec049420 0x7f6bec049850 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f6be8009870 tx=0x7f6be802a230 comp rx=0 tx=0).stop 2024-04-24T23:16:13.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.874+0000 7fa436600640 1 -- 172.21.15.6:0/2030432410 wait complete. 2024-04-24T23:16:13.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.875+0000 7f6bfbc00640 1 -- 172.21.15.6:0/545794384 shutdown_connections 2024-04-24T23:16:13.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.875+0000 7f6bfbc00640 1 --2- 172.21.15.6:0/545794384 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6bec049420 0x7f6bec049850 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:13.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.875+0000 7f6bfbc00640 1 --2- 172.21.15.6:0/545794384 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6bec0d9040 0x7f6bec0061f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:13.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.875+0000 7f6bfbc00640 1 -- 172.21.15.6:0/545794384 >> 172.21.15.6:0/545794384 conn(0x7f6bec054580 msgr2=0x7f6bec054990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:13.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.875+0000 7f6bfbc00640 1 -- 172.21.15.6:0/545794384 shutdown_connections 2024-04-24T23:16:13.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.875+0000 7f6bfbc00640 1 -- 172.21.15.6:0/545794384 wait complete. 2024-04-24T23:16:13.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.877+0000 7f6bfbc00640 1 Processor -- start 2024-04-24T23:16:13.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.878+0000 7f6bfbc00640 1 -- start start 2024-04-24T23:16:13.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.880+0000 7f6bfbc00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6bec049420 0x7f6bec1767b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:13.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.880+0000 7f6bfbc00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6bec0508e0 0x7f6bec176cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:13.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.878+0000 7fa436600640 1 Processor -- start 2024-04-24T23:16:13.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.878+0000 7fa436600640 1 -- start start 2024-04-24T23:16:13.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.880+0000 7fa436600640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa4300c7810 0x7fa4301ec600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:13.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.880+0000 7fa436600640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa4300c8670 0x7fa4301efb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:13.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.880+0000 7fa436600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa4300c9770 0x7fa430148270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:13.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.880+0000 7fa436600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fa4300c7010 con 0x7fa4300c9770 2024-04-24T23:16:13.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.880+0000 7fa42ea00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa4300c8670 0x7fa4301efb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:13.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.880+0000 7fa436600640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fa4300c6d30 con 0x7fa4300c7810 2024-04-24T23:16:13.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.880+0000 7fa436600640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fa430157250 con 0x7fa4300c8670 2024-04-24T23:16:13.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.881+0000 7fa42ea00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa4300c8670 0x7fa4301efb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.6:53142/0 (socket says 172.21.15.6:53142) 2024-04-24T23:16:13.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.881+0000 7fa42ea00640 1 -- 172.21.15.6:0/2188732461 learned_addr learned my addr 172.21.15.6:0/2188732461 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:16:13.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.880+0000 7fa42f400640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa4300c7810 0x7fa4301ec600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:13.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.881+0000 7fa42fe00640 1 --2- 172.21.15.6:0/2188732461 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa4300c9770 0x7fa430148270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:13.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.881+0000 7f6bfbc00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6bec0d9040 0x7f6bec177560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:13.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.881+0000 7f6bf9200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6bec049420 0x7f6bec1767b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:13.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.881+0000 7fa42f400640 1 -- 172.21.15.6:0/2188732461 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa4300c8670 msgr2=0x7fa4301efb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:13.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.881+0000 7fa42f400640 1 --2- 172.21.15.6:0/2188732461 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa4300c8670 0x7fa4301efb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:13.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.881+0000 7fa42f400640 1 -- 172.21.15.6:0/2188732461 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa4300c9770 msgr2=0x7fa430148270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:13.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.881+0000 7fa42f400640 1 --2- 172.21.15.6:0/2188732461 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa4300c9770 0x7fa430148270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:13.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.881+0000 7fa42f400640 1 -- 172.21.15.6:0/2188732461 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4301f02b0 con 0x7fa4300c7810 2024-04-24T23:16:13.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.881+0000 7f6bf9200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6bec049420 0x7f6bec1767b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:60006/0 (socket says 172.21.15.6:60006) 2024-04-24T23:16:13.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.882+0000 7f6bf9200640 1 -- 172.21.15.6:0/1232895690 learned_addr learned my addr 172.21.15.6:0/1232895690 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:16:13.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.882+0000 7fc4bd400640 1 Processor -- start 2024-04-24T23:16:13.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.882+0000 7fa42fe00640 1 --2- 172.21.15.6:0/2188732461 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa4300c9770 0x7fa430148270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:16:13.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.882+0000 7f6bf9c00640 1 --2- 172.21.15.6:0/1232895690 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6bec0d9040 0x7f6bec177560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:13.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.882+0000 7f6bfbc00640 1 -- 172.21.15.6:0/1232895690 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f6bec0076c0 con 0x7f6bec049420 2024-04-24T23:16:13.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.882+0000 7fc4bd400640 1 -- start start 2024-04-24T23:16:13.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.882+0000 7f6bf3e00640 1 --2- 172.21.15.6:0/1232895690 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6bec0508e0 0x7f6bec176cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:13.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.882+0000 7f6bfbc00640 1 -- 172.21.15.6:0/1232895690 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f6bec0073e0 con 0x7f6bec0d9040 2024-04-24T23:16:13.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.882+0000 7f6bfbc00640 1 -- 172.21.15.6:0/1232895690 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f6bec0d9420 con 0x7f6bec0508e0 2024-04-24T23:16:13.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.883+0000 7f6bf9200640 1 -- 172.21.15.6:0/1232895690 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6bec0508e0 msgr2=0x7f6bec176cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:13.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.883+0000 7fa42f400640 1 --2- 172.21.15.6:0/2188732461 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa4300c7810 0x7fa4301ec600 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fa43006bf80 tx=0x7fa4180046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:13.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.884+0000 7f6bf9200640 1 --2- 172.21.15.6:0/1232895690 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6bec0508e0 0x7f6bec176cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:13.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.884+0000 7fa427400640 1 -- 172.21.15.6:0/2188732461 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa418040070 con 0x7fa4300c7810 2024-04-24T23:16:13.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.884+0000 7fa427400640 1 -- 172.21.15.6:0/2188732461 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa418019380 con 0x7fa4300c7810 2024-04-24T23:16:13.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.884+0000 7fa427400640 1 -- 172.21.15.6:0/2188732461 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa418039550 con 0x7fa4300c7810 2024-04-24T23:16:13.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.884+0000 7fc4bd400640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fc4b8154cb0 0x7fc4b806cd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:13.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.884+0000 7f6bf9200640 1 -- 172.21.15.6:0/1232895690 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6bec0d9040 msgr2=0x7f6bec177560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:13.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.884+0000 7f6bf9200640 1 --2- 172.21.15.6:0/1232895690 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6bec0d9040 0x7f6bec177560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:13.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.884+0000 7f6bf9200640 1 -- 172.21.15.6:0/1232895690 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6bec0dd2e0 con 0x7f6bec049420 2024-04-24T23:16:13.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.884+0000 7f6bf9c00640 1 --2- 172.21.15.6:0/1232895690 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6bec0d9040 0x7f6bec177560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:16:13.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.885+0000 7fc4bca00640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fc4b8154cb0 0x7fc4b806cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:13.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.885+0000 7fc4bca00640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fc4b8154cb0 0x7fc4b806cd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:59792/0 (socket says 172.21.15.6:59792) 2024-04-24T23:16:13.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.885+0000 7fc4bca00640 1 -- 172.21.15.6:0/1537732802 learned_addr learned my addr 172.21.15.6:0/1537732802 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:16:13.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.885+0000 7f6bf9200640 1 --2- 172.21.15.6:0/1232895690 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6bec049420 0x7f6bec1767b0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f6bf4074280 tx=0x7f6bf4076b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:13.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.885+0000 7fc4bd400640 1 -- 172.21.15.6:0/1537732802 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc4b806d430 con 0x7fc4b80c8770 2024-04-24T23:16:13.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.885+0000 7fc4bd400640 1 -- 172.21.15.6:0/1537732802 --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc4b8071b20 con 0x7fc4b80c8390 2024-04-24T23:16:13.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.885+0000 7fc4bd400640 1 -- 172.21.15.6:0/1537732802 --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7fc4b8071640 con 0x7fc4b8154cb0 2024-04-24T23:16:13.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.884+0000 7fa436600640 1 -- 172.21.15.6:0/2188732461 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4301f0590 con 0x7fa4300c7810 2024-04-24T23:16:13.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.884+0000 7fa436600640 1 -- 172.21.15.6:0/2188732461 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa430147a70 con 0x7fa4300c7810 2024-04-24T23:16:13.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.886+0000 7fa436600640 1 -- 172.21.15.6:0/2188732461 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fa408001030 con 0x7fa4300c7810 2024-04-24T23:16:13.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.886+0000 7f6bf0c00640 1 -- 172.21.15.6:0/1232895690 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bf4068b40 con 0x7f6bec049420 2024-04-24T23:16:13.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.886+0000 7f6bf0c00640 1 -- 172.21.15.6:0/1232895690 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6bf4090030 con 0x7f6bec049420 2024-04-24T23:16:13.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.886+0000 7f6bf0c00640 1 -- 172.21.15.6:0/1232895690 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bf4068f20 con 0x7f6bec049420 2024-04-24T23:16:13.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.887+0000 7f6bfbc00640 1 -- 172.21.15.6:0/1232895690 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6bec0dd5c0 con 0x7f6bec049420 2024-04-24T23:16:13.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.887+0000 7f6bfbc00640 1 -- 172.21.15.6:0/1232895690 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6bec0deca0 con 0x7f6bec049420 2024-04-24T23:16:13.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.886+0000 7fc4bca00640 1 -- 172.21.15.6:0/1537732802 >> v1:172.21.15.190:6789/0 conn(0x7fc4b80c8390 legacy=0x7fc4b8070f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:13.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.886+0000 7fc4bca00640 1 -- 172.21.15.6:0/1537732802 >> v1:172.21.15.6:6789/0 conn(0x7fc4b80c8770 legacy=0x7fc4b81545a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:13.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.886+0000 7fc4bca00640 1 -- 172.21.15.6:0/1537732802 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4b8071e00 con 0x7fc4b8154cb0 2024-04-24T23:16:13.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.888+0000 7f0e83000640 1 Processor -- start 2024-04-24T23:16:13.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.888+0000 7fc4bca00640 1 --2- 172.21.15.6:0/1537732802 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc4b8154cb0 0x7fc4b806cd30 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fc4a8009870 tx=0x7fc4a802a400 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d95fd02b5d3bde89 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:13.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.888+0000 7f0e83000640 1 -- start start 2024-04-24T23:16:13.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.888+0000 7fc4b6a00640 1 -- 172.21.15.6:0/1537732802 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4a8037070 con 0x7fc4b8154cb0 2024-04-24T23:16:13.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.889+0000 7fc4b6a00640 1 -- 172.21.15.6:0/1537732802 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc4a8032640 con 0x7fc4b8154cb0 2024-04-24T23:16:13.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.890+0000 7fc4b6a00640 1 -- 172.21.15.6:0/1537732802 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4a8032a50 con 0x7fc4b8154cb0 2024-04-24T23:16:13.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.888+0000 7f6bfbc00640 1 -- 172.21.15.6:0/1232895690 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f6bd0001030 con 0x7f6bec049420 2024-04-24T23:16:13.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.890+0000 7f6bf0c00640 1 -- 172.21.15.6:0/1232895690 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f6bf4095420 con 0x7f6bec049420 2024-04-24T23:16:13.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.890+0000 7f0e83000640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0e7c06bc70 0x7f0e7c15b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:13.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.890+0000 7f0e83000640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0e7c15f0e0 con 0x7f0e7c06c050 2024-04-24T23:16:13.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.890+0000 7f0e83000640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0e7c15bb30 con 0x7f0e7c0cdae0 2024-04-24T23:16:13.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.890+0000 7f0e83000640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f0e7c15e560 con 0x7f0e7c06bc70 2024-04-24T23:16:13.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.891+0000 7fc4bd400640 1 -- 172.21.15.6:0/1537732802 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc4b8154cb0 msgr2=0x7fc4b806cd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:13.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.891+0000 7fc4bd400640 1 --2- 172.21.15.6:0/1537732802 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc4b8154cb0 0x7fc4b806cd30 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fc4a8009870 tx=0x7fc4a802a400 comp rx=0 tx=0).stop 2024-04-24T23:16:13.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.891+0000 7f485b200640 1 Processor -- start 2024-04-24T23:16:13.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.891+0000 7fc4bd400640 1 -- 172.21.15.6:0/1537732802 shutdown_connections 2024-04-24T23:16:13.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.891+0000 7fc4bd400640 1 --2- 172.21.15.6:0/1537732802 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc4b8154cb0 0x7fc4b806cd30 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:13.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.891+0000 7fc4bd400640 1 -- 172.21.15.6:0/1537732802 >> 172.21.15.6:0/1537732802 conn(0x7fc4b80cc1e0 msgr2=0x7fc4b80cc5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:13.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.891+0000 7f0e7be00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0e7c06bc70 0x7f0e7c15b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:13.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.892+0000 7fc4bd400640 1 -- 172.21.15.6:0/1537732802 shutdown_connections 2024-04-24T23:16:13.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.892+0000 7fc4bd400640 1 -- 172.21.15.6:0/1537732802 wait complete. 2024-04-24T23:16:13.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.891+0000 7f0e81000640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f0e7c0cdae0 0x7f0e7c0cc2d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.6:50624/0 (socket says 172.21.15.6:50624) 2024-04-24T23:16:13.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.892+0000 7f0e81000640 1 -- 172.21.15.6:0/994627828 learned_addr learned my addr 172.21.15.6:0/994627828 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:16:13.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.891+0000 7f0e7b400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f0e7c06c050 0x7f0e7c154a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:43454/0 (socket says 172.21.15.6:43454) 2024-04-24T23:16:13.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.892+0000 7f5992600640 1 Processor -- start 2024-04-24T23:16:13.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.892+0000 7f0e7aa00640 1 -- 172.21.15.6:0/994627828 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2595398996 0 0) 0x7f0e68001110 con 0x7f0e7c0cdae0 2024-04-24T23:16:13.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.893+0000 7f0e7aa00640 1 -- 172.21.15.6:0/994627828 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0e60003520 con 0x7f0e7c0cdae0 2024-04-24T23:16:13.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.892+0000 7f6bf0c00640 1 --2- 172.21.15.6:0/1232895690 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f6bd406bf30 0x7f6bd406e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:13.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.892+0000 7f6bf0c00640 1 -- 172.21.15.6:0/1232895690 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f6bf41076c0 con 0x7f6bec049420 2024-04-24T23:16:13.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.893+0000 7f6bf0c00640 1 --2- 172.21.15.6:0/1232895690 >> [v2:172.21.15.6:6808/3022569437,v1:172.21.15.6:6809/3022569437] conn(0x7f6bd40756b0 0x7f6bd4077af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:13.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.893+0000 7f6bf0c00640 1 -- 172.21.15.6:0/1232895690 --> [v2:172.21.15.6:6808/3022569437,v1:172.21.15.6:6809/3022569437] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6bd40781c0 con 0x7f6bd40756b0 2024-04-24T23:16:13.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.893+0000 7f6bf0c00640 1 -- 172.21.15.6:0/1232895690 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f6bf410e050 con 0x7f6bec049420 2024-04-24T23:16:13.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.891+0000 7fa427400640 1 -- 172.21.15.6:0/2188732461 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fa4180396f0 con 0x7fa4300c7810 2024-04-24T23:16:13.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.892+0000 7fa427400640 1 --2- 172.21.15.6:0/2188732461 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fa40c06bf00 0x7fa40c06e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:13.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.892+0000 7fa427400640 1 -- 172.21.15.6:0/2188732461 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fa4180abef0 con 0x7fa4300c7810 2024-04-24T23:16:13.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.893+0000 7f0e7aa00640 1 -- 172.21.15.6:0/994627828 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2623660101 0 0) 0x7f0e60003520 con 0x7f0e7c0cdae0 2024-04-24T23:16:13.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.893+0000 7f0e7aa00640 1 -- 172.21.15.6:0/994627828 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0e7c06bc70 msgr2=0x7f0e7c15b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:16:13.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.893+0000 7f0e7aa00640 1 --2- 172.21.15.6:0/994627828 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0e7c06bc70 0x7f0e7c15b5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:13.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.893+0000 7f0e7aa00640 1 -- 172.21.15.6:0/994627828 >> v1:172.21.15.6:6789/0 conn(0x7f0e7c06c050 legacy=0x7f0e7c154a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:13.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.893+0000 7f0e7aa00640 1 -- 172.21.15.6:0/994627828 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e7c15e840 con 0x7f0e7c0cdae0 2024-04-24T23:16:13.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.893+0000 7fa42ea00640 1 --2- 172.21.15.6:0/2188732461 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fa40c06bf00 0x7fa40c06e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:13.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.893+0000 7fa427400640 1 --2- 172.21.15.6:0/2188732461 >> [v2:172.21.15.6:6816/3687836888,v1:172.21.15.6:6817/3687836888] conn(0x7fa40c075680 0x7fa40c077ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:13.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.893+0000 7fa427400640 1 -- 172.21.15.6:0/2188732461 --> [v2:172.21.15.6:6816/3687836888,v1:172.21.15.6:6817/3687836888] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa418013e90 con 0x7fa40c075680 2024-04-24T23:16:13.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.893+0000 7fa427400640 1 -- 172.21.15.6:0/2188732461 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fa4180ac2e0 con 0x7fa4300c7810 2024-04-24T23:16:13.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.893+0000 7f485b200640 1 -- start start 2024-04-24T23:16:13.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.893+0000 7f6bf3e00640 1 --2- 172.21.15.6:0/1232895690 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f6bd406bf30 0x7f6bd406e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:13.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.893+0000 7fa42ea00640 1 --2- 172.21.15.6:0/2188732461 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fa40c06bf00 0x7fa40c06e3c0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fa4300cb2a0 tx=0x7fa420001000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:13.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.893+0000 7f5992600640 1 -- start start 2024-04-24T23:16:13.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.894+0000 7fa42fe00640 1 --2- 172.21.15.6:0/2188732461 >> [v2:172.21.15.6:6816/3687836888,v1:172.21.15.6:6817/3687836888] conn(0x7fa40c075680 0x7fa40c077ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:13.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.894+0000 7efc78a00640 1 Processor -- start 2024-04-24T23:16:13.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.894+0000 7f6bf9c00640 1 --2- 172.21.15.6:0/1232895690 >> [v2:172.21.15.6:6808/3022569437,v1:172.21.15.6:6809/3022569437] conn(0x7f6bd40756b0 0x7f6bd4077af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:13.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.893+0000 7f0e7aa00640 1 -- 172.21.15.6:0/994627828 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3463268701 0 0) 0x7f0e68001820 con 0x7f0e7c0cdae0 2024-04-24T23:16:13.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.893+0000 7f0e7aa00640 1 -- 172.21.15.6:0/994627828 >> v1:172.21.15.190:6789/0 conn(0x7f0e7c0cdae0 legacy=0x7f0e7c0cc2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:13.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.894+0000 7f0e7aa00640 1 --2- 172.21.15.6:0/994627828 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0e600039e0 0x7f0e60003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:13.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.894+0000 7f0e7aa00640 1 --2- 172.21.15.6:0/994627828 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0e7c06bc70 0x7f0e60004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:13.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.894+0000 7f0e7aa00640 1 --2- 172.21.15.6:0/994627828 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0e60004d40 0x7f0e600051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:13.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.894+0000 7f0e7aa00640 1 -- 172.21.15.6:0/994627828 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f0e7c15e560 con 0x7f0e60004d40 2024-04-24T23:16:13.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.894+0000 7f0e7aa00640 1 -- 172.21.15.6:0/994627828 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f0e6000bbe0 con 0x7f0e600039e0 2024-04-24T23:16:13.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.894+0000 7f0e7aa00640 1 -- 172.21.15.6:0/994627828 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f0e6000bde0 con 0x7f0e7c06bc70 2024-04-24T23:16:13.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.895+0000 7f6bf3e00640 1 --2- 172.21.15.6:0/1232895690 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f6bd406bf30 0x7f6bd406e3f0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f6bec0e3310 tx=0x7f6be0007e30 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:13.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.895+0000 7f5992600640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f598c0b35b0 0x7f598c15ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:13.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.895+0000 7f5992600640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f598c15b0b0 0x7f598c0cb240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:13.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.895+0000 7f5992600640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f598c0b3150 con 0x7f598c154dd0 2024-04-24T23:16:13.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.895+0000 7fc4bd400640 1 Processor -- start 2024-04-24T23:16:13.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.895+0000 7f5992600640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f598c15f020 con 0x7f598c15b0b0 2024-04-24T23:16:13.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.895+0000 7f5992600640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f598c15e260 con 0x7f598c0b35b0 2024-04-24T23:16:13.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.895+0000 7f485b200640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f485404c210 0x7f48540df9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:13.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.895+0000 7f485b200640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f48540d5cc0 0x7f48540dc2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:13.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.895+0000 7f485b200640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f48540e2310 con 0x7f4854051220 2024-04-24T23:16:13.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.895+0000 7f485b200640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f48540dbe50 con 0x7f48540d5cc0 2024-04-24T23:16:13.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.895+0000 7f485b200640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f485404cb80 con 0x7f485404c210 2024-04-24T23:16:13.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.895+0000 7f4859400640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f48540d5cc0 0x7f48540dc2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:13.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.895+0000 7f4859400640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f48540d5cc0 0x7f48540dc2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:59806/0 (socket says 172.21.15.6:59806) 2024-04-24T23:16:13.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.895+0000 7f4859400640 1 -- 172.21.15.6:0/2434952094 learned_addr learned my addr 172.21.15.6:0/2434952094 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:16:13.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.895+0000 7f598be00640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f598c15b0b0 0x7f598c0cb240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:13.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.895+0000 7efc78a00640 1 -- start start 2024-04-24T23:16:13.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.896+0000 7f598be00640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f598c15b0b0 0x7f598c0cb240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:59820/0 (socket says 172.21.15.6:59820) 2024-04-24T23:16:13.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.895+0000 7f6bf9c00640 1 --2- 172.21.15.6:0/1232895690 >> [v2:172.21.15.6:6808/3022569437,v1:172.21.15.6:6809/3022569437] conn(0x7f6bd40756b0 0x7f6bd4077af0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:13.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.895+0000 7f0e7b400640 1 --2- 172.21.15.6:0/994627828 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0e7c06bc70 0x7f0e60004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:13.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.896+0000 7f0e7be00640 1 --2- 172.21.15.6:0/994627828 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0e600039e0 0x7f0e60003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:13.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.896+0000 7f598be00640 1 -- 172.21.15.6:0/2090237320 learned_addr learned my addr 172.21.15.6:0/2090237320 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:16:13.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.896+0000 7fc4bd400640 1 -- start start 2024-04-24T23:16:13.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.896+0000 7fa42fe00640 1 --2- 172.21.15.6:0/2188732461 >> [v2:172.21.15.6:6816/3687836888,v1:172.21.15.6:6817/3687836888] conn(0x7fa40c075680 0x7fa40c077ac0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:13.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.896+0000 7f598be00640 1 -- 172.21.15.6:0/2090237320 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f598c0b35b0 msgr2=0x7f598c15ab70 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-24T23:16:13.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.896+0000 7f598be00640 1 --2- 172.21.15.6:0/2090237320 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f598c0b35b0 0x7f598c15ab70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:13.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.896+0000 7f598be00640 1 -- 172.21.15.6:0/2090237320 >> v1:172.21.15.190:6789/0 conn(0x7f598c154dd0 legacy=0x7f598c0cbfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:16:13.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.896+0000 7f598be00640 1 -- 172.21.15.6:0/2090237320 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f598c15e540 con 0x7f598c15b0b0 2024-04-24T23:16:13.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.896+0000 7f0e7b400640 1 -- 172.21.15.6:0/994627828 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0e600039e0 msgr2=0x7f0e60003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:16:13.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.896+0000 7f0e7b400640 1 --2- 172.21.15.6:0/994627828 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0e600039e0 0x7f0e60003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:13.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.897+0000 7f0e7b400640 1 -- 172.21.15.6:0/994627828 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0e60004d40 msgr2=0x7f0e600051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:16:13.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.896+0000 7f4859e00640 1 --2- 172.21.15.6:0/2434952094 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f485404c210 0x7f48540df9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:13.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.897+0000 7f4859400640 1 -- 172.21.15.6:0/2434952094 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f485404c210 msgr2=0x7f48540df9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:16:13.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.897+0000 7f4859400640 1 --2- 172.21.15.6:0/2434952094 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f485404c210 0x7f48540df9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:13.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.897+0000 7f4859400640 1 -- 172.21.15.6:0/2434952094 >> v1:172.21.15.190:6789/0 conn(0x7f4854051220 legacy=0x7f485404dca0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-24T23:16:13.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.897+0000 7f4859400640 1 -- 172.21.15.6:0/2434952094 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48540d6120 con 0x7f48540d5cc0 2024-04-24T23:16:13.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.897+0000 7f0e81000640 1 --2- 172.21.15.6:0/994627828 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0e60004d40 0x7f0e600051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:13.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.897+0000 7f6bf0c00640 1 -- 172.21.15.6:0/1232895690 <== osd.0 v2:172.21.15.6:6808/3022569437 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6bd40781c0 con 0x7f6bd40756b0 2024-04-24T23:16:13.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.897+0000 7f0e7b400640 1 --2- 172.21.15.6:0/994627828 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0e60004d40 0x7f0e600051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:13.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.897+0000 7efc78a00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efc740b4bc0 0x7efc74162e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:13.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.897+0000 7efc78a00640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7efc740d03c0 0x7efc740d0810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:13.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.898+0000 7efc78a00640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efc740b5100 con 0x7efc7415fa00 2024-04-24T23:16:13.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.898+0000 7efc78a00640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7efc740b53c0 con 0x7efc740d03c0 2024-04-24T23:16:13.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.898+0000 7efc78a00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7efc740b56a0 con 0x7efc740b4bc0 2024-04-24T23:16:13.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.898+0000 7efc73200640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7efc740d03c0 0x7efc740d0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:13.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.898+0000 7efc72800640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7efc7415fa00 0x7efc740720b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.6:50658/0 (socket says 172.21.15.6:50658) 2024-04-24T23:16:13.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.898+0000 7efc72800640 1 -- 172.21.15.6:0/2775045399 learned_addr learned my addr 172.21.15.6:0/2775045399 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:16:13.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.898+0000 7fa427400640 1 -- 172.21.15.6:0/2188732461 <== osd.1 v2:172.21.15.6:6816/3687836888 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa418013e90 con 0x7fa40c075680 2024-04-24T23:16:13.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.897+0000 7f0e7b400640 1 -- 172.21.15.6:0/994627828 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e60003520 con 0x7f0e7c06bc70 2024-04-24T23:16:13.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.898+0000 7f0e81000640 1 --2- 172.21.15.6:0/994627828 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0e60004d40 0x7f0e600051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:16:13.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.898+0000 7efc71e00640 1 --2- 172.21.15.6:0/2775045399 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efc740b4bc0 0x7efc74162e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:13.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.898+0000 7efc71400640 1 -- 172.21.15.6:0/2775045399 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3721788867 0 0) 0x7efc740b5100 con 0x7efc7415fa00 2024-04-24T23:16:13.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.898+0000 7f598be00640 1 --2- 172.21.15.6:0/2090237320 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f598c15b0b0 0x7f598c0cb240 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f5984009e50 tx=0x7f598402a6f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3c5304fc6b8d67aa server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:13.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.899+0000 7f598a000640 1 -- 172.21.15.6:0/2090237320 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5984036070 con 0x7f598c15b0b0 2024-04-24T23:16:13.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.899+0000 7f598a000640 1 -- 172.21.15.6:0/2090237320 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5984031600 con 0x7f598c15b0b0 2024-04-24T23:16:13.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.899+0000 7f598a000640 1 -- 172.21.15.6:0/2090237320 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59840319d0 con 0x7f598c15b0b0 2024-04-24T23:16:13.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.899+0000 7efc71400640 1 -- 172.21.15.6:0/2775045399 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efc58003520 con 0x7efc7415fa00 2024-04-24T23:16:13.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.899+0000 7efc73200640 1 -- 172.21.15.6:0/2775045399 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efc740b4bc0 msgr2=0x7efc74162e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:16:13.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.899+0000 7efc73200640 1 --2- 172.21.15.6:0/2775045399 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efc740b4bc0 0x7efc74162e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:13.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.899+0000 7efc73200640 1 -- 172.21.15.6:0/2775045399 >> v1:172.21.15.190:6789/0 conn(0x7efc7415fa00 legacy=0x7efc740720b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:13.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.899+0000 7efc73200640 1 -- 172.21.15.6:0/2775045399 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc74162680 con 0x7efc740d03c0 2024-04-24T23:16:13.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.899+0000 7f0e7b400640 1 --2- 172.21.15.6:0/994627828 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0e7c06bc70 0x7f0e60004800 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f0e74001e50 tx=0x7f0e60005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=db24fd3dc64b5f33 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:13.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.899+0000 7f4882a00640 1 Processor -- start 2024-04-24T23:16:13.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.899+0000 7fc4bd400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc4b80c8390 0x7fc4b806e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:13.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.899+0000 7fc4bd400640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc4b80c8770 0x7fc4b81f4b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:13.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.899+0000 7f4859400640 1 --2- 172.21.15.6:0/2434952094 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f48540d5cc0 0x7f48540dc2e0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f48540087d0 tx=0x7f484802a2a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eccfe8ff5857dd67 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:13.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.899+0000 7f4858a00640 1 -- 172.21.15.6:0/2434952094 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4848037070 con 0x7f48540d5cc0 2024-04-24T23:16:13.919 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.900+0000 7f0e7aa00640 1 -- 172.21.15.6:0/994627828 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e74001aa0 con 0x7f0e7c06bc70 2024-04-24T23:16:13.919 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.900+0000 7f0e7aa00640 1 -- 172.21.15.6:0/994627828 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0e7402e070 con 0x7f0e7c06bc70 2024-04-24T23:16:13.919 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.900+0000 7f0e7aa00640 1 -- 172.21.15.6:0/994627828 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e74004d80 con 0x7f0e7c06bc70 2024-04-24T23:16:13.919 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.900+0000 7f5992600640 1 -- 172.21.15.6:0/2090237320 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f598c15b0b0 msgr2=0x7f598c0cb240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:13.920 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.900+0000 7f5992600640 1 --2- 172.21.15.6:0/2090237320 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f598c15b0b0 0x7f598c0cb240 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f5984009e50 tx=0x7f598402a6f0 comp rx=0 tx=0).stop 2024-04-24T23:16:13.920 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.900+0000 7f4858a00640 1 -- 172.21.15.6:0/2434952094 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4848032440 con 0x7f48540d5cc0 2024-04-24T23:16:13.920 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.900+0000 7f4858a00640 1 -- 172.21.15.6:0/2434952094 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4848032850 con 0x7f48540d5cc0 2024-04-24T23:16:13.920 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.900+0000 7f5992600640 1 -- 172.21.15.6:0/2090237320 shutdown_connections 2024-04-24T23:16:13.920 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.900+0000 7f5992600640 1 --2- 172.21.15.6:0/2090237320 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f598c15b0b0 0x7f598c0cb240 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:13.920 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.900+0000 7f5992600640 1 --2- 172.21.15.6:0/2090237320 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f598c0b35b0 0x7f598c15ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:13.920 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.900+0000 7f5992600640 1 -- 172.21.15.6:0/2090237320 >> 172.21.15.6:0/2090237320 conn(0x7f598c066ea0 msgr2=0x7f598c0672b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:13.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.900+0000 7fc4b7e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc4b80c8390 0x7fc4b806e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:13.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.901+0000 7fc4b7e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc4b80c8390 0x7fc4b806e7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:60036/0 (socket says 172.21.15.6:60036) 2024-04-24T23:16:13.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.901+0000 7f0e83000640 1 -- 172.21.15.6:0/994627828 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0e7c06bc70 msgr2=0x7f0e60004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:13.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.901+0000 7f0e83000640 1 --2- 172.21.15.6:0/994627828 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0e7c06bc70 0x7f0e60004800 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f0e74001e50 tx=0x7f0e60005830 comp rx=0 tx=0).stop 2024-04-24T23:16:13.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.901+0000 7fc4b7e00640 1 -- 172.21.15.6:0/2181508344 learned_addr learned my addr 172.21.15.6:0/2181508344 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:16:13.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.901+0000 7efc73200640 1 --2- 172.21.15.6:0/2775045399 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efc740d03c0 0x7efc740d0810 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7efc6c009d90 tx=0x7efc6c02a6f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=65826eca96675667 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:13.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.901+0000 7f0e81000640 1 -- 172.21.15.6:0/994627828 reap_dead start 2024-04-24T23:16:13.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.901+0000 7fc4bd400640 1 --2- 172.21.15.6:0/2181508344 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc4b8154cb0 0x7fc4b81f5060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:13.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.901+0000 7fc4b7400640 1 --2- 172.21.15.6:0/2181508344 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc4b80c8770 0x7fc4b81f4b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:13.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.901+0000 7f485b200640 1 -- 172.21.15.6:0/2434952094 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f48540d5cc0 msgr2=0x7f48540dc2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:13.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.901+0000 7f485b200640 1 --2- 172.21.15.6:0/2434952094 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f48540d5cc0 0x7f48540dc2e0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f48540087d0 tx=0x7f484802a2a0 comp rx=0 tx=0).stop 2024-04-24T23:16:13.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.901+0000 7f5992600640 1 -- 172.21.15.6:0/2090237320 shutdown_connections 2024-04-24T23:16:13.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.901+0000 7efc71400640 1 -- 172.21.15.6:0/2775045399 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc6c007740 con 0x7efc740d03c0 2024-04-24T23:16:13.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.901+0000 7f0e83000640 1 -- 172.21.15.6:0/994627828 shutdown_connections 2024-04-24T23:16:13.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.901+0000 7f0e83000640 1 -- 172.21.15.6:0/994627828 >> 172.21.15.6:0/994627828 conn(0x7f0e7c072e40 msgr2=0x7f0e7c06d8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:13.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.901+0000 7efc71400640 1 -- 172.21.15.6:0/2775045399 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efc6c0078e0 con 0x7efc740d03c0 2024-04-24T23:16:13.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.901+0000 7efc71400640 1 -- 172.21.15.6:0/2775045399 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc6c007cb0 con 0x7efc740d03c0 2024-04-24T23:16:13.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.900+0000 7f4882a00640 1 -- start start 2024-04-24T23:16:13.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.901+0000 7f485b200640 1 -- 172.21.15.6:0/2434952094 shutdown_connections 2024-04-24T23:16:13.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.901+0000 7f485b200640 1 --2- 172.21.15.6:0/2434952094 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f48540d5cc0 0x7f48540dc2e0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:13.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.901+0000 7f485b200640 1 --2- 172.21.15.6:0/2434952094 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f485404c210 0x7f48540df9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:13.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.901+0000 7f485b200640 1 -- 172.21.15.6:0/2434952094 >> 172.21.15.6:0/2434952094 conn(0x7f4854007c90 msgr2=0x7f4854008070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:13.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.902+0000 7f0e83000640 1 -- 172.21.15.6:0/994627828 shutdown_connections 2024-04-24T23:16:13.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.902+0000 7fc4bd400640 1 -- 172.21.15.6:0/2181508344 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fc4b815aa70 con 0x7fc4b80c8390 2024-04-24T23:16:13.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.902+0000 7fc4bd400640 1 -- 172.21.15.6:0/2181508344 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fc4b80718d0 con 0x7fc4b80c8770 2024-04-24T23:16:13.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.902+0000 7f485b200640 1 -- 172.21.15.6:0/2434952094 shutdown_connections 2024-04-24T23:16:13.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.902+0000 7fc4bd400640 1 -- 172.21.15.6:0/2181508344 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fc4b815abf0 con 0x7fc4b8154cb0 2024-04-24T23:16:13.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.902+0000 7fc4bca00640 1 --2- 172.21.15.6:0/2181508344 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc4b8154cb0 0x7fc4b81f5060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:13.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.902+0000 7efc78a00640 1 -- 172.21.15.6:0/2775045399 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efc740d03c0 msgr2=0x7efc740d0810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:13.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.902+0000 7efc78a00640 1 --2- 172.21.15.6:0/2775045399 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efc740d03c0 0x7efc740d0810 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7efc6c009d90 tx=0x7efc6c02a6f0 comp rx=0 tx=0).stop 2024-04-24T23:16:13.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.901+0000 7f5992600640 1 -- 172.21.15.6:0/2090237320 wait complete. 2024-04-24T23:16:13.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.903+0000 7efc78a00640 1 -- 172.21.15.6:0/2775045399 shutdown_connections 2024-04-24T23:16:13.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.903+0000 7efc78a00640 1 --2- 172.21.15.6:0/2775045399 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efc740d03c0 0x7efc740d0810 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:13.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.903+0000 7efc78a00640 1 --2- 172.21.15.6:0/2775045399 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efc740b4bc0 0x7efc74162e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:13.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.903+0000 7efc78a00640 1 -- 172.21.15.6:0/2775045399 >> 172.21.15.6:0/2775045399 conn(0x7efc7406c990 msgr2=0x7efc7406cda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:13.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.902+0000 7f485b200640 1 -- 172.21.15.6:0/2434952094 wait complete. 2024-04-24T23:16:13.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.902+0000 7f0e83000640 1 -- 172.21.15.6:0/994627828 wait complete. 2024-04-24T23:16:13.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.903+0000 7fc4bca00640 1 -- 172.21.15.6:0/2181508344 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc4b80c8770 msgr2=0x7fc4b81f4b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:13.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.903+0000 7fc4bca00640 1 --2- 172.21.15.6:0/2181508344 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc4b80c8770 0x7fc4b81f4b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:13.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.903+0000 7fc4bca00640 1 -- 172.21.15.6:0/2181508344 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc4b80c8390 msgr2=0x7fc4b806e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:13.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.903+0000 7f4882a00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f487c0d03c0 0x7f487c0d0830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:13.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.903+0000 7f4881600640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f487c154d80 0x7f487c0c5310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.6:50660/0 (socket says 172.21.15.6:50660) 2024-04-24T23:16:13.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.904+0000 7f4882a00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f487c0cc720 con 0x7f487c0cdaa0 2024-04-24T23:16:13.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.904+0000 7f4882a00640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f487c0cca00 con 0x7f487c154d80 2024-04-24T23:16:13.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.904+0000 7f4882a00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f487c0cd610 con 0x7f487c0d03c0 2024-04-24T23:16:13.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.904+0000 7f5992600640 1 Processor -- start 2024-04-24T23:16:13.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.904+0000 7fc4bca00640 1 --2- 172.21.15.6:0/2181508344 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc4b80c8390 0x7fc4b806e7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:13.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.904+0000 7fc4bca00640 1 -- 172.21.15.6:0/2181508344 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4b81f8010 con 0x7fc4b8154cb0 2024-04-24T23:16:13.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.904+0000 7f4881600640 1 -- 172.21.15.6:0/2409701347 learned_addr learned my addr 172.21.15.6:0/2409701347 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:16:13.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.904+0000 7fc4b7e00640 1 --2- 172.21.15.6:0/2181508344 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc4b80c8390 0x7fc4b806e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:16:13.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.904+0000 7f485b200640 1 Processor -- start 2024-04-24T23:16:13.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.905+0000 7f485b200640 1 -- start start 2024-04-24T23:16:13.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.905+0000 7f0e83000640 1 Processor -- start 2024-04-24T23:16:13.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.905+0000 7fc4bca00640 1 --2- 172.21.15.6:0/2181508344 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc4b8154cb0 0x7fc4b81f5060 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fc4b80cce50 tx=0x7fc4a80046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:13.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.905+0000 7efc78a00640 1 -- 172.21.15.6:0/2775045399 shutdown_connections 2024-04-24T23:16:13.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.905+0000 7f487be00640 1 -- 172.21.15.6:0/2409701347 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3822272104 0 0) 0x7f487c0cca00 con 0x7f487c154d80 2024-04-24T23:16:13.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.905+0000 7f0e83000640 1 -- start start 2024-04-24T23:16:13.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.905+0000 7f487be00640 1 -- 172.21.15.6:0/2409701347 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4864003520 con 0x7f487c154d80 2024-04-24T23:16:13.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.905+0000 7f487be00640 1 -- 172.21.15.6:0/2409701347 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1132246982 0 0) 0x7f487c0cc720 con 0x7f487c0cdaa0 2024-04-24T23:16:13.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.905+0000 7f487be00640 1 -- 172.21.15.6:0/2409701347 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4864003880 con 0x7f487c0cdaa0 2024-04-24T23:16:13.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.905+0000 7f4882000640 1 --2- 172.21.15.6:0/2409701347 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f487c0d03c0 0x7f487c0d0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:13.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.905+0000 7f487be00640 1 -- 172.21.15.6:0/2409701347 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1266846456 0 0) 0x7f4864003520 con 0x7f487c154d80 2024-04-24T23:16:13.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.905+0000 7f487be00640 1 -- 172.21.15.6:0/2409701347 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f487c0d03c0 msgr2=0x7f487c0d0830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:16:13.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.905+0000 7f487be00640 1 --2- 172.21.15.6:0/2409701347 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f487c0d03c0 0x7f487c0d0830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:13.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.905+0000 7f487be00640 1 -- 172.21.15.6:0/2409701347 >> v1:172.21.15.6:6789/0 conn(0x7f487c0cdaa0 legacy=0x7f487c0cb260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:13.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.905+0000 7f487be00640 1 -- 172.21.15.6:0/2409701347 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f487c0ccce0 con 0x7f487c154d80 2024-04-24T23:16:13.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.905+0000 7f487be00640 1 -- 172.21.15.6:0/2409701347 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3463268701 0 0) 0x7f4870001950 con 0x7f487c154d80 2024-04-24T23:16:13.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.905+0000 7f487be00640 1 -- 172.21.15.6:0/2409701347 >> v1:172.21.15.190:6789/0 conn(0x7f487c154d80 legacy=0x7f487c0c5310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:13.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.905+0000 7f487be00640 1 --2- 172.21.15.6:0/2409701347 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4864003d30 0x7f4864004150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:13.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.905+0000 7f487be00640 1 --2- 172.21.15.6:0/2409701347 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4864004800 0x7f4864004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:13.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.905+0000 7f487be00640 1 --2- 172.21.15.6:0/2409701347 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f48640053f0 0x7f486400ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:13.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.905+0000 7f487be00640 1 -- 172.21.15.6:0/2409701347 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f487c0cd610 con 0x7f48640053f0 2024-04-24T23:16:13.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.905+0000 7f487be00640 1 -- 172.21.15.6:0/2409701347 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f486400c300 con 0x7f4864004800 2024-04-24T23:16:13.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.905+0000 7f487be00640 1 -- 172.21.15.6:0/2409701347 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f486400c500 con 0x7f4864003d30 2024-04-24T23:16:13.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.905+0000 7efc78a00640 1 -- 172.21.15.6:0/2775045399 wait complete. 2024-04-24T23:16:13.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.906+0000 7fc4afe00640 1 -- 172.21.15.6:0/2181508344 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4a8041020 con 0x7fc4b8154cb0 2024-04-24T23:16:13.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.906+0000 7fc4afe00640 1 -- 172.21.15.6:0/2181508344 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc4a80306f0 con 0x7fc4b8154cb0 2024-04-24T23:16:13.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.906+0000 7fc4afe00640 1 -- 172.21.15.6:0/2181508344 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4a8037040 con 0x7fc4b8154cb0 2024-04-24T23:16:13.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.906+0000 7fc4bd400640 1 -- 172.21.15.6:0/2181508344 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4b81f82f0 con 0x7fc4b8154cb0 2024-04-24T23:16:13.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.904+0000 7f5992600640 1 -- start start 2024-04-24T23:16:13.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.905+0000 7f5992600640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f598c0b35b0 0x7f598c14cf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:13.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.906+0000 7f5992600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f598c154dd0 0x7f598c14d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:13.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.906+0000 7f5992600640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f598c15b0b0 0x7f598c1e5930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:13.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.906+0000 7f5992600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f598c1551b0 con 0x7f598c154dd0 2024-04-24T23:16:13.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.906+0000 7f5992600640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f598c0b2e10 con 0x7f598c15b0b0 2024-04-24T23:16:13.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.906+0000 7f5992600640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f598c155330 con 0x7f598c0b35b0 2024-04-24T23:16:13.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.906+0000 7f598be00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f598c15b0b0 0x7f598c1e5930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:13.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.906+0000 7f4882000640 1 --2- 172.21.15.6:0/2409701347 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f487c0d03c0 0x7f487c0d0830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:16:13.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.906+0000 7f4881600640 1 --2- 172.21.15.6:0/2409701347 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4864003d30 0x7f4864004150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:13.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.906+0000 7f598be00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f598c15b0b0 0x7f598c1e5930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:59870/0 (socket says 172.21.15.6:59870) 2024-04-24T23:16:13.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.906+0000 7f598be00640 1 -- 172.21.15.6:0/2381668547 learned_addr learned my addr 172.21.15.6:0/2381668547 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:16:13.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.906+0000 7fc4bd400640 1 -- 172.21.15.6:0/2181508344 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4b81f88b0 con 0x7fc4b8154cb0 2024-04-24T23:16:13.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.907+0000 7f485b200640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f485404c210 0x7f4854048610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:13.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.907+0000 7f485b200640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4854051220 0x7f4854036b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:13.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.907+0000 7f485b200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f48540d5cc0 0x7f48540d0ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:13.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.907+0000 7f485b200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f4854046510 con 0x7f48540d5cc0 2024-04-24T23:16:13.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.907+0000 7f485b200640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f4854046230 con 0x7f4854051220 2024-04-24T23:16:13.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.907+0000 7f485b200640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f48540467f0 con 0x7f485404c210 2024-04-24T23:16:13.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.906+0000 7f598aa00640 1 --2- 172.21.15.6:0/2381668547 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f598c154dd0 0x7f598c14d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:13.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.907+0000 7f598be00640 1 -- 172.21.15.6:0/2381668547 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f598c0b35b0 msgr2=0x7f598c14cf80 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-24T23:16:13.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.907+0000 7f4882000640 1 --2- 172.21.15.6:0/2409701347 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f48640053f0 0x7f486400ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:13.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.907+0000 7f4880c00640 1 --2- 172.21.15.6:0/2409701347 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4864004800 0x7f4864004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:13.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.907+0000 7f4859e00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f485404c210 0x7f4854048610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:13.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.907+0000 7f4859e00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f485404c210 0x7f4854048610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.6:53242/0 (socket says 172.21.15.6:53242) 2024-04-24T23:16:13.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.907+0000 7f4859e00640 1 -- 172.21.15.6:0/2635184880 learned_addr learned my addr 172.21.15.6:0/2635184880 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:16:13.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.907+0000 7f0e83000640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0e7c06bc70 0x7f0e7c1f8820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:13.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.907+0000 7f598b400640 1 --2- 172.21.15.6:0/2381668547 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f598c0b35b0 0x7f598c14cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:13.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.907+0000 7f4882000640 1 -- 172.21.15.6:0/2409701347 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4864003d30 msgr2=0x7f4864004150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:16:13.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.907+0000 7f0e83000640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0e7c157d50 0x7f0e7c15bd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:13.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.908+0000 7f0e83000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0e7c15c270 0x7f0e7c150060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:13.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.908+0000 7f0e83000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f0e7c0cbdb0 con 0x7f0e7c15c270 2024-04-24T23:16:13.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.908+0000 7f0e83000640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f0e7c15ee00 con 0x7f0e7c06bc70 2024-04-24T23:16:13.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.908+0000 7f0e83000640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f0e7c15e160 con 0x7f0e7c157d50 2024-04-24T23:16:13.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.908+0000 7f0e7b400640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0e7c157d50 0x7f0e7c15bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:13.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.907+0000 7f598be00640 1 --2- 172.21.15.6:0/2381668547 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f598c0b35b0 0x7f598c14cf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:13.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.907+0000 7f598be00640 1 -- 172.21.15.6:0/2381668547 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f598c154dd0 msgr2=0x7f598c14d4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:13.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.907+0000 7f598be00640 1 --2- 172.21.15.6:0/2381668547 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f598c154dd0 0x7f598c14d4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:13.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.907+0000 7f598be00640 1 -- 172.21.15.6:0/2381668547 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f598c1e3eb0 con 0x7f598c15b0b0 2024-04-24T23:16:13.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.908+0000 7f0e7b400640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0e7c157d50 0x7f0e7c15bd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.6:53254/0 (socket says 172.21.15.6:53254) 2024-04-24T23:16:13.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.908+0000 7f0e7b400640 1 -- 172.21.15.6:0/1994744847 learned_addr learned my addr 172.21.15.6:0/1994744847 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:16:13.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.908+0000 7f0e7be00640 1 --2- 172.21.15.6:0/1994744847 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0e7c06bc70 0x7f0e7c1f8820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:13.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.908+0000 7f0e81000640 1 --2- 172.21.15.6:0/1994744847 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0e7c15c270 0x7f0e7c150060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:13.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.908+0000 7f598aa00640 1 --2- 172.21.15.6:0/2381668547 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f598c154dd0 0x7f598c14d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:16:13.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.908+0000 7f598b400640 1 --2- 172.21.15.6:0/2381668547 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f598c0b35b0 0x7f598c14cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:16:13.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.908+0000 7f4859400640 1 --2- 172.21.15.6:0/2635184880 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4854051220 0x7f4854036b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:13.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.909+0000 7f598be00640 1 --2- 172.21.15.6:0/2381668547 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f598c15b0b0 0x7f598c1e5930 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f5984009f70 tx=0x7f5984004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:13.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.909+0000 7f485a800640 1 --2- 172.21.15.6:0/2635184880 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f48540d5cc0 0x7f48540d0ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:13.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.909+0000 7f0e7be00640 1 -- 172.21.15.6:0/1994744847 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0e7c157d50 msgr2=0x7f0e7c15bd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:13.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.908+0000 7efc78a00640 1 Processor -- start 2024-04-24T23:16:13.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.909+0000 7efc78a00640 1 -- start start 2024-04-24T23:16:13.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.909+0000 7f4859e00640 1 -- 172.21.15.6:0/2635184880 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4854051220 msgr2=0x7f4854036b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:13.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.909+0000 7f4859e00640 1 --2- 172.21.15.6:0/2635184880 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4854051220 0x7f4854036b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:13.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.909+0000 7f4859e00640 1 -- 172.21.15.6:0/2635184880 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f48540d5cc0 msgr2=0x7f48540d0ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:13.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.909+0000 7f0e7be00640 1 --2- 172.21.15.6:0/1994744847 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0e7c157d50 0x7f0e7c15bd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:13.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.909+0000 7f0e7be00640 1 -- 172.21.15.6:0/1994744847 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0e7c15c270 msgr2=0x7f0e7c150060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:13.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.909+0000 7f0e7be00640 1 --2- 172.21.15.6:0/1994744847 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0e7c15c270 0x7f0e7c150060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:13.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.909+0000 7f0e7be00640 1 -- 172.21.15.6:0/1994744847 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e7c15c810 con 0x7f0e7c06bc70 2024-04-24T23:16:13.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.909+0000 7f5983400640 1 -- 172.21.15.6:0/2381668547 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5984041020 con 0x7f598c15b0b0 2024-04-24T23:16:13.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.910+0000 7f5983400640 1 -- 172.21.15.6:0/2381668547 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5984007780 con 0x7f598c15b0b0 2024-04-24T23:16:13.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.910+0000 7f5983400640 1 -- 172.21.15.6:0/2381668547 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5984036040 con 0x7f598c15b0b0 2024-04-24T23:16:13.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.909+0000 7f0e81000640 1 --2- 172.21.15.6:0/1994744847 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0e7c15c270 0x7f0e7c150060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:16:13.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.909+0000 7f4859e00640 1 --2- 172.21.15.6:0/2635184880 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f48540d5cc0 0x7f48540d0ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:13.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.910+0000 7f0e7be00640 1 --2- 172.21.15.6:0/1994744847 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0e7c06bc70 0x7f0e7c1f8820 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f0e6c011590 tx=0x7f0e6c011a50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:13.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.910+0000 7f4859e00640 1 -- 172.21.15.6:0/2635184880 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48540db030 con 0x7f485404c210 2024-04-24T23:16:13.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.910+0000 7f485a800640 1 --2- 172.21.15.6:0/2635184880 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f48540d5cc0 0x7f48540d0ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:16:13.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.907+0000 7f4882000640 1 --2- 172.21.15.6:0/2409701347 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4864003d30 0x7f4864004150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:13.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.907+0000 7f4882000640 1 -- 172.21.15.6:0/2409701347 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4864004800 msgr2=0x7f4864004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:16:13.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.907+0000 7f4882000640 1 --2- 172.21.15.6:0/2409701347 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4864004800 0x7f4864004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:13.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.907+0000 7f4882000640 1 -- 172.21.15.6:0/2409701347 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f487c0ccce0 con 0x7f48640053f0 2024-04-24T23:16:13.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.908+0000 7f4881600640 1 --2- 172.21.15.6:0/2409701347 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4864003d30 0x7f4864004150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:16:13.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.909+0000 7f4882000640 1 --2- 172.21.15.6:0/2409701347 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f48640053f0 0x7f486400ba20 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f487c0d07a0 tx=0x7f4864005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac5a071ce3aca35d server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:13.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.910+0000 7f0e73e00640 1 -- 172.21.15.6:0/1994744847 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e6c019070 con 0x7f0e7c06bc70 2024-04-24T23:16:13.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.910+0000 7f4859e00640 1 --2- 172.21.15.6:0/2635184880 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f485404c210 0x7f4854048610 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f484c00d830 tx=0x7f484c00dcf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:13.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.910+0000 7f0e73e00640 1 -- 172.21.15.6:0/1994744847 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0e6c003610 con 0x7f0e7c06bc70 2024-04-24T23:16:13.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.910+0000 7f0e73e00640 1 -- 172.21.15.6:0/1994744847 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e6c0039c0 con 0x7f0e7c06bc70 2024-04-24T23:16:13.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.910+0000 7f0e83000640 1 -- 172.21.15.6:0/1994744847 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e7c1f4940 con 0x7f0e7c06bc70 2024-04-24T23:16:13.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.911+0000 7f4852000640 1 -- 172.21.15.6:0/2635184880 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f484c00b6c0 con 0x7f485404c210 2024-04-24T23:16:13.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.911+0000 7f4852000640 1 -- 172.21.15.6:0/2635184880 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f484c019070 con 0x7f485404c210 2024-04-24T23:16:13.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.911+0000 7f4852000640 1 -- 172.21.15.6:0/2635184880 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f484c00ba60 con 0x7f485404c210 2024-04-24T23:16:13.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.911+0000 7f485b200640 1 -- 172.21.15.6:0/2635184880 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48540db310 con 0x7f485404c210 2024-04-24T23:16:13.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.911+0000 7f485b200640 1 -- 172.21.15.6:0/2635184880 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f4854036340 con 0x7f485404c210 2024-04-24T23:16:13.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.910+0000 7f0e83000640 1 -- 172.21.15.6:0/1994744847 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e7c1f4f00 con 0x7f0e7c06bc70 2024-04-24T23:16:13.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.911+0000 7efc78a00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efc740b4bc0 0x7efc74071310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:13.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.912+0000 7f5992600640 1 -- 172.21.15.6:0/2381668547 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f598c15dc80 con 0x7f598c15b0b0 2024-04-24T23:16:13.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.910+0000 7fc4bd400640 1 -- 172.21.15.6:0/2181508344 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fc490001030 con 0x7fc4b8154cb0 2024-04-24T23:16:13.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.912+0000 7f485b200640 1 -- 172.21.15.6:0/2635184880 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f48540052c0 con 0x7f485404c210 2024-04-24T23:16:13.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.913+0000 7fc4afe00640 1 -- 172.21.15.6:0/2181508344 <== mon.2 v2:172.21.15.190:3301/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fc4a8044020 con 0x7fc4b8154cb0 2024-04-24T23:16:13.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.913+0000 7f0e83000640 1 -- 172.21.15.6:0/1994744847 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f0e7c075c10 con 0x7f0e7c06bc70 2024-04-24T23:16:13.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.913+0000 7f487be00640 1 -- 172.21.15.6:0/2409701347 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4874037070 con 0x7f48640053f0 2024-04-24T23:16:13.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.913+0000 7f487be00640 1 -- 172.21.15.6:0/2409701347 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f487402aa50 con 0x7f48640053f0 2024-04-24T23:16:13.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.913+0000 7f0e73e00640 1 -- 172.21.15.6:0/1994744847 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f0e6c01c070 con 0x7f0e7c06bc70 2024-04-24T23:16:13.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.913+0000 7efc78a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efc740d03c0 0x7efc74071850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:13.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.913+0000 7efc78a00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efc7415fa00 0x7efc740c98a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:13.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.913+0000 7f5992600640 1 -- 172.21.15.6:0/2381668547 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f598c14a120 con 0x7f598c15b0b0 2024-04-24T23:16:13.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.914+0000 7f4882a00640 1 -- 172.21.15.6:0/2409701347 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f48640053f0 msgr2=0x7f486400ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:13.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.914+0000 7f4882a00640 1 --2- 172.21.15.6:0/2409701347 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f48640053f0 0x7f486400ba20 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f487c0d07a0 tx=0x7f4864005830 comp rx=0 tx=0).stop 2024-04-24T23:16:13.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.913+0000 7efc72800640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efc740b4bc0 0x7efc74071310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:13.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.914+0000 7efc73200640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efc7415fa00 0x7efc740c98a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:13.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.914+0000 7efc73200640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efc7415fa00 0x7efc740c98a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.6:53260/0 (socket says 172.21.15.6:53260) 2024-04-24T23:16:13.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.914+0000 7efc73200640 1 -- 172.21.15.6:0/1085726918 learned_addr learned my addr 172.21.15.6:0/1085726918 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:16:13.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.914+0000 7f4882000640 1 -- 172.21.15.6:0/2409701347 reap_dead start 2024-04-24T23:16:13.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.914+0000 7fc4afe00640 1 --2- 172.21.15.6:0/2181508344 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fc49806be80 0x7fc49806e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:13.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.914+0000 7f4882a00640 1 -- 172.21.15.6:0/2409701347 shutdown_connections 2024-04-24T23:16:13.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.914+0000 7f4882a00640 1 -- 172.21.15.6:0/2409701347 >> 172.21.15.6:0/2409701347 conn(0x7f487c06c990 msgr2=0x7f487c06cda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:13.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.914+0000 7efc78a00640 1 -- 172.21.15.6:0/1085726918 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7efc740b5980 con 0x7efc740d03c0 2024-04-24T23:16:13.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.914+0000 7fc4afe00640 1 -- 172.21.15.6:0/2181508344 <== mon.2 v2:172.21.15.190:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fc4a80abba0 con 0x7fc4b8154cb0 2024-04-24T23:16:13.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.914+0000 7f4882a00640 1 -- 172.21.15.6:0/2409701347 shutdown_connections 2024-04-24T23:16:13.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.914+0000 7fc4b7e00640 1 --2- 172.21.15.6:0/2181508344 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fc49806be80 0x7fc49806e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:13.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.914+0000 7efc78a00640 1 -- 172.21.15.6:0/1085726918 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7efc740b3610 con 0x7efc740b4bc0 2024-04-24T23:16:13.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.914+0000 7efc78a00640 1 -- 172.21.15.6:0/1085726918 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7efc741633c0 con 0x7efc7415fa00 2024-04-24T23:16:13.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.915+0000 7f4882a00640 1 -- 172.21.15.6:0/2409701347 wait complete. 2024-04-24T23:16:13.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.915+0000 7fc4afe00640 1 --2- 172.21.15.6:0/2181508344 >> [v2:172.21.15.6:6800/1020213724,v1:172.21.15.6:6801/1020213724] conn(0x7fc4980755f0 0x7fc498077a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:13.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.915+0000 7fc4afe00640 1 -- 172.21.15.6:0/2181508344 --> [v2:172.21.15.6:6800/1020213724,v1:172.21.15.6:6801/1020213724] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc4a8032640 con 0x7fc4980755f0 2024-04-24T23:16:13.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.915+0000 7efc72800640 1 -- 172.21.15.6:0/1085726918 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efc7415fa00 msgr2=0x7efc740c98a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:13.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.915+0000 7efc72800640 1 --2- 172.21.15.6:0/1085726918 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efc7415fa00 0x7efc740c98a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:13.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.915+0000 7efc72800640 1 -- 172.21.15.6:0/1085726918 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efc740d03c0 msgr2=0x7efc74071850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:13.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.915+0000 7fc4afe00640 1 -- 172.21.15.6:0/2181508344 <== mon.2 v2:172.21.15.190:3301/0 6 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fc4a80ac080 con 0x7fc4b8154cb0 2024-04-24T23:16:13.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.915+0000 7efc71e00640 1 --2- 172.21.15.6:0/1085726918 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efc740d03c0 0x7efc74071850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:13.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.915+0000 7efc72800640 1 --2- 172.21.15.6:0/1085726918 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efc740d03c0 0x7efc74071850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:13.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.915+0000 7efc72800640 1 -- 172.21.15.6:0/1085726918 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc740cf4d0 con 0x7efc740b4bc0 2024-04-24T23:16:13.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.915+0000 7efc71e00640 1 --2- 172.21.15.6:0/1085726918 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efc740d03c0 0x7efc74071850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:16:13.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.915+0000 7fc4b7400640 1 --2- 172.21.15.6:0/2181508344 >> [v2:172.21.15.6:6800/1020213724,v1:172.21.15.6:6801/1020213724] conn(0x7fc4980755f0 0x7fc498077a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:13.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.915+0000 7fc4b7e00640 1 --2- 172.21.15.6:0/2181508344 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fc49806be80 0x7fc49806e340 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fc4b0007fd0 tx=0x7fc4b0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:13.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.915+0000 7f0e73e00640 1 --2- 172.21.15.6:0/1994744847 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f0e5c06bf30 0x7f0e5c06e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:13.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.916+0000 7f0e7b400640 1 --2- 172.21.15.6:0/1994744847 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f0e5c06bf30 0x7f0e5c06e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:13.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.916+0000 7f0e73e00640 1 -- 172.21.15.6:0/1994744847 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f0e6c08b4d0 con 0x7f0e7c06bc70 2024-04-24T23:16:13.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.916+0000 7f5992600640 1 -- 172.21.15.6:0/2381668547 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f598c0767b0 con 0x7f598c15b0b0 2024-04-24T23:16:13.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.916+0000 7efc72800640 1 --2- 172.21.15.6:0/1085726918 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efc740b4bc0 0x7efc74071310 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7efc6400c420 tx=0x7efc64001e70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:13.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.916+0000 7f4852000640 1 -- 172.21.15.6:0/2635184880 <== mon.2 v2:172.21.15.190:3301/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f484c03b480 con 0x7f485404c210 2024-04-24T23:16:13.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.916+0000 7fc4b7400640 1 --2- 172.21.15.6:0/2181508344 >> [v2:172.21.15.6:6800/1020213724,v1:172.21.15.6:6801/1020213724] conn(0x7fc4980755f0 0x7fc498077a30 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:13.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.916+0000 7efc6aa00640 1 -- 172.21.15.6:0/1085726918 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc64017070 con 0x7efc740b4bc0 2024-04-24T23:16:13.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.916+0000 7efc6aa00640 1 -- 172.21.15.6:0/1085726918 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efc64005ce0 con 0x7efc740b4bc0 2024-04-24T23:16:13.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.916+0000 7efc6aa00640 1 -- 172.21.15.6:0/1085726918 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc640072d0 con 0x7efc740b4bc0 2024-04-24T23:16:13.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.916+0000 7efc78a00640 1 -- 172.21.15.6:0/1085726918 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc740b3030 con 0x7efc740b4bc0 2024-04-24T23:16:13.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.916+0000 7f0e73e00640 1 --2- 172.21.15.6:0/1994744847 >> [v2:172.21.15.190:6808/1989500934,v1:172.21.15.190:6809/1989500934] conn(0x7f0e5c0756a0 0x7f0e5c077ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:13.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.916+0000 7f0e73e00640 1 -- 172.21.15.6:0/1994744847 --> [v2:172.21.15.190:6808/1989500934,v1:172.21.15.190:6809/1989500934] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0e5c0781b0 con 0x7f0e5c0756a0 2024-04-24T23:16:13.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.916+0000 7f0e73e00640 1 -- 172.21.15.6:0/1994744847 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f0e6c092050 con 0x7f0e7c06bc70 2024-04-24T23:16:13.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.916+0000 7efc78a00640 1 -- 172.21.15.6:0/1085726918 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc741f9780 con 0x7efc740b4bc0 2024-04-24T23:16:13.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.917+0000 7f0e81000640 1 --2- 172.21.15.6:0/1994744847 >> [v2:172.21.15.190:6808/1989500934,v1:172.21.15.190:6809/1989500934] conn(0x7f0e5c0756a0 0x7f0e5c077ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:13.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.918+0000 7efc78a00640 1 -- 172.21.15.6:0/1085726918 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7efc74075ba0 con 0x7efc740b4bc0 2024-04-24T23:16:13.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.918+0000 7fc4afe00640 1 -- 172.21.15.6:0/2181508344 <== osd.2 v2:172.21.15.6:6800/1020213724 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc4a8032640 con 0x7fc4980755f0 2024-04-24T23:16:13.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.918+0000 7f4882a00640 1 Processor -- start 2024-04-24T23:16:13.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.918+0000 7f0e7b400640 1 --2- 172.21.15.6:0/1994744847 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f0e5c06bf30 0x7f0e5c06e3f0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f0e7c158f00 tx=0x7f0e74007000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:13.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.918+0000 7f4852000640 1 --2- 172.21.15.6:0/2635184880 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f483c06be80 0x7f483c06e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:13.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.919+0000 7f4852000640 1 -- 172.21.15.6:0/2635184880 <== mon.2 v2:172.21.15.190:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f484c08b8f0 con 0x7f485404c210 2024-04-24T23:16:13.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.919+0000 7f4852000640 1 --2- 172.21.15.6:0/2635184880 >> [v2:172.21.15.6:6824/3163883468,v1:172.21.15.6:6825/3163883468] conn(0x7f483c0755f0 0x7f483c077a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:13.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.919+0000 7f4852000640 1 -- 172.21.15.6:0/2635184880 --> [v2:172.21.15.6:6824/3163883468,v1:172.21.15.6:6825/3163883468] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f483c078100 con 0x7f483c0755f0 2024-04-24T23:16:13.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.919+0000 7f4852000640 1 -- 172.21.15.6:0/2635184880 <== mon.2 v2:172.21.15.190:3301/0 6 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f484c08bdd0 con 0x7f485404c210 2024-04-24T23:16:13.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.920+0000 7f4859400640 1 --2- 172.21.15.6:0/2635184880 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f483c06be80 0x7f483c06e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:13.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.920+0000 7f485a800640 1 --2- 172.21.15.6:0/2635184880 >> [v2:172.21.15.6:6824/3163883468,v1:172.21.15.6:6825/3163883468] conn(0x7f483c0755f0 0x7f483c077a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:13.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.918+0000 7f5983400640 1 -- 172.21.15.6:0/2381668547 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f5984044020 con 0x7f598c15b0b0 2024-04-24T23:16:13.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.919+0000 7f5983400640 1 --2- 172.21.15.6:0/2381668547 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f596c06be80 0x7f596c06e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:13.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.919+0000 7f5983400640 1 -- 172.21.15.6:0/2381668547 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f59840abb40 con 0x7f598c15b0b0 2024-04-24T23:16:13.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.920+0000 7f5983400640 1 --2- 172.21.15.6:0/2381668547 >> [v2:172.21.15.190:6800/3687548412,v1:172.21.15.190:6801/3687548412] conn(0x7f596c0755f0 0x7f596c077a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:13.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.920+0000 7f5983400640 1 -- 172.21.15.6:0/2381668547 --> [v2:172.21.15.190:6800/3687548412,v1:172.21.15.190:6801/3687548412] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5984008e50 con 0x7f596c0755f0 2024-04-24T23:16:13.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.920+0000 7f5983400640 1 -- 172.21.15.6:0/2381668547 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f59840ae290 con 0x7f598c15b0b0 2024-04-24T23:16:13.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.920+0000 7f598b400640 1 --2- 172.21.15.6:0/2381668547 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f596c06be80 0x7f596c06e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:13.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.920+0000 7f598aa00640 1 --2- 172.21.15.6:0/2381668547 >> [v2:172.21.15.190:6800/3687548412,v1:172.21.15.190:6801/3687548412] conn(0x7f596c0755f0 0x7f596c077a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:13.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.918+0000 7f4882a00640 1 -- start start 2024-04-24T23:16:13.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.920+0000 7f4882a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f48640053f0 0x7f487c15c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:13.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.920+0000 7f4882a00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f487c15e160 0x7f487c1f1360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:13.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.920+0000 7f4882a00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f487c0c87c0 0x7f487c151310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:13.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.921+0000 7f4882a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f487c15bb00 con 0x7f48640053f0 2024-04-24T23:16:13.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.921+0000 7f0e81000640 1 --2- 172.21.15.6:0/1994744847 >> [v2:172.21.15.190:6808/1989500934,v1:172.21.15.190:6809/1989500934] conn(0x7f0e5c0756a0 0x7f0e5c077ae0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:13.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.921+0000 7efc6aa00640 1 -- 172.21.15.6:0/1085726918 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7efc6401a070 con 0x7efc740b4bc0 2024-04-24T23:16:13.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.921+0000 7f4882000640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f487c0c87c0 0x7f487c151310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:13.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.921+0000 7f4882000640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f487c0c87c0 0x7f487c151310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.6:53270/0 (socket says 172.21.15.6:53270) 2024-04-24T23:16:13.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.921+0000 7f4882000640 1 -- 172.21.15.6:0/1975709464 learned_addr learned my addr 172.21.15.6:0/1975709464 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:16:13.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.921+0000 7f4882a00640 1 -- 172.21.15.6:0/1975709464 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f487c0cd2a0 con 0x7f487c15e160 2024-04-24T23:16:13.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.921+0000 7f4880c00640 1 --2- 172.21.15.6:0/1975709464 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f487c15e160 0x7f487c1f1360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:13.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.921+0000 7f485a800640 1 --2- 172.21.15.6:0/2635184880 >> [v2:172.21.15.6:6824/3163883468,v1:172.21.15.6:6825/3163883468] conn(0x7f483c0755f0 0x7f483c077a30 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:13.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.921+0000 7f598b400640 1 --2- 172.21.15.6:0/2381668547 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f596c06be80 0x7f596c06e340 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f597c003eb0 tx=0x7f597c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:13.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.921+0000 7f4881600640 1 --2- 172.21.15.6:0/1975709464 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f48640053f0 0x7f487c15c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:13.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.921+0000 7f4882a00640 1 -- 172.21.15.6:0/1975709464 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f487c0cb970 con 0x7f487c0c87c0 2024-04-24T23:16:13.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.921+0000 7f598aa00640 1 --2- 172.21.15.6:0/2381668547 >> [v2:172.21.15.190:6800/3687548412,v1:172.21.15.190:6801/3687548412] conn(0x7f596c0755f0 0x7f596c077a30 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:13.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.921+0000 7f4859400640 1 --2- 172.21.15.6:0/2635184880 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f483c06be80 0x7f483c06e340 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f4848034000 tx=0x7f484802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:13.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.922+0000 7f0e73e00640 1 -- 172.21.15.6:0/1994744847 <== osd.4 v2:172.21.15.190:6808/1989500934 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f0e5c0781b0 con 0x7f0e5c0756a0 2024-04-24T23:16:13.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.922+0000 7f4882000640 1 -- 172.21.15.6:0/1975709464 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f487c15e160 msgr2=0x7f487c1f1360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:13.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.922+0000 7f4882000640 1 --2- 172.21.15.6:0/1975709464 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f487c15e160 0x7f487c1f1360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:13.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.922+0000 7f4882000640 1 -- 172.21.15.6:0/1975709464 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f48640053f0 msgr2=0x7f487c15c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:13.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.922+0000 7f4882000640 1 --2- 172.21.15.6:0/1975709464 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f48640053f0 0x7f487c15c7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:13.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.922+0000 7f4882000640 1 -- 172.21.15.6:0/1975709464 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f487c1f08e0 con 0x7f487c0c87c0 2024-04-24T23:16:13.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.922+0000 7f4881600640 1 --2- 172.21.15.6:0/1975709464 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f48640053f0 0x7f487c15c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:16:13.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.923+0000 7f4852000640 1 -- 172.21.15.6:0/2635184880 <== osd.3 v2:172.21.15.6:6824/3163883468 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f483c078100 con 0x7f483c0755f0 2024-04-24T23:16:13.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.923+0000 7f4882000640 1 --2- 172.21.15.6:0/1975709464 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f487c0c87c0 0x7f487c151310 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f487400a920 tx=0x7f4874004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:13.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.923+0000 7efc6aa00640 1 --2- 172.21.15.6:0/1085726918 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7efc5406bf00 0x7efc5406e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:13.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.923+0000 7efc6aa00640 1 -- 172.21.15.6:0/1085726918 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7efc6408a6f0 con 0x7efc740b4bc0 2024-04-24T23:16:13.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.923+0000 7efc6aa00640 1 --2- 172.21.15.6:0/1085726918 >> [v2:172.21.15.190:6816/870571564,v1:172.21.15.190:6817/870571564] conn(0x7efc54075670 0x7efc54077ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:13.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.923+0000 7efc6aa00640 1 -- 172.21.15.6:0/1085726918 --> [v2:172.21.15.190:6816/870571564,v1:172.21.15.190:6817/870571564] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7efc6400de50 con 0x7efc54075670 2024-04-24T23:16:13.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.923+0000 7efc6aa00640 1 -- 172.21.15.6:0/1085726918 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7efc6408aae0 con 0x7efc740b4bc0 2024-04-24T23:16:13.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.923+0000 7f4879600640 1 -- 172.21.15.6:0/1975709464 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4874037040 con 0x7f487c0c87c0 2024-04-24T23:16:13.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.923+0000 7efc73200640 1 --2- 172.21.15.6:0/1085726918 >> [v2:172.21.15.190:6816/870571564,v1:172.21.15.190:6817/870571564] conn(0x7efc54075670 0x7efc54077ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:13.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.923+0000 7f4879600640 1 -- 172.21.15.6:0/1975709464 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4874033e20 con 0x7f487c0c87c0 2024-04-24T23:16:13.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.924+0000 7f4882a00640 1 -- 172.21.15.6:0/1975709464 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f487c15e670 con 0x7f487c0c87c0 2024-04-24T23:16:13.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.924+0000 7efc71e00640 1 --2- 172.21.15.6:0/1085726918 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7efc5406bf00 0x7efc5406e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:13.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.924+0000 7f4879600640 1 -- 172.21.15.6:0/1975709464 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f487000a330 con 0x7f487c0c87c0 2024-04-24T23:16:13.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.924+0000 7f4882a00640 1 -- 172.21.15.6:0/1975709464 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f487c15ec30 con 0x7f487c0c87c0 2024-04-24T23:16:13.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.924+0000 7efc73200640 1 --2- 172.21.15.6:0/1085726918 >> [v2:172.21.15.190:6816/870571564,v1:172.21.15.190:6817/870571564] conn(0x7efc54075670 0x7efc54077ab0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:13.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.925+0000 7efc71e00640 1 --2- 172.21.15.6:0/1085726918 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7efc5406bf00 0x7efc5406e3c0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7efc740c8930 tx=0x7efc60008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:13.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.925+0000 7efc6aa00640 1 -- 172.21.15.6:0/1085726918 <== osd.6 v2:172.21.15.190:6816/870571564 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7efc6400de50 con 0x7efc54075670 2024-04-24T23:16:13.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.925+0000 7f4882a00640 1 -- 172.21.15.6:0/1975709464 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f4858001030 con 0x7f487c0c87c0 2024-04-24T23:16:13.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.927+0000 7f4879600640 1 -- 172.21.15.6:0/1975709464 <== mon.2 v2:172.21.15.190:3301/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f4874046020 con 0x7f487c0c87c0 2024-04-24T23:16:13.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.929+0000 7f4879600640 1 --2- 172.21.15.6:0/1975709464 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f485c06bf30 0x7f485c06e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:13.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.929+0000 7f4879600640 1 -- 172.21.15.6:0/1975709464 <== mon.2 v2:172.21.15.190:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f48740aad70 con 0x7f487c0c87c0 2024-04-24T23:16:13.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.929+0000 7f4879600640 1 --2- 172.21.15.6:0/1975709464 >> [v2:172.21.15.190:6824/123639197,v1:172.21.15.190:6825/123639197] conn(0x7f485c0756a0 0x7f485c077ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:13.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.929+0000 7f4879600640 1 -- 172.21.15.6:0/1975709464 --> [v2:172.21.15.190:6824/123639197,v1:172.21.15.190:6825/123639197] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f487404ae80 con 0x7f485c0756a0 2024-04-24T23:16:13.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.929+0000 7f4879600640 1 -- 172.21.15.6:0/1975709464 <== mon.2 v2:172.21.15.190:3301/0 6 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f48740ab160 con 0x7f487c0c87c0 2024-04-24T23:16:13.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.930+0000 7f4881600640 1 --2- 172.21.15.6:0/1975709464 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f485c06bf30 0x7f485c06e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:13.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.930+0000 7f4880c00640 1 --2- 172.21.15.6:0/1975709464 >> [v2:172.21.15.190:6824/123639197,v1:172.21.15.190:6825/123639197] conn(0x7f485c0756a0 0x7f485c077ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:13.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.931+0000 7f4881600640 1 --2- 172.21.15.6:0/1975709464 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f485c06bf30 0x7f485c06e3f0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f4870000f80 tx=0x7f4870006040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:13.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.932+0000 7f4880c00640 1 --2- 172.21.15.6:0/1975709464 >> [v2:172.21.15.190:6824/123639197,v1:172.21.15.190:6825/123639197] conn(0x7f485c0756a0 0x7f485c077ae0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:13.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.932+0000 7f4879600640 1 -- 172.21.15.6:0/1975709464 <== osd.7 v2:172.21.15.190:6824/123639197 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f487404ae80 con 0x7f485c0756a0 2024-04-24T23:16:13.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.935+0000 7f6bfbc00640 1 -- 172.21.15.6:0/1232895690 --> [v2:172.21.15.6:6808/3022569437,v1:172.21.15.6:6809/3022569437] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f6bd0002dd0 con 0x7f6bd40756b0 2024-04-24T23:16:13.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.938+0000 7fa436600640 1 -- 172.21.15.6:0/2188732461 --> [v2:172.21.15.6:6816/3687836888,v1:172.21.15.6:6817/3687836888] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fa408002dd0 con 0x7fa40c075680 2024-04-24T23:16:13.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.938+0000 7f6bf0c00640 1 -- 172.21.15.6:0/1232895690 <== osd.0 v2:172.21.15.6:6808/3022569437 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f6bd0002dd0 con 0x7f6bd40756b0 2024-04-24T23:16:13.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.938+0000 7fa427400640 1 -- 172.21.15.6:0/2188732461 <== osd.1 v2:172.21.15.6:6816/3687836888 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fa408002dd0 con 0x7fa40c075680 2024-04-24T23:16:13.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.939+0000 7f6bfbc00640 1 -- 172.21.15.6:0/1232895690 >> [v2:172.21.15.6:6808/3022569437,v1:172.21.15.6:6809/3022569437] conn(0x7f6bd40756b0 msgr2=0x7f6bd4077af0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:13.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.939+0000 7f6bfbc00640 1 --2- 172.21.15.6:0/1232895690 >> [v2:172.21.15.6:6808/3022569437,v1:172.21.15.6:6809/3022569437] conn(0x7f6bd40756b0 0x7f6bd4077af0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:13.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.939+0000 7f6bfbc00640 1 -- 172.21.15.6:0/1232895690 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f6bd406bf30 msgr2=0x7f6bd406e3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:13.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.939+0000 7f6bfbc00640 1 --2- 172.21.15.6:0/1232895690 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f6bd406bf30 0x7f6bd406e3f0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f6bec0e3310 tx=0x7f6be0007e30 comp rx=0 tx=0).stop 2024-04-24T23:16:13.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.939+0000 7fa436600640 1 -- 172.21.15.6:0/2188732461 >> [v2:172.21.15.6:6816/3687836888,v1:172.21.15.6:6817/3687836888] conn(0x7fa40c075680 msgr2=0x7fa40c077ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:13.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.939+0000 7fa436600640 1 --2- 172.21.15.6:0/2188732461 >> [v2:172.21.15.6:6816/3687836888,v1:172.21.15.6:6817/3687836888] conn(0x7fa40c075680 0x7fa40c077ac0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:13.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.939+0000 7f6bfbc00640 1 -- 172.21.15.6:0/1232895690 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6bec049420 msgr2=0x7f6bec1767b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:13.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.939+0000 7f6bfbc00640 1 --2- 172.21.15.6:0/1232895690 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6bec049420 0x7f6bec1767b0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f6bf4074280 tx=0x7f6bf4076b80 comp rx=0 tx=0).stop 2024-04-24T23:16:13.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.940+0000 7fa436600640 1 -- 172.21.15.6:0/2188732461 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fa40c06bf00 msgr2=0x7fa40c06e3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:13.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.940+0000 7fa436600640 1 --2- 172.21.15.6:0/2188732461 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fa40c06bf00 0x7fa40c06e3c0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fa4300cb2a0 tx=0x7fa420001000 comp rx=0 tx=0).stop 2024-04-24T23:16:13.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.940+0000 7fa436600640 1 -- 172.21.15.6:0/2188732461 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa4300c7810 msgr2=0x7fa4301ec600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:13.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.940+0000 7fa436600640 1 --2- 172.21.15.6:0/2188732461 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa4300c7810 0x7fa4301ec600 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fa43006bf80 tx=0x7fa4180046c0 comp rx=0 tx=0).stop 2024-04-24T23:16:13.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.940+0000 7fa42fe00640 1 -- 172.21.15.6:0/2188732461 reap_dead start 2024-04-24T23:16:13.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.940+0000 7f6bf9c00640 1 -- 172.21.15.6:0/1232895690 reap_dead start 2024-04-24T23:16:13.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.940+0000 7fa436600640 1 -- 172.21.15.6:0/2188732461 shutdown_connections 2024-04-24T23:16:13.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.940+0000 7fa436600640 1 -- 172.21.15.6:0/2188732461 >> 172.21.15.6:0/2188732461 conn(0x7fa4300c8c50 msgr2=0x7fa43014f710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:13.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.940+0000 7f6bfbc00640 1 -- 172.21.15.6:0/1232895690 shutdown_connections 2024-04-24T23:16:13.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.940+0000 7f6bfbc00640 1 -- 172.21.15.6:0/1232895690 >> 172.21.15.6:0/1232895690 conn(0x7f6bec054580 msgr2=0x7f6bec0d3d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:13.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.940+0000 7f6bfbc00640 1 -- 172.21.15.6:0/1232895690 shutdown_connections 2024-04-24T23:16:13.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.940+0000 7f6bfbc00640 1 -- 172.21.15.6:0/1232895690 wait complete. 2024-04-24T23:16:13.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.941+0000 7fa436600640 1 -- 172.21.15.6:0/2188732461 shutdown_connections 2024-04-24T23:16:13.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.941+0000 7fa436600640 1 -- 172.21.15.6:0/2188732461 wait complete. 2024-04-24T23:16:13.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.948+0000 7fc4bd400640 1 -- 172.21.15.6:0/2181508344 --> [v2:172.21.15.6:6800/1020213724,v1:172.21.15.6:6801/1020213724] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fc490002dd0 con 0x7fc4980755f0 2024-04-24T23:16:13.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.949+0000 7fc4afe00640 1 -- 172.21.15.6:0/2181508344 <== osd.2 v2:172.21.15.6:6800/1020213724 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fc490002dd0 con 0x7fc4980755f0 2024-04-24T23:16:13.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.950+0000 7fc4bd400640 1 -- 172.21.15.6:0/2181508344 >> [v2:172.21.15.6:6800/1020213724,v1:172.21.15.6:6801/1020213724] conn(0x7fc4980755f0 msgr2=0x7fc498077a30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:13.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.950+0000 7fc4bd400640 1 --2- 172.21.15.6:0/2181508344 >> [v2:172.21.15.6:6800/1020213724,v1:172.21.15.6:6801/1020213724] conn(0x7fc4980755f0 0x7fc498077a30 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:13.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.951+0000 7fc4bd400640 1 -- 172.21.15.6:0/2181508344 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fc49806be80 msgr2=0x7fc49806e340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:13.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.951+0000 7fc4bd400640 1 --2- 172.21.15.6:0/2181508344 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fc49806be80 0x7fc49806e340 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fc4b0007fd0 tx=0x7fc4b0009040 comp rx=0 tx=0).stop 2024-04-24T23:16:13.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.951+0000 7fc4bd400640 1 -- 172.21.15.6:0/2181508344 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc4b8154cb0 msgr2=0x7fc4b81f5060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:13.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.951+0000 7fc4bd400640 1 --2- 172.21.15.6:0/2181508344 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc4b8154cb0 0x7fc4b81f5060 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fc4b80cce50 tx=0x7fc4a80046c0 comp rx=0 tx=0).stop 2024-04-24T23:16:13.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.951+0000 7fc4bca00640 1 -- 172.21.15.6:0/2181508344 reap_dead start 2024-04-24T23:16:13.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.951+0000 7fc4bd400640 1 -- 172.21.15.6:0/2181508344 shutdown_connections 2024-04-24T23:16:13.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.951+0000 7fc4bd400640 1 -- 172.21.15.6:0/2181508344 >> 172.21.15.6:0/2181508344 conn(0x7fc4b80cc1e0 msgr2=0x7fc4b80c6ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:13.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.951+0000 7fc4bd400640 1 -- 172.21.15.6:0/2181508344 shutdown_connections 2024-04-24T23:16:13.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.951+0000 7fc4bd400640 1 -- 172.21.15.6:0/2181508344 wait complete. 2024-04-24T23:16:13.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.958+0000 7f0e83000640 1 -- 172.21.15.6:0/1994744847 --> [v2:172.21.15.190:6808/1989500934,v1:172.21.15.190:6809/1989500934] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f0e7c0ca8d0 con 0x7f0e5c0756a0 2024-04-24T23:16:13.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.960+0000 7f485b200640 1 -- 172.21.15.6:0/2635184880 --> [v2:172.21.15.6:6824/3163883468,v1:172.21.15.6:6825/3163883468] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f485404cd70 con 0x7f483c0755f0 2024-04-24T23:16:13.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.960+0000 7f0e73e00640 1 -- 172.21.15.6:0/1994744847 <== osd.4 v2:172.21.15.190:6808/1989500934 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f0e7c0ca8d0 con 0x7f0e5c0756a0 2024-04-24T23:16:13.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.961+0000 7f4852000640 1 -- 172.21.15.6:0/2635184880 <== osd.3 v2:172.21.15.6:6824/3163883468 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f485404cd70 con 0x7f483c0755f0 2024-04-24T23:16:13.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.961+0000 7f0e83000640 1 -- 172.21.15.6:0/1994744847 >> [v2:172.21.15.190:6808/1989500934,v1:172.21.15.190:6809/1989500934] conn(0x7f0e5c0756a0 msgr2=0x7f0e5c077ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:13.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.961+0000 7f0e83000640 1 --2- 172.21.15.6:0/1994744847 >> [v2:172.21.15.190:6808/1989500934,v1:172.21.15.190:6809/1989500934] conn(0x7f0e5c0756a0 0x7f0e5c077ae0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:13.989 INFO:teuthology.orchestra.run.smithi006.stdout:55834574861 2024-04-24T23:16:13.990 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-24T23:16:13.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.961+0000 7f0e83000640 1 -- 172.21.15.6:0/1994744847 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f0e5c06bf30 msgr2=0x7f0e5c06e3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:13.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.961+0000 7f0e83000640 1 --2- 172.21.15.6:0/1994744847 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f0e5c06bf30 0x7f0e5c06e3f0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f0e7c158f00 tx=0x7f0e74007000 comp rx=0 tx=0).stop 2024-04-24T23:16:13.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.961+0000 7f0e83000640 1 -- 172.21.15.6:0/1994744847 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0e7c06bc70 msgr2=0x7f0e7c1f8820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:13.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.961+0000 7f0e83000640 1 --2- 172.21.15.6:0/1994744847 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0e7c06bc70 0x7f0e7c1f8820 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f0e6c011590 tx=0x7f0e6c011a50 comp rx=0 tx=0).stop 2024-04-24T23:16:13.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.961+0000 7f0e81000640 1 -- 172.21.15.6:0/1994744847 reap_dead start 2024-04-24T23:16:13.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.961+0000 7f0e83000640 1 -- 172.21.15.6:0/1994744847 shutdown_connections 2024-04-24T23:16:13.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.961+0000 7f0e83000640 1 -- 172.21.15.6:0/1994744847 >> 172.21.15.6:0/1994744847 conn(0x7f0e7c072e40 msgr2=0x7f0e7c14ddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:13.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.962+0000 7f485b200640 1 -- 172.21.15.6:0/2635184880 >> [v2:172.21.15.6:6824/3163883468,v1:172.21.15.6:6825/3163883468] conn(0x7f483c0755f0 msgr2=0x7f483c077a30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:13.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.962+0000 7f485b200640 1 --2- 172.21.15.6:0/2635184880 >> [v2:172.21.15.6:6824/3163883468,v1:172.21.15.6:6825/3163883468] conn(0x7f483c0755f0 0x7f483c077a30 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:13.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.962+0000 7f0e83000640 1 -- 172.21.15.6:0/1994744847 shutdown_connections 2024-04-24T23:16:13.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.962+0000 7f0e83000640 1 -- 172.21.15.6:0/1994744847 wait complete. 2024-04-24T23:16:13.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.962+0000 7f485b200640 1 -- 172.21.15.6:0/2635184880 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f483c06be80 msgr2=0x7f483c06e340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:13.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.962+0000 7f485b200640 1 --2- 172.21.15.6:0/2635184880 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f483c06be80 0x7f483c06e340 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f4848034000 tx=0x7f484802e000 comp rx=0 tx=0).stop 2024-04-24T23:16:13.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.962+0000 7f485b200640 1 -- 172.21.15.6:0/2635184880 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f485404c210 msgr2=0x7f4854048610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:13.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.962+0000 7f485b200640 1 --2- 172.21.15.6:0/2635184880 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f485404c210 0x7f4854048610 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f484c00d830 tx=0x7f484c00dcf0 comp rx=0 tx=0).stop 2024-04-24T23:16:13.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.962+0000 7f485a800640 1 -- 172.21.15.6:0/2635184880 reap_dead start 2024-04-24T23:16:13.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.963+0000 7f485b200640 1 -- 172.21.15.6:0/2635184880 shutdown_connections 2024-04-24T23:16:13.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.963+0000 7f485b200640 1 -- 172.21.15.6:0/2635184880 >> 172.21.15.6:0/2635184880 conn(0x7f4854007c90 msgr2=0x7f4854008070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:13.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.963+0000 7f485b200640 1 -- 172.21.15.6:0/2635184880 shutdown_connections 2024-04-24T23:16:13.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.963+0000 7f485b200640 1 -- 172.21.15.6:0/2635184880 wait complete. 2024-04-24T23:16:13.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.964+0000 7efc78a00640 1 -- 172.21.15.6:0/1085726918 --> [v2:172.21.15.190:6816/870571564,v1:172.21.15.190:6817/870571564] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7efc740cd640 con 0x7efc54075670 2024-04-24T23:16:13.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.964+0000 7f4882a00640 1 -- 172.21.15.6:0/1975709464 --> [v2:172.21.15.190:6824/123639197,v1:172.21.15.190:6825/123639197] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f4858002dd0 con 0x7f485c0756a0 2024-04-24T23:16:13.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.965+0000 7f4879600640 1 -- 172.21.15.6:0/1975709464 <== osd.7 v2:172.21.15.190:6824/123639197 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f4858002dd0 con 0x7f485c0756a0 2024-04-24T23:16:13.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.965+0000 7efc6aa00640 1 -- 172.21.15.6:0/1085726918 <== osd.6 v2:172.21.15.190:6816/870571564 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7efc740cd640 con 0x7efc54075670 2024-04-24T23:16:13.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.965+0000 7efc78a00640 1 -- 172.21.15.6:0/1085726918 >> [v2:172.21.15.190:6816/870571564,v1:172.21.15.190:6817/870571564] conn(0x7efc54075670 msgr2=0x7efc54077ab0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:13.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.965+0000 7efc78a00640 1 --2- 172.21.15.6:0/1085726918 >> [v2:172.21.15.190:6816/870571564,v1:172.21.15.190:6817/870571564] conn(0x7efc54075670 0x7efc54077ab0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:13.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.966+0000 7efc78a00640 1 -- 172.21.15.6:0/1085726918 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7efc5406bf00 msgr2=0x7efc5406e3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:13.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.966+0000 7efc78a00640 1 --2- 172.21.15.6:0/1085726918 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7efc5406bf00 0x7efc5406e3c0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7efc740c8930 tx=0x7efc60008040 comp rx=0 tx=0).stop 2024-04-24T23:16:13.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.966+0000 7efc78a00640 1 -- 172.21.15.6:0/1085726918 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efc740b4bc0 msgr2=0x7efc74071310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:13.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.966+0000 7efc78a00640 1 --2- 172.21.15.6:0/1085726918 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efc740b4bc0 0x7efc74071310 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7efc6400c420 tx=0x7efc64001e70 comp rx=0 tx=0).stop 2024-04-24T23:16:13.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.966+0000 7f4882a00640 1 -- 172.21.15.6:0/1975709464 >> [v2:172.21.15.190:6824/123639197,v1:172.21.15.190:6825/123639197] conn(0x7f485c0756a0 msgr2=0x7f485c077ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:13.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.966+0000 7f4882a00640 1 --2- 172.21.15.6:0/1975709464 >> [v2:172.21.15.190:6824/123639197,v1:172.21.15.190:6825/123639197] conn(0x7f485c0756a0 0x7f485c077ae0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:13.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.966+0000 7efc73200640 1 -- 172.21.15.6:0/1085726918 reap_dead start 2024-04-24T23:16:13.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.966+0000 7efc78a00640 1 -- 172.21.15.6:0/1085726918 shutdown_connections 2024-04-24T23:16:13.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.966+0000 7efc78a00640 1 -- 172.21.15.6:0/1085726918 >> 172.21.15.6:0/1085726918 conn(0x7efc7406c990 msgr2=0x7efc74071ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:13.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.966+0000 7f4882a00640 1 -- 172.21.15.6:0/1975709464 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f485c06bf30 msgr2=0x7f485c06e3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:13.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.966+0000 7f4882a00640 1 --2- 172.21.15.6:0/1975709464 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f485c06bf30 0x7f485c06e3f0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f4870000f80 tx=0x7f4870006040 comp rx=0 tx=0).stop 2024-04-24T23:16:13.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.966+0000 7f4882a00640 1 -- 172.21.15.6:0/1975709464 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f487c0c87c0 msgr2=0x7f487c151310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:13.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.966+0000 7f4882a00640 1 --2- 172.21.15.6:0/1975709464 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f487c0c87c0 0x7f487c151310 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f487400a920 tx=0x7f4874004730 comp rx=0 tx=0).stop 2024-04-24T23:16:13.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.966+0000 7efc78a00640 1 -- 172.21.15.6:0/1085726918 shutdown_connections 2024-04-24T23:16:13.998 INFO:teuthology.orchestra.run.smithi006.stdout:55834574861 2024-04-24T23:16:13.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.966+0000 7f4882000640 1 -- 172.21.15.6:0/1975709464 reap_dead start 2024-04-24T23:16:13.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.966+0000 7f4882a00640 1 -- 172.21.15.6:0/1975709464 shutdown_connections 2024-04-24T23:16:13.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.966+0000 7f4882a00640 1 -- 172.21.15.6:0/1975709464 >> 172.21.15.6:0/1975709464 conn(0x7f487c06c990 msgr2=0x7f487c154130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:13.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.967+0000 7efc78a00640 1 -- 172.21.15.6:0/1085726918 wait complete. 2024-04-24T23:16:14.000 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-24T23:16:14.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.967+0000 7f4882a00640 1 -- 172.21.15.6:0/1975709464 shutdown_connections 2024-04-24T23:16:14.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:13.967+0000 7f4882a00640 1 -- 172.21.15.6:0/1975709464 wait complete. 2024-04-24T23:16:14.001 INFO:teuthology.orchestra.run.smithi006.stdout:55834574861 2024-04-24T23:16:14.001 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-24T23:16:14.002 INFO:teuthology.orchestra.run.smithi006.stdout:55834574862 2024-04-24T23:16:14.002 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-04-24T23:16:14.003 INFO:teuthology.orchestra.run.smithi006.stdout:55834574862 2024-04-24T23:16:14.003 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-24T23:16:14.004 INFO:teuthology.orchestra.run.smithi006.stdout:55834574861 2024-04-24T23:16:14.005 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-04-24T23:16:14.005 INFO:teuthology.orchestra.run.smithi006.stdout:55834574861 2024-04-24T23:16:14.006 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-04-24T23:16:14.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.301+0000 7fd715e00640 1 Processor -- start 2024-04-24T23:16:14.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.301+0000 7fd715e00640 1 -- start start 2024-04-24T23:16:14.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.304+0000 7fd715e00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd7100cc4b0 0x7fd710070e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:14.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.304+0000 7fd715e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd7100cc890 0x7fd710162c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:14.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.305+0000 7fd70ea00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd7100cc4b0 0x7fd710070e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:14.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.305+0000 7fd70ea00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd7100cc4b0 0x7fd710070e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.6:53282/0 (socket says 172.21.15.6:53282) 2024-04-24T23:16:14.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.305+0000 7fd715e00640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7101631d0 con 0x7fd7100c8880 2024-04-24T23:16:14.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.305+0000 7fd715e00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7fd7100c73a0 con 0x7fd7100cc890 2024-04-24T23:16:14.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.305+0000 7fd715e00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fd710162800 con 0x7fd7100cc4b0 2024-04-24T23:16:14.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.305+0000 7fd70f400640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7fd7100c8880 0x7fd71006d2d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.6:50676/0 (socket says 172.21.15.6:50676) 2024-04-24T23:16:14.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.306+0000 7fd70f400640 1 -- 172.21.15.6:0/1792075310 learned_addr learned my addr 172.21.15.6:0/1792075310 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:16:14.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.307+0000 7fd70e000640 1 --2- 172.21.15.6:0/1792075310 >> v2:172.21.15.6:3300/0 conn(0x7fd7100cc890 0x7fd710162c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:14.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.308+0000 7fd70e000640 1 -- 172.21.15.6:0/1792075310 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd7100cc4b0 msgr2=0x7fd710070e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:16:14.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.308+0000 7fd70e000640 1 --2- 172.21.15.6:0/1792075310 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd7100cc4b0 0x7fd710070e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:14.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.308+0000 7fd70e000640 1 -- 172.21.15.6:0/1792075310 >> v1:172.21.15.190:6789/0 conn(0x7fd7100c8880 legacy=0x7fd71006d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:14.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.308+0000 7fd70e000640 1 -- 172.21.15.6:0/1792075310 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7100c75d0 con 0x7fd7100cc890 2024-04-24T23:16:14.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.310+0000 7fd70ea00640 1 --2- 172.21.15.6:0/1792075310 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd7100cc4b0 0x7fd710070e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:16:14.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.311+0000 7fd70e000640 1 --2- 172.21.15.6:0/1792075310 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd7100cc890 0x7fd710162c90 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fd7000097a0 tx=0x7fd70002a130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7e01f0236a9b5f7 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:14.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.311+0000 7fd70d600640 1 -- 172.21.15.6:0/1792075310 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd700037070 con 0x7fd7100cc890 2024-04-24T23:16:14.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.311+0000 7fd70d600640 1 -- 172.21.15.6:0/1792075310 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd70002adc0 con 0x7fd7100cc890 2024-04-24T23:16:14.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.311+0000 7fd70d600640 1 -- 172.21.15.6:0/1792075310 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd700032660 con 0x7fd7100cc890 2024-04-24T23:16:14.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.312+0000 7fd715e00640 1 -- 172.21.15.6:0/1792075310 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd7100cc890 msgr2=0x7fd710162c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:14.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.312+0000 7fd715e00640 1 --2- 172.21.15.6:0/1792075310 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd7100cc890 0x7fd710162c90 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fd7000097a0 tx=0x7fd70002a130 comp rx=0 tx=0).stop 2024-04-24T23:16:14.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.313+0000 7fd715e00640 1 -- 172.21.15.6:0/1792075310 shutdown_connections 2024-04-24T23:16:14.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.313+0000 7fd715e00640 1 --2- 172.21.15.6:0/1792075310 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd7100cc890 0x7fd710162c90 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:14.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.313+0000 7fd715e00640 1 --2- 172.21.15.6:0/1792075310 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd7100cc4b0 0x7fd710070e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:14.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.313+0000 7fd715e00640 1 -- 172.21.15.6:0/1792075310 >> 172.21.15.6:0/1792075310 conn(0x7fd71006ec20 msgr2=0x7fd71006f030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:14.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.314+0000 7fd715e00640 1 -- 172.21.15.6:0/1792075310 shutdown_connections 2024-04-24T23:16:14.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.314+0000 7fd715e00640 1 -- 172.21.15.6:0/1792075310 wait complete. 2024-04-24T23:16:14.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.317+0000 7fd715e00640 1 Processor -- start 2024-04-24T23:16:14.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.318+0000 7fd715e00640 1 -- start start 2024-04-24T23:16:14.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.320+0000 7fd715e00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd7100c8880 0x7fd710154470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:14.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.320+0000 7fd715e00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd7100cc4b0 0x7fd71015b230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:14.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.320+0000 7fd70ea00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd7100c8880 0x7fd710154470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:14.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.320+0000 7fd70ea00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd7100c8880 0x7fd710154470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:59920/0 (socket says 172.21.15.6:59920) 2024-04-24T23:16:14.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.321+0000 7fd70ea00640 1 -- 172.21.15.6:0/1469985703 learned_addr learned my addr 172.21.15.6:0/1469985703 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:16:14.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.321+0000 7fd715e00640 1 --2- 172.21.15.6:0/1469985703 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd7100cc890 0x7fd71015d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:14.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.321+0000 7fd70e000640 1 --2- 172.21.15.6:0/1469985703 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd7100cc4b0 0x7fd71015b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:14.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.321+0000 7fd715e00640 1 -- 172.21.15.6:0/1469985703 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fd710162490 con 0x7fd7100cc890 2024-04-24T23:16:14.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.322+0000 7fd715e00640 1 -- 172.21.15.6:0/1469985703 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fd7100c7b90 con 0x7fd7100c8880 2024-04-24T23:16:14.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.322+0000 7fd715e00640 1 -- 172.21.15.6:0/1469985703 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fd710158c40 con 0x7fd7100cc4b0 2024-04-24T23:16:14.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.322+0000 7fd70f400640 1 --2- 172.21.15.6:0/1469985703 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd7100cc890 0x7fd71015d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:14.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.323+0000 7fd70e000640 1 -- 172.21.15.6:0/1469985703 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd7100c8880 msgr2=0x7fd710154470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:14.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.323+0000 7fd70e000640 1 --2- 172.21.15.6:0/1469985703 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd7100c8880 0x7fd710154470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:14.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.323+0000 7fd70e000640 1 -- 172.21.15.6:0/1469985703 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd7100cc890 msgr2=0x7fd71015d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:14.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.323+0000 7fd70e000640 1 --2- 172.21.15.6:0/1469985703 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd7100cc890 0x7fd71015d410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:14.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.323+0000 7fd70e000640 1 -- 172.21.15.6:0/1469985703 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd71015f3a0 con 0x7fd7100cc4b0 2024-04-24T23:16:14.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.324+0000 7fd70ea00640 1 --2- 172.21.15.6:0/1469985703 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd7100c8880 0x7fd710154470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:16:14.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.324+0000 7fd70e000640 1 --2- 172.21.15.6:0/1469985703 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd7100cc4b0 0x7fd71015b230 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fd700031000 tx=0x7fd7000046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:14.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.325+0000 7f0666400640 1 Processor -- start 2024-04-24T23:16:14.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.325+0000 7f61d1e00640 1 Processor -- start 2024-04-24T23:16:14.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.325+0000 7fd707400640 1 -- 172.21.15.6:0/1469985703 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd700037040 con 0x7fd7100cc4b0 2024-04-24T23:16:14.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.325+0000 7fc2fbe00640 1 Processor -- start 2024-04-24T23:16:14.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.325+0000 7f0666400640 1 -- start start 2024-04-24T23:16:14.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.325+0000 7f61d1e00640 1 -- start start 2024-04-24T23:16:14.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.325+0000 7fd707400640 1 -- 172.21.15.6:0/1469985703 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd700004030 con 0x7fd7100cc4b0 2024-04-24T23:16:14.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.325+0000 7fd707400640 1 -- 172.21.15.6:0/1469985703 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd700004400 con 0x7fd7100cc4b0 2024-04-24T23:16:14.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.325+0000 7fc2fbe00640 1 -- start start 2024-04-24T23:16:14.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.325+0000 7fd715e00640 1 -- 172.21.15.6:0/1469985703 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd71015f640 con 0x7fd7100cc4b0 2024-04-24T23:16:14.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.326+0000 7fd715e00640 1 -- 172.21.15.6:0/1469985703 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd71015cc10 con 0x7fd7100cc4b0 2024-04-24T23:16:14.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.327+0000 7f0666400640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f06580d9090 0x7f06580deda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:14.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.327+0000 7f61d1e00640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f61c40dd3c0 0x7f61c4046460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:14.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.327+0000 7f61d1e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f61c404ba60 0x7f61c404be80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:14.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.327+0000 7f61d1e00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f61c40dd7a0 con 0x7f61c40e3090 2024-04-24T23:16:14.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.327+0000 7f61d1e00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f61c4045450 con 0x7f61c404ba60 2024-04-24T23:16:14.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.327+0000 7f61d1e00640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f61c4045fd0 con 0x7f61c40dd3c0 2024-04-24T23:16:14.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.327+0000 7f0666400640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f0658037870 0x7f06580505c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:14.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.327+0000 7fc2fbe00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc2f4037630 0x7fc2f404f520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:14.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.328+0000 7fc2fbe00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc2f4037a10 0x7fc2f40e3980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:14.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.328+0000 7fc2fbe00640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc2f404fa60 con 0x7fc2f4007810 2024-04-24T23:16:14.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.328+0000 7fc2fbe00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7fc2f403a2d0 con 0x7fc2f4037a10 2024-04-24T23:16:14.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.328+0000 7fc2fbe00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fc2f4039480 con 0x7fc2f4037630 2024-04-24T23:16:14.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.327+0000 7f0666400640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f06580df4a0 con 0x7f0658050180 2024-04-24T23:16:14.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.327+0000 7f0666400640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f0658038090 con 0x7f0658037870 2024-04-24T23:16:14.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.327+0000 7f0666400640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f06580df780 con 0x7f06580d9090 2024-04-24T23:16:14.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.328+0000 7f61cbe00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f61c404ba60 0x7f61c404be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:14.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.328+0000 7f0665000640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f06580d9090 0x7f06580deda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:14.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.328+0000 7f0665000640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f06580d9090 0x7f06580deda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.6:53296/0 (socket says 172.21.15.6:53296) 2024-04-24T23:16:14.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.328+0000 7f0665000640 1 -- 172.21.15.6:0/1868326940 learned_addr learned my addr 172.21.15.6:0/1868326940 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:16:14.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.328+0000 7f0665000640 1 -- 172.21.15.6:0/1868326940 >> v2:172.21.15.190:3300/0 conn(0x7f0658037870 msgr2=0x7f06580505c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-24T23:16:14.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.328+0000 7f0665000640 1 --2- 172.21.15.6:0/1868326940 >> v2:172.21.15.190:3300/0 conn(0x7f0658037870 0x7f06580505c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:14.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.328+0000 7f0665000640 1 -- 172.21.15.6:0/1868326940 >> v1:172.21.15.190:6789/0 conn(0x7f0658050180 legacy=0x7f0658037160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:16:14.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.328+0000 7f0665000640 1 -- 172.21.15.6:0/1868326940 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06580dfa60 con 0x7f06580d9090 2024-04-24T23:16:14.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.328+0000 7fc2faa00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc2f4037630 0x7fc2f404f520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:14.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.328+0000 7fc2fa000640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7fc2f4007810 0x7fc2f4038d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.6:50690/0 (socket says 172.21.15.6:50690) 2024-04-24T23:16:14.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.328+0000 7fc2fa000640 1 -- 172.21.15.6:0/157047690 learned_addr learned my addr 172.21.15.6:0/157047690 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:16:14.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.329+0000 7fc2faa00640 1 -- 172.21.15.6:0/157047690 >> v2:172.21.15.6:3300/0 conn(0x7fc2f4037a10 msgr2=0x7fc2f40e3980 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-24T23:16:14.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.329+0000 7fc2faa00640 1 --2- 172.21.15.6:0/157047690 >> v2:172.21.15.6:3300/0 conn(0x7fc2f4037a10 0x7fc2f40e3980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:14.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.329+0000 7fc2faa00640 1 -- 172.21.15.6:0/157047690 >> v1:172.21.15.190:6789/0 conn(0x7fc2f4007810 legacy=0x7fc2f4038d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:14.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.329+0000 7fc2faa00640 1 -- 172.21.15.6:0/157047690 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2f4006210 con 0x7fc2f4037630 2024-04-24T23:16:14.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.328+0000 7f61d1400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f61c40e3090 0x7f61c40e34c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:43466/0 (socket says 172.21.15.6:43466) 2024-04-24T23:16:14.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.328+0000 7f61d1400640 1 -- 172.21.15.6:0/4197114404 learned_addr learned my addr 172.21.15.6:0/4197114404 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:16:14.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.330+0000 7f61d0a00640 1 --2- 172.21.15.6:0/4197114404 >> v2:172.21.15.190:3300/0 conn(0x7f61c40dd3c0 0x7f61c4046460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:14.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.330+0000 7f0665000640 1 --2- 172.21.15.6:0/1868326940 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f06580d9090 0x7f06580deda0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f066006a6e0 tx=0x7f06600a6a60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5cce26c85174077 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:14.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.330+0000 7fd707400640 1 -- 172.21.15.6:0/1469985703 <== mon.2 v2:172.21.15.190:3301/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fd700003610 con 0x7fd7100cc4b0 2024-04-24T23:16:14.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.330+0000 7fd715e00640 1 -- 172.21.15.6:0/1469985703 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd71015fb60 con 0x7fd7100cc4b0 2024-04-24T23:16:14.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.330+0000 7f61d0a00640 1 -- 172.21.15.6:0/4197114404 >> v2:172.21.15.6:3300/0 conn(0x7f61c404ba60 msgr2=0x7f61c404be80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:16:14.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.331+0000 7f61cb400640 1 -- 172.21.15.6:0/4197114404 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2447571460 0 0) 0x7f61cc0503f0 con 0x7f61c40e3090 2024-04-24T23:16:14.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.331+0000 7fc2faa00640 1 --2- 172.21.15.6:0/157047690 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc2f4037630 0x7fc2f404f520 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fc2fc04fa20 tx=0x7fc2fc0a6ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6091a80f731beca2 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:14.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.331+0000 7f61d0a00640 1 --2- 172.21.15.6:0/4197114404 >> v2:172.21.15.6:3300/0 conn(0x7f61c404ba60 0x7f61c404be80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:14.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.331+0000 7f61d0a00640 1 -- 172.21.15.6:0/4197114404 >> v1:172.21.15.6:6789/0 conn(0x7f61c40e3090 legacy=0x7f61c40e34c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:14.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.331+0000 7f61d0a00640 1 -- 172.21.15.6:0/4197114404 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61c40e3bd0 con 0x7f61c40dd3c0 2024-04-24T23:16:14.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.332+0000 7f61cbe00640 1 --2- 172.21.15.6:0/4197114404 >> v2:172.21.15.6:3300/0 conn(0x7f61c404ba60 0x7f61c404be80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:16:14.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.332+0000 7f065f400640 1 -- 172.21.15.6:0/1868326940 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06600aa8e0 con 0x7f06580d9090 2024-04-24T23:16:14.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.332+0000 7fd707400640 1 --2- 172.21.15.6:0/1469985703 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fd6f006be30 0x7fd6f006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:14.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.332+0000 7fd707400640 1 -- 172.21.15.6:0/1469985703 <== mon.2 v2:172.21.15.190:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fd7000ab3d0 con 0x7fd7100cc4b0 2024-04-24T23:16:14.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.332+0000 7fd70ea00640 1 --2- 172.21.15.6:0/1469985703 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fd6f006be30 0x7fd6f006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:14.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.332+0000 7f065f400640 1 -- 172.21.15.6:0/1868326940 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f06600aaa80 con 0x7f06580d9090 2024-04-24T23:16:14.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.332+0000 7f065f400640 1 -- 172.21.15.6:0/1868326940 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06600b2400 con 0x7f06580d9090 2024-04-24T23:16:14.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.333+0000 7f61d0a00640 1 --2- 172.21.15.6:0/4197114404 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f61c40dd3c0 0x7f61c4046460 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f61c4007a10 tx=0x7f61c002a1a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=98c886a446df8150 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:14.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.333+0000 7f61cb400640 1 -- 172.21.15.6:0/4197114404 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61c0037070 con 0x7f61c40dd3c0 2024-04-24T23:16:14.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.333+0000 7f61cb400640 1 -- 172.21.15.6:0/4197114404 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f61c0032440 con 0x7f61c40dd3c0 2024-04-24T23:16:14.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.333+0000 7f61cb400640 1 -- 172.21.15.6:0/4197114404 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61c0032810 con 0x7f61c40dd3c0 2024-04-24T23:16:14.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.333+0000 7fd70ea00640 1 --2- 172.21.15.6:0/1469985703 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fd6f006be30 0x7fd6f006e2f0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fd6fc009560 tx=0x7fd6fc009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:14.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.333+0000 7f0666400640 1 -- 172.21.15.6:0/1868326940 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f06580d9090 msgr2=0x7f06580deda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:14.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.333+0000 7f0666400640 1 --2- 172.21.15.6:0/1868326940 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f06580d9090 0x7f06580deda0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f066006a6e0 tx=0x7f06600a6a60 comp rx=0 tx=0).stop 2024-04-24T23:16:14.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.333+0000 7f0666400640 1 -- 172.21.15.6:0/1868326940 shutdown_connections 2024-04-24T23:16:14.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.333+0000 7f0666400640 1 --2- 172.21.15.6:0/1868326940 >> v2:172.21.15.190:3300/0 conn(0x7f0658037870 0x7f06580505c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:14.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.334+0000 7f0666400640 1 --2- 172.21.15.6:0/1868326940 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f06580d9090 0x7f06580deda0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:14.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.334+0000 7f0666400640 1 -- 172.21.15.6:0/1868326940 >> 172.21.15.6:0/1868326940 conn(0x7f0658054610 msgr2=0x7f0658054a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:14.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.334+0000 7f0666400640 1 -- 172.21.15.6:0/1868326940 shutdown_connections 2024-04-24T23:16:14.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.334+0000 7f0666400640 1 -- 172.21.15.6:0/1868326940 wait complete. 2024-04-24T23:16:14.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.334+0000 7f61d1e00640 1 -- 172.21.15.6:0/4197114404 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f61c40dd3c0 msgr2=0x7f61c4046460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:14.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.334+0000 7f61d1e00640 1 --2- 172.21.15.6:0/4197114404 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f61c40dd3c0 0x7f61c4046460 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f61c4007a10 tx=0x7f61c002a1a0 comp rx=0 tx=0).stop 2024-04-24T23:16:14.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.334+0000 7f61d1e00640 1 -- 172.21.15.6:0/4197114404 shutdown_connections 2024-04-24T23:16:14.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.334+0000 7f61d1e00640 1 --2- 172.21.15.6:0/4197114404 >> v2:172.21.15.6:3300/0 conn(0x7f61c404ba60 0x7f61c404be80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:14.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.334+0000 7f61d1e00640 1 --2- 172.21.15.6:0/4197114404 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f61c40dd3c0 0x7f61c4046460 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:14.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.335+0000 7f61d1e00640 1 -- 172.21.15.6:0/4197114404 >> 172.21.15.6:0/4197114404 conn(0x7f61c4050180 msgr2=0x7f61c4050590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:14.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.335+0000 7f61d1e00640 1 -- 172.21.15.6:0/4197114404 shutdown_connections 2024-04-24T23:16:14.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.335+0000 7f61d1e00640 1 -- 172.21.15.6:0/4197114404 wait complete. 2024-04-24T23:16:14.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.335+0000 7f3dbae00640 1 Processor -- start 2024-04-24T23:16:14.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.336+0000 7f3dbae00640 1 -- start start 2024-04-24T23:16:14.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.336+0000 7fd707400640 1 -- 172.21.15.6:0/1469985703 <== mon.2 v2:172.21.15.190:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fd71015fb60 con 0x7fd7100cc4b0 2024-04-24T23:16:14.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.337+0000 7fc2f9600640 1 -- 172.21.15.6:0/157047690 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2fc0aa930 con 0x7fc2f4037630 2024-04-24T23:16:14.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.337+0000 7f0666400640 1 Processor -- start 2024-04-24T23:16:14.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.337+0000 7fc2f9600640 1 -- 172.21.15.6:0/157047690 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2fc0aab10 con 0x7fc2f4037630 2024-04-24T23:16:14.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.337+0000 7fc2f9600640 1 -- 172.21.15.6:0/157047690 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2fc0b2400 con 0x7fc2f4037630 2024-04-24T23:16:14.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.337+0000 7f7dac400640 1 Processor -- start 2024-04-24T23:16:14.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.338+0000 7f0666400640 1 -- start start 2024-04-24T23:16:14.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.338+0000 7f3dbae00640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f3db40b35b0 0x7f3db415ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:14.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.338+0000 7f3dbae00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f3db415b0b0 0x7f3db40cb240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:14.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.338+0000 7f3dbae00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3db40b3150 con 0x7f3db4154dd0 2024-04-24T23:16:14.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.338+0000 7f3dbae00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f3db415f020 con 0x7f3db415b0b0 2024-04-24T23:16:14.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.338+0000 7f3dbae00640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f3db415e260 con 0x7f3db40b35b0 2024-04-24T23:16:14.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.338+0000 7f61d1e00640 1 Processor -- start 2024-04-24T23:16:14.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.338+0000 7fc2fbe00640 1 -- 172.21.15.6:0/157047690 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc2f4037630 msgr2=0x7fc2f404f520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:14.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.338+0000 7fc2fbe00640 1 --2- 172.21.15.6:0/157047690 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc2f4037630 0x7fc2f404f520 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fc2fc04fa20 tx=0x7fc2fc0a6ab0 comp rx=0 tx=0).stop 2024-04-24T23:16:14.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.338+0000 7f61d1e00640 1 -- start start 2024-04-24T23:16:14.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.339+0000 7f0666400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0658037870 0x7f06580da0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:14.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.339+0000 7f0666400640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0658050180 0x7f06580d5010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:14.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.339+0000 7f0666400640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f06580d9090 0x7f065817a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:14.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.339+0000 7f3db3e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f3db4154dd0 0x7f3db40cbfa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:43472/0 (socket says 172.21.15.6:43472) 2024-04-24T23:16:14.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.339+0000 7f3db3e00640 1 -- 172.21.15.6:0/1712685435 learned_addr learned my addr 172.21.15.6:0/1712685435 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:16:14.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.339+0000 7fc2fbe00640 1 -- 172.21.15.6:0/157047690 shutdown_connections 2024-04-24T23:16:14.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.339+0000 7fc2fbe00640 1 --2- 172.21.15.6:0/157047690 >> v2:172.21.15.6:3300/0 conn(0x7fc2f4037a10 0x7fc2f40e3980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:14.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.339+0000 7fc2fbe00640 1 --2- 172.21.15.6:0/157047690 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc2f4037630 0x7fc2f404f520 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:14.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.340+0000 7fc2fbe00640 1 -- 172.21.15.6:0/157047690 >> 172.21.15.6:0/157047690 conn(0x7fc2f4054610 msgr2=0x7fc2f4054a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:14.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.338+0000 7f7dac400640 1 -- start start 2024-04-24T23:16:14.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.339+0000 7f0666400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f0658038520 con 0x7f0658037870 2024-04-24T23:16:14.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.339+0000 7f0666400640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f06580d9470 con 0x7f0658050180 2024-04-24T23:16:14.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.339+0000 7f0666400640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f06580386a0 con 0x7f06580d9090 2024-04-24T23:16:14.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.340+0000 7fc2fbe00640 1 -- 172.21.15.6:0/157047690 shutdown_connections 2024-04-24T23:16:14.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.340+0000 7fc2fbe00640 1 -- 172.21.15.6:0/157047690 wait complete. 2024-04-24T23:16:14.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.340+0000 7f3db3400640 1 --2- 172.21.15.6:0/1712685435 >> v2:172.21.15.190:3300/0 conn(0x7f3db40b35b0 0x7f3db415ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:14.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.340+0000 7f3db8e00640 1 --2- 172.21.15.6:0/1712685435 >> v2:172.21.15.6:3300/0 conn(0x7f3db415b0b0 0x7f3db40cb240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:14.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.340+0000 7f8110800640 1 Processor -- start 2024-04-24T23:16:14.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.340+0000 7f8110800640 1 -- start start 2024-04-24T23:16:14.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.340+0000 7f0665a00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f06580d9090 0x7f065817a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:14.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.340+0000 7f0665a00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f06580d9090 0x7f065817a3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.6:53318/0 (socket says 172.21.15.6:53318) 2024-04-24T23:16:14.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.340+0000 7f0665a00640 1 -- 172.21.15.6:0/424223695 learned_addr learned my addr 172.21.15.6:0/424223695 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:16:14.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.340+0000 7f065fe00640 1 --2- 172.21.15.6:0/424223695 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0658050180 0x7f06580d5010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:14.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.341+0000 7f0665000640 1 --2- 172.21.15.6:0/424223695 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0658037870 0x7f06580da0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:14.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.341+0000 7f0665a00640 1 -- 172.21.15.6:0/424223695 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0658050180 msgr2=0x7f06580d5010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:14.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.341+0000 7f0665a00640 1 --2- 172.21.15.6:0/424223695 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0658050180 0x7f06580d5010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:14.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.341+0000 7f0665a00640 1 -- 172.21.15.6:0/424223695 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0658037870 msgr2=0x7f06580da0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:14.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.341+0000 7f0665a00640 1 --2- 172.21.15.6:0/424223695 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0658037870 0x7f06580da0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:14.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.341+0000 7f0665a00640 1 -- 172.21.15.6:0/424223695 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f065817c520 con 0x7f06580d9090 2024-04-24T23:16:14.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.341+0000 7f61d1e00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f61c404ba60 0x7f61c4179200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:14.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.341+0000 7f3db2a00640 1 -- 172.21.15.6:0/1712685435 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 858852538 0 0) 0x7f3db40b3150 con 0x7f3db4154dd0 2024-04-24T23:16:14.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.341+0000 7f3db3400640 1 -- 172.21.15.6:0/1712685435 >> v2:172.21.15.6:3300/0 conn(0x7f3db415b0b0 msgr2=0x7f3db40cb240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:16:14.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.341+0000 7f7dac400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f7d9c0d8fd0 0x7f7d9c04f870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:14.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.342+0000 7f3db3400640 1 --2- 172.21.15.6:0/1712685435 >> v2:172.21.15.6:3300/0 conn(0x7f3db415b0b0 0x7f3db40cb240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:14.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.342+0000 7f3db3400640 1 -- 172.21.15.6:0/1712685435 >> v1:172.21.15.6:6789/0 conn(0x7f3db4154dd0 legacy=0x7f3db40cbfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:14.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.342+0000 7f3db3400640 1 -- 172.21.15.6:0/1712685435 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3db415e540 con 0x7f3db40b35b0 2024-04-24T23:16:14.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.342+0000 7f61d0a00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f61c404ba60 0x7f61c4179200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:14.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.342+0000 7f61d0a00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f61c404ba60 0x7f61c4179200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.6:53320/0 (socket says 172.21.15.6:53320) 2024-04-24T23:16:14.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.342+0000 7f0665a00640 1 --2- 172.21.15.6:0/424223695 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f06580d9090 0x7f065817a3d0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f065400b3f0 tx=0x7f065400b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:14.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.342+0000 7f61d1e00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f61c40dd3c0 0x7f61c40db810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:14.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.343+0000 7f61d1e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f61c40e3090 0x7f61c40dbd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:14.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.343+0000 7f61d1e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f61c4045980 con 0x7f61c40e3090 2024-04-24T23:16:14.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.343+0000 7f61d1e00640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f61c40456a0 con 0x7f61c40dd3c0 2024-04-24T23:16:14.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.343+0000 7f61d1e00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f61c4045c60 con 0x7f61c404ba60 2024-04-24T23:16:14.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.343+0000 7f8110800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f810806d6d0 0x7f81080c4060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:14.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.343+0000 7f8110800640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f81080c45a0 0x7f81080c49d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:14.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.343+0000 7f61d0a00640 1 -- 172.21.15.6:0/407495816 learned_addr learned my addr 172.21.15.6:0/407495816 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:16:14.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.343+0000 7f8110800640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8108153120 con 0x7f8108152d40 2024-04-24T23:16:14.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.343+0000 7f8110800640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f81081599c0 con 0x7f810806d6d0 2024-04-24T23:16:14.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.343+0000 7f8110800640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f81080c8aa0 con 0x7f81080c45a0 2024-04-24T23:16:14.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.343+0000 7f7dac400640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7d9c007590 0x7f7d9c04fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:14.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.343+0000 7f61cbe00640 1 --2- 172.21.15.6:0/407495816 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f61c40dd3c0 0x7f61c40db810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:14.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.343+0000 7f065cc00640 1 -- 172.21.15.6:0/424223695 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0654015020 con 0x7f06580d9090 2024-04-24T23:16:14.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.343+0000 7f065cc00640 1 -- 172.21.15.6:0/424223695 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0654011850 con 0x7f06580d9090 2024-04-24T23:16:14.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.343+0000 7f065cc00640 1 -- 172.21.15.6:0/424223695 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0654011c40 con 0x7f06580d9090 2024-04-24T23:16:14.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.343+0000 7f61d1400640 1 --2- 172.21.15.6:0/407495816 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f61c40e3090 0x7f61c40dbd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:14.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.343+0000 7f7dac400640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7d9c0e3250 con 0x7f7d9c00e930 2024-04-24T23:16:14.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.343+0000 7f7dac400640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f7d9c0e23f0 con 0x7f7d9c0d8fd0 2024-04-24T23:16:14.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.343+0000 7f7dac400640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f7d9c0e26d0 con 0x7f7d9c007590 2024-04-24T23:16:14.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.343+0000 7f810de00640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f8108152d40 0x7f81081592b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.6:50692/0 (socket says 172.21.15.6:50692) 2024-04-24T23:16:14.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.343+0000 7f810de00640 1 -- 172.21.15.6:0/3741974138 learned_addr learned my addr 172.21.15.6:0/3741974138 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:16:14.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.344+0000 7f61d0a00640 1 -- 172.21.15.6:0/407495816 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f61c40dd3c0 msgr2=0x7f61c40db810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:14.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.344+0000 7f61d0a00640 1 --2- 172.21.15.6:0/407495816 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f61c40dd3c0 0x7f61c40db810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:14.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.343+0000 7f7da9a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f7d9c0d8fd0 0x7f7d9c04f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:14.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.344+0000 7f61d0a00640 1 -- 172.21.15.6:0/407495816 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f61c40e3090 msgr2=0x7f61c40dbd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:14.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.344+0000 7f61d0a00640 1 --2- 172.21.15.6:0/407495816 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f61c40e3090 0x7f61c40dbd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:14.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.344+0000 7f61d0a00640 1 -- 172.21.15.6:0/407495816 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61c40da9c0 con 0x7f61c404ba60 2024-04-24T23:16:14.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.343+0000 7f810d400640 1 --2- 172.21.15.6:0/3741974138 >> v2:172.21.15.6:3300/0 conn(0x7f810806d6d0 0x7f81080c4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:14.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.344+0000 7f810ca00640 1 -- 172.21.15.6:0/3741974138 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 557753899 0 0) 0x7f8108153120 con 0x7f8108152d40 2024-04-24T23:16:14.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.344+0000 7f7da9a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f7d9c0d8fd0 0x7f7d9c04f870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:60110/0 (socket says 172.21.15.6:60110) 2024-04-24T23:16:14.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.344+0000 7f7da9000640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7d9c007590 0x7f7d9c04fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:14.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.344+0000 7f7da9a00640 1 -- 172.21.15.6:0/2480048197 learned_addr learned my addr 172.21.15.6:0/2480048197 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:16:14.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.344+0000 7f810ca00640 1 -- 172.21.15.6:0/3741974138 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f80f4003520 con 0x7f8108152d40 2024-04-24T23:16:14.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.344+0000 7f61d0a00640 1 --2- 172.21.15.6:0/407495816 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f61c404ba60 0x7f61c4179200 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f61c0000c00 tx=0x7f61c00046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:14.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.344+0000 7f3db3400640 1 --2- 172.21.15.6:0/1712685435 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3db40b35b0 0x7f3db415ab70 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f3db40b65c0 tx=0x7f3da002a110 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c4f64a6ff74c2ee6 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:14.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.344+0000 7f810e800640 1 --2- 172.21.15.6:0/3741974138 >> v2:172.21.15.190:3300/0 conn(0x7f81080c45a0 0x7f81080c49d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:14.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.344+0000 7f0666400640 1 -- 172.21.15.6:0/424223695 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f065817c800 con 0x7f06580d9090 2024-04-24T23:16:14.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.344+0000 7f0666400640 1 -- 172.21.15.6:0/424223695 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f065817cdc0 con 0x7f06580d9090 2024-04-24T23:16:14.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.345+0000 7f3db2a00640 1 -- 172.21.15.6:0/1712685435 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3da0037070 con 0x7f3db40b35b0 2024-04-24T23:16:14.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.345+0000 7f3db2a00640 1 -- 172.21.15.6:0/1712685435 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3da0032440 con 0x7f3db40b35b0 2024-04-24T23:16:14.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.345+0000 7f3db2a00640 1 -- 172.21.15.6:0/1712685435 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3da002ae60 con 0x7f3db40b35b0 2024-04-24T23:16:14.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.345+0000 7f7da9a00640 1 -- 172.21.15.6:0/2480048197 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7d9c007590 msgr2=0x7f7d9c04fdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:16:14.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.345+0000 7f7da9a00640 1 --2- 172.21.15.6:0/2480048197 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7d9c007590 0x7f7d9c04fdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:14.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.345+0000 7f7da9a00640 1 -- 172.21.15.6:0/2480048197 >> v1:172.21.15.190:6789/0 conn(0x7f7d9c00e930 legacy=0x7f7d9c049bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:14.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.345+0000 7f61c8c00640 1 -- 172.21.15.6:0/407495816 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61c0037040 con 0x7f61c404ba60 2024-04-24T23:16:14.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.345+0000 7f61c8c00640 1 -- 172.21.15.6:0/407495816 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f61c0042070 con 0x7f61c404ba60 2024-04-24T23:16:14.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.345+0000 7f61c8c00640 1 -- 172.21.15.6:0/407495816 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61c0030bd0 con 0x7f61c404ba60 2024-04-24T23:16:14.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.345+0000 7f7da9a00640 1 -- 172.21.15.6:0/2480048197 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d9c0e29b0 con 0x7f7d9c0d8fd0 2024-04-24T23:16:14.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.345+0000 7f61d1e00640 1 -- 172.21.15.6:0/407495816 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61c40dac60 con 0x7f61c404ba60 2024-04-24T23:16:14.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.345+0000 7fc2fbe00640 1 Processor -- start 2024-04-24T23:16:14.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.345+0000 7f61d1e00640 1 -- 172.21.15.6:0/407495816 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f61c40db220 con 0x7f61c404ba60 2024-04-24T23:16:14.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.345+0000 7fc2fbe00640 1 -- start start 2024-04-24T23:16:14.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.346+0000 7f3dbae00640 1 -- 172.21.15.6:0/1712685435 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3db40b35b0 msgr2=0x7f3db415ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:14.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.346+0000 7f3dbae00640 1 --2- 172.21.15.6:0/1712685435 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3db40b35b0 0x7f3db415ab70 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f3db40b65c0 tx=0x7f3da002a110 comp rx=0 tx=0).stop 2024-04-24T23:16:14.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.346+0000 7f3dbae00640 1 -- 172.21.15.6:0/1712685435 shutdown_connections 2024-04-24T23:16:14.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.346+0000 7f3dbae00640 1 --2- 172.21.15.6:0/1712685435 >> v2:172.21.15.6:3300/0 conn(0x7f3db415b0b0 0x7f3db40cb240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:14.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.346+0000 7f3dbae00640 1 --2- 172.21.15.6:0/1712685435 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3db40b35b0 0x7f3db415ab70 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:14.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.346+0000 7f3dbae00640 1 -- 172.21.15.6:0/1712685435 >> 172.21.15.6:0/1712685435 conn(0x7f3db4066ea0 msgr2=0x7f3db40672b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:14.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.346+0000 7f810d400640 1 -- 172.21.15.6:0/3741974138 >> v2:172.21.15.190:3300/0 conn(0x7f81080c45a0 msgr2=0x7f81080c49d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:16:14.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.346+0000 7f810d400640 1 --2- 172.21.15.6:0/3741974138 >> v2:172.21.15.190:3300/0 conn(0x7f81080c45a0 0x7f81080c49d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:14.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.346+0000 7f810d400640 1 -- 172.21.15.6:0/3741974138 >> v1:172.21.15.190:6789/0 conn(0x7f8108152d40 legacy=0x7f81081592b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:14.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.346+0000 7f810d400640 1 -- 172.21.15.6:0/3741974138 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f810806d230 con 0x7f810806d6d0 2024-04-24T23:16:14.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.346+0000 7f0666400640 1 -- 172.21.15.6:0/424223695 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f06580052c0 con 0x7f06580d9090 2024-04-24T23:16:14.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.347+0000 7fc2fbe00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc2f4007810 0x7fc2f404d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:14.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.347+0000 7fc2fbe00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc2f4037630 0x7fc2f404be30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:14.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.347+0000 7fc2fbe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc2f4037a10 0x7fc2f4049d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:14.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.347+0000 7fc2fbe00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fc2f4006a50 con 0x7fc2f4037a10 2024-04-24T23:16:14.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.347+0000 7fc2fbe00640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fc2f4006770 con 0x7fc2f4037630 2024-04-24T23:16:14.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.347+0000 7fc2fbe00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fc2f4006d30 con 0x7fc2f4007810 2024-04-24T23:16:14.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.347+0000 7f61d1e00640 1 -- 172.21.15.6:0/407495816 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f61a8005290 con 0x7f61c404ba60 2024-04-24T23:16:14.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.348+0000 7f3dbae00640 1 -- 172.21.15.6:0/1712685435 shutdown_connections 2024-04-24T23:16:14.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.348+0000 7f3dbae00640 1 -- 172.21.15.6:0/1712685435 wait complete. 2024-04-24T23:16:14.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.348+0000 7fc2fb400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc2f4037a10 0x7fc2f4049d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:14.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.349+0000 7f61c8c00640 1 -- 172.21.15.6:0/407495816 <== mon.2 v2:172.21.15.190:3301/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f61c0047460 con 0x7f61c404ba60 2024-04-24T23:16:14.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.348+0000 7f810d400640 1 --2- 172.21.15.6:0/3741974138 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f810806d6d0 0x7f81080c4060 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f810806ddd0 tx=0x7f80f802a400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb178d1a82071c71 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:14.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.348+0000 7fc2fa000640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc2f4037630 0x7fc2f404be30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:14.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.348+0000 7fc2fa000640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc2f4037630 0x7fc2f404be30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:59990/0 (socket says 172.21.15.6:59990) 2024-04-24T23:16:14.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.348+0000 7fc2fa000640 1 -- 172.21.15.6:0/1865741887 learned_addr learned my addr 172.21.15.6:0/1865741887 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:16:14.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.348+0000 7fc2faa00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc2f4007810 0x7fc2f404d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:14.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.350+0000 7f61c8c00640 1 --2- 172.21.15.6:0/407495816 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f61ac06bf30 0x7f61ac06e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:14.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.349+0000 7fc2fb400640 1 -- 172.21.15.6:0/1865741887 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc2f4007810 msgr2=0x7fc2f404d0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:14.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.350+0000 7f61c8c00640 1 -- 172.21.15.6:0/407495816 <== mon.2 v2:172.21.15.190:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f61c00abd40 con 0x7f61c404ba60 2024-04-24T23:16:14.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.349+0000 7f810ca00640 1 -- 172.21.15.6:0/3741974138 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80f8037070 con 0x7f810806d6d0 2024-04-24T23:16:14.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.349+0000 7f810ca00640 1 -- 172.21.15.6:0/3741974138 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f80f802ac80 con 0x7f810806d6d0 2024-04-24T23:16:14.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.349+0000 7f810ca00640 1 -- 172.21.15.6:0/3741974138 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8100001670 con 0x7f810806d6d0 2024-04-24T23:16:14.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.350+0000 7fc2fb400640 1 --2- 172.21.15.6:0/1865741887 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc2f4007810 0x7fc2f404d0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:14.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.350+0000 7fc2fb400640 1 -- 172.21.15.6:0/1865741887 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc2f4037630 msgr2=0x7fc2f404be30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:14.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.350+0000 7fc2fb400640 1 --2- 172.21.15.6:0/1865741887 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc2f4037630 0x7fc2f404be30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:14.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.350+0000 7f065cc00640 1 -- 172.21.15.6:0/424223695 <== mon.2 v2:172.21.15.190:3301/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f0654018020 con 0x7f06580d9090 2024-04-24T23:16:14.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.350+0000 7f61cbe00640 1 --2- 172.21.15.6:0/407495816 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f61ac06bf30 0x7f61ac06e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:14.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.350+0000 7fc2fb400640 1 -- 172.21.15.6:0/1865741887 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2f4181eb0 con 0x7fc2f4037a10 2024-04-24T23:16:14.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.350+0000 7fc2fa000640 1 --2- 172.21.15.6:0/1865741887 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc2f4037630 0x7fc2f404be30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:16:14.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.350+0000 7fc2faa00640 1 --2- 172.21.15.6:0/1865741887 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc2f4007810 0x7fc2f404d0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:16:14.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.348+0000 7f7da9a00640 1 --2- 172.21.15.6:0/2480048197 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7d9c0d8fd0 0x7f7d9c04f870 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f7d98009870 tx=0x7f7d9802a250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e166e20f4d6ebead server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:14.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.349+0000 7f7da3e00640 1 -- 172.21.15.6:0/2480048197 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d98037070 con 0x7f7d9c0d8fd0 2024-04-24T23:16:14.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.349+0000 7f7da3e00640 1 -- 172.21.15.6:0/2480048197 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d98032440 con 0x7f7d9c0d8fd0 2024-04-24T23:16:14.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.351+0000 7fc2fb400640 1 --2- 172.21.15.6:0/1865741887 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc2f4037a10 0x7fc2f4049d20 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fc2e800b500 tx=0x7fc2e800b9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:14.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.351+0000 7f61cbe00640 1 --2- 172.21.15.6:0/407495816 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f61ac06bf30 0x7f61ac06e3f0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f61c40d6f20 tx=0x7f61b8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:14.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.349+0000 7f7da3e00640 1 -- 172.21.15.6:0/2480048197 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d98032810 con 0x7f7d9c0d8fd0 2024-04-24T23:16:14.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.351+0000 7f7dac400640 1 -- 172.21.15.6:0/2480048197 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7d9c0d8fd0 msgr2=0x7f7d9c04f870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:14.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.351+0000 7f7dac400640 1 --2- 172.21.15.6:0/2480048197 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7d9c0d8fd0 0x7f7d9c04f870 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f7d98009870 tx=0x7f7d9802a250 comp rx=0 tx=0).stop 2024-04-24T23:16:14.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.351+0000 7f065cc00640 1 --2- 172.21.15.6:0/424223695 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f064006be80 0x7f064006e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:14.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.351+0000 7f3dbae00640 1 Processor -- start 2024-04-24T23:16:14.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.351+0000 7f065cc00640 1 -- 172.21.15.6:0/424223695 <== mon.2 v2:172.21.15.190:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f065408aff0 con 0x7f06580d9090 2024-04-24T23:16:14.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.351+0000 7fc2f2a00640 1 -- 172.21.15.6:0/1865741887 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2e8014070 con 0x7fc2f4037a10 2024-04-24T23:16:14.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.351+0000 7fc2f2a00640 1 -- 172.21.15.6:0/1865741887 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2e8010660 con 0x7fc2f4037a10 2024-04-24T23:16:14.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.351+0000 7fc2f2a00640 1 -- 172.21.15.6:0/1865741887 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2e8014070 con 0x7fc2f4037a10 2024-04-24T23:16:14.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.351+0000 7f0665000640 1 --2- 172.21.15.6:0/424223695 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f064006be80 0x7f064006e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:14.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.352+0000 7fc2fbe00640 1 -- 172.21.15.6:0/1865741887 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2f4182150 con 0x7fc2f4037a10 2024-04-24T23:16:14.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.352+0000 7f8110800640 1 -- 172.21.15.6:0/3741974138 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f810806d6d0 msgr2=0x7f81080c4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:14.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.352+0000 7f8110800640 1 --2- 172.21.15.6:0/3741974138 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f810806d6d0 0x7f81080c4060 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f810806ddd0 tx=0x7f80f802a400 comp rx=0 tx=0).stop 2024-04-24T23:16:14.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.352+0000 7f7dac400640 1 -- 172.21.15.6:0/2480048197 shutdown_connections 2024-04-24T23:16:14.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.352+0000 7fc2fbe00640 1 -- 172.21.15.6:0/1865741887 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2f4182710 con 0x7fc2f4037a10 2024-04-24T23:16:14.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.352+0000 7f7dac400640 1 --2- 172.21.15.6:0/2480048197 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7d9c007590 0x7f7d9c04fdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:14.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.352+0000 7f7dac400640 1 --2- 172.21.15.6:0/2480048197 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7d9c0d8fd0 0x7f7d9c04f870 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:14.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.352+0000 7f7dac400640 1 -- 172.21.15.6:0/2480048197 >> 172.21.15.6:0/2480048197 conn(0x7f7d9c054990 msgr2=0x7f7d9c054da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:14.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.352+0000 7f0665000640 1 --2- 172.21.15.6:0/424223695 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f064006be80 0x7f064006e340 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f0658039f90 tx=0x7f06600ad000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:14.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.352+0000 7f3dbae00640 1 -- start start 2024-04-24T23:16:14.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.353+0000 7f8110800640 1 -- 172.21.15.6:0/3741974138 shutdown_connections 2024-04-24T23:16:14.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.353+0000 7f8110800640 1 --2- 172.21.15.6:0/3741974138 >> v2:172.21.15.190:3300/0 conn(0x7f81080c45a0 0x7f81080c49d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:14.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.353+0000 7f8110800640 1 --2- 172.21.15.6:0/3741974138 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f810806d6d0 0x7f81080c4060 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:14.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.353+0000 7f8110800640 1 -- 172.21.15.6:0/3741974138 >> 172.21.15.6:0/3741974138 conn(0x7f81080c8c50 msgr2=0x7f81080c9060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:14.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.353+0000 7f7dac400640 1 -- 172.21.15.6:0/2480048197 shutdown_connections 2024-04-24T23:16:14.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.353+0000 7f8110800640 1 -- 172.21.15.6:0/3741974138 shutdown_connections 2024-04-24T23:16:14.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.354+0000 7f3dbae00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3db40b35b0 0x7f3db415d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:14.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.354+0000 7f3dbae00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3db4154dd0 0x7f3db4153210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:14.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.354+0000 7f3dbae00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3db415b0b0 0x7f3db4153750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:14.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.354+0000 7f3dbae00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f3db41551b0 con 0x7f3db415b0b0 2024-04-24T23:16:14.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.354+0000 7f3dbae00640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f3db40b2e10 con 0x7f3db40b35b0 2024-04-24T23:16:14.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.354+0000 7f3dbae00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f3db4155330 con 0x7f3db4154dd0 2024-04-24T23:16:14.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.354+0000 7f3db3400640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3db4154dd0 0x7f3db4153210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:14.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.354+0000 7f3db3400640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3db4154dd0 0x7f3db4153210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.6:53350/0 (socket says 172.21.15.6:53350) 2024-04-24T23:16:14.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.353+0000 7f7dac400640 1 -- 172.21.15.6:0/2480048197 wait complete. 2024-04-24T23:16:14.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.354+0000 7f3db3400640 1 -- 172.21.15.6:0/1283842620 learned_addr learned my addr 172.21.15.6:0/1283842620 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:16:14.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.354+0000 7f3db3e00640 1 --2- 172.21.15.6:0/1283842620 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3db40b35b0 0x7f3db415d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:14.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.353+0000 7fc2fbe00640 1 -- 172.21.15.6:0/1865741887 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc2f40052c0 con 0x7fc2f4037a10 2024-04-24T23:16:14.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.354+0000 7f3db8e00640 1 --2- 172.21.15.6:0/1283842620 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3db415b0b0 0x7f3db4153750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:14.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.355+0000 7f3db3400640 1 -- 172.21.15.6:0/1283842620 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3db40b35b0 msgr2=0x7f3db415d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:14.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.355+0000 7f3db3400640 1 --2- 172.21.15.6:0/1283842620 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3db40b35b0 0x7f3db415d2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:14.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.355+0000 7f3db3400640 1 -- 172.21.15.6:0/1283842620 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3db415b0b0 msgr2=0x7f3db4153750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:14.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.355+0000 7f3db3400640 1 --2- 172.21.15.6:0/1283842620 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3db415b0b0 0x7f3db4153750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:14.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.355+0000 7f3db3400640 1 -- 172.21.15.6:0/1283842620 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3db41580f0 con 0x7f3db4154dd0 2024-04-24T23:16:14.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.355+0000 7f3db8e00640 1 --2- 172.21.15.6:0/1283842620 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3db415b0b0 0x7f3db4153750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:16:14.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.355+0000 7f3db3400640 1 --2- 172.21.15.6:0/1283842620 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3db4154dd0 0x7f3db4153210 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f3da002a700 tx=0x7f3da00046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:14.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.353+0000 7f8110800640 1 -- 172.21.15.6:0/3741974138 wait complete. 2024-04-24T23:16:14.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.355+0000 7f61c8c00640 1 -- 172.21.15.6:0/407495816 <== mon.2 v2:172.21.15.190:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f61c0032e20 con 0x7f61c404ba60 2024-04-24T23:16:14.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.356+0000 7f7dac400640 1 Processor -- start 2024-04-24T23:16:14.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.356+0000 7f7dac400640 1 -- start start 2024-04-24T23:16:14.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.356+0000 7f8110800640 1 Processor -- start 2024-04-24T23:16:14.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.356+0000 7f8110800640 1 -- start start 2024-04-24T23:16:14.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.356+0000 7fc2f2a00640 1 -- 172.21.15.6:0/1865741887 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fc2e8017070 con 0x7fc2f4037a10 2024-04-24T23:16:14.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.357+0000 7f7dac400640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7d9c007590 0x7f7d9c177b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:14.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.357+0000 7f7dac400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7d9c00e930 0x7f7d9c178060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:14.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.357+0000 7f7dac400640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7d9c0d8fd0 0x7f7d9c175b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:14.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.357+0000 7f7dac400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f7d9c0d93b0 con 0x7f7d9c00e930 2024-04-24T23:16:14.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.357+0000 7f7dac400640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f7d9c0e2f70 con 0x7f7d9c0d8fd0 2024-04-24T23:16:14.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.357+0000 7f7dac400640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f7d9c0d9530 con 0x7f7d9c007590 2024-04-24T23:16:14.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.358+0000 7f7daa400640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7d9c0d8fd0 0x7f7d9c175b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:14.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.358+0000 7fc2f2a00640 1 --2- 172.21.15.6:0/1865741887 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fc2dc06c030 0x7fc2dc06e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:14.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.358+0000 7f7daa400640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7d9c0d8fd0 0x7f7d9c175b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:60000/0 (socket says 172.21.15.6:60000) 2024-04-24T23:16:14.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.358+0000 7f7daa400640 1 -- 172.21.15.6:0/3573754216 learned_addr learned my addr 172.21.15.6:0/3573754216 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:16:14.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.358+0000 7f8110800640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f810806d6d0 0x7f81081ebfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:14.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.358+0000 7f8110800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f81080c45a0 0x7f8108072420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:14.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.358+0000 7f8110800640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8108152d40 0x7f81081406f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:14.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.358+0000 7f8110800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f8108158ab0 con 0x7f81080c45a0 2024-04-24T23:16:14.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.358+0000 7f8110800640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f810806cf50 con 0x7f810806d6d0 2024-04-24T23:16:14.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.358+0000 7f8110800640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f810806f8d0 con 0x7f8108152d40 2024-04-24T23:16:14.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.358+0000 7fc2f2a00640 1 -- 172.21.15.6:0/1865741887 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fc2e808aae0 con 0x7fc2f4037a10 2024-04-24T23:16:14.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.359+0000 7f3dabe00640 1 -- 172.21.15.6:0/1283842620 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3da0037050 con 0x7f3db4154dd0 2024-04-24T23:16:14.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.359+0000 7f7da9a00640 1 --2- 172.21.15.6:0/3573754216 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7d9c007590 0x7f7d9c177b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:14.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.359+0000 7f3dabe00640 1 -- 172.21.15.6:0/1283842620 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3da0038040 con 0x7f3db4154dd0 2024-04-24T23:16:14.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.359+0000 7f3dabe00640 1 -- 172.21.15.6:0/1283842620 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3da003cb50 con 0x7f3db4154dd0 2024-04-24T23:16:14.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.359+0000 7fc2faa00640 1 --2- 172.21.15.6:0/1865741887 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fc2dc06c030 0x7fc2dc06e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:14.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.359+0000 7f3dbae00640 1 -- 172.21.15.6:0/1283842620 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3db41583d0 con 0x7f3db4154dd0 2024-04-24T23:16:14.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.360+0000 7f7da9a00640 1 -- 172.21.15.6:0/3573754216 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7d9c0d8fd0 msgr2=0x7f7d9c175b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:14.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.360+0000 7f7da9a00640 1 --2- 172.21.15.6:0/3573754216 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7d9c0d8fd0 0x7f7d9c175b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:14.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.360+0000 7f7da9a00640 1 -- 172.21.15.6:0/3573754216 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7d9c00e930 msgr2=0x7f7d9c178060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:14.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.360+0000 7f7da9a00640 1 --2- 172.21.15.6:0/3573754216 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7d9c00e930 0x7f7d9c178060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:14.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.360+0000 7f7da9a00640 1 -- 172.21.15.6:0/3573754216 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d9c0e0cb0 con 0x7f7d9c007590 2024-04-24T23:16:14.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.359+0000 7f810de00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f810806d6d0 0x7f81081ebfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:14.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.359+0000 7f810de00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f810806d6d0 0x7f81081ebfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:60004/0 (socket says 172.21.15.6:60004) 2024-04-24T23:16:14.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.359+0000 7f810de00640 1 -- 172.21.15.6:0/1311032804 learned_addr learned my addr 172.21.15.6:0/1311032804 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:16:14.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.359+0000 7f810e800640 1 --2- 172.21.15.6:0/1311032804 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8108152d40 0x7f81081406f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:14.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.360+0000 7f810d400640 1 --2- 172.21.15.6:0/1311032804 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f81080c45a0 0x7f8108072420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:14.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.360+0000 7fc2faa00640 1 --2- 172.21.15.6:0/1865741887 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fc2dc06c030 0x7fc2dc06e4f0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fc2f404bac0 tx=0x7fc2fc0ad000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:14.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.359+0000 7f065cc00640 1 -- 172.21.15.6:0/424223695 <== mon.2 v2:172.21.15.190:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f065401f020 con 0x7f06580d9090 2024-04-24T23:16:14.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.360+0000 7f810e800640 1 -- 172.21.15.6:0/1311032804 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f810806d6d0 msgr2=0x7f81081ebfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:14.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.360+0000 7f810e800640 1 --2- 172.21.15.6:0/1311032804 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f810806d6d0 0x7f81081ebfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:14.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.360+0000 7f810e800640 1 -- 172.21.15.6:0/1311032804 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f81080c45a0 msgr2=0x7f8108072420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:14.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.360+0000 7f810e800640 1 --2- 172.21.15.6:0/1311032804 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f81080c45a0 0x7f8108072420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:14.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.360+0000 7f810e800640 1 -- 172.21.15.6:0/1311032804 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8108140c30 con 0x7f8108152d40 2024-04-24T23:16:14.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.360+0000 7f7da9a00640 1 --2- 172.21.15.6:0/3573754216 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7d9c007590 0x7f7d9c177b20 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f7d98031000 tx=0x7f7d980046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:14.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.361+0000 7f810de00640 1 --2- 172.21.15.6:0/1311032804 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f810806d6d0 0x7f81081ebfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:16:14.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.359+0000 7f3dbae00640 1 -- 172.21.15.6:0/1283842620 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f3db4152a10 con 0x7f3db4154dd0 2024-04-24T23:16:14.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.361+0000 7f7da1600640 1 -- 172.21.15.6:0/3573754216 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d98041020 con 0x7f7d9c007590 2024-04-24T23:16:14.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.361+0000 7f5983400640 1 -- 172.21.15.6:0/2381668547 <== osd.5 v2:172.21.15.190:6800/3687548412 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f5984008e50 con 0x7f596c0755f0 2024-04-24T23:16:14.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.361+0000 7f810e800640 1 --2- 172.21.15.6:0/1311032804 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8108152d40 0x7f81081406f0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f8104007d90 tx=0x7f810400e4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:14.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.363+0000 7f3dabe00640 1 -- 172.21.15.6:0/1283842620 <== mon.2 v2:172.21.15.190:3301/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f3da003ccf0 con 0x7f3db4154dd0 2024-04-24T23:16:14.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.364+0000 7f3dbae00640 1 -- 172.21.15.6:0/1283842620 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3db40cf680 con 0x7f3db4154dd0 2024-04-24T23:16:14.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.362+0000 7f80fe000640 1 -- 172.21.15.6:0/1311032804 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f810400edd0 con 0x7f8108152d40 2024-04-24T23:16:14.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.362+0000 7f80fe000640 1 -- 172.21.15.6:0/1311032804 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8104003990 con 0x7f8108152d40 2024-04-24T23:16:14.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.362+0000 7f80fe000640 1 -- 172.21.15.6:0/1311032804 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8104003d80 con 0x7f8108152d40 2024-04-24T23:16:14.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.362+0000 7f8110800640 1 -- 172.21.15.6:0/1311032804 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81080710a0 con 0x7f8108152d40 2024-04-24T23:16:14.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.362+0000 7f8110800640 1 -- 172.21.15.6:0/1311032804 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f8108071660 con 0x7f8108152d40 2024-04-24T23:16:14.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.365+0000 7f7da1600640 1 -- 172.21.15.6:0/3573754216 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d980308f0 con 0x7f7d9c007590 2024-04-24T23:16:14.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.365+0000 7f7da1600640 1 -- 172.21.15.6:0/3573754216 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d98037040 con 0x7f7d9c007590 2024-04-24T23:16:14.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.364+0000 7f3dabe00640 1 --2- 172.21.15.6:0/1283842620 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f3d9406bf00 0x7f3d9406e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:14.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.364+0000 7f3dabe00640 1 -- 172.21.15.6:0/1283842620 <== mon.2 v2:172.21.15.190:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f3da00abbd0 con 0x7f3db4154dd0 2024-04-24T23:16:14.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.368+0000 7f3db3e00640 1 --2- 172.21.15.6:0/1283842620 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f3d9406bf00 0x7f3d9406e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:14.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.367+0000 7f7dac400640 1 -- 172.21.15.6:0/3573754216 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d9c178760 con 0x7f7d9c007590 2024-04-24T23:16:14.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.368+0000 7f7dac400640 1 -- 172.21.15.6:0/3573754216 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d9c0e1580 con 0x7f7d9c007590 2024-04-24T23:16:14.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.366+0000 7f80fe000640 1 -- 172.21.15.6:0/1311032804 <== mon.2 v2:172.21.15.190:3301/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f810401b020 con 0x7f8108152d40 2024-04-24T23:16:14.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.369+0000 7f80fe000640 1 --2- 172.21.15.6:0/1311032804 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f80ec06be60 0x7f80ec06e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:14.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.369+0000 7f8110800640 1 -- 172.21.15.6:0/1311032804 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f80e8005290 con 0x7f8108152d40 2024-04-24T23:16:14.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.369+0000 7f80fe000640 1 -- 172.21.15.6:0/1311032804 <== mon.2 v2:172.21.15.190:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f810408b320 con 0x7f8108152d40 2024-04-24T23:16:14.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.370+0000 7f810de00640 1 --2- 172.21.15.6:0/1311032804 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f80ec06be60 0x7f80ec06e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:14.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.370+0000 7fc2f2a00640 1 -- 172.21.15.6:0/1865741887 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fc2e80590c0 con 0x7fc2f4037a10 2024-04-24T23:16:14.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.371+0000 7f810de00640 1 --2- 172.21.15.6:0/1311032804 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f80ec06be60 0x7f80ec06e320 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f8100000f60 tx=0x7f8100001990 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:14.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.370+0000 7f3db3e00640 1 --2- 172.21.15.6:0/1283842620 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f3d9406bf00 0x7f3d9406e3c0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f3da4000f60 tx=0x7f3da4007e40 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:14.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.371+0000 7f3dabe00640 1 -- 172.21.15.6:0/1283842620 <== mon.2 v2:172.21.15.190:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f3da00abe20 con 0x7f3db4154dd0 2024-04-24T23:16:14.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.370+0000 7f7dac400640 1 -- 172.21.15.6:0/3573754216 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d9c006440 con 0x7f7d9c007590 2024-04-24T23:16:14.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.373+0000 7f7da1600640 1 -- 172.21.15.6:0/3573754216 <== mon.2 v2:172.21.15.190:3301/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f7d98044020 con 0x7f7d9c007590 2024-04-24T23:16:14.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.374+0000 7f7da1600640 1 --2- 172.21.15.6:0/3573754216 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f7d8406beb0 0x7f7d8406e370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:14.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.375+0000 7f7da1600640 1 -- 172.21.15.6:0/3573754216 <== mon.2 v2:172.21.15.190:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f7d9803ee10 con 0x7f7d9c007590 2024-04-24T23:16:14.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.375+0000 7f7da9000640 1 --2- 172.21.15.6:0/3573754216 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f7d8406beb0 0x7f7d8406e370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:14.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.376+0000 7f7da9000640 1 --2- 172.21.15.6:0/3573754216 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f7d8406beb0 0x7f7d8406e370 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f7d90006a90 tx=0x7f7d90006a20 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:14.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.378+0000 7f80fe000640 1 -- 172.21.15.6:0/1311032804 <== mon.2 v2:172.21.15.190:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f81040599d0 con 0x7f8108152d40 2024-04-24T23:16:14.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.380+0000 7f7da1600640 1 -- 172.21.15.6:0/3573754216 <== mon.2 v2:172.21.15.190:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f7d9804b070 con 0x7f7d9c007590 2024-04-24T23:16:14.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.394+0000 7f5992600640 1 -- 172.21.15.6:0/2381668547 --> [v2:172.21.15.190:6800/3687548412,v1:172.21.15.190:6801/3687548412] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f598c0c72e0 con 0x7f596c0755f0 2024-04-24T23:16:14.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.395+0000 7f5983400640 1 -- 172.21.15.6:0/2381668547 <== osd.5 v2:172.21.15.190:6800/3687548412 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f598c0c72e0 con 0x7f596c0755f0 2024-04-24T23:16:14.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.399+0000 7f5992600640 1 -- 172.21.15.6:0/2381668547 >> [v2:172.21.15.190:6800/3687548412,v1:172.21.15.190:6801/3687548412] conn(0x7f596c0755f0 msgr2=0x7f596c077a30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:14.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.399+0000 7f5992600640 1 --2- 172.21.15.6:0/2381668547 >> [v2:172.21.15.190:6800/3687548412,v1:172.21.15.190:6801/3687548412] conn(0x7f596c0755f0 0x7f596c077a30 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:14.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.399+0000 7f5992600640 1 -- 172.21.15.6:0/2381668547 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f596c06be80 msgr2=0x7f596c06e340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:14.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.399+0000 7f5992600640 1 --2- 172.21.15.6:0/2381668547 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f596c06be80 0x7f596c06e340 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f597c003eb0 tx=0x7f597c009040 comp rx=0 tx=0).stop 2024-04-24T23:16:14.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.399+0000 7f5992600640 1 -- 172.21.15.6:0/2381668547 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f598c15b0b0 msgr2=0x7f598c1e5930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:14.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.399+0000 7f5992600640 1 --2- 172.21.15.6:0/2381668547 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f598c15b0b0 0x7f598c1e5930 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f5984009f70 tx=0x7f5984004a40 comp rx=0 tx=0).stop 2024-04-24T23:16:14.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.401+0000 7f598be00640 1 -- 172.21.15.6:0/2381668547 reap_dead start 2024-04-24T23:16:14.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.401+0000 7f5992600640 1 -- 172.21.15.6:0/2381668547 shutdown_connections 2024-04-24T23:16:14.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.401+0000 7f5992600640 1 -- 172.21.15.6:0/2381668547 >> 172.21.15.6:0/2381668547 conn(0x7f598c066ea0 msgr2=0x7f598c0b4260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:14.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.402+0000 7f5992600640 1 -- 172.21.15.6:0/2381668547 shutdown_connections 2024-04-24T23:16:14.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.402+0000 7f5992600640 1 -- 172.21.15.6:0/2381668547 wait complete. 2024-04-24T23:16:14.434 INFO:teuthology.orchestra.run.smithi006.stdout:55834574861 2024-04-24T23:16:14.434 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-04-24T23:16:14.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.803+0000 7fd715e00640 1 -- 172.21.15.6:0/1469985703 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7fd7100c66f0 con 0x7fd7100cc4b0 2024-04-24T23:16:14.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.805+0000 7fd707400640 1 -- 172.21.15.6:0/1469985703 <== mon.2 v2:172.21.15.190:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fd7100c66f0 con 0x7fd7100cc4b0 2024-04-24T23:16:14.810 INFO:teuthology.orchestra.run.smithi006.stdout:55834574860 2024-04-24T23:16:14.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.812+0000 7f8110800640 1 -- 172.21.15.6:0/1311032804 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f80e8005850 con 0x7f8108152d40 2024-04-24T23:16:14.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.813+0000 7f80fe000640 1 -- 172.21.15.6:0/1311032804 <== mon.2 v2:172.21.15.190:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f81040592d0 con 0x7f8108152d40 2024-04-24T23:16:14.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.813+0000 7fd715e00640 1 -- 172.21.15.6:0/1469985703 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fd6f006be30 msgr2=0x7fd6f006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:14.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.813+0000 7fd715e00640 1 --2- 172.21.15.6:0/1469985703 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fd6f006be30 0x7fd6f006e2f0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fd6fc009560 tx=0x7fd6fc009190 comp rx=0 tx=0).stop 2024-04-24T23:16:14.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.813+0000 7fd715e00640 1 -- 172.21.15.6:0/1469985703 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd7100cc4b0 msgr2=0x7fd71015b230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:14.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.813+0000 7fd715e00640 1 --2- 172.21.15.6:0/1469985703 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd7100cc4b0 0x7fd71015b230 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fd700031000 tx=0x7fd7000046c0 comp rx=0 tx=0).stop 2024-04-24T23:16:14.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.813+0000 7fd715e00640 1 -- 172.21.15.6:0/1469985703 shutdown_connections 2024-04-24T23:16:14.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.814+0000 7fd715e00640 1 --2- 172.21.15.6:0/1469985703 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fd6f006be30 0x7fd6f006e2f0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:14.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.814+0000 7fd715e00640 1 --2- 172.21.15.6:0/1469985703 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd7100cc4b0 0x7fd71015b230 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:14.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.814+0000 7fd715e00640 1 --2- 172.21.15.6:0/1469985703 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd7100cc890 0x7fd71015d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:14.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.814+0000 7fd715e00640 1 --2- 172.21.15.6:0/1469985703 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd7100c8880 0x7fd710154470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:14.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.814+0000 7fd715e00640 1 -- 172.21.15.6:0/1469985703 >> 172.21.15.6:0/1469985703 conn(0x7fd71006ec20 msgr2=0x7fd7100caa80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:14.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.814+0000 7fd715e00640 1 -- 172.21.15.6:0/1469985703 shutdown_connections 2024-04-24T23:16:14.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.814+0000 7fd715e00640 1 -- 172.21.15.6:0/1469985703 wait complete. 2024-04-24T23:16:14.819 INFO:teuthology.orchestra.run.smithi006.stdout:55834574860 2024-04-24T23:16:14.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.820+0000 7f8110800640 1 -- 172.21.15.6:0/1311032804 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f80ec06be60 msgr2=0x7f80ec06e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:14.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.820+0000 7f8110800640 1 --2- 172.21.15.6:0/1311032804 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f80ec06be60 0x7f80ec06e320 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f8100000f60 tx=0x7f8100001990 comp rx=0 tx=0).stop 2024-04-24T23:16:14.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.820+0000 7f8110800640 1 -- 172.21.15.6:0/1311032804 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8108152d40 msgr2=0x7f81081406f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:14.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.820+0000 7f8110800640 1 --2- 172.21.15.6:0/1311032804 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8108152d40 0x7f81081406f0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f8104007d90 tx=0x7f810400e4b0 comp rx=0 tx=0).stop 2024-04-24T23:16:14.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.820+0000 7f8110800640 1 -- 172.21.15.6:0/1311032804 shutdown_connections 2024-04-24T23:16:14.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.820+0000 7f8110800640 1 --2- 172.21.15.6:0/1311032804 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f80ec06be60 0x7f80ec06e320 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:14.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.820+0000 7f8110800640 1 --2- 172.21.15.6:0/1311032804 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8108152d40 0x7f81081406f0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:14.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.820+0000 7f8110800640 1 --2- 172.21.15.6:0/1311032804 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f81080c45a0 0x7f8108072420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:14.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.820+0000 7f8110800640 1 --2- 172.21.15.6:0/1311032804 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f810806d6d0 0x7f81081ebfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:14.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.820+0000 7f8110800640 1 -- 172.21.15.6:0/1311032804 >> 172.21.15.6:0/1311032804 conn(0x7f81080c8c50 msgr2=0x7f8108151e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:14.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.822+0000 7f8110800640 1 -- 172.21.15.6:0/1311032804 shutdown_connections 2024-04-24T23:16:14.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.822+0000 7f8110800640 1 -- 172.21.15.6:0/1311032804 wait complete. 2024-04-24T23:16:14.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.830+0000 7f61d1e00640 1 -- 172.21.15.6:0/407495816 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f61a8005580 con 0x7f61c404ba60 2024-04-24T23:16:14.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.832+0000 7f61c8c00640 1 -- 172.21.15.6:0/407495816 <== mon.2 v2:172.21.15.190:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f61c007a390 con 0x7f61c404ba60 2024-04-24T23:16:14.835 INFO:teuthology.orchestra.run.smithi006.stdout:55834574861 2024-04-24T23:16:14.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.833+0000 7f3dbae00640 1 -- 172.21.15.6:0/1283842620 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f3db41569f0 con 0x7f3db4154dd0 2024-04-24T23:16:14.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.834+0000 7f3dabe00640 1 -- 172.21.15.6:0/1283842620 <== mon.2 v2:172.21.15.190:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f3db40cf680 con 0x7f3db4154dd0 2024-04-24T23:16:14.838 INFO:teuthology.orchestra.run.smithi006.stdout:55834574861 2024-04-24T23:16:14.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.837+0000 7f61d1e00640 1 -- 172.21.15.6:0/407495816 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f61ac06bf30 msgr2=0x7f61ac06e3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:14.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.837+0000 7f61d1e00640 1 --2- 172.21.15.6:0/407495816 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f61ac06bf30 0x7f61ac06e3f0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f61c40d6f20 tx=0x7f61b8008040 comp rx=0 tx=0).stop 2024-04-24T23:16:14.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.837+0000 7f61d1e00640 1 -- 172.21.15.6:0/407495816 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f61c404ba60 msgr2=0x7f61c4179200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:14.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.837+0000 7f61d1e00640 1 --2- 172.21.15.6:0/407495816 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f61c404ba60 0x7f61c4179200 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f61c0000c00 tx=0x7f61c00046c0 comp rx=0 tx=0).stop 2024-04-24T23:16:14.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.837+0000 7f61d1e00640 1 -- 172.21.15.6:0/407495816 shutdown_connections 2024-04-24T23:16:14.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.837+0000 7f61d1e00640 1 --2- 172.21.15.6:0/407495816 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f61ac06bf30 0x7f61ac06e3f0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:14.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.837+0000 7f61d1e00640 1 --2- 172.21.15.6:0/407495816 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f61c40e3090 0x7f61c40dbd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:14.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.837+0000 7f61d1e00640 1 --2- 172.21.15.6:0/407495816 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f61c40dd3c0 0x7f61c40db810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:14.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.837+0000 7f61d1e00640 1 --2- 172.21.15.6:0/407495816 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f61c404ba60 0x7f61c4179200 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:14.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.837+0000 7f61d1e00640 1 -- 172.21.15.6:0/407495816 >> 172.21.15.6:0/407495816 conn(0x7f61c4050180 msgr2=0x7f61c40d86e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:14.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.837+0000 7f61d1e00640 1 -- 172.21.15.6:0/407495816 shutdown_connections 2024-04-24T23:16:14.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.837+0000 7f61d1e00640 1 -- 172.21.15.6:0/407495816 wait complete. 2024-04-24T23:16:14.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.842+0000 7f3dbae00640 1 -- 172.21.15.6:0/1283842620 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f3d9406bf00 msgr2=0x7f3d9406e3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:14.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.842+0000 7f3dbae00640 1 --2- 172.21.15.6:0/1283842620 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f3d9406bf00 0x7f3d9406e3c0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f3da4000f60 tx=0x7f3da4007e40 comp rx=0 tx=0).stop 2024-04-24T23:16:14.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.842+0000 7f3dbae00640 1 -- 172.21.15.6:0/1283842620 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3db4154dd0 msgr2=0x7f3db4153210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:14.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.842+0000 7f3dbae00640 1 --2- 172.21.15.6:0/1283842620 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3db4154dd0 0x7f3db4153210 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f3da002a700 tx=0x7f3da00046c0 comp rx=0 tx=0).stop 2024-04-24T23:16:14.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.843+0000 7f3dbae00640 1 -- 172.21.15.6:0/1283842620 shutdown_connections 2024-04-24T23:16:14.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.843+0000 7f3dbae00640 1 --2- 172.21.15.6:0/1283842620 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f3d9406bf00 0x7f3d9406e3c0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:14.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.843+0000 7f3dbae00640 1 --2- 172.21.15.6:0/1283842620 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3db4154dd0 0x7f3db4153210 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:14.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.843+0000 7f3dbae00640 1 --2- 172.21.15.6:0/1283842620 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3db415b0b0 0x7f3db4153750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:14.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.843+0000 7f3dbae00640 1 --2- 172.21.15.6:0/1283842620 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3db40b35b0 0x7f3db415d2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:14.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.843+0000 7f3dbae00640 1 -- 172.21.15.6:0/1283842620 >> 172.21.15.6:0/1283842620 conn(0x7f3db4066ea0 msgr2=0x7f3db40cb200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:14.846 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574861 got 55834574860 for osd.0 2024-04-24T23:16:14.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.843+0000 7f3dbae00640 1 -- 172.21.15.6:0/1283842620 shutdown_connections 2024-04-24T23:16:14.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.843+0000 7f3dbae00640 1 -- 172.21.15.6:0/1283842620 wait complete. 2024-04-24T23:16:14.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.850+0000 7f7dac400640 1 -- 172.21.15.6:0/3573754216 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f7d9c04f650 con 0x7f7d9c007590 2024-04-24T23:16:14.856 INFO:teuthology.orchestra.run.smithi006.stdout:55834574860 2024-04-24T23:16:14.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.851+0000 7f7da1600640 1 -- 172.21.15.6:0/3573754216 <== mon.2 v2:172.21.15.190:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f7d98079df0 con 0x7f7d9c007590 2024-04-24T23:16:14.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.855+0000 7f7dac400640 1 -- 172.21.15.6:0/3573754216 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f7d8406beb0 msgr2=0x7f7d8406e370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:14.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.855+0000 7f7dac400640 1 --2- 172.21.15.6:0/3573754216 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f7d8406beb0 0x7f7d8406e370 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f7d90006a90 tx=0x7f7d90006a20 comp rx=0 tx=0).stop 2024-04-24T23:16:14.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.855+0000 7f7dac400640 1 -- 172.21.15.6:0/3573754216 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7d9c007590 msgr2=0x7f7d9c177b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:14.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.855+0000 7f7dac400640 1 --2- 172.21.15.6:0/3573754216 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7d9c007590 0x7f7d9c177b20 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f7d98031000 tx=0x7f7d980046c0 comp rx=0 tx=0).stop 2024-04-24T23:16:14.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.856+0000 7f7dac400640 1 -- 172.21.15.6:0/3573754216 shutdown_connections 2024-04-24T23:16:14.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.856+0000 7f7dac400640 1 --2- 172.21.15.6:0/3573754216 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f7d8406beb0 0x7f7d8406e370 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:14.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.857+0000 7f7dac400640 1 --2- 172.21.15.6:0/3573754216 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7d9c0d8fd0 0x7f7d9c175b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:14.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.857+0000 7f7dac400640 1 --2- 172.21.15.6:0/3573754216 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7d9c00e930 0x7f7d9c178060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:14.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.857+0000 7f7dac400640 1 --2- 172.21.15.6:0/3573754216 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7d9c007590 0x7f7d9c177b20 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:14.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.857+0000 7f7dac400640 1 -- 172.21.15.6:0/3573754216 >> 172.21.15.6:0/3573754216 conn(0x7f7d9c054990 msgr2=0x7f7d9c052dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:14.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.857+0000 7f7dac400640 1 -- 172.21.15.6:0/3573754216 shutdown_connections 2024-04-24T23:16:14.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.857+0000 7f7dac400640 1 -- 172.21.15.6:0/3573754216 wait complete. 2024-04-24T23:16:14.865 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574861 got 55834574860 for osd.6 2024-04-24T23:16:14.868 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574862 got 55834574861 for osd.4 2024-04-24T23:16:14.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.866+0000 7fc2fbe00640 1 -- 172.21.15.6:0/1865741887 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7fc2f4039ea0 con 0x7fc2f4037a10 2024-04-24T23:16:14.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.867+0000 7f0666400640 1 -- 172.21.15.6:0/424223695 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f0658050ba0 con 0x7f06580d9090 2024-04-24T23:16:14.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.867+0000 7fc2f2a00640 1 -- 172.21.15.6:0/1865741887 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fc2e80589c0 con 0x7fc2f4037a10 2024-04-24T23:16:14.870 INFO:teuthology.orchestra.run.smithi006.stdout:55834574860 2024-04-24T23:16:14.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.867+0000 7f065cc00640 1 -- 172.21.15.6:0/424223695 <== mon.2 v2:172.21.15.190:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f06540596f0 con 0x7f06580d9090 2024-04-24T23:16:14.870 INFO:teuthology.orchestra.run.smithi006.stdout:55834574859 2024-04-24T23:16:14.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.871+0000 7fc2fbe00640 1 -- 172.21.15.6:0/1865741887 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fc2dc06c030 msgr2=0x7fc2dc06e4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:14.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.871+0000 7fc2fbe00640 1 --2- 172.21.15.6:0/1865741887 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fc2dc06c030 0x7fc2dc06e4f0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fc2f404bac0 tx=0x7fc2fc0ad000 comp rx=0 tx=0).stop 2024-04-24T23:16:14.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.871+0000 7fc2fbe00640 1 -- 172.21.15.6:0/1865741887 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc2f4037a10 msgr2=0x7fc2f4049d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:14.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.871+0000 7fc2fbe00640 1 --2- 172.21.15.6:0/1865741887 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc2f4037a10 0x7fc2f4049d20 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fc2e800b500 tx=0x7fc2e800b9c0 comp rx=0 tx=0).stop 2024-04-24T23:16:14.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.871+0000 7fc2fbe00640 1 -- 172.21.15.6:0/1865741887 shutdown_connections 2024-04-24T23:16:14.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.871+0000 7fc2fbe00640 1 --2- 172.21.15.6:0/1865741887 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fc2dc06c030 0x7fc2dc06e4f0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:14.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.871+0000 7fc2fbe00640 1 --2- 172.21.15.6:0/1865741887 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc2f4037a10 0x7fc2f4049d20 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:14.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.871+0000 7fc2fbe00640 1 --2- 172.21.15.6:0/1865741887 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc2f4037630 0x7fc2f404be30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:14.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.871+0000 7fc2fbe00640 1 --2- 172.21.15.6:0/1865741887 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc2f4007810 0x7fc2f404d0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:14.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.871+0000 7fc2fbe00640 1 -- 172.21.15.6:0/1865741887 >> 172.21.15.6:0/1865741887 conn(0x7fc2f4054610 msgr2=0x7fc2f40e3750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:14.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.871+0000 7fc2fbe00640 1 -- 172.21.15.6:0/1865741887 shutdown_connections 2024-04-24T23:16:14.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.871+0000 7fc2fbe00640 1 -- 172.21.15.6:0/1865741887 wait complete. 2024-04-24T23:16:14.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.871+0000 7f0666400640 1 -- 172.21.15.6:0/424223695 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f064006be80 msgr2=0x7f064006e340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:14.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.871+0000 7f0666400640 1 --2- 172.21.15.6:0/424223695 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f064006be80 0x7f064006e340 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f0658039f90 tx=0x7f06600ad000 comp rx=0 tx=0).stop 2024-04-24T23:16:14.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.871+0000 7f0666400640 1 -- 172.21.15.6:0/424223695 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f06580d9090 msgr2=0x7f065817a3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:14.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.871+0000 7f0666400640 1 --2- 172.21.15.6:0/424223695 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f06580d9090 0x7f065817a3d0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f065400b3f0 tx=0x7f065400b8b0 comp rx=0 tx=0).stop 2024-04-24T23:16:14.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.872+0000 7f0666400640 1 -- 172.21.15.6:0/424223695 shutdown_connections 2024-04-24T23:16:14.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.872+0000 7f0666400640 1 --2- 172.21.15.6:0/424223695 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f064006be80 0x7f064006e340 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:14.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.872+0000 7f0666400640 1 --2- 172.21.15.6:0/424223695 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f06580d9090 0x7f065817a3d0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:14.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.872+0000 7f0666400640 1 --2- 172.21.15.6:0/424223695 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0658050180 0x7f06580d5010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:14.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.872+0000 7f0666400640 1 --2- 172.21.15.6:0/424223695 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0658037870 0x7f06580da0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:14.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.872+0000 7f0666400640 1 -- 172.21.15.6:0/424223695 >> 172.21.15.6:0/424223695 conn(0x7f0658054610 msgr2=0x7f0658038ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:14.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.872+0000 7f0666400640 1 -- 172.21.15.6:0/424223695 shutdown_connections 2024-04-24T23:16:14.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.872+0000 7f0666400640 1 -- 172.21.15.6:0/424223695 wait complete. 2024-04-24T23:16:14.878 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574862 got 55834574861 for osd.3 2024-04-24T23:16:14.884 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574861 got 55834574860 for osd.7 2024-04-24T23:16:14.893 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574861 got 55834574860 for osd.2 2024-04-24T23:16:14.902 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574861 got 55834574859 for osd.1 2024-04-24T23:16:14.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.908+0000 7f981f800640 1 Processor -- start 2024-04-24T23:16:14.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.909+0000 7f981f800640 1 -- start start 2024-04-24T23:16:14.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.910+0000 7f981f800640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f981806d720 0x7f981806c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:14.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.910+0000 7f981f800640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f981814ebb0 con 0x7f981814e7d0 2024-04-24T23:16:14.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.910+0000 7f981f800640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9818157920 con 0x7f98180c9770 2024-04-24T23:16:14.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.910+0000 7f981f800640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f9818071000 con 0x7f981806d720 2024-04-24T23:16:14.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.910+0000 7f981ce00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f981814e7d0 0x7f98180c7c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:39628/0 (socket says 172.21.15.6:39628) 2024-04-24T23:16:14.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.910+0000 7f981ce00640 1 -- 172.21.15.6:0/3082086695 learned_addr learned my addr 172.21.15.6:0/3082086695 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:16:14.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.910+0000 7f9817e00640 1 --2- 172.21.15.6:0/3082086695 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f981806d720 0x7f981806c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:14.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.911+0000 7f9817400640 1 -- 172.21.15.6:0/3082086695 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 199339199 0 0) 0x7f980c001110 con 0x7f981814e7d0 2024-04-24T23:16:14.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.911+0000 7f9817400640 1 -- 172.21.15.6:0/3082086695 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9800003520 con 0x7f981814e7d0 2024-04-24T23:16:14.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.911+0000 7f9817400640 1 -- 172.21.15.6:0/3082086695 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1391346085 0 0) 0x7f9818157920 con 0x7f98180c9770 2024-04-24T23:16:14.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.911+0000 7f9817400640 1 -- 172.21.15.6:0/3082086695 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f980c001110 con 0x7f98180c9770 2024-04-24T23:16:14.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.911+0000 7f9817e00640 1 -- 172.21.15.6:0/3082086695 >> v1:172.21.15.190:6789/0 conn(0x7f98180c9770 legacy=0x7f9818162220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:14.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.911+0000 7f9817e00640 1 -- 172.21.15.6:0/3082086695 >> v1:172.21.15.6:6789/0 conn(0x7f981814e7d0 legacy=0x7f98180c7c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:14.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.911+0000 7f9817e00640 1 -- 172.21.15.6:0/3082086695 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9818071250 con 0x7f981806d720 2024-04-24T23:16:14.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.912+0000 7f9817e00640 1 --2- 172.21.15.6:0/3082086695 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f981806d720 0x7f981806c3a0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f98040097d0 tx=0x7f980402a210 comp rx=0 tx=0).ready entity=mon.2 client_cookie=81feb69eeb163633 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:14.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.912+0000 7f9817400640 1 -- 172.21.15.6:0/3082086695 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9804037070 con 0x7f981806d720 2024-04-24T23:16:14.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.912+0000 7f9817400640 1 -- 172.21.15.6:0/3082086695 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9804032440 con 0x7f981806d720 2024-04-24T23:16:14.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.912+0000 7f9817400640 1 -- 172.21.15.6:0/3082086695 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9804032810 con 0x7f981806d720 2024-04-24T23:16:14.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.913+0000 7f981f800640 1 -- 172.21.15.6:0/3082086695 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f981806d720 msgr2=0x7f981806c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:14.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.913+0000 7f981f800640 1 --2- 172.21.15.6:0/3082086695 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f981806d720 0x7f981806c3a0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f98040097d0 tx=0x7f980402a210 comp rx=0 tx=0).stop 2024-04-24T23:16:14.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.913+0000 7f981f800640 1 -- 172.21.15.6:0/3082086695 shutdown_connections 2024-04-24T23:16:14.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.913+0000 7f981f800640 1 --2- 172.21.15.6:0/3082086695 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f981806d720 0x7f981806c3a0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:14.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.913+0000 7f981f800640 1 -- 172.21.15.6:0/3082086695 >> 172.21.15.6:0/3082086695 conn(0x7f98180c8c50 msgr2=0x7f98180c9060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:14.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.913+0000 7f981f800640 1 -- 172.21.15.6:0/3082086695 shutdown_connections 2024-04-24T23:16:14.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.913+0000 7f981f800640 1 -- 172.21.15.6:0/3082086695 wait complete. 2024-04-24T23:16:14.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.914+0000 7f981f800640 1 Processor -- start 2024-04-24T23:16:14.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.914+0000 7f981f800640 1 -- start start 2024-04-24T23:16:14.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.915+0000 7f981f800640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f981806d720 0x7f981815a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:14.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.915+0000 7f981f800640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f98180c9770 0x7f98181f3f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:14.919 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.915+0000 7f981f800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f981814e7d0 0x7f98181f4490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:14.919 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.915+0000 7f981f800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f9818157be0 con 0x7f981814e7d0 2024-04-24T23:16:14.919 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.915+0000 7f981f800640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f9818157f50 con 0x7f981806d720 2024-04-24T23:16:14.919 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.915+0000 7f981f800640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f98180c8370 con 0x7f98180c9770 2024-04-24T23:16:14.919 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.916+0000 7f981d800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f981814e7d0 0x7f98181f4490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:14.919 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.916+0000 7f981d800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f981814e7d0 0x7f98181f4490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:43348/0 (socket says 172.21.15.6:43348) 2024-04-24T23:16:14.919 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.916+0000 7f981d800640 1 -- 172.21.15.6:0/3927498563 learned_addr learned my addr 172.21.15.6:0/3927498563 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:16:14.919 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.916+0000 7f9817e00640 1 --2- 172.21.15.6:0/3927498563 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f98180c9770 0x7f98181f3f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:14.919 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.916+0000 7f981ce00640 1 --2- 172.21.15.6:0/3927498563 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f981806d720 0x7f981815a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:14.919 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.916+0000 7f9817e00640 1 -- 172.21.15.6:0/3927498563 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f981806d720 msgr2=0x7f981815a0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:14.919 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.916+0000 7f9817e00640 1 --2- 172.21.15.6:0/3927498563 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f981806d720 0x7f981815a0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:14.919 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.916+0000 7f9817e00640 1 -- 172.21.15.6:0/3927498563 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f981814e7d0 msgr2=0x7f98181f4490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:14.919 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.916+0000 7f9817e00640 1 --2- 172.21.15.6:0/3927498563 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f981814e7d0 0x7f98181f4490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:14.919 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.916+0000 7f9817e00640 1 -- 172.21.15.6:0/3927498563 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9818158a80 con 0x7f98180c9770 2024-04-24T23:16:14.920 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.916+0000 7f981d800640 1 --2- 172.21.15.6:0/3927498563 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f981814e7d0 0x7f98181f4490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:16:14.920 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.917+0000 7f9817e00640 1 --2- 172.21.15.6:0/3927498563 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f98180c9770 0x7f98181f3f50 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f9804031000 tx=0x7f98040046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:14.920 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.917+0000 7f9814c00640 1 -- 172.21.15.6:0/3927498563 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9804041030 con 0x7f98180c9770 2024-04-24T23:16:14.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.917+0000 7f9814c00640 1 -- 172.21.15.6:0/3927498563 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f980403d4e0 con 0x7f98180c9770 2024-04-24T23:16:14.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.917+0000 7f981f800640 1 -- 172.21.15.6:0/3927498563 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f981815a5e0 con 0x7f98180c9770 2024-04-24T23:16:14.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.917+0000 7f9814c00640 1 -- 172.21.15.6:0/3927498563 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9804037040 con 0x7f98180c9770 2024-04-24T23:16:14.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.917+0000 7f981f800640 1 -- 172.21.15.6:0/3927498563 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f9818159000 con 0x7f98180c9770 2024-04-24T23:16:14.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.918+0000 7f981f800640 1 -- 172.21.15.6:0/3927498563 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9818159bd0 con 0x7f98180c9770 2024-04-24T23:16:14.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.919+0000 7f9814c00640 1 -- 172.21.15.6:0/3927498563 <== mon.2 v2:172.21.15.190:3301/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f980403d8b0 con 0x7f98180c9770 2024-04-24T23:16:14.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.920+0000 7f9814c00640 1 --2- 172.21.15.6:0/3927498563 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f97f806be30 0x7f97f806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:14.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.920+0000 7f9814c00640 1 -- 172.21.15.6:0/3927498563 <== mon.2 v2:172.21.15.190:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f98040ab260 con 0x7f98180c9770 2024-04-24T23:16:14.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.920+0000 7f981ce00640 1 --2- 172.21.15.6:0/3927498563 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f97f806be30 0x7f97f806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:14.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.921+0000 7f981ce00640 1 --2- 172.21.15.6:0/3927498563 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f97f806be30 0x7f97f806e2f0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f980c0013b0 tx=0x7f980c007d30 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:14.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:14.925+0000 7f9814c00640 1 -- 172.21.15.6:0/3927498563 <== mon.2 v2:172.21.15.190:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f980404b020 con 0x7f98180c9770 2024-04-24T23:16:15.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:15.118+0000 7f981f800640 1 -- 172.21.15.6:0/3927498563 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f981815bd30 con 0x7f98180c9770 2024-04-24T23:16:15.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:15.118+0000 7f9814c00640 1 -- 172.21.15.6:0/3927498563 <== mon.2 v2:172.21.15.190:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f9804079270 con 0x7f98180c9770 2024-04-24T23:16:15.121 INFO:teuthology.orchestra.run.smithi006.stdout:55834574860 2024-04-24T23:16:15.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:15.121+0000 7f981f800640 1 -- 172.21.15.6:0/3927498563 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f97f806be30 msgr2=0x7f97f806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:15.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:15.121+0000 7f981f800640 1 --2- 172.21.15.6:0/3927498563 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f97f806be30 0x7f97f806e2f0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f980c0013b0 tx=0x7f980c007d30 comp rx=0 tx=0).stop 2024-04-24T23:16:15.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:15.121+0000 7f981f800640 1 -- 172.21.15.6:0/3927498563 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f98180c9770 msgr2=0x7f98181f3f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:15.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:15.121+0000 7f981f800640 1 --2- 172.21.15.6:0/3927498563 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f98180c9770 0x7f98181f3f50 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f9804031000 tx=0x7f98040046c0 comp rx=0 tx=0).stop 2024-04-24T23:16:15.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:15.121+0000 7f981f800640 1 -- 172.21.15.6:0/3927498563 shutdown_connections 2024-04-24T23:16:15.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:15.121+0000 7f981f800640 1 --2- 172.21.15.6:0/3927498563 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f97f806be30 0x7f97f806e2f0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:15.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:15.121+0000 7f981f800640 1 --2- 172.21.15.6:0/3927498563 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f98180c9770 0x7f98181f3f50 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:15.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:15.121+0000 7f981f800640 1 --2- 172.21.15.6:0/3927498563 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f981814e7d0 0x7f98181f4490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:15.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:15.121+0000 7f981f800640 1 --2- 172.21.15.6:0/3927498563 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f981806d720 0x7f981815a0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:15.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:15.121+0000 7f981f800640 1 -- 172.21.15.6:0/3927498563 >> 172.21.15.6:0/3927498563 conn(0x7f98180c8c50 msgr2=0x7f98180c5080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:15.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:15.121+0000 7f981f800640 1 -- 172.21.15.6:0/3927498563 shutdown_connections 2024-04-24T23:16:15.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:15.121+0000 7f981f800640 1 -- 172.21.15.6:0/3927498563 wait complete. 2024-04-24T23:16:15.141 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574861 got 55834574860 for osd.5 2024-04-24T23:16:15.847 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-24T23:16:15.866 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-04-24T23:16:15.869 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-04-24T23:16:15.879 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-24T23:16:15.885 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-04-24T23:16:15.894 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-24T23:16:15.903 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-24T23:16:16.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.113+0000 7f5e55e00640 1 Processor -- start 2024-04-24T23:16:16.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.113+0000 7f5e55e00640 1 -- start start 2024-04-24T23:16:16.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.117+0000 7f5e55e00640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f5e480deda0 0x7f5e48037160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:16.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.117+0000 7f5e55e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f5e480376a0 0x7f5e48038520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:16.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.117+0000 7f5e55e00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5e480df1e0 con 0x7f5e480d9090 2024-04-24T23:16:16.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.117+0000 7f5e55e00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f5e48037ae0 con 0x7f5e480376a0 2024-04-24T23:16:16.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.118+0000 7f5e55e00640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f5e48038090 con 0x7f5e480deda0 2024-04-24T23:16:16.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.118+0000 7f5e4ea00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f5e480d9090 0x7f5e48050180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:39644/0 (socket says 172.21.15.6:39644) 2024-04-24T23:16:16.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.118+0000 7f5e4ea00640 1 -- 172.21.15.6:0/1753391979 learned_addr learned my addr 172.21.15.6:0/1753391979 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:16:16.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.118+0000 7f5e4e000640 1 --2- 172.21.15.6:0/1753391979 >> v2:172.21.15.190:3300/0 conn(0x7f5e480deda0 0x7f5e48037160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:16.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.120+0000 7f5e4f400640 1 --2- 172.21.15.6:0/1753391979 >> v2:172.21.15.6:3300/0 conn(0x7f5e480376a0 0x7f5e48038520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:16.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.120+0000 7f5e4d600640 1 -- 172.21.15.6:0/1753391979 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3165429151 0 0) 0x7f5e480df1e0 con 0x7f5e480d9090 2024-04-24T23:16:16.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.120+0000 7f5e4d600640 1 -- 172.21.15.6:0/1753391979 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5e34003520 con 0x7f5e480d9090 2024-04-24T23:16:16.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.120+0000 7f5e4d600640 1 -- 172.21.15.6:0/1753391979 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3668343022 0 0) 0x7f5e34003520 con 0x7f5e480d9090 2024-04-24T23:16:16.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.120+0000 7f5e4d600640 1 -- 172.21.15.6:0/1753391979 >> v2:172.21.15.190:3300/0 conn(0x7f5e480deda0 msgr2=0x7f5e48037160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:16:16.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.120+0000 7f5e4d600640 1 --2- 172.21.15.6:0/1753391979 >> v2:172.21.15.190:3300/0 conn(0x7f5e480deda0 0x7f5e48037160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:16.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.120+0000 7f5e4d600640 1 -- 172.21.15.6:0/1753391979 >> v2:172.21.15.6:3300/0 conn(0x7f5e480376a0 msgr2=0x7f5e48038520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:16:16.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.121+0000 7f5e4d600640 1 --2- 172.21.15.6:0/1753391979 >> v2:172.21.15.6:3300/0 conn(0x7f5e480376a0 0x7f5e48038520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:16.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.121+0000 7f5e4d600640 1 -- 172.21.15.6:0/1753391979 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e480df4a0 con 0x7f5e480d9090 2024-04-24T23:16:16.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.121+0000 7f5e4d600640 1 -- 172.21.15.6:0/1753391979 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3463268701 0 0) 0x7f5e3c0016c0 con 0x7f5e480d9090 2024-04-24T23:16:16.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.121+0000 7f5e4d600640 1 -- 172.21.15.6:0/1753391979 >> v1:172.21.15.6:6789/0 conn(0x7f5e480d9090 legacy=0x7f5e48050180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:16.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.121+0000 7f5e4d600640 1 --2- 172.21.15.6:0/1753391979 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5e340039b0 0x7f5e34003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:16.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.121+0000 7f5e4d600640 1 --2- 172.21.15.6:0/1753391979 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5e34004800 0x7f5e34004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:16.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.121+0000 7f5e4d600640 1 --2- 172.21.15.6:0/1753391979 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5e340051c0 0x7f5e3400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:16.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.121+0000 7f5e4d600640 1 -- 172.21.15.6:0/1753391979 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f5e48037ae0 con 0x7f5e34004800 2024-04-24T23:16:16.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.121+0000 7f5e4d600640 1 -- 172.21.15.6:0/1753391979 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f5e48038090 con 0x7f5e340051c0 2024-04-24T23:16:16.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.121+0000 7f5e4d600640 1 -- 172.21.15.6:0/1753391979 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f5e3400c030 con 0x7f5e340039b0 2024-04-24T23:16:16.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.121+0000 7f5e4f400640 1 --2- 172.21.15.6:0/1753391979 >> v2:172.21.15.6:3300/0 conn(0x7f5e480376a0 0x7f5e48038520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:16:16.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.122+0000 7f5e4ea00640 1 --2- 172.21.15.6:0/1753391979 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5e340039b0 0x7f5e34003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:16.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.122+0000 7f5e4ea00640 1 -- 172.21.15.6:0/1753391979 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5e340051c0 msgr2=0x7f5e3400b9c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-24T23:16:16.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.122+0000 7f5e4ea00640 1 --2- 172.21.15.6:0/1753391979 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5e340051c0 0x7f5e3400b9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:16.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.122+0000 7f5e4ea00640 1 -- 172.21.15.6:0/1753391979 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5e34004800 msgr2=0x7f5e34004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:16:16.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.122+0000 7f5e4ea00640 1 --2- 172.21.15.6:0/1753391979 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5e34004800 0x7f5e34004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:16.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.122+0000 7f5e4ea00640 1 -- 172.21.15.6:0/1753391979 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e480df4a0 con 0x7f5e340039b0 2024-04-24T23:16:16.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.123+0000 7f5e4f400640 1 -- 172.21.15.6:0/1753391979 reap_dead start 2024-04-24T23:16:16.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.124+0000 7f5e4ea00640 1 --2- 172.21.15.6:0/1753391979 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5e340039b0 0x7f5e34003dd0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f5e3c00c290 tx=0x7f5e34005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=699e3eca81e17a9e server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:16.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.124+0000 7f5e4d600640 1 -- 172.21.15.6:0/1753391979 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e3c007d60 con 0x7f5e340039b0 2024-04-24T23:16:16.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.124+0000 7f5e4d600640 1 -- 172.21.15.6:0/1753391979 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5e3c005af0 con 0x7f5e340039b0 2024-04-24T23:16:16.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.125+0000 7f5e55e00640 1 -- 172.21.15.6:0/1753391979 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5e340039b0 msgr2=0x7f5e34003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:16.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.125+0000 7f5e55e00640 1 --2- 172.21.15.6:0/1753391979 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5e340039b0 0x7f5e34003dd0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f5e3c00c290 tx=0x7f5e34005830 comp rx=0 tx=0).stop 2024-04-24T23:16:16.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.126+0000 7f5e55e00640 1 -- 172.21.15.6:0/1753391979 shutdown_connections 2024-04-24T23:16:16.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.126+0000 7f5e55e00640 1 --2- 172.21.15.6:0/1753391979 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5e340039b0 0x7f5e34003dd0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:16.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.126+0000 7f5e55e00640 1 -- 172.21.15.6:0/1753391979 >> 172.21.15.6:0/1753391979 conn(0x7f5e48054610 msgr2=0x7f5e48054a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:16.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.127+0000 7f5e55e00640 1 -- 172.21.15.6:0/1753391979 shutdown_connections 2024-04-24T23:16:16.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.127+0000 7f5e55e00640 1 -- 172.21.15.6:0/1753391979 wait complete. 2024-04-24T23:16:16.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.130+0000 7f5e55e00640 1 Processor -- start 2024-04-24T23:16:16.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.130+0000 7f5e55e00640 1 -- start start 2024-04-24T23:16:16.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.133+0000 7f5e55e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5e340039b0 0x7f5e48052950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:16.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.133+0000 7f5e55e00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5e480523f0 0x7f5e48181ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:16.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.133+0000 7f5e55e00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5e48185030 0x7f5e48006ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:16.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.134+0000 7f5e55e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f5e48037d20 con 0x7f5e340039b0 2024-04-24T23:16:16.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.134+0000 7f5e55e00640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f5e480dfa60 con 0x7f5e48185030 2024-04-24T23:16:16.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.134+0000 7f5e55e00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f5e480d9470 con 0x7f5e480523f0 2024-04-24T23:16:16.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.135+0000 7f5e4ea00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5e340039b0 0x7f5e48052950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:16.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.135+0000 7f5e4ea00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5e340039b0 0x7f5e48052950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:43354/0 (socket says 172.21.15.6:43354) 2024-04-24T23:16:16.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.135+0000 7f5e4ea00640 1 -- 172.21.15.6:0/1916418933 learned_addr learned my addr 172.21.15.6:0/1916418933 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:16:16.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.136+0000 7f5e4ea00640 1 -- 172.21.15.6:0/1916418933 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5e480523f0 msgr2=0x7f5e48181ba0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-24T23:16:16.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.136+0000 7f5e4ea00640 1 --2- 172.21.15.6:0/1916418933 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5e480523f0 0x7f5e48181ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:16.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.137+0000 7f5e4f400640 1 --2- 172.21.15.6:0/1916418933 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5e48185030 0x7f5e48006ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:16.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.137+0000 7f5e4ea00640 1 -- 172.21.15.6:0/1916418933 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5e48185030 msgr2=0x7f5e48006ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:16.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.137+0000 7f5e4ea00640 1 --2- 172.21.15.6:0/1916418933 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5e48185030 0x7f5e48006ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:16.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.137+0000 7f5e4ea00640 1 -- 172.21.15.6:0/1916418933 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e48053e70 con 0x7f5e340039b0 2024-04-24T23:16:16.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.138+0000 7f5e4ea00640 1 --2- 172.21.15.6:0/1916418933 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5e340039b0 0x7f5e48052950 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f5e3c00c3b0 tx=0x7f5e3c001030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:16.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.139+0000 7f5e47400640 1 -- 172.21.15.6:0/1916418933 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e3c042020 con 0x7f5e340039b0 2024-04-24T23:16:16.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.139+0000 7f5e47400640 1 -- 172.21.15.6:0/1916418933 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5e3c0049e0 con 0x7f5e340039b0 2024-04-24T23:16:16.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.139+0000 7f5e55e00640 1 -- 172.21.15.6:0/1916418933 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e481810e0 con 0x7f5e340039b0 2024-04-24T23:16:16.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.139+0000 7f5e55e00640 1 -- 172.21.15.6:0/1916418933 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e481813a0 con 0x7f5e340039b0 2024-04-24T23:16:16.142 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-04-24T23:16:16.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.142+0000 7f5e55e00640 1 -- 172.21.15.6:0/1916418933 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5e480052c0 con 0x7f5e340039b0 2024-04-24T23:16:16.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.143+0000 7f5e47400640 1 -- 172.21.15.6:0/1916418933 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e3c007d60 con 0x7f5e340039b0 2024-04-24T23:16:16.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.150+0000 7f5e47400640 1 -- 172.21.15.6:0/1916418933 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f5e3c045020 con 0x7f5e340039b0 2024-04-24T23:16:16.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.152+0000 7f5e47400640 1 --2- 172.21.15.6:0/1916418933 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f5e3006bf30 0x7f5e3006e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:16.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.153+0000 7f5e47400640 1 -- 172.21.15.6:0/1916418933 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f5e3c0abb40 con 0x7f5e340039b0 2024-04-24T23:16:16.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.153+0000 7f5e4e000640 1 --2- 172.21.15.6:0/1916418933 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f5e3006bf30 0x7f5e3006e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:16.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.157+0000 7f5e47400640 1 -- 172.21.15.6:0/1916418933 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f5e3c04c030 con 0x7f5e340039b0 2024-04-24T23:16:16.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.158+0000 7f5e4e000640 1 --2- 172.21.15.6:0/1916418933 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f5e3006bf30 0x7f5e3006e3f0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f5e48007390 tx=0x7f5e40009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:16.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.176+0000 7f9908400640 1 Processor -- start 2024-04-24T23:16:16.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.176+0000 7f9908400640 1 -- start start 2024-04-24T23:16:16.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.178+0000 7f9908400640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f9900159210 0x7f99000c74a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:16.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.178+0000 7f9908400640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f99000c8420 0x7f9900162500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:16.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.179+0000 7f9908400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f9900162a40 0x7f9900162e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:16.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.179+0000 7f9908400640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f99000c6a50 con 0x7f9900162a40 2024-04-24T23:16:16.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.179+0000 7f9908400640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f99000c7010 con 0x7f9900159210 2024-04-24T23:16:16.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.179+0000 7f9908400640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f99000c89a0 con 0x7f99000c8420 2024-04-24T23:16:16.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.179+0000 7f9906400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f9900162a40 0x7f9900162e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:16.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.179+0000 7f9906400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f9900162a40 0x7f9900162e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:43358/0 (socket says 172.21.15.6:43358) 2024-04-24T23:16:16.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.180+0000 7f24b9800640 1 Processor -- start 2024-04-24T23:16:16.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.180+0000 7f24b9800640 1 -- start start 2024-04-24T23:16:16.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.179+0000 7f9906400640 1 -- 172.21.15.6:0/143081999 learned_addr learned my addr 172.21.15.6:0/143081999 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:16:16.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.180+0000 7f9905a00640 1 --2- 172.21.15.6:0/143081999 >> v2:172.21.15.190:3300/0 conn(0x7f9900159210 0x7f99000c74a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:16.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.180+0000 7f9905000640 1 --2- 172.21.15.6:0/143081999 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f99000c8420 0x7f9900162500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:16.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.180+0000 7f9905a00640 1 -- 172.21.15.6:0/143081999 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f99000c8420 msgr2=0x7f9900162500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:16:16.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.181+0000 7f9905a00640 1 --2- 172.21.15.6:0/143081999 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f99000c8420 0x7f9900162500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:16.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.181+0000 7f9905a00640 1 -- 172.21.15.6:0/143081999 >> v2:172.21.15.6:3300/0 conn(0x7f9900162a40 msgr2=0x7f9900162e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:16:16.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.181+0000 7f0fb6600640 1 Processor -- start 2024-04-24T23:16:16.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.183+0000 7f9905a00640 1 --2- 172.21.15.6:0/143081999 >> v2:172.21.15.6:3300/0 conn(0x7f9900162a40 0x7f9900162e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:16.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.183+0000 7f9905a00640 1 -- 172.21.15.6:0/143081999 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f990006cdb0 con 0x7f9900159210 2024-04-24T23:16:16.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.182+0000 7f0fb6600640 1 -- start start 2024-04-24T23:16:16.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.183+0000 7f9905000640 1 --2- 172.21.15.6:0/143081999 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f99000c8420 0x7f9900162500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:16:16.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.183+0000 7f24b9800640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f24b40cc790 0x7f24b415b5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:16.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.184+0000 7f24b3400640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f24b40cc790 0x7f24b415b5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:16.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.184+0000 7f24b3400640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f24b40cc790 0x7f24b415b5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:34040/0 (socket says 172.21.15.6:34040) 2024-04-24T23:16:16.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.184+0000 7f24b2a00640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f24b4154d00 0x7f24b406b950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.6:45130/0 (socket says 172.21.15.6:45130) 2024-04-24T23:16:16.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.184+0000 7f24b3400640 1 -- 172.21.15.6:0/3556079617 learned_addr learned my addr 172.21.15.6:0/3556079617 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:16:16.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.185+0000 7f24b9800640 1 -- 172.21.15.6:0/3556079617 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f24b415e360 con 0x7f24b40cae00 2024-04-24T23:16:16.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.185+0000 7f24b9800640 1 -- 172.21.15.6:0/3556079617 --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f24b415e640 con 0x7f24b4154d00 2024-04-24T23:16:16.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.185+0000 7f24b9800640 1 -- 172.21.15.6:0/3556079617 --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f24b415bae0 con 0x7f24b40cc790 2024-04-24T23:16:16.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.185+0000 7f0fb6600640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0fb006ef70 0x7f0fb00cef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:16.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.185+0000 7f0fb6600640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f0fb00cab20 0x7f0fb015c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:16.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.185+0000 7f0fb6600640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0fb006e200 con 0x7f0fb006f350 2024-04-24T23:16:16.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.185+0000 7f0fb6600640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f0fb0159870 con 0x7f0fb00cab20 2024-04-24T23:16:16.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.185+0000 7f0fb6600640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f0fb0158a10 con 0x7f0fb006ef70 2024-04-24T23:16:16.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.186+0000 7f0fafe00640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f0fb006f350 0x7f0fb00c8fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.6:45136/0 (socket says 172.21.15.6:45136) 2024-04-24T23:16:16.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.186+0000 7f0fafe00640 1 -- 172.21.15.6:0/3948633507 learned_addr learned my addr 172.21.15.6:0/3948633507 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:16:16.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.186+0000 7f24b3400640 1 -- 172.21.15.6:0/3556079617 >> v1:172.21.15.190:6789/0 conn(0x7f24b4154d00 legacy=0x7f24b406b950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:16.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.186+0000 7f24b3400640 1 -- 172.21.15.6:0/3556079617 >> v1:172.21.15.6:6789/0 conn(0x7f24b40cae00 legacy=0x7f24b40cc080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:16.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.186+0000 7f24b3400640 1 -- 172.21.15.6:0/3556079617 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24b415e920 con 0x7f24b40cc790 2024-04-24T23:16:16.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.187+0000 7f0fae000640 1 -- 172.21.15.6:0/3948633507 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1033010291 0 0) 0x7f0fb006e200 con 0x7f0fb006f350 2024-04-24T23:16:16.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.187+0000 7f0fae000640 1 -- 172.21.15.6:0/3948633507 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0f94003520 con 0x7f0fb006f350 2024-04-24T23:16:16.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.187+0000 7f0faea00640 1 --2- 172.21.15.6:0/3948633507 >> v2:172.21.15.190:3300/0 conn(0x7f0fb00cab20 0x7f0fb015c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:16.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.188+0000 7f9906400640 1 --2- 172.21.15.6:0/143081999 >> v2:172.21.15.6:3300/0 conn(0x7f9900162a40 0x7f9900162e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:16:16.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.188+0000 7f0faf400640 1 --2- 172.21.15.6:0/3948633507 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0fb006ef70 0x7f0fb00cef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:16.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.188+0000 7f0fae000640 1 -- 172.21.15.6:0/3948633507 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2026372058 0 0) 0x7f0f94003520 con 0x7f0fb006f350 2024-04-24T23:16:16.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.188+0000 7f0fae000640 1 -- 172.21.15.6:0/3948633507 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0fb006ef70 msgr2=0x7f0fb00cef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:16:16.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.188+0000 7f9905a00640 1 --2- 172.21.15.6:0/143081999 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9900159210 0x7f99000c74a0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f990006c090 tx=0x7f98f402a680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8a3a0a33a801a6b7 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:16.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.188+0000 7f0fae000640 1 --2- 172.21.15.6:0/3948633507 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0fb006ef70 0x7f0fb00cef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:16.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.188+0000 7f0fae000640 1 -- 172.21.15.6:0/3948633507 >> v2:172.21.15.190:3300/0 conn(0x7f0fb00cab20 msgr2=0x7f0fb015c1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:16:16.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.188+0000 7f0fae000640 1 --2- 172.21.15.6:0/3948633507 >> v2:172.21.15.190:3300/0 conn(0x7f0fb00cab20 0x7f0fb015c1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:16.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.188+0000 7f0fae000640 1 -- 172.21.15.6:0/3948633507 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0fb0158cf0 con 0x7f0fb006f350 2024-04-24T23:16:16.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.188+0000 7f0fae000640 1 -- 172.21.15.6:0/3948633507 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3463268701 0 0) 0x7f0f9c001670 con 0x7f0fb006f350 2024-04-24T23:16:16.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.188+0000 7f0fae000640 1 -- 172.21.15.6:0/3948633507 >> v1:172.21.15.190:6789/0 conn(0x7f0fb006f350 legacy=0x7f0fb00c8fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:16.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.188+0000 7f0fae000640 1 --2- 172.21.15.6:0/3948633507 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0f940039b0 0x7f0f94003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:16.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.188+0000 7f0fae000640 1 --2- 172.21.15.6:0/3948633507 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0f94004800 0x7f0f94004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:16.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.188+0000 7f0fae000640 1 --2- 172.21.15.6:0/3948633507 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0f940051c0 0x7f0f9400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:16.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.188+0000 7f0fae000640 1 -- 172.21.15.6:0/3948633507 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f0fb0159870 con 0x7f0f94004800 2024-04-24T23:16:16.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.188+0000 7f0fae000640 1 -- 172.21.15.6:0/3948633507 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f0fb0158a10 con 0x7f0f940051c0 2024-04-24T23:16:16.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.188+0000 7f0fae000640 1 -- 172.21.15.6:0/3948633507 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f0f9400bfe0 con 0x7f0f940039b0 2024-04-24T23:16:16.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.188+0000 7f24b3400640 1 --2- 172.21.15.6:0/3556079617 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f24b40cc790 0x7f24b415b5a0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f24ac009e50 tx=0x7f24ac02a740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=760fa03f1f86508e server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:16.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.189+0000 7f0faf400640 1 --2- 172.21.15.6:0/3948633507 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0f940039b0 0x7f0f94003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:16.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.189+0000 7f0fafe00640 1 --2- 172.21.15.6:0/3948633507 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0f940051c0 0x7f0f9400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:16.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.189+0000 7f24b1600640 1 -- 172.21.15.6:0/3556079617 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24ac036070 con 0x7f24b40cc790 2024-04-24T23:16:16.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.189+0000 7f24b1600640 1 -- 172.21.15.6:0/3556079617 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f24ac031600 con 0x7f24b40cc790 2024-04-24T23:16:16.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.189+0000 7f24b1600640 1 -- 172.21.15.6:0/3556079617 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24ac0319d0 con 0x7f24b40cc790 2024-04-24T23:16:16.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.189+0000 7f0faf400640 1 -- 172.21.15.6:0/3948633507 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0f940051c0 msgr2=0x7f0f9400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:16:16.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.189+0000 7f0faf400640 1 --2- 172.21.15.6:0/3948633507 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0f940051c0 0x7f0f9400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:16.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.190+0000 7f0faf400640 1 -- 172.21.15.6:0/3948633507 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0f94004800 msgr2=0x7f0f94004c80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-24T23:16:16.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.190+0000 7f0faea00640 1 --2- 172.21.15.6:0/3948633507 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0f94004800 0x7f0f94004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:16.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.190+0000 7f0faf400640 1 --2- 172.21.15.6:0/3948633507 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0f94004800 0x7f0f94004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:16.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.190+0000 7f24b9800640 1 -- 172.21.15.6:0/3556079617 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f24b40cc790 msgr2=0x7f24b415b5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:16.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.190+0000 7f24b9800640 1 --2- 172.21.15.6:0/3556079617 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f24b40cc790 0x7f24b415b5a0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f24ac009e50 tx=0x7f24ac02a740 comp rx=0 tx=0).stop 2024-04-24T23:16:16.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.191+0000 7f24b9800640 1 -- 172.21.15.6:0/3556079617 shutdown_connections 2024-04-24T23:16:16.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.191+0000 7f24b9800640 1 --2- 172.21.15.6:0/3556079617 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f24b40cc790 0x7f24b415b5a0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:16.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.191+0000 7f24b9800640 1 -- 172.21.15.6:0/3556079617 >> 172.21.15.6:0/3556079617 conn(0x7f24b4072c60 msgr2=0x7f24b406d8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:16.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.191+0000 7f24b9800640 1 -- 172.21.15.6:0/3556079617 shutdown_connections 2024-04-24T23:16:16.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.190+0000 7f0faf400640 1 -- 172.21.15.6:0/3948633507 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0fb0158cf0 con 0x7f0f940039b0 2024-04-24T23:16:16.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.191+0000 7f0faea00640 1 --2- 172.21.15.6:0/3948633507 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0f94004800 0x7f0f94004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:16:16.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.191+0000 7f24b9800640 1 -- 172.21.15.6:0/3556079617 wait complete. 2024-04-24T23:16:16.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.192+0000 7f98ffe00640 1 -- 172.21.15.6:0/143081999 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98f4007740 con 0x7f9900159210 2024-04-24T23:16:16.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.192+0000 7f98ffe00640 1 -- 172.21.15.6:0/143081999 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f98f40078e0 con 0x7f9900159210 2024-04-24T23:16:16.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.192+0000 7f98ffe00640 1 -- 172.21.15.6:0/143081999 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98f4007cf0 con 0x7f9900159210 2024-04-24T23:16:16.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.193+0000 7f9908400640 1 -- 172.21.15.6:0/143081999 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9900159210 msgr2=0x7f99000c74a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:16.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.193+0000 7f9908400640 1 --2- 172.21.15.6:0/143081999 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9900159210 0x7f99000c74a0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f990006c090 tx=0x7f98f402a680 comp rx=0 tx=0).stop 2024-04-24T23:16:16.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.193+0000 7f0faf400640 1 --2- 172.21.15.6:0/3948633507 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0f940039b0 0x7f0f94003dd0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f0fb0067e40 tx=0x7f0f94005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6e422f96bd7d7e58 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:16.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.194+0000 7f9908400640 1 -- 172.21.15.6:0/143081999 shutdown_connections 2024-04-24T23:16:16.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.194+0000 7f9908400640 1 --2- 172.21.15.6:0/143081999 >> v2:172.21.15.6:3300/0 conn(0x7f9900162a40 0x7f9900162e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:16.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.194+0000 7f9908400640 1 --2- 172.21.15.6:0/143081999 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f99000c8420 0x7f9900162500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:16.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.194+0000 7f9908400640 1 --2- 172.21.15.6:0/143081999 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9900159210 0x7f99000c74a0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:16.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.194+0000 7f9908400640 1 -- 172.21.15.6:0/143081999 >> 172.21.15.6:0/143081999 conn(0x7f99000c8c50 msgr2=0x7f99000c9060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:16.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.194+0000 7f0fae000640 1 -- 172.21.15.6:0/3948633507 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fa0030070 con 0x7f0f940039b0 2024-04-24T23:16:16.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.194+0000 7f0fae000640 1 -- 172.21.15.6:0/3948633507 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0fa002acc0 con 0x7f0f940039b0 2024-04-24T23:16:16.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.195+0000 7f24b9800640 1 Processor -- start 2024-04-24T23:16:16.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.195+0000 7fa41f200640 1 Processor -- start 2024-04-24T23:16:16.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.195+0000 7f0fb6600640 1 -- 172.21.15.6:0/3948633507 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0f940039b0 msgr2=0x7f0f94003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:16.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.195+0000 7f0fb6600640 1 --2- 172.21.15.6:0/3948633507 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0f940039b0 0x7f0f94003dd0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f0fb0067e40 tx=0x7f0f94005830 comp rx=0 tx=0).stop 2024-04-24T23:16:16.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.197+0000 7f0fafe00640 1 -- 172.21.15.6:0/3948633507 reap_dead start 2024-04-24T23:16:16.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.194+0000 7f9908400640 1 -- 172.21.15.6:0/143081999 shutdown_connections 2024-04-24T23:16:16.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.197+0000 7f0fb6600640 1 -- 172.21.15.6:0/3948633507 shutdown_connections 2024-04-24T23:16:16.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.197+0000 7f0fb6600640 1 -- 172.21.15.6:0/3948633507 >> 172.21.15.6:0/3948633507 conn(0x7f0fb0066ea0 msgr2=0x7f0fb0067280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:16.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.194+0000 7f9908400640 1 -- 172.21.15.6:0/143081999 wait complete. 2024-04-24T23:16:16.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.195+0000 7f24b9800640 1 -- start start 2024-04-24T23:16:16.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.197+0000 7f9908400640 1 Processor -- start 2024-04-24T23:16:16.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.198+0000 7f24b9800640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f24b40cae00 0x7f24b4152fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:16.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.198+0000 7f0fb6600640 1 -- 172.21.15.6:0/3948633507 shutdown_connections 2024-04-24T23:16:16.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.198+0000 7f0fb6600640 1 -- 172.21.15.6:0/3948633507 wait complete. 2024-04-24T23:16:16.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.195+0000 7fa41f200640 1 -- start start 2024-04-24T23:16:16.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.198+0000 7f24b9800640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f24b40cc790 0x7f24b4159af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:16.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.198+0000 7fa41f200640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa41815f910 0x7fa4180cae10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:16.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.198+0000 7fa41f200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa4181625b0 con 0x7fa4180cda60 2024-04-24T23:16:16.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.198+0000 7fa41f200640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa418162890 con 0x7fa4181590b0 2024-04-24T23:16:16.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.198+0000 7fa41f200640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fa41815fda0 con 0x7fa41815f910 2024-04-24T23:16:16.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.198+0000 7f9908400640 1 -- start start 2024-04-24T23:16:16.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.198+0000 7f24b9800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f24b4154d00 0x7f24b415ca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:16.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.198+0000 7f24b9800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f24b40cbbf0 con 0x7f24b4154d00 2024-04-24T23:16:16.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.198+0000 7f24b9800640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f24b415e110 con 0x7f24b40cc790 2024-04-24T23:16:16.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.198+0000 7f24b9800640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f24b415eee0 con 0x7f24b40cae00 2024-04-24T23:16:16.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.198+0000 7f24b2000640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f24b40cc790 0x7f24b4159af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:16.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.199+0000 7f24b2000640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f24b40cc790 0x7f24b4159af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:34080/0 (socket says 172.21.15.6:34080) 2024-04-24T23:16:16.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.199+0000 7f24b2000640 1 -- 172.21.15.6:0/3050276165 learned_addr learned my addr 172.21.15.6:0/3050276165 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:16:16.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.199+0000 7f24b3400640 1 --2- 172.21.15.6:0/3050276165 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f24b4154d00 0x7f24b415ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:16.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.199+0000 7f24b2a00640 1 --2- 172.21.15.6:0/3050276165 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f24b40cae00 0x7f24b4152fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:16.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.199+0000 7f24b2000640 1 -- 172.21.15.6:0/3050276165 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f24b40cae00 msgr2=0x7f24b4152fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:16.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.199+0000 7fa417e00640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7fa4181590b0 0x7fa41806b910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.6:45152/0 (socket says 172.21.15.6:45152) 2024-04-24T23:16:16.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.199+0000 7fa417e00640 1 -- 172.21.15.6:0/3944647631 learned_addr learned my addr 172.21.15.6:0/3944647631 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:16:16.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.199+0000 7f24b2000640 1 --2- 172.21.15.6:0/3050276165 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f24b40cae00 0x7f24b4152fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:16.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.199+0000 7f24b2000640 1 -- 172.21.15.6:0/3050276165 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f24b4154d00 msgr2=0x7f24b415ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:16.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.199+0000 7f24b2000640 1 --2- 172.21.15.6:0/3050276165 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f24b4154d00 0x7f24b415ca60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:16.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.199+0000 7f24b2000640 1 -- 172.21.15.6:0/3050276165 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24b40cf970 con 0x7f24b40cc790 2024-04-24T23:16:16.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.200+0000 7f9908400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f99000c8420 0x7f99000cb850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:16.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.200+0000 7f24b3400640 1 --2- 172.21.15.6:0/3050276165 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f24b4154d00 0x7f24b415ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:16:16.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.200+0000 7fa417400640 1 --2- 172.21.15.6:0/3944647631 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa41815f910 0x7fa4180cae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:16.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.201+0000 7fa416a00640 1 -- 172.21.15.6:0/3944647631 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3665649588 0 0) 0x7fa4181625b0 con 0x7fa4180cda60 2024-04-24T23:16:16.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.201+0000 7fa416a00640 1 -- 172.21.15.6:0/3944647631 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa3fc003520 con 0x7fa4180cda60 2024-04-24T23:16:16.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.201+0000 7fa416a00640 1 -- 172.21.15.6:0/3944647631 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1081026100 0 0) 0x7fa418162890 con 0x7fa4181590b0 2024-04-24T23:16:16.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.201+0000 7f9908400640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9900159210 0x7f990015b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:16.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.201+0000 7f9905a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f99000c8420 0x7f99000cb850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:16.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.201+0000 7f9908400640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9900162a40 0x7f990015ba30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:16.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.201+0000 7f24b2000640 1 --2- 172.21.15.6:0/3050276165 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f24b40cc790 0x7f24b4159af0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f24a400c780 tx=0x7f24a400cc40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:16.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.201+0000 7f9908400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f99000c6ca0 con 0x7f99000c8420 2024-04-24T23:16:16.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.201+0000 7f9908400640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f990006d050 con 0x7f9900162a40 2024-04-24T23:16:16.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.201+0000 7f9908400640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f99001595f0 con 0x7f9900159210 2024-04-24T23:16:16.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.202+0000 7f9906400640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9900162a40 0x7f990015ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:16.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.202+0000 7f9906400640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9900162a40 0x7f990015ba30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:34094/0 (socket says 172.21.15.6:34094) 2024-04-24T23:16:16.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.202+0000 7f9906400640 1 -- 172.21.15.6:0/3051729946 learned_addr learned my addr 172.21.15.6:0/3051729946 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:16:16.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.202+0000 7f9905000640 1 --2- 172.21.15.6:0/3051729946 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9900159210 0x7f990015b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:16.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.202+0000 7fa416a00640 1 -- 172.21.15.6:0/3944647631 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa3fc0048e0 con 0x7fa4181590b0 2024-04-24T23:16:16.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.202+0000 7f9906400640 1 -- 172.21.15.6:0/3051729946 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9900159210 msgr2=0x7f990015b4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:16.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.202+0000 7f9906400640 1 --2- 172.21.15.6:0/3051729946 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9900159210 0x7f990015b4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:16.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.202+0000 7f9906400640 1 -- 172.21.15.6:0/3051729946 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f99000c8420 msgr2=0x7f99000cb850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:16.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.202+0000 7f24aaa00640 1 -- 172.21.15.6:0/3050276165 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24a4017070 con 0x7f24b40cc790 2024-04-24T23:16:16.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.202+0000 7f24aaa00640 1 -- 172.21.15.6:0/3050276165 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f24a4005290 con 0x7f24b40cc790 2024-04-24T23:16:16.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.202+0000 7f24aaa00640 1 -- 172.21.15.6:0/3050276165 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24a400a880 con 0x7f24b40cc790 2024-04-24T23:16:16.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.202+0000 7fa416a00640 1 -- 172.21.15.6:0/3944647631 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2582446937 0 0) 0x7fa3fc003520 con 0x7fa4180cda60 2024-04-24T23:16:16.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.203+0000 7fa416a00640 1 -- 172.21.15.6:0/3944647631 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa41815f910 msgr2=0x7fa4180cae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:16:16.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.203+0000 7fa416a00640 1 --2- 172.21.15.6:0/3944647631 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa41815f910 0x7fa4180cae10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:16.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.203+0000 7fa416a00640 1 -- 172.21.15.6:0/3944647631 >> v1:172.21.15.190:6789/0 conn(0x7fa4181590b0 legacy=0x7fa41806b910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:16.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.203+0000 7fa416a00640 1 -- 172.21.15.6:0/3944647631 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa418162b70 con 0x7fa4180cda60 2024-04-24T23:16:16.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.203+0000 7f24b9800640 1 -- 172.21.15.6:0/3050276165 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24b4152470 con 0x7f24b40cc790 2024-04-24T23:16:16.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.203+0000 7f24b9800640 1 -- 172.21.15.6:0/3050276165 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24b4153ce0 con 0x7f24b40cc790 2024-04-24T23:16:16.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.204+0000 7f9906400640 1 --2- 172.21.15.6:0/3051729946 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f99000c8420 0x7f99000cb850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:16.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.204+0000 7f9906400640 1 -- 172.21.15.6:0/3051729946 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9900150a60 con 0x7f9900162a40 2024-04-24T23:16:16.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.204+0000 7f9905a00640 1 --2- 172.21.15.6:0/3051729946 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f99000c8420 0x7f99000cb850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:16:16.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.203+0000 7fa416a00640 1 -- 172.21.15.6:0/3944647631 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3463268701 0 0) 0x7fa4100019d0 con 0x7fa4180cda60 2024-04-24T23:16:16.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.203+0000 7fa416a00640 1 -- 172.21.15.6:0/3944647631 >> v1:172.21.15.6:6789/0 conn(0x7fa4180cda60 legacy=0x7fa4180c5630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:16.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.203+0000 7fa416a00640 1 --2- 172.21.15.6:0/3944647631 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa3fc004dc0 0x7fa3fc0051e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:16.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.203+0000 7fa416a00640 1 --2- 172.21.15.6:0/3944647631 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa3fc0037a0 0x7fa3fc003c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:16.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.203+0000 7fa416a00640 1 --2- 172.21.15.6:0/3944647631 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa3fc004380 0x7fa3fc00ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:16.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.203+0000 7fa416a00640 1 -- 172.21.15.6:0/3944647631 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fa41815fda0 con 0x7fa3fc004380 2024-04-24T23:16:16.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.203+0000 7fa416a00640 1 -- 172.21.15.6:0/3944647631 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fa3fc00c300 con 0x7fa3fc0037a0 2024-04-24T23:16:16.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.203+0000 7fa416a00640 1 -- 172.21.15.6:0/3944647631 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fa3fc00c500 con 0x7fa3fc004dc0 2024-04-24T23:16:16.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.205+0000 7f24b9800640 1 -- 172.21.15.6:0/3050276165 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f24b40767b0 con 0x7f24b40cc790 2024-04-24T23:16:16.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.204+0000 7fa417400640 1 --2- 172.21.15.6:0/3944647631 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa3fc0037a0 0x7fa3fc003c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:16.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.204+0000 7fa417e00640 1 --2- 172.21.15.6:0/3944647631 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa3fc004dc0 0x7fa3fc0051e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:16.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.205+0000 7fa41d200640 1 --2- 172.21.15.6:0/3944647631 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa3fc004380 0x7fa3fc00ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:16.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.206+0000 7f9906400640 1 --2- 172.21.15.6:0/3051729946 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9900162a40 0x7f990015ba30 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f98f800e810 tx=0x7f98f800ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:16.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.206+0000 7fa417400640 1 -- 172.21.15.6:0/3944647631 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa3fc004dc0 msgr2=0x7fa3fc0051e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:16:16.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.206+0000 7fa417400640 1 --2- 172.21.15.6:0/3944647631 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa3fc004dc0 0x7fa3fc0051e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:16.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.206+0000 7fa417400640 1 -- 172.21.15.6:0/3944647631 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa3fc004380 msgr2=0x7fa3fc00ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:16:16.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.206+0000 7fa417400640 1 --2- 172.21.15.6:0/3944647631 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa3fc004380 0x7fa3fc00ba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:16.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.206+0000 7fa417400640 1 -- 172.21.15.6:0/3944647631 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3fc003520 con 0x7fa3fc0037a0 2024-04-24T23:16:16.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.207+0000 7fa41d200640 1 --2- 172.21.15.6:0/3944647631 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa3fc004380 0x7fa3fc00ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:16:16.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.208+0000 7fa417400640 1 --2- 172.21.15.6:0/3944647631 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa3fc0037a0 0x7fa3fc003c20 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fa4180cd150 tx=0x7fa3fc005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4cec3e71a03f76cf server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:16.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.200+0000 7f0fb6600640 1 Processor -- start 2024-04-24T23:16:16.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.201+0000 7f0fb6600640 1 -- start start 2024-04-24T23:16:16.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.203+0000 7f0fb6600640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0fb014f410 0x7f0fb014f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:16.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.203+0000 7f0fb6600640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0fb014fd30 0x7f0fb00b1e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:16.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.209+0000 7f24aaa00640 1 -- 172.21.15.6:0/3050276165 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f24a401a070 con 0x7f24b40cc790 2024-04-24T23:16:16.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.203+0000 7f0fb6600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0fb00b2370 0x7f0fb0154200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:16.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.205+0000 7f0faea00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0fb014fd30 0x7f0fb00b1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:16.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.205+0000 7f0faea00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0fb014fd30 0x7f0fb00b1e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:34106/0 (socket says 172.21.15.6:34106) 2024-04-24T23:16:16.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.205+0000 7f0faf400640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0fb014f410 0x7f0fb014f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:16.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.205+0000 7f0fafe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0fb00b2370 0x7f0fb0154200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:16.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.205+0000 7f0fafe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0fb00b2370 0x7f0fb0154200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:43406/0 (socket says 172.21.15.6:43406) 2024-04-24T23:16:16.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.205+0000 7f0faf400640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0fb014f410 0x7f0fb014f7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.6:60688/0 (socket says 172.21.15.6:60688) 2024-04-24T23:16:16.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.205+0000 7f0faea00640 1 -- 172.21.15.6:0/1521201146 learned_addr learned my addr 172.21.15.6:0/1521201146 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:16:16.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.206+0000 7f0fb6600640 1 -- 172.21.15.6:0/1521201146 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f0fb0159590 con 0x7f0fb00b2370 2024-04-24T23:16:16.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.206+0000 7f0fb6600640 1 -- 172.21.15.6:0/1521201146 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f0fb01592b0 con 0x7f0fb014fd30 2024-04-24T23:16:16.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.206+0000 7f0fb6600640 1 -- 172.21.15.6:0/1521201146 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f0fb00cbb80 con 0x7f0fb014f410 2024-04-24T23:16:16.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.206+0000 7f0faea00640 1 -- 172.21.15.6:0/1521201146 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0fb014f410 msgr2=0x7f0fb014f7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:16.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.206+0000 7f0faea00640 1 --2- 172.21.15.6:0/1521201146 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0fb014f410 0x7f0fb014f7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:16.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.206+0000 7f0faea00640 1 -- 172.21.15.6:0/1521201146 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0fb00b2370 msgr2=0x7f0fb0154200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:16.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.206+0000 7f0faea00640 1 --2- 172.21.15.6:0/1521201146 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0fb00b2370 0x7f0fb0154200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:16.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.206+0000 7f0faea00640 1 -- 172.21.15.6:0/1521201146 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0fb00b27e0 con 0x7f0fb014fd30 2024-04-24T23:16:16.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.207+0000 7f0faea00640 1 --2- 172.21.15.6:0/1521201146 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0fb014fd30 0x7f0fb00b1e30 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f0fa8007f70 tx=0x7f0fa800f5e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:16.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.211+0000 7f24aaa00640 1 --2- 172.21.15.6:0/3050276165 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f249406bf00 0x7f249406e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:16.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.211+0000 7f24aaa00640 1 -- 172.21.15.6:0/3050276165 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f24a408af30 con 0x7f24b40cc790 2024-04-24T23:16:16.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.211+0000 7f24b2a00640 1 --2- 172.21.15.6:0/3050276165 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f249406bf00 0x7f249406e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:16.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.212+0000 7f24aaa00640 1 -- 172.21.15.6:0/3050276165 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f24a40594f0 con 0x7f24b40cc790 2024-04-24T23:16:16.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.213+0000 7f24b2a00640 1 --2- 172.21.15.6:0/3050276165 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f249406bf00 0x7f249406e3c0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f24a0000f60 tx=0x7f24a0007c80 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:16.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.213+0000 7f0fa7400640 1 -- 172.21.15.6:0/1521201146 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fa8018020 con 0x7f0fb014fd30 2024-04-24T23:16:16.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.213+0000 7fa416a00640 1 -- 172.21.15.6:0/3944647631 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa404036070 con 0x7fa3fc0037a0 2024-04-24T23:16:16.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.214+0000 7f0fa7400640 1 -- 172.21.15.6:0/1521201146 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0fa80145f0 con 0x7f0fb014fd30 2024-04-24T23:16:16.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.214+0000 7f0fa7400640 1 -- 172.21.15.6:0/1521201146 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fa80149a0 con 0x7f0fb014fd30 2024-04-24T23:16:16.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.214+0000 7f98fd600640 1 -- 172.21.15.6:0/3051729946 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98f800f040 con 0x7f9900162a40 2024-04-24T23:16:16.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.214+0000 7f9908400640 1 -- 172.21.15.6:0/3051729946 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9900150120 con 0x7f9900162a40 2024-04-24T23:16:16.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.215+0000 7f0fb6600640 1 -- 172.21.15.6:0/1521201146 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0fb00c8c40 con 0x7f0fb014fd30 2024-04-24T23:16:16.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.214+0000 7fa416a00640 1 -- 172.21.15.6:0/3944647631 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa404031400 con 0x7fa3fc0037a0 2024-04-24T23:16:16.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.215+0000 7fa41f200640 1 -- 172.21.15.6:0/3944647631 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa3fc0037a0 msgr2=0x7fa3fc003c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:16.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.214+0000 7f9908400640 1 -- 172.21.15.6:0/3051729946 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9900159910 con 0x7f9900162a40 2024-04-24T23:16:16.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.216+0000 7fa416a00640 1 -- 172.21.15.6:0/3944647631 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa404036070 con 0x7fa3fc0037a0 2024-04-24T23:16:16.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.216+0000 7fa41f200640 1 --2- 172.21.15.6:0/3944647631 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa3fc0037a0 0x7fa3fc003c20 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fa4180cd150 tx=0x7fa3fc005830 comp rx=0 tx=0).stop 2024-04-24T23:16:16.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.216+0000 7fa41d200640 1 -- 172.21.15.6:0/3944647631 reap_dead start 2024-04-24T23:16:16.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.216+0000 7fa41f200640 1 -- 172.21.15.6:0/3944647631 shutdown_connections 2024-04-24T23:16:16.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.216+0000 7fa41f200640 1 -- 172.21.15.6:0/3944647631 >> 172.21.15.6:0/3944647631 conn(0x7fa41806d080 msgr2=0x7fa41806d8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:16.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.217+0000 7fa41f200640 1 -- 172.21.15.6:0/3944647631 shutdown_connections 2024-04-24T23:16:16.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.217+0000 7fa41f200640 1 -- 172.21.15.6:0/3944647631 wait complete. 2024-04-24T23:16:16.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.215+0000 7f0fb6600640 1 -- 172.21.15.6:0/1521201146 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0fb015afc0 con 0x7f0fb014fd30 2024-04-24T23:16:16.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.217+0000 7f0fb6600640 1 -- 172.21.15.6:0/1521201146 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0f88005290 con 0x7f0fb014fd30 2024-04-24T23:16:16.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.220+0000 7f6228a00640 1 Processor -- start 2024-04-24T23:16:16.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.215+0000 7f98fd600640 1 -- 172.21.15.6:0/3051729946 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f98f8004030 con 0x7f9900162a40 2024-04-24T23:16:16.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.215+0000 7f98fd600640 1 -- 172.21.15.6:0/3051729946 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98f800f040 con 0x7f9900162a40 2024-04-24T23:16:16.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.220+0000 7f9908400640 1 -- 172.21.15.6:0/3051729946 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f98dc005290 con 0x7f9900162a40 2024-04-24T23:16:16.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.221+0000 7fa41f200640 1 Processor -- start 2024-04-24T23:16:16.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.221+0000 7f6228a00640 1 -- start start 2024-04-24T23:16:16.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.221+0000 7fa41f200640 1 -- start start 2024-04-24T23:16:16.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.220+0000 7f98fd600640 1 -- 172.21.15.6:0/3051729946 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f98f80041e0 con 0x7f9900162a40 2024-04-24T23:16:16.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.223+0000 7fa41f200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa4180cf660 0x7fa4181f5290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:16.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.224+0000 7fa41f200640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa4180cfa40 0x7fa4181f57d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:16.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.223+0000 7f6228a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f6224150d40 0x7f62240c6f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:16.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.224+0000 7f6228a00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f62240c89b0 con 0x7f6224071590 2024-04-24T23:16:16.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.224+0000 7f6228a00640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f622415a210 con 0x7f62240c74b0 2024-04-24T23:16:16.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.224+0000 7f6228a00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f622415a4f0 con 0x7f6224150d40 2024-04-24T23:16:16.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.224+0000 7fa41f200640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa4180d0a50 0x7fa4181524d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:16.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.224+0000 7f6222800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f6224150d40 0x7f62240c6f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:16.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.225+0000 7f6222800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f6224150d40 0x7f62240c6f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:43422/0 (socket says 172.21.15.6:43422) 2024-04-24T23:16:16.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.225+0000 7f6223200640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f62240c74b0 0x7f6224156cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.6:45168/0 (socket says 172.21.15.6:45168) 2024-04-24T23:16:16.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.225+0000 7f6223200640 1 -- 172.21.15.6:0/547983819 learned_addr learned my addr 172.21.15.6:0/547983819 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:16:16.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.225+0000 7fa417400640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa4180cfa40 0x7fa4181f57d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:16.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.225+0000 7fa41d200640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa4180d0a50 0x7fa4181524d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:16.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.225+0000 7fa41d200640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa4180d0a50 0x7fa4181524d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:34118/0 (socket says 172.21.15.6:34118) 2024-04-24T23:16:16.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.225+0000 7fa41d200640 1 -- 172.21.15.6:0/3868174122 learned_addr learned my addr 172.21.15.6:0/3868174122 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:16:16.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.225+0000 7fa41f200640 1 -- 172.21.15.6:0/3868174122 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fa418159490 con 0x7fa4180cf660 2024-04-24T23:16:16.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.225+0000 7fa41f200640 1 -- 172.21.15.6:0/3868174122 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fa418163130 con 0x7fa4180d0a50 2024-04-24T23:16:16.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.225+0000 7fa41f200640 1 -- 172.21.15.6:0/3868174122 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fa41806c020 con 0x7fa4180cfa40 2024-04-24T23:16:16.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.225+0000 7fa417e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa4180cf660 0x7fa4181f5290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:16.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.226+0000 7f6222800640 1 -- 172.21.15.6:0/547983819 >> v1:172.21.15.190:6789/0 conn(0x7f62240c74b0 legacy=0x7f6224156cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:16.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.226+0000 7f6222800640 1 -- 172.21.15.6:0/547983819 >> v1:172.21.15.6:6789/0 conn(0x7f6224071590 legacy=0x7f62240c7cf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-24T23:16:16.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.226+0000 7f6222800640 1 -- 172.21.15.6:0/547983819 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f622415a7d0 con 0x7f6224150d40 2024-04-24T23:16:16.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.226+0000 7fa417400640 1 -- 172.21.15.6:0/3868174122 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa4180d0a50 msgr2=0x7fa4181524d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:16.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.226+0000 7fa417400640 1 --2- 172.21.15.6:0/3868174122 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa4180d0a50 0x7fa4181524d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:16.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.226+0000 7f98fd600640 1 --2- 172.21.15.6:0/3051729946 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f98e006c030 0x7f98e006e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:16.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.227+0000 7f98fd600640 1 -- 172.21.15.6:0/3051729946 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f98f808aba0 con 0x7f9900162a40 2024-04-24T23:16:16.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.227+0000 7f9905a00640 1 --2- 172.21.15.6:0/3051729946 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f98e006c030 0x7f98e006e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:16.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.227+0000 7fa417400640 1 -- 172.21.15.6:0/3868174122 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa4180cf660 msgr2=0x7fa4181f5290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:16.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.227+0000 7fa417400640 1 --2- 172.21.15.6:0/3868174122 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa4180cf660 0x7fa4181f5290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:16.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.227+0000 7fa417400640 1 -- 172.21.15.6:0/3868174122 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4180d0860 con 0x7fa4180cfa40 2024-04-24T23:16:16.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.227+0000 7f6222800640 1 --2- 172.21.15.6:0/547983819 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6224150d40 0x7f62240c6f70 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f621c00c2e0 tx=0x7f621c02cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a02e8ea62079f35 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:16.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.227+0000 7fa417400640 1 --2- 172.21.15.6:0/3868174122 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa4180cfa40 0x7fa4181f57d0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fa404031000 tx=0x7fa404004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:16.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.227+0000 7f0fa7400640 1 -- 172.21.15.6:0/1521201146 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f0fa801b020 con 0x7f0fb014fd30 2024-04-24T23:16:16.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.228+0000 7f6221400640 1 -- 172.21.15.6:0/547983819 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f621c02da60 con 0x7f6224150d40 2024-04-24T23:16:16.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.228+0000 7f6221400640 1 -- 172.21.15.6:0/547983819 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f621c02dc00 con 0x7f6224150d40 2024-04-24T23:16:16.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.228+0000 7fa40fe00640 1 -- 172.21.15.6:0/3868174122 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa404040070 con 0x7fa4180cfa40 2024-04-24T23:16:16.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.228+0000 7f9905a00640 1 --2- 172.21.15.6:0/3051729946 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f98e006c030 0x7f98e006e4f0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f98f4034000 tx=0x7f98f402f000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:16.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.228+0000 7fa40fe00640 1 -- 172.21.15.6:0/3868174122 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa40403b660 con 0x7fa4180cfa40 2024-04-24T23:16:16.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.228+0000 7fa40fe00640 1 -- 172.21.15.6:0/3868174122 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa404036040 con 0x7fa4180cfa40 2024-04-24T23:16:16.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.228+0000 7fa41f200640 1 -- 172.21.15.6:0/3868174122 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa418161ed0 con 0x7fa4180cfa40 2024-04-24T23:16:16.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.229+0000 7fa41f200640 1 -- 172.21.15.6:0/3868174122 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4181f8a80 con 0x7fa4180cfa40 2024-04-24T23:16:16.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.230+0000 7f6221400640 1 -- 172.21.15.6:0/547983819 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f621c02da60 con 0x7f6224150d40 2024-04-24T23:16:16.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.230+0000 7f6228a00640 1 -- 172.21.15.6:0/547983819 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6224150d40 msgr2=0x7f62240c6f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:16.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.230+0000 7f6228a00640 1 --2- 172.21.15.6:0/547983819 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6224150d40 0x7f62240c6f70 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f621c00c2e0 tx=0x7f621c02cd50 comp rx=0 tx=0).stop 2024-04-24T23:16:16.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.231+0000 7f6228a00640 1 -- 172.21.15.6:0/547983819 shutdown_connections 2024-04-24T23:16:16.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.231+0000 7f6228a00640 1 --2- 172.21.15.6:0/547983819 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6224150d40 0x7f62240c6f70 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:16.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.231+0000 7f6228a00640 1 -- 172.21.15.6:0/547983819 >> 172.21.15.6:0/547983819 conn(0x7f62240cc1e0 msgr2=0x7f62240cc5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:16.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.231+0000 7f98fd600640 1 -- 172.21.15.6:0/3051729946 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f98f8017020 con 0x7f9900162a40 2024-04-24T23:16:16.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.231+0000 7f6228a00640 1 -- 172.21.15.6:0/547983819 shutdown_connections 2024-04-24T23:16:16.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.231+0000 7f0fa7400640 1 --2- 172.21.15.6:0/1521201146 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f0f9006be60 0x7f0f9006e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:16.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.231+0000 7f0fa7400640 1 -- 172.21.15.6:0/1521201146 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f0fa808c2d0 con 0x7f0fb014fd30 2024-04-24T23:16:16.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.232+0000 7f0fa7400640 1 -- 172.21.15.6:0/1521201146 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f0fa8022020 con 0x7f0fb014fd30 2024-04-24T23:16:16.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.231+0000 7fa41f200640 1 -- 172.21.15.6:0/3868174122 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa4180767b0 con 0x7fa4180cfa40 2024-04-24T23:16:16.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.233+0000 7fa40fe00640 1 -- 172.21.15.6:0/3868174122 <== mon.2 v2:172.21.15.190:3301/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fa404039070 con 0x7fa4180cfa40 2024-04-24T23:16:16.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.231+0000 7f6228a00640 1 -- 172.21.15.6:0/547983819 wait complete. 2024-04-24T23:16:16.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.234+0000 7f6228a00640 1 Processor -- start 2024-04-24T23:16:16.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.235+0000 7f6228a00640 1 -- start start 2024-04-24T23:16:16.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.235+0000 7fa40fe00640 1 --2- 172.21.15.6:0/3868174122 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fa3f806bf30 0x7fa3f806e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:16.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.235+0000 7fa40fe00640 1 -- 172.21.15.6:0/3868174122 <== mon.2 v2:172.21.15.190:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fa4040ab060 con 0x7fa4180cfa40 2024-04-24T23:16:16.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.233+0000 7f0faf400640 1 --2- 172.21.15.6:0/1521201146 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f0f9006be60 0x7f0f9006e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:16.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.236+0000 7fa417e00640 1 --2- 172.21.15.6:0/3868174122 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fa3f806bf30 0x7fa3f806e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:16.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.237+0000 7f6228a00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6224071590 0x7f62241f0300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:16.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.237+0000 7f6228a00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f62240c74b0 0x7f6224145120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:16.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.237+0000 7f6222800640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6224071590 0x7f62241f0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:16.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.237+0000 7f6228a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6224150d40 0x7f6224151940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:16.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.237+0000 7f6222800640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6224071590 0x7f62241f0300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.6:60696/0 (socket says 172.21.15.6:60696) 2024-04-24T23:16:16.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.238+0000 7f6228a00640 1 -- 172.21.15.6:0/1196377210 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f6224151120 con 0x7f6224150d40 2024-04-24T23:16:16.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.238+0000 7f6228a00640 1 -- 172.21.15.6:0/1196377210 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f622415ad90 con 0x7f62240c74b0 2024-04-24T23:16:16.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.238+0000 7f6228a00640 1 -- 172.21.15.6:0/1196377210 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f62240c8400 con 0x7f6224071590 2024-04-24T23:16:16.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.238+0000 7f6222800640 1 -- 172.21.15.6:0/1196377210 learned_addr learned my addr 172.21.15.6:0/1196377210 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:16:16.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.238+0000 7f6221e00640 1 --2- 172.21.15.6:0/1196377210 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f62240c74b0 0x7f6224145120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:16.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.238+0000 7f6223200640 1 --2- 172.21.15.6:0/1196377210 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6224150d40 0x7f6224151940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:16.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.238+0000 7fa417e00640 1 --2- 172.21.15.6:0/3868174122 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fa3f806bf30 0x7fa3f806e3f0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fa408004840 tx=0x7fa408009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:16.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.238+0000 7f0faf400640 1 --2- 172.21.15.6:0/1521201146 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f0f9006be60 0x7f0f9006e320 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f0fb00c36e0 tx=0x7f0fa0030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:16.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.238+0000 7fa40fe00640 1 -- 172.21.15.6:0/3868174122 <== mon.2 v2:172.21.15.190:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fa4040790f0 con 0x7fa4180cfa40 2024-04-24T23:16:16.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.239+0000 7f6222800640 1 -- 172.21.15.6:0/1196377210 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f62240c74b0 msgr2=0x7f6224145120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:16.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.239+0000 7f6222800640 1 --2- 172.21.15.6:0/1196377210 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f62240c74b0 0x7f6224145120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:16.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.239+0000 7f6222800640 1 -- 172.21.15.6:0/1196377210 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6224150d40 msgr2=0x7f6224151940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:16.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.242+0000 7f6222800640 1 --2- 172.21.15.6:0/1196377210 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6224150d40 0x7f6224151940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:16.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.242+0000 7f6222800640 1 -- 172.21.15.6:0/1196377210 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6224145b00 con 0x7f6224071590 2024-04-24T23:16:16.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.242+0000 7f6223200640 1 --2- 172.21.15.6:0/1196377210 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6224150d40 0x7f6224151940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:16:16.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.244+0000 7f6222800640 1 --2- 172.21.15.6:0/1196377210 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6224071590 0x7f62241f0300 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f621c037040 tx=0x7f621c004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:16.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.244+0000 7f621aa00640 1 -- 172.21.15.6:0/1196377210 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f621c045030 con 0x7f6224071590 2024-04-24T23:16:16.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.244+0000 7f6228a00640 1 -- 172.21.15.6:0/1196377210 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62241ec6d0 con 0x7f6224071590 2024-04-24T23:16:16.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.245+0000 7f621aa00640 1 -- 172.21.15.6:0/1196377210 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f621c004680 con 0x7f6224071590 2024-04-24T23:16:16.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.245+0000 7f6228a00640 1 -- 172.21.15.6:0/1196377210 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f62241ecc90 con 0x7f6224071590 2024-04-24T23:16:16.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.245+0000 7f621aa00640 1 -- 172.21.15.6:0/1196377210 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f621c045030 con 0x7f6224071590 2024-04-24T23:16:16.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.249+0000 7f6228a00640 1 -- 172.21.15.6:0/1196377210 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f61fc005290 con 0x7f6224071590 2024-04-24T23:16:16.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.252+0000 7f621aa00640 1 -- 172.21.15.6:0/1196377210 <== mon.2 v2:172.21.15.190:3301/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f621c048070 con 0x7f6224071590 2024-04-24T23:16:16.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.257+0000 7fc2ef000640 1 Processor -- start 2024-04-24T23:16:16.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.257+0000 7fc2ef000640 1 -- start start 2024-04-24T23:16:16.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.257+0000 7f621aa00640 1 --2- 172.21.15.6:0/1196377210 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f6204070440 0x7f6204072900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:16.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.257+0000 7f621aa00640 1 -- 172.21.15.6:0/1196377210 <== mon.2 v2:172.21.15.190:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f621c0ac170 con 0x7f6224071590 2024-04-24T23:16:16.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.258+0000 7f621aa00640 1 -- 172.21.15.6:0/1196377210 <== mon.2 v2:172.21.15.190:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f621c04e020 con 0x7f6224071590 2024-04-24T23:16:16.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.260+0000 7f6221e00640 1 --2- 172.21.15.6:0/1196377210 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f6204070440 0x7f6204072900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:16.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.259+0000 7fc2ef000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc2e8156cc0 0x7fc2e8157110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:16.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.259+0000 7fc2ef000640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc2e8157650 con 0x7fc2e8071590 2024-04-24T23:16:16.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.259+0000 7fc2ef000640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc2e80c89b0 con 0x7fc2e8150d40 2024-04-24T23:16:16.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.259+0000 7fc2ef000640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7fc2e815a210 con 0x7fc2e8156cc0 2024-04-24T23:16:16.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.260+0000 7fc2edc00640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7fc2e8150d40 0x7fc2e80c6f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.6:45170/0 (socket says 172.21.15.6:45170) 2024-04-24T23:16:16.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.260+0000 7fc2edc00640 1 -- 172.21.15.6:0/148960437 learned_addr learned my addr 172.21.15.6:0/148960437 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:16:16.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.260+0000 7fc2ee600640 1 --2- 172.21.15.6:0/148960437 >> v2:172.21.15.6:3300/0 conn(0x7fc2e8156cc0 0x7fc2e8157110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:16.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.262+0000 7fc2e7e00640 1 -- 172.21.15.6:0/148960437 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1555766852 0 0) 0x7fc2e8157650 con 0x7fc2e8071590 2024-04-24T23:16:16.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.262+0000 7fc2e7e00640 1 -- 172.21.15.6:0/148960437 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc2d0003520 con 0x7fc2e8071590 2024-04-24T23:16:16.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.262+0000 7fc2ee600640 1 -- 172.21.15.6:0/148960437 >> v1:172.21.15.190:6789/0 conn(0x7fc2e8150d40 legacy=0x7fc2e80c6f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:16.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.262+0000 7fc2ee600640 1 -- 172.21.15.6:0/148960437 >> v1:172.21.15.6:6789/0 conn(0x7fc2e8071590 legacy=0x7fc2e80c7cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:16.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.262+0000 7fc2ee600640 1 -- 172.21.15.6:0/148960437 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2e815a4f0 con 0x7fc2e8156cc0 2024-04-24T23:16:16.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.268+0000 7fc2ee600640 1 --2- 172.21.15.6:0/148960437 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc2e8156cc0 0x7fc2e8157110 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fc2e0009d90 tx=0x7fc2e002c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd1d86f7e69d4e93 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:16.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.269+0000 7f6221e00640 1 --2- 172.21.15.6:0/1196377210 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f6204070440 0x7f6204072900 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f622406f890 tx=0x7f62140090e0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:16.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.272+0000 7fc2e7e00640 1 -- 172.21.15.6:0/148960437 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2e0036020 con 0x7fc2e8156cc0 2024-04-24T23:16:16.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.272+0000 7fc2e7e00640 1 -- 172.21.15.6:0/148960437 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2e0032830 con 0x7fc2e8156cc0 2024-04-24T23:16:16.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.272+0000 7fc2e7e00640 1 -- 172.21.15.6:0/148960437 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2e0032c40 con 0x7fc2e8156cc0 2024-04-24T23:16:16.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.273+0000 7fc2ef000640 1 -- 172.21.15.6:0/148960437 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc2e8156cc0 msgr2=0x7fc2e8157110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:16.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.273+0000 7fc2ef000640 1 --2- 172.21.15.6:0/148960437 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc2e8156cc0 0x7fc2e8157110 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fc2e0009d90 tx=0x7fc2e002c970 comp rx=0 tx=0).stop 2024-04-24T23:16:16.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.277+0000 7fc2ef000640 1 -- 172.21.15.6:0/148960437 shutdown_connections 2024-04-24T23:16:16.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.277+0000 7fc2ef000640 1 --2- 172.21.15.6:0/148960437 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc2e8156cc0 0x7fc2e8157110 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:16.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.277+0000 7fc2ef000640 1 -- 172.21.15.6:0/148960437 >> 172.21.15.6:0/148960437 conn(0x7fc2e80cc1e0 msgr2=0x7fc2e80cc5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:16.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.277+0000 7fc2ef000640 1 -- 172.21.15.6:0/148960437 shutdown_connections 2024-04-24T23:16:16.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.277+0000 7fc2ef000640 1 -- 172.21.15.6:0/148960437 wait complete. 2024-04-24T23:16:16.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.280+0000 7fc2ef000640 1 Processor -- start 2024-04-24T23:16:16.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.280+0000 7fc2ef000640 1 -- start start 2024-04-24T23:16:16.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.282+0000 7fc2ef000640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc2e8071590 0x7fc2e813fe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:16.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.283+0000 7fc2ef000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc2e8150d40 0x7fc2e8158e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:16.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.283+0000 7fc2edc00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc2e8071590 0x7fc2e813fe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:16.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.284+0000 7fc2edc00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc2e8071590 0x7fc2e813fe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.6:60702/0 (socket says 172.21.15.6:60702) 2024-04-24T23:16:16.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.287+0000 7fc2ed200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc2e8150d40 0x7fc2e8158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:16.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.287+0000 7fc2ed200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc2e8150d40 0x7fc2e8158e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:43466/0 (socket says 172.21.15.6:43466) 2024-04-24T23:16:16.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.287+0000 7fc2ef000640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc2e8156cc0 0x7fc2e81593c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:16.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.287+0000 7fc2ed200640 1 -- 172.21.15.6:0/3248479006 learned_addr learned my addr 172.21.15.6:0/3248479006 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:16:16.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.287+0000 7fc2ef000640 1 -- 172.21.15.6:0/3248479006 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fc2e815ad90 con 0x7fc2e8150d40 2024-04-24T23:16:16.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.287+0000 7fc2ef000640 1 -- 172.21.15.6:0/3248479006 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fc2e815aab0 con 0x7fc2e8156cc0 2024-04-24T23:16:16.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.287+0000 7fc2ef000640 1 -- 172.21.15.6:0/3248479006 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fc2e80c7680 con 0x7fc2e8071590 2024-04-24T23:16:16.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.287+0000 7fc2ee600640 1 --2- 172.21.15.6:0/3248479006 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc2e8156cc0 0x7fc2e81593c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:16.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.289+0000 7fc2edc00640 1 -- 172.21.15.6:0/3248479006 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc2e8156cc0 msgr2=0x7fc2e81593c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:16.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.289+0000 7fc2edc00640 1 --2- 172.21.15.6:0/3248479006 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc2e8156cc0 0x7fc2e81593c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:16.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.289+0000 7fc2edc00640 1 -- 172.21.15.6:0/3248479006 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc2e8150d40 msgr2=0x7fc2e8158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:16.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.289+0000 7fc2edc00640 1 --2- 172.21.15.6:0/3248479006 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc2e8150d40 0x7fc2e8158e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:16.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.289+0000 7fc2edc00640 1 -- 172.21.15.6:0/3248479006 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2e80ca210 con 0x7fc2e8071590 2024-04-24T23:16:16.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.289+0000 7fc2ed200640 1 --2- 172.21.15.6:0/3248479006 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc2e8150d40 0x7fc2e8158e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:16:16.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.293+0000 7fc2edc00640 1 --2- 172.21.15.6:0/3248479006 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc2e8071590 0x7fc2e813fe80 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fc2dc00c470 tx=0x7fc2dc001e70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:16.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.293+0000 7fc2e5600640 1 -- 172.21.15.6:0/3248479006 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2dc0058b0 con 0x7fc2e8071590 2024-04-24T23:16:16.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.293+0000 7fc2e5600640 1 -- 172.21.15.6:0/3248479006 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2dc019030 con 0x7fc2e8071590 2024-04-24T23:16:16.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.293+0000 7fc2e5600640 1 -- 172.21.15.6:0/3248479006 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2dc005c70 con 0x7fc2e8071590 2024-04-24T23:16:16.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.293+0000 7fc2ef000640 1 -- 172.21.15.6:0/3248479006 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2e8159e30 con 0x7fc2e8071590 2024-04-24T23:16:16.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.294+0000 7fc2ef000640 1 -- 172.21.15.6:0/3248479006 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2e81ec950 con 0x7fc2e8071590 2024-04-24T23:16:16.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.301+0000 7fc2e5600640 1 -- 172.21.15.6:0/3248479006 <== mon.2 v2:172.21.15.190:3301/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fc2dc007070 con 0x7fc2e8071590 2024-04-24T23:16:16.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.303+0000 7fc2e5600640 1 --2- 172.21.15.6:0/3248479006 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fc2c807cec0 0x7fc2c807f380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:16.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.303+0000 7fc2e5600640 1 -- 172.21.15.6:0/3248479006 <== mon.2 v2:172.21.15.190:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fc2dc08aa80 con 0x7fc2e8071590 2024-04-24T23:16:16.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.304+0000 7fc2ed200640 1 --2- 172.21.15.6:0/3248479006 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fc2c807cec0 0x7fc2c807f380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:16.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.306+0000 7fc2ed200640 1 --2- 172.21.15.6:0/3248479006 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fc2c807cec0 0x7fc2c807f380 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fc2e80cc970 tx=0x7fc2d4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:16.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.306+0000 7fc2ef000640 1 -- 172.21.15.6:0/3248479006 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc2e8075ba0 con 0x7fc2e8071590 2024-04-24T23:16:16.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.312+0000 7fc2e5600640 1 -- 172.21.15.6:0/3248479006 <== mon.2 v2:172.21.15.190:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fc2dc058a80 con 0x7fc2e8071590 2024-04-24T23:16:16.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.468+0000 7f1ca2a00640 1 Processor -- start 2024-04-24T23:16:16.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.472+0000 7f1ca2a00640 1 -- start start 2024-04-24T23:16:16.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.474+0000 7f1ca2a00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1c9c155010 0x7f1c9c0b2f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:16.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.474+0000 7f1ca2a00640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f1c9c06be80 0x7f1c9c06c310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:16.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.474+0000 7f1ca2a00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1c9c06c850 con 0x7f1c9c15b690 2024-04-24T23:16:16.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.474+0000 7f1ca2a00640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f1c9c1553f0 con 0x7f1c9c06be80 2024-04-24T23:16:16.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.474+0000 7f1ca2a00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f1c9c0cc280 con 0x7f1c9c155010 2024-04-24T23:16:16.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.474+0000 7f1ca2000640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f1c9c06be80 0x7f1c9c06c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:16.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.474+0000 7f1ca1600640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1c9c155010 0x7f1c9c0b2f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:16.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.474+0000 7f1ca2000640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f1c9c06be80 0x7f1c9c06c310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:34152/0 (socket says 172.21.15.6:34152) 2024-04-24T23:16:16.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.475+0000 7f1ca2000640 1 -- 172.21.15.6:0/2920774288 learned_addr learned my addr 172.21.15.6:0/2920774288 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:16:16.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.475+0000 7f1ca2000640 1 -- 172.21.15.6:0/2920774288 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1c9c155010 msgr2=0x7f1c9c0b2f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:16:16.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.475+0000 7f1ca2000640 1 --2- 172.21.15.6:0/2920774288 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1c9c155010 0x7f1c9c0b2f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:16.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.475+0000 7f1ca2000640 1 -- 172.21.15.6:0/2920774288 >> v1:172.21.15.6:6789/0 conn(0x7f1c9c15b690 legacy=0x7f1c9c0cd1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:16:16.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.475+0000 7f1ca2000640 1 -- 172.21.15.6:0/2920774288 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c9c0cc560 con 0x7f1c9c06be80 2024-04-24T23:16:16.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.475+0000 7f1ca1600640 1 --2- 172.21.15.6:0/2920774288 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1c9c155010 0x7f1c9c0b2f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:16:16.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.489+0000 7f1ca2000640 1 --2- 172.21.15.6:0/2920774288 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1c9c06be80 0x7f1c9c06c310 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f1c8c009870 tx=0x7f1c8c02a180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4fde7b0d6f48d2ed server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:16.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.493+0000 7f1c9be00640 1 -- 172.21.15.6:0/2920774288 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c8c037070 con 0x7f1c9c06be80 2024-04-24T23:16:16.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.493+0000 7f1c9be00640 1 -- 172.21.15.6:0/2920774288 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1c8c032440 con 0x7f1c9c06be80 2024-04-24T23:16:16.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.493+0000 7f1c9be00640 1 -- 172.21.15.6:0/2920774288 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c8c032810 con 0x7f1c9c06be80 2024-04-24T23:16:16.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.507+0000 7f1ca2a00640 1 -- 172.21.15.6:0/2920774288 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1c9c06be80 msgr2=0x7f1c9c06c310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:16.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.507+0000 7f1ca2a00640 1 --2- 172.21.15.6:0/2920774288 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1c9c06be80 0x7f1c9c06c310 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f1c8c009870 tx=0x7f1c8c02a180 comp rx=0 tx=0).stop 2024-04-24T23:16:16.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.515+0000 7f1ca2a00640 1 -- 172.21.15.6:0/2920774288 shutdown_connections 2024-04-24T23:16:16.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.515+0000 7f1ca2a00640 1 --2- 172.21.15.6:0/2920774288 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1c9c06be80 0x7f1c9c06c310 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:16.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.515+0000 7f1ca2a00640 1 --2- 172.21.15.6:0/2920774288 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1c9c155010 0x7f1c9c0b2f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:16.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.515+0000 7f1ca2a00640 1 -- 172.21.15.6:0/2920774288 >> 172.21.15.6:0/2920774288 conn(0x7f1c9c067080 msgr2=0x7f1c9c067490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:16.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.518+0000 7f1ca2a00640 1 -- 172.21.15.6:0/2920774288 shutdown_connections 2024-04-24T23:16:16.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.518+0000 7f1ca2a00640 1 -- 172.21.15.6:0/2920774288 wait complete. 2024-04-24T23:16:16.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.520+0000 7f1ca2a00640 1 Processor -- start 2024-04-24T23:16:16.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.534+0000 7f1ca2a00640 1 -- start start 2024-04-24T23:16:16.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.539+0000 7f1ca2a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1c9c06be80 0x7f1c9c0c6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:16.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.539+0000 7f1ca2a00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1c9c155010 0x7f1c9c1f7e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:16.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.539+0000 7f1ca2a00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1c9c15b690 0x7f1c9c1f83a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:16.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.539+0000 7f1ca2a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f1c9c0b3930 con 0x7f1c9c06be80 2024-04-24T23:16:16.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.539+0000 7f1ca2a00640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f1c9c15bad0 con 0x7f1c9c15b690 2024-04-24T23:16:16.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.539+0000 7f1ca2a00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f1c9c0cd8c0 con 0x7f1c9c155010 2024-04-24T23:16:16.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.547+0000 7f1ca1600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1c9c06be80 0x7f1c9c0c6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:16.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.548+0000 7f1ca1600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1c9c06be80 0x7f1c9c0c6380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:43474/0 (socket says 172.21.15.6:43474) 2024-04-24T23:16:16.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.548+0000 7f1ca1600640 1 -- 172.21.15.6:0/3287254073 learned_addr learned my addr 172.21.15.6:0/3287254073 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:16:16.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.549+0000 7f1ca0c00640 1 --2- 172.21.15.6:0/3287254073 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1c9c155010 0x7f1c9c1f7e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:16.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.549+0000 7f1ca2000640 1 --2- 172.21.15.6:0/3287254073 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1c9c15b690 0x7f1c9c1f83a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:16.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.551+0000 7f1ca2000640 1 -- 172.21.15.6:0/3287254073 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1c9c155010 msgr2=0x7f1c9c1f7e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:16.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.551+0000 7f1ca2000640 1 --2- 172.21.15.6:0/3287254073 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1c9c155010 0x7f1c9c1f7e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:16.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.551+0000 7f1ca2000640 1 -- 172.21.15.6:0/3287254073 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1c9c06be80 msgr2=0x7f1c9c0c6380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:16.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.553+0000 7f1ca2000640 1 --2- 172.21.15.6:0/3287254073 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1c9c06be80 0x7f1c9c0c6380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:16.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.553+0000 7f1ca2000640 1 -- 172.21.15.6:0/3287254073 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c9c0c3db0 con 0x7f1c9c15b690 2024-04-24T23:16:16.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.555+0000 7f1ca1600640 1 --2- 172.21.15.6:0/3287254073 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1c9c06be80 0x7f1c9c0c6380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:16:16.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.555+0000 7f1ca2000640 1 --2- 172.21.15.6:0/3287254073 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1c9c15b690 0x7f1c9c1f83a0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f1c9c08df20 tx=0x7f1c8c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:16.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.559+0000 7f1c99600640 1 -- 172.21.15.6:0/3287254073 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c8c041020 con 0x7f1c9c15b690 2024-04-24T23:16:16.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.561+0000 7f1c99600640 1 -- 172.21.15.6:0/3287254073 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1c8c03d6e0 con 0x7f1c9c15b690 2024-04-24T23:16:16.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.561+0000 7f1c99600640 1 -- 172.21.15.6:0/3287254073 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c8c037040 con 0x7f1c9c15b690 2024-04-24T23:16:16.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.561+0000 7f1ca2a00640 1 -- 172.21.15.6:0/3287254073 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c9c0c4090 con 0x7f1c9c15b690 2024-04-24T23:16:16.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.561+0000 7f1ca2a00640 1 -- 172.21.15.6:0/3287254073 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c9c0c4650 con 0x7f1c9c15b690 2024-04-24T23:16:16.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.568+0000 7f1ca2a00640 1 -- 172.21.15.6:0/3287254073 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1c9c075ba0 con 0x7f1c9c15b690 2024-04-24T23:16:16.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.585+0000 7f1c99600640 1 -- 172.21.15.6:0/3287254073 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f1c8c044020 con 0x7f1c9c15b690 2024-04-24T23:16:16.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.594+0000 7f1c99600640 1 --2- 172.21.15.6:0/3287254073 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f1c8006be80 0x7f1c8006e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:16.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.594+0000 7f1c99600640 1 -- 172.21.15.6:0/3287254073 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f1c8c0ab870 con 0x7f1c9c15b690 2024-04-24T23:16:16.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.595+0000 7f1c99600640 1 -- 172.21.15.6:0/3287254073 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f1c8c04b030 con 0x7f1c9c15b690 2024-04-24T23:16:16.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.607+0000 7f1ca1600640 1 --2- 172.21.15.6:0/3287254073 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f1c8006be80 0x7f1c8006e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:16.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.610+0000 7f1ca1600640 1 --2- 172.21.15.6:0/3287254073 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f1c8006be80 0x7f1c8006e340 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f1c94009bd0 tx=0x7f1c9400b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:16.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.612+0000 7f5e55e00640 1 -- 172.21.15.6:0/1916418933 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f5e48053bc0 con 0x7f5e340039b0 2024-04-24T23:16:16.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.613+0000 7f5e47400640 1 -- 172.21.15.6:0/1916418933 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f5e3c07a220 con 0x7f5e340039b0 2024-04-24T23:16:16.616 INFO:teuthology.orchestra.run.smithi006.stdout:55834574861 2024-04-24T23:16:16.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.617+0000 7f5e55e00640 1 -- 172.21.15.6:0/1916418933 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f5e3006bf30 msgr2=0x7f5e3006e3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:16.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.617+0000 7f5e55e00640 1 --2- 172.21.15.6:0/1916418933 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f5e3006bf30 0x7f5e3006e3f0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f5e48007390 tx=0x7f5e40009190 comp rx=0 tx=0).stop 2024-04-24T23:16:16.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.617+0000 7f5e55e00640 1 -- 172.21.15.6:0/1916418933 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5e340039b0 msgr2=0x7f5e48052950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:16.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.618+0000 7f5e55e00640 1 --2- 172.21.15.6:0/1916418933 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5e340039b0 0x7f5e48052950 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f5e3c00c3b0 tx=0x7f5e3c001030 comp rx=0 tx=0).stop 2024-04-24T23:16:16.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.618+0000 7f5e55e00640 1 -- 172.21.15.6:0/1916418933 shutdown_connections 2024-04-24T23:16:16.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.618+0000 7f5e55e00640 1 --2- 172.21.15.6:0/1916418933 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f5e3006bf30 0x7f5e3006e3f0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:16.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.619+0000 7f5e55e00640 1 --2- 172.21.15.6:0/1916418933 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5e480523f0 0x7f5e48181ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:16.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.619+0000 7f5e55e00640 1 --2- 172.21.15.6:0/1916418933 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5e48185030 0x7f5e48006ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:16.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.619+0000 7f5e55e00640 1 --2- 172.21.15.6:0/1916418933 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5e340039b0 0x7f5e48052950 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:16.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.619+0000 7f5e55e00640 1 -- 172.21.15.6:0/1916418933 >> 172.21.15.6:0/1916418933 conn(0x7f5e48054610 msgr2=0x7f5e48006100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:16.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.619+0000 7f5e55e00640 1 -- 172.21.15.6:0/1916418933 shutdown_connections 2024-04-24T23:16:16.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.620+0000 7f5e55e00640 1 -- 172.21.15.6:0/1916418933 wait complete. 2024-04-24T23:16:16.648 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574861 got 55834574861 for osd.0 2024-04-24T23:16:16.649 DEBUG:teuthology.parallel:result is None 2024-04-24T23:16:16.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.676+0000 7f24b9800640 1 -- 172.21.15.6:0/3050276165 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f24b40688c0 con 0x7f24b40cc790 2024-04-24T23:16:16.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.676+0000 7f24aaa00640 1 -- 172.21.15.6:0/3050276165 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f24a4058df0 con 0x7f24b40cc790 2024-04-24T23:16:16.679 INFO:teuthology.orchestra.run.smithi006.stdout:55834574862 2024-04-24T23:16:16.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.680+0000 7f24b9800640 1 -- 172.21.15.6:0/3050276165 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f249406bf00 msgr2=0x7f249406e3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:16.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.680+0000 7f24b9800640 1 --2- 172.21.15.6:0/3050276165 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f249406bf00 0x7f249406e3c0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f24a0000f60 tx=0x7f24a0007c80 comp rx=0 tx=0).stop 2024-04-24T23:16:16.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.681+0000 7f24b9800640 1 -- 172.21.15.6:0/3050276165 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f24b40cc790 msgr2=0x7f24b4159af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:16.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.681+0000 7f24b9800640 1 --2- 172.21.15.6:0/3050276165 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f24b40cc790 0x7f24b4159af0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f24a400c780 tx=0x7f24a400cc40 comp rx=0 tx=0).stop 2024-04-24T23:16:16.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.681+0000 7f24b9800640 1 -- 172.21.15.6:0/3050276165 shutdown_connections 2024-04-24T23:16:16.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.681+0000 7f24b9800640 1 --2- 172.21.15.6:0/3050276165 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f249406bf00 0x7f249406e3c0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:16.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.681+0000 7f24b9800640 1 --2- 172.21.15.6:0/3050276165 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f24b4154d00 0x7f24b415ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:16.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.681+0000 7f24b9800640 1 --2- 172.21.15.6:0/3050276165 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f24b40cc790 0x7f24b4159af0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:16.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.681+0000 7f24b9800640 1 --2- 172.21.15.6:0/3050276165 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f24b40cae00 0x7f24b4152fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:16.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.681+0000 7f24b9800640 1 -- 172.21.15.6:0/3050276165 >> 172.21.15.6:0/3050276165 conn(0x7f24b4072c60 msgr2=0x7f24b4140b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:16.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.682+0000 7f24b9800640 1 -- 172.21.15.6:0/3050276165 shutdown_connections 2024-04-24T23:16:16.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.683+0000 7f24b9800640 1 -- 172.21.15.6:0/3050276165 wait complete. 2024-04-24T23:16:16.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.683+0000 7f9908400640 1 -- 172.21.15.6:0/3051729946 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f98dc005580 con 0x7f9900162a40 2024-04-24T23:16:16.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.684+0000 7f98fd600640 1 -- 172.21.15.6:0/3051729946 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f98f8059120 con 0x7f9900162a40 2024-04-24T23:16:16.687 INFO:teuthology.orchestra.run.smithi006.stdout:55834574861 2024-04-24T23:16:16.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.691+0000 7f9908400640 1 -- 172.21.15.6:0/3051729946 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f98e006c030 msgr2=0x7f98e006e4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:16.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.691+0000 7f9908400640 1 --2- 172.21.15.6:0/3051729946 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f98e006c030 0x7f98e006e4f0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f98f4034000 tx=0x7f98f402f000 comp rx=0 tx=0).stop 2024-04-24T23:16:16.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.691+0000 7f9908400640 1 -- 172.21.15.6:0/3051729946 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9900162a40 msgr2=0x7f990015ba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:16.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.691+0000 7f9908400640 1 --2- 172.21.15.6:0/3051729946 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9900162a40 0x7f990015ba30 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f98f800e810 tx=0x7f98f800ecd0 comp rx=0 tx=0).stop 2024-04-24T23:16:16.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.692+0000 7f9908400640 1 -- 172.21.15.6:0/3051729946 shutdown_connections 2024-04-24T23:16:16.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.692+0000 7f9908400640 1 --2- 172.21.15.6:0/3051729946 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f98e006c030 0x7f98e006e4f0 secure :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f98f4034000 tx=0x7f98f402f000 comp rx=0 tx=0).stop 2024-04-24T23:16:16.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.692+0000 7f9908400640 1 --2- 172.21.15.6:0/3051729946 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9900159210 0x7f990015b4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:16.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.692+0000 7f9908400640 1 --2- 172.21.15.6:0/3051729946 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9900162a40 0x7f990015ba30 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:16.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.692+0000 7f9908400640 1 --2- 172.21.15.6:0/3051729946 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f99000c8420 0x7f99000cb850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:16.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.692+0000 7f9908400640 1 -- 172.21.15.6:0/3051729946 >> 172.21.15.6:0/3051729946 conn(0x7f99000c8c50 msgr2=0x7f9900157fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:16.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.692+0000 7f9908400640 1 -- 172.21.15.6:0/3051729946 shutdown_connections 2024-04-24T23:16:16.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.693+0000 7f9908400640 1 -- 172.21.15.6:0/3051729946 wait complete. 2024-04-24T23:16:16.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.694+0000 7fa41f200640 1 -- 172.21.15.6:0/3868174122 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7fa418160ef0 con 0x7fa4180cfa40 2024-04-24T23:16:16.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.694+0000 7fa40fe00640 1 -- 172.21.15.6:0/3868174122 <== mon.2 v2:172.21.15.190:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fa40407d3f0 con 0x7fa4180cfa40 2024-04-24T23:16:16.697 INFO:teuthology.orchestra.run.smithi006.stdout:55834574861 2024-04-24T23:16:16.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.699+0000 7fa41f200640 1 -- 172.21.15.6:0/3868174122 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fa3f806bf30 msgr2=0x7fa3f806e3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:16.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.699+0000 7fa41f200640 1 --2- 172.21.15.6:0/3868174122 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fa3f806bf30 0x7fa3f806e3f0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fa408004840 tx=0x7fa408009040 comp rx=0 tx=0).stop 2024-04-24T23:16:16.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.699+0000 7fa41f200640 1 -- 172.21.15.6:0/3868174122 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa4180cfa40 msgr2=0x7fa4181f57d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:16.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.699+0000 7fa41f200640 1 --2- 172.21.15.6:0/3868174122 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa4180cfa40 0x7fa4181f57d0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fa404031000 tx=0x7fa404004730 comp rx=0 tx=0).stop 2024-04-24T23:16:16.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.699+0000 7fa41f200640 1 -- 172.21.15.6:0/3868174122 shutdown_connections 2024-04-24T23:16:16.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.699+0000 7fa41f200640 1 --2- 172.21.15.6:0/3868174122 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fa3f806bf30 0x7fa3f806e3f0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:16.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.699+0000 7fa41f200640 1 --2- 172.21.15.6:0/3868174122 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa4180cfa40 0x7fa4181f57d0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:16.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.699+0000 7fa41f200640 1 --2- 172.21.15.6:0/3868174122 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa4180d0a50 0x7fa4181524d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:16.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.699+0000 7fa41f200640 1 --2- 172.21.15.6:0/3868174122 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa4180cf660 0x7fa4181f5290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:16.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.700+0000 7fa41f200640 1 -- 172.21.15.6:0/3868174122 >> 172.21.15.6:0/3868174122 conn(0x7fa41806d080 msgr2=0x7fa41815e020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:16.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.702+0000 7fa41f200640 1 -- 172.21.15.6:0/3868174122 shutdown_connections 2024-04-24T23:16:16.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.702+0000 7fa41f200640 1 -- 172.21.15.6:0/3868174122 wait complete. 2024-04-24T23:16:16.714 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574862 got 55834574862 for osd.3 2024-04-24T23:16:16.714 DEBUG:teuthology.parallel:result is None 2024-04-24T23:16:16.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.718+0000 7f0fb6600640 1 -- 172.21.15.6:0/1521201146 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f0f88005850 con 0x7f0fb014fd30 2024-04-24T23:16:16.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.719+0000 7f0fa7400640 1 -- 172.21.15.6:0/1521201146 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f0fa805a920 con 0x7f0fb014fd30 2024-04-24T23:16:16.721 INFO:teuthology.orchestra.run.smithi006.stdout:55834574862 2024-04-24T23:16:16.722 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574861 got 55834574861 for osd.6 2024-04-24T23:16:16.722 DEBUG:teuthology.parallel:result is None 2024-04-24T23:16:16.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.722+0000 7f0fb6600640 1 -- 172.21.15.6:0/1521201146 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f0f9006be60 msgr2=0x7f0f9006e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:16.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.722+0000 7f0fb6600640 1 --2- 172.21.15.6:0/1521201146 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f0f9006be60 0x7f0f9006e320 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f0fb00c36e0 tx=0x7f0fa0030000 comp rx=0 tx=0).stop 2024-04-24T23:16:16.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.723+0000 7f0fb6600640 1 -- 172.21.15.6:0/1521201146 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0fb014fd30 msgr2=0x7f0fb00b1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:16.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.723+0000 7f0fb6600640 1 --2- 172.21.15.6:0/1521201146 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0fb014fd30 0x7f0fb00b1e30 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f0fa8007f70 tx=0x7f0fa800f5e0 comp rx=0 tx=0).stop 2024-04-24T23:16:16.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.723+0000 7f0fb6600640 1 -- 172.21.15.6:0/1521201146 shutdown_connections 2024-04-24T23:16:16.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.723+0000 7f0fb6600640 1 --2- 172.21.15.6:0/1521201146 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f0f9006be60 0x7f0f9006e320 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:16.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.723+0000 7f0fb6600640 1 --2- 172.21.15.6:0/1521201146 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0fb00b2370 0x7f0fb0154200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:16.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.723+0000 7f0fb6600640 1 --2- 172.21.15.6:0/1521201146 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0fb014fd30 0x7f0fb00b1e30 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:16.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.723+0000 7f0fb6600640 1 --2- 172.21.15.6:0/1521201146 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0fb014f410 0x7f0fb014f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:16.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.723+0000 7f0fb6600640 1 -- 172.21.15.6:0/1521201146 >> 172.21.15.6:0/1521201146 conn(0x7f0fb0066ea0 msgr2=0x7f0fb00c3710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:16.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.723+0000 7f0fb6600640 1 -- 172.21.15.6:0/1521201146 shutdown_connections 2024-04-24T23:16:16.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.723+0000 7f0fb6600640 1 -- 172.21.15.6:0/1521201146 wait complete. 2024-04-24T23:16:16.730 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574861 got 55834574861 for osd.7 2024-04-24T23:16:16.730 DEBUG:teuthology.parallel:result is None 2024-04-24T23:16:16.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.730+0000 7f6228a00640 1 -- 172.21.15.6:0/1196377210 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f61fc005850 con 0x7f6224071590 2024-04-24T23:16:16.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.731+0000 7f621aa00640 1 -- 172.21.15.6:0/1196377210 <== mon.2 v2:172.21.15.190:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f621c07a290 con 0x7f6224071590 2024-04-24T23:16:16.733 INFO:teuthology.orchestra.run.smithi006.stdout:55834574861 2024-04-24T23:16:16.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.733+0000 7f6228a00640 1 -- 172.21.15.6:0/1196377210 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f6204070440 msgr2=0x7f6204072900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:16.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.733+0000 7f6228a00640 1 --2- 172.21.15.6:0/1196377210 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f6204070440 0x7f6204072900 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f622406f890 tx=0x7f62140090e0 comp rx=0 tx=0).stop 2024-04-24T23:16:16.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.733+0000 7f6228a00640 1 -- 172.21.15.6:0/1196377210 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6224071590 msgr2=0x7f62241f0300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:16.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.733+0000 7f6228a00640 1 --2- 172.21.15.6:0/1196377210 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6224071590 0x7f62241f0300 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f621c037040 tx=0x7f621c004a40 comp rx=0 tx=0).stop 2024-04-24T23:16:16.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.734+0000 7f6228a00640 1 -- 172.21.15.6:0/1196377210 shutdown_connections 2024-04-24T23:16:16.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.734+0000 7f6228a00640 1 --2- 172.21.15.6:0/1196377210 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f6204070440 0x7f6204072900 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:16.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.734+0000 7f6228a00640 1 --2- 172.21.15.6:0/1196377210 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6224150d40 0x7f6224151940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:16.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.734+0000 7f6228a00640 1 --2- 172.21.15.6:0/1196377210 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f62240c74b0 0x7f6224145120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:16.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.734+0000 7f6228a00640 1 --2- 172.21.15.6:0/1196377210 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6224071590 0x7f62241f0300 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:16.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.734+0000 7f6228a00640 1 -- 172.21.15.6:0/1196377210 >> 172.21.15.6:0/1196377210 conn(0x7f62240cc1e0 msgr2=0x7f6224071d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:16.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.735+0000 7f6228a00640 1 -- 172.21.15.6:0/1196377210 shutdown_connections 2024-04-24T23:16:16.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.735+0000 7f6228a00640 1 -- 172.21.15.6:0/1196377210 wait complete. 2024-04-24T23:16:16.742 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574862 got 55834574862 for osd.4 2024-04-24T23:16:16.742 DEBUG:teuthology.parallel:result is None 2024-04-24T23:16:16.753 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574861 got 55834574861 for osd.2 2024-04-24T23:16:16.753 DEBUG:teuthology.parallel:result is None 2024-04-24T23:16:16.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.791+0000 7fc2ef000640 1 -- 172.21.15.6:0/3248479006 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7fc2e814dff0 con 0x7fc2e8071590 2024-04-24T23:16:16.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.792+0000 7fc2e5600640 1 -- 172.21.15.6:0/3248479006 <== mon.2 v2:172.21.15.190:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fc2dc05cd80 con 0x7fc2e8071590 2024-04-24T23:16:16.794 INFO:teuthology.orchestra.run.smithi006.stdout:55834574861 2024-04-24T23:16:16.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.795+0000 7fc2ef000640 1 -- 172.21.15.6:0/3248479006 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fc2c807cec0 msgr2=0x7fc2c807f380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:16.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.795+0000 7fc2ef000640 1 --2- 172.21.15.6:0/3248479006 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fc2c807cec0 0x7fc2c807f380 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fc2e80cc970 tx=0x7fc2d4009040 comp rx=0 tx=0).stop 2024-04-24T23:16:16.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.795+0000 7fc2ef000640 1 -- 172.21.15.6:0/3248479006 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc2e8071590 msgr2=0x7fc2e813fe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:16.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.795+0000 7fc2ef000640 1 --2- 172.21.15.6:0/3248479006 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc2e8071590 0x7fc2e813fe80 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fc2dc00c470 tx=0x7fc2dc001e70 comp rx=0 tx=0).stop 2024-04-24T23:16:16.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.795+0000 7fc2ef000640 1 -- 172.21.15.6:0/3248479006 shutdown_connections 2024-04-24T23:16:16.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.795+0000 7fc2ef000640 1 --2- 172.21.15.6:0/3248479006 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fc2c807cec0 0x7fc2c807f380 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:16.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.795+0000 7fc2ef000640 1 --2- 172.21.15.6:0/3248479006 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc2e8156cc0 0x7fc2e81593c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:16.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.795+0000 7fc2ef000640 1 --2- 172.21.15.6:0/3248479006 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc2e8150d40 0x7fc2e8158e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:16.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.795+0000 7fc2ef000640 1 --2- 172.21.15.6:0/3248479006 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc2e8071590 0x7fc2e813fe80 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:16.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.795+0000 7fc2ef000640 1 -- 172.21.15.6:0/3248479006 >> 172.21.15.6:0/3248479006 conn(0x7fc2e80cc1e0 msgr2=0x7fc2e80c7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:16.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.795+0000 7fc2ef000640 1 -- 172.21.15.6:0/3248479006 shutdown_connections 2024-04-24T23:16:16.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.795+0000 7fc2ef000640 1 -- 172.21.15.6:0/3248479006 wait complete. 2024-04-24T23:16:16.814 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574861 got 55834574861 for osd.1 2024-04-24T23:16:16.814 DEBUG:teuthology.parallel:result is None 2024-04-24T23:16:16.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.877+0000 7f1ca2a00640 1 -- 172.21.15.6:0/3287254073 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f1c9c0688c0 con 0x7f1c9c15b690 2024-04-24T23:16:16.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.878+0000 7f1c99600640 1 -- 172.21.15.6:0/3287254073 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f1c8c0798f0 con 0x7f1c9c15b690 2024-04-24T23:16:16.880 INFO:teuthology.orchestra.run.smithi006.stdout:55834574861 2024-04-24T23:16:16.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.881+0000 7f1ca2a00640 1 -- 172.21.15.6:0/3287254073 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f1c8006be80 msgr2=0x7f1c8006e340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:16.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.881+0000 7f1ca2a00640 1 --2- 172.21.15.6:0/3287254073 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f1c8006be80 0x7f1c8006e340 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f1c94009bd0 tx=0x7f1c9400b040 comp rx=0 tx=0).stop 2024-04-24T23:16:16.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.881+0000 7f1ca2a00640 1 -- 172.21.15.6:0/3287254073 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1c9c15b690 msgr2=0x7f1c9c1f83a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:16.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.881+0000 7f1ca2a00640 1 --2- 172.21.15.6:0/3287254073 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1c9c15b690 0x7f1c9c1f83a0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f1c9c08df20 tx=0x7f1c8c0046c0 comp rx=0 tx=0).stop 2024-04-24T23:16:16.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.882+0000 7f1ca2a00640 1 -- 172.21.15.6:0/3287254073 shutdown_connections 2024-04-24T23:16:16.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.882+0000 7f1ca2a00640 1 --2- 172.21.15.6:0/3287254073 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f1c8006be80 0x7f1c8006e340 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:16.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.882+0000 7f1ca2a00640 1 --2- 172.21.15.6:0/3287254073 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1c9c155010 0x7f1c9c1f7e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:16.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.882+0000 7f1ca2a00640 1 --2- 172.21.15.6:0/3287254073 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1c9c15b690 0x7f1c9c1f83a0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:16.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.882+0000 7f1ca2a00640 1 --2- 172.21.15.6:0/3287254073 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1c9c06be80 0x7f1c9c0c6380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:16.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.882+0000 7f1ca2a00640 1 -- 172.21.15.6:0/3287254073 >> 172.21.15.6:0/3287254073 conn(0x7f1c9c067080 msgr2=0x7f1c9c0b4940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:16.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.882+0000 7f1ca2a00640 1 -- 172.21.15.6:0/3287254073 shutdown_connections 2024-04-24T23:16:16.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:16.882+0000 7f1ca2a00640 1 -- 172.21.15.6:0/3287254073 wait complete. 2024-04-24T23:16:16.894 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-04-24T23:16:16.892+0000 7fef1ce00640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-04-24T23:16:16.902 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574861 got 55834574861 for osd.5 2024-04-24T23:16:16.902 DEBUG:teuthology.parallel:result is None 2024-04-24T23:16:16.902 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-24T23:16:16.902 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T23:16:17.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.093+0000 7f4d01400640 1 Processor -- start 2024-04-24T23:16:17.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.093+0000 7f4d01400640 1 -- start start 2024-04-24T23:16:17.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.094+0000 7f4d01400640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4cfc154d90 0x7f4cfc0cbe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:17.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.094+0000 7f4d01400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f4cfc0cb090 0x7f4cfc0cc3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:17.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.094+0000 7f4d01400640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4cfc155170 con 0x7f4cfc0d0820 2024-04-24T23:16:17.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.094+0000 7f4d01400640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f4cfc0cc9a0 con 0x7f4cfc0cb090 2024-04-24T23:16:17.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.094+0000 7f4d01400640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f4cfc15ad60 con 0x7f4cfc154d90 2024-04-24T23:16:17.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.095+0000 7f4cfb400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f4cfc0cb090 0x7f4cfc0cc3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:17.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.095+0000 7f4cfb400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f4cfc0cb090 0x7f4cfc0cc3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:43478/0 (socket says 172.21.15.6:43478) 2024-04-24T23:16:17.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.095+0000 7f4cfb400640 1 -- 172.21.15.6:0/178932655 learned_addr learned my addr 172.21.15.6:0/178932655 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:16:17.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.095+0000 7f4d00a00640 1 --1- 172.21.15.6:0/178932655 >> v1:172.21.15.6:6789/0 conn(0x7f4cfc0d0820 0x7f4cfc15e000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:39688/0 (socket says 172.21.15.6:39688) 2024-04-24T23:16:17.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.095+0000 7f4cfbe00640 1 --2- 172.21.15.6:0/178932655 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4cfc154d90 0x7f4cfc0cbe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:17.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.095+0000 7f4cfb400640 1 -- 172.21.15.6:0/178932655 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4cfc154d90 msgr2=0x7f4cfc0cbe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:16:17.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.095+0000 7f4cfb400640 1 --2- 172.21.15.6:0/178932655 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4cfc154d90 0x7f4cfc0cbe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:17.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.095+0000 7f4cfb400640 1 -- 172.21.15.6:0/178932655 >> v1:172.21.15.6:6789/0 conn(0x7f4cfc0d0820 legacy=0x7f4cfc15e000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:17.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.095+0000 7f4cfb400640 1 -- 172.21.15.6:0/178932655 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4cfc15b040 con 0x7f4cfc0cb090 2024-04-24T23:16:17.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.097+0000 7f4cfb400640 1 --2- 172.21.15.6:0/178932655 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4cfc0cb090 0x7f4cfc0cc3c0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f4ce4009870 tx=0x7f4ce402a300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48c40eef5c51ef52 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:17.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.097+0000 7f4cfaa00640 1 -- 172.21.15.6:0/178932655 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ce4037070 con 0x7f4cfc0cb090 2024-04-24T23:16:17.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.097+0000 7f4cfaa00640 1 -- 172.21.15.6:0/178932655 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ce402ae50 con 0x7f4cfc0cb090 2024-04-24T23:16:17.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.097+0000 7f4cfaa00640 1 -- 172.21.15.6:0/178932655 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ce4032860 con 0x7f4cfc0cb090 2024-04-24T23:16:17.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.097+0000 7f4d01400640 1 -- 172.21.15.6:0/178932655 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4cfc0cb090 msgr2=0x7f4cfc0cc3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:17.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.097+0000 7f4d01400640 1 --2- 172.21.15.6:0/178932655 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4cfc0cb090 0x7f4cfc0cc3c0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f4ce4009870 tx=0x7f4ce402a300 comp rx=0 tx=0).stop 2024-04-24T23:16:17.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.097+0000 7f4d01400640 1 -- 172.21.15.6:0/178932655 shutdown_connections 2024-04-24T23:16:17.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.097+0000 7f4d01400640 1 --2- 172.21.15.6:0/178932655 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4cfc0cb090 0x7f4cfc0cc3c0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:17.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.097+0000 7f4d01400640 1 --2- 172.21.15.6:0/178932655 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4cfc154d90 0x7f4cfc0cbe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:17.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.097+0000 7f4d01400640 1 -- 172.21.15.6:0/178932655 >> 172.21.15.6:0/178932655 conn(0x7f4cfc06be10 msgr2=0x7f4cfc06c1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:17.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.097+0000 7f4d01400640 1 -- 172.21.15.6:0/178932655 shutdown_connections 2024-04-24T23:16:17.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.097+0000 7f4d01400640 1 -- 172.21.15.6:0/178932655 wait complete. 2024-04-24T23:16:17.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.099+0000 7f4d01400640 1 Processor -- start 2024-04-24T23:16:17.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.099+0000 7f4d01400640 1 -- start start 2024-04-24T23:16:17.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.101+0000 7f4d01400640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4cfc0cb090 0x7f4cfc159ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:17.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.101+0000 7f4d01400640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4cfc0d0820 0x7f4cfc159800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:17.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.101+0000 7f4d01400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4cfc154d90 0x7f4cfc0cfa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:17.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.101+0000 7f4d01400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f4cfc15aa30 con 0x7f4cfc154d90 2024-04-24T23:16:17.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.101+0000 7f4d01400640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f4cfc15b600 con 0x7f4cfc0d0820 2024-04-24T23:16:17.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.101+0000 7f4d01400640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f4cfc15e710 con 0x7f4cfc0cb090 2024-04-24T23:16:17.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.101+0000 7f4d00a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4cfc154d90 0x7f4cfc0cfa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:17.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.101+0000 7f4d00a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4cfc154d90 0x7f4cfc0cfa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:43492/0 (socket says 172.21.15.6:43492) 2024-04-24T23:16:17.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.101+0000 7f4d00a00640 1 -- 172.21.15.6:0/2503261045 learned_addr learned my addr 172.21.15.6:0/2503261045 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:16:17.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.101+0000 7f4cfb400640 1 --2- 172.21.15.6:0/2503261045 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4cfc0d0820 0x7f4cfc159800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:17.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.101+0000 7f4cfbe00640 1 --2- 172.21.15.6:0/2503261045 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4cfc0cb090 0x7f4cfc159ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:17.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.101+0000 7f4d00a00640 1 -- 172.21.15.6:0/2503261045 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4cfc0cb090 msgr2=0x7f4cfc159ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:17.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.101+0000 7f4d00a00640 1 --2- 172.21.15.6:0/2503261045 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4cfc0cb090 0x7f4cfc159ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:17.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.101+0000 7f4d00a00640 1 -- 172.21.15.6:0/2503261045 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4cfc0d0820 msgr2=0x7f4cfc159800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:17.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.101+0000 7f4d00a00640 1 --2- 172.21.15.6:0/2503261045 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4cfc0d0820 0x7f4cfc159800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:17.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.101+0000 7f4d00a00640 1 -- 172.21.15.6:0/2503261045 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4cfc0cc020 con 0x7f4cfc154d90 2024-04-24T23:16:17.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.101+0000 7f4d00a00640 1 --2- 172.21.15.6:0/2503261045 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4cfc154d90 0x7f4cfc0cfa70 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f4cf400c7b0 tx=0x7f4cf400cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:17.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.101+0000 7f4cf3e00640 1 -- 172.21.15.6:0/2503261045 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cf4017070 con 0x7f4cfc154d90 2024-04-24T23:16:17.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.101+0000 7f4cf3e00640 1 -- 172.21.15.6:0/2503261045 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4cf400a640 con 0x7f4cfc154d90 2024-04-24T23:16:17.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.101+0000 7f4cf3e00640 1 -- 172.21.15.6:0/2503261045 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cf400a9f0 con 0x7f4cfc154d90 2024-04-24T23:16:17.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.101+0000 7f4d01400640 1 -- 172.21.15.6:0/2503261045 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4cfc15be00 con 0x7f4cfc154d90 2024-04-24T23:16:17.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.101+0000 7f4d01400640 1 -- 172.21.15.6:0/2503261045 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4cfc158d20 con 0x7f4cfc154d90 2024-04-24T23:16:17.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.103+0000 7f4d01400640 1 -- 172.21.15.6:0/2503261045 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4cfc075b30 con 0x7f4cfc154d90 2024-04-24T23:16:17.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.104+0000 7f4cf3e00640 1 -- 172.21.15.6:0/2503261045 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f4cf401a070 con 0x7f4cfc154d90 2024-04-24T23:16:17.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.104+0000 7f4cf3e00640 1 --2- 172.21.15.6:0/2503261045 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f4cd806be30 0x7f4cd806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:17.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.106+0000 7f4cf3e00640 1 -- 172.21.15.6:0/2503261045 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f4cf408a6a0 con 0x7f4cfc154d90 2024-04-24T23:16:17.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.106+0000 7f4cfbe00640 1 --2- 172.21.15.6:0/2503261045 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f4cd806be30 0x7f4cd806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:17.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.107+0000 7f4cfbe00640 1 --2- 172.21.15.6:0/2503261045 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f4cd806be30 0x7f4cd806e2f0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f4cec009aa0 tx=0x7f4cec009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:17.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.109+0000 7f4cf3e00640 1 -- 172.21.15.6:0/2503261045 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f4cf40586b0 con 0x7f4cfc154d90 2024-04-24T23:16:17.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.310+0000 7f4d01400640 1 -- 172.21.15.6:0/2503261045 --> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f4cfc0cc370 con 0x7f4cd806be30 2024-04-24T23:16:17.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.319+0000 7f4cf3e00640 1 -- 172.21.15.6:0/2503261045 <== mgr.4105 v2:172.21.15.190:6832/806769663 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+332574 (secure 0 0 0) 0x7f4cfc0cc370 con 0x7f4cd806be30 2024-04-24T23:16:17.325 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:16:17.327 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-04-24T23:16:17.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.324+0000 7f4d01400640 1 -- 172.21.15.6:0/2503261045 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f4cd806be30 msgr2=0x7f4cd806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:17.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.324+0000 7f4d01400640 1 --2- 172.21.15.6:0/2503261045 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f4cd806be30 0x7f4cd806e2f0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f4cec009aa0 tx=0x7f4cec009190 comp rx=0 tx=0).stop 2024-04-24T23:16:17.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.324+0000 7f4d01400640 1 -- 172.21.15.6:0/2503261045 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4cfc154d90 msgr2=0x7f4cfc0cfa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:17.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.324+0000 7f4d01400640 1 --2- 172.21.15.6:0/2503261045 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4cfc154d90 0x7f4cfc0cfa70 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f4cf400c7b0 tx=0x7f4cf400cc70 comp rx=0 tx=0).stop 2024-04-24T23:16:17.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.325+0000 7f4d01400640 1 -- 172.21.15.6:0/2503261045 shutdown_connections 2024-04-24T23:16:17.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.325+0000 7f4d01400640 1 --2- 172.21.15.6:0/2503261045 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f4cd806be30 0x7f4cd806e2f0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:17.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.325+0000 7f4d01400640 1 --2- 172.21.15.6:0/2503261045 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4cfc154d90 0x7f4cfc0cfa70 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:17.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.325+0000 7f4d01400640 1 --2- 172.21.15.6:0/2503261045 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4cfc0d0820 0x7f4cfc159800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:17.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.325+0000 7f4d01400640 1 --2- 172.21.15.6:0/2503261045 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4cfc0cb090 0x7f4cfc159ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:17.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.325+0000 7f4d01400640 1 -- 172.21.15.6:0/2503261045 >> 172.21.15.6:0/2503261045 conn(0x7f4cfc06be10 msgr2=0x7f4cfc14fad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:17.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.325+0000 7f4d01400640 1 -- 172.21.15.6:0/2503261045 shutdown_connections 2024-04-24T23:16:17.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.325+0000 7f4d01400640 1 -- 172.21.15.6:0/2503261045 wait complete. 2024-04-24T23:16:17.344 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":44,"stamp":"2024-04-24T23:16:17.219061+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596414,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":247,"num_write_kb":2447,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":26,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":194,"ondisk_log_size":194,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219384,"kb_used_data":3252,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755336,"statfs":{"total":773094113280,"available":772869464064,"internally_reserved":0,"allocated":3330048,"data_stored":1994850,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1406,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":39,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.532076"},"pg_stats":[{"pgid":"2.1f","version":"23'4","reported_seq":26,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.051292+0000","last_change":"2024-04-24T23:15:35.964646+0000","last_active":"2024-04-24T23:15:42.051292+0000","last_peered":"2024-04-24T23:15:42.051292+0000","last_clean":"2024-04-24T23:15:42.051292+0000","last_became_active":"2024-04-24T23:15:35.964357+0000","last_became_peered":"2024-04-24T23:15:35.964357+0000","last_unstale":"2024-04-24T23:15:42.051292+0000","last_undegraded":"2024-04-24T23:15:42.051292+0000","last_fullsized":"2024-04-24T23:15:42.051292+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:01:17.912993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.295716+0000","last_change":"2024-04-24T23:15:36.989772+0000","last_active":"2024-04-24T23:15:42.295716+0000","last_peered":"2024-04-24T23:15:42.295716+0000","last_clean":"2024-04-24T23:15:42.295716+0000","last_became_active":"2024-04-24T23:15:36.989543+0000","last_became_peered":"2024-04-24T23:15:36.989543+0000","last_unstale":"2024-04-24T23:15:42.295716+0000","last_undegraded":"2024-04-24T23:15:42.295716+0000","last_fullsized":"2024-04-24T23:15:42.295716+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:37:16.737354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.187704+0000","last_change":"2024-04-24T23:15:41.017182+0000","last_active":"2024-04-24T23:15:43.187704+0000","last_peered":"2024-04-24T23:15:43.187704+0000","last_clean":"2024-04-24T23:15:43.187704+0000","last_became_active":"2024-04-24T23:15:41.016719+0000","last_became_peered":"2024-04-24T23:15:41.016719+0000","last_unstale":"2024-04-24T23:15:43.187704+0000","last_undegraded":"2024-04-24T23:15:43.187704+0000","last_fullsized":"2024-04-24T23:15:43.187704+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:42:11.186486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4,1],"acting":[7,6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1e","version":"23'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.295817+0000","last_change":"2024-04-24T23:15:35.946083+0000","last_active":"2024-04-24T23:15:42.295817+0000","last_peered":"2024-04-24T23:15:42.295817+0000","last_clean":"2024-04-24T23:15:42.295817+0000","last_became_active":"2024-04-24T23:15:35.945227+0000","last_became_peered":"2024-04-24T23:15:35.945227+0000","last_unstale":"2024-04-24T23:15:42.295817+0000","last_undegraded":"2024-04-24T23:15:42.295817+0000","last_fullsized":"2024-04-24T23:15:42.295817+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:05:17.727725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.295775+0000","last_change":"2024-04-24T23:15:36.998245+0000","last_active":"2024-04-24T23:15:42.295775+0000","last_peered":"2024-04-24T23:15:42.295775+0000","last_clean":"2024-04-24T23:15:42.295775+0000","last_became_active":"2024-04-24T23:15:36.998021+0000","last_became_peered":"2024-04-24T23:15:36.998021+0000","last_unstale":"2024-04-24T23:15:42.295775+0000","last_undegraded":"2024-04-24T23:15:42.295775+0000","last_fullsized":"2024-04-24T23:15:42.295775+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:03:42.359117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.106527+0000","last_change":"2024-04-24T23:15:41.008313+0000","last_active":"2024-04-24T23:15:42.106527+0000","last_peered":"2024-04-24T23:15:42.106527+0000","last_clean":"2024-04-24T23:15:42.106527+0000","last_became_active":"2024-04-24T23:15:41.007908+0000","last_became_peered":"2024-04-24T23:15:41.007908+0000","last_unstale":"2024-04-24T23:15:42.106527+0000","last_undegraded":"2024-04-24T23:15:42.106527+0000","last_fullsized":"2024-04-24T23:15:42.106527+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:02:11.902575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4,6],"acting":[2,7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1d","version":"23'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.193739+0000","last_change":"2024-04-24T23:15:35.968312+0000","last_active":"2024-04-24T23:15:43.193739+0000","last_peered":"2024-04-24T23:15:43.193739+0000","last_clean":"2024-04-24T23:15:43.193739+0000","last_became_active":"2024-04-24T23:15:35.967873+0000","last_became_peered":"2024-04-24T23:15:35.967873+0000","last_unstale":"2024-04-24T23:15:43.193739+0000","last_undegraded":"2024-04-24T23:15:43.193739+0000","last_fullsized":"2024-04-24T23:15:43.193739+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:21:40.116977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.733492+0000","last_change":"2024-04-24T23:15:36.997361+0000","last_active":"2024-04-24T23:15:42.733492+0000","last_peered":"2024-04-24T23:15:42.733492+0000","last_clean":"2024-04-24T23:15:42.733492+0000","last_became_active":"2024-04-24T23:15:36.997083+0000","last_became_peered":"2024-04-24T23:15:36.997083+0000","last_unstale":"2024-04-24T23:15:42.733492+0000","last_undegraded":"2024-04-24T23:15:42.733492+0000","last_fullsized":"2024-04-24T23:15:42.733492+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:47:59.334093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.107355+0000","last_change":"2024-04-24T23:15:41.025153+0000","last_active":"2024-04-24T23:15:42.107355+0000","last_peered":"2024-04-24T23:15:42.107355+0000","last_clean":"2024-04-24T23:15:42.107355+0000","last_became_active":"2024-04-24T23:15:41.024739+0000","last_became_peered":"2024-04-24T23:15:41.024739+0000","last_unstale":"2024-04-24T23:15:42.107355+0000","last_undegraded":"2024-04-24T23:15:42.107355+0000","last_fullsized":"2024-04-24T23:15:42.107355+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:31:00.160213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3,6],"acting":[2,5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1c","version":"23'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.193705+0000","last_change":"2024-04-24T23:15:35.967338+0000","last_active":"2024-04-24T23:15:43.193705+0000","last_peered":"2024-04-24T23:15:43.193705+0000","last_clean":"2024-04-24T23:15:43.193705+0000","last_became_active":"2024-04-24T23:15:35.966951+0000","last_became_peered":"2024-04-24T23:15:35.966951+0000","last_unstale":"2024-04-24T23:15:43.193705+0000","last_undegraded":"2024-04-24T23:15:43.193705+0000","last_fullsized":"2024-04-24T23:15:43.193705+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:08:05.248330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.191745+0000","last_change":"2024-04-24T23:15:36.998072+0000","last_active":"2024-04-24T23:15:43.191745+0000","last_peered":"2024-04-24T23:15:43.191745+0000","last_clean":"2024-04-24T23:15:43.191745+0000","last_became_active":"2024-04-24T23:15:36.997735+0000","last_became_peered":"2024-04-24T23:15:36.997735+0000","last_unstale":"2024-04-24T23:15:43.191745+0000","last_undegraded":"2024-04-24T23:15:43.191745+0000","last_fullsized":"2024-04-24T23:15:43.191745+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:22:06.869644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.106640+0000","last_change":"2024-04-24T23:15:41.008895+0000","last_active":"2024-04-24T23:15:42.106640+0000","last_peered":"2024-04-24T23:15:42.106640+0000","last_clean":"2024-04-24T23:15:42.106640+0000","last_became_active":"2024-04-24T23:15:41.006199+0000","last_became_peered":"2024-04-24T23:15:41.006199+0000","last_unstale":"2024-04-24T23:15:42.106640+0000","last_undegraded":"2024-04-24T23:15:42.106640+0000","last_fullsized":"2024-04-24T23:15:42.106640+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:13:34.583602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5,7],"acting":[2,6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.191106+0000","last_change":"2024-04-24T23:15:37.004074+0000","last_active":"2024-04-24T23:15:43.191106+0000","last_peered":"2024-04-24T23:15:43.191106+0000","last_clean":"2024-04-24T23:15:43.191106+0000","last_became_active":"2024-04-24T23:15:37.003544+0000","last_became_peered":"2024-04-24T23:15:37.003544+0000","last_unstale":"2024-04-24T23:15:43.191106+0000","last_undegraded":"2024-04-24T23:15:43.191106+0000","last_fullsized":"2024-04-24T23:15:43.191106+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:38:28.024702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"24'4","reported_seq":26,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:16:13.162728+0000","last_change":"2024-04-24T23:15:35.963000+0000","last_active":"2024-04-24T23:16:13.162728+0000","last_peered":"2024-04-24T23:16:13.162728+0000","last_clean":"2024-04-24T23:16:13.162728+0000","last_became_active":"2024-04-24T23:15:35.961960+0000","last_became_peered":"2024-04-24T23:15:35.961960+0000","last_unstale":"2024-04-24T23:16:13.162728+0000","last_undegraded":"2024-04-24T23:16:13.162728+0000","last_fullsized":"2024-04-24T23:16:13.162728+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:56:26.651193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.191430+0000","last_change":"2024-04-24T23:15:41.022481+0000","last_active":"2024-04-24T23:15:43.191430+0000","last_peered":"2024-04-24T23:15:43.191430+0000","last_clean":"2024-04-24T23:15:43.191430+0000","last_became_active":"2024-04-24T23:15:41.020034+0000","last_became_peered":"2024-04-24T23:15:41.020034+0000","last_unstale":"2024-04-24T23:15:43.191430+0000","last_undegraded":"2024-04-24T23:15:43.191430+0000","last_fullsized":"2024-04-24T23:15:43.191430+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:27:25.337503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,5],"acting":[3,1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.187801+0000","last_change":"2024-04-24T23:15:36.972020+0000","last_active":"2024-04-24T23:15:43.187801+0000","last_peered":"2024-04-24T23:15:43.187801+0000","last_clean":"2024-04-24T23:15:43.187801+0000","last_became_active":"2024-04-24T23:15:36.971583+0000","last_became_peered":"2024-04-24T23:15:36.971583+0000","last_unstale":"2024-04-24T23:15:43.187801+0000","last_undegraded":"2024-04-24T23:15:43.187801+0000","last_fullsized":"2024-04-24T23:15:43.187801+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:29:05.568608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.050091+0000","last_change":"2024-04-24T23:15:35.945916+0000","last_active":"2024-04-24T23:15:42.050091+0000","last_peered":"2024-04-24T23:15:42.050091+0000","last_clean":"2024-04-24T23:15:42.050091+0000","last_became_active":"2024-04-24T23:15:35.945610+0000","last_became_peered":"2024-04-24T23:15:35.945610+0000","last_unstale":"2024-04-24T23:15:42.050091+0000","last_undegraded":"2024-04-24T23:15:42.050091+0000","last_fullsized":"2024-04-24T23:15:42.050091+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:57:05.976074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.192627+0000","last_change":"2024-04-24T23:15:41.030165+0000","last_active":"2024-04-24T23:15:43.192627+0000","last_peered":"2024-04-24T23:15:43.192627+0000","last_clean":"2024-04-24T23:15:43.192627+0000","last_became_active":"2024-04-24T23:15:41.029685+0000","last_became_peered":"2024-04-24T23:15:41.029685+0000","last_unstale":"2024-04-24T23:15:43.192627+0000","last_undegraded":"2024-04-24T23:15:43.192627+0000","last_fullsized":"2024-04-24T23:15:43.192627+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:20:43.498190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6,1],"acting":[0,4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.049168+0000","last_change":"2024-04-24T23:15:36.965949+0000","last_active":"2024-04-24T23:15:42.049168+0000","last_peered":"2024-04-24T23:15:42.049168+0000","last_clean":"2024-04-24T23:15:42.049168+0000","last_became_active":"2024-04-24T23:15:36.965658+0000","last_became_peered":"2024-04-24T23:15:36.965658+0000","last_unstale":"2024-04-24T23:15:42.049168+0000","last_undegraded":"2024-04-24T23:15:42.049168+0000","last_fullsized":"2024-04-24T23:15:42.049168+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:38:28.024702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"24'3","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:16:13.164854+0000","last_change":"2024-04-24T23:15:35.972165+0000","last_active":"2024-04-24T23:16:13.164854+0000","last_peered":"2024-04-24T23:16:13.164854+0000","last_clean":"2024-04-24T23:16:13.164854+0000","last_became_active":"2024-04-24T23:15:35.971972+0000","last_became_peered":"2024-04-24T23:15:35.971972+0000","last_unstale":"2024-04-24T23:16:13.164854+0000","last_undegraded":"2024-04-24T23:16:13.164854+0000","last_fullsized":"2024-04-24T23:16:13.164854+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:11:12.088698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.188224+0000","last_change":"2024-04-24T23:15:41.019799+0000","last_active":"2024-04-24T23:15:43.188224+0000","last_peered":"2024-04-24T23:15:43.188224+0000","last_clean":"2024-04-24T23:15:43.188224+0000","last_became_active":"2024-04-24T23:15:41.019339+0000","last_became_peered":"2024-04-24T23:15:41.019339+0000","last_unstale":"2024-04-24T23:15:43.188224+0000","last_undegraded":"2024-04-24T23:15:43.188224+0000","last_fullsized":"2024-04-24T23:15:43.188224+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:59:34.601004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2,3],"acting":[7,4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.190629+0000","last_change":"2024-04-24T23:15:36.991991+0000","last_active":"2024-04-24T23:15:43.190629+0000","last_peered":"2024-04-24T23:15:43.190629+0000","last_clean":"2024-04-24T23:15:43.190629+0000","last_became_active":"2024-04-24T23:15:36.989336+0000","last_became_peered":"2024-04-24T23:15:36.989336+0000","last_unstale":"2024-04-24T23:15:43.190629+0000","last_undegraded":"2024-04-24T23:15:43.190629+0000","last_fullsized":"2024-04-24T23:15:43.190629+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:25:56.038903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.8","version":"24'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:16:13.164232+0000","last_change":"2024-04-24T23:15:35.965443+0000","last_active":"2024-04-24T23:16:13.164232+0000","last_peered":"2024-04-24T23:16:13.164232+0000","last_clean":"2024-04-24T23:16:13.164232+0000","last_became_active":"2024-04-24T23:15:35.965043+0000","last_became_peered":"2024-04-24T23:15:35.965043+0000","last_unstale":"2024-04-24T23:16:13.164232+0000","last_undegraded":"2024-04-24T23:16:13.164232+0000","last_fullsized":"2024-04-24T23:16:13.164232+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:07:23.320042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.107146+0000","last_change":"2024-04-24T23:15:41.025131+0000","last_active":"2024-04-24T23:15:42.107146+0000","last_peered":"2024-04-24T23:15:42.107146+0000","last_clean":"2024-04-24T23:15:42.107146+0000","last_became_active":"2024-04-24T23:15:41.024731+0000","last_became_peered":"2024-04-24T23:15:41.024731+0000","last_unstale":"2024-04-24T23:15:42.107146+0000","last_undegraded":"2024-04-24T23:15:42.107146+0000","last_fullsized":"2024-04-24T23:15:42.107146+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:28:07.952854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,7],"acting":[2,1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.049954+0000","last_change":"2024-04-24T23:15:36.965850+0000","last_active":"2024-04-24T23:15:42.049954+0000","last_peered":"2024-04-24T23:15:42.049954+0000","last_clean":"2024-04-24T23:15:42.049954+0000","last_became_active":"2024-04-24T23:15:36.965547+0000","last_became_peered":"2024-04-24T23:15:36.965547+0000","last_unstale":"2024-04-24T23:15:42.049954+0000","last_undegraded":"2024-04-24T23:15:42.049954+0000","last_fullsized":"2024-04-24T23:15:42.049954+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:06:37.125573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"24'4","reported_seq":26,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:16:13.163117+0000","last_change":"2024-04-24T23:15:35.971040+0000","last_active":"2024-04-24T23:16:13.163117+0000","last_peered":"2024-04-24T23:16:13.163117+0000","last_clean":"2024-04-24T23:16:13.163117+0000","last_became_active":"2024-04-24T23:15:35.970184+0000","last_became_peered":"2024-04-24T23:15:35.970184+0000","last_unstale":"2024-04-24T23:16:13.163117+0000","last_undegraded":"2024-04-24T23:16:13.163117+0000","last_fullsized":"2024-04-24T23:16:13.163117+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:45:20.913759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.106686+0000","last_change":"2024-04-24T23:15:41.025813+0000","last_active":"2024-04-24T23:15:42.106686+0000","last_peered":"2024-04-24T23:15:42.106686+0000","last_clean":"2024-04-24T23:15:42.106686+0000","last_became_active":"2024-04-24T23:15:41.025137+0000","last_became_peered":"2024-04-24T23:15:41.025137+0000","last_unstale":"2024-04-24T23:15:42.106686+0000","last_undegraded":"2024-04-24T23:15:42.106686+0000","last_fullsized":"2024-04-24T23:15:42.106686+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:35:17.784854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,3,1],"acting":[2,6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.051163+0000","last_change":"2024-04-24T23:15:36.967476+0000","last_active":"2024-04-24T23:15:42.051163+0000","last_peered":"2024-04-24T23:15:42.051163+0000","last_clean":"2024-04-24T23:15:42.051163+0000","last_became_active":"2024-04-24T23:15:36.967098+0000","last_became_peered":"2024-04-24T23:15:36.967098+0000","last_unstale":"2024-04-24T23:15:42.051163+0000","last_undegraded":"2024-04-24T23:15:42.051163+0000","last_fullsized":"2024-04-24T23:15:42.051163+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:25:56.038903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"24'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.050091+0000","last_change":"2024-04-24T23:15:35.970630+0000","last_active":"2024-04-24T23:15:42.050091+0000","last_peered":"2024-04-24T23:15:42.050091+0000","last_clean":"2024-04-24T23:15:42.050091+0000","last_became_active":"2024-04-24T23:15:35.970430+0000","last_became_peered":"2024-04-24T23:15:35.970430+0000","last_unstale":"2024-04-24T23:15:42.050091+0000","last_undegraded":"2024-04-24T23:15:42.050091+0000","last_fullsized":"2024-04-24T23:15:42.050091+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:15:59.219084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.049399+0000","last_change":"2024-04-24T23:15:41.005022+0000","last_active":"2024-04-24T23:15:42.049399+0000","last_peered":"2024-04-24T23:15:42.049399+0000","last_clean":"2024-04-24T23:15:42.049399+0000","last_became_active":"2024-04-24T23:15:41.004648+0000","last_became_peered":"2024-04-24T23:15:41.004648+0000","last_unstale":"2024-04-24T23:15:42.049399+0000","last_undegraded":"2024-04-24T23:15:42.049399+0000","last_fullsized":"2024-04-24T23:15:42.049399+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:06:01.559760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6,3],"acting":[4,1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.051008+0000","last_change":"2024-04-24T23:15:36.967053+0000","last_active":"2024-04-24T23:15:42.051008+0000","last_peered":"2024-04-24T23:15:42.051008+0000","last_clean":"2024-04-24T23:15:42.051008+0000","last_became_active":"2024-04-24T23:15:36.966746+0000","last_became_peered":"2024-04-24T23:15:36.966746+0000","last_unstale":"2024-04-24T23:15:42.051008+0000","last_undegraded":"2024-04-24T23:15:42.051008+0000","last_fullsized":"2024-04-24T23:15:42.051008+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:16:08.779860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"24'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.192404+0000","last_change":"2024-04-24T23:15:35.965890+0000","last_active":"2024-04-24T23:15:43.192404+0000","last_peered":"2024-04-24T23:15:43.192404+0000","last_clean":"2024-04-24T23:15:43.192404+0000","last_became_active":"2024-04-24T23:15:35.965631+0000","last_became_peered":"2024-04-24T23:15:35.965631+0000","last_unstale":"2024-04-24T23:15:43.192404+0000","last_undegraded":"2024-04-24T23:15:43.192404+0000","last_fullsized":"2024-04-24T23:15:43.192404+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:17:35.650731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.050382+0000","last_change":"2024-04-24T23:15:41.011405+0000","last_active":"2024-04-24T23:15:42.050382+0000","last_peered":"2024-04-24T23:15:42.050382+0000","last_clean":"2024-04-24T23:15:42.050382+0000","last_became_active":"2024-04-24T23:15:41.010982+0000","last_became_peered":"2024-04-24T23:15:41.010982+0000","last_unstale":"2024-04-24T23:15:42.050382+0000","last_undegraded":"2024-04-24T23:15:42.050382+0000","last_fullsized":"2024-04-24T23:15:42.050382+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:08:00.914296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2,7],"acting":[6,0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.106945+0000","last_change":"2024-04-24T23:15:36.975244+0000","last_active":"2024-04-24T23:15:42.106945+0000","last_peered":"2024-04-24T23:15:42.106945+0000","last_clean":"2024-04-24T23:15:42.106945+0000","last_became_active":"2024-04-24T23:15:36.974872+0000","last_became_peered":"2024-04-24T23:15:36.974872+0000","last_unstale":"2024-04-24T23:15:42.106945+0000","last_undegraded":"2024-04-24T23:15:42.106945+0000","last_fullsized":"2024-04-24T23:15:42.106945+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:45:57.018205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.295386+0000","last_change":"2024-04-24T23:15:35.945792+0000","last_active":"2024-04-24T23:15:42.295386+0000","last_peered":"2024-04-24T23:15:42.295386+0000","last_clean":"2024-04-24T23:15:42.295386+0000","last_became_active":"2024-04-24T23:15:35.945094+0000","last_became_peered":"2024-04-24T23:15:35.945094+0000","last_unstale":"2024-04-24T23:15:42.295386+0000","last_undegraded":"2024-04-24T23:15:42.295386+0000","last_fullsized":"2024-04-24T23:15:42.295386+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:49:12.719127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.049802+0000","last_change":"2024-04-24T23:15:41.022105+0000","last_active":"2024-04-24T23:15:42.049802+0000","last_peered":"2024-04-24T23:15:42.049802+0000","last_clean":"2024-04-24T23:15:42.049802+0000","last_became_active":"2024-04-24T23:15:41.021645+0000","last_became_peered":"2024-04-24T23:15:41.021645+0000","last_unstale":"2024-04-24T23:15:42.049802+0000","last_undegraded":"2024-04-24T23:15:42.049802+0000","last_fullsized":"2024-04-24T23:15:42.049802+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:03:52.974835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5,7],"acting":[6,3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.192545+0000","last_change":"2024-04-24T23:15:36.991239+0000","last_active":"2024-04-24T23:15:43.192545+0000","last_peered":"2024-04-24T23:15:43.192545+0000","last_clean":"2024-04-24T23:15:43.192545+0000","last_became_active":"2024-04-24T23:15:36.990963+0000","last_became_peered":"2024-04-24T23:15:36.990963+0000","last_unstale":"2024-04-24T23:15:43.192545+0000","last_undegraded":"2024-04-24T23:15:43.192545+0000","last_fullsized":"2024-04-24T23:15:43.192545+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:53:28.101334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.191521+0000","last_change":"2024-04-24T23:15:35.957931+0000","last_active":"2024-04-24T23:15:43.191521+0000","last_peered":"2024-04-24T23:15:43.191521+0000","last_clean":"2024-04-24T23:15:43.191521+0000","last_became_active":"2024-04-24T23:15:35.957432+0000","last_became_peered":"2024-04-24T23:15:35.957432+0000","last_unstale":"2024-04-24T23:15:43.191521+0000","last_undegraded":"2024-04-24T23:15:43.191521+0000","last_fullsized":"2024-04-24T23:15:43.191521+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:11:38.773425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.050839+0000","last_change":"2024-04-24T23:15:41.007857+0000","last_active":"2024-04-24T23:15:42.050839+0000","last_peered":"2024-04-24T23:15:42.050839+0000","last_clean":"2024-04-24T23:15:42.050839+0000","last_became_active":"2024-04-24T23:15:41.007449+0000","last_became_peered":"2024-04-24T23:15:41.007449+0000","last_unstale":"2024-04-24T23:15:42.050839+0000","last_undegraded":"2024-04-24T23:15:42.050839+0000","last_fullsized":"2024-04-24T23:15:42.050839+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:05:06.145440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0,3],"acting":[6,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.295077+0000","last_change":"2024-04-24T23:15:36.970444+0000","last_active":"2024-04-24T23:15:42.295077+0000","last_peered":"2024-04-24T23:15:42.295077+0000","last_clean":"2024-04-24T23:15:42.295077+0000","last_became_active":"2024-04-24T23:15:36.970159+0000","last_became_peered":"2024-04-24T23:15:36.970159+0000","last_unstale":"2024-04-24T23:15:42.295077+0000","last_undegraded":"2024-04-24T23:15:42.295077+0000","last_fullsized":"2024-04-24T23:15:42.295077+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:37:02.956224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"24'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.187846+0000","last_change":"2024-04-24T23:15:35.945461+0000","last_active":"2024-04-24T23:15:43.187846+0000","last_peered":"2024-04-24T23:15:43.187846+0000","last_clean":"2024-04-24T23:15:43.187846+0000","last_became_active":"2024-04-24T23:15:35.945185+0000","last_became_peered":"2024-04-24T23:15:35.945185+0000","last_unstale":"2024-04-24T23:15:43.187846+0000","last_undegraded":"2024-04-24T23:15:43.187846+0000","last_fullsized":"2024-04-24T23:15:43.187846+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:37:34.175539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.050260+0000","last_change":"2024-04-24T23:15:41.014584+0000","last_active":"2024-04-24T23:15:42.050260+0000","last_peered":"2024-04-24T23:15:42.050260+0000","last_clean":"2024-04-24T23:15:42.050260+0000","last_became_active":"2024-04-24T23:15:41.014137+0000","last_became_peered":"2024-04-24T23:15:41.014137+0000","last_unstale":"2024-04-24T23:15:42.050260+0000","last_undegraded":"2024-04-24T23:15:42.050260+0000","last_fullsized":"2024-04-24T23:15:42.050260+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:23:50.387635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,7],"acting":[6,0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.049485+0000","last_change":"2024-04-24T23:15:36.981783+0000","last_active":"2024-04-24T23:15:42.049485+0000","last_peered":"2024-04-24T23:15:42.049485+0000","last_clean":"2024-04-24T23:15:42.049485+0000","last_became_active":"2024-04-24T23:15:36.981502+0000","last_became_peered":"2024-04-24T23:15:36.981502+0000","last_unstale":"2024-04-24T23:15:42.049485+0000","last_undegraded":"2024-04-24T23:15:42.049485+0000","last_fullsized":"2024-04-24T23:15:42.049485+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:56:08.483410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"24'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.190892+0000","last_change":"2024-04-24T23:15:35.960519+0000","last_active":"2024-04-24T23:15:43.190892+0000","last_peered":"2024-04-24T23:15:43.190892+0000","last_clean":"2024-04-24T23:15:43.190892+0000","last_became_active":"2024-04-24T23:15:35.960180+0000","last_became_peered":"2024-04-24T23:15:35.960180+0000","last_unstale":"2024-04-24T23:15:43.190892+0000","last_undegraded":"2024-04-24T23:15:43.190892+0000","last_fullsized":"2024-04-24T23:15:43.190892+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:50:00.199120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.295661+0000","last_change":"2024-04-24T23:15:41.012876+0000","last_active":"2024-04-24T23:15:42.295661+0000","last_peered":"2024-04-24T23:15:42.295661+0000","last_clean":"2024-04-24T23:15:42.295661+0000","last_became_active":"2024-04-24T23:15:41.012493+0000","last_became_peered":"2024-04-24T23:15:41.012493+0000","last_unstale":"2024-04-24T23:15:42.295661+0000","last_undegraded":"2024-04-24T23:15:42.295661+0000","last_fullsized":"2024-04-24T23:15:42.295661+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:21:28.287059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6,1],"acting":[5,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.050057+0000","last_change":"2024-04-24T23:15:36.981449+0000","last_active":"2024-04-24T23:15:42.050057+0000","last_peered":"2024-04-24T23:15:42.050057+0000","last_clean":"2024-04-24T23:15:42.050057+0000","last_became_active":"2024-04-24T23:15:36.980837+0000","last_became_peered":"2024-04-24T23:15:36.980837+0000","last_unstale":"2024-04-24T23:15:42.050057+0000","last_undegraded":"2024-04-24T23:15:42.050057+0000","last_fullsized":"2024-04-24T23:15:42.050057+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:15:13.513341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.3","version":"24'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:16:13.164506+0000","last_change":"2024-04-24T23:15:35.952947+0000","last_active":"2024-04-24T23:16:13.164506+0000","last_peered":"2024-04-24T23:16:13.164506+0000","last_clean":"2024-04-24T23:16:13.164506+0000","last_became_active":"2024-04-24T23:15:35.952655+0000","last_became_peered":"2024-04-24T23:15:35.952655+0000","last_unstale":"2024-04-24T23:16:13.164506+0000","last_undegraded":"2024-04-24T23:16:13.164506+0000","last_fullsized":"2024-04-24T23:16:13.164506+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:50:10.939396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":178,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.732636+0000","last_change":"2024-04-24T23:15:25.826673+0000","last_active":"2024-04-24T23:15:42.732636+0000","last_peered":"2024-04-24T23:15:42.732636+0000","last_clean":"2024-04-24T23:15:42.732636+0000","last_became_active":"2024-04-24T23:15:25.820030+0000","last_became_peered":"2024-04-24T23:15:25.820030+0000","last_unstale":"2024-04-24T23:15:42.732636+0000","last_undegraded":"2024-04-24T23:15:42.732636+0000","last_fullsized":"2024-04-24T23:15:42.732636+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:24.790928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:24.790928+0000","last_clean_scrub_stamp":"2024-04-24T23:15:24.790928+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:37:24.054332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.192066+0000","last_change":"2024-04-24T23:15:41.024712+0000","last_active":"2024-04-24T23:15:43.192066+0000","last_peered":"2024-04-24T23:15:43.192066+0000","last_clean":"2024-04-24T23:15:43.192066+0000","last_became_active":"2024-04-24T23:15:41.022697+0000","last_became_peered":"2024-04-24T23:15:41.022697+0000","last_unstale":"2024-04-24T23:15:43.192066+0000","last_undegraded":"2024-04-24T23:15:43.192066+0000","last_fullsized":"2024-04-24T23:15:43.192066+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:37:08.231105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2,0],"acting":[3,1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.191847+0000","last_change":"2024-04-24T23:15:37.000489+0000","last_active":"2024-04-24T23:15:43.191847+0000","last_peered":"2024-04-24T23:15:43.191847+0000","last_clean":"2024-04-24T23:15:43.191847+0000","last_became_active":"2024-04-24T23:15:37.000179+0000","last_became_peered":"2024-04-24T23:15:37.000179+0000","last_unstale":"2024-04-24T23:15:43.191847+0000","last_undegraded":"2024-04-24T23:15:43.191847+0000","last_fullsized":"2024-04-24T23:15:43.191847+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:06:37.125573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.051120+0000","last_change":"2024-04-24T23:15:35.966020+0000","last_active":"2024-04-24T23:15:42.051120+0000","last_peered":"2024-04-24T23:15:42.051120+0000","last_clean":"2024-04-24T23:15:42.051120+0000","last_became_active":"2024-04-24T23:15:35.965758+0000","last_became_peered":"2024-04-24T23:15:35.965758+0000","last_unstale":"2024-04-24T23:15:42.051120+0000","last_undegraded":"2024-04-24T23:15:42.051120+0000","last_fullsized":"2024-04-24T23:15:42.051120+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:39:21.233461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.106667+0000","last_change":"2024-04-24T23:15:41.024489+0000","last_active":"2024-04-24T23:15:42.106667+0000","last_peered":"2024-04-24T23:15:42.106667+0000","last_clean":"2024-04-24T23:15:42.106667+0000","last_became_active":"2024-04-24T23:15:41.024041+0000","last_became_peered":"2024-04-24T23:15:41.024041+0000","last_unstale":"2024-04-24T23:15:42.106667+0000","last_undegraded":"2024-04-24T23:15:42.106667+0000","last_fullsized":"2024-04-24T23:15:42.106667+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:27:18.197003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7,4],"acting":[2,3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.193516+0000","last_change":"2024-04-24T23:15:36.979887+0000","last_active":"2024-04-24T23:15:43.193516+0000","last_peered":"2024-04-24T23:15:43.193516+0000","last_clean":"2024-04-24T23:15:43.193516+0000","last_became_active":"2024-04-24T23:15:36.979611+0000","last_became_peered":"2024-04-24T23:15:36.979611+0000","last_unstale":"2024-04-24T23:15:43.193516+0000","last_undegraded":"2024-04-24T23:15:43.193516+0000","last_fullsized":"2024-04-24T23:15:43.193516+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:07:18.261077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"23'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.295532+0000","last_change":"2024-04-24T23:15:35.965192+0000","last_active":"2024-04-24T23:15:42.295532+0000","last_peered":"2024-04-24T23:15:42.295532+0000","last_clean":"2024-04-24T23:15:42.295532+0000","last_became_active":"2024-04-24T23:15:35.964873+0000","last_became_peered":"2024-04-24T23:15:35.964873+0000","last_unstale":"2024-04-24T23:15:42.295532+0000","last_undegraded":"2024-04-24T23:15:42.295532+0000","last_fullsized":"2024-04-24T23:15:42.295532+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:14:27.884944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.295549+0000","last_change":"2024-04-24T23:15:41.004002+0000","last_active":"2024-04-24T23:15:42.295549+0000","last_peered":"2024-04-24T23:15:42.295549+0000","last_clean":"2024-04-24T23:15:42.295549+0000","last_became_active":"2024-04-24T23:15:41.003335+0000","last_became_peered":"2024-04-24T23:15:41.003335+0000","last_unstale":"2024-04-24T23:15:42.295549+0000","last_undegraded":"2024-04-24T23:15:42.295549+0000","last_fullsized":"2024-04-24T23:15:42.295549+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:49:36.206903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1,3],"acting":[5,4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.193583+0000","last_change":"2024-04-24T23:15:37.002854+0000","last_active":"2024-04-24T23:15:43.193583+0000","last_peered":"2024-04-24T23:15:43.193583+0000","last_clean":"2024-04-24T23:15:43.193583+0000","last_became_active":"2024-04-24T23:15:37.002583+0000","last_became_peered":"2024-04-24T23:15:37.002583+0000","last_unstale":"2024-04-24T23:15:43.193583+0000","last_undegraded":"2024-04-24T23:15:43.193583+0000","last_fullsized":"2024-04-24T23:15:43.193583+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:27:04.251547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"23'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.050875+0000","last_change":"2024-04-24T23:15:35.948752+0000","last_active":"2024-04-24T23:15:42.050875+0000","last_peered":"2024-04-24T23:15:42.050875+0000","last_clean":"2024-04-24T23:15:42.050875+0000","last_became_active":"2024-04-24T23:15:35.948438+0000","last_became_peered":"2024-04-24T23:15:35.948438+0000","last_unstale":"2024-04-24T23:15:42.050875+0000","last_undegraded":"2024-04-24T23:15:42.050875+0000","last_fullsized":"2024-04-24T23:15:42.050875+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:45:56.006278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.190120+0000","last_change":"2024-04-24T23:15:41.017524+0000","last_active":"2024-04-24T23:15:43.190120+0000","last_peered":"2024-04-24T23:15:43.190120+0000","last_clean":"2024-04-24T23:15:43.190120+0000","last_became_active":"2024-04-24T23:15:41.017092+0000","last_became_peered":"2024-04-24T23:15:41.017092+0000","last_unstale":"2024-04-24T23:15:43.190120+0000","last_undegraded":"2024-04-24T23:15:43.190120+0000","last_fullsized":"2024-04-24T23:15:43.190120+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:04:53.235038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.107289+0000","last_change":"2024-04-24T23:15:36.990441+0000","last_active":"2024-04-24T23:15:42.107289+0000","last_peered":"2024-04-24T23:15:42.107289+0000","last_clean":"2024-04-24T23:15:42.107289+0000","last_became_active":"2024-04-24T23:15:36.989863+0000","last_became_peered":"2024-04-24T23:15:36.989863+0000","last_unstale":"2024-04-24T23:15:42.107289+0000","last_undegraded":"2024-04-24T23:15:42.107289+0000","last_fullsized":"2024-04-24T23:15:42.107289+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:50:07.975347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.f","version":"23'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.193960+0000","last_change":"2024-04-24T23:15:35.969317+0000","last_active":"2024-04-24T23:15:43.193960+0000","last_peered":"2024-04-24T23:15:43.193960+0000","last_clean":"2024-04-24T23:15:43.193960+0000","last_became_active":"2024-04-24T23:15:35.968994+0000","last_became_peered":"2024-04-24T23:15:35.968994+0000","last_unstale":"2024-04-24T23:15:43.193960+0000","last_undegraded":"2024-04-24T23:15:43.193960+0000","last_fullsized":"2024-04-24T23:15:43.193960+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:51:57.100772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.733165+0000","last_change":"2024-04-24T23:15:41.010103+0000","last_active":"2024-04-24T23:15:42.733165+0000","last_peered":"2024-04-24T23:15:42.733165+0000","last_clean":"2024-04-24T23:15:42.733165+0000","last_became_active":"2024-04-24T23:15:41.009328+0000","last_became_peered":"2024-04-24T23:15:41.009328+0000","last_unstale":"2024-04-24T23:15:42.733165+0000","last_undegraded":"2024-04-24T23:15:42.733165+0000","last_fullsized":"2024-04-24T23:15:42.733165+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:46:49.579501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,2],"acting":[1,5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.190520+0000","last_change":"2024-04-24T23:15:36.996765+0000","last_active":"2024-04-24T23:15:43.190520+0000","last_peered":"2024-04-24T23:15:43.190520+0000","last_clean":"2024-04-24T23:15:43.190520+0000","last_became_active":"2024-04-24T23:15:36.995801+0000","last_became_peered":"2024-04-24T23:15:36.995801+0000","last_unstale":"2024-04-24T23:15:43.190520+0000","last_undegraded":"2024-04-24T23:15:43.190520+0000","last_fullsized":"2024-04-24T23:15:43.190520+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:25:19.421667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"24'10","reported_seq":32,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:16:13.160400+0000","last_change":"2024-04-24T23:15:35.944023+0000","last_active":"2024-04-24T23:16:13.160400+0000","last_peered":"2024-04-24T23:16:13.160400+0000","last_clean":"2024-04-24T23:16:13.160400+0000","last_became_active":"2024-04-24T23:15:35.943710+0000","last_became_peered":"2024-04-24T23:15:35.943710+0000","last_unstale":"2024-04-24T23:16:13.160400+0000","last_undegraded":"2024-04-24T23:16:13.160400+0000","last_fullsized":"2024-04-24T23:16:13.160400+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:46:48.090958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.295767+0000","last_change":"2024-04-24T23:15:41.013027+0000","last_active":"2024-04-24T23:15:42.295767+0000","last_peered":"2024-04-24T23:15:42.295767+0000","last_clean":"2024-04-24T23:15:42.295767+0000","last_became_active":"2024-04-24T23:15:41.012587+0000","last_became_peered":"2024-04-24T23:15:41.012587+0000","last_unstale":"2024-04-24T23:15:42.295767+0000","last_undegraded":"2024-04-24T23:15:42.295767+0000","last_fullsized":"2024-04-24T23:15:42.295767+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:27:04.276246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1,6],"acting":[5,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.732647+0000","last_change":"2024-04-24T23:15:36.979702+0000","last_active":"2024-04-24T23:15:42.732647+0000","last_peered":"2024-04-24T23:15:42.732647+0000","last_clean":"2024-04-24T23:15:42.732647+0000","last_became_active":"2024-04-24T23:15:36.979356+0000","last_became_peered":"2024-04-24T23:15:36.979356+0000","last_unstale":"2024-04-24T23:15:42.732647+0000","last_undegraded":"2024-04-24T23:15:42.732647+0000","last_fullsized":"2024-04-24T23:15:42.732647+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:14:24.309192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.193698+0000","last_change":"2024-04-24T23:15:35.973026+0000","last_active":"2024-04-24T23:15:43.193698+0000","last_peered":"2024-04-24T23:15:43.193698+0000","last_clean":"2024-04-24T23:15:43.193698+0000","last_became_active":"2024-04-24T23:15:35.972655+0000","last_became_peered":"2024-04-24T23:15:35.972655+0000","last_unstale":"2024-04-24T23:15:43.193698+0000","last_undegraded":"2024-04-24T23:15:43.193698+0000","last_fullsized":"2024-04-24T23:15:43.193698+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:25:34.352908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.295901+0000","last_change":"2024-04-24T23:15:41.004221+0000","last_active":"2024-04-24T23:15:42.295901+0000","last_peered":"2024-04-24T23:15:42.295901+0000","last_clean":"2024-04-24T23:15:42.295901+0000","last_became_active":"2024-04-24T23:15:41.003790+0000","last_became_peered":"2024-04-24T23:15:41.003790+0000","last_unstale":"2024-04-24T23:15:42.295901+0000","last_undegraded":"2024-04-24T23:15:42.295901+0000","last_fullsized":"2024-04-24T23:15:42.295901+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:28:58.866024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6,4],"acting":[5,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.191414+0000","last_change":"2024-04-24T23:15:36.997389+0000","last_active":"2024-04-24T23:15:43.191414+0000","last_peered":"2024-04-24T23:15:43.191414+0000","last_clean":"2024-04-24T23:15:43.191414+0000","last_became_active":"2024-04-24T23:15:36.995923+0000","last_became_peered":"2024-04-24T23:15:36.995923+0000","last_unstale":"2024-04-24T23:15:43.191414+0000","last_undegraded":"2024-04-24T23:15:43.191414+0000","last_fullsized":"2024-04-24T23:15:43.191414+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:55:47.134320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"24'3","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:16:13.164411+0000","last_change":"2024-04-24T23:15:35.957715+0000","last_active":"2024-04-24T23:16:13.164411+0000","last_peered":"2024-04-24T23:16:13.164411+0000","last_clean":"2024-04-24T23:16:13.164411+0000","last_became_active":"2024-04-24T23:15:35.957088+0000","last_became_peered":"2024-04-24T23:15:35.957088+0000","last_unstale":"2024-04-24T23:16:13.164411+0000","last_undegraded":"2024-04-24T23:16:13.164411+0000","last_fullsized":"2024-04-24T23:16:13.164411+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:30:34.483669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.050833+0000","last_change":"2024-04-24T23:15:41.008165+0000","last_active":"2024-04-24T23:15:42.050833+0000","last_peered":"2024-04-24T23:15:42.050833+0000","last_clean":"2024-04-24T23:15:42.050833+0000","last_became_active":"2024-04-24T23:15:41.007555+0000","last_became_peered":"2024-04-24T23:15:41.007555+0000","last_unstale":"2024-04-24T23:15:42.050833+0000","last_undegraded":"2024-04-24T23:15:42.050833+0000","last_fullsized":"2024-04-24T23:15:42.050833+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:18:08.195747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0,7],"acting":[6,4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.050006+0000","last_change":"2024-04-24T23:15:36.991382+0000","last_active":"2024-04-24T23:15:42.050006+0000","last_peered":"2024-04-24T23:15:42.050006+0000","last_clean":"2024-04-24T23:15:42.050006+0000","last_became_active":"2024-04-24T23:15:36.991115+0000","last_became_peered":"2024-04-24T23:15:36.991115+0000","last_unstale":"2024-04-24T23:15:42.050006+0000","last_undegraded":"2024-04-24T23:15:42.050006+0000","last_fullsized":"2024-04-24T23:15:42.050006+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:21:51.863947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"24'4","reported_seq":26,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:16:13.164725+0000","last_change":"2024-04-24T23:15:35.964245+0000","last_active":"2024-04-24T23:16:13.164725+0000","last_peered":"2024-04-24T23:16:13.164725+0000","last_clean":"2024-04-24T23:16:13.164725+0000","last_became_active":"2024-04-24T23:15:35.963907+0000","last_became_peered":"2024-04-24T23:15:35.963907+0000","last_unstale":"2024-04-24T23:16:13.164725+0000","last_undegraded":"2024-04-24T23:16:13.164725+0000","last_fullsized":"2024-04-24T23:16:13.164725+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:15:00.913899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.192087+0000","last_change":"2024-04-24T23:15:41.021981+0000","last_active":"2024-04-24T23:15:43.192087+0000","last_peered":"2024-04-24T23:15:43.192087+0000","last_clean":"2024-04-24T23:15:43.192087+0000","last_became_active":"2024-04-24T23:15:41.019193+0000","last_became_peered":"2024-04-24T23:15:41.019193+0000","last_unstale":"2024-04-24T23:15:43.192087+0000","last_undegraded":"2024-04-24T23:15:43.192087+0000","last_fullsized":"2024-04-24T23:15:43.192087+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:04:39.828502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5,6],"acting":[3,0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.193203+0000","last_change":"2024-04-24T23:15:36.986833+0000","last_active":"2024-04-24T23:15:43.193203+0000","last_peered":"2024-04-24T23:15:43.193203+0000","last_clean":"2024-04-24T23:15:43.193203+0000","last_became_active":"2024-04-24T23:15:36.986484+0000","last_became_peered":"2024-04-24T23:15:36.986484+0000","last_unstale":"2024-04-24T23:15:43.193203+0000","last_undegraded":"2024-04-24T23:15:43.193203+0000","last_fullsized":"2024-04-24T23:15:43.193203+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:12:32.245267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.14","version":"24'12","reported_seq":34,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:16:13.164529+0000","last_change":"2024-04-24T23:15:35.957124+0000","last_active":"2024-04-24T23:16:13.164529+0000","last_peered":"2024-04-24T23:16:13.164529+0000","last_clean":"2024-04-24T23:16:13.164529+0000","last_became_active":"2024-04-24T23:15:35.951923+0000","last_became_peered":"2024-04-24T23:15:35.951923+0000","last_unstale":"2024-04-24T23:16:13.164529+0000","last_undegraded":"2024-04-24T23:16:13.164529+0000","last_fullsized":"2024-04-24T23:16:13.164529+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:44:16.820640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.049936+0000","last_change":"2024-04-24T23:15:41.015494+0000","last_active":"2024-04-24T23:15:42.049936+0000","last_peered":"2024-04-24T23:15:42.049936+0000","last_clean":"2024-04-24T23:15:42.049936+0000","last_became_active":"2024-04-24T23:15:41.015179+0000","last_became_peered":"2024-04-24T23:15:41.015179+0000","last_unstale":"2024-04-24T23:15:42.049936+0000","last_undegraded":"2024-04-24T23:15:42.049936+0000","last_fullsized":"2024-04-24T23:15:42.049936+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:18:32.208506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5,0],"acting":[4,3,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.190450+0000","last_change":"2024-04-24T23:15:36.999334+0000","last_active":"2024-04-24T23:15:43.190450+0000","last_peered":"2024-04-24T23:15:43.190450+0000","last_clean":"2024-04-24T23:15:43.190450+0000","last_became_active":"2024-04-24T23:15:36.995470+0000","last_became_peered":"2024-04-24T23:15:36.995470+0000","last_unstale":"2024-04-24T23:15:43.190450+0000","last_undegraded":"2024-04-24T23:15:43.190450+0000","last_fullsized":"2024-04-24T23:15:43.190450+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:19:57.973402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.15","version":"24'4","reported_seq":26,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.188040+0000","last_change":"2024-04-24T23:15:35.946041+0000","last_active":"2024-04-24T23:15:43.188040+0000","last_peered":"2024-04-24T23:15:43.188040+0000","last_clean":"2024-04-24T23:15:43.188040+0000","last_became_active":"2024-04-24T23:15:35.944805+0000","last_became_peered":"2024-04-24T23:15:35.944805+0000","last_unstale":"2024-04-24T23:15:43.188040+0000","last_undegraded":"2024-04-24T23:15:43.188040+0000","last_fullsized":"2024-04-24T23:15:43.188040+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:52:19.261306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.107105+0000","last_change":"2024-04-24T23:15:41.024392+0000","last_active":"2024-04-24T23:15:42.107105+0000","last_peered":"2024-04-24T23:15:42.107105+0000","last_clean":"2024-04-24T23:15:42.107105+0000","last_became_active":"2024-04-24T23:15:41.023951+0000","last_became_peered":"2024-04-24T23:15:41.023951+0000","last_unstale":"2024-04-24T23:15:42.107105+0000","last_undegraded":"2024-04-24T23:15:42.107105+0000","last_fullsized":"2024-04-24T23:15:42.107105+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:24:17.810117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1,7],"acting":[2,3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.295845+0000","last_change":"2024-04-24T23:15:36.992243+0000","last_active":"2024-04-24T23:15:42.295845+0000","last_peered":"2024-04-24T23:15:42.295845+0000","last_clean":"2024-04-24T23:15:42.295845+0000","last_became_active":"2024-04-24T23:15:36.992002+0000","last_became_peered":"2024-04-24T23:15:36.992002+0000","last_unstale":"2024-04-24T23:15:42.295845+0000","last_undegraded":"2024-04-24T23:15:42.295845+0000","last_fullsized":"2024-04-24T23:15:42.295845+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:52:15.911009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"24'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:16:13.163477+0000","last_change":"2024-04-24T23:15:35.967408+0000","last_active":"2024-04-24T23:16:13.163477+0000","last_peered":"2024-04-24T23:16:13.163477+0000","last_clean":"2024-04-24T23:16:13.163477+0000","last_became_active":"2024-04-24T23:15:35.966646+0000","last_became_peered":"2024-04-24T23:15:35.966646+0000","last_unstale":"2024-04-24T23:16:13.163477+0000","last_undegraded":"2024-04-24T23:16:13.163477+0000","last_fullsized":"2024-04-24T23:16:13.163477+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:07:07.931773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.187622+0000","last_change":"2024-04-24T23:15:41.021697+0000","last_active":"2024-04-24T23:15:43.187622+0000","last_peered":"2024-04-24T23:15:43.187622+0000","last_clean":"2024-04-24T23:15:43.187622+0000","last_became_active":"2024-04-24T23:15:41.021281+0000","last_became_peered":"2024-04-24T23:15:41.021281+0000","last_unstale":"2024-04-24T23:15:43.187622+0000","last_undegraded":"2024-04-24T23:15:43.187622+0000","last_fullsized":"2024-04-24T23:15:43.187622+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:59:37.306593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3,2],"acting":[7,1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.107218+0000","last_change":"2024-04-24T23:15:36.990988+0000","last_active":"2024-04-24T23:15:42.107218+0000","last_peered":"2024-04-24T23:15:42.107218+0000","last_clean":"2024-04-24T23:15:42.107218+0000","last_became_active":"2024-04-24T23:15:36.990711+0000","last_became_peered":"2024-04-24T23:15:36.990711+0000","last_unstale":"2024-04-24T23:15:42.107218+0000","last_undegraded":"2024-04-24T23:15:42.107218+0000","last_fullsized":"2024-04-24T23:15:42.107218+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:33:20.663776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.17","version":"24'3","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:16:13.163955+0000","last_change":"2024-04-24T23:15:35.957838+0000","last_active":"2024-04-24T23:16:13.163955+0000","last_peered":"2024-04-24T23:16:13.163955+0000","last_clean":"2024-04-24T23:16:13.163955+0000","last_became_active":"2024-04-24T23:15:35.956870+0000","last_became_peered":"2024-04-24T23:15:35.956870+0000","last_unstale":"2024-04-24T23:16:13.163955+0000","last_undegraded":"2024-04-24T23:16:13.163955+0000","last_fullsized":"2024-04-24T23:16:13.163955+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:56:03.889693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.049546+0000","last_change":"2024-04-24T23:15:41.006098+0000","last_active":"2024-04-24T23:15:42.049546+0000","last_peered":"2024-04-24T23:15:42.049546+0000","last_clean":"2024-04-24T23:15:42.049546+0000","last_became_active":"2024-04-24T23:15:41.005578+0000","last_became_peered":"2024-04-24T23:15:41.005578+0000","last_unstale":"2024-04-24T23:15:42.049546+0000","last_undegraded":"2024-04-24T23:15:42.049546+0000","last_fullsized":"2024-04-24T23:15:42.049546+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:21:12.802112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3,7],"acting":[4,0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.049896+0000","last_change":"2024-04-24T23:15:41.022122+0000","last_active":"2024-04-24T23:15:42.049896+0000","last_peered":"2024-04-24T23:15:42.049896+0000","last_clean":"2024-04-24T23:15:42.049896+0000","last_became_active":"2024-04-24T23:15:41.021698+0000","last_became_peered":"2024-04-24T23:15:41.021698+0000","last_unstale":"2024-04-24T23:15:42.049896+0000","last_undegraded":"2024-04-24T23:15:42.049896+0000","last_fullsized":"2024-04-24T23:15:42.049896+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:13:56.730050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5,3],"acting":[4,1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.049600+0000","last_change":"2024-04-24T23:15:36.979816+0000","last_active":"2024-04-24T23:15:42.049600+0000","last_peered":"2024-04-24T23:15:42.049600+0000","last_clean":"2024-04-24T23:15:42.049600+0000","last_became_active":"2024-04-24T23:15:36.979553+0000","last_became_peered":"2024-04-24T23:15:36.979553+0000","last_unstale":"2024-04-24T23:15:42.049600+0000","last_undegraded":"2024-04-24T23:15:42.049600+0000","last_fullsized":"2024-04-24T23:15:42.049600+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:55:47.134320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.18","version":"24'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.190707+0000","last_change":"2024-04-24T23:15:35.964744+0000","last_active":"2024-04-24T23:15:43.190707+0000","last_peered":"2024-04-24T23:15:43.190707+0000","last_clean":"2024-04-24T23:15:43.190707+0000","last_became_active":"2024-04-24T23:15:35.964434+0000","last_became_peered":"2024-04-24T23:15:35.964434+0000","last_unstale":"2024-04-24T23:15:43.190707+0000","last_undegraded":"2024-04-24T23:15:43.190707+0000","last_fullsized":"2024-04-24T23:15:43.190707+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:26:32.587197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.191463+0000","last_change":"2024-04-24T23:15:41.022032+0000","last_active":"2024-04-24T23:15:43.191463+0000","last_peered":"2024-04-24T23:15:43.191463+0000","last_clean":"2024-04-24T23:15:43.191463+0000","last_became_active":"2024-04-24T23:15:41.021616+0000","last_became_peered":"2024-04-24T23:15:41.021616+0000","last_unstale":"2024-04-24T23:15:43.191463+0000","last_undegraded":"2024-04-24T23:15:43.191463+0000","last_fullsized":"2024-04-24T23:15:43.191463+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:08:59.242392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5,1],"acting":[3,2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.295164+0000","last_change":"2024-04-24T23:15:36.967198+0000","last_active":"2024-04-24T23:15:42.295164+0000","last_peered":"2024-04-24T23:15:42.295164+0000","last_clean":"2024-04-24T23:15:42.295164+0000","last_became_active":"2024-04-24T23:15:36.966910+0000","last_became_peered":"2024-04-24T23:15:36.966910+0000","last_unstale":"2024-04-24T23:15:42.295164+0000","last_undegraded":"2024-04-24T23:15:42.295164+0000","last_fullsized":"2024-04-24T23:15:42.295164+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:51:19.788437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.049504+0000","last_change":"2024-04-24T23:15:35.967169+0000","last_active":"2024-04-24T23:15:42.049504+0000","last_peered":"2024-04-24T23:15:42.049504+0000","last_clean":"2024-04-24T23:15:42.049504+0000","last_became_active":"2024-04-24T23:15:35.966939+0000","last_became_peered":"2024-04-24T23:15:35.966939+0000","last_unstale":"2024-04-24T23:15:42.049504+0000","last_undegraded":"2024-04-24T23:15:42.049504+0000","last_fullsized":"2024-04-24T23:15:42.049504+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:38:55.460569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"23'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.191308+0000","last_change":"2024-04-24T23:15:35.971236+0000","last_active":"2024-04-24T23:15:43.191308+0000","last_peered":"2024-04-24T23:15:43.191308+0000","last_clean":"2024-04-24T23:15:43.191308+0000","last_became_active":"2024-04-24T23:15:35.970908+0000","last_became_peered":"2024-04-24T23:15:35.970908+0000","last_unstale":"2024-04-24T23:15:43.191308+0000","last_undegraded":"2024-04-24T23:15:43.191308+0000","last_fullsized":"2024-04-24T23:15:43.191308+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:14:15.548024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.050320+0000","last_change":"2024-04-24T23:15:36.988156+0000","last_active":"2024-04-24T23:15:42.050320+0000","last_peered":"2024-04-24T23:15:42.050320+0000","last_clean":"2024-04-24T23:15:42.050320+0000","last_became_active":"2024-04-24T23:15:36.987896+0000","last_became_peered":"2024-04-24T23:15:36.987896+0000","last_unstale":"2024-04-24T23:15:42.050320+0000","last_undegraded":"2024-04-24T23:15:42.050320+0000","last_fullsized":"2024-04-24T23:15:42.050320+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:22:06.869644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.050441+0000","last_change":"2024-04-24T23:15:40.997559+0000","last_active":"2024-04-24T23:15:42.050441+0000","last_peered":"2024-04-24T23:15:42.050441+0000","last_clean":"2024-04-24T23:15:42.050441+0000","last_became_active":"2024-04-24T23:15:40.997102+0000","last_became_peered":"2024-04-24T23:15:40.997102+0000","last_unstale":"2024-04-24T23:15:42.050441+0000","last_undegraded":"2024-04-24T23:15:42.050441+0000","last_fullsized":"2024-04-24T23:15:42.050441+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:25:02.046761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,7],"acting":[4,6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"24'5","reported_seq":27,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:16:13.163435+0000","last_change":"2024-04-24T23:15:35.944842+0000","last_active":"2024-04-24T23:16:13.163435+0000","last_peered":"2024-04-24T23:16:13.163435+0000","last_clean":"2024-04-24T23:16:13.163435+0000","last_became_active":"2024-04-24T23:15:35.944563+0000","last_became_peered":"2024-04-24T23:15:35.944563+0000","last_unstale":"2024-04-24T23:16:13.163435+0000","last_undegraded":"2024-04-24T23:16:13.163435+0000","last_fullsized":"2024-04-24T23:16:13.163435+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:40:58.933384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.732992+0000","last_change":"2024-04-24T23:15:36.981103+0000","last_active":"2024-04-24T23:15:42.732992+0000","last_peered":"2024-04-24T23:15:42.732992+0000","last_clean":"2024-04-24T23:15:42.732992+0000","last_became_active":"2024-04-24T23:15:36.979637+0000","last_became_peered":"2024-04-24T23:15:36.979637+0000","last_unstale":"2024-04-24T23:15:42.732992+0000","last_undegraded":"2024-04-24T23:15:42.732992+0000","last_fullsized":"2024-04-24T23:15:42.732992+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:09:39.750272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.188153+0000","last_change":"2024-04-24T23:15:41.017087+0000","last_active":"2024-04-24T23:15:43.188153+0000","last_peered":"2024-04-24T23:15:43.188153+0000","last_clean":"2024-04-24T23:15:43.188153+0000","last_became_active":"2024-04-24T23:15:41.016696+0000","last_became_peered":"2024-04-24T23:15:41.016696+0000","last_unstale":"2024-04-24T23:15:43.188153+0000","last_undegraded":"2024-04-24T23:15:43.188153+0000","last_fullsized":"2024-04-24T23:15:43.188153+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:05:57.823447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1,0],"acting":[7,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.106609+0000","last_change":"2024-04-24T23:15:36.970211+0000","last_active":"2024-04-24T23:15:42.106609+0000","last_peered":"2024-04-24T23:15:42.106609+0000","last_clean":"2024-04-24T23:15:42.106609+0000","last_became_active":"2024-04-24T23:15:36.969090+0000","last_became_peered":"2024-04-24T23:15:36.969090+0000","last_unstale":"2024-04-24T23:15:42.106609+0000","last_undegraded":"2024-04-24T23:15:42.106609+0000","last_fullsized":"2024-04-24T23:15:42.106609+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:16:00.231012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.187632+0000","last_change":"2024-04-24T23:15:36.982528+0000","last_active":"2024-04-24T23:15:43.187632+0000","last_peered":"2024-04-24T23:15:43.187632+0000","last_clean":"2024-04-24T23:15:43.187632+0000","last_became_active":"2024-04-24T23:15:36.982299+0000","last_became_peered":"2024-04-24T23:15:36.982299+0000","last_unstale":"2024-04-24T23:15:43.187632+0000","last_undegraded":"2024-04-24T23:15:43.187632+0000","last_fullsized":"2024-04-24T23:15:43.187632+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:02:17.732324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.191005+0000","last_change":"2024-04-24T23:15:36.989899+0000","last_active":"2024-04-24T23:15:43.191005+0000","last_peered":"2024-04-24T23:15:43.191005+0000","last_clean":"2024-04-24T23:15:43.191005+0000","last_became_active":"2024-04-24T23:15:36.989557+0000","last_became_peered":"2024-04-24T23:15:36.989557+0000","last_unstale":"2024-04-24T23:15:43.191005+0000","last_undegraded":"2024-04-24T23:15:43.191005+0000","last_fullsized":"2024-04-24T23:15:43.191005+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:15:13.513341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.295325+0000","last_change":"2024-04-24T23:15:36.969628+0000","last_active":"2024-04-24T23:15:42.295325+0000","last_peered":"2024-04-24T23:15:42.295325+0000","last_clean":"2024-04-24T23:15:42.295325+0000","last_became_active":"2024-04-24T23:15:36.969327+0000","last_became_peered":"2024-04-24T23:15:36.969327+0000","last_unstale":"2024-04-24T23:15:42.295325+0000","last_undegraded":"2024-04-24T23:15:42.295325+0000","last_fullsized":"2024-04-24T23:15:42.295325+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:08:47.021999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.050888+0000","last_change":"2024-04-24T23:15:36.986343+0000","last_active":"2024-04-24T23:15:42.050888+0000","last_peered":"2024-04-24T23:15:42.050888+0000","last_clean":"2024-04-24T23:15:42.050888+0000","last_became_active":"2024-04-24T23:15:36.986026+0000","last_became_peered":"2024-04-24T23:15:36.986026+0000","last_unstale":"2024-04-24T23:15:42.050888+0000","last_undegraded":"2024-04-24T23:15:42.050888+0000","last_fullsized":"2024-04-24T23:15:42.050888+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:10:00.347826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.049809+0000","last_change":"2024-04-24T23:15:36.968538+0000","last_active":"2024-04-24T23:15:42.049809+0000","last_peered":"2024-04-24T23:15:42.049809+0000","last_clean":"2024-04-24T23:15:42.049809+0000","last_became_active":"2024-04-24T23:15:36.968263+0000","last_became_peered":"2024-04-24T23:15:36.968263+0000","last_unstale":"2024-04-24T23:15:42.049809+0000","last_undegraded":"2024-04-24T23:15:42.049809+0000","last_fullsized":"2024-04-24T23:15:42.049809+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:07:43.731738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.107186+0000","last_change":"2024-04-24T23:15:36.991008+0000","last_active":"2024-04-24T23:15:42.107186+0000","last_peered":"2024-04-24T23:15:42.107186+0000","last_clean":"2024-04-24T23:15:42.107186+0000","last_became_active":"2024-04-24T23:15:36.990764+0000","last_became_peered":"2024-04-24T23:15:36.990764+0000","last_unstale":"2024-04-24T23:15:42.107186+0000","last_undegraded":"2024-04-24T23:15:42.107186+0000","last_fullsized":"2024-04-24T23:15:42.107186+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:50:21.865233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.295685+0000","last_change":"2024-04-24T23:15:36.993550+0000","last_active":"2024-04-24T23:15:42.295685+0000","last_peered":"2024-04-24T23:15:42.295685+0000","last_clean":"2024-04-24T23:15:42.295685+0000","last_became_active":"2024-04-24T23:15:36.993310+0000","last_became_peered":"2024-04-24T23:15:36.993310+0000","last_unstale":"2024-04-24T23:15:42.295685+0000","last_undegraded":"2024-04-24T23:15:42.295685+0000","last_fullsized":"2024-04-24T23:15:42.295685+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:21:20.617940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.190455+0000","last_change":"2024-04-24T23:15:36.987810+0000","last_active":"2024-04-24T23:15:43.190455+0000","last_peered":"2024-04-24T23:15:43.190455+0000","last_clean":"2024-04-24T23:15:43.190455+0000","last_became_active":"2024-04-24T23:15:36.987540+0000","last_became_peered":"2024-04-24T23:15:36.987540+0000","last_unstale":"2024-04-24T23:15:43.190455+0000","last_undegraded":"2024-04-24T23:15:43.190455+0000","last_fullsized":"2024-04-24T23:15:43.190455+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:49:56.176021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.295070+0000","last_change":"2024-04-24T23:15:36.981187+0000","last_active":"2024-04-24T23:15:42.295070+0000","last_peered":"2024-04-24T23:15:42.295070+0000","last_clean":"2024-04-24T23:15:42.295070+0000","last_became_active":"2024-04-24T23:15:36.980838+0000","last_became_peered":"2024-04-24T23:15:36.980838+0000","last_unstale":"2024-04-24T23:15:42.295070+0000","last_undegraded":"2024-04-24T23:15:42.295070+0000","last_fullsized":"2024-04-24T23:15:42.295070+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:57:41.178367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.732913+0000","last_change":"2024-04-24T23:15:36.999238+0000","last_active":"2024-04-24T23:15:42.732913+0000","last_peered":"2024-04-24T23:15:42.732913+0000","last_clean":"2024-04-24T23:15:42.732913+0000","last_became_active":"2024-04-24T23:15:36.996245+0000","last_became_peered":"2024-04-24T23:15:36.996245+0000","last_unstale":"2024-04-24T23:15:42.732913+0000","last_undegraded":"2024-04-24T23:15:42.732913+0000","last_fullsized":"2024-04-24T23:15:42.732913+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:21:20.443640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.050214+0000","last_change":"2024-04-24T23:15:36.981219+0000","last_active":"2024-04-24T23:15:42.050214+0000","last_peered":"2024-04-24T23:15:42.050214+0000","last_clean":"2024-04-24T23:15:42.050214+0000","last_became_active":"2024-04-24T23:15:36.980911+0000","last_became_peered":"2024-04-24T23:15:36.980911+0000","last_unstale":"2024-04-24T23:15:42.050214+0000","last_undegraded":"2024-04-24T23:15:42.050214+0000","last_fullsized":"2024-04-24T23:15:42.050214+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:19:57.973402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.193222+0000","last_change":"2024-04-24T23:15:36.991418+0000","last_active":"2024-04-24T23:15:43.193222+0000","last_peered":"2024-04-24T23:15:43.193222+0000","last_clean":"2024-04-24T23:15:43.193222+0000","last_became_active":"2024-04-24T23:15:36.989792+0000","last_became_peered":"2024-04-24T23:15:36.989792+0000","last_unstale":"2024-04-24T23:15:43.193222+0000","last_undegraded":"2024-04-24T23:15:43.193222+0000","last_fullsized":"2024-04-24T23:15:43.193222+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:02:53.601311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.192125+0000","last_change":"2024-04-24T23:15:36.987810+0000","last_active":"2024-04-24T23:15:43.192125+0000","last_peered":"2024-04-24T23:15:43.192125+0000","last_clean":"2024-04-24T23:15:43.192125+0000","last_became_active":"2024-04-24T23:15:36.987494+0000","last_became_peered":"2024-04-24T23:15:36.987494+0000","last_unstale":"2024-04-24T23:15:43.192125+0000","last_undegraded":"2024-04-24T23:15:43.192125+0000","last_fullsized":"2024-04-24T23:15:43.192125+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:43:21.141172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.188217+0000","last_change":"2024-04-24T23:15:36.986576+0000","last_active":"2024-04-24T23:15:43.188217+0000","last_peered":"2024-04-24T23:15:43.188217+0000","last_clean":"2024-04-24T23:15:43.188217+0000","last_became_active":"2024-04-24T23:15:36.986272+0000","last_became_peered":"2024-04-24T23:15:36.986272+0000","last_unstale":"2024-04-24T23:15:43.188217+0000","last_undegraded":"2024-04-24T23:15:43.188217+0000","last_fullsized":"2024-04-24T23:15:43.188217+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:55:08.894945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.192001+0000","last_change":"2024-04-24T23:15:36.998812+0000","last_active":"2024-04-24T23:15:43.192001+0000","last_peered":"2024-04-24T23:15:43.192001+0000","last_clean":"2024-04-24T23:15:43.192001+0000","last_became_active":"2024-04-24T23:15:36.998456+0000","last_became_peered":"2024-04-24T23:15:36.998456+0000","last_unstale":"2024-04-24T23:15:43.192001+0000","last_undegraded":"2024-04-24T23:15:43.192001+0000","last_fullsized":"2024-04-24T23:15:43.192001+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:33:11.424218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.190262+0000","last_change":"2024-04-24T23:15:36.996521+0000","last_active":"2024-04-24T23:15:43.190262+0000","last_peered":"2024-04-24T23:15:43.190262+0000","last_clean":"2024-04-24T23:15:43.190262+0000","last_became_active":"2024-04-24T23:15:36.995507+0000","last_became_peered":"2024-04-24T23:15:36.995507+0000","last_unstale":"2024-04-24T23:15:43.190262+0000","last_undegraded":"2024-04-24T23:15:43.190262+0000","last_fullsized":"2024-04-24T23:15:43.190262+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:58:50.153471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.294996+0000","last_change":"2024-04-24T23:15:36.984994+0000","last_active":"2024-04-24T23:15:42.294996+0000","last_peered":"2024-04-24T23:15:42.294996+0000","last_clean":"2024-04-24T23:15:42.294996+0000","last_became_active":"2024-04-24T23:15:36.984741+0000","last_became_peered":"2024-04-24T23:15:36.984741+0000","last_unstale":"2024-04-24T23:15:42.294996+0000","last_undegraded":"2024-04-24T23:15:42.294996+0000","last_fullsized":"2024-04-24T23:15:42.294996+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:45:23.429836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.049917+0000","last_change":"2024-04-24T23:15:36.966165+0000","last_active":"2024-04-24T23:15:42.049917+0000","last_peered":"2024-04-24T23:15:42.049917+0000","last_clean":"2024-04-24T23:15:42.049917+0000","last_became_active":"2024-04-24T23:15:36.965877+0000","last_became_peered":"2024-04-24T23:15:36.965877+0000","last_unstale":"2024-04-24T23:15:42.049917+0000","last_undegraded":"2024-04-24T23:15:42.049917+0000","last_fullsized":"2024-04-24T23:15:42.049917+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:30:12.390978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.050537+0000","last_change":"2024-04-24T23:15:36.990667+0000","last_active":"2024-04-24T23:15:42.050537+0000","last_peered":"2024-04-24T23:15:42.050537+0000","last_clean":"2024-04-24T23:15:42.050537+0000","last_became_active":"2024-04-24T23:15:36.990383+0000","last_became_peered":"2024-04-24T23:15:36.990383+0000","last_unstale":"2024-04-24T23:15:42.050537+0000","last_undegraded":"2024-04-24T23:15:42.050537+0000","last_fullsized":"2024-04-24T23:15:42.050537+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:36:43.850046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.193024+0000","last_change":"2024-04-24T23:15:36.991686+0000","last_active":"2024-04-24T23:15:43.193024+0000","last_peered":"2024-04-24T23:15:43.193024+0000","last_clean":"2024-04-24T23:15:43.193024+0000","last_became_active":"2024-04-24T23:15:36.990155+0000","last_became_peered":"2024-04-24T23:15:36.990155+0000","last_unstale":"2024-04-24T23:15:43.193024+0000","last_undegraded":"2024-04-24T23:15:43.193024+0000","last_fullsized":"2024-04-24T23:15:43.193024+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:00:52.424019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.192909+0000","last_change":"2024-04-24T23:15:36.992958+0000","last_active":"2024-04-24T23:15:43.192909+0000","last_peered":"2024-04-24T23:15:43.192909+0000","last_clean":"2024-04-24T23:15:43.192909+0000","last_became_active":"2024-04-24T23:15:36.992608+0000","last_became_peered":"2024-04-24T23:15:36.992608+0000","last_unstale":"2024-04-24T23:15:43.192909+0000","last_undegraded":"2024-04-24T23:15:43.192909+0000","last_fullsized":"2024-04-24T23:15:43.192909+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:15:30.416763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.192849+0000","last_change":"2024-04-24T23:15:36.992943+0000","last_active":"2024-04-24T23:15:43.192849+0000","last_peered":"2024-04-24T23:15:43.192849+0000","last_clean":"2024-04-24T23:15:43.192849+0000","last_became_active":"2024-04-24T23:15:36.992567+0000","last_became_peered":"2024-04-24T23:15:36.992567+0000","last_unstale":"2024-04-24T23:15:43.192849+0000","last_undegraded":"2024-04-24T23:15:43.192849+0000","last_fullsized":"2024-04-24T23:15:43.192849+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:50:43.209232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.734006+0000","last_change":"2024-04-24T23:15:36.999697+0000","last_active":"2024-04-24T23:15:42.734006+0000","last_peered":"2024-04-24T23:15:42.734006+0000","last_clean":"2024-04-24T23:15:42.734006+0000","last_became_active":"2024-04-24T23:15:36.999413+0000","last_became_peered":"2024-04-24T23:15:36.999413+0000","last_unstale":"2024-04-24T23:15:42.734006+0000","last_undegraded":"2024-04-24T23:15:42.734006+0000","last_fullsized":"2024-04-24T23:15:42.734006+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:25:42.856471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.295006+0000","last_change":"2024-04-24T23:15:36.986742+0000","last_active":"2024-04-24T23:15:42.295006+0000","last_peered":"2024-04-24T23:15:42.295006+0000","last_clean":"2024-04-24T23:15:42.295006+0000","last_became_active":"2024-04-24T23:15:36.986013+0000","last_became_peered":"2024-04-24T23:15:36.986013+0000","last_unstale":"2024-04-24T23:15:42.295006+0000","last_undegraded":"2024-04-24T23:15:42.295006+0000","last_fullsized":"2024-04-24T23:15:42.295006+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:16:15.203151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.193480+0000","last_change":"2024-04-24T23:15:36.984336+0000","last_active":"2024-04-24T23:15:43.193480+0000","last_peered":"2024-04-24T23:15:43.193480+0000","last_clean":"2024-04-24T23:15:43.193480+0000","last_became_active":"2024-04-24T23:15:36.983966+0000","last_became_peered":"2024-04-24T23:15:36.983966+0000","last_unstale":"2024-04-24T23:15:43.193480+0000","last_undegraded":"2024-04-24T23:15:43.193480+0000","last_fullsized":"2024-04-24T23:15:43.193480+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:54:16.319608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.106762+0000","last_change":"2024-04-24T23:15:36.979079+0000","last_active":"2024-04-24T23:15:42.106762+0000","last_peered":"2024-04-24T23:15:42.106762+0000","last_clean":"2024-04-24T23:15:42.106762+0000","last_became_active":"2024-04-24T23:15:36.978706+0000","last_became_peered":"2024-04-24T23:15:36.978706+0000","last_unstale":"2024-04-24T23:15:42.106762+0000","last_undegraded":"2024-04-24T23:15:42.106762+0000","last_fullsized":"2024-04-24T23:15:42.106762+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:53:39.888150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.193929+0000","last_change":"2024-04-24T23:15:36.986008+0000","last_active":"2024-04-24T23:15:43.193929+0000","last_peered":"2024-04-24T23:15:43.193929+0000","last_clean":"2024-04-24T23:15:43.193929+0000","last_became_active":"2024-04-24T23:15:36.985552+0000","last_became_peered":"2024-04-24T23:15:36.985552+0000","last_unstale":"2024-04-24T23:15:43.193929+0000","last_undegraded":"2024-04-24T23:15:43.193929+0000","last_fullsized":"2024-04-24T23:15:43.193929+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:36:50.827503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.106908+0000","last_change":"2024-04-24T23:15:36.978673+0000","last_active":"2024-04-24T23:15:42.106908+0000","last_peered":"2024-04-24T23:15:42.106908+0000","last_clean":"2024-04-24T23:15:42.106908+0000","last_became_active":"2024-04-24T23:15:36.972744+0000","last_became_peered":"2024-04-24T23:15:36.972744+0000","last_unstale":"2024-04-24T23:15:42.106908+0000","last_undegraded":"2024-04-24T23:15:42.106908+0000","last_fullsized":"2024-04-24T23:15:42.106908+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:28:57.028825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.295395+0000","last_change":"2024-04-24T23:15:36.970238+0000","last_active":"2024-04-24T23:15:42.295395+0000","last_peered":"2024-04-24T23:15:42.295395+0000","last_clean":"2024-04-24T23:15:42.295395+0000","last_became_active":"2024-04-24T23:15:36.969751+0000","last_became_peered":"2024-04-24T23:15:36.969751+0000","last_unstale":"2024-04-24T23:15:42.295395+0000","last_undegraded":"2024-04-24T23:15:42.295395+0000","last_fullsized":"2024-04-24T23:15:42.295395+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:19:56.456801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.295592+0000","last_change":"2024-04-24T23:15:36.968825+0000","last_active":"2024-04-24T23:15:42.295592+0000","last_peered":"2024-04-24T23:15:42.295592+0000","last_clean":"2024-04-24T23:15:42.295592+0000","last_became_active":"2024-04-24T23:15:36.968543+0000","last_became_peered":"2024-04-24T23:15:36.968543+0000","last_unstale":"2024-04-24T23:15:42.295592+0000","last_undegraded":"2024-04-24T23:15:42.295592+0000","last_fullsized":"2024-04-24T23:15:42.295592+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:36:48.795062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.193226+0000","last_change":"2024-04-24T23:15:36.992800+0000","last_active":"2024-04-24T23:15:43.193226+0000","last_peered":"2024-04-24T23:15:43.193226+0000","last_clean":"2024-04-24T23:15:43.193226+0000","last_became_active":"2024-04-24T23:15:36.992476+0000","last_became_peered":"2024-04-24T23:15:36.992476+0000","last_unstale":"2024-04-24T23:15:43.193226+0000","last_undegraded":"2024-04-24T23:15:43.193226+0000","last_fullsized":"2024-04-24T23:15:43.193226+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:01:21.103559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":6046,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":88,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":26,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":12092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77,"ondisk_log_size":77,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574862,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":266240,"data_stored":100970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574862,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27272,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":262144,"data_stored":100886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574862,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27860,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343980,"statfs":{"total":96636764160,"available":96608235520,"internally_reserved":0,"allocated":864256,"data_stored":691346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574863,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":284,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":290816,"data_stored":105562,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574863,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27272,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":262144,"data_stored":100868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574862,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":266240,"data_stored":104106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574861,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27852,"kb_used_data":832,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343988,"statfs":{"total":96636764160,"available":96608243712,"internally_reserved":0,"allocated":851968,"data_stored":690702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574862,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":266240,"data_stored":100410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":5298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":714,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T23:16:17.350 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T23:16:17.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.575+0000 7fdbb0000640 1 Processor -- start 2024-04-24T23:16:17.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.575+0000 7fdbb0000640 1 -- start start 2024-04-24T23:16:17.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.577+0000 7fdbb0000640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdba815b530 0x7fdba8166380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:17.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.577+0000 7fdbb0000640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdba806ffd0 con 0x7fdba8154e10 2024-04-24T23:16:17.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.577+0000 7fdbb0000640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdba815e470 con 0x7fdba815e020 2024-04-24T23:16:17.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.577+0000 7fdbb0000640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fdba81551f0 con 0x7fdba815b530 2024-04-24T23:16:17.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.577+0000 7fdbad600640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fdba8154e10 0x7fdba806f8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:39704/0 (socket says 172.21.15.6:39704) 2024-04-24T23:16:17.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.577+0000 7fdbad600640 1 -- 172.21.15.6:0/861645294 learned_addr learned my addr 172.21.15.6:0/861645294 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:16:17.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.578+0000 7fdbae000640 1 --2- 172.21.15.6:0/861645294 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdba815b530 0x7fdba8166380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:17.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.578+0000 7fdba7e00640 1 -- 172.21.15.6:0/861645294 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1949634266 0 0) 0x7fdba806ffd0 con 0x7fdba8154e10 2024-04-24T23:16:17.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.578+0000 7fdba7e00640 1 -- 172.21.15.6:0/861645294 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdb90003520 con 0x7fdba8154e10 2024-04-24T23:16:17.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.578+0000 7fdba7e00640 1 -- 172.21.15.6:0/861645294 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 469395382 0 0) 0x7fdba815e470 con 0x7fdba815e020 2024-04-24T23:16:17.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.578+0000 7fdba7e00640 1 -- 172.21.15.6:0/861645294 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdba806ffd0 con 0x7fdba815e020 2024-04-24T23:16:17.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.578+0000 7fdba7e00640 1 -- 172.21.15.6:0/861645294 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3072271180 0 0) 0x7fdb90003520 con 0x7fdba8154e10 2024-04-24T23:16:17.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.578+0000 7fdba7e00640 1 -- 172.21.15.6:0/861645294 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdba815b530 msgr2=0x7fdba8166380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:16:17.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.578+0000 7fdba7e00640 1 --2- 172.21.15.6:0/861645294 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdba815b530 0x7fdba8166380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:17.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.578+0000 7fdba7e00640 1 -- 172.21.15.6:0/861645294 >> v1:172.21.15.190:6789/0 conn(0x7fdba815e020 legacy=0x7fdba80caf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:17.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.578+0000 7fdba7e00640 1 -- 172.21.15.6:0/861645294 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdba815e720 con 0x7fdba8154e10 2024-04-24T23:16:17.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.578+0000 7fdba7e00640 1 -- 172.21.15.6:0/861645294 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3463268701 0 0) 0x7fdb9c001670 con 0x7fdba8154e10 2024-04-24T23:16:17.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.578+0000 7fdba7e00640 1 -- 172.21.15.6:0/861645294 >> v1:172.21.15.6:6789/0 conn(0x7fdba8154e10 legacy=0x7fdba806f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:17.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.578+0000 7fdba7e00640 1 --2- 172.21.15.6:0/861645294 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdb90003b70 0x7fdb90003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:17.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.578+0000 7fdba7e00640 1 --2- 172.21.15.6:0/861645294 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdb90004800 0x7fdb90004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:17.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.578+0000 7fdba7e00640 1 --2- 172.21.15.6:0/861645294 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdb900051c0 0x7fdb9000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:17.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.578+0000 7fdba7e00640 1 -- 172.21.15.6:0/861645294 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fdba81551f0 con 0x7fdb90003b70 2024-04-24T23:16:17.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.578+0000 7fdba7e00640 1 -- 172.21.15.6:0/861645294 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fdb9000bfe0 con 0x7fdb90004800 2024-04-24T23:16:17.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.578+0000 7fdba7e00640 1 -- 172.21.15.6:0/861645294 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fdb9000c1e0 con 0x7fdb900051c0 2024-04-24T23:16:17.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.578+0000 7fdbae000640 1 --2- 172.21.15.6:0/861645294 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdba815b530 0x7fdba8166380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:16:17.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.579+0000 7fdbad600640 1 --2- 172.21.15.6:0/861645294 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdb90003b70 0x7fdb90003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:17.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.579+0000 7fdbae000640 1 --2- 172.21.15.6:0/861645294 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdb900051c0 0x7fdb9000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:17.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.579+0000 7fdbacc00640 1 --2- 172.21.15.6:0/861645294 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdb90004800 0x7fdb90004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:17.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.579+0000 7fdbad600640 1 -- 172.21.15.6:0/861645294 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdb900051c0 msgr2=0x7fdb9000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:16:17.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.579+0000 7fdbad600640 1 --2- 172.21.15.6:0/861645294 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdb900051c0 0x7fdb9000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:17.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.579+0000 7fdbad600640 1 -- 172.21.15.6:0/861645294 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdb90004800 msgr2=0x7fdb90004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:16:17.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.579+0000 7fdbad600640 1 --2- 172.21.15.6:0/861645294 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdb90004800 0x7fdb90004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:17.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.579+0000 7fdbad600640 1 -- 172.21.15.6:0/861645294 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb90003520 con 0x7fdb90003b70 2024-04-24T23:16:17.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.580+0000 7fdbad600640 1 --2- 172.21.15.6:0/861645294 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdb90003b70 0x7fdb90003f70 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fdb9c00c1c0 tx=0x7fdb90005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5ad5a1d593c3ea3 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:17.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.581+0000 7fdba7e00640 1 -- 172.21.15.6:0/861645294 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb9c007cc0 con 0x7fdb90003b70 2024-04-24T23:16:17.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.581+0000 7fdba7e00640 1 -- 172.21.15.6:0/861645294 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb9c005af0 con 0x7fdb90003b70 2024-04-24T23:16:17.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.581+0000 7fdba7e00640 1 -- 172.21.15.6:0/861645294 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb9c007070 con 0x7fdb90003b70 2024-04-24T23:16:17.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.581+0000 7fdbb0000640 1 -- 172.21.15.6:0/861645294 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdb90003b70 msgr2=0x7fdb90003f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:17.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.581+0000 7fdbb0000640 1 --2- 172.21.15.6:0/861645294 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdb90003b70 0x7fdb90003f70 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fdb9c00c1c0 tx=0x7fdb90005830 comp rx=0 tx=0).stop 2024-04-24T23:16:17.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.581+0000 7fdbae000640 1 -- 172.21.15.6:0/861645294 reap_dead start 2024-04-24T23:16:17.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.581+0000 7fdbb0000640 1 -- 172.21.15.6:0/861645294 shutdown_connections 2024-04-24T23:16:17.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.581+0000 7fdbb0000640 1 -- 172.21.15.6:0/861645294 >> 172.21.15.6:0/861645294 conn(0x7fdba8070ce0 msgr2=0x7fdba80710f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:17.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.582+0000 7fdbb0000640 1 -- 172.21.15.6:0/861645294 shutdown_connections 2024-04-24T23:16:17.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.582+0000 7fdbb0000640 1 -- 172.21.15.6:0/861645294 wait complete. 2024-04-24T23:16:17.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.583+0000 7fdbb0000640 1 Processor -- start 2024-04-24T23:16:17.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.584+0000 7fdbb0000640 1 -- start start 2024-04-24T23:16:17.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.584+0000 7fdbb0000640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdb90003b70 0x7fdba81f11e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:17.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.585+0000 7fdbb0000640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdba81562a0 0x7fdba80af170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:17.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.585+0000 7fdbb0000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdba8158780 0x7fdba80c3bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:17.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.585+0000 7fdbb0000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fdba806f0c0 con 0x7fdba8158780 2024-04-24T23:16:17.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.585+0000 7fdbb0000640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fdba815ece0 con 0x7fdba81562a0 2024-04-24T23:16:17.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.585+0000 7fdbb0000640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fdba806f3a0 con 0x7fdb90003b70 2024-04-24T23:16:17.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.585+0000 7fdbae000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdba8158780 0x7fdba80c3bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:17.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.585+0000 7fdbad600640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdb90003b70 0x7fdba81f11e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:17.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.585+0000 7fdbacc00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdba81562a0 0x7fdba80af170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:17.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.585+0000 7fdbae000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdba8158780 0x7fdba80c3bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:43514/0 (socket says 172.21.15.6:43514) 2024-04-24T23:16:17.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.585+0000 7fdbad600640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdb90003b70 0x7fdba81f11e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.6:60784/0 (socket says 172.21.15.6:60784) 2024-04-24T23:16:17.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.585+0000 7fdbae000640 1 -- 172.21.15.6:0/1029704429 learned_addr learned my addr 172.21.15.6:0/1029704429 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:16:17.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.585+0000 7fdbae000640 1 -- 172.21.15.6:0/1029704429 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdb90003b70 msgr2=0x7fdba81f11e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:17.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.585+0000 7fdbae000640 1 --2- 172.21.15.6:0/1029704429 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdb90003b70 0x7fdba81f11e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:17.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.585+0000 7fdbae000640 1 -- 172.21.15.6:0/1029704429 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdba81562a0 msgr2=0x7fdba80af170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:17.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.585+0000 7fdbae000640 1 --2- 172.21.15.6:0/1029704429 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdba81562a0 0x7fdba80af170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:17.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.585+0000 7fdbae000640 1 -- 172.21.15.6:0/1029704429 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdba81f1720 con 0x7fdba8158780 2024-04-24T23:16:17.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.586+0000 7fdbae000640 1 --2- 172.21.15.6:0/1029704429 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdba8158780 0x7fdba80c3bf0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fdba815b910 tx=0x7fdba000f7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:17.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.586+0000 7fdba5600640 1 -- 172.21.15.6:0/1029704429 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdba001e070 con 0x7fdba8158780 2024-04-24T23:16:17.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.586+0000 7fdba5600640 1 -- 172.21.15.6:0/1029704429 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdba0003990 con 0x7fdba8158780 2024-04-24T23:16:17.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.586+0000 7fdba5600640 1 -- 172.21.15.6:0/1029704429 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdba0003d80 con 0x7fdba8158780 2024-04-24T23:16:17.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.586+0000 7fdbb0000640 1 -- 172.21.15.6:0/1029704429 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdba80af6b0 con 0x7fdba8158780 2024-04-24T23:16:17.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.586+0000 7fdbb0000640 1 -- 172.21.15.6:0/1029704429 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdba8157200 con 0x7fdba8158780 2024-04-24T23:16:17.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.587+0000 7fdbb0000640 1 -- 172.21.15.6:0/1029704429 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdba815b9c0 con 0x7fdba8158780 2024-04-24T23:16:17.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.588+0000 7fdba5600640 1 -- 172.21.15.6:0/1029704429 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fdba0021070 con 0x7fdba8158780 2024-04-24T23:16:17.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.589+0000 7fdba5600640 1 --2- 172.21.15.6:0/1029704429 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fdb8806be30 0x7fdb8806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:17.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.589+0000 7fdba5600640 1 -- 172.21.15.6:0/1029704429 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6590+0+0 (secure 0 0 0) 0x7fdba008b2a0 con 0x7fdba8158780 2024-04-24T23:16:17.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.590+0000 7fdbad600640 1 --2- 172.21.15.6:0/1029704429 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fdb8806be30 0x7fdb8806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:17.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.590+0000 7fdbad600640 1 --2- 172.21.15.6:0/1029704429 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fdb8806be30 0x7fdb8806e2f0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fdb9c032000 tx=0x7fdb9c034040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:17.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.598+0000 7fdba5600640 1 -- 172.21.15.6:0/1029704429 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fdba815b9c0 con 0x7fdba8158780 2024-04-24T23:16:17.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.786+0000 7fdbb0000640 1 -- 172.21.15.6:0/1029704429 --> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fdba8068430 con 0x7fdb8806be30 2024-04-24T23:16:17.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.796+0000 7fdba5600640 1 -- 172.21.15.6:0/1029704429 <== mgr.4105 v2:172.21.15.190:6832/806769663 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+332574 (secure 0 0 0) 0x7fdba8068430 con 0x7fdb8806be30 2024-04-24T23:16:17.802 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:16:17.804 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-04-24T23:16:17.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.802+0000 7fdbb0000640 1 -- 172.21.15.6:0/1029704429 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fdb8806be30 msgr2=0x7fdb8806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:17.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.802+0000 7fdbb0000640 1 --2- 172.21.15.6:0/1029704429 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fdb8806be30 0x7fdb8806e2f0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fdb9c032000 tx=0x7fdb9c034040 comp rx=0 tx=0).stop 2024-04-24T23:16:17.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.802+0000 7fdbb0000640 1 -- 172.21.15.6:0/1029704429 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdba8158780 msgr2=0x7fdba80c3bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:17.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.802+0000 7fdbb0000640 1 --2- 172.21.15.6:0/1029704429 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdba8158780 0x7fdba80c3bf0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fdba815b910 tx=0x7fdba000f7b0 comp rx=0 tx=0).stop 2024-04-24T23:16:17.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.802+0000 7fdbb0000640 1 -- 172.21.15.6:0/1029704429 shutdown_connections 2024-04-24T23:16:17.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.802+0000 7fdbb0000640 1 --2- 172.21.15.6:0/1029704429 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fdb8806be30 0x7fdb8806e2f0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:17.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.802+0000 7fdbb0000640 1 --2- 172.21.15.6:0/1029704429 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdba8158780 0x7fdba80c3bf0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:17.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.802+0000 7fdbb0000640 1 --2- 172.21.15.6:0/1029704429 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdba81562a0 0x7fdba80af170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:17.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.802+0000 7fdbb0000640 1 --2- 172.21.15.6:0/1029704429 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdb90003b70 0x7fdba81f11e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:17.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.802+0000 7fdbb0000640 1 -- 172.21.15.6:0/1029704429 >> 172.21.15.6:0/1029704429 conn(0x7fdba8070ce0 msgr2=0x7fdba80cd4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:17.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.802+0000 7fdbb0000640 1 -- 172.21.15.6:0/1029704429 shutdown_connections 2024-04-24T23:16:17.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:17.802+0000 7fdbb0000640 1 -- 172.21.15.6:0/1029704429 wait complete. 2024-04-24T23:16:17.819 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":44,"stamp":"2024-04-24T23:16:17.219061+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596414,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":247,"num_write_kb":2447,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":26,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":194,"ondisk_log_size":194,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219384,"kb_used_data":3252,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755336,"statfs":{"total":773094113280,"available":772869464064,"internally_reserved":0,"allocated":3330048,"data_stored":1994850,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1406,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":39,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.532076"},"pg_stats":[{"pgid":"2.1f","version":"23'4","reported_seq":26,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.051292+0000","last_change":"2024-04-24T23:15:35.964646+0000","last_active":"2024-04-24T23:15:42.051292+0000","last_peered":"2024-04-24T23:15:42.051292+0000","last_clean":"2024-04-24T23:15:42.051292+0000","last_became_active":"2024-04-24T23:15:35.964357+0000","last_became_peered":"2024-04-24T23:15:35.964357+0000","last_unstale":"2024-04-24T23:15:42.051292+0000","last_undegraded":"2024-04-24T23:15:42.051292+0000","last_fullsized":"2024-04-24T23:15:42.051292+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:01:17.912993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.295716+0000","last_change":"2024-04-24T23:15:36.989772+0000","last_active":"2024-04-24T23:15:42.295716+0000","last_peered":"2024-04-24T23:15:42.295716+0000","last_clean":"2024-04-24T23:15:42.295716+0000","last_became_active":"2024-04-24T23:15:36.989543+0000","last_became_peered":"2024-04-24T23:15:36.989543+0000","last_unstale":"2024-04-24T23:15:42.295716+0000","last_undegraded":"2024-04-24T23:15:42.295716+0000","last_fullsized":"2024-04-24T23:15:42.295716+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:37:16.737354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.187704+0000","last_change":"2024-04-24T23:15:41.017182+0000","last_active":"2024-04-24T23:15:43.187704+0000","last_peered":"2024-04-24T23:15:43.187704+0000","last_clean":"2024-04-24T23:15:43.187704+0000","last_became_active":"2024-04-24T23:15:41.016719+0000","last_became_peered":"2024-04-24T23:15:41.016719+0000","last_unstale":"2024-04-24T23:15:43.187704+0000","last_undegraded":"2024-04-24T23:15:43.187704+0000","last_fullsized":"2024-04-24T23:15:43.187704+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:42:11.186486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4,1],"acting":[7,6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1e","version":"23'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.295817+0000","last_change":"2024-04-24T23:15:35.946083+0000","last_active":"2024-04-24T23:15:42.295817+0000","last_peered":"2024-04-24T23:15:42.295817+0000","last_clean":"2024-04-24T23:15:42.295817+0000","last_became_active":"2024-04-24T23:15:35.945227+0000","last_became_peered":"2024-04-24T23:15:35.945227+0000","last_unstale":"2024-04-24T23:15:42.295817+0000","last_undegraded":"2024-04-24T23:15:42.295817+0000","last_fullsized":"2024-04-24T23:15:42.295817+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:05:17.727725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.295775+0000","last_change":"2024-04-24T23:15:36.998245+0000","last_active":"2024-04-24T23:15:42.295775+0000","last_peered":"2024-04-24T23:15:42.295775+0000","last_clean":"2024-04-24T23:15:42.295775+0000","last_became_active":"2024-04-24T23:15:36.998021+0000","last_became_peered":"2024-04-24T23:15:36.998021+0000","last_unstale":"2024-04-24T23:15:42.295775+0000","last_undegraded":"2024-04-24T23:15:42.295775+0000","last_fullsized":"2024-04-24T23:15:42.295775+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:03:42.359117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.106527+0000","last_change":"2024-04-24T23:15:41.008313+0000","last_active":"2024-04-24T23:15:42.106527+0000","last_peered":"2024-04-24T23:15:42.106527+0000","last_clean":"2024-04-24T23:15:42.106527+0000","last_became_active":"2024-04-24T23:15:41.007908+0000","last_became_peered":"2024-04-24T23:15:41.007908+0000","last_unstale":"2024-04-24T23:15:42.106527+0000","last_undegraded":"2024-04-24T23:15:42.106527+0000","last_fullsized":"2024-04-24T23:15:42.106527+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:02:11.902575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4,6],"acting":[2,7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1d","version":"23'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.193739+0000","last_change":"2024-04-24T23:15:35.968312+0000","last_active":"2024-04-24T23:15:43.193739+0000","last_peered":"2024-04-24T23:15:43.193739+0000","last_clean":"2024-04-24T23:15:43.193739+0000","last_became_active":"2024-04-24T23:15:35.967873+0000","last_became_peered":"2024-04-24T23:15:35.967873+0000","last_unstale":"2024-04-24T23:15:43.193739+0000","last_undegraded":"2024-04-24T23:15:43.193739+0000","last_fullsized":"2024-04-24T23:15:43.193739+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:21:40.116977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.733492+0000","last_change":"2024-04-24T23:15:36.997361+0000","last_active":"2024-04-24T23:15:42.733492+0000","last_peered":"2024-04-24T23:15:42.733492+0000","last_clean":"2024-04-24T23:15:42.733492+0000","last_became_active":"2024-04-24T23:15:36.997083+0000","last_became_peered":"2024-04-24T23:15:36.997083+0000","last_unstale":"2024-04-24T23:15:42.733492+0000","last_undegraded":"2024-04-24T23:15:42.733492+0000","last_fullsized":"2024-04-24T23:15:42.733492+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:47:59.334093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.107355+0000","last_change":"2024-04-24T23:15:41.025153+0000","last_active":"2024-04-24T23:15:42.107355+0000","last_peered":"2024-04-24T23:15:42.107355+0000","last_clean":"2024-04-24T23:15:42.107355+0000","last_became_active":"2024-04-24T23:15:41.024739+0000","last_became_peered":"2024-04-24T23:15:41.024739+0000","last_unstale":"2024-04-24T23:15:42.107355+0000","last_undegraded":"2024-04-24T23:15:42.107355+0000","last_fullsized":"2024-04-24T23:15:42.107355+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:31:00.160213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3,6],"acting":[2,5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1c","version":"23'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.193705+0000","last_change":"2024-04-24T23:15:35.967338+0000","last_active":"2024-04-24T23:15:43.193705+0000","last_peered":"2024-04-24T23:15:43.193705+0000","last_clean":"2024-04-24T23:15:43.193705+0000","last_became_active":"2024-04-24T23:15:35.966951+0000","last_became_peered":"2024-04-24T23:15:35.966951+0000","last_unstale":"2024-04-24T23:15:43.193705+0000","last_undegraded":"2024-04-24T23:15:43.193705+0000","last_fullsized":"2024-04-24T23:15:43.193705+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:08:05.248330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.191745+0000","last_change":"2024-04-24T23:15:36.998072+0000","last_active":"2024-04-24T23:15:43.191745+0000","last_peered":"2024-04-24T23:15:43.191745+0000","last_clean":"2024-04-24T23:15:43.191745+0000","last_became_active":"2024-04-24T23:15:36.997735+0000","last_became_peered":"2024-04-24T23:15:36.997735+0000","last_unstale":"2024-04-24T23:15:43.191745+0000","last_undegraded":"2024-04-24T23:15:43.191745+0000","last_fullsized":"2024-04-24T23:15:43.191745+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:22:06.869644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.106640+0000","last_change":"2024-04-24T23:15:41.008895+0000","last_active":"2024-04-24T23:15:42.106640+0000","last_peered":"2024-04-24T23:15:42.106640+0000","last_clean":"2024-04-24T23:15:42.106640+0000","last_became_active":"2024-04-24T23:15:41.006199+0000","last_became_peered":"2024-04-24T23:15:41.006199+0000","last_unstale":"2024-04-24T23:15:42.106640+0000","last_undegraded":"2024-04-24T23:15:42.106640+0000","last_fullsized":"2024-04-24T23:15:42.106640+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:13:34.583602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5,7],"acting":[2,6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.191106+0000","last_change":"2024-04-24T23:15:37.004074+0000","last_active":"2024-04-24T23:15:43.191106+0000","last_peered":"2024-04-24T23:15:43.191106+0000","last_clean":"2024-04-24T23:15:43.191106+0000","last_became_active":"2024-04-24T23:15:37.003544+0000","last_became_peered":"2024-04-24T23:15:37.003544+0000","last_unstale":"2024-04-24T23:15:43.191106+0000","last_undegraded":"2024-04-24T23:15:43.191106+0000","last_fullsized":"2024-04-24T23:15:43.191106+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:38:28.024702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"24'4","reported_seq":26,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:16:13.162728+0000","last_change":"2024-04-24T23:15:35.963000+0000","last_active":"2024-04-24T23:16:13.162728+0000","last_peered":"2024-04-24T23:16:13.162728+0000","last_clean":"2024-04-24T23:16:13.162728+0000","last_became_active":"2024-04-24T23:15:35.961960+0000","last_became_peered":"2024-04-24T23:15:35.961960+0000","last_unstale":"2024-04-24T23:16:13.162728+0000","last_undegraded":"2024-04-24T23:16:13.162728+0000","last_fullsized":"2024-04-24T23:16:13.162728+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:56:26.651193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.191430+0000","last_change":"2024-04-24T23:15:41.022481+0000","last_active":"2024-04-24T23:15:43.191430+0000","last_peered":"2024-04-24T23:15:43.191430+0000","last_clean":"2024-04-24T23:15:43.191430+0000","last_became_active":"2024-04-24T23:15:41.020034+0000","last_became_peered":"2024-04-24T23:15:41.020034+0000","last_unstale":"2024-04-24T23:15:43.191430+0000","last_undegraded":"2024-04-24T23:15:43.191430+0000","last_fullsized":"2024-04-24T23:15:43.191430+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:27:25.337503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,5],"acting":[3,1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.187801+0000","last_change":"2024-04-24T23:15:36.972020+0000","last_active":"2024-04-24T23:15:43.187801+0000","last_peered":"2024-04-24T23:15:43.187801+0000","last_clean":"2024-04-24T23:15:43.187801+0000","last_became_active":"2024-04-24T23:15:36.971583+0000","last_became_peered":"2024-04-24T23:15:36.971583+0000","last_unstale":"2024-04-24T23:15:43.187801+0000","last_undegraded":"2024-04-24T23:15:43.187801+0000","last_fullsized":"2024-04-24T23:15:43.187801+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:29:05.568608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.050091+0000","last_change":"2024-04-24T23:15:35.945916+0000","last_active":"2024-04-24T23:15:42.050091+0000","last_peered":"2024-04-24T23:15:42.050091+0000","last_clean":"2024-04-24T23:15:42.050091+0000","last_became_active":"2024-04-24T23:15:35.945610+0000","last_became_peered":"2024-04-24T23:15:35.945610+0000","last_unstale":"2024-04-24T23:15:42.050091+0000","last_undegraded":"2024-04-24T23:15:42.050091+0000","last_fullsized":"2024-04-24T23:15:42.050091+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:57:05.976074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.192627+0000","last_change":"2024-04-24T23:15:41.030165+0000","last_active":"2024-04-24T23:15:43.192627+0000","last_peered":"2024-04-24T23:15:43.192627+0000","last_clean":"2024-04-24T23:15:43.192627+0000","last_became_active":"2024-04-24T23:15:41.029685+0000","last_became_peered":"2024-04-24T23:15:41.029685+0000","last_unstale":"2024-04-24T23:15:43.192627+0000","last_undegraded":"2024-04-24T23:15:43.192627+0000","last_fullsized":"2024-04-24T23:15:43.192627+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:20:43.498190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6,1],"acting":[0,4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.049168+0000","last_change":"2024-04-24T23:15:36.965949+0000","last_active":"2024-04-24T23:15:42.049168+0000","last_peered":"2024-04-24T23:15:42.049168+0000","last_clean":"2024-04-24T23:15:42.049168+0000","last_became_active":"2024-04-24T23:15:36.965658+0000","last_became_peered":"2024-04-24T23:15:36.965658+0000","last_unstale":"2024-04-24T23:15:42.049168+0000","last_undegraded":"2024-04-24T23:15:42.049168+0000","last_fullsized":"2024-04-24T23:15:42.049168+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:38:28.024702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"24'3","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:16:13.164854+0000","last_change":"2024-04-24T23:15:35.972165+0000","last_active":"2024-04-24T23:16:13.164854+0000","last_peered":"2024-04-24T23:16:13.164854+0000","last_clean":"2024-04-24T23:16:13.164854+0000","last_became_active":"2024-04-24T23:15:35.971972+0000","last_became_peered":"2024-04-24T23:15:35.971972+0000","last_unstale":"2024-04-24T23:16:13.164854+0000","last_undegraded":"2024-04-24T23:16:13.164854+0000","last_fullsized":"2024-04-24T23:16:13.164854+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:11:12.088698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.188224+0000","last_change":"2024-04-24T23:15:41.019799+0000","last_active":"2024-04-24T23:15:43.188224+0000","last_peered":"2024-04-24T23:15:43.188224+0000","last_clean":"2024-04-24T23:15:43.188224+0000","last_became_active":"2024-04-24T23:15:41.019339+0000","last_became_peered":"2024-04-24T23:15:41.019339+0000","last_unstale":"2024-04-24T23:15:43.188224+0000","last_undegraded":"2024-04-24T23:15:43.188224+0000","last_fullsized":"2024-04-24T23:15:43.188224+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:59:34.601004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2,3],"acting":[7,4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.190629+0000","last_change":"2024-04-24T23:15:36.991991+0000","last_active":"2024-04-24T23:15:43.190629+0000","last_peered":"2024-04-24T23:15:43.190629+0000","last_clean":"2024-04-24T23:15:43.190629+0000","last_became_active":"2024-04-24T23:15:36.989336+0000","last_became_peered":"2024-04-24T23:15:36.989336+0000","last_unstale":"2024-04-24T23:15:43.190629+0000","last_undegraded":"2024-04-24T23:15:43.190629+0000","last_fullsized":"2024-04-24T23:15:43.190629+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:25:56.038903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.8","version":"24'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:16:13.164232+0000","last_change":"2024-04-24T23:15:35.965443+0000","last_active":"2024-04-24T23:16:13.164232+0000","last_peered":"2024-04-24T23:16:13.164232+0000","last_clean":"2024-04-24T23:16:13.164232+0000","last_became_active":"2024-04-24T23:15:35.965043+0000","last_became_peered":"2024-04-24T23:15:35.965043+0000","last_unstale":"2024-04-24T23:16:13.164232+0000","last_undegraded":"2024-04-24T23:16:13.164232+0000","last_fullsized":"2024-04-24T23:16:13.164232+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:07:23.320042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.107146+0000","last_change":"2024-04-24T23:15:41.025131+0000","last_active":"2024-04-24T23:15:42.107146+0000","last_peered":"2024-04-24T23:15:42.107146+0000","last_clean":"2024-04-24T23:15:42.107146+0000","last_became_active":"2024-04-24T23:15:41.024731+0000","last_became_peered":"2024-04-24T23:15:41.024731+0000","last_unstale":"2024-04-24T23:15:42.107146+0000","last_undegraded":"2024-04-24T23:15:42.107146+0000","last_fullsized":"2024-04-24T23:15:42.107146+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:28:07.952854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,7],"acting":[2,1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.049954+0000","last_change":"2024-04-24T23:15:36.965850+0000","last_active":"2024-04-24T23:15:42.049954+0000","last_peered":"2024-04-24T23:15:42.049954+0000","last_clean":"2024-04-24T23:15:42.049954+0000","last_became_active":"2024-04-24T23:15:36.965547+0000","last_became_peered":"2024-04-24T23:15:36.965547+0000","last_unstale":"2024-04-24T23:15:42.049954+0000","last_undegraded":"2024-04-24T23:15:42.049954+0000","last_fullsized":"2024-04-24T23:15:42.049954+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:06:37.125573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"24'4","reported_seq":26,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:16:13.163117+0000","last_change":"2024-04-24T23:15:35.971040+0000","last_active":"2024-04-24T23:16:13.163117+0000","last_peered":"2024-04-24T23:16:13.163117+0000","last_clean":"2024-04-24T23:16:13.163117+0000","last_became_active":"2024-04-24T23:15:35.970184+0000","last_became_peered":"2024-04-24T23:15:35.970184+0000","last_unstale":"2024-04-24T23:16:13.163117+0000","last_undegraded":"2024-04-24T23:16:13.163117+0000","last_fullsized":"2024-04-24T23:16:13.163117+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:45:20.913759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.106686+0000","last_change":"2024-04-24T23:15:41.025813+0000","last_active":"2024-04-24T23:15:42.106686+0000","last_peered":"2024-04-24T23:15:42.106686+0000","last_clean":"2024-04-24T23:15:42.106686+0000","last_became_active":"2024-04-24T23:15:41.025137+0000","last_became_peered":"2024-04-24T23:15:41.025137+0000","last_unstale":"2024-04-24T23:15:42.106686+0000","last_undegraded":"2024-04-24T23:15:42.106686+0000","last_fullsized":"2024-04-24T23:15:42.106686+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:35:17.784854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,3,1],"acting":[2,6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.051163+0000","last_change":"2024-04-24T23:15:36.967476+0000","last_active":"2024-04-24T23:15:42.051163+0000","last_peered":"2024-04-24T23:15:42.051163+0000","last_clean":"2024-04-24T23:15:42.051163+0000","last_became_active":"2024-04-24T23:15:36.967098+0000","last_became_peered":"2024-04-24T23:15:36.967098+0000","last_unstale":"2024-04-24T23:15:42.051163+0000","last_undegraded":"2024-04-24T23:15:42.051163+0000","last_fullsized":"2024-04-24T23:15:42.051163+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:25:56.038903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"24'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.050091+0000","last_change":"2024-04-24T23:15:35.970630+0000","last_active":"2024-04-24T23:15:42.050091+0000","last_peered":"2024-04-24T23:15:42.050091+0000","last_clean":"2024-04-24T23:15:42.050091+0000","last_became_active":"2024-04-24T23:15:35.970430+0000","last_became_peered":"2024-04-24T23:15:35.970430+0000","last_unstale":"2024-04-24T23:15:42.050091+0000","last_undegraded":"2024-04-24T23:15:42.050091+0000","last_fullsized":"2024-04-24T23:15:42.050091+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:15:59.219084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.049399+0000","last_change":"2024-04-24T23:15:41.005022+0000","last_active":"2024-04-24T23:15:42.049399+0000","last_peered":"2024-04-24T23:15:42.049399+0000","last_clean":"2024-04-24T23:15:42.049399+0000","last_became_active":"2024-04-24T23:15:41.004648+0000","last_became_peered":"2024-04-24T23:15:41.004648+0000","last_unstale":"2024-04-24T23:15:42.049399+0000","last_undegraded":"2024-04-24T23:15:42.049399+0000","last_fullsized":"2024-04-24T23:15:42.049399+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:06:01.559760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6,3],"acting":[4,1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.051008+0000","last_change":"2024-04-24T23:15:36.967053+0000","last_active":"2024-04-24T23:15:42.051008+0000","last_peered":"2024-04-24T23:15:42.051008+0000","last_clean":"2024-04-24T23:15:42.051008+0000","last_became_active":"2024-04-24T23:15:36.966746+0000","last_became_peered":"2024-04-24T23:15:36.966746+0000","last_unstale":"2024-04-24T23:15:42.051008+0000","last_undegraded":"2024-04-24T23:15:42.051008+0000","last_fullsized":"2024-04-24T23:15:42.051008+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:16:08.779860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"24'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.192404+0000","last_change":"2024-04-24T23:15:35.965890+0000","last_active":"2024-04-24T23:15:43.192404+0000","last_peered":"2024-04-24T23:15:43.192404+0000","last_clean":"2024-04-24T23:15:43.192404+0000","last_became_active":"2024-04-24T23:15:35.965631+0000","last_became_peered":"2024-04-24T23:15:35.965631+0000","last_unstale":"2024-04-24T23:15:43.192404+0000","last_undegraded":"2024-04-24T23:15:43.192404+0000","last_fullsized":"2024-04-24T23:15:43.192404+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:17:35.650731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.050382+0000","last_change":"2024-04-24T23:15:41.011405+0000","last_active":"2024-04-24T23:15:42.050382+0000","last_peered":"2024-04-24T23:15:42.050382+0000","last_clean":"2024-04-24T23:15:42.050382+0000","last_became_active":"2024-04-24T23:15:41.010982+0000","last_became_peered":"2024-04-24T23:15:41.010982+0000","last_unstale":"2024-04-24T23:15:42.050382+0000","last_undegraded":"2024-04-24T23:15:42.050382+0000","last_fullsized":"2024-04-24T23:15:42.050382+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:08:00.914296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2,7],"acting":[6,0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.106945+0000","last_change":"2024-04-24T23:15:36.975244+0000","last_active":"2024-04-24T23:15:42.106945+0000","last_peered":"2024-04-24T23:15:42.106945+0000","last_clean":"2024-04-24T23:15:42.106945+0000","last_became_active":"2024-04-24T23:15:36.974872+0000","last_became_peered":"2024-04-24T23:15:36.974872+0000","last_unstale":"2024-04-24T23:15:42.106945+0000","last_undegraded":"2024-04-24T23:15:42.106945+0000","last_fullsized":"2024-04-24T23:15:42.106945+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:45:57.018205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.295386+0000","last_change":"2024-04-24T23:15:35.945792+0000","last_active":"2024-04-24T23:15:42.295386+0000","last_peered":"2024-04-24T23:15:42.295386+0000","last_clean":"2024-04-24T23:15:42.295386+0000","last_became_active":"2024-04-24T23:15:35.945094+0000","last_became_peered":"2024-04-24T23:15:35.945094+0000","last_unstale":"2024-04-24T23:15:42.295386+0000","last_undegraded":"2024-04-24T23:15:42.295386+0000","last_fullsized":"2024-04-24T23:15:42.295386+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:49:12.719127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.049802+0000","last_change":"2024-04-24T23:15:41.022105+0000","last_active":"2024-04-24T23:15:42.049802+0000","last_peered":"2024-04-24T23:15:42.049802+0000","last_clean":"2024-04-24T23:15:42.049802+0000","last_became_active":"2024-04-24T23:15:41.021645+0000","last_became_peered":"2024-04-24T23:15:41.021645+0000","last_unstale":"2024-04-24T23:15:42.049802+0000","last_undegraded":"2024-04-24T23:15:42.049802+0000","last_fullsized":"2024-04-24T23:15:42.049802+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:03:52.974835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5,7],"acting":[6,3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.192545+0000","last_change":"2024-04-24T23:15:36.991239+0000","last_active":"2024-04-24T23:15:43.192545+0000","last_peered":"2024-04-24T23:15:43.192545+0000","last_clean":"2024-04-24T23:15:43.192545+0000","last_became_active":"2024-04-24T23:15:36.990963+0000","last_became_peered":"2024-04-24T23:15:36.990963+0000","last_unstale":"2024-04-24T23:15:43.192545+0000","last_undegraded":"2024-04-24T23:15:43.192545+0000","last_fullsized":"2024-04-24T23:15:43.192545+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:53:28.101334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.191521+0000","last_change":"2024-04-24T23:15:35.957931+0000","last_active":"2024-04-24T23:15:43.191521+0000","last_peered":"2024-04-24T23:15:43.191521+0000","last_clean":"2024-04-24T23:15:43.191521+0000","last_became_active":"2024-04-24T23:15:35.957432+0000","last_became_peered":"2024-04-24T23:15:35.957432+0000","last_unstale":"2024-04-24T23:15:43.191521+0000","last_undegraded":"2024-04-24T23:15:43.191521+0000","last_fullsized":"2024-04-24T23:15:43.191521+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:11:38.773425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.050839+0000","last_change":"2024-04-24T23:15:41.007857+0000","last_active":"2024-04-24T23:15:42.050839+0000","last_peered":"2024-04-24T23:15:42.050839+0000","last_clean":"2024-04-24T23:15:42.050839+0000","last_became_active":"2024-04-24T23:15:41.007449+0000","last_became_peered":"2024-04-24T23:15:41.007449+0000","last_unstale":"2024-04-24T23:15:42.050839+0000","last_undegraded":"2024-04-24T23:15:42.050839+0000","last_fullsized":"2024-04-24T23:15:42.050839+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:05:06.145440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0,3],"acting":[6,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.295077+0000","last_change":"2024-04-24T23:15:36.970444+0000","last_active":"2024-04-24T23:15:42.295077+0000","last_peered":"2024-04-24T23:15:42.295077+0000","last_clean":"2024-04-24T23:15:42.295077+0000","last_became_active":"2024-04-24T23:15:36.970159+0000","last_became_peered":"2024-04-24T23:15:36.970159+0000","last_unstale":"2024-04-24T23:15:42.295077+0000","last_undegraded":"2024-04-24T23:15:42.295077+0000","last_fullsized":"2024-04-24T23:15:42.295077+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:37:02.956224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"24'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.187846+0000","last_change":"2024-04-24T23:15:35.945461+0000","last_active":"2024-04-24T23:15:43.187846+0000","last_peered":"2024-04-24T23:15:43.187846+0000","last_clean":"2024-04-24T23:15:43.187846+0000","last_became_active":"2024-04-24T23:15:35.945185+0000","last_became_peered":"2024-04-24T23:15:35.945185+0000","last_unstale":"2024-04-24T23:15:43.187846+0000","last_undegraded":"2024-04-24T23:15:43.187846+0000","last_fullsized":"2024-04-24T23:15:43.187846+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:37:34.175539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.050260+0000","last_change":"2024-04-24T23:15:41.014584+0000","last_active":"2024-04-24T23:15:42.050260+0000","last_peered":"2024-04-24T23:15:42.050260+0000","last_clean":"2024-04-24T23:15:42.050260+0000","last_became_active":"2024-04-24T23:15:41.014137+0000","last_became_peered":"2024-04-24T23:15:41.014137+0000","last_unstale":"2024-04-24T23:15:42.050260+0000","last_undegraded":"2024-04-24T23:15:42.050260+0000","last_fullsized":"2024-04-24T23:15:42.050260+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:23:50.387635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,7],"acting":[6,0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.049485+0000","last_change":"2024-04-24T23:15:36.981783+0000","last_active":"2024-04-24T23:15:42.049485+0000","last_peered":"2024-04-24T23:15:42.049485+0000","last_clean":"2024-04-24T23:15:42.049485+0000","last_became_active":"2024-04-24T23:15:36.981502+0000","last_became_peered":"2024-04-24T23:15:36.981502+0000","last_unstale":"2024-04-24T23:15:42.049485+0000","last_undegraded":"2024-04-24T23:15:42.049485+0000","last_fullsized":"2024-04-24T23:15:42.049485+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:56:08.483410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"24'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.190892+0000","last_change":"2024-04-24T23:15:35.960519+0000","last_active":"2024-04-24T23:15:43.190892+0000","last_peered":"2024-04-24T23:15:43.190892+0000","last_clean":"2024-04-24T23:15:43.190892+0000","last_became_active":"2024-04-24T23:15:35.960180+0000","last_became_peered":"2024-04-24T23:15:35.960180+0000","last_unstale":"2024-04-24T23:15:43.190892+0000","last_undegraded":"2024-04-24T23:15:43.190892+0000","last_fullsized":"2024-04-24T23:15:43.190892+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:50:00.199120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.295661+0000","last_change":"2024-04-24T23:15:41.012876+0000","last_active":"2024-04-24T23:15:42.295661+0000","last_peered":"2024-04-24T23:15:42.295661+0000","last_clean":"2024-04-24T23:15:42.295661+0000","last_became_active":"2024-04-24T23:15:41.012493+0000","last_became_peered":"2024-04-24T23:15:41.012493+0000","last_unstale":"2024-04-24T23:15:42.295661+0000","last_undegraded":"2024-04-24T23:15:42.295661+0000","last_fullsized":"2024-04-24T23:15:42.295661+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:21:28.287059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6,1],"acting":[5,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.050057+0000","last_change":"2024-04-24T23:15:36.981449+0000","last_active":"2024-04-24T23:15:42.050057+0000","last_peered":"2024-04-24T23:15:42.050057+0000","last_clean":"2024-04-24T23:15:42.050057+0000","last_became_active":"2024-04-24T23:15:36.980837+0000","last_became_peered":"2024-04-24T23:15:36.980837+0000","last_unstale":"2024-04-24T23:15:42.050057+0000","last_undegraded":"2024-04-24T23:15:42.050057+0000","last_fullsized":"2024-04-24T23:15:42.050057+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:15:13.513341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.3","version":"24'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:16:13.164506+0000","last_change":"2024-04-24T23:15:35.952947+0000","last_active":"2024-04-24T23:16:13.164506+0000","last_peered":"2024-04-24T23:16:13.164506+0000","last_clean":"2024-04-24T23:16:13.164506+0000","last_became_active":"2024-04-24T23:15:35.952655+0000","last_became_peered":"2024-04-24T23:15:35.952655+0000","last_unstale":"2024-04-24T23:16:13.164506+0000","last_undegraded":"2024-04-24T23:16:13.164506+0000","last_fullsized":"2024-04-24T23:16:13.164506+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:50:10.939396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":178,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.732636+0000","last_change":"2024-04-24T23:15:25.826673+0000","last_active":"2024-04-24T23:15:42.732636+0000","last_peered":"2024-04-24T23:15:42.732636+0000","last_clean":"2024-04-24T23:15:42.732636+0000","last_became_active":"2024-04-24T23:15:25.820030+0000","last_became_peered":"2024-04-24T23:15:25.820030+0000","last_unstale":"2024-04-24T23:15:42.732636+0000","last_undegraded":"2024-04-24T23:15:42.732636+0000","last_fullsized":"2024-04-24T23:15:42.732636+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:24.790928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:24.790928+0000","last_clean_scrub_stamp":"2024-04-24T23:15:24.790928+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:37:24.054332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.192066+0000","last_change":"2024-04-24T23:15:41.024712+0000","last_active":"2024-04-24T23:15:43.192066+0000","last_peered":"2024-04-24T23:15:43.192066+0000","last_clean":"2024-04-24T23:15:43.192066+0000","last_became_active":"2024-04-24T23:15:41.022697+0000","last_became_peered":"2024-04-24T23:15:41.022697+0000","last_unstale":"2024-04-24T23:15:43.192066+0000","last_undegraded":"2024-04-24T23:15:43.192066+0000","last_fullsized":"2024-04-24T23:15:43.192066+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:37:08.231105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2,0],"acting":[3,1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.191847+0000","last_change":"2024-04-24T23:15:37.000489+0000","last_active":"2024-04-24T23:15:43.191847+0000","last_peered":"2024-04-24T23:15:43.191847+0000","last_clean":"2024-04-24T23:15:43.191847+0000","last_became_active":"2024-04-24T23:15:37.000179+0000","last_became_peered":"2024-04-24T23:15:37.000179+0000","last_unstale":"2024-04-24T23:15:43.191847+0000","last_undegraded":"2024-04-24T23:15:43.191847+0000","last_fullsized":"2024-04-24T23:15:43.191847+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:06:37.125573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.051120+0000","last_change":"2024-04-24T23:15:35.966020+0000","last_active":"2024-04-24T23:15:42.051120+0000","last_peered":"2024-04-24T23:15:42.051120+0000","last_clean":"2024-04-24T23:15:42.051120+0000","last_became_active":"2024-04-24T23:15:35.965758+0000","last_became_peered":"2024-04-24T23:15:35.965758+0000","last_unstale":"2024-04-24T23:15:42.051120+0000","last_undegraded":"2024-04-24T23:15:42.051120+0000","last_fullsized":"2024-04-24T23:15:42.051120+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:39:21.233461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.106667+0000","last_change":"2024-04-24T23:15:41.024489+0000","last_active":"2024-04-24T23:15:42.106667+0000","last_peered":"2024-04-24T23:15:42.106667+0000","last_clean":"2024-04-24T23:15:42.106667+0000","last_became_active":"2024-04-24T23:15:41.024041+0000","last_became_peered":"2024-04-24T23:15:41.024041+0000","last_unstale":"2024-04-24T23:15:42.106667+0000","last_undegraded":"2024-04-24T23:15:42.106667+0000","last_fullsized":"2024-04-24T23:15:42.106667+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:27:18.197003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7,4],"acting":[2,3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.193516+0000","last_change":"2024-04-24T23:15:36.979887+0000","last_active":"2024-04-24T23:15:43.193516+0000","last_peered":"2024-04-24T23:15:43.193516+0000","last_clean":"2024-04-24T23:15:43.193516+0000","last_became_active":"2024-04-24T23:15:36.979611+0000","last_became_peered":"2024-04-24T23:15:36.979611+0000","last_unstale":"2024-04-24T23:15:43.193516+0000","last_undegraded":"2024-04-24T23:15:43.193516+0000","last_fullsized":"2024-04-24T23:15:43.193516+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:07:18.261077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"23'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.295532+0000","last_change":"2024-04-24T23:15:35.965192+0000","last_active":"2024-04-24T23:15:42.295532+0000","last_peered":"2024-04-24T23:15:42.295532+0000","last_clean":"2024-04-24T23:15:42.295532+0000","last_became_active":"2024-04-24T23:15:35.964873+0000","last_became_peered":"2024-04-24T23:15:35.964873+0000","last_unstale":"2024-04-24T23:15:42.295532+0000","last_undegraded":"2024-04-24T23:15:42.295532+0000","last_fullsized":"2024-04-24T23:15:42.295532+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:14:27.884944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.295549+0000","last_change":"2024-04-24T23:15:41.004002+0000","last_active":"2024-04-24T23:15:42.295549+0000","last_peered":"2024-04-24T23:15:42.295549+0000","last_clean":"2024-04-24T23:15:42.295549+0000","last_became_active":"2024-04-24T23:15:41.003335+0000","last_became_peered":"2024-04-24T23:15:41.003335+0000","last_unstale":"2024-04-24T23:15:42.295549+0000","last_undegraded":"2024-04-24T23:15:42.295549+0000","last_fullsized":"2024-04-24T23:15:42.295549+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:49:36.206903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1,3],"acting":[5,4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.193583+0000","last_change":"2024-04-24T23:15:37.002854+0000","last_active":"2024-04-24T23:15:43.193583+0000","last_peered":"2024-04-24T23:15:43.193583+0000","last_clean":"2024-04-24T23:15:43.193583+0000","last_became_active":"2024-04-24T23:15:37.002583+0000","last_became_peered":"2024-04-24T23:15:37.002583+0000","last_unstale":"2024-04-24T23:15:43.193583+0000","last_undegraded":"2024-04-24T23:15:43.193583+0000","last_fullsized":"2024-04-24T23:15:43.193583+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:27:04.251547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"23'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.050875+0000","last_change":"2024-04-24T23:15:35.948752+0000","last_active":"2024-04-24T23:15:42.050875+0000","last_peered":"2024-04-24T23:15:42.050875+0000","last_clean":"2024-04-24T23:15:42.050875+0000","last_became_active":"2024-04-24T23:15:35.948438+0000","last_became_peered":"2024-04-24T23:15:35.948438+0000","last_unstale":"2024-04-24T23:15:42.050875+0000","last_undegraded":"2024-04-24T23:15:42.050875+0000","last_fullsized":"2024-04-24T23:15:42.050875+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:45:56.006278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.190120+0000","last_change":"2024-04-24T23:15:41.017524+0000","last_active":"2024-04-24T23:15:43.190120+0000","last_peered":"2024-04-24T23:15:43.190120+0000","last_clean":"2024-04-24T23:15:43.190120+0000","last_became_active":"2024-04-24T23:15:41.017092+0000","last_became_peered":"2024-04-24T23:15:41.017092+0000","last_unstale":"2024-04-24T23:15:43.190120+0000","last_undegraded":"2024-04-24T23:15:43.190120+0000","last_fullsized":"2024-04-24T23:15:43.190120+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:04:53.235038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.107289+0000","last_change":"2024-04-24T23:15:36.990441+0000","last_active":"2024-04-24T23:15:42.107289+0000","last_peered":"2024-04-24T23:15:42.107289+0000","last_clean":"2024-04-24T23:15:42.107289+0000","last_became_active":"2024-04-24T23:15:36.989863+0000","last_became_peered":"2024-04-24T23:15:36.989863+0000","last_unstale":"2024-04-24T23:15:42.107289+0000","last_undegraded":"2024-04-24T23:15:42.107289+0000","last_fullsized":"2024-04-24T23:15:42.107289+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:50:07.975347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.f","version":"23'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.193960+0000","last_change":"2024-04-24T23:15:35.969317+0000","last_active":"2024-04-24T23:15:43.193960+0000","last_peered":"2024-04-24T23:15:43.193960+0000","last_clean":"2024-04-24T23:15:43.193960+0000","last_became_active":"2024-04-24T23:15:35.968994+0000","last_became_peered":"2024-04-24T23:15:35.968994+0000","last_unstale":"2024-04-24T23:15:43.193960+0000","last_undegraded":"2024-04-24T23:15:43.193960+0000","last_fullsized":"2024-04-24T23:15:43.193960+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:51:57.100772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.733165+0000","last_change":"2024-04-24T23:15:41.010103+0000","last_active":"2024-04-24T23:15:42.733165+0000","last_peered":"2024-04-24T23:15:42.733165+0000","last_clean":"2024-04-24T23:15:42.733165+0000","last_became_active":"2024-04-24T23:15:41.009328+0000","last_became_peered":"2024-04-24T23:15:41.009328+0000","last_unstale":"2024-04-24T23:15:42.733165+0000","last_undegraded":"2024-04-24T23:15:42.733165+0000","last_fullsized":"2024-04-24T23:15:42.733165+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:46:49.579501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,2],"acting":[1,5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.190520+0000","last_change":"2024-04-24T23:15:36.996765+0000","last_active":"2024-04-24T23:15:43.190520+0000","last_peered":"2024-04-24T23:15:43.190520+0000","last_clean":"2024-04-24T23:15:43.190520+0000","last_became_active":"2024-04-24T23:15:36.995801+0000","last_became_peered":"2024-04-24T23:15:36.995801+0000","last_unstale":"2024-04-24T23:15:43.190520+0000","last_undegraded":"2024-04-24T23:15:43.190520+0000","last_fullsized":"2024-04-24T23:15:43.190520+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:25:19.421667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"24'10","reported_seq":32,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:16:13.160400+0000","last_change":"2024-04-24T23:15:35.944023+0000","last_active":"2024-04-24T23:16:13.160400+0000","last_peered":"2024-04-24T23:16:13.160400+0000","last_clean":"2024-04-24T23:16:13.160400+0000","last_became_active":"2024-04-24T23:15:35.943710+0000","last_became_peered":"2024-04-24T23:15:35.943710+0000","last_unstale":"2024-04-24T23:16:13.160400+0000","last_undegraded":"2024-04-24T23:16:13.160400+0000","last_fullsized":"2024-04-24T23:16:13.160400+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:46:48.090958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.295767+0000","last_change":"2024-04-24T23:15:41.013027+0000","last_active":"2024-04-24T23:15:42.295767+0000","last_peered":"2024-04-24T23:15:42.295767+0000","last_clean":"2024-04-24T23:15:42.295767+0000","last_became_active":"2024-04-24T23:15:41.012587+0000","last_became_peered":"2024-04-24T23:15:41.012587+0000","last_unstale":"2024-04-24T23:15:42.295767+0000","last_undegraded":"2024-04-24T23:15:42.295767+0000","last_fullsized":"2024-04-24T23:15:42.295767+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:27:04.276246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1,6],"acting":[5,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.732647+0000","last_change":"2024-04-24T23:15:36.979702+0000","last_active":"2024-04-24T23:15:42.732647+0000","last_peered":"2024-04-24T23:15:42.732647+0000","last_clean":"2024-04-24T23:15:42.732647+0000","last_became_active":"2024-04-24T23:15:36.979356+0000","last_became_peered":"2024-04-24T23:15:36.979356+0000","last_unstale":"2024-04-24T23:15:42.732647+0000","last_undegraded":"2024-04-24T23:15:42.732647+0000","last_fullsized":"2024-04-24T23:15:42.732647+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:14:24.309192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.193698+0000","last_change":"2024-04-24T23:15:35.973026+0000","last_active":"2024-04-24T23:15:43.193698+0000","last_peered":"2024-04-24T23:15:43.193698+0000","last_clean":"2024-04-24T23:15:43.193698+0000","last_became_active":"2024-04-24T23:15:35.972655+0000","last_became_peered":"2024-04-24T23:15:35.972655+0000","last_unstale":"2024-04-24T23:15:43.193698+0000","last_undegraded":"2024-04-24T23:15:43.193698+0000","last_fullsized":"2024-04-24T23:15:43.193698+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:25:34.352908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.295901+0000","last_change":"2024-04-24T23:15:41.004221+0000","last_active":"2024-04-24T23:15:42.295901+0000","last_peered":"2024-04-24T23:15:42.295901+0000","last_clean":"2024-04-24T23:15:42.295901+0000","last_became_active":"2024-04-24T23:15:41.003790+0000","last_became_peered":"2024-04-24T23:15:41.003790+0000","last_unstale":"2024-04-24T23:15:42.295901+0000","last_undegraded":"2024-04-24T23:15:42.295901+0000","last_fullsized":"2024-04-24T23:15:42.295901+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:28:58.866024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6,4],"acting":[5,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.191414+0000","last_change":"2024-04-24T23:15:36.997389+0000","last_active":"2024-04-24T23:15:43.191414+0000","last_peered":"2024-04-24T23:15:43.191414+0000","last_clean":"2024-04-24T23:15:43.191414+0000","last_became_active":"2024-04-24T23:15:36.995923+0000","last_became_peered":"2024-04-24T23:15:36.995923+0000","last_unstale":"2024-04-24T23:15:43.191414+0000","last_undegraded":"2024-04-24T23:15:43.191414+0000","last_fullsized":"2024-04-24T23:15:43.191414+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:55:47.134320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"24'3","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:16:13.164411+0000","last_change":"2024-04-24T23:15:35.957715+0000","last_active":"2024-04-24T23:16:13.164411+0000","last_peered":"2024-04-24T23:16:13.164411+0000","last_clean":"2024-04-24T23:16:13.164411+0000","last_became_active":"2024-04-24T23:15:35.957088+0000","last_became_peered":"2024-04-24T23:15:35.957088+0000","last_unstale":"2024-04-24T23:16:13.164411+0000","last_undegraded":"2024-04-24T23:16:13.164411+0000","last_fullsized":"2024-04-24T23:16:13.164411+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:30:34.483669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.050833+0000","last_change":"2024-04-24T23:15:41.008165+0000","last_active":"2024-04-24T23:15:42.050833+0000","last_peered":"2024-04-24T23:15:42.050833+0000","last_clean":"2024-04-24T23:15:42.050833+0000","last_became_active":"2024-04-24T23:15:41.007555+0000","last_became_peered":"2024-04-24T23:15:41.007555+0000","last_unstale":"2024-04-24T23:15:42.050833+0000","last_undegraded":"2024-04-24T23:15:42.050833+0000","last_fullsized":"2024-04-24T23:15:42.050833+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:18:08.195747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0,7],"acting":[6,4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.050006+0000","last_change":"2024-04-24T23:15:36.991382+0000","last_active":"2024-04-24T23:15:42.050006+0000","last_peered":"2024-04-24T23:15:42.050006+0000","last_clean":"2024-04-24T23:15:42.050006+0000","last_became_active":"2024-04-24T23:15:36.991115+0000","last_became_peered":"2024-04-24T23:15:36.991115+0000","last_unstale":"2024-04-24T23:15:42.050006+0000","last_undegraded":"2024-04-24T23:15:42.050006+0000","last_fullsized":"2024-04-24T23:15:42.050006+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:21:51.863947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"24'4","reported_seq":26,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:16:13.164725+0000","last_change":"2024-04-24T23:15:35.964245+0000","last_active":"2024-04-24T23:16:13.164725+0000","last_peered":"2024-04-24T23:16:13.164725+0000","last_clean":"2024-04-24T23:16:13.164725+0000","last_became_active":"2024-04-24T23:15:35.963907+0000","last_became_peered":"2024-04-24T23:15:35.963907+0000","last_unstale":"2024-04-24T23:16:13.164725+0000","last_undegraded":"2024-04-24T23:16:13.164725+0000","last_fullsized":"2024-04-24T23:16:13.164725+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:15:00.913899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.192087+0000","last_change":"2024-04-24T23:15:41.021981+0000","last_active":"2024-04-24T23:15:43.192087+0000","last_peered":"2024-04-24T23:15:43.192087+0000","last_clean":"2024-04-24T23:15:43.192087+0000","last_became_active":"2024-04-24T23:15:41.019193+0000","last_became_peered":"2024-04-24T23:15:41.019193+0000","last_unstale":"2024-04-24T23:15:43.192087+0000","last_undegraded":"2024-04-24T23:15:43.192087+0000","last_fullsized":"2024-04-24T23:15:43.192087+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:04:39.828502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5,6],"acting":[3,0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.193203+0000","last_change":"2024-04-24T23:15:36.986833+0000","last_active":"2024-04-24T23:15:43.193203+0000","last_peered":"2024-04-24T23:15:43.193203+0000","last_clean":"2024-04-24T23:15:43.193203+0000","last_became_active":"2024-04-24T23:15:36.986484+0000","last_became_peered":"2024-04-24T23:15:36.986484+0000","last_unstale":"2024-04-24T23:15:43.193203+0000","last_undegraded":"2024-04-24T23:15:43.193203+0000","last_fullsized":"2024-04-24T23:15:43.193203+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:12:32.245267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.14","version":"24'12","reported_seq":34,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:16:13.164529+0000","last_change":"2024-04-24T23:15:35.957124+0000","last_active":"2024-04-24T23:16:13.164529+0000","last_peered":"2024-04-24T23:16:13.164529+0000","last_clean":"2024-04-24T23:16:13.164529+0000","last_became_active":"2024-04-24T23:15:35.951923+0000","last_became_peered":"2024-04-24T23:15:35.951923+0000","last_unstale":"2024-04-24T23:16:13.164529+0000","last_undegraded":"2024-04-24T23:16:13.164529+0000","last_fullsized":"2024-04-24T23:16:13.164529+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:44:16.820640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.049936+0000","last_change":"2024-04-24T23:15:41.015494+0000","last_active":"2024-04-24T23:15:42.049936+0000","last_peered":"2024-04-24T23:15:42.049936+0000","last_clean":"2024-04-24T23:15:42.049936+0000","last_became_active":"2024-04-24T23:15:41.015179+0000","last_became_peered":"2024-04-24T23:15:41.015179+0000","last_unstale":"2024-04-24T23:15:42.049936+0000","last_undegraded":"2024-04-24T23:15:42.049936+0000","last_fullsized":"2024-04-24T23:15:42.049936+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:18:32.208506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5,0],"acting":[4,3,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.190450+0000","last_change":"2024-04-24T23:15:36.999334+0000","last_active":"2024-04-24T23:15:43.190450+0000","last_peered":"2024-04-24T23:15:43.190450+0000","last_clean":"2024-04-24T23:15:43.190450+0000","last_became_active":"2024-04-24T23:15:36.995470+0000","last_became_peered":"2024-04-24T23:15:36.995470+0000","last_unstale":"2024-04-24T23:15:43.190450+0000","last_undegraded":"2024-04-24T23:15:43.190450+0000","last_fullsized":"2024-04-24T23:15:43.190450+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:19:57.973402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.15","version":"24'4","reported_seq":26,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.188040+0000","last_change":"2024-04-24T23:15:35.946041+0000","last_active":"2024-04-24T23:15:43.188040+0000","last_peered":"2024-04-24T23:15:43.188040+0000","last_clean":"2024-04-24T23:15:43.188040+0000","last_became_active":"2024-04-24T23:15:35.944805+0000","last_became_peered":"2024-04-24T23:15:35.944805+0000","last_unstale":"2024-04-24T23:15:43.188040+0000","last_undegraded":"2024-04-24T23:15:43.188040+0000","last_fullsized":"2024-04-24T23:15:43.188040+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:52:19.261306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.107105+0000","last_change":"2024-04-24T23:15:41.024392+0000","last_active":"2024-04-24T23:15:42.107105+0000","last_peered":"2024-04-24T23:15:42.107105+0000","last_clean":"2024-04-24T23:15:42.107105+0000","last_became_active":"2024-04-24T23:15:41.023951+0000","last_became_peered":"2024-04-24T23:15:41.023951+0000","last_unstale":"2024-04-24T23:15:42.107105+0000","last_undegraded":"2024-04-24T23:15:42.107105+0000","last_fullsized":"2024-04-24T23:15:42.107105+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:24:17.810117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1,7],"acting":[2,3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.295845+0000","last_change":"2024-04-24T23:15:36.992243+0000","last_active":"2024-04-24T23:15:42.295845+0000","last_peered":"2024-04-24T23:15:42.295845+0000","last_clean":"2024-04-24T23:15:42.295845+0000","last_became_active":"2024-04-24T23:15:36.992002+0000","last_became_peered":"2024-04-24T23:15:36.992002+0000","last_unstale":"2024-04-24T23:15:42.295845+0000","last_undegraded":"2024-04-24T23:15:42.295845+0000","last_fullsized":"2024-04-24T23:15:42.295845+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:52:15.911009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"24'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:16:13.163477+0000","last_change":"2024-04-24T23:15:35.967408+0000","last_active":"2024-04-24T23:16:13.163477+0000","last_peered":"2024-04-24T23:16:13.163477+0000","last_clean":"2024-04-24T23:16:13.163477+0000","last_became_active":"2024-04-24T23:15:35.966646+0000","last_became_peered":"2024-04-24T23:15:35.966646+0000","last_unstale":"2024-04-24T23:16:13.163477+0000","last_undegraded":"2024-04-24T23:16:13.163477+0000","last_fullsized":"2024-04-24T23:16:13.163477+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:07:07.931773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.187622+0000","last_change":"2024-04-24T23:15:41.021697+0000","last_active":"2024-04-24T23:15:43.187622+0000","last_peered":"2024-04-24T23:15:43.187622+0000","last_clean":"2024-04-24T23:15:43.187622+0000","last_became_active":"2024-04-24T23:15:41.021281+0000","last_became_peered":"2024-04-24T23:15:41.021281+0000","last_unstale":"2024-04-24T23:15:43.187622+0000","last_undegraded":"2024-04-24T23:15:43.187622+0000","last_fullsized":"2024-04-24T23:15:43.187622+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:59:37.306593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3,2],"acting":[7,1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.107218+0000","last_change":"2024-04-24T23:15:36.990988+0000","last_active":"2024-04-24T23:15:42.107218+0000","last_peered":"2024-04-24T23:15:42.107218+0000","last_clean":"2024-04-24T23:15:42.107218+0000","last_became_active":"2024-04-24T23:15:36.990711+0000","last_became_peered":"2024-04-24T23:15:36.990711+0000","last_unstale":"2024-04-24T23:15:42.107218+0000","last_undegraded":"2024-04-24T23:15:42.107218+0000","last_fullsized":"2024-04-24T23:15:42.107218+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:33:20.663776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.17","version":"24'3","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:16:13.163955+0000","last_change":"2024-04-24T23:15:35.957838+0000","last_active":"2024-04-24T23:16:13.163955+0000","last_peered":"2024-04-24T23:16:13.163955+0000","last_clean":"2024-04-24T23:16:13.163955+0000","last_became_active":"2024-04-24T23:15:35.956870+0000","last_became_peered":"2024-04-24T23:15:35.956870+0000","last_unstale":"2024-04-24T23:16:13.163955+0000","last_undegraded":"2024-04-24T23:16:13.163955+0000","last_fullsized":"2024-04-24T23:16:13.163955+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:56:03.889693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.049546+0000","last_change":"2024-04-24T23:15:41.006098+0000","last_active":"2024-04-24T23:15:42.049546+0000","last_peered":"2024-04-24T23:15:42.049546+0000","last_clean":"2024-04-24T23:15:42.049546+0000","last_became_active":"2024-04-24T23:15:41.005578+0000","last_became_peered":"2024-04-24T23:15:41.005578+0000","last_unstale":"2024-04-24T23:15:42.049546+0000","last_undegraded":"2024-04-24T23:15:42.049546+0000","last_fullsized":"2024-04-24T23:15:42.049546+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:21:12.802112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3,7],"acting":[4,0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.049896+0000","last_change":"2024-04-24T23:15:41.022122+0000","last_active":"2024-04-24T23:15:42.049896+0000","last_peered":"2024-04-24T23:15:42.049896+0000","last_clean":"2024-04-24T23:15:42.049896+0000","last_became_active":"2024-04-24T23:15:41.021698+0000","last_became_peered":"2024-04-24T23:15:41.021698+0000","last_unstale":"2024-04-24T23:15:42.049896+0000","last_undegraded":"2024-04-24T23:15:42.049896+0000","last_fullsized":"2024-04-24T23:15:42.049896+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:13:56.730050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5,3],"acting":[4,1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.049600+0000","last_change":"2024-04-24T23:15:36.979816+0000","last_active":"2024-04-24T23:15:42.049600+0000","last_peered":"2024-04-24T23:15:42.049600+0000","last_clean":"2024-04-24T23:15:42.049600+0000","last_became_active":"2024-04-24T23:15:36.979553+0000","last_became_peered":"2024-04-24T23:15:36.979553+0000","last_unstale":"2024-04-24T23:15:42.049600+0000","last_undegraded":"2024-04-24T23:15:42.049600+0000","last_fullsized":"2024-04-24T23:15:42.049600+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:55:47.134320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.18","version":"24'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.190707+0000","last_change":"2024-04-24T23:15:35.964744+0000","last_active":"2024-04-24T23:15:43.190707+0000","last_peered":"2024-04-24T23:15:43.190707+0000","last_clean":"2024-04-24T23:15:43.190707+0000","last_became_active":"2024-04-24T23:15:35.964434+0000","last_became_peered":"2024-04-24T23:15:35.964434+0000","last_unstale":"2024-04-24T23:15:43.190707+0000","last_undegraded":"2024-04-24T23:15:43.190707+0000","last_fullsized":"2024-04-24T23:15:43.190707+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:26:32.587197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.191463+0000","last_change":"2024-04-24T23:15:41.022032+0000","last_active":"2024-04-24T23:15:43.191463+0000","last_peered":"2024-04-24T23:15:43.191463+0000","last_clean":"2024-04-24T23:15:43.191463+0000","last_became_active":"2024-04-24T23:15:41.021616+0000","last_became_peered":"2024-04-24T23:15:41.021616+0000","last_unstale":"2024-04-24T23:15:43.191463+0000","last_undegraded":"2024-04-24T23:15:43.191463+0000","last_fullsized":"2024-04-24T23:15:43.191463+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:08:59.242392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5,1],"acting":[3,2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.295164+0000","last_change":"2024-04-24T23:15:36.967198+0000","last_active":"2024-04-24T23:15:42.295164+0000","last_peered":"2024-04-24T23:15:42.295164+0000","last_clean":"2024-04-24T23:15:42.295164+0000","last_became_active":"2024-04-24T23:15:36.966910+0000","last_became_peered":"2024-04-24T23:15:36.966910+0000","last_unstale":"2024-04-24T23:15:42.295164+0000","last_undegraded":"2024-04-24T23:15:42.295164+0000","last_fullsized":"2024-04-24T23:15:42.295164+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:51:19.788437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.049504+0000","last_change":"2024-04-24T23:15:35.967169+0000","last_active":"2024-04-24T23:15:42.049504+0000","last_peered":"2024-04-24T23:15:42.049504+0000","last_clean":"2024-04-24T23:15:42.049504+0000","last_became_active":"2024-04-24T23:15:35.966939+0000","last_became_peered":"2024-04-24T23:15:35.966939+0000","last_unstale":"2024-04-24T23:15:42.049504+0000","last_undegraded":"2024-04-24T23:15:42.049504+0000","last_fullsized":"2024-04-24T23:15:42.049504+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:38:55.460569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"23'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.191308+0000","last_change":"2024-04-24T23:15:35.971236+0000","last_active":"2024-04-24T23:15:43.191308+0000","last_peered":"2024-04-24T23:15:43.191308+0000","last_clean":"2024-04-24T23:15:43.191308+0000","last_became_active":"2024-04-24T23:15:35.970908+0000","last_became_peered":"2024-04-24T23:15:35.970908+0000","last_unstale":"2024-04-24T23:15:43.191308+0000","last_undegraded":"2024-04-24T23:15:43.191308+0000","last_fullsized":"2024-04-24T23:15:43.191308+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:14:15.548024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.050320+0000","last_change":"2024-04-24T23:15:36.988156+0000","last_active":"2024-04-24T23:15:42.050320+0000","last_peered":"2024-04-24T23:15:42.050320+0000","last_clean":"2024-04-24T23:15:42.050320+0000","last_became_active":"2024-04-24T23:15:36.987896+0000","last_became_peered":"2024-04-24T23:15:36.987896+0000","last_unstale":"2024-04-24T23:15:42.050320+0000","last_undegraded":"2024-04-24T23:15:42.050320+0000","last_fullsized":"2024-04-24T23:15:42.050320+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:22:06.869644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.050441+0000","last_change":"2024-04-24T23:15:40.997559+0000","last_active":"2024-04-24T23:15:42.050441+0000","last_peered":"2024-04-24T23:15:42.050441+0000","last_clean":"2024-04-24T23:15:42.050441+0000","last_became_active":"2024-04-24T23:15:40.997102+0000","last_became_peered":"2024-04-24T23:15:40.997102+0000","last_unstale":"2024-04-24T23:15:42.050441+0000","last_undegraded":"2024-04-24T23:15:42.050441+0000","last_fullsized":"2024-04-24T23:15:42.050441+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:25:02.046761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,7],"acting":[4,6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"24'5","reported_seq":27,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:16:13.163435+0000","last_change":"2024-04-24T23:15:35.944842+0000","last_active":"2024-04-24T23:16:13.163435+0000","last_peered":"2024-04-24T23:16:13.163435+0000","last_clean":"2024-04-24T23:16:13.163435+0000","last_became_active":"2024-04-24T23:15:35.944563+0000","last_became_peered":"2024-04-24T23:15:35.944563+0000","last_unstale":"2024-04-24T23:16:13.163435+0000","last_undegraded":"2024-04-24T23:16:13.163435+0000","last_fullsized":"2024-04-24T23:16:13.163435+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:40:58.933384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.732992+0000","last_change":"2024-04-24T23:15:36.981103+0000","last_active":"2024-04-24T23:15:42.732992+0000","last_peered":"2024-04-24T23:15:42.732992+0000","last_clean":"2024-04-24T23:15:42.732992+0000","last_became_active":"2024-04-24T23:15:36.979637+0000","last_became_peered":"2024-04-24T23:15:36.979637+0000","last_unstale":"2024-04-24T23:15:42.732992+0000","last_undegraded":"2024-04-24T23:15:42.732992+0000","last_fullsized":"2024-04-24T23:15:42.732992+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:09:39.750272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.188153+0000","last_change":"2024-04-24T23:15:41.017087+0000","last_active":"2024-04-24T23:15:43.188153+0000","last_peered":"2024-04-24T23:15:43.188153+0000","last_clean":"2024-04-24T23:15:43.188153+0000","last_became_active":"2024-04-24T23:15:41.016696+0000","last_became_peered":"2024-04-24T23:15:41.016696+0000","last_unstale":"2024-04-24T23:15:43.188153+0000","last_undegraded":"2024-04-24T23:15:43.188153+0000","last_fullsized":"2024-04-24T23:15:43.188153+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:05:57.823447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1,0],"acting":[7,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.106609+0000","last_change":"2024-04-24T23:15:36.970211+0000","last_active":"2024-04-24T23:15:42.106609+0000","last_peered":"2024-04-24T23:15:42.106609+0000","last_clean":"2024-04-24T23:15:42.106609+0000","last_became_active":"2024-04-24T23:15:36.969090+0000","last_became_peered":"2024-04-24T23:15:36.969090+0000","last_unstale":"2024-04-24T23:15:42.106609+0000","last_undegraded":"2024-04-24T23:15:42.106609+0000","last_fullsized":"2024-04-24T23:15:42.106609+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:16:00.231012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.187632+0000","last_change":"2024-04-24T23:15:36.982528+0000","last_active":"2024-04-24T23:15:43.187632+0000","last_peered":"2024-04-24T23:15:43.187632+0000","last_clean":"2024-04-24T23:15:43.187632+0000","last_became_active":"2024-04-24T23:15:36.982299+0000","last_became_peered":"2024-04-24T23:15:36.982299+0000","last_unstale":"2024-04-24T23:15:43.187632+0000","last_undegraded":"2024-04-24T23:15:43.187632+0000","last_fullsized":"2024-04-24T23:15:43.187632+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:02:17.732324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.191005+0000","last_change":"2024-04-24T23:15:36.989899+0000","last_active":"2024-04-24T23:15:43.191005+0000","last_peered":"2024-04-24T23:15:43.191005+0000","last_clean":"2024-04-24T23:15:43.191005+0000","last_became_active":"2024-04-24T23:15:36.989557+0000","last_became_peered":"2024-04-24T23:15:36.989557+0000","last_unstale":"2024-04-24T23:15:43.191005+0000","last_undegraded":"2024-04-24T23:15:43.191005+0000","last_fullsized":"2024-04-24T23:15:43.191005+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:15:13.513341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.295325+0000","last_change":"2024-04-24T23:15:36.969628+0000","last_active":"2024-04-24T23:15:42.295325+0000","last_peered":"2024-04-24T23:15:42.295325+0000","last_clean":"2024-04-24T23:15:42.295325+0000","last_became_active":"2024-04-24T23:15:36.969327+0000","last_became_peered":"2024-04-24T23:15:36.969327+0000","last_unstale":"2024-04-24T23:15:42.295325+0000","last_undegraded":"2024-04-24T23:15:42.295325+0000","last_fullsized":"2024-04-24T23:15:42.295325+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:08:47.021999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.050888+0000","last_change":"2024-04-24T23:15:36.986343+0000","last_active":"2024-04-24T23:15:42.050888+0000","last_peered":"2024-04-24T23:15:42.050888+0000","last_clean":"2024-04-24T23:15:42.050888+0000","last_became_active":"2024-04-24T23:15:36.986026+0000","last_became_peered":"2024-04-24T23:15:36.986026+0000","last_unstale":"2024-04-24T23:15:42.050888+0000","last_undegraded":"2024-04-24T23:15:42.050888+0000","last_fullsized":"2024-04-24T23:15:42.050888+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:10:00.347826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.049809+0000","last_change":"2024-04-24T23:15:36.968538+0000","last_active":"2024-04-24T23:15:42.049809+0000","last_peered":"2024-04-24T23:15:42.049809+0000","last_clean":"2024-04-24T23:15:42.049809+0000","last_became_active":"2024-04-24T23:15:36.968263+0000","last_became_peered":"2024-04-24T23:15:36.968263+0000","last_unstale":"2024-04-24T23:15:42.049809+0000","last_undegraded":"2024-04-24T23:15:42.049809+0000","last_fullsized":"2024-04-24T23:15:42.049809+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:07:43.731738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.107186+0000","last_change":"2024-04-24T23:15:36.991008+0000","last_active":"2024-04-24T23:15:42.107186+0000","last_peered":"2024-04-24T23:15:42.107186+0000","last_clean":"2024-04-24T23:15:42.107186+0000","last_became_active":"2024-04-24T23:15:36.990764+0000","last_became_peered":"2024-04-24T23:15:36.990764+0000","last_unstale":"2024-04-24T23:15:42.107186+0000","last_undegraded":"2024-04-24T23:15:42.107186+0000","last_fullsized":"2024-04-24T23:15:42.107186+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:50:21.865233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.295685+0000","last_change":"2024-04-24T23:15:36.993550+0000","last_active":"2024-04-24T23:15:42.295685+0000","last_peered":"2024-04-24T23:15:42.295685+0000","last_clean":"2024-04-24T23:15:42.295685+0000","last_became_active":"2024-04-24T23:15:36.993310+0000","last_became_peered":"2024-04-24T23:15:36.993310+0000","last_unstale":"2024-04-24T23:15:42.295685+0000","last_undegraded":"2024-04-24T23:15:42.295685+0000","last_fullsized":"2024-04-24T23:15:42.295685+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:21:20.617940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.190455+0000","last_change":"2024-04-24T23:15:36.987810+0000","last_active":"2024-04-24T23:15:43.190455+0000","last_peered":"2024-04-24T23:15:43.190455+0000","last_clean":"2024-04-24T23:15:43.190455+0000","last_became_active":"2024-04-24T23:15:36.987540+0000","last_became_peered":"2024-04-24T23:15:36.987540+0000","last_unstale":"2024-04-24T23:15:43.190455+0000","last_undegraded":"2024-04-24T23:15:43.190455+0000","last_fullsized":"2024-04-24T23:15:43.190455+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:49:56.176021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.295070+0000","last_change":"2024-04-24T23:15:36.981187+0000","last_active":"2024-04-24T23:15:42.295070+0000","last_peered":"2024-04-24T23:15:42.295070+0000","last_clean":"2024-04-24T23:15:42.295070+0000","last_became_active":"2024-04-24T23:15:36.980838+0000","last_became_peered":"2024-04-24T23:15:36.980838+0000","last_unstale":"2024-04-24T23:15:42.295070+0000","last_undegraded":"2024-04-24T23:15:42.295070+0000","last_fullsized":"2024-04-24T23:15:42.295070+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:57:41.178367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.732913+0000","last_change":"2024-04-24T23:15:36.999238+0000","last_active":"2024-04-24T23:15:42.732913+0000","last_peered":"2024-04-24T23:15:42.732913+0000","last_clean":"2024-04-24T23:15:42.732913+0000","last_became_active":"2024-04-24T23:15:36.996245+0000","last_became_peered":"2024-04-24T23:15:36.996245+0000","last_unstale":"2024-04-24T23:15:42.732913+0000","last_undegraded":"2024-04-24T23:15:42.732913+0000","last_fullsized":"2024-04-24T23:15:42.732913+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:21:20.443640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.050214+0000","last_change":"2024-04-24T23:15:36.981219+0000","last_active":"2024-04-24T23:15:42.050214+0000","last_peered":"2024-04-24T23:15:42.050214+0000","last_clean":"2024-04-24T23:15:42.050214+0000","last_became_active":"2024-04-24T23:15:36.980911+0000","last_became_peered":"2024-04-24T23:15:36.980911+0000","last_unstale":"2024-04-24T23:15:42.050214+0000","last_undegraded":"2024-04-24T23:15:42.050214+0000","last_fullsized":"2024-04-24T23:15:42.050214+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:19:57.973402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.193222+0000","last_change":"2024-04-24T23:15:36.991418+0000","last_active":"2024-04-24T23:15:43.193222+0000","last_peered":"2024-04-24T23:15:43.193222+0000","last_clean":"2024-04-24T23:15:43.193222+0000","last_became_active":"2024-04-24T23:15:36.989792+0000","last_became_peered":"2024-04-24T23:15:36.989792+0000","last_unstale":"2024-04-24T23:15:43.193222+0000","last_undegraded":"2024-04-24T23:15:43.193222+0000","last_fullsized":"2024-04-24T23:15:43.193222+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:02:53.601311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.192125+0000","last_change":"2024-04-24T23:15:36.987810+0000","last_active":"2024-04-24T23:15:43.192125+0000","last_peered":"2024-04-24T23:15:43.192125+0000","last_clean":"2024-04-24T23:15:43.192125+0000","last_became_active":"2024-04-24T23:15:36.987494+0000","last_became_peered":"2024-04-24T23:15:36.987494+0000","last_unstale":"2024-04-24T23:15:43.192125+0000","last_undegraded":"2024-04-24T23:15:43.192125+0000","last_fullsized":"2024-04-24T23:15:43.192125+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:43:21.141172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.188217+0000","last_change":"2024-04-24T23:15:36.986576+0000","last_active":"2024-04-24T23:15:43.188217+0000","last_peered":"2024-04-24T23:15:43.188217+0000","last_clean":"2024-04-24T23:15:43.188217+0000","last_became_active":"2024-04-24T23:15:36.986272+0000","last_became_peered":"2024-04-24T23:15:36.986272+0000","last_unstale":"2024-04-24T23:15:43.188217+0000","last_undegraded":"2024-04-24T23:15:43.188217+0000","last_fullsized":"2024-04-24T23:15:43.188217+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:55:08.894945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.192001+0000","last_change":"2024-04-24T23:15:36.998812+0000","last_active":"2024-04-24T23:15:43.192001+0000","last_peered":"2024-04-24T23:15:43.192001+0000","last_clean":"2024-04-24T23:15:43.192001+0000","last_became_active":"2024-04-24T23:15:36.998456+0000","last_became_peered":"2024-04-24T23:15:36.998456+0000","last_unstale":"2024-04-24T23:15:43.192001+0000","last_undegraded":"2024-04-24T23:15:43.192001+0000","last_fullsized":"2024-04-24T23:15:43.192001+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:33:11.424218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.190262+0000","last_change":"2024-04-24T23:15:36.996521+0000","last_active":"2024-04-24T23:15:43.190262+0000","last_peered":"2024-04-24T23:15:43.190262+0000","last_clean":"2024-04-24T23:15:43.190262+0000","last_became_active":"2024-04-24T23:15:36.995507+0000","last_became_peered":"2024-04-24T23:15:36.995507+0000","last_unstale":"2024-04-24T23:15:43.190262+0000","last_undegraded":"2024-04-24T23:15:43.190262+0000","last_fullsized":"2024-04-24T23:15:43.190262+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:58:50.153471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.294996+0000","last_change":"2024-04-24T23:15:36.984994+0000","last_active":"2024-04-24T23:15:42.294996+0000","last_peered":"2024-04-24T23:15:42.294996+0000","last_clean":"2024-04-24T23:15:42.294996+0000","last_became_active":"2024-04-24T23:15:36.984741+0000","last_became_peered":"2024-04-24T23:15:36.984741+0000","last_unstale":"2024-04-24T23:15:42.294996+0000","last_undegraded":"2024-04-24T23:15:42.294996+0000","last_fullsized":"2024-04-24T23:15:42.294996+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:45:23.429836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.049917+0000","last_change":"2024-04-24T23:15:36.966165+0000","last_active":"2024-04-24T23:15:42.049917+0000","last_peered":"2024-04-24T23:15:42.049917+0000","last_clean":"2024-04-24T23:15:42.049917+0000","last_became_active":"2024-04-24T23:15:36.965877+0000","last_became_peered":"2024-04-24T23:15:36.965877+0000","last_unstale":"2024-04-24T23:15:42.049917+0000","last_undegraded":"2024-04-24T23:15:42.049917+0000","last_fullsized":"2024-04-24T23:15:42.049917+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:30:12.390978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.050537+0000","last_change":"2024-04-24T23:15:36.990667+0000","last_active":"2024-04-24T23:15:42.050537+0000","last_peered":"2024-04-24T23:15:42.050537+0000","last_clean":"2024-04-24T23:15:42.050537+0000","last_became_active":"2024-04-24T23:15:36.990383+0000","last_became_peered":"2024-04-24T23:15:36.990383+0000","last_unstale":"2024-04-24T23:15:42.050537+0000","last_undegraded":"2024-04-24T23:15:42.050537+0000","last_fullsized":"2024-04-24T23:15:42.050537+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:36:43.850046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.193024+0000","last_change":"2024-04-24T23:15:36.991686+0000","last_active":"2024-04-24T23:15:43.193024+0000","last_peered":"2024-04-24T23:15:43.193024+0000","last_clean":"2024-04-24T23:15:43.193024+0000","last_became_active":"2024-04-24T23:15:36.990155+0000","last_became_peered":"2024-04-24T23:15:36.990155+0000","last_unstale":"2024-04-24T23:15:43.193024+0000","last_undegraded":"2024-04-24T23:15:43.193024+0000","last_fullsized":"2024-04-24T23:15:43.193024+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:00:52.424019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.192909+0000","last_change":"2024-04-24T23:15:36.992958+0000","last_active":"2024-04-24T23:15:43.192909+0000","last_peered":"2024-04-24T23:15:43.192909+0000","last_clean":"2024-04-24T23:15:43.192909+0000","last_became_active":"2024-04-24T23:15:36.992608+0000","last_became_peered":"2024-04-24T23:15:36.992608+0000","last_unstale":"2024-04-24T23:15:43.192909+0000","last_undegraded":"2024-04-24T23:15:43.192909+0000","last_fullsized":"2024-04-24T23:15:43.192909+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:15:30.416763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.192849+0000","last_change":"2024-04-24T23:15:36.992943+0000","last_active":"2024-04-24T23:15:43.192849+0000","last_peered":"2024-04-24T23:15:43.192849+0000","last_clean":"2024-04-24T23:15:43.192849+0000","last_became_active":"2024-04-24T23:15:36.992567+0000","last_became_peered":"2024-04-24T23:15:36.992567+0000","last_unstale":"2024-04-24T23:15:43.192849+0000","last_undegraded":"2024-04-24T23:15:43.192849+0000","last_fullsized":"2024-04-24T23:15:43.192849+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:50:43.209232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.734006+0000","last_change":"2024-04-24T23:15:36.999697+0000","last_active":"2024-04-24T23:15:42.734006+0000","last_peered":"2024-04-24T23:15:42.734006+0000","last_clean":"2024-04-24T23:15:42.734006+0000","last_became_active":"2024-04-24T23:15:36.999413+0000","last_became_peered":"2024-04-24T23:15:36.999413+0000","last_unstale":"2024-04-24T23:15:42.734006+0000","last_undegraded":"2024-04-24T23:15:42.734006+0000","last_fullsized":"2024-04-24T23:15:42.734006+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:25:42.856471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.295006+0000","last_change":"2024-04-24T23:15:36.986742+0000","last_active":"2024-04-24T23:15:42.295006+0000","last_peered":"2024-04-24T23:15:42.295006+0000","last_clean":"2024-04-24T23:15:42.295006+0000","last_became_active":"2024-04-24T23:15:36.986013+0000","last_became_peered":"2024-04-24T23:15:36.986013+0000","last_unstale":"2024-04-24T23:15:42.295006+0000","last_undegraded":"2024-04-24T23:15:42.295006+0000","last_fullsized":"2024-04-24T23:15:42.295006+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:16:15.203151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.193480+0000","last_change":"2024-04-24T23:15:36.984336+0000","last_active":"2024-04-24T23:15:43.193480+0000","last_peered":"2024-04-24T23:15:43.193480+0000","last_clean":"2024-04-24T23:15:43.193480+0000","last_became_active":"2024-04-24T23:15:36.983966+0000","last_became_peered":"2024-04-24T23:15:36.983966+0000","last_unstale":"2024-04-24T23:15:43.193480+0000","last_undegraded":"2024-04-24T23:15:43.193480+0000","last_fullsized":"2024-04-24T23:15:43.193480+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:54:16.319608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.106762+0000","last_change":"2024-04-24T23:15:36.979079+0000","last_active":"2024-04-24T23:15:42.106762+0000","last_peered":"2024-04-24T23:15:42.106762+0000","last_clean":"2024-04-24T23:15:42.106762+0000","last_became_active":"2024-04-24T23:15:36.978706+0000","last_became_peered":"2024-04-24T23:15:36.978706+0000","last_unstale":"2024-04-24T23:15:42.106762+0000","last_undegraded":"2024-04-24T23:15:42.106762+0000","last_fullsized":"2024-04-24T23:15:42.106762+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:53:39.888150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.193929+0000","last_change":"2024-04-24T23:15:36.986008+0000","last_active":"2024-04-24T23:15:43.193929+0000","last_peered":"2024-04-24T23:15:43.193929+0000","last_clean":"2024-04-24T23:15:43.193929+0000","last_became_active":"2024-04-24T23:15:36.985552+0000","last_became_peered":"2024-04-24T23:15:36.985552+0000","last_unstale":"2024-04-24T23:15:43.193929+0000","last_undegraded":"2024-04-24T23:15:43.193929+0000","last_fullsized":"2024-04-24T23:15:43.193929+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:36:50.827503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.106908+0000","last_change":"2024-04-24T23:15:36.978673+0000","last_active":"2024-04-24T23:15:42.106908+0000","last_peered":"2024-04-24T23:15:42.106908+0000","last_clean":"2024-04-24T23:15:42.106908+0000","last_became_active":"2024-04-24T23:15:36.972744+0000","last_became_peered":"2024-04-24T23:15:36.972744+0000","last_unstale":"2024-04-24T23:15:42.106908+0000","last_undegraded":"2024-04-24T23:15:42.106908+0000","last_fullsized":"2024-04-24T23:15:42.106908+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:28:57.028825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.295395+0000","last_change":"2024-04-24T23:15:36.970238+0000","last_active":"2024-04-24T23:15:42.295395+0000","last_peered":"2024-04-24T23:15:42.295395+0000","last_clean":"2024-04-24T23:15:42.295395+0000","last_became_active":"2024-04-24T23:15:36.969751+0000","last_became_peered":"2024-04-24T23:15:36.969751+0000","last_unstale":"2024-04-24T23:15:42.295395+0000","last_undegraded":"2024-04-24T23:15:42.295395+0000","last_fullsized":"2024-04-24T23:15:42.295395+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:19:56.456801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:42.295592+0000","last_change":"2024-04-24T23:15:36.968825+0000","last_active":"2024-04-24T23:15:42.295592+0000","last_peered":"2024-04-24T23:15:42.295592+0000","last_clean":"2024-04-24T23:15:42.295592+0000","last_became_active":"2024-04-24T23:15:36.968543+0000","last_became_peered":"2024-04-24T23:15:36.968543+0000","last_unstale":"2024-04-24T23:15:42.295592+0000","last_undegraded":"2024-04-24T23:15:42.295592+0000","last_fullsized":"2024-04-24T23:15:42.295592+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:36:48.795062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-24T23:15:43.193226+0000","last_change":"2024-04-24T23:15:36.992800+0000","last_active":"2024-04-24T23:15:43.193226+0000","last_peered":"2024-04-24T23:15:43.193226+0000","last_clean":"2024-04-24T23:15:43.193226+0000","last_became_active":"2024-04-24T23:15:36.992476+0000","last_became_peered":"2024-04-24T23:15:36.992476+0000","last_unstale":"2024-04-24T23:15:43.193226+0000","last_undegraded":"2024-04-24T23:15:43.193226+0000","last_fullsized":"2024-04-24T23:15:43.193226+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:01:21.103559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":6046,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":88,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":26,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":12092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77,"ondisk_log_size":77,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574862,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":266240,"data_stored":100970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574862,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27272,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":262144,"data_stored":100886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574862,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27860,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343980,"statfs":{"total":96636764160,"available":96608235520,"internally_reserved":0,"allocated":864256,"data_stored":691346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574863,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":284,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":290816,"data_stored":105562,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574863,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27272,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":262144,"data_stored":100868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574862,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":266240,"data_stored":104106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574861,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27852,"kb_used_data":832,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343988,"statfs":{"total":96636764160,"available":96608243712,"internally_reserved":0,"allocated":851968,"data_stored":690702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574862,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":266240,"data_stored":100410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":5298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":714,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T23:16:17.824 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-24T23:16:17.824 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-24T23:16:17.824 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-04-24T23:16:17.825 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-04-24T23:16:18.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:18.055+0000 7f0cb2a00640 1 Processor -- start 2024-04-24T23:16:18.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:18.056+0000 7f0cb2a00640 1 -- start start 2024-04-24T23:16:18.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:18.057+0000 7f0cb2a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f0cac154b00 0x7f0cac0c6230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:18.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:18.057+0000 7f0cb2a00640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f0cac067940 0x7f0cac0cbad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:18.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:18.057+0000 7f0cb2a00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0cac06bfd0 0x7f0cac06c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:18.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:18.057+0000 7f0cb2a00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f0cac15bbe0 con 0x7f0cac154b00 2024-04-24T23:16:18.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:18.057+0000 7f0cb2a00640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f0cac0c5da0 con 0x7f0cac067940 2024-04-24T23:16:18.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:18.057+0000 7f0cb2a00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f0cac0c54b0 con 0x7f0cac06bfd0 2024-04-24T23:16:18.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:18.057+0000 7f0cabe00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f0cac154b00 0x7f0cac0c6230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:18.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:18.058+0000 7f0cabe00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f0cac154b00 0x7f0cac0c6230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:43516/0 (socket says 172.21.15.6:43516) 2024-04-24T23:16:18.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:18.058+0000 7f0cabe00640 1 -- 172.21.15.6:0/3493486562 learned_addr learned my addr 172.21.15.6:0/3493486562 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:16:18.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:18.058+0000 7f0ca3e00640 1 --2- 172.21.15.6:0/3493486562 >> v2:172.21.15.190:3300/0 conn(0x7f0cac067940 0x7f0cac0cbad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:18.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:18.058+0000 7f0cb0a00640 1 --2- 172.21.15.6:0/3493486562 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0cac06bfd0 0x7f0cac06c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:18.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:18.058+0000 7f0cabe00640 1 -- 172.21.15.6:0/3493486562 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0cac06bfd0 msgr2=0x7f0cac06c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:16:18.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:18.058+0000 7f0cabe00640 1 --2- 172.21.15.6:0/3493486562 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0cac06bfd0 0x7f0cac06c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:18.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:18.058+0000 7f0cabe00640 1 -- 172.21.15.6:0/3493486562 >> v2:172.21.15.190:3300/0 conn(0x7f0cac067940 msgr2=0x7f0cac0cbad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:16:18.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:18.058+0000 7f0cabe00640 1 --2- 172.21.15.6:0/3493486562 >> v2:172.21.15.190:3300/0 conn(0x7f0cac067940 0x7f0cac0cbad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:18.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:18.058+0000 7f0cabe00640 1 -- 172.21.15.6:0/3493486562 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0cac15b640 con 0x7f0cac154b00 2024-04-24T23:16:18.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:18.058+0000 7f0ca3e00640 1 --2- 172.21.15.6:0/3493486562 >> v2:172.21.15.190:3300/0 conn(0x7f0cac067940 0x7f0cac0cbad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:16:18.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:18.059+0000 7f0cabe00640 1 --2- 172.21.15.6:0/3493486562 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0cac154b00 0x7f0cac0c6230 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f0cac0c7760 tx=0x7f0c9c02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dfd5c424172cd624 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:18.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:18.059+0000 7f0cab400640 1 -- 172.21.15.6:0/3493486562 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c9c037070 con 0x7f0cac154b00 2024-04-24T23:16:18.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:18.060+0000 7f0cab400640 1 -- 172.21.15.6:0/3493486562 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c9c032440 con 0x7f0cac154b00 2024-04-24T23:16:18.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:18.060+0000 7f0cab400640 1 -- 172.21.15.6:0/3493486562 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c9c032810 con 0x7f0cac154b00 2024-04-24T23:16:18.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:18.060+0000 7f0cb2a00640 1 -- 172.21.15.6:0/3493486562 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0cac154b00 msgr2=0x7f0cac0c6230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:18.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:18.060+0000 7f0cb2a00640 1 --2- 172.21.15.6:0/3493486562 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0cac154b00 0x7f0cac0c6230 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f0cac0c7760 tx=0x7f0c9c02a270 comp rx=0 tx=0).stop 2024-04-24T23:16:18.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:18.060+0000 7f0cb2a00640 1 -- 172.21.15.6:0/3493486562 shutdown_connections 2024-04-24T23:16:18.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:18.060+0000 7f0cb2a00640 1 --2- 172.21.15.6:0/3493486562 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0cac06bfd0 0x7f0cac06c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:18.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:18.060+0000 7f0cb2a00640 1 --2- 172.21.15.6:0/3493486562 >> v2:172.21.15.190:3300/0 conn(0x7f0cac067940 0x7f0cac0cbad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:18.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:18.060+0000 7f0cb2a00640 1 --2- 172.21.15.6:0/3493486562 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0cac154b00 0x7f0cac0c6230 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:18.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:18.060+0000 7f0cb2a00640 1 -- 172.21.15.6:0/3493486562 >> 172.21.15.6:0/3493486562 conn(0x7f0cac072810 msgr2=0x7f0cac066ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:18.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:18.060+0000 7f0cb2a00640 1 -- 172.21.15.6:0/3493486562 shutdown_connections 2024-04-24T23:16:18.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:18.060+0000 7f0cb2a00640 1 -- 172.21.15.6:0/3493486562 wait complete. 2024-04-24T23:16:18.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:18.063+0000 7f0cb2a00640 1 Processor -- start 2024-04-24T23:16:18.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:18.063+0000 7f0cb2a00640 1 -- start start 2024-04-24T23:16:18.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:18.064+0000 7f0cb2a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0cac067940 0x7f0cac1e70f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:18.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:18.064+0000 7f0cb2a00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0cac06bfd0 0x7f0cac1e55e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:18.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:18.064+0000 7f0cb2a00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0cac154b00 0x7f0cac144ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:18.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:18.064+0000 7f0cb2a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f0cac0c5a30 con 0x7f0cac067940 2024-04-24T23:16:18.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:18.064+0000 7f0cb2a00640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f0cac0c5750 con 0x7f0cac154b00 2024-04-24T23:16:18.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:18.064+0000 7f0cb2a00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f0cac06d360 con 0x7f0cac06bfd0 2024-04-24T23:16:18.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:18.065+0000 7f0cabe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0cac067940 0x7f0cac1e70f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:18.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:18.065+0000 7f0cabe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0cac067940 0x7f0cac1e70f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:43522/0 (socket says 172.21.15.6:43522) 2024-04-24T23:16:18.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:18.065+0000 7f0cabe00640 1 -- 172.21.15.6:0/4144534117 learned_addr learned my addr 172.21.15.6:0/4144534117 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:16:18.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:18.065+0000 7f0ca3e00640 1 --2- 172.21.15.6:0/4144534117 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0cac06bfd0 0x7f0cac1e55e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:18.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:18.065+0000 7f0cb0a00640 1 --2- 172.21.15.6:0/4144534117 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0cac154b00 0x7f0cac144ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:18.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:18.065+0000 7f0cabe00640 1 -- 172.21.15.6:0/4144534117 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0cac06bfd0 msgr2=0x7f0cac1e55e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:18.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:18.065+0000 7f0cabe00640 1 --2- 172.21.15.6:0/4144534117 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0cac06bfd0 0x7f0cac1e55e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:18.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:18.065+0000 7f0cabe00640 1 -- 172.21.15.6:0/4144534117 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0cac154b00 msgr2=0x7f0cac144ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:18.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:18.065+0000 7f0cabe00640 1 --2- 172.21.15.6:0/4144534117 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0cac154b00 0x7f0cac144ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:18.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:18.065+0000 7f0cabe00640 1 -- 172.21.15.6:0/4144534117 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0cac15a960 con 0x7f0cac067940 2024-04-24T23:16:18.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:18.065+0000 7f0cb0a00640 1 --2- 172.21.15.6:0/4144534117 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0cac154b00 0x7f0cac144ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:16:18.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:18.065+0000 7f0cabe00640 1 --2- 172.21.15.6:0/4144534117 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0cac067940 0x7f0cac1e70f0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f0c9c000c00 tx=0x7f0c9c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:18.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:18.066+0000 7f0ca8c00640 1 -- 172.21.15.6:0/4144534117 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c9c041020 con 0x7f0cac067940 2024-04-24T23:16:18.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:18.066+0000 7f0ca8c00640 1 -- 172.21.15.6:0/4144534117 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c9c03d620 con 0x7f0cac067940 2024-04-24T23:16:18.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:18.066+0000 7f0ca8c00640 1 -- 172.21.15.6:0/4144534117 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c9c037040 con 0x7f0cac067940 2024-04-24T23:16:18.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:18.066+0000 7f0cb2a00640 1 -- 172.21.15.6:0/4144534117 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0cac158860 con 0x7f0cac067940 2024-04-24T23:16:18.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:18.066+0000 7f0cb2a00640 1 -- 172.21.15.6:0/4144534117 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0cac158e20 con 0x7f0cac067940 2024-04-24T23:16:18.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:18.067+0000 7f0cb2a00640 1 -- 172.21.15.6:0/4144534117 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0cac068430 con 0x7f0cac067940 2024-04-24T23:16:18.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:18.068+0000 7f0ca8c00640 1 -- 172.21.15.6:0/4144534117 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f0c9c044020 con 0x7f0cac067940 2024-04-24T23:16:18.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:18.069+0000 7f0ca8c00640 1 --2- 172.21.15.6:0/4144534117 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f0c8806be30 0x7f0c8806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:18.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:18.070+0000 7f0ca8c00640 1 -- 172.21.15.6:0/4144534117 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6590+0+0 (secure 0 0 0) 0x7f0c9c0ab7f0 con 0x7f0cac067940 2024-04-24T23:16:18.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:18.070+0000 7f0ca3e00640 1 --2- 172.21.15.6:0/4144534117 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f0c8806be30 0x7f0c8806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:18.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:18.071+0000 7f0ca3e00640 1 --2- 172.21.15.6:0/4144534117 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f0c8806be30 0x7f0c8806e2f0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f0cac159960 tx=0x7f0c94009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:18.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:18.074+0000 7f0ca8c00640 1 -- 172.21.15.6:0/4144534117 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f0c9c04b030 con 0x7f0cac067940 2024-04-24T23:16:18.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:18.322+0000 7f0cb2a00640 1 -- 172.21.15.6:0/4144534117 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f0cac0cb3f0 con 0x7f0cac067940 2024-04-24T23:16:18.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:18.323+0000 7f0ca8c00640 1 -- 172.21.15.6:0/4144534117 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+196 (secure 0 0 0) 0x7f0c9c079e20 con 0x7f0cac067940 2024-04-24T23:16:18.324 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:16:18.324 INFO:teuthology.orchestra.run.smithi006.stdout:{"status":"HEALTH_WARN","checks":{"MDS_UP_LESS_THAN_MAX":{"severity":"HEALTH_WARN","summary":{"message":"1 filesystem is online with fewer MDS than max_mds","count":1},"muted":false}},"mutes":[]} 2024-04-24T23:16:18.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:18.325+0000 7f0cb2a00640 1 -- 172.21.15.6:0/4144534117 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f0c8806be30 msgr2=0x7f0c8806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:18.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:18.325+0000 7f0cb2a00640 1 --2- 172.21.15.6:0/4144534117 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f0c8806be30 0x7f0c8806e2f0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f0cac159960 tx=0x7f0c94009190 comp rx=0 tx=0).stop 2024-04-24T23:16:18.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:18.325+0000 7f0cb2a00640 1 -- 172.21.15.6:0/4144534117 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0cac067940 msgr2=0x7f0cac1e70f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:18.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:18.325+0000 7f0cb2a00640 1 --2- 172.21.15.6:0/4144534117 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0cac067940 0x7f0cac1e70f0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f0c9c000c00 tx=0x7f0c9c0046c0 comp rx=0 tx=0).stop 2024-04-24T23:16:18.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:18.325+0000 7f0cb2a00640 1 -- 172.21.15.6:0/4144534117 shutdown_connections 2024-04-24T23:16:18.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:18.325+0000 7f0cb2a00640 1 --2- 172.21.15.6:0/4144534117 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f0c8806be30 0x7f0c8806e2f0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:18.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:18.325+0000 7f0cb2a00640 1 --2- 172.21.15.6:0/4144534117 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0cac06bfd0 0x7f0cac1e55e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:18.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:18.325+0000 7f0cb2a00640 1 --2- 172.21.15.6:0/4144534117 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0cac154b00 0x7f0cac144ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:18.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:18.325+0000 7f0cb2a00640 1 --2- 172.21.15.6:0/4144534117 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0cac067940 0x7f0cac1e70f0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:18.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:18.325+0000 7f0cb2a00640 1 -- 172.21.15.6:0/4144534117 >> 172.21.15.6:0/4144534117 conn(0x7f0cac072810 msgr2=0x7f0cac06c420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:18.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:18.326+0000 7f0cb2a00640 1 -- 172.21.15.6:0/4144534117 shutdown_connections 2024-04-24T23:16:18.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:18.326+0000 7f0cb2a00640 1 -- 172.21.15.6:0/4144534117 wait complete. 2024-04-24T23:16:21.344 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-04-24T23:16:21.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:21.538+0000 7fd50b200640 1 Processor -- start 2024-04-24T23:16:21.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:21.538+0000 7fd50b200640 1 -- start start 2024-04-24T23:16:21.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:21.540+0000 7fd50b200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd504152bd0 0x7fd504158e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:21.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:21.540+0000 7fd50b200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd50406d390 con 0x7fd50406ce30 2024-04-24T23:16:21.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:21.540+0000 7fd50b200640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd50406dbc0 con 0x7fd5040c6be0 2024-04-24T23:16:21.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:21.540+0000 7fd50b200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7fd50406d970 con 0x7fd504152bd0 2024-04-24T23:16:21.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:21.540+0000 7fd509e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd504152bd0 0x7fd504158e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:21.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:21.540+0000 7fd509e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd504152bd0 0x7fd504158e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:43536/0 (socket says 172.21.15.6:43536) 2024-04-24T23:16:21.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:21.540+0000 7fd509e00640 1 -- 172.21.15.6:0/3777710274 learned_addr learned my addr 172.21.15.6:0/3777710274 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:16:21.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:21.540+0000 7fd509400640 1 --1- 172.21.15.6:0/3777710274 >> v1:172.21.15.6:6789/0 conn(0x7fd50406ce30 0x7fd5040cc600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:39712/0 (socket says 172.21.15.6:39712) 2024-04-24T23:16:21.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:21.541+0000 7fd509e00640 1 -- 172.21.15.6:0/3777710274 >> v1:172.21.15.190:6789/0 conn(0x7fd5040c6be0 legacy=0x7fd504163090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:21.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:21.541+0000 7fd509e00640 1 -- 172.21.15.6:0/3777710274 >> v1:172.21.15.6:6789/0 conn(0x7fd50406ce30 legacy=0x7fd5040cc600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:21.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:21.541+0000 7fd509e00640 1 -- 172.21.15.6:0/3777710274 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd50406dea0 con 0x7fd504152bd0 2024-04-24T23:16:21.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:21.542+0000 7fd509e00640 1 --2- 172.21.15.6:0/3777710274 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd504152bd0 0x7fd504158e40 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fd4f8009870 tx=0x7fd4f802a300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa7fa8001be14c14 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:21.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:21.542+0000 7fd508a00640 1 -- 172.21.15.6:0/3777710274 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4f8037070 con 0x7fd504152bd0 2024-04-24T23:16:21.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:21.542+0000 7fd508a00640 1 -- 172.21.15.6:0/3777710274 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd4f802ae50 con 0x7fd504152bd0 2024-04-24T23:16:21.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:21.542+0000 7fd508a00640 1 -- 172.21.15.6:0/3777710274 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4f8032860 con 0x7fd504152bd0 2024-04-24T23:16:21.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:21.543+0000 7fd50b200640 1 -- 172.21.15.6:0/3777710274 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd504152bd0 msgr2=0x7fd504158e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:21.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:21.543+0000 7fd50b200640 1 --2- 172.21.15.6:0/3777710274 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd504152bd0 0x7fd504158e40 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fd4f8009870 tx=0x7fd4f802a300 comp rx=0 tx=0).stop 2024-04-24T23:16:21.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:21.543+0000 7fd50b200640 1 -- 172.21.15.6:0/3777710274 shutdown_connections 2024-04-24T23:16:21.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:21.543+0000 7fd50b200640 1 --2- 172.21.15.6:0/3777710274 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd504152bd0 0x7fd504158e40 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:21.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:21.543+0000 7fd50b200640 1 -- 172.21.15.6:0/3777710274 >> 172.21.15.6:0/3777710274 conn(0x7fd50406ed80 msgr2=0x7fd50406f190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:21.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:21.543+0000 7fd50b200640 1 -- 172.21.15.6:0/3777710274 shutdown_connections 2024-04-24T23:16:21.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:21.543+0000 7fd50b200640 1 -- 172.21.15.6:0/3777710274 wait complete. 2024-04-24T23:16:21.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:21.546+0000 7fd50b200640 1 Processor -- start 2024-04-24T23:16:21.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:21.546+0000 7fd50b200640 1 -- start start 2024-04-24T23:16:21.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:21.547+0000 7fd50b200640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd50406ce30 0x7fd5040c5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:21.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:21.547+0000 7fd50b200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd5040c6be0 0x7fd5041e6f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:21.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:21.547+0000 7fd50b200640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd504152bd0 0x7fd5041f4300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:21.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:21.547+0000 7fd50b200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fd504072650 con 0x7fd5040c6be0 2024-04-24T23:16:21.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:21.547+0000 7fd50b200640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fd504071890 con 0x7fd504152bd0 2024-04-24T23:16:21.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:21.547+0000 7fd50b200640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fd5040727d0 con 0x7fd50406ce30 2024-04-24T23:16:21.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:21.548+0000 7fd509400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd5040c6be0 0x7fd5041e6f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:21.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:21.548+0000 7fd509400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd5040c6be0 0x7fd5041e6f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:43540/0 (socket says 172.21.15.6:43540) 2024-04-24T23:16:21.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:21.548+0000 7fd509400640 1 -- 172.21.15.6:0/406312574 learned_addr learned my addr 172.21.15.6:0/406312574 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:16:21.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:21.548+0000 7fd50a800640 1 --2- 172.21.15.6:0/406312574 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd504152bd0 0x7fd5041f4300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:21.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:21.548+0000 7fd509e00640 1 --2- 172.21.15.6:0/406312574 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd50406ce30 0x7fd5040c5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:21.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:21.548+0000 7fd509400640 1 -- 172.21.15.6:0/406312574 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd50406ce30 msgr2=0x7fd5040c5160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:21.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:21.548+0000 7fd509400640 1 --2- 172.21.15.6:0/406312574 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd50406ce30 0x7fd5040c5160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:21.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:21.548+0000 7fd509400640 1 -- 172.21.15.6:0/406312574 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd504152bd0 msgr2=0x7fd5041f4300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:21.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:21.548+0000 7fd509400640 1 --2- 172.21.15.6:0/406312574 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd504152bd0 0x7fd5041f4300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:21.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:21.548+0000 7fd509400640 1 -- 172.21.15.6:0/406312574 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5040c56a0 con 0x7fd5040c6be0 2024-04-24T23:16:21.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:21.548+0000 7fd509400640 1 --2- 172.21.15.6:0/406312574 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd5040c6be0 0x7fd5041e6f70 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fd4f400c4d0 tx=0x7fd4f400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:21.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:21.549+0000 7fd4fe000640 1 -- 172.21.15.6:0/406312574 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4f4018070 con 0x7fd5040c6be0 2024-04-24T23:16:21.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:21.549+0000 7fd4fe000640 1 -- 172.21.15.6:0/406312574 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd4f4007480 con 0x7fd5040c6be0 2024-04-24T23:16:21.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:21.549+0000 7fd4fe000640 1 -- 172.21.15.6:0/406312574 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4f40049e0 con 0x7fd5040c6be0 2024-04-24T23:16:21.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:21.549+0000 7fd50b200640 1 -- 172.21.15.6:0/406312574 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd504155240 con 0x7fd5040c6be0 2024-04-24T23:16:21.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:21.549+0000 7fd50b200640 1 -- 172.21.15.6:0/406312574 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5041557e0 con 0x7fd5040c6be0 2024-04-24T23:16:21.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:21.550+0000 7fd50b200640 1 -- 172.21.15.6:0/406312574 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd504075b30 con 0x7fd5040c6be0 2024-04-24T23:16:21.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:21.551+0000 7fd4fe000640 1 -- 172.21.15.6:0/406312574 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fd4f4004b80 con 0x7fd5040c6be0 2024-04-24T23:16:21.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:21.552+0000 7fd4fe000640 1 --2- 172.21.15.6:0/406312574 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fd4e806be30 0x7fd4e806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:21.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:21.552+0000 7fd4fe000640 1 -- 172.21.15.6:0/406312574 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6590+0+0 (secure 0 0 0) 0x7fd4f408a420 con 0x7fd5040c6be0 2024-04-24T23:16:21.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:21.552+0000 7fd509e00640 1 --2- 172.21.15.6:0/406312574 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fd4e806be30 0x7fd4e806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:21.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:21.553+0000 7fd509e00640 1 --2- 172.21.15.6:0/406312574 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fd4e806be30 0x7fd4e806e2f0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fd4f8034000 tx=0x7fd4f802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:21.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:21.556+0000 7fd4fe000640 1 -- 172.21.15.6:0/406312574 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fd4f4007670 con 0x7fd5040c6be0 2024-04-24T23:16:21.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:21.800+0000 7fd50b200640 1 -- 172.21.15.6:0/406312574 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fd5041474f0 con 0x7fd5040c6be0 2024-04-24T23:16:21.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:21.800+0000 7fd4fe000640 1 -- 172.21.15.6:0/406312574 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+196 (secure 0 0 0) 0x7fd4f4058370 con 0x7fd5040c6be0 2024-04-24T23:16:21.802 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:16:21.802 INFO:teuthology.orchestra.run.smithi006.stdout:{"status":"HEALTH_WARN","checks":{"MDS_UP_LESS_THAN_MAX":{"severity":"HEALTH_WARN","summary":{"message":"1 filesystem is online with fewer MDS than max_mds","count":1},"muted":false}},"mutes":[]} 2024-04-24T23:16:21.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:21.803+0000 7fd50b200640 1 -- 172.21.15.6:0/406312574 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fd4e806be30 msgr2=0x7fd4e806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:21.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:21.803+0000 7fd50b200640 1 --2- 172.21.15.6:0/406312574 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fd4e806be30 0x7fd4e806e2f0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fd4f8034000 tx=0x7fd4f802e000 comp rx=0 tx=0).stop 2024-04-24T23:16:21.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:21.803+0000 7fd50b200640 1 -- 172.21.15.6:0/406312574 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd5040c6be0 msgr2=0x7fd5041e6f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:21.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:21.803+0000 7fd50b200640 1 --2- 172.21.15.6:0/406312574 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd5040c6be0 0x7fd5041e6f70 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fd4f400c4d0 tx=0x7fd4f400c990 comp rx=0 tx=0).stop 2024-04-24T23:16:21.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:21.804+0000 7fd50b200640 1 -- 172.21.15.6:0/406312574 shutdown_connections 2024-04-24T23:16:21.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:21.804+0000 7fd50b200640 1 --2- 172.21.15.6:0/406312574 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fd4e806be30 0x7fd4e806e2f0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:21.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:21.804+0000 7fd50b200640 1 --2- 172.21.15.6:0/406312574 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd504152bd0 0x7fd5041f4300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:21.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:21.804+0000 7fd50b200640 1 --2- 172.21.15.6:0/406312574 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd5040c6be0 0x7fd5041e6f70 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:21.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:21.804+0000 7fd50b200640 1 --2- 172.21.15.6:0/406312574 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd50406ce30 0x7fd5040c5160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:21.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:21.804+0000 7fd50b200640 1 -- 172.21.15.6:0/406312574 >> 172.21.15.6:0/406312574 conn(0x7fd50406ed80 msgr2=0x7fd504071c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:21.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:21.804+0000 7fd50b200640 1 -- 172.21.15.6:0/406312574 shutdown_connections 2024-04-24T23:16:21.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:21.804+0000 7fd50b200640 1 -- 172.21.15.6:0/406312574 wait complete. 2024-04-24T23:16:21.895 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-04-24T23:16:21.894+0000 7fef1ce00640 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-04-24T23:16:24.823 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-04-24T23:16:25.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:25.014+0000 7f9d40e00640 1 Processor -- start 2024-04-24T23:16:25.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:25.014+0000 7f9d40e00640 1 -- start start 2024-04-24T23:16:25.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:25.015+0000 7f9d40e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f9d3c0c8f50 0x7f9d3c06e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:25.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:25.015+0000 7f9d40e00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9d3c0c9330 0x7f9d3c06e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:25.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:25.015+0000 7f9d40e00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9d3c0c6e70 con 0x7f9d3c152bc0 2024-04-24T23:16:25.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:25.015+0000 7f9d40e00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f9d3c0c73d0 con 0x7f9d3c0c8f50 2024-04-24T23:16:25.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:25.015+0000 7f9d40e00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f9d3c0c8ac0 con 0x7f9d3c0c9330 2024-04-24T23:16:25.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:25.016+0000 7f9d39e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f9d3c152bc0 0x7f9d3c159390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:57078/0 (socket says 172.21.15.6:57078) 2024-04-24T23:16:25.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:25.016+0000 7f9d39e00640 1 -- 172.21.15.6:0/835448836 learned_addr learned my addr 172.21.15.6:0/835448836 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:16:25.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:25.016+0000 7f9d31400640 1 --2- 172.21.15.6:0/835448836 >> v2:172.21.15.6:3300/0 conn(0x7f9d3c0c8f50 0x7f9d3c06e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:25.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:25.016+0000 7f9d3a800640 1 --2- 172.21.15.6:0/835448836 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9d3c0c9330 0x7f9d3c06e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:25.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:25.016+0000 7f9d39400640 1 -- 172.21.15.6:0/835448836 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1403729063 0 0) 0x7f9d3c0c6e70 con 0x7f9d3c152bc0 2024-04-24T23:16:25.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:25.016+0000 7f9d31400640 1 -- 172.21.15.6:0/835448836 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9d3c0c9330 msgr2=0x7f9d3c06e8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:16:25.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:25.016+0000 7f9d31400640 1 --2- 172.21.15.6:0/835448836 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9d3c0c9330 0x7f9d3c06e8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:25.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:25.016+0000 7f9d31400640 1 -- 172.21.15.6:0/835448836 >> v1:172.21.15.6:6789/0 conn(0x7f9d3c152bc0 legacy=0x7f9d3c159390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:25.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:25.016+0000 7f9d31400640 1 -- 172.21.15.6:0/835448836 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d3c0c7110 con 0x7f9d3c0c8f50 2024-04-24T23:16:25.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:25.017+0000 7f9d31400640 1 --2- 172.21.15.6:0/835448836 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9d3c0c8f50 0x7f9d3c06e3a0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f9d240097a0 tx=0x7f9d2402a100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0213446e75d39f9 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:25.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:25.018+0000 7f9d39400640 1 -- 172.21.15.6:0/835448836 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d24037070 con 0x7f9d3c0c8f50 2024-04-24T23:16:25.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:25.018+0000 7f9d39400640 1 -- 172.21.15.6:0/835448836 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9d2402ae50 con 0x7f9d3c0c8f50 2024-04-24T23:16:25.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:25.018+0000 7f9d39400640 1 -- 172.21.15.6:0/835448836 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d24032660 con 0x7f9d3c0c8f50 2024-04-24T23:16:25.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:25.018+0000 7f9d40e00640 1 -- 172.21.15.6:0/835448836 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9d3c0c8f50 msgr2=0x7f9d3c06e3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:25.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:25.018+0000 7f9d40e00640 1 --2- 172.21.15.6:0/835448836 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9d3c0c8f50 0x7f9d3c06e3a0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f9d240097a0 tx=0x7f9d2402a100 comp rx=0 tx=0).stop 2024-04-24T23:16:25.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:25.019+0000 7f9d40e00640 1 -- 172.21.15.6:0/835448836 shutdown_connections 2024-04-24T23:16:25.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:25.019+0000 7f9d40e00640 1 --2- 172.21.15.6:0/835448836 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9d3c0c9330 0x7f9d3c06e8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:25.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:25.019+0000 7f9d40e00640 1 --2- 172.21.15.6:0/835448836 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9d3c0c8f50 0x7f9d3c06e3a0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:25.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:25.019+0000 7f9d40e00640 1 -- 172.21.15.6:0/835448836 >> 172.21.15.6:0/835448836 conn(0x7f9d3c06b0e0 msgr2=0x7f9d3c06b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:25.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:25.019+0000 7f9d40e00640 1 -- 172.21.15.6:0/835448836 shutdown_connections 2024-04-24T23:16:25.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:25.019+0000 7f9d40e00640 1 -- 172.21.15.6:0/835448836 wait complete. 2024-04-24T23:16:25.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:25.020+0000 7f9d40e00640 1 Processor -- start 2024-04-24T23:16:25.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:25.021+0000 7f9d40e00640 1 -- start start 2024-04-24T23:16:25.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:25.021+0000 7f9d40e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9d3c0c8f50 0x7f9d3c144100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:25.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:25.021+0000 7f9d40e00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9d3c0c9330 0x7f9d3c1f18b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:25.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:25.022+0000 7f9d40e00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9d3c152bc0 0x7f9d3c1f39b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:25.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:25.022+0000 7f9d40e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f9d3c0c8750 con 0x7f9d3c0c8f50 2024-04-24T23:16:25.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:25.022+0000 7f9d40e00640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f9d3c0c8470 con 0x7f9d3c152bc0 2024-04-24T23:16:25.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:25.022+0000 7f9d40e00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f9d3c152fa0 con 0x7f9d3c0c9330 2024-04-24T23:16:25.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:25.022+0000 7f9d39e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9d3c0c8f50 0x7f9d3c144100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:25.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:25.022+0000 7f9d39e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9d3c0c8f50 0x7f9d3c144100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:35456/0 (socket says 172.21.15.6:35456) 2024-04-24T23:16:25.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:25.022+0000 7f9d39e00640 1 -- 172.21.15.6:0/268905471 learned_addr learned my addr 172.21.15.6:0/268905471 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:16:25.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:25.022+0000 7f9d31400640 1 --2- 172.21.15.6:0/268905471 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9d3c0c9330 0x7f9d3c1f18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:25.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:25.022+0000 7f9d3a800640 1 --2- 172.21.15.6:0/268905471 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9d3c152bc0 0x7f9d3c1f39b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:25.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:25.022+0000 7f9d39e00640 1 -- 172.21.15.6:0/268905471 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9d3c0c9330 msgr2=0x7f9d3c1f18b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:25.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:25.022+0000 7f9d39e00640 1 --2- 172.21.15.6:0/268905471 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9d3c0c9330 0x7f9d3c1f18b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:25.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:25.022+0000 7f9d39e00640 1 -- 172.21.15.6:0/268905471 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9d3c152bc0 msgr2=0x7f9d3c1f39b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:25.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:25.022+0000 7f9d39e00640 1 --2- 172.21.15.6:0/268905471 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9d3c152bc0 0x7f9d3c1f39b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:25.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:25.022+0000 7f9d39e00640 1 -- 172.21.15.6:0/268905471 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d3c1f0650 con 0x7f9d3c0c8f50 2024-04-24T23:16:25.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:25.022+0000 7f9d39e00640 1 --2- 172.21.15.6:0/268905471 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9d3c0c8f50 0x7f9d3c144100 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f9d2c00cc40 tx=0x7f9d2c00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:25.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:25.023+0000 7f9d32a00640 1 -- 172.21.15.6:0/268905471 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d2c017070 con 0x7f9d3c0c8f50 2024-04-24T23:16:25.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:25.023+0000 7f9d32a00640 1 -- 172.21.15.6:0/268905471 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9d2c00ce50 con 0x7f9d3c0c8f50 2024-04-24T23:16:25.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:25.023+0000 7f9d40e00640 1 -- 172.21.15.6:0/268905471 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d3c15aed0 con 0x7f9d3c0c8f50 2024-04-24T23:16:25.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:25.023+0000 7f9d32a00640 1 -- 172.21.15.6:0/268905471 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d2c012640 con 0x7f9d3c0c8f50 2024-04-24T23:16:25.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:25.023+0000 7f9d40e00640 1 -- 172.21.15.6:0/268905471 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d3c15b490 con 0x7f9d3c0c8f50 2024-04-24T23:16:25.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:25.024+0000 7f9d40e00640 1 -- 172.21.15.6:0/268905471 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9d3c0767b0 con 0x7f9d3c0c8f50 2024-04-24T23:16:25.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:25.025+0000 7f9d32a00640 1 -- 172.21.15.6:0/268905471 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f9d2c01a070 con 0x7f9d3c0c8f50 2024-04-24T23:16:25.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:25.026+0000 7f9d32a00640 1 --2- 172.21.15.6:0/268905471 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f9d1806be60 0x7f9d1806e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:25.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:25.026+0000 7f9d32a00640 1 -- 172.21.15.6:0/268905471 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6590+0+0 (secure 0 0 0) 0x7f9d2c08b000 con 0x7f9d3c0c8f50 2024-04-24T23:16:25.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:25.027+0000 7f9d31400640 1 --2- 172.21.15.6:0/268905471 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f9d1806be60 0x7f9d1806e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:25.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:25.027+0000 7f9d31400640 1 --2- 172.21.15.6:0/268905471 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f9d1806be60 0x7f9d1806e320 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f9d24034000 tx=0x7f9d2402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:25.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:25.030+0000 7f9d32a00640 1 -- 172.21.15.6:0/268905471 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f9d2c058fe0 con 0x7f9d3c0c8f50 2024-04-24T23:16:25.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:25.282+0000 7f9d40e00640 1 -- 172.21.15.6:0/268905471 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f9d3c0c35d0 con 0x7f9d3c0c8f50 2024-04-24T23:16:25.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:25.282+0000 7f9d32a00640 1 -- 172.21.15.6:0/268905471 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+196 (secure 0 0 0) 0x7f9d2c05d2e0 con 0x7f9d3c0c8f50 2024-04-24T23:16:25.283 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:16:25.284 INFO:teuthology.orchestra.run.smithi006.stdout:{"status":"HEALTH_WARN","checks":{"MDS_UP_LESS_THAN_MAX":{"severity":"HEALTH_WARN","summary":{"message":"1 filesystem is online with fewer MDS than max_mds","count":1},"muted":false}},"mutes":[]} 2024-04-24T23:16:25.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:25.285+0000 7f9d40e00640 1 -- 172.21.15.6:0/268905471 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f9d1806be60 msgr2=0x7f9d1806e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:25.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:25.285+0000 7f9d40e00640 1 --2- 172.21.15.6:0/268905471 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f9d1806be60 0x7f9d1806e320 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f9d24034000 tx=0x7f9d2402e000 comp rx=0 tx=0).stop 2024-04-24T23:16:25.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:25.285+0000 7f9d40e00640 1 -- 172.21.15.6:0/268905471 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9d3c0c8f50 msgr2=0x7f9d3c144100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:25.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:25.285+0000 7f9d40e00640 1 --2- 172.21.15.6:0/268905471 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9d3c0c8f50 0x7f9d3c144100 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f9d2c00cc40 tx=0x7f9d2c00a430 comp rx=0 tx=0).stop 2024-04-24T23:16:25.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:25.285+0000 7f9d40e00640 1 -- 172.21.15.6:0/268905471 shutdown_connections 2024-04-24T23:16:25.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:25.285+0000 7f9d40e00640 1 --2- 172.21.15.6:0/268905471 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f9d1806be60 0x7f9d1806e320 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:25.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:25.285+0000 7f9d40e00640 1 --2- 172.21.15.6:0/268905471 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9d3c0c9330 0x7f9d3c1f18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:25.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:25.285+0000 7f9d40e00640 1 --2- 172.21.15.6:0/268905471 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9d3c152bc0 0x7f9d3c1f39b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:25.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:25.285+0000 7f9d40e00640 1 --2- 172.21.15.6:0/268905471 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9d3c0c8f50 0x7f9d3c144100 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:25.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:25.285+0000 7f9d40e00640 1 -- 172.21.15.6:0/268905471 >> 172.21.15.6:0/268905471 conn(0x7f9d3c06b0e0 msgr2=0x7f9d3c06e130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:25.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:25.285+0000 7f9d40e00640 1 -- 172.21.15.6:0/268905471 shutdown_connections 2024-04-24T23:16:25.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:25.285+0000 7f9d40e00640 1 -- 172.21.15.6:0/268905471 wait complete. 2024-04-24T23:16:26.895 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-04-24T23:16:26.894+0000 7fef1ce00640 -1 mds.pinger is_rank_lagging: rank=3 was never sent ping request. 2024-04-24T23:16:28.306 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-04-24T23:16:28.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:28.500+0000 7fd155000640 1 Processor -- start 2024-04-24T23:16:28.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:28.500+0000 7fd155000640 1 -- start start 2024-04-24T23:16:28.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:28.501+0000 7fd155000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd150152bd0 0x7fd150158e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:28.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:28.501+0000 7fd155000640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd15006d390 con 0x7fd1500c6be0 2024-04-24T23:16:28.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:28.501+0000 7fd155000640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd15006dbc0 con 0x7fd15006ce30 2024-04-24T23:16:28.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:28.501+0000 7fd155000640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7fd15006d970 con 0x7fd150152bd0 2024-04-24T23:16:28.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:28.501+0000 7fd14f400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd150152bd0 0x7fd150158e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:28.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:28.501+0000 7fd14fe00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fd1500c6be0 0x7fd150163090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:57080/0 (socket says 172.21.15.6:57080) 2024-04-24T23:16:28.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:28.501+0000 7fd14fe00640 1 -- 172.21.15.6:0/2997468916 learned_addr learned my addr 172.21.15.6:0/2997468916 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:16:28.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:28.502+0000 7fd14f400640 1 -- 172.21.15.6:0/2997468916 >> v1:172.21.15.190:6789/0 conn(0x7fd15006ce30 legacy=0x7fd1500cc600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:28.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:28.502+0000 7fd14f400640 1 -- 172.21.15.6:0/2997468916 >> v1:172.21.15.6:6789/0 conn(0x7fd1500c6be0 legacy=0x7fd150163090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:28.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:28.502+0000 7fd14f400640 1 -- 172.21.15.6:0/2997468916 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd15006dea0 con 0x7fd150152bd0 2024-04-24T23:16:28.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:28.502+0000 7fd14e000640 1 -- 172.21.15.6:0/2997468916 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3522915708 0 0) 0x7fd15006d390 con 0x7fd1500c6be0 2024-04-24T23:16:28.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:28.503+0000 7fd14f400640 1 --2- 172.21.15.6:0/2997468916 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd150152bd0 0x7fd150158e40 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fd1400097a0 tx=0x7fd14002a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8dd25886221c85c0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:28.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:28.503+0000 7fd14e000640 1 -- 172.21.15.6:0/2997468916 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd140037070 con 0x7fd150152bd0 2024-04-24T23:16:28.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:28.503+0000 7fd14e000640 1 -- 172.21.15.6:0/2997468916 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd140032440 con 0x7fd150152bd0 2024-04-24T23:16:28.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:28.503+0000 7fd14e000640 1 -- 172.21.15.6:0/2997468916 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd140032810 con 0x7fd150152bd0 2024-04-24T23:16:28.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:28.504+0000 7fd155000640 1 -- 172.21.15.6:0/2997468916 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd150152bd0 msgr2=0x7fd150158e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:28.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:28.504+0000 7fd155000640 1 --2- 172.21.15.6:0/2997468916 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd150152bd0 0x7fd150158e40 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fd1400097a0 tx=0x7fd14002a230 comp rx=0 tx=0).stop 2024-04-24T23:16:28.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:28.504+0000 7fd155000640 1 -- 172.21.15.6:0/2997468916 shutdown_connections 2024-04-24T23:16:28.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:28.504+0000 7fd155000640 1 --2- 172.21.15.6:0/2997468916 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd150152bd0 0x7fd150158e40 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:28.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:28.504+0000 7fd155000640 1 -- 172.21.15.6:0/2997468916 >> 172.21.15.6:0/2997468916 conn(0x7fd15006ed80 msgr2=0x7fd15006f190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:28.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:28.504+0000 7fd155000640 1 -- 172.21.15.6:0/2997468916 shutdown_connections 2024-04-24T23:16:28.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:28.504+0000 7fd155000640 1 -- 172.21.15.6:0/2997468916 wait complete. 2024-04-24T23:16:28.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:28.506+0000 7fd155000640 1 Processor -- start 2024-04-24T23:16:28.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:28.506+0000 7fd155000640 1 -- start start 2024-04-24T23:16:28.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:28.507+0000 7fd155000640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd15006ce30 0x7fd1501ee430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:28.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:28.507+0000 7fd155000640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd1500c6be0 0x7fd1501514f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:28.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:28.507+0000 7fd155000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd150152bd0 0x7fd150155470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:28.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:28.507+0000 7fd155000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fd150072650 con 0x7fd150152bd0 2024-04-24T23:16:28.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:28.507+0000 7fd155000640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fd150071890 con 0x7fd15006ce30 2024-04-24T23:16:28.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:28.507+0000 7fd155000640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fd1500727d0 con 0x7fd1500c6be0 2024-04-24T23:16:28.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:28.507+0000 7fd14fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd150152bd0 0x7fd150155470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:28.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:28.507+0000 7fd14fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd150152bd0 0x7fd150155470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:35472/0 (socket says 172.21.15.6:35472) 2024-04-24T23:16:28.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:28.507+0000 7fd14fe00640 1 -- 172.21.15.6:0/1095528168 learned_addr learned my addr 172.21.15.6:0/1095528168 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:16:28.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:28.507+0000 7fd14f400640 1 --2- 172.21.15.6:0/1095528168 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd15006ce30 0x7fd1501ee430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:28.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:28.507+0000 7fd14ea00640 1 --2- 172.21.15.6:0/1095528168 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd1500c6be0 0x7fd1501514f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:28.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:28.508+0000 7fd14fe00640 1 -- 172.21.15.6:0/1095528168 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd1500c6be0 msgr2=0x7fd1501514f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:28.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:28.508+0000 7fd14fe00640 1 --2- 172.21.15.6:0/1095528168 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd1500c6be0 0x7fd1501514f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:28.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:28.508+0000 7fd14fe00640 1 -- 172.21.15.6:0/1095528168 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd15006ce30 msgr2=0x7fd1501ee430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:28.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:28.508+0000 7fd14fe00640 1 --2- 172.21.15.6:0/1095528168 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd15006ce30 0x7fd1501ee430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:28.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:28.508+0000 7fd14fe00640 1 -- 172.21.15.6:0/1095528168 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd150159a50 con 0x7fd150152bd0 2024-04-24T23:16:28.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:28.508+0000 7fd14f400640 1 --2- 172.21.15.6:0/1095528168 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd15006ce30 0x7fd1501ee430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:16:28.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:28.508+0000 7fd14fe00640 1 --2- 172.21.15.6:0/1095528168 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd150152bd0 0x7fd150155470 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fd14800c480 tx=0x7fd14800c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:28.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:28.508+0000 7fd147400640 1 -- 172.21.15.6:0/1095528168 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd148018070 con 0x7fd150152bd0 2024-04-24T23:16:28.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:28.508+0000 7fd147400640 1 -- 172.21.15.6:0/1095528168 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd148007500 con 0x7fd150152bd0 2024-04-24T23:16:28.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:28.508+0000 7fd147400640 1 -- 172.21.15.6:0/1095528168 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd148004a20 con 0x7fd150152bd0 2024-04-24T23:16:28.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:28.508+0000 7fd155000640 1 -- 172.21.15.6:0/1095528168 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd150159d30 con 0x7fd150152bd0 2024-04-24T23:16:28.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:28.508+0000 7fd155000640 1 -- 172.21.15.6:0/1095528168 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd15015a2f0 con 0x7fd150152bd0 2024-04-24T23:16:28.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:28.509+0000 7fd155000640 1 -- 172.21.15.6:0/1095528168 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd150075b30 con 0x7fd150152bd0 2024-04-24T23:16:28.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:28.511+0000 7fd147400640 1 -- 172.21.15.6:0/1095528168 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fd148004bc0 con 0x7fd150152bd0 2024-04-24T23:16:28.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:28.512+0000 7fd147400640 1 --2- 172.21.15.6:0/1095528168 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fd12c06be30 0x7fd12c06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:28.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:28.513+0000 7fd147400640 1 -- 172.21.15.6:0/1095528168 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6590+0+0 (secure 0 0 0) 0x7fd14808a430 con 0x7fd150152bd0 2024-04-24T23:16:28.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:28.513+0000 7fd14f400640 1 --2- 172.21.15.6:0/1095528168 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fd12c06be30 0x7fd12c06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:28.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:28.513+0000 7fd14f400640 1 --2- 172.21.15.6:0/1095528168 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fd12c06be30 0x7fd12c06e2f0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fd140034000 tx=0x7fd14002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:28.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:28.518+0000 7fd147400640 1 -- 172.21.15.6:0/1095528168 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fd148058380 con 0x7fd150152bd0 2024-04-24T23:16:28.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:28.760+0000 7fd155000640 1 -- 172.21.15.6:0/1095528168 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fd15006d750 con 0x7fd150152bd0 2024-04-24T23:16:28.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:28.761+0000 7fd147400640 1 -- 172.21.15.6:0/1095528168 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7fd14805c680 con 0x7fd150152bd0 2024-04-24T23:16:28.763 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:16:28.763 INFO:teuthology.orchestra.run.smithi006.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-24T23:16:28.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:28.763+0000 7fd155000640 1 -- 172.21.15.6:0/1095528168 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fd12c06be30 msgr2=0x7fd12c06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:28.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:28.763+0000 7fd155000640 1 --2- 172.21.15.6:0/1095528168 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fd12c06be30 0x7fd12c06e2f0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fd140034000 tx=0x7fd14002e000 comp rx=0 tx=0).stop 2024-04-24T23:16:28.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:28.763+0000 7fd155000640 1 -- 172.21.15.6:0/1095528168 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd150152bd0 msgr2=0x7fd150155470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:28.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:28.763+0000 7fd155000640 1 --2- 172.21.15.6:0/1095528168 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd150152bd0 0x7fd150155470 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fd14800c480 tx=0x7fd14800c940 comp rx=0 tx=0).stop 2024-04-24T23:16:28.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:28.764+0000 7fd155000640 1 -- 172.21.15.6:0/1095528168 shutdown_connections 2024-04-24T23:16:28.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:28.764+0000 7fd155000640 1 --2- 172.21.15.6:0/1095528168 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fd12c06be30 0x7fd12c06e2f0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:28.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:28.764+0000 7fd155000640 1 --2- 172.21.15.6:0/1095528168 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd1500c6be0 0x7fd1501514f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:28.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:28.764+0000 7fd155000640 1 --2- 172.21.15.6:0/1095528168 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd150152bd0 0x7fd150155470 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:28.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:28.764+0000 7fd155000640 1 --2- 172.21.15.6:0/1095528168 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd15006ce30 0x7fd1501ee430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:28.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:28.764+0000 7fd155000640 1 -- 172.21.15.6:0/1095528168 >> 172.21.15.6:0/1095528168 conn(0x7fd15006ed80 msgr2=0x7fd1500cb220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:28.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:28.764+0000 7fd155000640 1 -- 172.21.15.6:0/1095528168 shutdown_connections 2024-04-24T23:16:28.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:28.764+0000 7fd155000640 1 -- 172.21.15.6:0/1095528168 wait complete. 2024-04-24T23:16:28.782 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-04-24T23:16:28.783 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T23:16:28.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:28.956+0000 7fd734000640 1 Processor -- start 2024-04-24T23:16:28.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:28.957+0000 7fd734000640 1 -- start start 2024-04-24T23:16:28.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:28.958+0000 7fd734000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd72c152bc0 0x7fd72c159390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:28.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:28.959+0000 7fd734000640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fd72c0c9360 0x7fd72c15c610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:28.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:28.959+0000 7fd734000640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd72c0c6e70 con 0x7fd72c0c8f50 2024-04-24T23:16:28.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:28.959+0000 7fd734000640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7fd72c0c73d0 con 0x7fd72c152bc0 2024-04-24T23:16:28.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:28.959+0000 7fd734000640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7fd72c0c8ac0 con 0x7fd72c0c9360 2024-04-24T23:16:28.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:28.959+0000 7fd730c00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fd72c0c8f50 0x7fd72c15bf00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:57096/0 (socket says 172.21.15.6:57096) 2024-04-24T23:16:28.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:28.959+0000 7fd731600640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd72c152bc0 0x7fd72c159390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:28.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:28.959+0000 7fd730c00640 1 -- 172.21.15.6:0/3610692319 learned_addr learned my addr 172.21.15.6:0/3610692319 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:16:28.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:28.959+0000 7fd732000640 1 --2- 172.21.15.6:0/3610692319 >> v2:172.21.15.190:3300/0 conn(0x7fd72c0c9360 0x7fd72c15c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:28.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:28.959+0000 7fd732000640 1 -- 172.21.15.6:0/3610692319 >> v2:172.21.15.6:3300/0 conn(0x7fd72c152bc0 msgr2=0x7fd72c159390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:16:28.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:28.959+0000 7fd732000640 1 --2- 172.21.15.6:0/3610692319 >> v2:172.21.15.6:3300/0 conn(0x7fd72c152bc0 0x7fd72c159390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:28.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:28.960+0000 7fd732000640 1 -- 172.21.15.6:0/3610692319 >> v1:172.21.15.6:6789/0 conn(0x7fd72c0c8f50 legacy=0x7fd72c15bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:28.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:28.960+0000 7fd732000640 1 -- 172.21.15.6:0/3610692319 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd72c1598d0 con 0x7fd72c0c9360 2024-04-24T23:16:28.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:28.961+0000 7fd732000640 1 --2- 172.21.15.6:0/3610692319 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd72c0c9360 0x7fd72c15c610 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fd72c0c7550 tx=0x7fd72402a5f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f1914153286e24ea server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:28.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:28.961+0000 7fd72be00640 1 -- 172.21.15.6:0/3610692319 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd724031600 con 0x7fd72c0c9360 2024-04-24T23:16:28.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:28.962+0000 7fd72be00640 1 -- 172.21.15.6:0/3610692319 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7240317a0 con 0x7fd72c0c9360 2024-04-24T23:16:28.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:28.962+0000 7fd72be00640 1 -- 172.21.15.6:0/3610692319 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd724031bd0 con 0x7fd72c0c9360 2024-04-24T23:16:28.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:28.962+0000 7fd734000640 1 -- 172.21.15.6:0/3610692319 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd72c0c9360 msgr2=0x7fd72c15c610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:28.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:28.962+0000 7fd734000640 1 --2- 172.21.15.6:0/3610692319 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd72c0c9360 0x7fd72c15c610 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fd72c0c7550 tx=0x7fd72402a5f0 comp rx=0 tx=0).stop 2024-04-24T23:16:28.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:28.962+0000 7fd734000640 1 -- 172.21.15.6:0/3610692319 shutdown_connections 2024-04-24T23:16:28.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:28.962+0000 7fd734000640 1 --2- 172.21.15.6:0/3610692319 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd72c0c9360 0x7fd72c15c610 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:28.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:28.962+0000 7fd734000640 1 --2- 172.21.15.6:0/3610692319 >> v2:172.21.15.6:3300/0 conn(0x7fd72c152bc0 0x7fd72c159390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:28.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:28.962+0000 7fd734000640 1 -- 172.21.15.6:0/3610692319 >> 172.21.15.6:0/3610692319 conn(0x7fd72c06b0e0 msgr2=0x7fd72c06b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:28.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:28.962+0000 7fd734000640 1 -- 172.21.15.6:0/3610692319 shutdown_connections 2024-04-24T23:16:28.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:28.962+0000 7fd734000640 1 -- 172.21.15.6:0/3610692319 wait complete. 2024-04-24T23:16:28.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:28.964+0000 7fd734000640 1 Processor -- start 2024-04-24T23:16:28.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:28.964+0000 7fd734000640 1 -- start start 2024-04-24T23:16:28.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:28.965+0000 7fd734000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd72c0c8f50 0x7fd72c1455b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:28.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:28.965+0000 7fd734000640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd72c0c9360 0x7fd72c151c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:28.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:28.965+0000 7fd734000640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd72c152bc0 0x7fd72c1f0730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:28.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:28.965+0000 7fd734000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fd72c0c8470 con 0x7fd72c0c8f50 2024-04-24T23:16:28.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:28.965+0000 7fd734000640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fd72c0c6700 con 0x7fd72c0c9360 2024-04-24T23:16:28.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:28.965+0000 7fd734000640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fd72c0c8750 con 0x7fd72c152bc0 2024-04-24T23:16:28.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:28.965+0000 7fd731600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd72c0c8f50 0x7fd72c1455b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:28.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:28.965+0000 7fd730c00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd72c0c9360 0x7fd72c151c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:28.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:28.965+0000 7fd732000640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd72c152bc0 0x7fd72c1f0730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:28.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:28.965+0000 7fd730c00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd72c0c9360 0x7fd72c151c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:58274/0 (socket says 172.21.15.6:58274) 2024-04-24T23:16:28.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:28.965+0000 7fd731600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd72c0c8f50 0x7fd72c1455b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:35490/0 (socket says 172.21.15.6:35490) 2024-04-24T23:16:28.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:28.965+0000 7fd731600640 1 -- 172.21.15.6:0/2051795081 learned_addr learned my addr 172.21.15.6:0/2051795081 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:16:28.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:28.965+0000 7fd732000640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd72c152bc0 0x7fd72c1f0730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.6:55324/0 (socket says 172.21.15.6:55324) 2024-04-24T23:16:28.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:28.966+0000 7fd730c00640 1 -- 172.21.15.6:0/2051795081 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd72c152bc0 msgr2=0x7fd72c1f0730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:28.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:28.966+0000 7fd730c00640 1 --2- 172.21.15.6:0/2051795081 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd72c152bc0 0x7fd72c1f0730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:28.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:28.966+0000 7fd730c00640 1 -- 172.21.15.6:0/2051795081 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd72c0c8f50 msgr2=0x7fd72c1455b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:28.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:28.966+0000 7fd730c00640 1 --2- 172.21.15.6:0/2051795081 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd72c0c8f50 0x7fd72c1455b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:28.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:28.966+0000 7fd730c00640 1 -- 172.21.15.6:0/2051795081 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd72c159c50 con 0x7fd72c0c9360 2024-04-24T23:16:28.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:28.966+0000 7fd730c00640 1 --2- 172.21.15.6:0/2051795081 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd72c0c9360 0x7fd72c151c80 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fd718000f60 tx=0x7fd71800c760 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:28.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:28.966+0000 7fd729600640 1 -- 172.21.15.6:0/2051795081 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd718017070 con 0x7fd72c0c9360 2024-04-24T23:16:28.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:28.966+0000 7fd729600640 1 -- 172.21.15.6:0/2051795081 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd718005ce0 con 0x7fd72c0c9360 2024-04-24T23:16:28.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:28.966+0000 7fd734000640 1 -- 172.21.15.6:0/2051795081 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd72c159f10 con 0x7fd72c0c9360 2024-04-24T23:16:28.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:28.966+0000 7fd729600640 1 -- 172.21.15.6:0/2051795081 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd718007310 con 0x7fd72c0c9360 2024-04-24T23:16:28.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:28.967+0000 7fd734000640 1 -- 172.21.15.6:0/2051795081 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd72c1461d0 con 0x7fd72c0c9360 2024-04-24T23:16:28.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:28.968+0000 7fd734000640 1 -- 172.21.15.6:0/2051795081 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd72c14e6c0 con 0x7fd72c0c9360 2024-04-24T23:16:28.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:28.969+0000 7fd729600640 1 -- 172.21.15.6:0/2051795081 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fd71801a070 con 0x7fd72c0c9360 2024-04-24T23:16:28.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:28.970+0000 7fd729600640 1 --2- 172.21.15.6:0/2051795081 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fd70c06be60 0x7fd70c06e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:28.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:28.970+0000 7fd729600640 1 -- 172.21.15.6:0/2051795081 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6590+0+0 (secure 0 0 0) 0x7fd71808a400 con 0x7fd72c0c9360 2024-04-24T23:16:28.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:28.970+0000 7fd731600640 1 --2- 172.21.15.6:0/2051795081 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fd70c06be60 0x7fd70c06e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:28.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:28.971+0000 7fd731600640 1 --2- 172.21.15.6:0/2051795081 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fd70c06be60 0x7fd70c06e320 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fd720009c00 tx=0x7fd720009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:28.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:28.974+0000 7fd729600640 1 -- 172.21.15.6:0/2051795081 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fd72c14e6c0 con 0x7fd72c0c9360 2024-04-24T23:16:29.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:29.183+0000 7fd734000640 1 -- 172.21.15.6:0/2051795081 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd72c0767b0 con 0x7fd72c0c9360 2024-04-24T23:16:29.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:29.185+0000 7fd729600640 1 -- 172.21.15.6:0/2051795081 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+18554 (secure 0 0 0) 0x7fd718058350 con 0x7fd72c0c9360 2024-04-24T23:16:29.187 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:16:29.187 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":26,"fsid":"18bef68d-5e6d-4ec5-862b-b6f206e2588d","created":"2024-04-24T23:15:12.535072+0000","modified":"2024-04-24T23:16:18.218008+0000","last_up_change":"2024-04-24T23:15:23.773223+0000","last_in_change":"2024-04-24T23:15:14.905328+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T23:15:24.747441+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-24T23:15:34.051800+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-24T23:15:35.436870+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-24T23:15:39.034257+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"3b7a212d-d498-462e-90bf-ed493c803a09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":3022569437},{"type":"v1","addr":"172.21.15.6:6809","nonce":3022569437}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":3022569437},{"type":"v1","addr":"172.21.15.6:6811","nonce":3022569437}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":3022569437},{"type":"v1","addr":"172.21.15.6:6815","nonce":3022569437}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":3022569437},{"type":"v1","addr":"172.21.15.6:6813","nonce":3022569437}]},"public_addr":"172.21.15.6:6809/3022569437","cluster_addr":"172.21.15.6:6811/3022569437","heartbeat_back_addr":"172.21.15.6:6815/3022569437","heartbeat_front_addr":"172.21.15.6:6813/3022569437","state":["exists","up"]},{"osd":1,"uuid":"7c97f5a2-1735-4775-9f15-eb50099b19d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":3687836888},{"type":"v1","addr":"172.21.15.6:6817","nonce":3687836888}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":3687836888},{"type":"v1","addr":"172.21.15.6:6819","nonce":3687836888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":3687836888},{"type":"v1","addr":"172.21.15.6:6823","nonce":3687836888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":3687836888},{"type":"v1","addr":"172.21.15.6:6821","nonce":3687836888}]},"public_addr":"172.21.15.6:6817/3687836888","cluster_addr":"172.21.15.6:6819/3687836888","heartbeat_back_addr":"172.21.15.6:6823/3687836888","heartbeat_front_addr":"172.21.15.6:6821/3687836888","state":["exists","up"]},{"osd":2,"uuid":"d86bbc18-438c-4ec2-8018-ba7dd78f790b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":1020213724},{"type":"v1","addr":"172.21.15.6:6801","nonce":1020213724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":1020213724},{"type":"v1","addr":"172.21.15.6:6803","nonce":1020213724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":1020213724},{"type":"v1","addr":"172.21.15.6:6807","nonce":1020213724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":1020213724},{"type":"v1","addr":"172.21.15.6:6805","nonce":1020213724}]},"public_addr":"172.21.15.6:6801/1020213724","cluster_addr":"172.21.15.6:6803/1020213724","heartbeat_back_addr":"172.21.15.6:6807/1020213724","heartbeat_front_addr":"172.21.15.6:6805/1020213724","state":["exists","up"]},{"osd":3,"uuid":"3754ae16-4248-40dd-95f9-0f59c0eb5426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6824","nonce":3163883468},{"type":"v1","addr":"172.21.15.6:6825","nonce":3163883468}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":3163883468},{"type":"v1","addr":"172.21.15.6:6827","nonce":3163883468}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":3163883468},{"type":"v1","addr":"172.21.15.6:6831","nonce":3163883468}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":3163883468},{"type":"v1","addr":"172.21.15.6:6829","nonce":3163883468}]},"public_addr":"172.21.15.6:6825/3163883468","cluster_addr":"172.21.15.6:6827/3163883468","heartbeat_back_addr":"172.21.15.6:6831/3163883468","heartbeat_front_addr":"172.21.15.6:6829/3163883468","state":["exists","up"]},{"osd":4,"uuid":"8b2913a0-d006-479c-84ed-5b21e0e6c392","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":1989500934},{"type":"v1","addr":"172.21.15.190:6809","nonce":1989500934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":1989500934},{"type":"v1","addr":"172.21.15.190:6811","nonce":1989500934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":1989500934},{"type":"v1","addr":"172.21.15.190:6815","nonce":1989500934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":1989500934},{"type":"v1","addr":"172.21.15.190:6813","nonce":1989500934}]},"public_addr":"172.21.15.190:6809/1989500934","cluster_addr":"172.21.15.190:6811/1989500934","heartbeat_back_addr":"172.21.15.190:6815/1989500934","heartbeat_front_addr":"172.21.15.190:6813/1989500934","state":["exists","up"]},{"osd":5,"uuid":"499b79d0-0585-4dae-ab69-196a551793da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":3687548412},{"type":"v1","addr":"172.21.15.190:6801","nonce":3687548412}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":3687548412},{"type":"v1","addr":"172.21.15.190:6803","nonce":3687548412}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":3687548412},{"type":"v1","addr":"172.21.15.190:6807","nonce":3687548412}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":3687548412},{"type":"v1","addr":"172.21.15.190:6805","nonce":3687548412}]},"public_addr":"172.21.15.190:6801/3687548412","cluster_addr":"172.21.15.190:6803/3687548412","heartbeat_back_addr":"172.21.15.190:6807/3687548412","heartbeat_front_addr":"172.21.15.190:6805/3687548412","state":["exists","up"]},{"osd":6,"uuid":"6ba6df3b-4a6b-4a89-a890-0f79f387d4d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":870571564},{"type":"v1","addr":"172.21.15.190:6817","nonce":870571564}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":870571564},{"type":"v1","addr":"172.21.15.190:6819","nonce":870571564}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":870571564},{"type":"v1","addr":"172.21.15.190:6823","nonce":870571564}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":870571564},{"type":"v1","addr":"172.21.15.190:6821","nonce":870571564}]},"public_addr":"172.21.15.190:6817/870571564","cluster_addr":"172.21.15.190:6819/870571564","heartbeat_back_addr":"172.21.15.190:6823/870571564","heartbeat_front_addr":"172.21.15.190:6821/870571564","state":["exists","up"]},{"osd":7,"uuid":"e9668b1c-a000-4332-86e8-4efd73b02409","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":123639197},{"type":"v1","addr":"172.21.15.190:6825","nonce":123639197}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":123639197},{"type":"v1","addr":"172.21.15.190:6827","nonce":123639197}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":123639197},{"type":"v1","addr":"172.21.15.190:6831","nonce":123639197}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":123639197},{"type":"v1","addr":"172.21.15.190:6829","nonce":123639197}]},"public_addr":"172.21.15.190:6825/123639197","cluster_addr":"172.21.15.190:6827/123639197","heartbeat_back_addr":"172.21.15.190:6831/123639197","heartbeat_front_addr":"172.21.15.190:6829/123639197","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.321646+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.363873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.466963+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.531989+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:19.990707+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.043543+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.032492+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.115004+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.2f","mappings":[{"from":0,"to":7}]},{"pgid":"4.19","mappings":[{"from":7,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T23:16:29.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:29.187+0000 7fd734000640 1 -- 172.21.15.6:0/2051795081 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fd70c06be60 msgr2=0x7fd70c06e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:29.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:29.188+0000 7fd734000640 1 --2- 172.21.15.6:0/2051795081 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fd70c06be60 0x7fd70c06e320 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fd720009c00 tx=0x7fd720009190 comp rx=0 tx=0).stop 2024-04-24T23:16:29.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:29.188+0000 7fd734000640 1 -- 172.21.15.6:0/2051795081 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd72c0c9360 msgr2=0x7fd72c151c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:29.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:29.188+0000 7fd734000640 1 --2- 172.21.15.6:0/2051795081 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd72c0c9360 0x7fd72c151c80 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fd718000f60 tx=0x7fd71800c760 comp rx=0 tx=0).stop 2024-04-24T23:16:29.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:29.188+0000 7fd734000640 1 -- 172.21.15.6:0/2051795081 shutdown_connections 2024-04-24T23:16:29.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:29.188+0000 7fd734000640 1 --2- 172.21.15.6:0/2051795081 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fd70c06be60 0x7fd70c06e320 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:29.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:29.188+0000 7fd734000640 1 --2- 172.21.15.6:0/2051795081 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd72c152bc0 0x7fd72c1f0730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:29.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:29.188+0000 7fd734000640 1 --2- 172.21.15.6:0/2051795081 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd72c0c9360 0x7fd72c151c80 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:29.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:29.188+0000 7fd734000640 1 --2- 172.21.15.6:0/2051795081 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd72c0c8f50 0x7fd72c1455b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:29.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:29.188+0000 7fd734000640 1 -- 172.21.15.6:0/2051795081 >> 172.21.15.6:0/2051795081 conn(0x7fd72c06b0e0 msgr2=0x7fd72c14dbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:29.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:29.188+0000 7fd734000640 1 -- 172.21.15.6:0/2051795081 shutdown_connections 2024-04-24T23:16:29.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:29.188+0000 7fd734000640 1 -- 172.21.15.6:0/2051795081 wait complete. 2024-04-24T23:16:29.207 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T23:15:24.747441+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-24T23:15:34.051800+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-24T23:15:35.436870+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-24T23:15:39.034257+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-24T23:16:29.207 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-24T23:16:29.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:29.378+0000 7f5ab2400640 1 Processor -- start 2024-04-24T23:16:29.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:29.378+0000 7f5ab2400640 1 -- start start 2024-04-24T23:16:29.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:29.379+0000 7f5ab2400640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f5aac15c1b0 0x7f5aac0cb810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:29.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:29.379+0000 7f5ab2400640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5aac0ca900 con 0x7f5aac0ca520 2024-04-24T23:16:29.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:29.379+0000 7f5ab2400640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5aac0cc360 con 0x7f5aac15e2b0 2024-04-24T23:16:29.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:29.379+0000 7f5ab2400640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f5aac0ccf70 con 0x7f5aac15c1b0 2024-04-24T23:16:29.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:29.379+0000 7f5aab400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f5aac0ca520 0x7f5aac0cd400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:57100/0 (socket says 172.21.15.6:57100) 2024-04-24T23:16:29.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:29.379+0000 7f5aab400640 1 -- 172.21.15.6:0/3845549337 learned_addr learned my addr 172.21.15.6:0/3845549337 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:16:29.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:29.380+0000 7f5aaaa00640 1 --2- 172.21.15.6:0/3845549337 >> v2:172.21.15.190:3300/0 conn(0x7f5aac15c1b0 0x7f5aac0cb810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:29.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:29.380+0000 7f5aaa000640 1 -- 172.21.15.6:0/3845549337 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 618486969 0 0) 0x7f5aac0ca900 con 0x7f5aac0ca520 2024-04-24T23:16:29.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:29.380+0000 7f5aaa000640 1 -- 172.21.15.6:0/3845549337 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5a8c003520 con 0x7f5aac0ca520 2024-04-24T23:16:29.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:29.380+0000 7f5aaa000640 1 -- 172.21.15.6:0/3845549337 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 204013862 0 0) 0x7f5aac0cc360 con 0x7f5aac15e2b0 2024-04-24T23:16:29.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:29.380+0000 7f5aaa000640 1 -- 172.21.15.6:0/3845549337 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5aac0ca900 con 0x7f5aac15e2b0 2024-04-24T23:16:29.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:29.380+0000 7f5aaaa00640 1 -- 172.21.15.6:0/3845549337 >> v1:172.21.15.190:6789/0 conn(0x7f5aac15e2b0 legacy=0x7f5aac15e740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:29.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:29.380+0000 7f5aaaa00640 1 -- 172.21.15.6:0/3845549337 >> v1:172.21.15.6:6789/0 conn(0x7f5aac0ca520 legacy=0x7f5aac0cd400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:29.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:29.380+0000 7f5aaaa00640 1 -- 172.21.15.6:0/3845549337 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5aac0cc640 con 0x7f5aac15c1b0 2024-04-24T23:16:29.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:29.381+0000 7f5aaaa00640 1 --2- 172.21.15.6:0/3845549337 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5aac15c1b0 0x7f5aac0cb810 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f5aac0caad0 tx=0x7f5a9402a780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=122add17af874394 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:29.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:29.382+0000 7f5aaa000640 1 -- 172.21.15.6:0/3845549337 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a94037070 con 0x7f5aac15c1b0 2024-04-24T23:16:29.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:29.382+0000 7f5aaa000640 1 -- 172.21.15.6:0/3845549337 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5a94007740 con 0x7f5aac15c1b0 2024-04-24T23:16:29.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:29.382+0000 7f5aaa000640 1 -- 172.21.15.6:0/3845549337 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a94007b10 con 0x7f5aac15c1b0 2024-04-24T23:16:29.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:29.382+0000 7f5ab2400640 1 -- 172.21.15.6:0/3845549337 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5aac15c1b0 msgr2=0x7f5aac0cb810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:29.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:29.382+0000 7f5ab2400640 1 --2- 172.21.15.6:0/3845549337 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5aac15c1b0 0x7f5aac0cb810 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f5aac0caad0 tx=0x7f5a9402a780 comp rx=0 tx=0).stop 2024-04-24T23:16:29.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:29.382+0000 7f5ab2400640 1 -- 172.21.15.6:0/3845549337 shutdown_connections 2024-04-24T23:16:29.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:29.382+0000 7f5ab2400640 1 --2- 172.21.15.6:0/3845549337 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5aac15c1b0 0x7f5aac0cb810 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:29.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:29.382+0000 7f5ab2400640 1 -- 172.21.15.6:0/3845549337 >> 172.21.15.6:0/3845549337 conn(0x7f5aac06c0f0 msgr2=0x7f5aac066ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:29.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:29.382+0000 7f5ab2400640 1 -- 172.21.15.6:0/3845549337 shutdown_connections 2024-04-24T23:16:29.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:29.383+0000 7f5ab2400640 1 -- 172.21.15.6:0/3845549337 wait complete. 2024-04-24T23:16:29.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:29.384+0000 7f5ab2400640 1 Processor -- start 2024-04-24T23:16:29.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:29.384+0000 7f5ab2400640 1 -- start start 2024-04-24T23:16:29.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:29.385+0000 7f5ab2400640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5aac0ca520 0x7f5aac158280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:29.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:29.385+0000 7f5ab2400640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5aac15c1b0 0x7f5aac0c7e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:29.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:29.385+0000 7f5ab2400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5aac15e2b0 0x7f5aac1f3730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:29.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:29.385+0000 7f5ab2400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f5aac067890 con 0x7f5aac15e2b0 2024-04-24T23:16:29.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:29.385+0000 7f5ab2400640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f5aac0ccc00 con 0x7f5aac15c1b0 2024-04-24T23:16:29.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:29.385+0000 7f5ab2400640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f5aac067bc0 con 0x7f5aac0ca520 2024-04-24T23:16:29.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:29.385+0000 7f5aab400640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5aac0ca520 0x7f5aac158280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:29.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:29.385+0000 7f5aaaa00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5aac15c1b0 0x7f5aac0c7e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:29.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:29.385+0000 7f5aabe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5aac15e2b0 0x7f5aac1f3730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:29.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:29.385+0000 7f5aaaa00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5aac15c1b0 0x7f5aac0c7e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:58292/0 (socket says 172.21.15.6:58292) 2024-04-24T23:16:29.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:29.385+0000 7f5aab400640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5aac0ca520 0x7f5aac158280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.6:55332/0 (socket says 172.21.15.6:55332) 2024-04-24T23:16:29.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:29.385+0000 7f5aaaa00640 1 -- 172.21.15.6:0/1575216936 learned_addr learned my addr 172.21.15.6:0/1575216936 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:16:29.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:29.386+0000 7f5aaaa00640 1 -- 172.21.15.6:0/1575216936 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5aac0ca520 msgr2=0x7f5aac158280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:29.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:29.386+0000 7f5aaaa00640 1 --2- 172.21.15.6:0/1575216936 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5aac0ca520 0x7f5aac158280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:29.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:29.386+0000 7f5aaaa00640 1 -- 172.21.15.6:0/1575216936 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5aac15e2b0 msgr2=0x7f5aac1f3730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:29.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:29.386+0000 7f5aaaa00640 1 --2- 172.21.15.6:0/1575216936 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5aac15e2b0 0x7f5aac1f3730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:29.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:29.386+0000 7f5aaaa00640 1 -- 172.21.15.6:0/1575216936 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5aac155af0 con 0x7f5aac15c1b0 2024-04-24T23:16:29.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:29.386+0000 7f5aaaa00640 1 --2- 172.21.15.6:0/1575216936 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5aac15c1b0 0x7f5aac0c7e20 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f5a94035040 tx=0x7f5a94004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:29.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:29.387+0000 7f5aa3400640 1 -- 172.21.15.6:0/1575216936 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a94042020 con 0x7f5aac15c1b0 2024-04-24T23:16:29.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:29.387+0000 7f5aa3400640 1 -- 172.21.15.6:0/1575216936 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5a94032870 con 0x7f5aac15c1b0 2024-04-24T23:16:29.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:29.387+0000 7f5aa3400640 1 -- 172.21.15.6:0/1575216936 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a94037040 con 0x7f5aac15c1b0 2024-04-24T23:16:29.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:29.387+0000 7f5ab2400640 1 -- 172.21.15.6:0/1575216936 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5aac1f1100 con 0x7f5aac15c1b0 2024-04-24T23:16:29.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:29.387+0000 7f5ab2400640 1 -- 172.21.15.6:0/1575216936 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5aac1f0340 con 0x7f5aac15c1b0 2024-04-24T23:16:29.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:29.388+0000 7f5ab2400640 1 -- 172.21.15.6:0/1575216936 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5aac0767b0 con 0x7f5aac15c1b0 2024-04-24T23:16:29.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:29.389+0000 7f5aa3400640 1 -- 172.21.15.6:0/1575216936 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f5a94005d10 con 0x7f5aac15c1b0 2024-04-24T23:16:29.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:29.390+0000 7f5aa3400640 1 --2- 172.21.15.6:0/1575216936 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f5a8806beb0 0x7f5a8806e370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:29.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:29.390+0000 7f5aa3400640 1 -- 172.21.15.6:0/1575216936 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6590+0+0 (secure 0 0 0) 0x7f5a940ac160 con 0x7f5aac15c1b0 2024-04-24T23:16:29.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:29.390+0000 7f5aab400640 1 --2- 172.21.15.6:0/1575216936 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f5a8806beb0 0x7f5a8806e370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:29.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:29.391+0000 7f5aab400640 1 --2- 172.21.15.6:0/1575216936 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f5a8806beb0 0x7f5a8806e370 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f5a9c000f60 tx=0x7f5a9c007d70 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:29.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:29.394+0000 7f5aa3400640 1 -- 172.21.15.6:0/1575216936 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f5a9404c070 con 0x7f5aac15c1b0 2024-04-24T23:16:29.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:29.601+0000 7f5ab2400640 1 -- 172.21.15.6:0/1575216936 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f5aac15e020 con 0x7f5aac15c1b0 2024-04-24T23:16:29.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:29.602+0000 7f5aa3400640 1 -- 172.21.15.6:0/1575216936 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) ==== 93+0+10 (secure 0 0 0) 0x7f5a9407a0a0 con 0x7f5aac15c1b0 2024-04-24T23:16:29.603 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-04-24T23:16:29.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:29.604+0000 7f5ab2400640 1 -- 172.21.15.6:0/1575216936 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f5a8806beb0 msgr2=0x7f5a8806e370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:29.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:29.604+0000 7f5ab2400640 1 --2- 172.21.15.6:0/1575216936 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f5a8806beb0 0x7f5a8806e370 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f5a9c000f60 tx=0x7f5a9c007d70 comp rx=0 tx=0).stop 2024-04-24T23:16:29.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:29.604+0000 7f5ab2400640 1 -- 172.21.15.6:0/1575216936 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5aac15c1b0 msgr2=0x7f5aac0c7e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:29.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:29.604+0000 7f5ab2400640 1 --2- 172.21.15.6:0/1575216936 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5aac15c1b0 0x7f5aac0c7e20 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f5a94035040 tx=0x7f5a94004a40 comp rx=0 tx=0).stop 2024-04-24T23:16:29.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:29.604+0000 7f5ab2400640 1 -- 172.21.15.6:0/1575216936 shutdown_connections 2024-04-24T23:16:29.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:29.604+0000 7f5ab2400640 1 --2- 172.21.15.6:0/1575216936 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f5a8806beb0 0x7f5a8806e370 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:29.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:29.604+0000 7f5ab2400640 1 --2- 172.21.15.6:0/1575216936 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5aac15e2b0 0x7f5aac1f3730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:29.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:29.604+0000 7f5ab2400640 1 --2- 172.21.15.6:0/1575216936 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5aac15c1b0 0x7f5aac0c7e20 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:29.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:29.604+0000 7f5ab2400640 1 --2- 172.21.15.6:0/1575216936 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5aac0ca520 0x7f5aac158280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:29.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:29.604+0000 7f5ab2400640 1 -- 172.21.15.6:0/1575216936 >> 172.21.15.6:0/1575216936 conn(0x7f5aac06c0f0 msgr2=0x7f5aac0cfc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:29.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:29.604+0000 7f5ab2400640 1 -- 172.21.15.6:0/1575216936 shutdown_connections 2024-04-24T23:16:29.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:29.604+0000 7f5ab2400640 1 -- 172.21.15.6:0/1575216936 wait complete. 2024-04-24T23:16:29.624 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-24T23:16:29.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:29.798+0000 7f844a200640 1 Processor -- start 2024-04-24T23:16:29.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:29.798+0000 7f844a200640 1 -- start start 2024-04-24T23:16:29.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:29.800+0000 7f844a200640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f84440c9330 0x7f84440c6ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:29.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:29.800+0000 7f844a200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f8444070e00 0x7f844415ab60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:29.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:29.800+0000 7f844a200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8444071a40 con 0x7f8444154940 2024-04-24T23:16:29.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:29.800+0000 7f844a200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f8444071d20 con 0x7f8444070e00 2024-04-24T23:16:29.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:29.800+0000 7f844a200640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f8444072000 con 0x7f84440c9330 2024-04-24T23:16:29.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:29.800+0000 7f8448e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f8444154940 0x7f844406e910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:57104/0 (socket says 172.21.15.6:57104) 2024-04-24T23:16:29.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:29.800+0000 7f8448e00640 1 -- 172.21.15.6:0/3335085870 learned_addr learned my addr 172.21.15.6:0/3335085870 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:16:29.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:29.800+0000 7f8449800640 1 --2- 172.21.15.6:0/3335085870 >> v2:172.21.15.6:3300/0 conn(0x7f8444070e00 0x7f844415ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:29.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:29.800+0000 7f8443e00640 1 --2- 172.21.15.6:0/3335085870 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f84440c9330 0x7f84440c6ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:29.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:29.801+0000 7f8449800640 1 -- 172.21.15.6:0/3335085870 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f84440c9330 msgr2=0x7f84440c6ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:16:29.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:29.801+0000 7f8449800640 1 --2- 172.21.15.6:0/3335085870 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f84440c9330 0x7f84440c6ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:29.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:29.801+0000 7f8449800640 1 -- 172.21.15.6:0/3335085870 >> v1:172.21.15.6:6789/0 conn(0x7f8444154940 legacy=0x7f844406e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:29.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:29.801+0000 7f8449800640 1 -- 172.21.15.6:0/3335085870 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84440722e0 con 0x7f8444070e00 2024-04-24T23:16:29.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:29.802+0000 7f8449800640 1 --2- 172.21.15.6:0/3335085870 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8444070e00 0x7f844415ab60 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f843c009d90 tx=0x7f843c02a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68b5d9efd3184161 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:29.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:29.802+0000 7f8443400640 1 -- 172.21.15.6:0/3335085870 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f843c007740 con 0x7f8444070e00 2024-04-24T23:16:29.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:29.802+0000 7f8443400640 1 -- 172.21.15.6:0/3335085870 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f843c0078e0 con 0x7f8444070e00 2024-04-24T23:16:29.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:29.802+0000 7f8443400640 1 -- 172.21.15.6:0/3335085870 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f843c007cb0 con 0x7f8444070e00 2024-04-24T23:16:29.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:29.803+0000 7f844a200640 1 -- 172.21.15.6:0/3335085870 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8444070e00 msgr2=0x7f844415ab60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:29.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:29.803+0000 7f844a200640 1 --2- 172.21.15.6:0/3335085870 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8444070e00 0x7f844415ab60 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f843c009d90 tx=0x7f843c02a6f0 comp rx=0 tx=0).stop 2024-04-24T23:16:29.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:29.803+0000 7f844a200640 1 -- 172.21.15.6:0/3335085870 shutdown_connections 2024-04-24T23:16:29.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:29.803+0000 7f844a200640 1 --2- 172.21.15.6:0/3335085870 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8444070e00 0x7f844415ab60 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:29.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:29.803+0000 7f844a200640 1 --2- 172.21.15.6:0/3335085870 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f84440c9330 0x7f84440c6ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:29.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:29.803+0000 7f844a200640 1 -- 172.21.15.6:0/3335085870 >> 172.21.15.6:0/3335085870 conn(0x7f844406cc40 msgr2=0x7f844406d050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:29.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:29.803+0000 7f844a200640 1 -- 172.21.15.6:0/3335085870 shutdown_connections 2024-04-24T23:16:29.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:29.803+0000 7f844a200640 1 -- 172.21.15.6:0/3335085870 wait complete. 2024-04-24T23:16:29.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:29.805+0000 7f844a200640 1 Processor -- start 2024-04-24T23:16:29.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:29.805+0000 7f844a200640 1 -- start start 2024-04-24T23:16:29.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:29.806+0000 7f844a200640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8444070e00 0x7f8444157bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:29.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:29.806+0000 7f844a200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f84440c9330 0x7f8444159030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:29.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:29.806+0000 7f844a200640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8444154940 0x7f8444159570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:29.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:29.806+0000 7f844a200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f844406f020 con 0x7f84440c9330 2024-04-24T23:16:29.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:29.806+0000 7f844a200640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f8444154d20 con 0x7f8444154940 2024-04-24T23:16:29.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:29.806+0000 7f844a200640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f844406f1a0 con 0x7f8444070e00 2024-04-24T23:16:29.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:29.806+0000 7f8443e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f84440c9330 0x7f8444159030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:29.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:29.806+0000 7f8443e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f84440c9330 0x7f8444159030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:35524/0 (socket says 172.21.15.6:35524) 2024-04-24T23:16:29.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:29.806+0000 7f8443e00640 1 -- 172.21.15.6:0/3803146708 learned_addr learned my addr 172.21.15.6:0/3803146708 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:16:29.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:29.806+0000 7f8448e00640 1 --2- 172.21.15.6:0/3803146708 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8444070e00 0x7f8444157bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:29.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:29.806+0000 7f8449800640 1 --2- 172.21.15.6:0/3803146708 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8444154940 0x7f8444159570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:29.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:29.807+0000 7f8443e00640 1 -- 172.21.15.6:0/3803146708 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8444070e00 msgr2=0x7f8444157bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:29.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:29.807+0000 7f8443e00640 1 --2- 172.21.15.6:0/3803146708 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8444070e00 0x7f8444157bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:29.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:29.807+0000 7f8443e00640 1 -- 172.21.15.6:0/3803146708 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8444154940 msgr2=0x7f8444159570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:29.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:29.807+0000 7f8443e00640 1 --2- 172.21.15.6:0/3803146708 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8444154940 0x7f8444159570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:29.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:29.807+0000 7f8443e00640 1 -- 172.21.15.6:0/3803146708 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84441e6c40 con 0x7f84440c9330 2024-04-24T23:16:29.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:29.807+0000 7f8443e00640 1 --2- 172.21.15.6:0/3803146708 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f84440c9330 0x7f8444159030 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f843000e9b0 tx=0x7f843000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:29.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:29.807+0000 7f8440c00640 1 -- 172.21.15.6:0/3803146708 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8430017070 con 0x7f84440c9330 2024-04-24T23:16:29.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:29.807+0000 7f8440c00640 1 -- 172.21.15.6:0/3803146708 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f843000c9a0 con 0x7f84440c9330 2024-04-24T23:16:29.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:29.807+0000 7f8440c00640 1 -- 172.21.15.6:0/3803146708 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f843000cd70 con 0x7f84440c9330 2024-04-24T23:16:29.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:29.807+0000 7f844a200640 1 -- 172.21.15.6:0/3803146708 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8444156ff0 con 0x7f84440c9330 2024-04-24T23:16:29.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:29.807+0000 7f844a200640 1 -- 172.21.15.6:0/3803146708 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84441f1130 con 0x7f84440c9330 2024-04-24T23:16:29.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:29.809+0000 7f844a200640 1 -- 172.21.15.6:0/3803146708 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8444158bc0 con 0x7f84440c9330 2024-04-24T23:16:29.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:29.810+0000 7f8440c00640 1 -- 172.21.15.6:0/3803146708 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f843001a070 con 0x7f84440c9330 2024-04-24T23:16:29.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:29.811+0000 7f8440c00640 1 --2- 172.21.15.6:0/3803146708 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f842406be30 0x7f842406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:29.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:29.811+0000 7f8440c00640 1 -- 172.21.15.6:0/3803146708 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6590+0+0 (secure 0 0 0) 0x7f843008b0c0 con 0x7f84440c9330 2024-04-24T23:16:29.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:29.811+0000 7f8448e00640 1 --2- 172.21.15.6:0/3803146708 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f842406be30 0x7f842406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:29.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:29.812+0000 7f8448e00640 1 --2- 172.21.15.6:0/3803146708 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f842406be30 0x7f842406e2f0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f8438007fd0 tx=0x7f8438007e80 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:29.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:29.819+0000 7f8440c00640 1 -- 172.21.15.6:0/3803146708 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f8444158bc0 con 0x7f84440c9330 2024-04-24T23:16:30.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.014+0000 7f844a200640 1 -- 172.21.15.6:0/3803146708 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f8444075b30 con 0x7f84440c9330 2024-04-24T23:16:30.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.015+0000 7f8440c00640 1 -- 172.21.15.6:0/3803146708 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) ==== 104+0+11 (secure 0 0 0) 0x7f8444156ff0 con 0x7f84440c9330 2024-04-24T23:16:30.016 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 32 2024-04-24T23:16:30.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.017+0000 7f844a200640 1 -- 172.21.15.6:0/3803146708 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f842406be30 msgr2=0x7f842406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:30.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.017+0000 7f844a200640 1 --2- 172.21.15.6:0/3803146708 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f842406be30 0x7f842406e2f0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f8438007fd0 tx=0x7f8438007e80 comp rx=0 tx=0).stop 2024-04-24T23:16:30.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.017+0000 7f844a200640 1 -- 172.21.15.6:0/3803146708 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f84440c9330 msgr2=0x7f8444159030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:30.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.017+0000 7f844a200640 1 --2- 172.21.15.6:0/3803146708 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f84440c9330 0x7f8444159030 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f843000e9b0 tx=0x7f843000ee70 comp rx=0 tx=0).stop 2024-04-24T23:16:30.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.017+0000 7f844a200640 1 -- 172.21.15.6:0/3803146708 shutdown_connections 2024-04-24T23:16:30.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.017+0000 7f844a200640 1 --2- 172.21.15.6:0/3803146708 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f842406be30 0x7f842406e2f0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:30.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.017+0000 7f844a200640 1 --2- 172.21.15.6:0/3803146708 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8444154940 0x7f8444159570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:30.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.017+0000 7f844a200640 1 --2- 172.21.15.6:0/3803146708 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f84440c9330 0x7f8444159030 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:30.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.017+0000 7f844a200640 1 --2- 172.21.15.6:0/3803146708 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8444070e00 0x7f8444157bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:30.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.017+0000 7f844a200640 1 -- 172.21.15.6:0/3803146708 >> 172.21.15.6:0/3803146708 conn(0x7f844406cc40 msgr2=0x7f844415a1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:30.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.017+0000 7f844a200640 1 -- 172.21.15.6:0/3803146708 shutdown_connections 2024-04-24T23:16:30.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.017+0000 7f844a200640 1 -- 172.21.15.6:0/3803146708 wait complete. 2024-04-24T23:16:30.034 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-24T23:16:30.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.210+0000 7ff240400640 1 Processor -- start 2024-04-24T23:16:30.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.210+0000 7ff240400640 1 -- start start 2024-04-24T23:16:30.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.212+0000 7ff240400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ff23815ec80 0x7ff23815b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:30.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.212+0000 7ff240400640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7ff2380cb3c0 0x7ff2380c9170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:30.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.212+0000 7ff240400640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff2380cc710 con 0x7ff238154ef0 2024-04-24T23:16:30.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.212+0000 7ff240400640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7ff2380cbca0 con 0x7ff23815ec80 2024-04-24T23:16:30.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.212+0000 7ff240400640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7ff23815bc90 con 0x7ff2380cb3c0 2024-04-24T23:16:30.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.212+0000 7ff23d000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ff23815ec80 0x7ff23815b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:30.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.212+0000 7ff23e400640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7ff2380cb3c0 0x7ff2380c9170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:30.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.212+0000 7ff23da00640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7ff238154ef0 0x7ff23815e570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.6:43620/0 (socket says 172.21.15.6:43620) 2024-04-24T23:16:30.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.212+0000 7ff23d000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ff23815ec80 0x7ff23815b750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:35532/0 (socket says 172.21.15.6:35532) 2024-04-24T23:16:30.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.212+0000 7ff23da00640 1 -- 172.21.15.6:0/4102677310 learned_addr learned my addr 172.21.15.6:0/4102677310 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:16:30.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.213+0000 7ff23e400640 1 -- 172.21.15.6:0/4102677310 >> v2:172.21.15.6:3300/0 conn(0x7ff23815ec80 msgr2=0x7ff23815b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:16:30.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.213+0000 7ff23e400640 1 --2- 172.21.15.6:0/4102677310 >> v2:172.21.15.6:3300/0 conn(0x7ff23815ec80 0x7ff23815b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:30.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.213+0000 7ff23e400640 1 -- 172.21.15.6:0/4102677310 >> v1:172.21.15.190:6789/0 conn(0x7ff238154ef0 legacy=0x7ff23815e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:30.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.213+0000 7ff23e400640 1 -- 172.21.15.6:0/4102677310 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2380cab00 con 0x7ff2380cb3c0 2024-04-24T23:16:30.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.214+0000 7ff23e400640 1 --2- 172.21.15.6:0/4102677310 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff2380cb3c0 0x7ff2380c9170 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7ff2380ca4b0 tx=0x7ff23002a560 comp rx=0 tx=0).ready entity=mon.1 client_cookie=928b7ed1be3a8091 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:30.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.214+0000 7ff237e00640 1 -- 172.21.15.6:0/4102677310 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff230007740 con 0x7ff2380cb3c0 2024-04-24T23:16:30.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.214+0000 7ff237e00640 1 -- 172.21.15.6:0/4102677310 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff2300078e0 con 0x7ff2380cb3c0 2024-04-24T23:16:30.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.214+0000 7ff237e00640 1 -- 172.21.15.6:0/4102677310 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff230007cb0 con 0x7ff2380cb3c0 2024-04-24T23:16:30.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.215+0000 7ff240400640 1 -- 172.21.15.6:0/4102677310 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff2380cb3c0 msgr2=0x7ff2380c9170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:30.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.215+0000 7ff240400640 1 --2- 172.21.15.6:0/4102677310 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff2380cb3c0 0x7ff2380c9170 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7ff2380ca4b0 tx=0x7ff23002a560 comp rx=0 tx=0).stop 2024-04-24T23:16:30.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.215+0000 7ff240400640 1 -- 172.21.15.6:0/4102677310 shutdown_connections 2024-04-24T23:16:30.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.215+0000 7ff240400640 1 --2- 172.21.15.6:0/4102677310 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff2380cb3c0 0x7ff2380c9170 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:30.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.215+0000 7ff240400640 1 --2- 172.21.15.6:0/4102677310 >> v2:172.21.15.6:3300/0 conn(0x7ff23815ec80 0x7ff23815b750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:30.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.215+0000 7ff240400640 1 -- 172.21.15.6:0/4102677310 >> 172.21.15.6:0/4102677310 conn(0x7ff2380cc9a0 msgr2=0x7ff2380ccd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:30.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.215+0000 7ff240400640 1 -- 172.21.15.6:0/4102677310 shutdown_connections 2024-04-24T23:16:30.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.215+0000 7ff240400640 1 -- 172.21.15.6:0/4102677310 wait complete. 2024-04-24T23:16:30.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.216+0000 7ff240400640 1 Processor -- start 2024-04-24T23:16:30.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.217+0000 7ff240400640 1 -- start start 2024-04-24T23:16:30.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.217+0000 7ff240400640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff2380cb3c0 0x7ff23815be30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:30.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.217+0000 7ff240400640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff238154ef0 0x7ff2381f12c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:30.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.217+0000 7ff240400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff23815ec80 0x7ff2381f3d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:30.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.217+0000 7ff240400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7ff2380cc150 con 0x7ff23815ec80 2024-04-24T23:16:30.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.217+0000 7ff240400640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7ff2380cbf00 con 0x7ff2380cb3c0 2024-04-24T23:16:30.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.217+0000 7ff240400640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7ff2380cc430 con 0x7ff238154ef0 2024-04-24T23:16:30.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.218+0000 7ff23e400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff23815ec80 0x7ff2381f3d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:30.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.218+0000 7ff23e400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff23815ec80 0x7ff2381f3d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:35546/0 (socket says 172.21.15.6:35546) 2024-04-24T23:16:30.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.218+0000 7ff23e400640 1 -- 172.21.15.6:0/1326153517 learned_addr learned my addr 172.21.15.6:0/1326153517 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:16:30.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.218+0000 7ff23da00640 1 --2- 172.21.15.6:0/1326153517 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff2380cb3c0 0x7ff23815be30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:30.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.218+0000 7ff23d000640 1 --2- 172.21.15.6:0/1326153517 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff238154ef0 0x7ff2381f12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:30.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.218+0000 7ff23e400640 1 -- 172.21.15.6:0/1326153517 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff238154ef0 msgr2=0x7ff2381f12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:30.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.218+0000 7ff23e400640 1 --2- 172.21.15.6:0/1326153517 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff238154ef0 0x7ff2381f12c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:30.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.218+0000 7ff23e400640 1 -- 172.21.15.6:0/1326153517 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff2380cb3c0 msgr2=0x7ff23815be30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:30.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.218+0000 7ff23e400640 1 --2- 172.21.15.6:0/1326153517 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff2380cb3c0 0x7ff23815be30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:30.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.218+0000 7ff23e400640 1 -- 172.21.15.6:0/1326153517 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2381f42b0 con 0x7ff23815ec80 2024-04-24T23:16:30.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.218+0000 7ff23d000640 1 --2- 172.21.15.6:0/1326153517 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff238154ef0 0x7ff2381f12c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:16:30.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.219+0000 7ff23e400640 1 --2- 172.21.15.6:0/1326153517 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff23815ec80 0x7ff2381f3d70 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7ff23002ab50 tx=0x7ff230004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:30.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.219+0000 7ff235600640 1 -- 172.21.15.6:0/1326153517 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff23003f070 con 0x7ff23815ec80 2024-04-24T23:16:30.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.219+0000 7ff235600640 1 -- 172.21.15.6:0/1326153517 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff230031bc0 con 0x7ff23815ec80 2024-04-24T23:16:30.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.219+0000 7ff235600640 1 -- 172.21.15.6:0/1326153517 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff230009450 con 0x7ff23815ec80 2024-04-24T23:16:30.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.219+0000 7ff240400640 1 -- 172.21.15.6:0/1326153517 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2381f0800 con 0x7ff23815ec80 2024-04-24T23:16:30.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.219+0000 7ff240400640 1 -- 172.21.15.6:0/1326153517 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2381f4820 con 0x7ff23815ec80 2024-04-24T23:16:30.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.221+0000 7ff240400640 1 -- 172.21.15.6:0/1326153517 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff2380767b0 con 0x7ff23815ec80 2024-04-24T23:16:30.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.222+0000 7ff235600640 1 -- 172.21.15.6:0/1326153517 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7ff23003b410 con 0x7ff23815ec80 2024-04-24T23:16:30.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.223+0000 7ff235600640 1 --2- 172.21.15.6:0/1326153517 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7ff21806be30 0x7ff21806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:30.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.223+0000 7ff235600640 1 -- 172.21.15.6:0/1326153517 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6590+0+0 (secure 0 0 0) 0x7ff2300ac900 con 0x7ff23815ec80 2024-04-24T23:16:30.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.223+0000 7ff23da00640 1 --2- 172.21.15.6:0/1326153517 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7ff21806be30 0x7ff21806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:30.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.224+0000 7ff23da00640 1 --2- 172.21.15.6:0/1326153517 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7ff21806be30 0x7ff21806e2f0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7ff22c007fd0 tx=0x7ff22c007e80 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:30.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.231+0000 7ff235600640 1 -- 172.21.15.6:0/1326153517 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7ff23007af30 con 0x7ff23815ec80 2024-04-24T23:16:30.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.429+0000 7ff240400640 1 -- 172.21.15.6:0/1326153517 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7ff2380cb1a0 con 0x7ff23815ec80 2024-04-24T23:16:30.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.430+0000 7ff235600640 1 -- 172.21.15.6:0/1326153517 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) ==== 100+0+11 (secure 0 0 0) 0x7ff23007a830 con 0x7ff23815ec80 2024-04-24T23:16:30.431 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 64 2024-04-24T23:16:30.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.432+0000 7ff240400640 1 -- 172.21.15.6:0/1326153517 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7ff21806be30 msgr2=0x7ff21806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:30.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.432+0000 7ff240400640 1 --2- 172.21.15.6:0/1326153517 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7ff21806be30 0x7ff21806e2f0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7ff22c007fd0 tx=0x7ff22c007e80 comp rx=0 tx=0).stop 2024-04-24T23:16:30.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.432+0000 7ff240400640 1 -- 172.21.15.6:0/1326153517 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff23815ec80 msgr2=0x7ff2381f3d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:30.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.432+0000 7ff240400640 1 --2- 172.21.15.6:0/1326153517 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff23815ec80 0x7ff2381f3d70 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7ff23002ab50 tx=0x7ff230004a40 comp rx=0 tx=0).stop 2024-04-24T23:16:30.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.432+0000 7ff240400640 1 -- 172.21.15.6:0/1326153517 shutdown_connections 2024-04-24T23:16:30.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.432+0000 7ff240400640 1 --2- 172.21.15.6:0/1326153517 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7ff21806be30 0x7ff21806e2f0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:30.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.432+0000 7ff240400640 1 --2- 172.21.15.6:0/1326153517 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff238154ef0 0x7ff2381f12c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:30.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.432+0000 7ff240400640 1 --2- 172.21.15.6:0/1326153517 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff23815ec80 0x7ff2381f3d70 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:30.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.432+0000 7ff240400640 1 --2- 172.21.15.6:0/1326153517 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff2380cb3c0 0x7ff23815be30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:30.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.432+0000 7ff240400640 1 -- 172.21.15.6:0/1326153517 >> 172.21.15.6:0/1326153517 conn(0x7ff2380cc9a0 msgr2=0x7ff2380c79c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:30.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.432+0000 7ff240400640 1 -- 172.21.15.6:0/1326153517 shutdown_connections 2024-04-24T23:16:30.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.432+0000 7ff240400640 1 -- 172.21.15.6:0/1326153517 wait complete. 2024-04-24T23:16:30.451 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-24T23:16:30.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.643+0000 7f71dac00640 1 Processor -- start 2024-04-24T23:16:30.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.644+0000 7f71dac00640 1 -- start start 2024-04-24T23:16:30.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.645+0000 7f71dac00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f71d4154f60 0x7f71d406c640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:30.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.645+0000 7f71dac00640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f71d40b35f0 0x7f71d406cb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:30.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.645+0000 7f71dac00640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f71d40b46f0 con 0x7f71d415ab20 2024-04-24T23:16:30.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.645+0000 7f71dac00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f71d40b49d0 con 0x7f71d4154f60 2024-04-24T23:16:30.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.645+0000 7f71dac00640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f71d40b4cb0 con 0x7f71d40b35f0 2024-04-24T23:16:30.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.646+0000 7f71d8e00640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f71d40b35f0 0x7f71d406cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:30.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.646+0000 7f71da200640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f71d415ab20 0x7f71d415af30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.6:43628/0 (socket says 172.21.15.6:43628) 2024-04-24T23:16:30.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.646+0000 7f71da200640 1 -- 172.21.15.6:0/221117295 learned_addr learned my addr 172.21.15.6:0/221117295 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:16:30.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.646+0000 7f71d9800640 1 --2- 172.21.15.6:0/221117295 >> v2:172.21.15.6:3300/0 conn(0x7f71d4154f60 0x7f71d406c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:30.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.646+0000 7f71d3e00640 1 -- 172.21.15.6:0/221117295 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3368247150 0 0) 0x7f71d40b46f0 con 0x7f71d415ab20 2024-04-24T23:16:30.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.646+0000 7f71d3e00640 1 -- 172.21.15.6:0/221117295 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f71bc003520 con 0x7f71d415ab20 2024-04-24T23:16:30.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.646+0000 7f71d8e00640 1 -- 172.21.15.6:0/221117295 >> v2:172.21.15.6:3300/0 conn(0x7f71d4154f60 msgr2=0x7f71d406c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:16:30.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.646+0000 7f71d8e00640 1 --2- 172.21.15.6:0/221117295 >> v2:172.21.15.6:3300/0 conn(0x7f71d4154f60 0x7f71d406c640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:30.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.646+0000 7f71d8e00640 1 -- 172.21.15.6:0/221117295 >> v1:172.21.15.190:6789/0 conn(0x7f71d415ab20 legacy=0x7f71d415af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:30.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.646+0000 7f71d8e00640 1 -- 172.21.15.6:0/221117295 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71d40b4f90 con 0x7f71d40b35f0 2024-04-24T23:16:30.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.646+0000 7f71d9800640 1 --2- 172.21.15.6:0/221117295 >> v2:172.21.15.6:3300/0 conn(0x7f71d4154f60 0x7f71d406c640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:16:30.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.647+0000 7f71d8e00640 1 --2- 172.21.15.6:0/221117295 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f71d40b35f0 0x7f71d406cb80 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f71d40b62f0 tx=0x7f71c402a250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b037326f2ec62307 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:30.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.648+0000 7f71d3e00640 1 -- 172.21.15.6:0/221117295 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71c4037070 con 0x7f71d40b35f0 2024-04-24T23:16:30.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.648+0000 7f71d3e00640 1 -- 172.21.15.6:0/221117295 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f71c4032440 con 0x7f71d40b35f0 2024-04-24T23:16:30.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.648+0000 7f71d3e00640 1 -- 172.21.15.6:0/221117295 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71c4032810 con 0x7f71d40b35f0 2024-04-24T23:16:30.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.648+0000 7f71dac00640 1 -- 172.21.15.6:0/221117295 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f71d40b35f0 msgr2=0x7f71d406cb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:30.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.648+0000 7f71dac00640 1 --2- 172.21.15.6:0/221117295 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f71d40b35f0 0x7f71d406cb80 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f71d40b62f0 tx=0x7f71c402a250 comp rx=0 tx=0).stop 2024-04-24T23:16:30.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.648+0000 7f71dac00640 1 -- 172.21.15.6:0/221117295 shutdown_connections 2024-04-24T23:16:30.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.648+0000 7f71dac00640 1 --2- 172.21.15.6:0/221117295 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f71d40b35f0 0x7f71d406cb80 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:30.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.648+0000 7f71dac00640 1 --2- 172.21.15.6:0/221117295 >> v2:172.21.15.6:3300/0 conn(0x7f71d4154f60 0x7f71d406c640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:30.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.648+0000 7f71dac00640 1 -- 172.21.15.6:0/221117295 >> 172.21.15.6:0/221117295 conn(0x7f71d40674d0 msgr2=0x7f71d40d0320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:30.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.648+0000 7f71dac00640 1 -- 172.21.15.6:0/221117295 shutdown_connections 2024-04-24T23:16:30.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.649+0000 7f71dac00640 1 -- 172.21.15.6:0/221117295 wait complete. 2024-04-24T23:16:30.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.650+0000 7f71dac00640 1 Processor -- start 2024-04-24T23:16:30.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.650+0000 7f71dac00640 1 -- start start 2024-04-24T23:16:30.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.651+0000 7f71dac00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f71d40b35f0 0x7f71d40cf5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:30.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.651+0000 7f71dac00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f71d4154f60 0x7f71d415d560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:30.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.651+0000 7f71dac00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f71d415ab20 0x7f71d415daa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:30.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.651+0000 7f71dac00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f71d40b3240 con 0x7f71d415ab20 2024-04-24T23:16:30.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.651+0000 7f71dac00640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f71d406c120 con 0x7f71d40b35f0 2024-04-24T23:16:30.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.651+0000 7f71dac00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f71d40b33c0 con 0x7f71d4154f60 2024-04-24T23:16:30.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.652+0000 7f71da200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f71d415ab20 0x7f71d415daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:30.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.652+0000 7f71d9800640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f71d40b35f0 0x7f71d40cf5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:30.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.652+0000 7f71d8e00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f71d4154f60 0x7f71d415d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:30.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.652+0000 7f71d9800640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f71d40b35f0 0x7f71d40cf5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:58350/0 (socket says 172.21.15.6:58350) 2024-04-24T23:16:30.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.652+0000 7f71d9800640 1 -- 172.21.15.6:0/854134303 learned_addr learned my addr 172.21.15.6:0/854134303 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:16:30.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.652+0000 7f71d9800640 1 -- 172.21.15.6:0/854134303 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f71d4154f60 msgr2=0x7f71d415d560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:30.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.652+0000 7f71d9800640 1 --2- 172.21.15.6:0/854134303 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f71d4154f60 0x7f71d415d560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:30.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.652+0000 7f71d9800640 1 -- 172.21.15.6:0/854134303 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f71d415ab20 msgr2=0x7f71d415daa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:30.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.652+0000 7f71d9800640 1 --2- 172.21.15.6:0/854134303 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f71d415ab20 0x7f71d415daa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:30.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.652+0000 7f71d9800640 1 -- 172.21.15.6:0/854134303 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71d415cdd0 con 0x7f71d40b35f0 2024-04-24T23:16:30.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.652+0000 7f71d8e00640 1 --2- 172.21.15.6:0/854134303 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f71d4154f60 0x7f71d415d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:16:30.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.652+0000 7f71da200640 1 --2- 172.21.15.6:0/854134303 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f71d415ab20 0x7f71d415daa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:16:30.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.652+0000 7f71d9800640 1 --2- 172.21.15.6:0/854134303 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f71d40b35f0 0x7f71d40cf5b0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f71c800ebe0 tx=0x7f71c800c520 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:30.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.653+0000 7f71d1600640 1 -- 172.21.15.6:0/854134303 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71c8017070 con 0x7f71d40b35f0 2024-04-24T23:16:30.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.653+0000 7f71d1600640 1 -- 172.21.15.6:0/854134303 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f71c800cc30 con 0x7f71d40b35f0 2024-04-24T23:16:30.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.653+0000 7f71d1600640 1 -- 172.21.15.6:0/854134303 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71c8013470 con 0x7f71d40b35f0 2024-04-24T23:16:30.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.653+0000 7f71dac00640 1 -- 172.21.15.6:0/854134303 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71d40cfee0 con 0x7f71d40b35f0 2024-04-24T23:16:30.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.653+0000 7f71dac00640 1 -- 172.21.15.6:0/854134303 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71d41f48e0 con 0x7f71d40b35f0 2024-04-24T23:16:30.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.654+0000 7f71dac00640 1 -- 172.21.15.6:0/854134303 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f71d4075b30 con 0x7f71d40b35f0 2024-04-24T23:16:30.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.655+0000 7f71d1600640 1 -- 172.21.15.6:0/854134303 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f71c801a070 con 0x7f71d40b35f0 2024-04-24T23:16:30.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.656+0000 7f71d1600640 1 --2- 172.21.15.6:0/854134303 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f71b806be30 0x7f71b806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:30.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.656+0000 7f71d1600640 1 -- 172.21.15.6:0/854134303 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6590+0+0 (secure 0 0 0) 0x7f71c808b1b0 con 0x7f71d40b35f0 2024-04-24T23:16:30.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.656+0000 7f71d8e00640 1 --2- 172.21.15.6:0/854134303 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f71b806be30 0x7f71b806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:30.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.657+0000 7f71d8e00640 1 --2- 172.21.15.6:0/854134303 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f71b806be30 0x7f71b806e2f0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f71c4034000 tx=0x7f71c402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:30.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.660+0000 7f71d1600640 1 -- 172.21.15.6:0/854134303 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f71c8059070 con 0x7f71d40b35f0 2024-04-24T23:16:30.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.858+0000 7f71dac00640 1 -- 172.21.15.6:0/854134303 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f71d40b2a60 con 0x7f71d40b35f0 2024-04-24T23:16:30.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.859+0000 7f71d1600640 1 -- 172.21.15.6:0/854134303 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) ==== 103+0+11 (secure 0 0 0) 0x7f71c805d370 con 0x7f71d40b35f0 2024-04-24T23:16:30.860 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 32 2024-04-24T23:16:30.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.861+0000 7f71dac00640 1 -- 172.21.15.6:0/854134303 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f71b806be30 msgr2=0x7f71b806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:30.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.861+0000 7f71dac00640 1 --2- 172.21.15.6:0/854134303 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f71b806be30 0x7f71b806e2f0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f71c4034000 tx=0x7f71c402e000 comp rx=0 tx=0).stop 2024-04-24T23:16:30.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.861+0000 7f71dac00640 1 -- 172.21.15.6:0/854134303 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f71d40b35f0 msgr2=0x7f71d40cf5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:30.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.861+0000 7f71dac00640 1 --2- 172.21.15.6:0/854134303 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f71d40b35f0 0x7f71d40cf5b0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f71c800ebe0 tx=0x7f71c800c520 comp rx=0 tx=0).stop 2024-04-24T23:16:30.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.861+0000 7f71dac00640 1 -- 172.21.15.6:0/854134303 shutdown_connections 2024-04-24T23:16:30.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.861+0000 7f71dac00640 1 --2- 172.21.15.6:0/854134303 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f71b806be30 0x7f71b806e2f0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:30.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.861+0000 7f71dac00640 1 --2- 172.21.15.6:0/854134303 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f71d4154f60 0x7f71d415d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:30.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.861+0000 7f71dac00640 1 --2- 172.21.15.6:0/854134303 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f71d415ab20 0x7f71d415daa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:30.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.861+0000 7f71dac00640 1 --2- 172.21.15.6:0/854134303 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f71d40b35f0 0x7f71d40cf5b0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:30.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.861+0000 7f71dac00640 1 -- 172.21.15.6:0/854134303 >> 172.21.15.6:0/854134303 conn(0x7f71d40674d0 msgr2=0x7f71d40cb540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:30.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.862+0000 7f71dac00640 1 -- 172.21.15.6:0/854134303 shutdown_connections 2024-04-24T23:16:30.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:30.862+0000 7f71dac00640 1 -- 172.21.15.6:0/854134303 wait complete. 2024-04-24T23:16:30.879 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:16:31.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:31.071+0000 7f06df200640 1 Processor -- start 2024-04-24T23:16:31.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:31.071+0000 7f06df200640 1 -- start start 2024-04-24T23:16:31.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:31.073+0000 7f06df200640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f06e0152bd0 0x7f06e0158e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:31.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:31.073+0000 7f06df200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f06e006d390 con 0x7f06e00c6be0 2024-04-24T23:16:31.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:31.073+0000 7f06df200640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f06e006dbc0 con 0x7f06e006ce30 2024-04-24T23:16:31.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:31.073+0000 7f06df200640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f06e006d970 con 0x7f06e0152bd0 2024-04-24T23:16:31.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:31.073+0000 7f06de800640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f06e00c6be0 0x7f06e0163090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:57120/0 (socket says 172.21.15.6:57120) 2024-04-24T23:16:31.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:31.073+0000 7f06de800640 1 -- 172.21.15.6:0/3236365973 learned_addr learned my addr 172.21.15.6:0/3236365973 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:16:31.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:31.073+0000 7f06dde00640 1 --2- 172.21.15.6:0/3236365973 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f06e0152bd0 0x7f06e0158e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:31.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:31.074+0000 7f06dca00640 1 -- 172.21.15.6:0/3236365973 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 411561617 0 0) 0x7f06e006dbc0 con 0x7f06e006ce30 2024-04-24T23:16:31.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:31.074+0000 7f06dca00640 1 -- 172.21.15.6:0/3236365973 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f06c4003520 con 0x7f06e006ce30 2024-04-24T23:16:31.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:31.074+0000 7f06dca00640 1 -- 172.21.15.6:0/3236365973 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 123620859 0 0) 0x7f06e006d390 con 0x7f06e00c6be0 2024-04-24T23:16:31.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:31.074+0000 7f06dca00640 1 -- 172.21.15.6:0/3236365973 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f06e006dbc0 con 0x7f06e00c6be0 2024-04-24T23:16:31.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:31.074+0000 7f06dde00640 1 -- 172.21.15.6:0/3236365973 >> v1:172.21.15.190:6789/0 conn(0x7f06e006ce30 legacy=0x7f06e00cc600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:31.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:31.074+0000 7f06dde00640 1 -- 172.21.15.6:0/3236365973 >> v1:172.21.15.6:6789/0 conn(0x7f06e00c6be0 legacy=0x7f06e0163090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:31.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:31.074+0000 7f06dde00640 1 -- 172.21.15.6:0/3236365973 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06e006dea0 con 0x7f06e0152bd0 2024-04-24T23:16:31.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:31.076+0000 7f06dde00640 1 --2- 172.21.15.6:0/3236365973 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f06e0152bd0 0x7f06e0158e40 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f06d00097d0 tx=0x7f06d002a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f1b0cfd0d777cf77 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:31.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:31.076+0000 7f06dca00640 1 -- 172.21.15.6:0/3236365973 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06d0037070 con 0x7f06e0152bd0 2024-04-24T23:16:31.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:31.076+0000 7f06dca00640 1 -- 172.21.15.6:0/3236365973 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f06d0032440 con 0x7f06e0152bd0 2024-04-24T23:16:31.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:31.076+0000 7f06dca00640 1 -- 172.21.15.6:0/3236365973 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06d00327f0 con 0x7f06e0152bd0 2024-04-24T23:16:31.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:31.077+0000 7f06df200640 1 -- 172.21.15.6:0/3236365973 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f06e0152bd0 msgr2=0x7f06e0158e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:31.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:31.077+0000 7f06df200640 1 --2- 172.21.15.6:0/3236365973 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f06e0152bd0 0x7f06e0158e40 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f06d00097d0 tx=0x7f06d002a180 comp rx=0 tx=0).stop 2024-04-24T23:16:31.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:31.077+0000 7f06df200640 1 -- 172.21.15.6:0/3236365973 shutdown_connections 2024-04-24T23:16:31.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:31.077+0000 7f06df200640 1 --2- 172.21.15.6:0/3236365973 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f06e0152bd0 0x7f06e0158e40 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:31.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:31.077+0000 7f06df200640 1 -- 172.21.15.6:0/3236365973 >> 172.21.15.6:0/3236365973 conn(0x7f06e006ed80 msgr2=0x7f06e006f190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:31.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:31.077+0000 7f06df200640 1 -- 172.21.15.6:0/3236365973 shutdown_connections 2024-04-24T23:16:31.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:31.077+0000 7f06df200640 1 -- 172.21.15.6:0/3236365973 wait complete. 2024-04-24T23:16:31.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:31.078+0000 7f06df200640 1 Processor -- start 2024-04-24T23:16:31.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:31.079+0000 7f06df200640 1 -- start start 2024-04-24T23:16:31.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:31.079+0000 7f06df200640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f06e006ce30 0x7f06e01f0d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:31.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:31.079+0000 7f06df200640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f06e00c6be0 0x7f06e01598d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:31.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:31.079+0000 7f06df200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f06e0152bd0 0x7f06e015c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:31.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:31.079+0000 7f06df200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f06e0072650 con 0x7f06e0152bd0 2024-04-24T23:16:31.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:31.079+0000 7f06df200640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f06e0071890 con 0x7f06e006ce30 2024-04-24T23:16:31.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:31.079+0000 7f06df200640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f06e00727d0 con 0x7f06e00c6be0 2024-04-24T23:16:31.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:31.080+0000 7f06de800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f06e0152bd0 0x7f06e015c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:31.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:31.080+0000 7f06de800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f06e0152bd0 0x7f06e015c3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:35578/0 (socket says 172.21.15.6:35578) 2024-04-24T23:16:31.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:31.080+0000 7f06de800640 1 -- 172.21.15.6:0/3131135673 learned_addr learned my addr 172.21.15.6:0/3131135673 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:16:31.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:31.080+0000 7f06dd400640 1 --2- 172.21.15.6:0/3131135673 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f06e00c6be0 0x7f06e01598d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:31.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:31.080+0000 7f06dde00640 1 --2- 172.21.15.6:0/3131135673 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f06e006ce30 0x7f06e01f0d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:31.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:31.080+0000 7f06de800640 1 -- 172.21.15.6:0/3131135673 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f06e00c6be0 msgr2=0x7f06e01598d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:31.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:31.080+0000 7f06de800640 1 --2- 172.21.15.6:0/3131135673 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f06e00c6be0 0x7f06e01598d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:31.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:31.080+0000 7f06de800640 1 -- 172.21.15.6:0/3131135673 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f06e006ce30 msgr2=0x7f06e01f0d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:31.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:31.080+0000 7f06de800640 1 --2- 172.21.15.6:0/3131135673 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f06e006ce30 0x7f06e01f0d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:31.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:31.080+0000 7f06de800640 1 -- 172.21.15.6:0/3131135673 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06e0153000 con 0x7f06e0152bd0 2024-04-24T23:16:31.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:31.080+0000 7f06de800640 1 --2- 172.21.15.6:0/3131135673 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f06e0152bd0 0x7f06e015c3f0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f06d800caa0 tx=0x7f06d800cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:31.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:31.081+0000 7f06d6000640 1 -- 172.21.15.6:0/3131135673 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06d8017070 con 0x7f06e0152bd0 2024-04-24T23:16:31.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:31.081+0000 7f06d6000640 1 -- 172.21.15.6:0/3131135673 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f06d800aba0 con 0x7f06e0152bd0 2024-04-24T23:16:31.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:31.081+0000 7f06d6000640 1 -- 172.21.15.6:0/3131135673 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06d8012400 con 0x7f06e0152bd0 2024-04-24T23:16:31.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:31.081+0000 7f06df200640 1 -- 172.21.15.6:0/3131135673 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06e01532e0 con 0x7f06e0152bd0 2024-04-24T23:16:31.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:31.081+0000 7f06df200640 1 -- 172.21.15.6:0/3131135673 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06e01523d0 con 0x7f06e0152bd0 2024-04-24T23:16:31.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:31.082+0000 7f06df200640 1 -- 172.21.15.6:0/3131135673 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f06e0075b30 con 0x7f06e0152bd0 2024-04-24T23:16:31.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:31.083+0000 7f06d6000640 1 -- 172.21.15.6:0/3131135673 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f06d801a070 con 0x7f06e0152bd0 2024-04-24T23:16:31.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:31.084+0000 7f06d6000640 1 --2- 172.21.15.6:0/3131135673 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f06c006be30 0x7f06c006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:31.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:31.084+0000 7f06d6000640 1 -- 172.21.15.6:0/3131135673 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6590+0+0 (secure 0 0 0) 0x7f06d808af90 con 0x7f06e0152bd0 2024-04-24T23:16:31.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:31.084+0000 7f06dde00640 1 --2- 172.21.15.6:0/3131135673 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f06c006be30 0x7f06c006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:31.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:31.085+0000 7f06dde00640 1 --2- 172.21.15.6:0/3131135673 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f06c006be30 0x7f06c006e2f0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f06d0034000 tx=0x7f06d002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:31.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:31.088+0000 7f06d6000640 1 -- 172.21.15.6:0/3131135673 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f06d8058ee0 con 0x7f06e0152bd0 2024-04-24T23:16:31.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:31.334+0000 7f06df200640 1 -- 172.21.15.6:0/3131135673 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f06e006d750 con 0x7f06e0152bd0 2024-04-24T23:16:31.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:31.335+0000 7f06d6000640 1 -- 172.21.15.6:0/3131135673 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+6898 (secure 0 0 0) 0x7f06d805d1e0 con 0x7f06e0152bd0 2024-04-24T23:16:31.336 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:16:31.337 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4197,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/105267896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":105267896},{"type":"v1","addr":"172.21.15.6:6834","nonce":105267896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:16:26.281859+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4282,"mds_2":4235,"mds_3":4232,"mds_4":4203},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4203":{"gid":4203,"name":"e","rank":4,"incarnation":17,"state":"up:creating","state_seq":1,"addr":"172.21.15.6:6837/2590421094","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":2590421094},{"type":"v1","addr":"172.21.15.6:6837","nonce":2590421094}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4232":{"gid":4232,"name":"b","rank":3,"incarnation":15,"state":"up:active","state_seq":15,"addr":"172.21.15.190:6838/2032280158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2032280158},{"type":"v1","addr":"172.21.15.190:6838","nonce":2032280158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4235":{"gid":4235,"name":"d","rank":2,"incarnation":13,"state":"up:active","state_seq":14,"addr":"172.21.15.190:6839/1572872451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":1572872451},{"type":"v1","addr":"172.21.15.190:6839","nonce":1572872451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4282":{"gid":4282,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":12,"addr":"172.21.15.190:6837/580315269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":580315269},{"type":"v1","addr":"172.21.15.190:6837","nonce":580315269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/489092393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":489092393},{"type":"v1","addr":"172.21.15.6:6835","nonce":489092393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4306,"qdb_cluster":[4306,4282,4235,4232]},"id":1}]} 2024-04-24T23:16:31.337 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 17 2024-04-24T23:16:31.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:31.338+0000 7f06df200640 1 -- 172.21.15.6:0/3131135673 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f06c006be30 msgr2=0x7f06c006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:31.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:31.338+0000 7f06df200640 1 --2- 172.21.15.6:0/3131135673 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f06c006be30 0x7f06c006e2f0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f06d0034000 tx=0x7f06d002e000 comp rx=0 tx=0).stop 2024-04-24T23:16:31.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:31.338+0000 7f06df200640 1 -- 172.21.15.6:0/3131135673 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f06e0152bd0 msgr2=0x7f06e015c3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:31.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:31.338+0000 7f06df200640 1 --2- 172.21.15.6:0/3131135673 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f06e0152bd0 0x7f06e015c3f0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f06d800caa0 tx=0x7f06d800cf60 comp rx=0 tx=0).stop 2024-04-24T23:16:31.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:31.338+0000 7f06df200640 1 -- 172.21.15.6:0/3131135673 shutdown_connections 2024-04-24T23:16:31.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:31.338+0000 7f06df200640 1 --2- 172.21.15.6:0/3131135673 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f06c006be30 0x7f06c006e2f0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:31.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:31.338+0000 7f06df200640 1 --2- 172.21.15.6:0/3131135673 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f06e00c6be0 0x7f06e01598d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:31.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:31.338+0000 7f06df200640 1 --2- 172.21.15.6:0/3131135673 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f06e0152bd0 0x7f06e015c3f0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:31.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:31.338+0000 7f06df200640 1 --2- 172.21.15.6:0/3131135673 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f06e006ce30 0x7f06e01f0d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:31.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:31.338+0000 7f06df200640 1 -- 172.21.15.6:0/3131135673 >> 172.21.15.6:0/3131135673 conn(0x7f06e006ed80 msgr2=0x7f06e01519a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:31.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:31.338+0000 7f06df200640 1 -- 172.21.15.6:0/3131135673 shutdown_connections 2024-04-24T23:16:31.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:31.338+0000 7f06df200640 1 -- 172.21.15.6:0/3131135673 wait complete. 2024-04-24T23:16:31.356 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T23:16:31.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:31.538+0000 7f2c23200640 1 Processor -- start 2024-04-24T23:16:31.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:31.538+0000 7f2c23200640 1 -- start start 2024-04-24T23:16:31.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:31.539+0000 7f2c23200640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2c240cb090 0x7f2c240d0820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:31.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:31.539+0000 7f2c23200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2c24155170 con 0x7f2c24154d90 2024-04-24T23:16:31.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:31.539+0000 7f2c23200640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2c240cc9a0 con 0x7f2c240d0d60 2024-04-24T23:16:31.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:31.539+0000 7f2c23200640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f2c2415ad60 con 0x7f2c240cb090 2024-04-24T23:16:31.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:31.540+0000 7f2c21e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f2c24154d90 0x7f2c240cbe80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:57132/0 (socket says 172.21.15.6:57132) 2024-04-24T23:16:31.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:31.540+0000 7f2c21e00640 1 -- 172.21.15.6:0/140974754 learned_addr learned my addr 172.21.15.6:0/140974754 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:16:31.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:31.540+0000 7f2c21400640 1 --2- 172.21.15.6:0/140974754 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2c240cb090 0x7f2c240d0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:31.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:31.540+0000 7f2c21400640 1 -- 172.21.15.6:0/140974754 >> v1:172.21.15.190:6789/0 conn(0x7f2c240d0d60 legacy=0x7f2c2415e000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:31.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:31.540+0000 7f2c20a00640 1 -- 172.21.15.6:0/140974754 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 248716757 0 0) 0x7f2c240cc9a0 con 0x7f2c240d0d60 2024-04-24T23:16:31.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:31.540+0000 7f2c21400640 1 -- 172.21.15.6:0/140974754 >> v1:172.21.15.6:6789/0 conn(0x7f2c24154d90 legacy=0x7f2c240cbe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:31.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:31.540+0000 7f2c21400640 1 -- 172.21.15.6:0/140974754 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c2415b040 con 0x7f2c240cb090 2024-04-24T23:16:31.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:31.541+0000 7f2c21400640 1 --2- 172.21.15.6:0/140974754 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2c240cb090 0x7f2c240d0820 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f2c0c0097d0 tx=0x7f2c0c02a1a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3c4c3f7c7287d0a3 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:31.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:31.542+0000 7f2c20a00640 1 -- 172.21.15.6:0/140974754 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c0c037070 con 0x7f2c240cb090 2024-04-24T23:16:31.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:31.542+0000 7f2c20a00640 1 -- 172.21.15.6:0/140974754 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2c0c02ae30 con 0x7f2c240cb090 2024-04-24T23:16:31.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:31.542+0000 7f2c20a00640 1 -- 172.21.15.6:0/140974754 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c0c032640 con 0x7f2c240cb090 2024-04-24T23:16:31.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:31.542+0000 7f2c23200640 1 -- 172.21.15.6:0/140974754 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2c240cb090 msgr2=0x7f2c240d0820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:31.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:31.542+0000 7f2c23200640 1 --2- 172.21.15.6:0/140974754 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2c240cb090 0x7f2c240d0820 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f2c0c0097d0 tx=0x7f2c0c02a1a0 comp rx=0 tx=0).stop 2024-04-24T23:16:31.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:31.542+0000 7f2c23200640 1 -- 172.21.15.6:0/140974754 shutdown_connections 2024-04-24T23:16:31.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:31.542+0000 7f2c23200640 1 --2- 172.21.15.6:0/140974754 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2c240cb090 0x7f2c240d0820 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:31.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:31.542+0000 7f2c23200640 1 -- 172.21.15.6:0/140974754 >> 172.21.15.6:0/140974754 conn(0x7f2c2406be10 msgr2=0x7f2c2406c1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:31.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:31.542+0000 7f2c23200640 1 -- 172.21.15.6:0/140974754 shutdown_connections 2024-04-24T23:16:31.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:31.542+0000 7f2c23200640 1 -- 172.21.15.6:0/140974754 wait complete. 2024-04-24T23:16:31.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:31.544+0000 7f2c23200640 1 Processor -- start 2024-04-24T23:16:31.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:31.544+0000 7f2c23200640 1 -- start start 2024-04-24T23:16:31.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:31.545+0000 7f2c23200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2c240cb090 0x7f2c241f3c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:31.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:31.545+0000 7f2c23200640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2c240d0d60 0x7f2c2414c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:31.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:31.545+0000 7f2c23200640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2c24154d90 0x7f2c24158bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:31.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:31.545+0000 7f2c23200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f2c2415aa30 con 0x7f2c240cb090 2024-04-24T23:16:31.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:31.545+0000 7f2c23200640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f2c2415b600 con 0x7f2c240d0d60 2024-04-24T23:16:31.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:31.545+0000 7f2c23200640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f2c240cc590 con 0x7f2c24154d90 2024-04-24T23:16:31.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:31.546+0000 7f2c21e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2c240cb090 0x7f2c241f3c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:31.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:31.546+0000 7f2c21400640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2c240d0d60 0x7f2c2414c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:31.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:31.546+0000 7f2c21e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2c240cb090 0x7f2c241f3c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:35590/0 (socket says 172.21.15.6:35590) 2024-04-24T23:16:31.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:31.546+0000 7f2c21e00640 1 -- 172.21.15.6:0/1421484340 learned_addr learned my addr 172.21.15.6:0/1421484340 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:16:31.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:31.546+0000 7f2c22800640 1 --2- 172.21.15.6:0/1421484340 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2c24154d90 0x7f2c24158bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:31.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:31.546+0000 7f2c21e00640 1 -- 172.21.15.6:0/1421484340 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2c24154d90 msgr2=0x7f2c24158bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:31.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:31.546+0000 7f2c21e00640 1 --2- 172.21.15.6:0/1421484340 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2c24154d90 0x7f2c24158bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:31.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:31.546+0000 7f2c21e00640 1 -- 172.21.15.6:0/1421484340 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2c240d0d60 msgr2=0x7f2c2414c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:31.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:31.546+0000 7f2c21e00640 1 --2- 172.21.15.6:0/1421484340 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2c240d0d60 0x7f2c2414c5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:31.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:31.546+0000 7f2c21e00640 1 -- 172.21.15.6:0/1421484340 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c2415a050 con 0x7f2c240cb090 2024-04-24T23:16:31.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:31.546+0000 7f2c21400640 1 --2- 172.21.15.6:0/1421484340 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2c240d0d60 0x7f2c2414c5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:16:31.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:31.547+0000 7f2c21e00640 1 --2- 172.21.15.6:0/1421484340 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2c240cb090 0x7f2c241f3c00 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f2c1400c910 tx=0x7f2c1400cdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:31.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:31.547+0000 7f2c1a000640 1 -- 172.21.15.6:0/1421484340 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c14018070 con 0x7f2c240cb090 2024-04-24T23:16:31.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:31.547+0000 7f2c1a000640 1 -- 172.21.15.6:0/1421484340 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2c1400a700 con 0x7f2c240cb090 2024-04-24T23:16:31.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:31.547+0000 7f2c23200640 1 -- 172.21.15.6:0/1421484340 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c241597b0 con 0x7f2c240cb090 2024-04-24T23:16:31.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:31.547+0000 7f2c1a000640 1 -- 172.21.15.6:0/1421484340 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c1400aaf0 con 0x7f2c240cb090 2024-04-24T23:16:31.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:31.547+0000 7f2c23200640 1 -- 172.21.15.6:0/1421484340 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c24159d70 con 0x7f2c240cb090 2024-04-24T23:16:31.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:31.548+0000 7f2c23200640 1 -- 172.21.15.6:0/1421484340 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2c24075b30 con 0x7f2c240cb090 2024-04-24T23:16:31.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:31.549+0000 7f2c1a000640 1 -- 172.21.15.6:0/1421484340 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f2c1401b070 con 0x7f2c240cb090 2024-04-24T23:16:31.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:31.551+0000 7f2c1a000640 1 --2- 172.21.15.6:0/1421484340 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f2c0006be60 0x7f2c0006e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:31.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:31.551+0000 7f2c1a000640 1 -- 172.21.15.6:0/1421484340 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6590+0+0 (secure 0 0 0) 0x7f2c1408a840 con 0x7f2c240cb090 2024-04-24T23:16:31.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:31.551+0000 7f2c21400640 1 --2- 172.21.15.6:0/1421484340 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f2c0006be60 0x7f2c0006e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:31.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:31.552+0000 7f2c21400640 1 --2- 172.21.15.6:0/1421484340 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f2c0006be60 0x7f2c0006e320 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f2c0c034000 tx=0x7f2c0c02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:31.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:31.556+0000 7f2c1a000640 1 -- 172.21.15.6:0/1421484340 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f2c14058820 con 0x7f2c240cb090 2024-04-24T23:16:31.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:31.763+0000 7f2c23200640 1 -- 172.21.15.6:0/1421484340 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2c240767b0 con 0x7f2c240cb090 2024-04-24T23:16:31.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:31.766+0000 7f2c1a000640 1 -- 172.21.15.6:0/1421484340 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+18554 (secure 0 0 0) 0x7f2c1405cb20 con 0x7f2c240cb090 2024-04-24T23:16:31.768 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:16:31.768 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":26,"fsid":"18bef68d-5e6d-4ec5-862b-b6f206e2588d","created":"2024-04-24T23:15:12.535072+0000","modified":"2024-04-24T23:16:18.218008+0000","last_up_change":"2024-04-24T23:15:23.773223+0000","last_in_change":"2024-04-24T23:15:14.905328+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T23:15:24.747441+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-24T23:15:34.051800+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-24T23:15:35.436870+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-24T23:15:39.034257+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"3b7a212d-d498-462e-90bf-ed493c803a09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":3022569437},{"type":"v1","addr":"172.21.15.6:6809","nonce":3022569437}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":3022569437},{"type":"v1","addr":"172.21.15.6:6811","nonce":3022569437}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":3022569437},{"type":"v1","addr":"172.21.15.6:6815","nonce":3022569437}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":3022569437},{"type":"v1","addr":"172.21.15.6:6813","nonce":3022569437}]},"public_addr":"172.21.15.6:6809/3022569437","cluster_addr":"172.21.15.6:6811/3022569437","heartbeat_back_addr":"172.21.15.6:6815/3022569437","heartbeat_front_addr":"172.21.15.6:6813/3022569437","state":["exists","up"]},{"osd":1,"uuid":"7c97f5a2-1735-4775-9f15-eb50099b19d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":3687836888},{"type":"v1","addr":"172.21.15.6:6817","nonce":3687836888}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":3687836888},{"type":"v1","addr":"172.21.15.6:6819","nonce":3687836888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":3687836888},{"type":"v1","addr":"172.21.15.6:6823","nonce":3687836888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":3687836888},{"type":"v1","addr":"172.21.15.6:6821","nonce":3687836888}]},"public_addr":"172.21.15.6:6817/3687836888","cluster_addr":"172.21.15.6:6819/3687836888","heartbeat_back_addr":"172.21.15.6:6823/3687836888","heartbeat_front_addr":"172.21.15.6:6821/3687836888","state":["exists","up"]},{"osd":2,"uuid":"d86bbc18-438c-4ec2-8018-ba7dd78f790b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":1020213724},{"type":"v1","addr":"172.21.15.6:6801","nonce":1020213724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":1020213724},{"type":"v1","addr":"172.21.15.6:6803","nonce":1020213724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":1020213724},{"type":"v1","addr":"172.21.15.6:6807","nonce":1020213724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":1020213724},{"type":"v1","addr":"172.21.15.6:6805","nonce":1020213724}]},"public_addr":"172.21.15.6:6801/1020213724","cluster_addr":"172.21.15.6:6803/1020213724","heartbeat_back_addr":"172.21.15.6:6807/1020213724","heartbeat_front_addr":"172.21.15.6:6805/1020213724","state":["exists","up"]},{"osd":3,"uuid":"3754ae16-4248-40dd-95f9-0f59c0eb5426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6824","nonce":3163883468},{"type":"v1","addr":"172.21.15.6:6825","nonce":3163883468}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":3163883468},{"type":"v1","addr":"172.21.15.6:6827","nonce":3163883468}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":3163883468},{"type":"v1","addr":"172.21.15.6:6831","nonce":3163883468}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":3163883468},{"type":"v1","addr":"172.21.15.6:6829","nonce":3163883468}]},"public_addr":"172.21.15.6:6825/3163883468","cluster_addr":"172.21.15.6:6827/3163883468","heartbeat_back_addr":"172.21.15.6:6831/3163883468","heartbeat_front_addr":"172.21.15.6:6829/3163883468","state":["exists","up"]},{"osd":4,"uuid":"8b2913a0-d006-479c-84ed-5b21e0e6c392","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":1989500934},{"type":"v1","addr":"172.21.15.190:6809","nonce":1989500934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":1989500934},{"type":"v1","addr":"172.21.15.190:6811","nonce":1989500934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":1989500934},{"type":"v1","addr":"172.21.15.190:6815","nonce":1989500934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":1989500934},{"type":"v1","addr":"172.21.15.190:6813","nonce":1989500934}]},"public_addr":"172.21.15.190:6809/1989500934","cluster_addr":"172.21.15.190:6811/1989500934","heartbeat_back_addr":"172.21.15.190:6815/1989500934","heartbeat_front_addr":"172.21.15.190:6813/1989500934","state":["exists","up"]},{"osd":5,"uuid":"499b79d0-0585-4dae-ab69-196a551793da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":3687548412},{"type":"v1","addr":"172.21.15.190:6801","nonce":3687548412}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":3687548412},{"type":"v1","addr":"172.21.15.190:6803","nonce":3687548412}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":3687548412},{"type":"v1","addr":"172.21.15.190:6807","nonce":3687548412}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":3687548412},{"type":"v1","addr":"172.21.15.190:6805","nonce":3687548412}]},"public_addr":"172.21.15.190:6801/3687548412","cluster_addr":"172.21.15.190:6803/3687548412","heartbeat_back_addr":"172.21.15.190:6807/3687548412","heartbeat_front_addr":"172.21.15.190:6805/3687548412","state":["exists","up"]},{"osd":6,"uuid":"6ba6df3b-4a6b-4a89-a890-0f79f387d4d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":870571564},{"type":"v1","addr":"172.21.15.190:6817","nonce":870571564}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":870571564},{"type":"v1","addr":"172.21.15.190:6819","nonce":870571564}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":870571564},{"type":"v1","addr":"172.21.15.190:6823","nonce":870571564}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":870571564},{"type":"v1","addr":"172.21.15.190:6821","nonce":870571564}]},"public_addr":"172.21.15.190:6817/870571564","cluster_addr":"172.21.15.190:6819/870571564","heartbeat_back_addr":"172.21.15.190:6823/870571564","heartbeat_front_addr":"172.21.15.190:6821/870571564","state":["exists","up"]},{"osd":7,"uuid":"e9668b1c-a000-4332-86e8-4efd73b02409","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":123639197},{"type":"v1","addr":"172.21.15.190:6825","nonce":123639197}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":123639197},{"type":"v1","addr":"172.21.15.190:6827","nonce":123639197}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":123639197},{"type":"v1","addr":"172.21.15.190:6831","nonce":123639197}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":123639197},{"type":"v1","addr":"172.21.15.190:6829","nonce":123639197}]},"public_addr":"172.21.15.190:6825/123639197","cluster_addr":"172.21.15.190:6827/123639197","heartbeat_back_addr":"172.21.15.190:6831/123639197","heartbeat_front_addr":"172.21.15.190:6829/123639197","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.321646+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.363873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.466963+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.531989+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:19.990707+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.043543+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.032492+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.115004+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.2f","mappings":[{"from":0,"to":7}]},{"pgid":"4.19","mappings":[{"from":7,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T23:16:31.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:31.769+0000 7f2c23200640 1 -- 172.21.15.6:0/1421484340 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f2c0006be60 msgr2=0x7f2c0006e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:31.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:31.769+0000 7f2c23200640 1 --2- 172.21.15.6:0/1421484340 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f2c0006be60 0x7f2c0006e320 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f2c0c034000 tx=0x7f2c0c02e000 comp rx=0 tx=0).stop 2024-04-24T23:16:31.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:31.769+0000 7f2c23200640 1 -- 172.21.15.6:0/1421484340 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2c240cb090 msgr2=0x7f2c241f3c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:31.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:31.769+0000 7f2c23200640 1 --2- 172.21.15.6:0/1421484340 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2c240cb090 0x7f2c241f3c00 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f2c1400c910 tx=0x7f2c1400cdd0 comp rx=0 tx=0).stop 2024-04-24T23:16:31.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:31.769+0000 7f2c23200640 1 -- 172.21.15.6:0/1421484340 shutdown_connections 2024-04-24T23:16:31.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:31.769+0000 7f2c23200640 1 --2- 172.21.15.6:0/1421484340 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f2c0006be60 0x7f2c0006e320 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:31.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:31.769+0000 7f2c23200640 1 --2- 172.21.15.6:0/1421484340 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2c24154d90 0x7f2c24158bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:31.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:31.769+0000 7f2c23200640 1 --2- 172.21.15.6:0/1421484340 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2c240d0d60 0x7f2c2414c5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:31.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:31.769+0000 7f2c23200640 1 --2- 172.21.15.6:0/1421484340 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2c240cb090 0x7f2c241f3c00 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:31.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:31.769+0000 7f2c23200640 1 -- 172.21.15.6:0/1421484340 >> 172.21.15.6:0/1421484340 conn(0x7f2c2406be10 msgr2=0x7f2c2406c1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:31.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:31.769+0000 7f2c23200640 1 -- 172.21.15.6:0/1421484340 shutdown_connections 2024-04-24T23:16:31.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:31.769+0000 7f2c23200640 1 -- 172.21.15.6:0/1421484340 wait complete. 2024-04-24T23:16:31.790 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T23:15:24.747441+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-24T23:15:34.051800+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-24T23:15:35.436870+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-24T23:15:39.034257+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-24T23:16:31.790 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-24T23:16:32.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:31.999+0000 7fb311800640 1 Processor -- start 2024-04-24T23:16:32.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:31.999+0000 7fb311800640 1 -- start start 2024-04-24T23:16:32.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.001+0000 7fb311800640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fb30c0c5e40 0x7fb30c0c6ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:32.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.001+0000 7fb311800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fb30c0c8570 0x7fb30c1524a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:32.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.001+0000 7fb311800640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb30c0cca80 con 0x7fb30c0c5a60 2024-04-24T23:16:32.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.001+0000 7fb311800640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7fb30c15c440 con 0x7fb30c0c8570 2024-04-24T23:16:32.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.001+0000 7fb311800640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7fb30c0ccd60 con 0x7fb30c0c5e40 2024-04-24T23:16:32.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.001+0000 7fb310e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fb30c0c8570 0x7fb30c1524a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:32.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.001+0000 7fb30be00640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7fb30c0c5a60 0x7fb30c15bd30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.6:43660/0 (socket says 172.21.15.6:43660) 2024-04-24T23:16:32.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.001+0000 7fb30be00640 1 -- 172.21.15.6:0/2974845910 learned_addr learned my addr 172.21.15.6:0/2974845910 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:16:32.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.001+0000 7fb30b400640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fb30c0c5e40 0x7fb30c0c6ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:32.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.002+0000 7fb30b400640 1 -- 172.21.15.6:0/2974845910 >> v2:172.21.15.6:3300/0 conn(0x7fb30c0c8570 msgr2=0x7fb30c1524a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:16:32.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.002+0000 7fb30b400640 1 --2- 172.21.15.6:0/2974845910 >> v2:172.21.15.6:3300/0 conn(0x7fb30c0c8570 0x7fb30c1524a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:32.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.002+0000 7fb30b400640 1 -- 172.21.15.6:0/2974845910 >> v1:172.21.15.190:6789/0 conn(0x7fb30c0c5a60 legacy=0x7fb30c15bd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:32.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.002+0000 7fb30b400640 1 -- 172.21.15.6:0/2974845910 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb30c06eab0 con 0x7fb30c0c5e40 2024-04-24T23:16:32.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.002+0000 7fb30aa00640 1 -- 172.21.15.6:0/2974845910 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1139060894 0 0) 0x7fb30c0cca80 con 0x7fb30c0c5a60 2024-04-24T23:16:32.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.002+0000 7fb310e00640 1 --2- 172.21.15.6:0/2974845910 >> v2:172.21.15.6:3300/0 conn(0x7fb30c0c8570 0x7fb30c1524a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:16:32.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.003+0000 7fb30b400640 1 --2- 172.21.15.6:0/2974845910 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb30c0c5e40 0x7fb30c0c6ae0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fb30c0cc8e0 tx=0x7fb2f802a1a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=611b3b210122a664 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:32.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.003+0000 7fb30aa00640 1 -- 172.21.15.6:0/2974845910 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2f8037070 con 0x7fb30c0c5e40 2024-04-24T23:16:32.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.003+0000 7fb30aa00640 1 -- 172.21.15.6:0/2974845910 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb2f8032440 con 0x7fb30c0c5e40 2024-04-24T23:16:32.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.003+0000 7fb30aa00640 1 -- 172.21.15.6:0/2974845910 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2f8032810 con 0x7fb30c0c5e40 2024-04-24T23:16:32.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.003+0000 7fb311800640 1 -- 172.21.15.6:0/2974845910 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb30c0c5e40 msgr2=0x7fb30c0c6ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:32.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.003+0000 7fb311800640 1 --2- 172.21.15.6:0/2974845910 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb30c0c5e40 0x7fb30c0c6ae0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fb30c0cc8e0 tx=0x7fb2f802a1a0 comp rx=0 tx=0).stop 2024-04-24T23:16:32.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.004+0000 7fb311800640 1 -- 172.21.15.6:0/2974845910 shutdown_connections 2024-04-24T23:16:32.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.004+0000 7fb311800640 1 --2- 172.21.15.6:0/2974845910 >> v2:172.21.15.6:3300/0 conn(0x7fb30c0c8570 0x7fb30c1524a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:32.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.004+0000 7fb311800640 1 --2- 172.21.15.6:0/2974845910 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb30c0c5e40 0x7fb30c0c6ae0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:32.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.004+0000 7fb311800640 1 -- 172.21.15.6:0/2974845910 >> 172.21.15.6:0/2974845910 conn(0x7fb30c06cc40 msgr2=0x7fb30c06d050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:32.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.004+0000 7fb311800640 1 -- 172.21.15.6:0/2974845910 shutdown_connections 2024-04-24T23:16:32.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.004+0000 7fb311800640 1 -- 172.21.15.6:0/2974845910 wait complete. 2024-04-24T23:16:32.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.005+0000 7fb311800640 1 Processor -- start 2024-04-24T23:16:32.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.005+0000 7fb311800640 1 -- start start 2024-04-24T23:16:32.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.006+0000 7fb311800640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb30c0c5a60 0x7fb30c14dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:32.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.006+0000 7fb311800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb30c0c5e40 0x7fb30c0ca8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:32.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.006+0000 7fb311800640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb30c0c8570 0x7fb30c1e1c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:32.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.006+0000 7fb311800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fb30c06f590 con 0x7fb30c0c5e40 2024-04-24T23:16:32.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.006+0000 7fb311800640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fb30c06f070 con 0x7fb30c0c5a60 2024-04-24T23:16:32.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.006+0000 7fb311800640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fb30c14e9a0 con 0x7fb30c0c8570 2024-04-24T23:16:32.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.006+0000 7fb30b400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb30c0c5e40 0x7fb30c0ca8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:32.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.007+0000 7fb30b400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb30c0c5e40 0x7fb30c0ca8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:35608/0 (socket says 172.21.15.6:35608) 2024-04-24T23:16:32.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.007+0000 7fb30b400640 1 -- 172.21.15.6:0/3806403795 learned_addr learned my addr 172.21.15.6:0/3806403795 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:16:32.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.007+0000 7fb310e00640 1 --2- 172.21.15.6:0/3806403795 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb30c0c8570 0x7fb30c1e1c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:32.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.007+0000 7fb30be00640 1 --2- 172.21.15.6:0/3806403795 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb30c0c5a60 0x7fb30c14dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:32.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.007+0000 7fb30b400640 1 -- 172.21.15.6:0/3806403795 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb30c0c8570 msgr2=0x7fb30c1e1c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:32.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.007+0000 7fb30b400640 1 --2- 172.21.15.6:0/3806403795 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb30c0c8570 0x7fb30c1e1c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:32.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.007+0000 7fb30b400640 1 -- 172.21.15.6:0/3806403795 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb30c0c5a60 msgr2=0x7fb30c14dda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:32.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.007+0000 7fb30b400640 1 --2- 172.21.15.6:0/3806403795 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb30c0c5a60 0x7fb30c14dda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:32.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.007+0000 7fb30b400640 1 -- 172.21.15.6:0/3806403795 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb30c0cb180 con 0x7fb30c0c5e40 2024-04-24T23:16:32.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.007+0000 7fb30b400640 1 --2- 172.21.15.6:0/3806403795 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb30c0c5e40 0x7fb30c0ca8a0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fb2f8035040 tx=0x7fb2f80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:32.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.007+0000 7fb303e00640 1 -- 172.21.15.6:0/3806403795 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2f8037050 con 0x7fb30c0c5e40 2024-04-24T23:16:32.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.008+0000 7fb303e00640 1 -- 172.21.15.6:0/3806403795 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb2f8038040 con 0x7fb30c0c5e40 2024-04-24T23:16:32.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.008+0000 7fb303e00640 1 -- 172.21.15.6:0/3806403795 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2f8004450 con 0x7fb30c0c5e40 2024-04-24T23:16:32.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.008+0000 7fb311800640 1 -- 172.21.15.6:0/3806403795 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb30c148c60 con 0x7fb30c0c5e40 2024-04-24T23:16:32.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.008+0000 7fb311800640 1 -- 172.21.15.6:0/3806403795 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb30c0cb6d0 con 0x7fb30c0c5e40 2024-04-24T23:16:32.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.009+0000 7fb311800640 1 -- 172.21.15.6:0/3806403795 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb30c075b30 con 0x7fb30c0c5e40 2024-04-24T23:16:32.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.010+0000 7fb303e00640 1 -- 172.21.15.6:0/3806403795 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fb2f803c6d0 con 0x7fb30c0c5e40 2024-04-24T23:16:32.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.010+0000 7fb303e00640 1 --2- 172.21.15.6:0/3806403795 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fb2ec06be60 0x7fb2ec06e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:32.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.011+0000 7fb303e00640 1 -- 172.21.15.6:0/3806403795 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6590+0+0 (secure 0 0 0) 0x7fb2f80abaa0 con 0x7fb30c0c5e40 2024-04-24T23:16:32.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.011+0000 7fb30be00640 1 --2- 172.21.15.6:0/3806403795 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fb2ec06be60 0x7fb2ec06e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:32.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.011+0000 7fb30be00640 1 --2- 172.21.15.6:0/3806403795 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fb2ec06be60 0x7fb2ec06e320 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fb2fc007f70 tx=0x7fb2fc007f00 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:32.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.015+0000 7fb303e00640 1 -- 172.21.15.6:0/3806403795 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fb30c148c60 con 0x7fb30c0c5e40 2024-04-24T23:16:32.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.214+0000 7fb311800640 1 -- 172.21.15.6:0/3806403795 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fb30c159d60 con 0x7fb30c0c5e40 2024-04-24T23:16:32.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.215+0000 7fb303e00640 1 -- 172.21.15.6:0/3806403795 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) ==== 93+0+10 (secure 0 0 0) 0x7fb2f80799f0 con 0x7fb30c0c5e40 2024-04-24T23:16:32.219 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-04-24T23:16:32.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.217+0000 7fb311800640 1 -- 172.21.15.6:0/3806403795 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fb2ec06be60 msgr2=0x7fb2ec06e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:32.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.217+0000 7fb311800640 1 --2- 172.21.15.6:0/3806403795 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fb2ec06be60 0x7fb2ec06e320 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fb2fc007f70 tx=0x7fb2fc007f00 comp rx=0 tx=0).stop 2024-04-24T23:16:32.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.217+0000 7fb311800640 1 -- 172.21.15.6:0/3806403795 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb30c0c5e40 msgr2=0x7fb30c0ca8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:32.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.217+0000 7fb311800640 1 --2- 172.21.15.6:0/3806403795 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb30c0c5e40 0x7fb30c0ca8a0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fb2f8035040 tx=0x7fb2f80046c0 comp rx=0 tx=0).stop 2024-04-24T23:16:32.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.218+0000 7fb311800640 1 -- 172.21.15.6:0/3806403795 shutdown_connections 2024-04-24T23:16:32.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.218+0000 7fb311800640 1 --2- 172.21.15.6:0/3806403795 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fb2ec06be60 0x7fb2ec06e320 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:32.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.218+0000 7fb311800640 1 --2- 172.21.15.6:0/3806403795 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb30c0c8570 0x7fb30c1e1c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:32.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.218+0000 7fb311800640 1 --2- 172.21.15.6:0/3806403795 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb30c0c5e40 0x7fb30c0ca8a0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:32.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.218+0000 7fb311800640 1 --2- 172.21.15.6:0/3806403795 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb30c0c5a60 0x7fb30c14dda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:32.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.218+0000 7fb311800640 1 -- 172.21.15.6:0/3806403795 >> 172.21.15.6:0/3806403795 conn(0x7fb30c06cc40 msgr2=0x7fb30c157160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:32.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.218+0000 7fb311800640 1 -- 172.21.15.6:0/3806403795 shutdown_connections 2024-04-24T23:16:32.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.218+0000 7fb311800640 1 -- 172.21.15.6:0/3806403795 wait complete. 2024-04-24T23:16:32.239 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-24T23:16:32.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.454+0000 7fbf07e00640 1 Processor -- start 2024-04-24T23:16:32.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.455+0000 7fbf07e00640 1 -- start start 2024-04-24T23:16:32.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.456+0000 7fbf07e00640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fbf00154e10 0x7fbf0015e020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:32.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.456+0000 7fbf07e00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbf0006ffd0 con 0x7fbf000caf20 2024-04-24T23:16:32.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.456+0000 7fbf07e00640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbf001551f0 con 0x7fbf0015b530 2024-04-24T23:16:32.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.456+0000 7fbf07e00640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7fbf0015e720 con 0x7fbf00154e10 2024-04-24T23:16:32.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.457+0000 7fbf05400640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fbf00154e10 0x7fbf0015e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:32.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.457+0000 7fbf04a00640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7fbf0015b530 0x7fbf0006f8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.6:43670/0 (socket says 172.21.15.6:43670) 2024-04-24T23:16:32.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.457+0000 7fbf04a00640 1 -- 172.21.15.6:0/760126922 learned_addr learned my addr 172.21.15.6:0/760126922 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:16:32.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.457+0000 7fbf05e00640 1 --1- 172.21.15.6:0/760126922 >> v1:172.21.15.6:6789/0 conn(0x7fbf000caf20 0x7fbf00165370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:57142/0 (socket says 172.21.15.6:57142) 2024-04-24T23:16:32.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.457+0000 7fbf05400640 1 -- 172.21.15.6:0/760126922 >> v1:172.21.15.190:6789/0 conn(0x7fbf0015b530 legacy=0x7fbf0006f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:32.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.457+0000 7fbf05400640 1 -- 172.21.15.6:0/760126922 >> v1:172.21.15.6:6789/0 conn(0x7fbf000caf20 legacy=0x7fbf00165370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:32.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.457+0000 7fbf05400640 1 -- 172.21.15.6:0/760126922 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf0015ea00 con 0x7fbf00154e10 2024-04-24T23:16:32.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.457+0000 7fbeffe00640 1 -- 172.21.15.6:0/760126922 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 990295845 0 0) 0x7fbf001551f0 con 0x7fbf0015b530 2024-04-24T23:16:32.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.458+0000 7fbf05400640 1 --2- 172.21.15.6:0/760126922 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbf00154e10 0x7fbf0015e020 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fbef40097d0 tx=0x7fbef402a160 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ed3ed1d8b98e3792 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:32.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.459+0000 7fbeffe00640 1 -- 172.21.15.6:0/760126922 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbef4037070 con 0x7fbf00154e10 2024-04-24T23:16:32.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.459+0000 7fbeffe00640 1 -- 172.21.15.6:0/760126922 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbef402adf0 con 0x7fbf00154e10 2024-04-24T23:16:32.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.459+0000 7fbeffe00640 1 -- 172.21.15.6:0/760126922 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbef4032660 con 0x7fbf00154e10 2024-04-24T23:16:32.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.459+0000 7fbf07e00640 1 -- 172.21.15.6:0/760126922 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbf00154e10 msgr2=0x7fbf0015e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:32.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.459+0000 7fbf07e00640 1 --2- 172.21.15.6:0/760126922 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbf00154e10 0x7fbf0015e020 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fbef40097d0 tx=0x7fbef402a160 comp rx=0 tx=0).stop 2024-04-24T23:16:32.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.459+0000 7fbf07e00640 1 -- 172.21.15.6:0/760126922 shutdown_connections 2024-04-24T23:16:32.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.459+0000 7fbf07e00640 1 --2- 172.21.15.6:0/760126922 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbf00154e10 0x7fbf0015e020 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:32.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.459+0000 7fbf07e00640 1 -- 172.21.15.6:0/760126922 >> 172.21.15.6:0/760126922 conn(0x7fbf00070ce0 msgr2=0x7fbf000710f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:32.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.459+0000 7fbf07e00640 1 -- 172.21.15.6:0/760126922 shutdown_connections 2024-04-24T23:16:32.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.459+0000 7fbf07e00640 1 -- 172.21.15.6:0/760126922 wait complete. 2024-04-24T23:16:32.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.461+0000 7fbf07e00640 1 Processor -- start 2024-04-24T23:16:32.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.461+0000 7fbf07e00640 1 -- start start 2024-04-24T23:16:32.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.462+0000 7fbf07e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbf000caf20 0x7fbf000c7e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:32.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.462+0000 7fbf07e00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbf00154e10 0x7fbf000b2850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:32.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.462+0000 7fbf07e00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbf0015b530 0x7fbf00153c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:32.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.462+0000 7fbf07e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fbf0006f3a0 con 0x7fbf000caf20 2024-04-24T23:16:32.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.462+0000 7fbf07e00640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fbf0006f0c0 con 0x7fbf0015b530 2024-04-24T23:16:32.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.462+0000 7fbf07e00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fbf0015b980 con 0x7fbf00154e10 2024-04-24T23:16:32.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.462+0000 7fbf05400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbf000caf20 0x7fbf000c7e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:32.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.462+0000 7fbf04a00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbf00154e10 0x7fbf000b2850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:32.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.462+0000 7fbf05e00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbf0015b530 0x7fbf00153c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:32.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.462+0000 7fbf04a00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbf00154e10 0x7fbf000b2850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.6:55430/0 (socket says 172.21.15.6:55430) 2024-04-24T23:16:32.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.462+0000 7fbf04a00640 1 -- 172.21.15.6:0/2257700181 learned_addr learned my addr 172.21.15.6:0/2257700181 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:16:32.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.463+0000 7fbf04a00640 1 -- 172.21.15.6:0/2257700181 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbf0015b530 msgr2=0x7fbf00153c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:32.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.463+0000 7fbf04a00640 1 --2- 172.21.15.6:0/2257700181 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbf0015b530 0x7fbf00153c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:32.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.463+0000 7fbf04a00640 1 -- 172.21.15.6:0/2257700181 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbf000caf20 msgr2=0x7fbf000c7e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:32.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.463+0000 7fbf04a00640 1 --2- 172.21.15.6:0/2257700181 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbf000caf20 0x7fbf000c7e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:32.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.463+0000 7fbf04a00640 1 -- 172.21.15.6:0/2257700181 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf000c67c0 con 0x7fbf00154e10 2024-04-24T23:16:32.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.463+0000 7fbf05e00640 1 --2- 172.21.15.6:0/2257700181 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbf0015b530 0x7fbf00153c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:16:32.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.463+0000 7fbf05400640 1 --2- 172.21.15.6:0/2257700181 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbf000caf20 0x7fbf000c7e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:16:32.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.463+0000 7fbf04a00640 1 --2- 172.21.15.6:0/2257700181 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbf00154e10 0x7fbf000b2850 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fbeec00c420 tx=0x7fbeec00c8e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:32.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.464+0000 7fbefd600640 1 -- 172.21.15.6:0/2257700181 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbeec0074e0 con 0x7fbf00154e10 2024-04-24T23:16:32.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.464+0000 7fbefd600640 1 -- 172.21.15.6:0/2257700181 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbeec0047e0 con 0x7fbf00154e10 2024-04-24T23:16:32.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.464+0000 7fbefd600640 1 -- 172.21.15.6:0/2257700181 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbeec004c10 con 0x7fbf00154e10 2024-04-24T23:16:32.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.464+0000 7fbf07e00640 1 -- 172.21.15.6:0/2257700181 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf000c6a80 con 0x7fbf00154e10 2024-04-24T23:16:32.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.464+0000 7fbf07e00640 1 -- 172.21.15.6:0/2257700181 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf000c7040 con 0x7fbf00154e10 2024-04-24T23:16:32.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.465+0000 7fbf07e00640 1 -- 172.21.15.6:0/2257700181 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbf00068430 con 0x7fbf00154e10 2024-04-24T23:16:32.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.466+0000 7fbefd600640 1 -- 172.21.15.6:0/2257700181 <== mon.2 v2:172.21.15.190:3301/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fbeec011070 con 0x7fbf00154e10 2024-04-24T23:16:32.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.467+0000 7fbefd600640 1 --2- 172.21.15.6:0/2257700181 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fbee006be80 0x7fbee006e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:32.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.467+0000 7fbefd600640 1 -- 172.21.15.6:0/2257700181 <== mon.2 v2:172.21.15.190:3301/0 5 ==== osd_map(26..26 src has 1..26) ==== 6590+0+0 (secure 0 0 0) 0x7fbeec08acf0 con 0x7fbf00154e10 2024-04-24T23:16:32.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.467+0000 7fbf05400640 1 --2- 172.21.15.6:0/2257700181 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fbee006be80 0x7fbee006e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:32.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.468+0000 7fbf05400640 1 --2- 172.21.15.6:0/2257700181 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fbee006be80 0x7fbee006e340 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fbef4034000 tx=0x7fbef402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:32.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.472+0000 7fbefd600640 1 -- 172.21.15.6:0/2257700181 <== mon.2 v2:172.21.15.190:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fbeec058c30 con 0x7fbf00154e10 2024-04-24T23:16:32.519 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-24T23:16:32.526 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-24T23:16:32.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.672+0000 7fbf07e00640 1 -- 172.21.15.6:0/2257700181 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fbf000cc740 con 0x7fbf00154e10 2024-04-24T23:16:32.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.673+0000 7fbefd600640 1 -- 172.21.15.6:0/2257700181 <== mon.2 v2:172.21.15.190:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) ==== 104+0+11 (secure 0 0 0) 0x7fbeec05cf30 con 0x7fbf00154e10 2024-04-24T23:16:32.674 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 32 2024-04-24T23:16:32.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.676+0000 7fbf07e00640 1 -- 172.21.15.6:0/2257700181 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fbee006be80 msgr2=0x7fbee006e340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:32.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.676+0000 7fbf07e00640 1 --2- 172.21.15.6:0/2257700181 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fbee006be80 0x7fbee006e340 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fbef4034000 tx=0x7fbef402e000 comp rx=0 tx=0).stop 2024-04-24T23:16:32.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.676+0000 7fbf07e00640 1 -- 172.21.15.6:0/2257700181 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbf00154e10 msgr2=0x7fbf000b2850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:32.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.676+0000 7fbf07e00640 1 --2- 172.21.15.6:0/2257700181 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbf00154e10 0x7fbf000b2850 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fbeec00c420 tx=0x7fbeec00c8e0 comp rx=0 tx=0).stop 2024-04-24T23:16:32.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.676+0000 7fbf07e00640 1 -- 172.21.15.6:0/2257700181 shutdown_connections 2024-04-24T23:16:32.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.676+0000 7fbf07e00640 1 --2- 172.21.15.6:0/2257700181 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fbee006be80 0x7fbee006e340 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:32.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.676+0000 7fbf07e00640 1 --2- 172.21.15.6:0/2257700181 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbf00154e10 0x7fbf000b2850 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:32.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.676+0000 7fbf07e00640 1 --2- 172.21.15.6:0/2257700181 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbf0015b530 0x7fbf00153c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:32.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.676+0000 7fbf07e00640 1 --2- 172.21.15.6:0/2257700181 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbf000caf20 0x7fbf000c7e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:32.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.676+0000 7fbf07e00640 1 -- 172.21.15.6:0/2257700181 >> 172.21.15.6:0/2257700181 conn(0x7fbf00070ce0 msgr2=0x7fbf0015de30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:32.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.676+0000 7fbf07e00640 1 -- 172.21.15.6:0/2257700181 shutdown_connections 2024-04-24T23:16:32.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.676+0000 7fbf07e00640 1 -- 172.21.15.6:0/2257700181 wait complete. 2024-04-24T23:16:32.695 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-24T23:16:32.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.887+0000 7fc7cb400640 1 Processor -- start 2024-04-24T23:16:32.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.888+0000 7fc7cb400640 1 -- start start 2024-04-24T23:16:32.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.889+0000 7fc7cb400640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fc7c4152bc0 0x7fc7c4159390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:32.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.889+0000 7fc7cb400640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc7c40c9360 0x7fc7c415c610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:32.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.889+0000 7fc7cb400640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc7c40c6e70 con 0x7fc7c40c8f50 2024-04-24T23:16:32.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.889+0000 7fc7cb400640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7fc7c40c73d0 con 0x7fc7c4152bc0 2024-04-24T23:16:32.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.889+0000 7fc7cb400640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fc7c40c8ac0 con 0x7fc7c40c9360 2024-04-24T23:16:32.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.889+0000 7fc7c3e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fc7c40c8f50 0x7fc7c415bf00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:57154/0 (socket says 172.21.15.6:57154) 2024-04-24T23:16:32.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.889+0000 7fc7c3e00640 1 -- 172.21.15.6:0/2128169558 learned_addr learned my addr 172.21.15.6:0/2128169558 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:16:32.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.890+0000 7fc7c8a00640 1 --2- 172.21.15.6:0/2128169558 >> v2:172.21.15.190:3300/0 conn(0x7fc7c4152bc0 0x7fc7c4159390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:32.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.890+0000 7fc7c9400640 1 --2- 172.21.15.6:0/2128169558 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc7c40c9360 0x7fc7c415c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:32.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.890+0000 7fc7c3400640 1 -- 172.21.15.6:0/2128169558 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1006726965 0 0) 0x7fc7c40c6e70 con 0x7fc7c40c8f50 2024-04-24T23:16:32.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.890+0000 7fc7c8a00640 1 -- 172.21.15.6:0/2128169558 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc7c40c9360 msgr2=0x7fc7c415c610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:16:32.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.890+0000 7fc7c8a00640 1 --2- 172.21.15.6:0/2128169558 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc7c40c9360 0x7fc7c415c610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:32.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.890+0000 7fc7c8a00640 1 -- 172.21.15.6:0/2128169558 >> v1:172.21.15.6:6789/0 conn(0x7fc7c40c8f50 legacy=0x7fc7c415bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:32.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.890+0000 7fc7c8a00640 1 -- 172.21.15.6:0/2128169558 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7c41598d0 con 0x7fc7c4152bc0 2024-04-24T23:16:32.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.891+0000 7fc7c8a00640 1 --2- 172.21.15.6:0/2128169558 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc7c4152bc0 0x7fc7c4159390 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fc7b80097a0 tx=0x7fc7b802a100 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cc61a1fc8a206a9f server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:32.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.891+0000 7fc7c3400640 1 -- 172.21.15.6:0/2128169558 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7b8037070 con 0x7fc7c4152bc0 2024-04-24T23:16:32.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.892+0000 7fc7c3400640 1 -- 172.21.15.6:0/2128169558 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc7b802ae50 con 0x7fc7c4152bc0 2024-04-24T23:16:32.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.892+0000 7fc7c3400640 1 -- 172.21.15.6:0/2128169558 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7b8032660 con 0x7fc7c4152bc0 2024-04-24T23:16:32.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.892+0000 7fc7cb400640 1 -- 172.21.15.6:0/2128169558 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc7c4152bc0 msgr2=0x7fc7c4159390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:32.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.892+0000 7fc7cb400640 1 --2- 172.21.15.6:0/2128169558 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc7c4152bc0 0x7fc7c4159390 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fc7b80097a0 tx=0x7fc7b802a100 comp rx=0 tx=0).stop 2024-04-24T23:16:32.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.892+0000 7fc7cb400640 1 -- 172.21.15.6:0/2128169558 shutdown_connections 2024-04-24T23:16:32.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.892+0000 7fc7cb400640 1 --2- 172.21.15.6:0/2128169558 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc7c40c9360 0x7fc7c415c610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:32.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.892+0000 7fc7cb400640 1 --2- 172.21.15.6:0/2128169558 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc7c4152bc0 0x7fc7c4159390 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:32.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.892+0000 7fc7cb400640 1 -- 172.21.15.6:0/2128169558 >> 172.21.15.6:0/2128169558 conn(0x7fc7c406b0e0 msgr2=0x7fc7c406b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:32.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.892+0000 7fc7cb400640 1 -- 172.21.15.6:0/2128169558 shutdown_connections 2024-04-24T23:16:32.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.892+0000 7fc7cb400640 1 -- 172.21.15.6:0/2128169558 wait complete. 2024-04-24T23:16:32.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.894+0000 7fc7cb400640 1 Processor -- start 2024-04-24T23:16:32.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.894+0000 7fc7cb400640 1 -- start start 2024-04-24T23:16:32.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.895+0000 7fc7cb400640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc7c40c8f50 0x7fc7c415b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:32.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.895+0000 7fc7cb400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc7c40c9360 0x7fc7c415b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:32.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.895+0000 7fc7cb400640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc7c4152bc0 0x7fc7c414f460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:32.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.895+0000 7fc7cb400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fc7c40c8470 con 0x7fc7c40c9360 2024-04-24T23:16:32.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.895+0000 7fc7cb400640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fc7c40c6700 con 0x7fc7c40c8f50 2024-04-24T23:16:32.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.895+0000 7fc7cb400640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fc7c40c8750 con 0x7fc7c4152bc0 2024-04-24T23:16:32.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.895+0000 7fc7c3e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc7c40c9360 0x7fc7c415b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:32.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.895+0000 7fc7c3e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc7c40c9360 0x7fc7c415b8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:35622/0 (socket says 172.21.15.6:35622) 2024-04-24T23:16:32.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.895+0000 7fc7c3e00640 1 -- 172.21.15.6:0/625852955 learned_addr learned my addr 172.21.15.6:0/625852955 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:16:32.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.895+0000 7fc7c8a00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc7c40c8f50 0x7fc7c415b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:32.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.895+0000 7fc7c9400640 1 --2- 172.21.15.6:0/625852955 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc7c4152bc0 0x7fc7c414f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:32.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.896+0000 7fc7c3e00640 1 -- 172.21.15.6:0/625852955 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc7c4152bc0 msgr2=0x7fc7c414f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:32.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.896+0000 7fc7c3e00640 1 --2- 172.21.15.6:0/625852955 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc7c4152bc0 0x7fc7c414f460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:32.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.896+0000 7fc7c3e00640 1 -- 172.21.15.6:0/625852955 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc7c40c8f50 msgr2=0x7fc7c415b380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:32.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.896+0000 7fc7c3e00640 1 --2- 172.21.15.6:0/625852955 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc7c40c8f50 0x7fc7c415b380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:32.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.896+0000 7fc7c3e00640 1 -- 172.21.15.6:0/625852955 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7c41eea20 con 0x7fc7c40c9360 2024-04-24T23:16:32.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.896+0000 7fc7c8a00640 1 --2- 172.21.15.6:0/625852955 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc7c40c8f50 0x7fc7c415b380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:16:32.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.896+0000 7fc7c9400640 1 --2- 172.21.15.6:0/625852955 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc7c4152bc0 0x7fc7c414f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:16:32.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.896+0000 7fc7c3e00640 1 --2- 172.21.15.6:0/625852955 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc7c40c9360 0x7fc7c415b8c0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fc7b400c430 tx=0x7fc7b400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:32.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.896+0000 7fc7c0c00640 1 -- 172.21.15.6:0/625852955 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7b4017070 con 0x7fc7c40c9360 2024-04-24T23:16:32.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.896+0000 7fc7c0c00640 1 -- 172.21.15.6:0/625852955 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc7b4005ce0 con 0x7fc7c40c9360 2024-04-24T23:16:32.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.896+0000 7fc7c0c00640 1 -- 172.21.15.6:0/625852955 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7b4007310 con 0x7fc7c40c9360 2024-04-24T23:16:32.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.896+0000 7fc7cb400640 1 -- 172.21.15.6:0/625852955 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7c41eed00 con 0x7fc7c40c9360 2024-04-24T23:16:32.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.897+0000 7fc7cb400640 1 -- 172.21.15.6:0/625852955 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7c41ef2c0 con 0x7fc7c40c9360 2024-04-24T23:16:32.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.898+0000 7fc7cb400640 1 -- 172.21.15.6:0/625852955 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc7c406dd40 con 0x7fc7c40c9360 2024-04-24T23:16:32.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.899+0000 7fc7c0c00640 1 -- 172.21.15.6:0/625852955 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fc7b401a070 con 0x7fc7c40c9360 2024-04-24T23:16:32.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.900+0000 7fc7c0c00640 1 --2- 172.21.15.6:0/625852955 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fc7a806be30 0x7fc7a806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:32.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.900+0000 7fc7c0c00640 1 -- 172.21.15.6:0/625852955 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6590+0+0 (secure 0 0 0) 0x7fc7b408a4d0 con 0x7fc7c40c9360 2024-04-24T23:16:32.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.900+0000 7fc7c8a00640 1 --2- 172.21.15.6:0/625852955 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fc7a806be30 0x7fc7a806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:32.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.901+0000 7fc7c8a00640 1 --2- 172.21.15.6:0/625852955 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fc7a806be30 0x7fc7a806e2f0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fc7b8034000 tx=0x7fc7b802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:32.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:32.904+0000 7fc7c0c00640 1 -- 172.21.15.6:0/625852955 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fc7c406dd40 con 0x7fc7c40c9360 2024-04-24T23:16:33.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.108+0000 7fc7cb400640 1 -- 172.21.15.6:0/625852955 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fc7c40767b0 con 0x7fc7c40c9360 2024-04-24T23:16:33.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.108+0000 7fc7c0c00640 1 -- 172.21.15.6:0/625852955 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) ==== 100+0+11 (secure 0 0 0) 0x7fc7b4058550 con 0x7fc7c40c9360 2024-04-24T23:16:33.110 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 64 2024-04-24T23:16:33.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.111+0000 7fc7cb400640 1 -- 172.21.15.6:0/625852955 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fc7a806be30 msgr2=0x7fc7a806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:33.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.111+0000 7fc7cb400640 1 --2- 172.21.15.6:0/625852955 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fc7a806be30 0x7fc7a806e2f0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fc7b8034000 tx=0x7fc7b802e000 comp rx=0 tx=0).stop 2024-04-24T23:16:33.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.111+0000 7fc7cb400640 1 -- 172.21.15.6:0/625852955 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc7c40c9360 msgr2=0x7fc7c415b8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:33.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.111+0000 7fc7cb400640 1 --2- 172.21.15.6:0/625852955 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc7c40c9360 0x7fc7c415b8c0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fc7b400c430 tx=0x7fc7b400c8f0 comp rx=0 tx=0).stop 2024-04-24T23:16:33.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.111+0000 7fc7cb400640 1 -- 172.21.15.6:0/625852955 shutdown_connections 2024-04-24T23:16:33.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.111+0000 7fc7cb400640 1 --2- 172.21.15.6:0/625852955 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fc7a806be30 0x7fc7a806e2f0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:33.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.111+0000 7fc7cb400640 1 --2- 172.21.15.6:0/625852955 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc7c4152bc0 0x7fc7c414f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:33.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.111+0000 7fc7cb400640 1 --2- 172.21.15.6:0/625852955 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc7c40c9360 0x7fc7c415b8c0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:33.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.111+0000 7fc7cb400640 1 --2- 172.21.15.6:0/625852955 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc7c40c8f50 0x7fc7c415b380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:33.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.111+0000 7fc7cb400640 1 -- 172.21.15.6:0/625852955 >> 172.21.15.6:0/625852955 conn(0x7fc7c406b0e0 msgr2=0x7fc7c406b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:33.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.111+0000 7fc7cb400640 1 -- 172.21.15.6:0/625852955 shutdown_connections 2024-04-24T23:16:33.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.112+0000 7fc7cb400640 1 -- 172.21.15.6:0/625852955 wait complete. 2024-04-24T23:16:33.129 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-24T23:16:33.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.333+0000 7fe757e00640 1 Processor -- start 2024-04-24T23:16:33.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.334+0000 7fe757e00640 1 -- start start 2024-04-24T23:16:33.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.335+0000 7fe757e00640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fe75806cd50 0x7fe75815e090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:33.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.335+0000 7fe757e00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe7580b35f0 0x7fe75815e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:33.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.335+0000 7fe757e00640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe7580b46f0 con 0x7fe758154f60 2024-04-24T23:16:33.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.335+0000 7fe757e00640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7fe7580b49d0 con 0x7fe75806cd50 2024-04-24T23:16:33.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.335+0000 7fe757e00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fe7580b4cb0 con 0x7fe7580b35f0 2024-04-24T23:16:33.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.335+0000 7fe757400640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe7580b35f0 0x7fe75815e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:33.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.335+0000 7fe756a00640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7fe758154f60 0x7fe75806c640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.6:43676/0 (socket says 172.21.15.6:43676) 2024-04-24T23:16:33.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.335+0000 7fe756a00640 1 -- 172.21.15.6:0/291037326 learned_addr learned my addr 172.21.15.6:0/291037326 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:16:33.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.335+0000 7fe756000640 1 --2- 172.21.15.6:0/291037326 >> v2:172.21.15.190:3300/0 conn(0x7fe75806cd50 0x7fe75815e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:33.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.336+0000 7fe755600640 1 -- 172.21.15.6:0/291037326 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1140657300 0 0) 0x7fe7580b46f0 con 0x7fe758154f60 2024-04-24T23:16:33.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.336+0000 7fe755600640 1 -- 172.21.15.6:0/291037326 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe73c003520 con 0x7fe758154f60 2024-04-24T23:16:33.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.336+0000 7fe757400640 1 -- 172.21.15.6:0/291037326 >> v2:172.21.15.190:3300/0 conn(0x7fe75806cd50 msgr2=0x7fe75815e090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:16:33.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.336+0000 7fe757400640 1 --2- 172.21.15.6:0/291037326 >> v2:172.21.15.190:3300/0 conn(0x7fe75806cd50 0x7fe75815e090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:33.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.336+0000 7fe757400640 1 -- 172.21.15.6:0/291037326 >> v1:172.21.15.190:6789/0 conn(0x7fe758154f60 legacy=0x7fe75806c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:33.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.336+0000 7fe757400640 1 -- 172.21.15.6:0/291037326 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7580b4f90 con 0x7fe7580b35f0 2024-04-24T23:16:33.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.336+0000 7fe756000640 1 --2- 172.21.15.6:0/291037326 >> v2:172.21.15.190:3300/0 conn(0x7fe75806cd50 0x7fe75815e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:16:33.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.337+0000 7fe757400640 1 --2- 172.21.15.6:0/291037326 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe7580b35f0 0x7fe75815e5d0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fe750009d90 tx=0x7fe75002a6f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d87a19a18e6c7bb9 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:33.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.337+0000 7fe755600640 1 -- 172.21.15.6:0/291037326 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe750007740 con 0x7fe7580b35f0 2024-04-24T23:16:33.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.337+0000 7fe755600640 1 -- 172.21.15.6:0/291037326 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe7500078e0 con 0x7fe7580b35f0 2024-04-24T23:16:33.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.337+0000 7fe755600640 1 -- 172.21.15.6:0/291037326 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe750007cb0 con 0x7fe7580b35f0 2024-04-24T23:16:33.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.338+0000 7fe757e00640 1 -- 172.21.15.6:0/291037326 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe7580b35f0 msgr2=0x7fe75815e5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:33.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.338+0000 7fe757e00640 1 --2- 172.21.15.6:0/291037326 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe7580b35f0 0x7fe75815e5d0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fe750009d90 tx=0x7fe75002a6f0 comp rx=0 tx=0).stop 2024-04-24T23:16:33.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.338+0000 7fe757e00640 1 -- 172.21.15.6:0/291037326 shutdown_connections 2024-04-24T23:16:33.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.338+0000 7fe757e00640 1 --2- 172.21.15.6:0/291037326 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe7580b35f0 0x7fe75815e5d0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:33.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.338+0000 7fe757e00640 1 --2- 172.21.15.6:0/291037326 >> v2:172.21.15.190:3300/0 conn(0x7fe75806cd50 0x7fe75815e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:33.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.338+0000 7fe757e00640 1 -- 172.21.15.6:0/291037326 >> 172.21.15.6:0/291037326 conn(0x7fe7580674d0 msgr2=0x7fe7580d0320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:33.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.338+0000 7fe757e00640 1 -- 172.21.15.6:0/291037326 shutdown_connections 2024-04-24T23:16:33.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.338+0000 7fe757e00640 1 -- 172.21.15.6:0/291037326 wait complete. 2024-04-24T23:16:33.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.340+0000 7fe757e00640 1 Processor -- start 2024-04-24T23:16:33.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.340+0000 7fe757e00640 1 -- start start 2024-04-24T23:16:33.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.341+0000 7fe757e00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe75806cd50 0x7fe7580c4cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:33.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.341+0000 7fe757e00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe7580b35f0 0x7fe7581e3080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:33.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.341+0000 7fe757e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe758154f60 0x7fe7580c42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:33.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.341+0000 7fe757e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fe7580b3240 con 0x7fe758154f60 2024-04-24T23:16:33.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.341+0000 7fe757e00640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fe75806c120 con 0x7fe7580b35f0 2024-04-24T23:16:33.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.341+0000 7fe757e00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fe7580b33c0 con 0x7fe75806cd50 2024-04-24T23:16:33.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.341+0000 7fe756a00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe75806cd50 0x7fe7580c4cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:33.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.341+0000 7fe756000640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe7580b35f0 0x7fe7581e3080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:33.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.341+0000 7fe757400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe758154f60 0x7fe7580c42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:33.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.341+0000 7fe756000640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe7580b35f0 0x7fe7581e3080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:58440/0 (socket says 172.21.15.6:58440) 2024-04-24T23:16:33.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.341+0000 7fe756a00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe75806cd50 0x7fe7580c4cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.6:55484/0 (socket says 172.21.15.6:55484) 2024-04-24T23:16:33.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.341+0000 7fe756000640 1 -- 172.21.15.6:0/2038775857 learned_addr learned my addr 172.21.15.6:0/2038775857 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:16:33.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.341+0000 7fe756000640 1 -- 172.21.15.6:0/2038775857 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe75806cd50 msgr2=0x7fe7580c4cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:33.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.341+0000 7fe756000640 1 --2- 172.21.15.6:0/2038775857 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe75806cd50 0x7fe7580c4cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:33.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.342+0000 7fe756000640 1 -- 172.21.15.6:0/2038775857 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe758154f60 msgr2=0x7fe7580c42d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:33.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.342+0000 7fe756000640 1 --2- 172.21.15.6:0/2038775857 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe758154f60 0x7fe7580c42d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:33.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.342+0000 7fe756000640 1 -- 172.21.15.6:0/2038775857 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7580c5db0 con 0x7fe7580b35f0 2024-04-24T23:16:33.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.342+0000 7fe756a00640 1 --2- 172.21.15.6:0/2038775857 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe75806cd50 0x7fe7580c4cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:16:33.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.342+0000 7fe756000640 1 --2- 172.21.15.6:0/2038775857 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe7580b35f0 0x7fe7581e3080 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fe74800ebe0 tx=0x7fe74800c520 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:33.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.342+0000 7fe74ea00640 1 -- 172.21.15.6:0/2038775857 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe748017070 con 0x7fe7580b35f0 2024-04-24T23:16:33.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.342+0000 7fe74ea00640 1 -- 172.21.15.6:0/2038775857 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe74800cc30 con 0x7fe7580b35f0 2024-04-24T23:16:33.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.342+0000 7fe74ea00640 1 -- 172.21.15.6:0/2038775857 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe748013470 con 0x7fe7580b35f0 2024-04-24T23:16:33.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.342+0000 7fe757e00640 1 -- 172.21.15.6:0/2038775857 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7580c6090 con 0x7fe7580b35f0 2024-04-24T23:16:33.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.342+0000 7fe757e00640 1 -- 172.21.15.6:0/2038775857 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7581e2cb0 con 0x7fe7580b35f0 2024-04-24T23:16:33.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.343+0000 7fe757e00640 1 -- 172.21.15.6:0/2038775857 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe758154110 con 0x7fe7580b35f0 2024-04-24T23:16:33.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.344+0000 7fe74ea00640 1 -- 172.21.15.6:0/2038775857 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fe74801a070 con 0x7fe7580b35f0 2024-04-24T23:16:33.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.345+0000 7fe74ea00640 1 --2- 172.21.15.6:0/2038775857 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fe73406be30 0x7fe73406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:33.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.345+0000 7fe74ea00640 1 -- 172.21.15.6:0/2038775857 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6590+0+0 (secure 0 0 0) 0x7fe74808b540 con 0x7fe7580b35f0 2024-04-24T23:16:33.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.345+0000 7fe756a00640 1 --2- 172.21.15.6:0/2038775857 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fe73406be30 0x7fe73406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:33.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.346+0000 7fe756a00640 1 --2- 172.21.15.6:0/2038775857 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fe73406be30 0x7fe73406e2f0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fe740005880 tx=0x7fe740009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:33.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.350+0000 7fe74ea00640 1 -- 172.21.15.6:0/2038775857 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fe758154110 con 0x7fe7580b35f0 2024-04-24T23:16:33.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.553+0000 7fe757e00640 1 -- 172.21.15.6:0/2038775857 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fe758075b30 con 0x7fe7580b35f0 2024-04-24T23:16:33.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.554+0000 7fe74ea00640 1 -- 172.21.15.6:0/2038775857 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) ==== 103+0+11 (secure 0 0 0) 0x7fe748059400 con 0x7fe7580b35f0 2024-04-24T23:16:33.555 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 32 2024-04-24T23:16:33.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.557+0000 7fe757e00640 1 -- 172.21.15.6:0/2038775857 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fe73406be30 msgr2=0x7fe73406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:33.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.557+0000 7fe757e00640 1 --2- 172.21.15.6:0/2038775857 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fe73406be30 0x7fe73406e2f0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fe740005880 tx=0x7fe740009040 comp rx=0 tx=0).stop 2024-04-24T23:16:33.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.557+0000 7fe757e00640 1 -- 172.21.15.6:0/2038775857 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe7580b35f0 msgr2=0x7fe7581e3080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:33.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.557+0000 7fe757e00640 1 --2- 172.21.15.6:0/2038775857 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe7580b35f0 0x7fe7581e3080 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fe74800ebe0 tx=0x7fe74800c520 comp rx=0 tx=0).stop 2024-04-24T23:16:33.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.557+0000 7fe757e00640 1 -- 172.21.15.6:0/2038775857 shutdown_connections 2024-04-24T23:16:33.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.557+0000 7fe757e00640 1 --2- 172.21.15.6:0/2038775857 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fe73406be30 0x7fe73406e2f0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:33.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.557+0000 7fe757e00640 1 --2- 172.21.15.6:0/2038775857 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe758154f60 0x7fe7580c42d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:33.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.557+0000 7fe757e00640 1 --2- 172.21.15.6:0/2038775857 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe7580b35f0 0x7fe7581e3080 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:33.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.557+0000 7fe757e00640 1 --2- 172.21.15.6:0/2038775857 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe75806cd50 0x7fe7580c4cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:33.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.557+0000 7fe757e00640 1 -- 172.21.15.6:0/2038775857 >> 172.21.15.6:0/2038775857 conn(0x7fe7580674d0 msgr2=0x7fe7580d0b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:33.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.558+0000 7fe757e00640 1 -- 172.21.15.6:0/2038775857 shutdown_connections 2024-04-24T23:16:33.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.558+0000 7fe757e00640 1 -- 172.21.15.6:0/2038775857 wait complete. 2024-04-24T23:16:33.586 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:16:33.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.782+0000 7f7857000640 1 Processor -- start 2024-04-24T23:16:33.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.782+0000 7f7857000640 1 -- start start 2024-04-24T23:16:33.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.783+0000 7f7857000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f78500cb090 0x7f78500d0820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:33.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.783+0000 7f7857000640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f78500d0d60 0x7f785015e000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:33.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.783+0000 7f7857000640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7850155170 con 0x7f7850154d90 2024-04-24T23:16:33.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.783+0000 7f7857000640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f78500cc9a0 con 0x7f78500cb090 2024-04-24T23:16:33.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.783+0000 7f7857000640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f785015ad60 con 0x7f78500d0d60 2024-04-24T23:16:33.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.784+0000 7f7855200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f78500cb090 0x7f78500d0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:33.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.784+0000 7f7855c00640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f7850154d90 0x7f78500cbe80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.6:43688/0 (socket says 172.21.15.6:43688) 2024-04-24T23:16:33.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.784+0000 7f7855c00640 1 -- 172.21.15.6:0/4103748594 learned_addr learned my addr 172.21.15.6:0/4103748594 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:16:33.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.784+0000 7f7856600640 1 --2- 172.21.15.6:0/4103748594 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f78500d0d60 0x7f785015e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:33.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.784+0000 7f7855200640 1 -- 172.21.15.6:0/4103748594 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f78500d0d60 msgr2=0x7f785015e000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:16:33.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.784+0000 7f7855200640 1 --2- 172.21.15.6:0/4103748594 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f78500d0d60 0x7f785015e000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:33.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.784+0000 7f7855200640 1 -- 172.21.15.6:0/4103748594 >> v1:172.21.15.190:6789/0 conn(0x7f7850154d90 legacy=0x7f78500cbe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:33.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.784+0000 7f7855200640 1 -- 172.21.15.6:0/4103748594 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f785015b040 con 0x7f78500cb090 2024-04-24T23:16:33.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.784+0000 7f784fe00640 1 -- 172.21.15.6:0/4103748594 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 623407521 0 0) 0x7f7850155170 con 0x7f7850154d90 2024-04-24T23:16:33.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.785+0000 7f7855200640 1 --2- 172.21.15.6:0/4103748594 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f78500cb090 0x7f78500d0820 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f783c0097a0 tx=0x7f783c02a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ecf48875a08757a8 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:33.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.786+0000 7f784fe00640 1 -- 172.21.15.6:0/4103748594 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f783c037070 con 0x7f78500cb090 2024-04-24T23:16:33.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.786+0000 7f784fe00640 1 -- 172.21.15.6:0/4103748594 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f783c032440 con 0x7f78500cb090 2024-04-24T23:16:33.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.786+0000 7f784fe00640 1 -- 172.21.15.6:0/4103748594 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f783c032810 con 0x7f78500cb090 2024-04-24T23:16:33.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.786+0000 7f7857000640 1 -- 172.21.15.6:0/4103748594 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f78500cb090 msgr2=0x7f78500d0820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:33.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.786+0000 7f7857000640 1 --2- 172.21.15.6:0/4103748594 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f78500cb090 0x7f78500d0820 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f783c0097a0 tx=0x7f783c02a210 comp rx=0 tx=0).stop 2024-04-24T23:16:33.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.786+0000 7f7857000640 1 -- 172.21.15.6:0/4103748594 shutdown_connections 2024-04-24T23:16:33.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.786+0000 7f7857000640 1 --2- 172.21.15.6:0/4103748594 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f78500d0d60 0x7f785015e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:33.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.786+0000 7f7857000640 1 --2- 172.21.15.6:0/4103748594 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f78500cb090 0x7f78500d0820 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:33.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.786+0000 7f7857000640 1 -- 172.21.15.6:0/4103748594 >> 172.21.15.6:0/4103748594 conn(0x7f785006be10 msgr2=0x7f785006c1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:33.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.786+0000 7f7857000640 1 -- 172.21.15.6:0/4103748594 shutdown_connections 2024-04-24T23:16:33.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.786+0000 7f7857000640 1 -- 172.21.15.6:0/4103748594 wait complete. 2024-04-24T23:16:33.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.789+0000 7f7857000640 1 Processor -- start 2024-04-24T23:16:33.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.789+0000 7f7857000640 1 -- start start 2024-04-24T23:16:33.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.790+0000 7f7857000640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f78500cb090 0x7f7850154030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:33.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.790+0000 7f7857000640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f78500d0d60 0x7f78501f3e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:33.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.790+0000 7f7857000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7850154d90 0x7f785014b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:33.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.790+0000 7f7857000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f785015aa30 con 0x7f7850154d90 2024-04-24T23:16:33.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.790+0000 7f7857000640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f785015b600 con 0x7f78500d0d60 2024-04-24T23:16:33.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.790+0000 7f7857000640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f78500cc590 con 0x7f78500cb090 2024-04-24T23:16:33.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.791+0000 7f7856600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7850154d90 0x7f785014b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:33.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.791+0000 7f7856600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7850154d90 0x7f785014b340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:35652/0 (socket says 172.21.15.6:35652) 2024-04-24T23:16:33.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.791+0000 7f7856600640 1 -- 172.21.15.6:0/569353870 learned_addr learned my addr 172.21.15.6:0/569353870 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:16:33.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.791+0000 7f7855200640 1 --2- 172.21.15.6:0/569353870 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f78500d0d60 0x7f78501f3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:33.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.791+0000 7f7855c00640 1 --2- 172.21.15.6:0/569353870 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f78500cb090 0x7f7850154030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:33.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.791+0000 7f7855200640 1 -- 172.21.15.6:0/569353870 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f78500cb090 msgr2=0x7f7850154030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:33.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.791+0000 7f7855200640 1 --2- 172.21.15.6:0/569353870 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f78500cb090 0x7f7850154030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:33.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.791+0000 7f7855200640 1 -- 172.21.15.6:0/569353870 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7850154d90 msgr2=0x7f785014b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:33.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.791+0000 7f7855200640 1 --2- 172.21.15.6:0/569353870 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7850154d90 0x7f785014b340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:33.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.791+0000 7f7855200640 1 -- 172.21.15.6:0/569353870 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78501f4370 con 0x7f78500d0d60 2024-04-24T23:16:33.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.791+0000 7f7856600640 1 --2- 172.21.15.6:0/569353870 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7850154d90 0x7f785014b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:16:33.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.792+0000 7f7855200640 1 --2- 172.21.15.6:0/569353870 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f78500d0d60 0x7f78501f3e30 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f78500cde90 tx=0x7f783c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:33.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.792+0000 7f784d600640 1 -- 172.21.15.6:0/569353870 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f783c041030 con 0x7f78500d0d60 2024-04-24T23:16:33.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.792+0000 7f784d600640 1 -- 172.21.15.6:0/569353870 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f783c030500 con 0x7f78500d0d60 2024-04-24T23:16:33.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.792+0000 7f7857000640 1 -- 172.21.15.6:0/569353870 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f785015cad0 con 0x7f78500d0d60 2024-04-24T23:16:33.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.792+0000 7f784d600640 1 -- 172.21.15.6:0/569353870 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f783c037040 con 0x7f78500d0d60 2024-04-24T23:16:33.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.792+0000 7f7857000640 1 -- 172.21.15.6:0/569353870 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f785015d090 con 0x7f78500d0d60 2024-04-24T23:16:33.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.793+0000 7f7857000640 1 -- 172.21.15.6:0/569353870 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7850075b30 con 0x7f78500d0d60 2024-04-24T23:16:33.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.794+0000 7f784d600640 1 -- 172.21.15.6:0/569353870 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f783c044070 con 0x7f78500d0d60 2024-04-24T23:16:33.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.795+0000 7f784d600640 1 --2- 172.21.15.6:0/569353870 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f783006be30 0x7f783006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:33.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.795+0000 7f784d600640 1 -- 172.21.15.6:0/569353870 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6590+0+0 (secure 0 0 0) 0x7f783c0ab780 con 0x7f78500d0d60 2024-04-24T23:16:33.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.795+0000 7f7855c00640 1 --2- 172.21.15.6:0/569353870 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f783006be30 0x7f783006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:33.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.796+0000 7f7855c00640 1 --2- 172.21.15.6:0/569353870 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f783006be30 0x7f783006e2f0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f7840001920 tx=0x7f78400018b0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:33.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:33.799+0000 7f784d600640 1 -- 172.21.15.6:0/569353870 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f783c0796d0 con 0x7f78500d0d60 2024-04-24T23:16:34.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.035+0000 7f7857000640 1 -- 172.21.15.6:0/569353870 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f785014b880 con 0x7f78500d0d60 2024-04-24T23:16:34.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.036+0000 7f784d600640 1 -- 172.21.15.6:0/569353870 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) ==== 94+0+6902 (secure 0 0 0) 0x7f783c07d9d0 con 0x7f78500d0d60 2024-04-24T23:16:34.037 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:16:34.038 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":18,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4197,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/105267896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":105267896},{"type":"v1","addr":"172.21.15.6:6834","nonce":105267896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:16:32.310838+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4282,"mds_2":4235,"mds_3":4232,"mds_4":4203},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4203":{"gid":4203,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":17,"addr":"172.21.15.6:6837/2590421094","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":2590421094},{"type":"v1","addr":"172.21.15.6:6837","nonce":2590421094}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4232":{"gid":4232,"name":"b","rank":3,"incarnation":15,"state":"up:active","state_seq":15,"addr":"172.21.15.190:6838/2032280158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2032280158},{"type":"v1","addr":"172.21.15.190:6838","nonce":2032280158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4235":{"gid":4235,"name":"d","rank":2,"incarnation":13,"state":"up:active","state_seq":14,"addr":"172.21.15.190:6839/1572872451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":1572872451},{"type":"v1","addr":"172.21.15.190:6839","nonce":1572872451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4282":{"gid":4282,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":12,"addr":"172.21.15.190:6837/580315269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":580315269},{"type":"v1","addr":"172.21.15.190:6837","nonce":580315269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/489092393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":489092393},{"type":"v1","addr":"172.21.15.6:6835","nonce":489092393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4306,"qdb_cluster":[4203,4232,4235,4282,4306]},"id":1}]} 2024-04-24T23:16:34.038 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 18 2024-04-24T23:16:34.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.038+0000 7f7857000640 1 -- 172.21.15.6:0/569353870 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f783006be30 msgr2=0x7f783006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:34.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.038+0000 7f7857000640 1 --2- 172.21.15.6:0/569353870 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f783006be30 0x7f783006e2f0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f7840001920 tx=0x7f78400018b0 comp rx=0 tx=0).stop 2024-04-24T23:16:34.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.038+0000 7f7857000640 1 -- 172.21.15.6:0/569353870 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f78500d0d60 msgr2=0x7f78501f3e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:34.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.038+0000 7f7857000640 1 --2- 172.21.15.6:0/569353870 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f78500d0d60 0x7f78501f3e30 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f78500cde90 tx=0x7f783c0046c0 comp rx=0 tx=0).stop 2024-04-24T23:16:34.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.039+0000 7f7857000640 1 -- 172.21.15.6:0/569353870 shutdown_connections 2024-04-24T23:16:34.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.039+0000 7f7857000640 1 --2- 172.21.15.6:0/569353870 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f783006be30 0x7f783006e2f0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:34.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.039+0000 7f7857000640 1 --2- 172.21.15.6:0/569353870 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7850154d90 0x7f785014b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:34.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.039+0000 7f7857000640 1 --2- 172.21.15.6:0/569353870 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f78500d0d60 0x7f78501f3e30 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:34.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.039+0000 7f7857000640 1 --2- 172.21.15.6:0/569353870 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f78500cb090 0x7f7850154030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:34.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.039+0000 7f7857000640 1 -- 172.21.15.6:0/569353870 >> 172.21.15.6:0/569353870 conn(0x7f785006be10 msgr2=0x7f78500676b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:34.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.039+0000 7f7857000640 1 -- 172.21.15.6:0/569353870 shutdown_connections 2024-04-24T23:16:34.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.039+0000 7f7857000640 1 -- 172.21.15.6:0/569353870 wait complete. 2024-04-24T23:16:34.058 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T23:16:34.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.253+0000 7f4d94c00640 1 Processor -- start 2024-04-24T23:16:34.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.253+0000 7f4d94c00640 1 -- start start 2024-04-24T23:16:34.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.254+0000 7f4d94c00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4d90152bc0 0x7f4d90159390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:34.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.254+0000 7f4d94c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f4d900c8f50 0x7f4d9015bf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:34.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.254+0000 7f4d94c00640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f4d900c9330 0x7f4d9015c440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:34.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.254+0000 7f4d94c00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f4d900c73d0 con 0x7f4d900c8f50 2024-04-24T23:16:34.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.254+0000 7f4d94c00640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f4d900c8ac0 con 0x7f4d900c9330 2024-04-24T23:16:34.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.254+0000 7f4d94c00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f4d900c6e70 con 0x7f4d90152bc0 2024-04-24T23:16:34.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.254+0000 7f4d8d400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f4d900c8f50 0x7f4d9015bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:34.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.254+0000 7f4d8e800640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f4d900c9330 0x7f4d9015c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:34.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.254+0000 7f4d8de00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4d90152bc0 0x7f4d90159390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:34.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.254+0000 7f4d8e800640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f4d900c9330 0x7f4d9015c440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:58460/0 (socket says 172.21.15.6:58460) 2024-04-24T23:16:34.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.255+0000 7f4d8e800640 1 -- 172.21.15.6:0/197454943 learned_addr learned my addr 172.21.15.6:0/197454943 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:16:34.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.255+0000 7f4d8de00640 1 -- 172.21.15.6:0/197454943 >> v2:172.21.15.190:3300/0 conn(0x7f4d900c9330 msgr2=0x7f4d9015c440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:16:34.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.255+0000 7f4d8de00640 1 --2- 172.21.15.6:0/197454943 >> v2:172.21.15.190:3300/0 conn(0x7f4d900c9330 0x7f4d9015c440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:34.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.255+0000 7f4d8de00640 1 -- 172.21.15.6:0/197454943 >> v2:172.21.15.6:3300/0 conn(0x7f4d900c8f50 msgr2=0x7f4d9015bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:16:34.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.255+0000 7f4d8de00640 1 --2- 172.21.15.6:0/197454943 >> v2:172.21.15.6:3300/0 conn(0x7f4d900c8f50 0x7f4d9015bf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:34.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.255+0000 7f4d8de00640 1 -- 172.21.15.6:0/197454943 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d901598d0 con 0x7f4d90152bc0 2024-04-24T23:16:34.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.255+0000 7f4d8e800640 1 --2- 172.21.15.6:0/197454943 >> v2:172.21.15.190:3300/0 conn(0x7f4d900c9330 0x7f4d9015c440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:16:34.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.256+0000 7f4d8de00640 1 --2- 172.21.15.6:0/197454943 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4d90152bc0 0x7f4d90159390 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f4d90159aa0 tx=0x7f4d8002a060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5574102f97d65a7b server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:34.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.256+0000 7f4d8ca00640 1 -- 172.21.15.6:0/197454943 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d80037070 con 0x7f4d90152bc0 2024-04-24T23:16:34.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.257+0000 7f4d8ca00640 1 -- 172.21.15.6:0/197454943 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d8002adb0 con 0x7f4d90152bc0 2024-04-24T23:16:34.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.257+0000 7f4d8ca00640 1 -- 172.21.15.6:0/197454943 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d80032660 con 0x7f4d90152bc0 2024-04-24T23:16:34.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.257+0000 7f4d94c00640 1 -- 172.21.15.6:0/197454943 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4d90152bc0 msgr2=0x7f4d90159390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:34.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.257+0000 7f4d94c00640 1 --2- 172.21.15.6:0/197454943 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4d90152bc0 0x7f4d90159390 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f4d90159aa0 tx=0x7f4d8002a060 comp rx=0 tx=0).stop 2024-04-24T23:16:34.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.257+0000 7f4d94c00640 1 -- 172.21.15.6:0/197454943 shutdown_connections 2024-04-24T23:16:34.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.257+0000 7f4d94c00640 1 --2- 172.21.15.6:0/197454943 >> v2:172.21.15.190:3300/0 conn(0x7f4d900c9330 0x7f4d9015c440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:34.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.257+0000 7f4d94c00640 1 --2- 172.21.15.6:0/197454943 >> v2:172.21.15.6:3300/0 conn(0x7f4d900c8f50 0x7f4d9015bf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:34.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.257+0000 7f4d94c00640 1 --2- 172.21.15.6:0/197454943 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4d90152bc0 0x7f4d90159390 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:34.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.257+0000 7f4d94c00640 1 -- 172.21.15.6:0/197454943 >> 172.21.15.6:0/197454943 conn(0x7f4d9006b0e0 msgr2=0x7f4d9006b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:34.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.257+0000 7f4d94c00640 1 -- 172.21.15.6:0/197454943 shutdown_connections 2024-04-24T23:16:34.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.258+0000 7f4d94c00640 1 -- 172.21.15.6:0/197454943 wait complete. 2024-04-24T23:16:34.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.259+0000 7f4d94c00640 1 Processor -- start 2024-04-24T23:16:34.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.259+0000 7f4d94c00640 1 -- start start 2024-04-24T23:16:34.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.260+0000 7f4d94c00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4d900c8f50 0x7f4d90145010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:34.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.260+0000 7f4d94c00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4d900c9330 0x7f4d90145e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:34.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.260+0000 7f4d94c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4d90152bc0 0x7f4d90147290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:34.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.260+0000 7f4d94c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f4d900c8470 con 0x7f4d90152bc0 2024-04-24T23:16:34.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.260+0000 7f4d94c00640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f4d900c6700 con 0x7f4d900c8f50 2024-04-24T23:16:34.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.260+0000 7f4d94c00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f4d900c8750 con 0x7f4d900c9330 2024-04-24T23:16:34.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.261+0000 7f4d8e800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4d90152bc0 0x7f4d90147290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:34.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.261+0000 7f4d8de00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4d900c8f50 0x7f4d90145010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:34.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.261+0000 7f4d8d400640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4d900c9330 0x7f4d90145e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:34.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.261+0000 7f4d8de00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4d900c8f50 0x7f4d90145010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:58472/0 (socket says 172.21.15.6:58472) 2024-04-24T23:16:34.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.261+0000 7f4d8de00640 1 -- 172.21.15.6:0/2724005065 learned_addr learned my addr 172.21.15.6:0/2724005065 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:16:34.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.261+0000 7f4d8de00640 1 -- 172.21.15.6:0/2724005065 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4d900c9330 msgr2=0x7f4d90145e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:34.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.261+0000 7f4d8de00640 1 --2- 172.21.15.6:0/2724005065 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4d900c9330 0x7f4d90145e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:34.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.261+0000 7f4d8de00640 1 -- 172.21.15.6:0/2724005065 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4d90152bc0 msgr2=0x7f4d90147290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:34.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.261+0000 7f4d8de00640 1 --2- 172.21.15.6:0/2724005065 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4d90152bc0 0x7f4d90147290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:34.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.261+0000 7f4d8de00640 1 -- 172.21.15.6:0/2724005065 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d90159ad0 con 0x7f4d900c8f50 2024-04-24T23:16:34.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.261+0000 7f4d8de00640 1 --2- 172.21.15.6:0/2724005065 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4d900c8f50 0x7f4d90145010 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f4d80000c00 tx=0x7f4d800046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:34.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.262+0000 7f4d86000640 1 -- 172.21.15.6:0/2724005065 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d80037040 con 0x7f4d900c8f50 2024-04-24T23:16:34.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.262+0000 7f4d86000640 1 -- 172.21.15.6:0/2724005065 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d80042070 con 0x7f4d900c8f50 2024-04-24T23:16:34.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.262+0000 7f4d86000640 1 -- 172.21.15.6:0/2724005065 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d80006e10 con 0x7f4d900c8f50 2024-04-24T23:16:34.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.262+0000 7f4d94c00640 1 -- 172.21.15.6:0/2724005065 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d90144a50 con 0x7f4d900c8f50 2024-04-24T23:16:34.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.262+0000 7f4d94c00640 1 -- 172.21.15.6:0/2724005065 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d90159d70 con 0x7f4d900c8f50 2024-04-24T23:16:34.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.263+0000 7f4d94c00640 1 -- 172.21.15.6:0/2724005065 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d900767b0 con 0x7f4d900c8f50 2024-04-24T23:16:34.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.264+0000 7f4d86000640 1 -- 172.21.15.6:0/2724005065 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f4d8003d660 con 0x7f4d900c8f50 2024-04-24T23:16:34.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.265+0000 7f4d86000640 1 --2- 172.21.15.6:0/2724005065 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f4d6c06be60 0x7f4d6c06e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:34.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.265+0000 7f4d86000640 1 -- 172.21.15.6:0/2724005065 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6590+0+0 (secure 0 0 0) 0x7f4d800ab5e0 con 0x7f4d900c8f50 2024-04-24T23:16:34.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.265+0000 7f4d8d400640 1 --2- 172.21.15.6:0/2724005065 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f4d6c06be60 0x7f4d6c06e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:34.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.266+0000 7f4d8d400640 1 --2- 172.21.15.6:0/2724005065 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f4d6c06be60 0x7f4d6c06e320 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f4d901f4cf0 tx=0x7f4d78009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:34.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.269+0000 7f4d86000640 1 -- 172.21.15.6:0/2724005065 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f4d80079530 con 0x7f4d900c8f50 2024-04-24T23:16:34.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.472+0000 7f4d94c00640 1 -- 172.21.15.6:0/2724005065 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4d901596b0 con 0x7f4d900c8f50 2024-04-24T23:16:34.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.474+0000 7f4d86000640 1 -- 172.21.15.6:0/2724005065 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+18554 (secure 0 0 0) 0x7f4d8007d830 con 0x7f4d900c8f50 2024-04-24T23:16:34.475 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:16:34.475 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":26,"fsid":"18bef68d-5e6d-4ec5-862b-b6f206e2588d","created":"2024-04-24T23:15:12.535072+0000","modified":"2024-04-24T23:16:18.218008+0000","last_up_change":"2024-04-24T23:15:23.773223+0000","last_in_change":"2024-04-24T23:15:14.905328+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T23:15:24.747441+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-24T23:15:34.051800+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-24T23:15:35.436870+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-24T23:15:39.034257+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"3b7a212d-d498-462e-90bf-ed493c803a09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":3022569437},{"type":"v1","addr":"172.21.15.6:6809","nonce":3022569437}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":3022569437},{"type":"v1","addr":"172.21.15.6:6811","nonce":3022569437}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":3022569437},{"type":"v1","addr":"172.21.15.6:6815","nonce":3022569437}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":3022569437},{"type":"v1","addr":"172.21.15.6:6813","nonce":3022569437}]},"public_addr":"172.21.15.6:6809/3022569437","cluster_addr":"172.21.15.6:6811/3022569437","heartbeat_back_addr":"172.21.15.6:6815/3022569437","heartbeat_front_addr":"172.21.15.6:6813/3022569437","state":["exists","up"]},{"osd":1,"uuid":"7c97f5a2-1735-4775-9f15-eb50099b19d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":3687836888},{"type":"v1","addr":"172.21.15.6:6817","nonce":3687836888}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":3687836888},{"type":"v1","addr":"172.21.15.6:6819","nonce":3687836888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":3687836888},{"type":"v1","addr":"172.21.15.6:6823","nonce":3687836888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":3687836888},{"type":"v1","addr":"172.21.15.6:6821","nonce":3687836888}]},"public_addr":"172.21.15.6:6817/3687836888","cluster_addr":"172.21.15.6:6819/3687836888","heartbeat_back_addr":"172.21.15.6:6823/3687836888","heartbeat_front_addr":"172.21.15.6:6821/3687836888","state":["exists","up"]},{"osd":2,"uuid":"d86bbc18-438c-4ec2-8018-ba7dd78f790b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":1020213724},{"type":"v1","addr":"172.21.15.6:6801","nonce":1020213724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":1020213724},{"type":"v1","addr":"172.21.15.6:6803","nonce":1020213724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":1020213724},{"type":"v1","addr":"172.21.15.6:6807","nonce":1020213724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":1020213724},{"type":"v1","addr":"172.21.15.6:6805","nonce":1020213724}]},"public_addr":"172.21.15.6:6801/1020213724","cluster_addr":"172.21.15.6:6803/1020213724","heartbeat_back_addr":"172.21.15.6:6807/1020213724","heartbeat_front_addr":"172.21.15.6:6805/1020213724","state":["exists","up"]},{"osd":3,"uuid":"3754ae16-4248-40dd-95f9-0f59c0eb5426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6824","nonce":3163883468},{"type":"v1","addr":"172.21.15.6:6825","nonce":3163883468}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":3163883468},{"type":"v1","addr":"172.21.15.6:6827","nonce":3163883468}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":3163883468},{"type":"v1","addr":"172.21.15.6:6831","nonce":3163883468}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":3163883468},{"type":"v1","addr":"172.21.15.6:6829","nonce":3163883468}]},"public_addr":"172.21.15.6:6825/3163883468","cluster_addr":"172.21.15.6:6827/3163883468","heartbeat_back_addr":"172.21.15.6:6831/3163883468","heartbeat_front_addr":"172.21.15.6:6829/3163883468","state":["exists","up"]},{"osd":4,"uuid":"8b2913a0-d006-479c-84ed-5b21e0e6c392","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":1989500934},{"type":"v1","addr":"172.21.15.190:6809","nonce":1989500934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":1989500934},{"type":"v1","addr":"172.21.15.190:6811","nonce":1989500934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":1989500934},{"type":"v1","addr":"172.21.15.190:6815","nonce":1989500934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":1989500934},{"type":"v1","addr":"172.21.15.190:6813","nonce":1989500934}]},"public_addr":"172.21.15.190:6809/1989500934","cluster_addr":"172.21.15.190:6811/1989500934","heartbeat_back_addr":"172.21.15.190:6815/1989500934","heartbeat_front_addr":"172.21.15.190:6813/1989500934","state":["exists","up"]},{"osd":5,"uuid":"499b79d0-0585-4dae-ab69-196a551793da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":3687548412},{"type":"v1","addr":"172.21.15.190:6801","nonce":3687548412}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":3687548412},{"type":"v1","addr":"172.21.15.190:6803","nonce":3687548412}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":3687548412},{"type":"v1","addr":"172.21.15.190:6807","nonce":3687548412}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":3687548412},{"type":"v1","addr":"172.21.15.190:6805","nonce":3687548412}]},"public_addr":"172.21.15.190:6801/3687548412","cluster_addr":"172.21.15.190:6803/3687548412","heartbeat_back_addr":"172.21.15.190:6807/3687548412","heartbeat_front_addr":"172.21.15.190:6805/3687548412","state":["exists","up"]},{"osd":6,"uuid":"6ba6df3b-4a6b-4a89-a890-0f79f387d4d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":870571564},{"type":"v1","addr":"172.21.15.190:6817","nonce":870571564}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":870571564},{"type":"v1","addr":"172.21.15.190:6819","nonce":870571564}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":870571564},{"type":"v1","addr":"172.21.15.190:6823","nonce":870571564}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":870571564},{"type":"v1","addr":"172.21.15.190:6821","nonce":870571564}]},"public_addr":"172.21.15.190:6817/870571564","cluster_addr":"172.21.15.190:6819/870571564","heartbeat_back_addr":"172.21.15.190:6823/870571564","heartbeat_front_addr":"172.21.15.190:6821/870571564","state":["exists","up"]},{"osd":7,"uuid":"e9668b1c-a000-4332-86e8-4efd73b02409","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":123639197},{"type":"v1","addr":"172.21.15.190:6825","nonce":123639197}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":123639197},{"type":"v1","addr":"172.21.15.190:6827","nonce":123639197}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":123639197},{"type":"v1","addr":"172.21.15.190:6831","nonce":123639197}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":123639197},{"type":"v1","addr":"172.21.15.190:6829","nonce":123639197}]},"public_addr":"172.21.15.190:6825/123639197","cluster_addr":"172.21.15.190:6827/123639197","heartbeat_back_addr":"172.21.15.190:6831/123639197","heartbeat_front_addr":"172.21.15.190:6829/123639197","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.321646+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.363873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.466963+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.531989+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:19.990707+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.043543+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.032492+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.115004+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.2f","mappings":[{"from":0,"to":7}]},{"pgid":"4.19","mappings":[{"from":7,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T23:16:34.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.476+0000 7f4d94c00640 1 -- 172.21.15.6:0/2724005065 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f4d6c06be60 msgr2=0x7f4d6c06e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:34.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.476+0000 7f4d94c00640 1 --2- 172.21.15.6:0/2724005065 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f4d6c06be60 0x7f4d6c06e320 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f4d901f4cf0 tx=0x7f4d78009190 comp rx=0 tx=0).stop 2024-04-24T23:16:34.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.476+0000 7f4d94c00640 1 -- 172.21.15.6:0/2724005065 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4d900c8f50 msgr2=0x7f4d90145010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:34.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.476+0000 7f4d94c00640 1 --2- 172.21.15.6:0/2724005065 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4d900c8f50 0x7f4d90145010 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f4d80000c00 tx=0x7f4d800046c0 comp rx=0 tx=0).stop 2024-04-24T23:16:34.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.476+0000 7f4d94c00640 1 -- 172.21.15.6:0/2724005065 shutdown_connections 2024-04-24T23:16:34.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.476+0000 7f4d94c00640 1 --2- 172.21.15.6:0/2724005065 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f4d6c06be60 0x7f4d6c06e320 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:34.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.476+0000 7f4d94c00640 1 --2- 172.21.15.6:0/2724005065 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4d900c9330 0x7f4d90145e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:34.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.476+0000 7f4d94c00640 1 --2- 172.21.15.6:0/2724005065 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4d90152bc0 0x7f4d90147290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:34.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.476+0000 7f4d94c00640 1 --2- 172.21.15.6:0/2724005065 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4d900c8f50 0x7f4d90145010 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:34.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.476+0000 7f4d94c00640 1 -- 172.21.15.6:0/2724005065 >> 172.21.15.6:0/2724005065 conn(0x7f4d9006b0e0 msgr2=0x7f4d90150280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:34.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.476+0000 7f4d94c00640 1 -- 172.21.15.6:0/2724005065 shutdown_connections 2024-04-24T23:16:34.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.477+0000 7f4d94c00640 1 -- 172.21.15.6:0/2724005065 wait complete. 2024-04-24T23:16:34.495 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:16:34.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.686+0000 7f18c4800640 1 Processor -- start 2024-04-24T23:16:34.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.687+0000 7f18c4800640 1 -- start start 2024-04-24T23:16:34.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.688+0000 7f18c4800640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f18bc0ca520 0x7f18bc0cb810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:34.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.688+0000 7f18c4800640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f18bc15e2b0 0x7f18bc15e760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:34.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.688+0000 7f18c4800640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f18bc0ca900 con 0x7f18bc15c1b0 2024-04-24T23:16:34.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.688+0000 7f18c4800640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f18bc0ccf70 con 0x7f18bc15e2b0 2024-04-24T23:16:34.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.688+0000 7f18c4800640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f18bc0cc360 con 0x7f18bc0ca520 2024-04-24T23:16:34.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.688+0000 7f18c2800640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f18bc15e2b0 0x7f18bc15e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:34.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.688+0000 7f18c1e00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f18bc0ca520 0x7f18bc0cb810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:34.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.688+0000 7f18c1400640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f18bc15c1b0 0x7f18bc0cd400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.6:43696/0 (socket says 172.21.15.6:43696) 2024-04-24T23:16:34.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.689+0000 7f18c2800640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f18bc15e2b0 0x7f18bc15e760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:58474/0 (socket says 172.21.15.6:58474) 2024-04-24T23:16:34.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.689+0000 7f18c1400640 1 -- 172.21.15.6:0/197204036 learned_addr learned my addr 172.21.15.6:0/197204036 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:16:34.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.689+0000 7f18c2800640 1 -- 172.21.15.6:0/197204036 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f18bc0ca520 msgr2=0x7f18bc0cb810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:16:34.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.689+0000 7f18c2800640 1 --2- 172.21.15.6:0/197204036 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f18bc0ca520 0x7f18bc0cb810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:34.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.689+0000 7f18c2800640 1 -- 172.21.15.6:0/197204036 >> v1:172.21.15.190:6789/0 conn(0x7f18bc15c1b0 legacy=0x7f18bc0cd400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:34.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.689+0000 7f18c2800640 1 -- 172.21.15.6:0/197204036 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18bc0cc640 con 0x7f18bc15e2b0 2024-04-24T23:16:34.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.689+0000 7f18c0a00640 1 -- 172.21.15.6:0/197204036 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2006801054 0 0) 0x7f18bc0ca900 con 0x7f18bc15c1b0 2024-04-24T23:16:34.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.689+0000 7f18c1e00640 1 --2- 172.21.15.6:0/197204036 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f18bc0ca520 0x7f18bc0cb810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:16:34.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.690+0000 7f18c2800640 1 --2- 172.21.15.6:0/197204036 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f18bc15e2b0 0x7f18bc15e760 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f18b8009d90 tx=0x7f18b802a6f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=15ea43ea3678f96b server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:34.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.690+0000 7f18c0a00640 1 -- 172.21.15.6:0/197204036 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18b8007740 con 0x7f18bc15e2b0 2024-04-24T23:16:34.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.690+0000 7f18c0a00640 1 -- 172.21.15.6:0/197204036 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f18b80078e0 con 0x7f18bc15e2b0 2024-04-24T23:16:34.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.690+0000 7f18c0a00640 1 -- 172.21.15.6:0/197204036 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18b8007cb0 con 0x7f18bc15e2b0 2024-04-24T23:16:34.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.691+0000 7f18c4800640 1 -- 172.21.15.6:0/197204036 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f18bc15e2b0 msgr2=0x7f18bc15e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:34.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.691+0000 7f18c4800640 1 --2- 172.21.15.6:0/197204036 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f18bc15e2b0 0x7f18bc15e760 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f18b8009d90 tx=0x7f18b802a6f0 comp rx=0 tx=0).stop 2024-04-24T23:16:34.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.691+0000 7f18c4800640 1 -- 172.21.15.6:0/197204036 shutdown_connections 2024-04-24T23:16:34.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.691+0000 7f18c4800640 1 --2- 172.21.15.6:0/197204036 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f18bc15e2b0 0x7f18bc15e760 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:34.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.691+0000 7f18c4800640 1 --2- 172.21.15.6:0/197204036 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f18bc0ca520 0x7f18bc0cb810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:34.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.691+0000 7f18c4800640 1 -- 172.21.15.6:0/197204036 >> 172.21.15.6:0/197204036 conn(0x7f18bc06c0f0 msgr2=0x7f18bc066ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:34.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.691+0000 7f18c4800640 1 -- 172.21.15.6:0/197204036 shutdown_connections 2024-04-24T23:16:34.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.691+0000 7f18c4800640 1 -- 172.21.15.6:0/197204036 wait complete. 2024-04-24T23:16:34.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.693+0000 7f18c4800640 1 Processor -- start 2024-04-24T23:16:34.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.693+0000 7f18c4800640 1 -- start start 2024-04-24T23:16:34.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.694+0000 7f18c4800640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f18bc0ca520 0x7f18bc156cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:34.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.694+0000 7f18c4800640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f18bc15c1b0 0x7f18bc1f0630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:34.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.694+0000 7f18c4800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f18bc15e2b0 0x7f18bc1f1b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:34.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.694+0000 7f18c4800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f18bc067890 con 0x7f18bc15e2b0 2024-04-24T23:16:34.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.694+0000 7f18c4800640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f18bc0ccc00 con 0x7f18bc0ca520 2024-04-24T23:16:34.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.694+0000 7f18c4800640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f18bc067bc0 con 0x7f18bc15c1b0 2024-04-24T23:16:34.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.694+0000 7f18c2800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f18bc15e2b0 0x7f18bc1f1b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:34.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.694+0000 7f18c2800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f18bc15e2b0 0x7f18bc1f1b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:35676/0 (socket says 172.21.15.6:35676) 2024-04-24T23:16:34.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.694+0000 7f18c2800640 1 -- 172.21.15.6:0/1241219125 learned_addr learned my addr 172.21.15.6:0/1241219125 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:16:34.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.694+0000 7f18c1400640 1 --2- 172.21.15.6:0/1241219125 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f18bc15c1b0 0x7f18bc1f0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:34.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.694+0000 7f18c1e00640 1 --2- 172.21.15.6:0/1241219125 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f18bc0ca520 0x7f18bc156cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:34.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.695+0000 7f18c2800640 1 -- 172.21.15.6:0/1241219125 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f18bc15c1b0 msgr2=0x7f18bc1f0630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:34.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.695+0000 7f18c2800640 1 --2- 172.21.15.6:0/1241219125 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f18bc15c1b0 0x7f18bc1f0630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:34.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.695+0000 7f18c2800640 1 -- 172.21.15.6:0/1241219125 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f18bc0ca520 msgr2=0x7f18bc156cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:34.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.695+0000 7f18c2800640 1 --2- 172.21.15.6:0/1241219125 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f18bc0ca520 0x7f18bc156cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:34.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.695+0000 7f18c2800640 1 -- 172.21.15.6:0/1241219125 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18bc1556b0 con 0x7f18bc15e2b0 2024-04-24T23:16:34.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.695+0000 7f18c2800640 1 --2- 172.21.15.6:0/1241219125 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f18bc15e2b0 0x7f18bc1f1b20 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f18b8009eb0 tx=0x7f18b8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:34.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.695+0000 7f18b6000640 1 -- 172.21.15.6:0/1241219125 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18b803f070 con 0x7f18bc15e2b0 2024-04-24T23:16:34.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.696+0000 7f18b6000640 1 -- 172.21.15.6:0/1241219125 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f18b8031bc0 con 0x7f18bc15e2b0 2024-04-24T23:16:34.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.696+0000 7f18c4800640 1 -- 172.21.15.6:0/1241219125 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18bc155970 con 0x7f18bc15e2b0 2024-04-24T23:16:34.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.696+0000 7f18b6000640 1 -- 172.21.15.6:0/1241219125 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18b8009450 con 0x7f18bc15e2b0 2024-04-24T23:16:34.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.696+0000 7f18c4800640 1 -- 172.21.15.6:0/1241219125 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18bc155f30 con 0x7f18bc15e2b0 2024-04-24T23:16:34.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.697+0000 7f18c4800640 1 -- 172.21.15.6:0/1241219125 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f18bc0767b0 con 0x7f18bc15e2b0 2024-04-24T23:16:34.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.698+0000 7f18b6000640 1 -- 172.21.15.6:0/1241219125 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f18b803b410 con 0x7f18bc15e2b0 2024-04-24T23:16:34.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.699+0000 7f18b6000640 1 --2- 172.21.15.6:0/1241219125 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f18a006be30 0x7f18a006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:34.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.699+0000 7f18b6000640 1 -- 172.21.15.6:0/1241219125 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6590+0+0 (secure 0 0 0) 0x7f18b80ac7b0 con 0x7f18bc15e2b0 2024-04-24T23:16:34.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.700+0000 7f18c1e00640 1 --2- 172.21.15.6:0/1241219125 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f18a006be30 0x7f18a006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:34.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.700+0000 7f18c1e00640 1 --2- 172.21.15.6:0/1241219125 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f18a006be30 0x7f18a006e2f0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f18b0009560 tx=0x7f18b0009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:34.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.704+0000 7f18b6000640 1 -- 172.21.15.6:0/1241219125 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f18b807a790 con 0x7f18bc15e2b0 2024-04-24T23:16:34.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.940+0000 7f18c4800640 1 -- 172.21.15.6:0/1241219125 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f18bc15e020 con 0x7f18bc15e2b0 2024-04-24T23:16:34.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.941+0000 7f18b6000640 1 -- 172.21.15.6:0/1241219125 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) ==== 94+0+6902 (secure 0 0 0) 0x7f18b807ea90 con 0x7f18bc15e2b0 2024-04-24T23:16:34.943 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:16:34.943 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":18,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4197,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/105267896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":105267896},{"type":"v1","addr":"172.21.15.6:6834","nonce":105267896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:16:32.310838+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4282,"mds_2":4235,"mds_3":4232,"mds_4":4203},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4203":{"gid":4203,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":17,"addr":"172.21.15.6:6837/2590421094","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":2590421094},{"type":"v1","addr":"172.21.15.6:6837","nonce":2590421094}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4232":{"gid":4232,"name":"b","rank":3,"incarnation":15,"state":"up:active","state_seq":15,"addr":"172.21.15.190:6838/2032280158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2032280158},{"type":"v1","addr":"172.21.15.190:6838","nonce":2032280158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4235":{"gid":4235,"name":"d","rank":2,"incarnation":13,"state":"up:active","state_seq":14,"addr":"172.21.15.190:6839/1572872451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":1572872451},{"type":"v1","addr":"172.21.15.190:6839","nonce":1572872451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4282":{"gid":4282,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":12,"addr":"172.21.15.190:6837/580315269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":580315269},{"type":"v1","addr":"172.21.15.190:6837","nonce":580315269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/489092393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":489092393},{"type":"v1","addr":"172.21.15.6:6835","nonce":489092393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4306,"qdb_cluster":[4203,4232,4235,4282,4306]},"id":1}]} 2024-04-24T23:16:34.943 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 18 2024-04-24T23:16:34.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.944+0000 7f18c4800640 1 -- 172.21.15.6:0/1241219125 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f18a006be30 msgr2=0x7f18a006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:34.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.944+0000 7f18c4800640 1 --2- 172.21.15.6:0/1241219125 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f18a006be30 0x7f18a006e2f0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f18b0009560 tx=0x7f18b0009190 comp rx=0 tx=0).stop 2024-04-24T23:16:34.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.944+0000 7f18c4800640 1 -- 172.21.15.6:0/1241219125 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f18bc15e2b0 msgr2=0x7f18bc1f1b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:34.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.944+0000 7f18c4800640 1 --2- 172.21.15.6:0/1241219125 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f18bc15e2b0 0x7f18bc1f1b20 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f18b8009eb0 tx=0x7f18b8004a40 comp rx=0 tx=0).stop 2024-04-24T23:16:34.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.944+0000 7f18c4800640 1 -- 172.21.15.6:0/1241219125 shutdown_connections 2024-04-24T23:16:34.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.944+0000 7f18c4800640 1 --2- 172.21.15.6:0/1241219125 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f18a006be30 0x7f18a006e2f0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:34.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.944+0000 7f18c4800640 1 --2- 172.21.15.6:0/1241219125 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f18bc15c1b0 0x7f18bc1f0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:34.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.944+0000 7f18c4800640 1 --2- 172.21.15.6:0/1241219125 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f18bc15e2b0 0x7f18bc1f1b20 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:34.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.944+0000 7f18c4800640 1 --2- 172.21.15.6:0/1241219125 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f18bc0ca520 0x7f18bc156cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:34.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.944+0000 7f18c4800640 1 -- 172.21.15.6:0/1241219125 >> 172.21.15.6:0/1241219125 conn(0x7f18bc06c0f0 msgr2=0x7f18bc0d0c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:34.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.944+0000 7f18c4800640 1 -- 172.21.15.6:0/1241219125 shutdown_connections 2024-04-24T23:16:34.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:34.945+0000 7f18c4800640 1 -- 172.21.15.6:0/1241219125 wait complete. 2024-04-24T23:16:35.568 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 18, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-24T23:15:36.400716+0000', 'modified': '2024-04-24T23:16:32.310838+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4306, 'mds_1': 4282, 'mds_2': 4235, 'mds_3': 4232, 'mds_4': 4203}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4203': {'gid': 4203, 'name': 'e', 'rank': 4, 'incarnation': 17, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.6:6837/2590421094', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6836', 'nonce': 2590421094}, {'type': 'v1', 'addr': '172.21.15.6:6837', 'nonce': 2590421094}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4232': {'gid': 4232, 'name': 'b', 'rank': 3, 'incarnation': 15, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.190:6838/2032280158', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6835', 'nonce': 2032280158}, {'type': 'v1', 'addr': '172.21.15.190:6838', 'nonce': 2032280158}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4235': {'gid': 4235, 'name': 'd', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.190:6839/1572872451', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6836', 'nonce': 1572872451}, {'type': 'v1', 'addr': '172.21.15.190:6839', 'nonce': 1572872451}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4282': {'gid': 4282, 'name': 'f', 'rank': 1, 'incarnation': 9, 'state': 'up:active', 'state_seq': 12, 'addr': '172.21.15.190:6837/580315269', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6834', 'nonce': 580315269}, {'type': 'v1', 'addr': '172.21.15.190:6837', 'nonce': 580315269}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4306': {'gid': 4306, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.6:6835/489092393', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6833', 'nonce': 489092393}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 489092393}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4306, 'qdb_cluster': [4203, 4232, 4235, 4282, 4306]} 2024-04-24T23:16:35.569 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-04-24T23:16:35.569 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.e status 2024-04-24T23:16:35.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:35.771+0000 7fcb72a00640 1 Processor -- start 2024-04-24T23:16:35.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:35.771+0000 7fcb72a00640 1 -- start start 2024-04-24T23:16:35.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:35.774+0000 7fcb72a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fcb6c152d40 0x7fcb6c1592b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:35.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:35.774+0000 7fcb72a00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcb6c153120 con 0x7fcb6c06d6d0 2024-04-24T23:16:35.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:35.774+0000 7fcb72a00640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcb6c06d230 con 0x7fcb6c0c4770 2024-04-24T23:16:35.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:35.774+0000 7fcb72a00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7fcb6c0c8aa0 con 0x7fcb6c152d40 2024-04-24T23:16:35.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:35.774+0000 7fcb63e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fcb6c06d6d0 0x7fcb6c0c4060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:40256/0 (socket says 172.21.15.6:40256) 2024-04-24T23:16:35.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:35.774+0000 7fcb63e00640 1 -- 172.21.15.6:0/3286637124 learned_addr learned my addr 172.21.15.6:0/3286637124 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:16:35.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:35.774+0000 7fcb6be00640 1 --2- 172.21.15.6:0/3286637124 >> v2:172.21.15.6:3300/0 conn(0x7fcb6c152d40 0x7fcb6c1592b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:35.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:35.775+0000 7fcb6b400640 1 -- 172.21.15.6:0/3286637124 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3942978544 0 0) 0x7fcb6c153120 con 0x7fcb6c06d6d0 2024-04-24T23:16:35.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:35.775+0000 7fcb6be00640 1 -- 172.21.15.6:0/3286637124 >> v1:172.21.15.190:6789/0 conn(0x7fcb6c0c4770 legacy=0x7fcb6c163320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:35.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:35.775+0000 7fcb6be00640 1 -- 172.21.15.6:0/3286637124 >> v1:172.21.15.6:6789/0 conn(0x7fcb6c06d6d0 legacy=0x7fcb6c0c4060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:35.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:35.775+0000 7fcb6be00640 1 -- 172.21.15.6:0/3286637124 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb6c0c63d0 con 0x7fcb6c152d40 2024-04-24T23:16:35.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:35.776+0000 7fcb6be00640 1 --2- 172.21.15.6:0/3286637124 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcb6c152d40 0x7fcb6c1592b0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fcb5c009720 tx=0x7fcb5c02a150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d5c3156e9e68a63 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:35.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:35.776+0000 7fcb6b400640 1 -- 172.21.15.6:0/3286637124 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb5c037070 con 0x7fcb6c152d40 2024-04-24T23:16:35.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:35.776+0000 7fcb6b400640 1 -- 172.21.15.6:0/3286637124 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb5c032440 con 0x7fcb6c152d40 2024-04-24T23:16:35.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:35.776+0000 7fcb6b400640 1 -- 172.21.15.6:0/3286637124 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb5c02ae60 con 0x7fcb6c152d40 2024-04-24T23:16:35.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:35.777+0000 7fcb72a00640 1 -- 172.21.15.6:0/3286637124 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcb6c152d40 msgr2=0x7fcb6c1592b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:35.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:35.777+0000 7fcb72a00640 1 --2- 172.21.15.6:0/3286637124 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcb6c152d40 0x7fcb6c1592b0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fcb5c009720 tx=0x7fcb5c02a150 comp rx=0 tx=0).stop 2024-04-24T23:16:35.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:35.777+0000 7fcb72a00640 1 -- 172.21.15.6:0/3286637124 shutdown_connections 2024-04-24T23:16:35.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:35.777+0000 7fcb72a00640 1 --2- 172.21.15.6:0/3286637124 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcb6c152d40 0x7fcb6c1592b0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:35.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:35.777+0000 7fcb72a00640 1 -- 172.21.15.6:0/3286637124 >> 172.21.15.6:0/3286637124 conn(0x7fcb6c0c8c50 msgr2=0x7fcb6c0c9060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:35.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:35.777+0000 7fcb72a00640 1 -- 172.21.15.6:0/3286637124 shutdown_connections 2024-04-24T23:16:35.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:35.777+0000 7fcb72a00640 1 -- 172.21.15.6:0/3286637124 wait complete. 2024-04-24T23:16:35.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:35.779+0000 7fcb72a00640 1 Processor -- start 2024-04-24T23:16:35.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:35.779+0000 7fcb72a00640 1 -- start start 2024-04-24T23:16:35.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:35.780+0000 7fcb72a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcb6c06d6d0 0x7fcb6c0cb9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:35.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:35.780+0000 7fcb72a00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcb6c0c4770 0x7fcb6c071210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:35.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:35.780+0000 7fcb72a00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcb6c152d40 0x7fcb6c141f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:35.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:35.780+0000 7fcb72a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fcb6c06f8d0 con 0x7fcb6c06d6d0 2024-04-24T23:16:35.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:35.780+0000 7fcb72a00640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fcb6c158ab0 con 0x7fcb6c0c4770 2024-04-24T23:16:35.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:35.780+0000 7fcb72a00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fcb6c0c6a50 con 0x7fcb6c152d40 2024-04-24T23:16:35.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:35.780+0000 7fcb6be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcb6c06d6d0 0x7fcb6c0cb9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:35.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:35.780+0000 7fcb6be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcb6c06d6d0 0x7fcb6c0cb9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:35482/0 (socket says 172.21.15.6:35482) 2024-04-24T23:16:35.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:35.780+0000 7fcb6be00640 1 -- 172.21.15.6:0/3574306764 learned_addr learned my addr 172.21.15.6:0/3574306764 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:16:35.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:35.780+0000 7fcb63e00640 1 --2- 172.21.15.6:0/3574306764 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcb6c0c4770 0x7fcb6c071210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:35.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:35.780+0000 7fcb70a00640 1 --2- 172.21.15.6:0/3574306764 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcb6c152d40 0x7fcb6c141f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:35.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:35.780+0000 7fcb6be00640 1 -- 172.21.15.6:0/3574306764 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcb6c152d40 msgr2=0x7fcb6c141f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:35.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:35.780+0000 7fcb6be00640 1 --2- 172.21.15.6:0/3574306764 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcb6c152d40 0x7fcb6c141f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:35.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:35.781+0000 7fcb6be00640 1 -- 172.21.15.6:0/3574306764 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcb6c0c4770 msgr2=0x7fcb6c071210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:35.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:35.781+0000 7fcb6be00640 1 --2- 172.21.15.6:0/3574306764 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcb6c0c4770 0x7fcb6c071210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:35.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:35.781+0000 7fcb6be00640 1 -- 172.21.15.6:0/3574306764 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb6c1e3750 con 0x7fcb6c06d6d0 2024-04-24T23:16:35.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:35.781+0000 7fcb6be00640 1 --2- 172.21.15.6:0/3574306764 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcb6c06d6d0 0x7fcb6c0cb9c0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fcb5c009720 tx=0x7fcb5c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:35.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:35.781+0000 7fcb68c00640 1 -- 172.21.15.6:0/3574306764 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb5c037040 con 0x7fcb6c06d6d0 2024-04-24T23:16:35.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:35.781+0000 7fcb68c00640 1 -- 172.21.15.6:0/3574306764 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb5c004230 con 0x7fcb6c06d6d0 2024-04-24T23:16:35.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:35.781+0000 7fcb68c00640 1 -- 172.21.15.6:0/3574306764 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb5c03c670 con 0x7fcb6c06d6d0 2024-04-24T23:16:35.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:35.781+0000 7fcb72a00640 1 -- 172.21.15.6:0/3574306764 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb6c1e1db0 con 0x7fcb6c06d6d0 2024-04-24T23:16:35.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:35.781+0000 7fcb72a00640 1 -- 172.21.15.6:0/3574306764 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb6c0cb400 con 0x7fcb6c06d6d0 2024-04-24T23:16:35.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:35.784+0000 7fcb68c00640 1 -- 172.21.15.6:0/3574306764 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fcb5c03a070 con 0x7fcb6c06d6d0 2024-04-24T23:16:35.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:35.785+0000 7fcb68c00640 1 --2- 172.21.15.6:0/3574306764 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fcb4806be60 0x7fcb4806e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:35.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:35.785+0000 7fcb68c00640 1 -- 172.21.15.6:0/3574306764 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6590+0+0 (secure 0 0 0) 0x7fcb5c0abad0 con 0x7fcb6c06d6d0 2024-04-24T23:16:35.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:35.785+0000 7fcb63e00640 1 --2- 172.21.15.6:0/3574306764 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fcb4806be60 0x7fcb4806e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:35.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:35.786+0000 7fcb757ab740 1 Processor -- start 2024-04-24T23:16:35.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:35.786+0000 7fcb63e00640 1 --2- 172.21.15.6:0/3574306764 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fcb4806be60 0x7fcb4806e320 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fcb54007f70 tx=0x7fcb54007f00 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:35.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:35.786+0000 7fcb757ab740 1 -- start start 2024-04-24T23:16:35.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:35.787+0000 7fcb757ab740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55ebe7907060 0x55ebe7922900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:35.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:35.787+0000 7fcb757ab740 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x55ebe78d9bf0 0x55ebe78dbb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:35.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:35.787+0000 7fcb757ab740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ebe78dc040 0x55ebe7907440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:35.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:35.787+0000 7fcb757ab740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x55ebe77dded0 con 0x55ebe7907060 2024-04-24T23:16:35.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:35.787+0000 7fcb757ab740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55ebe7840df0 con 0x55ebe78dc040 2024-04-24T23:16:35.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:35.787+0000 7fcb757ab740 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x55ebe77e98d0 con 0x55ebe78d9bf0 2024-04-24T23:16:35.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:35.787+0000 7fcb6be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55ebe7907060 0x55ebe7922900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:35.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:35.787+0000 7fcb6be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55ebe7907060 0x55ebe7922900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:35490/0 (socket says 172.21.15.6:35490) 2024-04-24T23:16:35.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:35.787+0000 7fcb6be00640 1 -- 172.21.15.6:0/1534001147 learned_addr learned my addr 172.21.15.6:0/1534001147 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:16:35.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:35.787+0000 7fcb63e00640 1 --2- 172.21.15.6:0/1534001147 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x55ebe78d9bf0 0x55ebe78dbb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:35.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:35.787+0000 7fcb70a00640 1 --2- 172.21.15.6:0/1534001147 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ebe78dc040 0x55ebe7907440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:35.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:35.788+0000 7fcb6be00640 1 -- 172.21.15.6:0/1534001147 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x55ebe78d9bf0 msgr2=0x55ebe78dbb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:16:35.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:35.788+0000 7fcb6be00640 1 --2- 172.21.15.6:0/1534001147 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x55ebe78d9bf0 0x55ebe78dbb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:35.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:35.788+0000 7fcb6be00640 1 -- 172.21.15.6:0/1534001147 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ebe78dc040 msgr2=0x55ebe7907440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:16:35.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:35.788+0000 7fcb6be00640 1 --2- 172.21.15.6:0/1534001147 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ebe78dc040 0x55ebe7907440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:35.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:35.788+0000 7fcb6be00640 1 -- 172.21.15.6:0/1534001147 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ebe75552d0 con 0x55ebe7907060 2024-04-24T23:16:35.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:35.788+0000 7fcb70a00640 1 --2- 172.21.15.6:0/1534001147 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ebe78dc040 0x55ebe7907440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:16:35.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:35.788+0000 7fcb6be00640 1 --2- 172.21.15.6:0/1534001147 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55ebe7907060 0x55ebe7922900 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fcb5c075f10 tx=0x7fcb5c0799b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9921ac1dfed1834a server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:35.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:35.788+0000 7fcb5be00640 1 -- 172.21.15.6:0/1534001147 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb5c04a070 con 0x55ebe7907060 2024-04-24T23:16:35.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:35.788+0000 7fcb5be00640 1 -- 172.21.15.6:0/1534001147 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb5c0825e0 con 0x55ebe7907060 2024-04-24T23:16:35.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:35.788+0000 7fcb5be00640 1 -- 172.21.15.6:0/1534001147 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb5c076120 con 0x55ebe7907060 2024-04-24T23:16:35.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:35.789+0000 7fcb757ab740 1 -- 172.21.15.6:0/1534001147 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55ebe7907060 msgr2=0x55ebe7922900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:35.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:35.789+0000 7fcb757ab740 1 --2- 172.21.15.6:0/1534001147 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55ebe7907060 0x55ebe7922900 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fcb5c075f10 tx=0x7fcb5c0799b0 comp rx=0 tx=0).stop 2024-04-24T23:16:35.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:35.789+0000 7fcb757ab740 1 -- 172.21.15.6:0/1534001147 shutdown_connections 2024-04-24T23:16:35.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:35.789+0000 7fcb757ab740 1 --2- 172.21.15.6:0/1534001147 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x55ebe78d9bf0 0x55ebe78dbb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:35.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:35.789+0000 7fcb757ab740 1 --2- 172.21.15.6:0/1534001147 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ebe78dc040 0x55ebe7907440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:35.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:35.789+0000 7fcb757ab740 1 --2- 172.21.15.6:0/1534001147 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55ebe7907060 0x55ebe7922900 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:35.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:35.789+0000 7fcb757ab740 1 -- 172.21.15.6:0/1534001147 >> 172.21.15.6:0/1534001147 conn(0x55ebe78daa80 msgr2=0x55ebe7921f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:35.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:35.789+0000 7fcb757ab740 1 -- 172.21.15.6:0/1534001147 shutdown_connections 2024-04-24T23:16:35.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:35.789+0000 7fcb757ab740 1 -- 172.21.15.6:0/1534001147 wait complete. 2024-04-24T23:16:35.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:35.790+0000 7fcb757ab740 1 -- start start 2024-04-24T23:16:35.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:35.790+0000 7fcb757ab740 1 Processor -- start 2024-04-24T23:16:35.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:35.791+0000 7fcb757ab740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55ebe7907060 0x55ebe7930540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:35.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:35.791+0000 7fcb757ab740 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x55ebe791a110 0x55ebe7932520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:35.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:35.791+0000 7fcb757ab740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ebe7932a60 0x55ebe7904ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:35.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:35.791+0000 7fcb757ab740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x55ebe77fcd20 con 0x55ebe7907060 2024-04-24T23:16:35.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:35.791+0000 7fcb757ab740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55ebe783cdb0 con 0x55ebe7932a60 2024-04-24T23:16:35.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:35.791+0000 7fcb757ab740 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x55ebe78da080 con 0x55ebe791a110 2024-04-24T23:16:35.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:35.791+0000 7fcb757ab740 10 client.0 _ll_register_callbacks cb 0x55ebe7837f10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T23:16:35.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:35.791+0000 7fcb6be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55ebe7907060 0x55ebe7930540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:35.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:35.791+0000 7fcb6be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55ebe7907060 0x55ebe7930540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:35500/0 (socket says 172.21.15.6:35500) 2024-04-24T23:16:35.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:35.791+0000 7fcb6be00640 1 -- 172.21.15.6:0/1368480259 learned_addr learned my addr 172.21.15.6:0/1368480259 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:16:35.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:35.791+0000 7fcb70a00640 1 --2- 172.21.15.6:0/1368480259 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ebe7932a60 0x55ebe7904ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:35.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:35.791+0000 7fcb63e00640 1 --2- 172.21.15.6:0/1368480259 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x55ebe791a110 0x55ebe7932520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:35.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:35.792+0000 7fcb6be00640 1 -- 172.21.15.6:0/1368480259 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x55ebe791a110 msgr2=0x55ebe7932520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:16:35.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:35.792+0000 7fcb6be00640 1 --2- 172.21.15.6:0/1368480259 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x55ebe791a110 0x55ebe7932520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:35.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:35.792+0000 7fcb6be00640 1 -- 172.21.15.6:0/1368480259 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ebe7932a60 msgr2=0x55ebe7904ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:16:35.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:35.792+0000 7fcb6be00640 1 --2- 172.21.15.6:0/1368480259 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ebe7932a60 0x55ebe7904ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:35.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:35.792+0000 7fcb6be00640 1 -- 172.21.15.6:0/1368480259 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ebe761f6b0 con 0x55ebe7907060 2024-04-24T23:16:35.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:35.792+0000 7fcb63e00640 1 --2- 172.21.15.6:0/1368480259 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x55ebe791a110 0x55ebe7932520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:16:35.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:35.792+0000 7fcb70a00640 1 --2- 172.21.15.6:0/1368480259 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ebe7932a60 0x55ebe7904ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:16:35.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:35.792+0000 7fcb6be00640 1 --2- 172.21.15.6:0/1368480259 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55ebe7907060 0x55ebe7930540 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fcb5c02a740 tx=0x7fcb5c009190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4162189418f02072 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:35.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:35.792+0000 7fcb58c00640 10 client.0 ms_handle_connect on v2:172.21.15.6:3300/0 2024-04-24T23:16:35.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:35.792+0000 7fcb58c00640 1 -- 172.21.15.6:0/1368480259 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb5c082350 con 0x55ebe7907060 2024-04-24T23:16:35.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:35.792+0000 7fcb58c00640 1 -- 172.21.15.6:0/1368480259 <== mon.0 v2:172.21.15.6:3300/0 2 ==== osd_map(26..26 src has 1..26) ==== 6590+0+0 (secure 0 0 0) 0x7fcb5c080070 con 0x55ebe7907060 2024-04-24T23:16:35.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:35.792+0000 7fcb757ab740 1 -- 172.21.15.6:0/1368480259 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcb5c0825e0 con 0x55ebe7907060 2024-04-24T23:16:35.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:35.792+0000 7fcb58c00640 1 -- 172.21.15.6:0/1368480259 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_get_version_reply(handle=1 version=18) ==== 24+0+0 (secure 0 0 0) 0x7fcb5c080270 con 0x55ebe7907060 2024-04-24T23:16:35.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:35.792+0000 7fcb757ab740 10 client.4782 fetch_fsmap learned FSMap version 18 2024-04-24T23:16:35.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:35.792+0000 7fcb757ab740 1 -- 172.21.15.6:0/1368480259 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,fsmap=18,monmap=2+}) -- 0x55ebe7930a80 con 0x55ebe7907060 2024-04-24T23:16:35.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:35.793+0000 7fcb58c00640 1 -- 172.21.15.6:0/1368480259 <== mon.0 v2:172.21.15.6:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb5c0825e0 con 0x55ebe7907060 2024-04-24T23:16:35.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:35.793+0000 7fcb58c00640 1 -- 172.21.15.6:0/1368480259 <== mon.0 v2:172.21.15.6:3300/0 5 ==== fsmap(e 18) ==== 4792+0+0 (secure 0 0 0) 0x7fcb5c078f00 con 0x55ebe7907060 2024-04-24T23:16:35.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:35.793+0000 7fcb757ab740 10 client.4782 fetch_fsmap finished waiting for FSMap version 18 2024-04-24T23:16:35.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:35.793+0000 7fcb757ab740 10 client.4782 resolve_mds: resolved name 'e' to daemon mds.e 2024-04-24T23:16:35.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:35.793+0000 7fcb757ab740 20 client.4782 populate_metadata read hostname 'smithi006' 2024-04-24T23:16:35.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:35.793+0000 7fcb757ab740 1 --2- 172.21.15.6:0/1368480259 >> [v2:172.21.15.6:6836/2590421094,v1:172.21.15.6:6837/2590421094] conn(0x55ebe793bab0 0x55ebe793dea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:35.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:35.793+0000 7fcb757ab740 4 client.4782 mds_command: new command op to 4203 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T23:16:35.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:35.793+0000 7fcb757ab740 1 -- 172.21.15.6:0/1368480259 --> [v2:172.21.15.6:6836/2590421094,v1:172.21.15.6:6837/2590421094] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ebe784e330 con 0x55ebe793bab0 2024-04-24T23:16:35.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:35.793+0000 7fcb63e00640 1 --2- 172.21.15.6:0/1368480259 >> [v2:172.21.15.6:6836/2590421094,v1:172.21.15.6:6837/2590421094] conn(0x55ebe793bab0 0x55ebe793dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:35.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:35.794+0000 7fcb63e00640 1 --2- 172.21.15.6:0/1368480259 >> [v2:172.21.15.6:6836/2590421094,v1:172.21.15.6:6837/2590421094] conn(0x55ebe793bab0 0x55ebe793dea0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=29bd69fc7dfd869a server_cookie=f9ed887ccb853da2 in_seq=0 out_seq=0 2024-04-24T23:16:35.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:35.794+0000 7fcb58c00640 10 client.4782 ms_handle_connect on v2:172.21.15.6:6836/2590421094 2024-04-24T23:16:36.896 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-04-24T23:16:36.895+0000 7fef1ce00640 -1 mds.pinger is_rank_lagging: rank=4 was never sent ping request. 2024-04-24T23:16:37.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.321+0000 7fcb58c00640 1 -- 172.21.15.6:0/1368480259 <== mds.4 v2:172.21.15.6:6836/2590421094 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ebe784e330 con 0x55ebe793bab0 2024-04-24T23:16:37.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.321+0000 7fcb58c00640 10 client.4782 handle_command_reply: tid=0 2024-04-24T23:16:37.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.321+0000 7fcb757ab740 1 client.4782 shutdown 2024-04-24T23:16:37.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.322+0000 7fcb757ab740 1 -- 172.21.15.6:0/1368480259 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55ebe7907060 msgr2=0x55ebe7930540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:37.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.322+0000 7fcb757ab740 1 --2- 172.21.15.6:0/1368480259 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55ebe7907060 0x55ebe7930540 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fcb5c02a740 tx=0x7fcb5c009190 comp rx=0 tx=0).stop 2024-04-24T23:16:37.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.322+0000 7fcb757ab740 1 -- 172.21.15.6:0/1368480259 shutdown_connections 2024-04-24T23:16:37.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.322+0000 7fcb757ab740 1 --2- 172.21.15.6:0/1368480259 >> [v2:172.21.15.6:6836/2590421094,v1:172.21.15.6:6837/2590421094] conn(0x55ebe793bab0 0x55ebe793dea0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:37.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.322+0000 7fcb757ab740 1 --2- 172.21.15.6:0/1368480259 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x55ebe791a110 0x55ebe7932520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:37.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.322+0000 7fcb757ab740 1 --2- 172.21.15.6:0/1368480259 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ebe7932a60 0x55ebe7904ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:37.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.322+0000 7fcb58c00640 0 client.4782 ms_handle_reset on v2:172.21.15.6:6836/2590421094 2024-04-24T23:16:37.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.322+0000 7fcb757ab740 1 --2- 172.21.15.6:0/1368480259 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55ebe7907060 0x55ebe7930540 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:37.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.322+0000 7fcb757ab740 1 -- 172.21.15.6:0/1368480259 >> 172.21.15.6:0/1368480259 conn(0x55ebe78dc040 msgr2=0x55ebe78d7680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:37.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.322+0000 7fcb757ab740 1 -- 172.21.15.6:0/1368480259 shutdown_connections 2024-04-24T23:16:37.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.322+0000 7fcb757ab740 1 -- 172.21.15.6:0/1368480259 wait complete. 2024-04-24T23:16:37.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.323+0000 7fcb757ab740 20 client.4782 trim_cache size 0 max 16384 2024-04-24T23:16:37.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.358+0000 7fcb757ab740 1 Processor -- start 2024-04-24T23:16:37.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.359+0000 7fcb757ab740 1 -- start start 2024-04-24T23:16:37.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.360+0000 7fcb757ab740 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x55ebe7907060 0x55ebe78dfb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:37.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.360+0000 7fcb757ab740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55ebe791a110 0x55ebe7930150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:37.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.360+0000 7fcb757ab740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ebe7932a60 0x55ebe7930690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:37.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.360+0000 7fcb757ab740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fcb6c0cabe0 con 0x55ebe791a110 2024-04-24T23:16:37.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.360+0000 7fcb757ab740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55ebe783cdb0 con 0x55ebe7932a60 2024-04-24T23:16:37.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.360+0000 7fcb757ab740 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x55ebe77fcd20 con 0x55ebe7907060 2024-04-24T23:16:37.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.361+0000 7fcb63e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55ebe791a110 0x55ebe7930150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:37.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.361+0000 7fcb63e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55ebe791a110 0x55ebe7930150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:35506/0 (socket says 172.21.15.6:35506) 2024-04-24T23:16:37.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.361+0000 7fcb63e00640 1 -- 172.21.15.6:0/946531367 learned_addr learned my addr 172.21.15.6:0/946531367 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:16:37.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.361+0000 7fcb6be00640 1 --2- 172.21.15.6:0/946531367 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x55ebe7907060 0x55ebe78dfb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:37.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.361+0000 7fcb70a00640 1 --2- 172.21.15.6:0/946531367 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ebe7932a60 0x55ebe7930690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:37.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.361+0000 7fcb63e00640 1 -- 172.21.15.6:0/946531367 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x55ebe7907060 msgr2=0x55ebe78dfb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:16:37.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.361+0000 7fcb63e00640 1 --2- 172.21.15.6:0/946531367 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x55ebe7907060 0x55ebe78dfb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:37.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.361+0000 7fcb63e00640 1 -- 172.21.15.6:0/946531367 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ebe7932a60 msgr2=0x55ebe7930690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:16:37.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.361+0000 7fcb63e00640 1 --2- 172.21.15.6:0/946531367 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ebe7932a60 0x55ebe7930690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:37.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.361+0000 7fcb63e00640 1 -- 172.21.15.6:0/946531367 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ebe78d90a0 con 0x55ebe791a110 2024-04-24T23:16:37.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.361+0000 7fcb63e00640 1 --2- 172.21.15.6:0/946531367 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55ebe791a110 0x55ebe7930150 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fcb6c06db10 tx=0x7fcb54013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67515870fdc715e3 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:37.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.362+0000 7fcb5b400640 1 -- 172.21.15.6:0/946531367 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb54024110 con 0x55ebe791a110 2024-04-24T23:16:37.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.362+0000 7fcb5b400640 1 -- 172.21.15.6:0/946531367 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb5402c020 con 0x55ebe791a110 2024-04-24T23:16:37.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.362+0000 7fcb5b400640 1 -- 172.21.15.6:0/946531367 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb540223f0 con 0x55ebe791a110 2024-04-24T23:16:37.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.362+0000 7fcb757ab740 1 -- 172.21.15.6:0/946531367 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55ebe791a110 msgr2=0x55ebe7930150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:37.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.362+0000 7fcb757ab740 1 --2- 172.21.15.6:0/946531367 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55ebe791a110 0x55ebe7930150 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fcb6c06db10 tx=0x7fcb54013000 comp rx=0 tx=0).stop 2024-04-24T23:16:37.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.363+0000 7fcb757ab740 1 -- 172.21.15.6:0/946531367 shutdown_connections 2024-04-24T23:16:37.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.363+0000 7fcb757ab740 1 --2- 172.21.15.6:0/946531367 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ebe7932a60 0x55ebe7930690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:37.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.363+0000 7fcb757ab740 1 --2- 172.21.15.6:0/946531367 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55ebe791a110 0x55ebe7930150 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:37.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.363+0000 7fcb757ab740 1 --2- 172.21.15.6:0/946531367 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x55ebe7907060 0x55ebe78dfb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:37.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.363+0000 7fcb757ab740 1 -- 172.21.15.6:0/946531367 >> 172.21.15.6:0/946531367 conn(0x55ebe78dc040 msgr2=0x55ebe78dc680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:37.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.363+0000 7fcb757ab740 1 -- 172.21.15.6:0/946531367 shutdown_connections 2024-04-24T23:16:37.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.363+0000 7fcb757ab740 1 -- 172.21.15.6:0/946531367 wait complete. 2024-04-24T23:16:37.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.363+0000 7fcb757ab740 1 -- start start 2024-04-24T23:16:37.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.364+0000 7fcb757ab740 1 Processor -- start 2024-04-24T23:16:37.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.365+0000 7fcb757ab740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ebe7907060 0x55ebe7930550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:37.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.365+0000 7fcb757ab740 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x55ebe791a110 0x55ebe7992e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:37.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.365+0000 7fcb757ab740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55ebe7932a60 0x55ebe7993370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:37.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.365+0000 7fcb757ab740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x55ebe78deb90 con 0x55ebe7932a60 2024-04-24T23:16:37.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.365+0000 7fcb757ab740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55ebe791b1f0 con 0x55ebe7907060 2024-04-24T23:16:37.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.365+0000 7fcb757ab740 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x55ebe792f220 con 0x55ebe791a110 2024-04-24T23:16:37.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.365+0000 7fcb757ab740 10 client.0 _ll_register_callbacks cb 0x7fcb54007340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T23:16:37.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.365+0000 7fcb70a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55ebe7932a60 0x55ebe7993370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:37.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.365+0000 7fcb70a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55ebe7932a60 0x55ebe7993370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:35510/0 (socket says 172.21.15.6:35510) 2024-04-24T23:16:37.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.365+0000 7fcb70a00640 1 -- 172.21.15.6:0/132741885 learned_addr learned my addr 172.21.15.6:0/132741885 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:16:37.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.365+0000 7fcb6be00640 1 --2- 172.21.15.6:0/132741885 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ebe7907060 0x55ebe7930550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:37.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.365+0000 7fcb63e00640 1 --2- 172.21.15.6:0/132741885 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x55ebe791a110 0x55ebe7992e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:37.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.365+0000 7fcb70a00640 1 -- 172.21.15.6:0/132741885 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x55ebe791a110 msgr2=0x55ebe7992e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:16:37.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.365+0000 7fcb70a00640 1 --2- 172.21.15.6:0/132741885 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x55ebe791a110 0x55ebe7992e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:37.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.365+0000 7fcb70a00640 1 -- 172.21.15.6:0/132741885 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ebe7907060 msgr2=0x55ebe7930550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:16:37.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.365+0000 7fcb70a00640 1 --2- 172.21.15.6:0/132741885 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ebe7907060 0x55ebe7930550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:37.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.365+0000 7fcb70a00640 1 -- 172.21.15.6:0/132741885 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ebe7922120 con 0x55ebe7932a60 2024-04-24T23:16:37.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.366+0000 7fcb70a00640 1 --2- 172.21.15.6:0/132741885 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55ebe7932a60 0x55ebe7993370 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fcb64016a10 tx=0x7fcb64016e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f82c28665586689 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:37.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.366+0000 7fcb58c00640 10 client.0 ms_handle_connect on v2:172.21.15.6:3300/0 2024-04-24T23:16:37.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.366+0000 7fcb58c00640 1 -- 172.21.15.6:0/132741885 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb64006d40 con 0x55ebe7932a60 2024-04-24T23:16:37.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.366+0000 7fcb58c00640 1 -- 172.21.15.6:0/132741885 <== mon.0 v2:172.21.15.6:3300/0 2 ==== osd_map(26..26 src has 1..26) ==== 6590+0+0 (secure 0 0 0) 0x7fcb64003a50 con 0x55ebe7932a60 2024-04-24T23:16:37.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.366+0000 7fcb757ab740 1 -- 172.21.15.6:0/132741885 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcb5402c020 con 0x55ebe7932a60 2024-04-24T23:16:37.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.366+0000 7fcb58c00640 1 -- 172.21.15.6:0/132741885 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_get_version_reply(handle=1 version=18) ==== 24+0+0 (secure 0 0 0) 0x7fcb64024050 con 0x55ebe7932a60 2024-04-24T23:16:37.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.366+0000 7fcb757ab740 10 client.4788 fetch_fsmap learned FSMap version 18 2024-04-24T23:16:37.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.366+0000 7fcb757ab740 1 -- 172.21.15.6:0/132741885 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,fsmap=18,monmap=2+}) -- 0x55ebe7931900 con 0x55ebe7932a60 2024-04-24T23:16:37.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.366+0000 7fcb58c00640 1 -- 172.21.15.6:0/132741885 <== mon.0 v2:172.21.15.6:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb5402c020 con 0x55ebe7932a60 2024-04-24T23:16:37.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.367+0000 7fcb58c00640 1 -- 172.21.15.6:0/132741885 <== mon.0 v2:172.21.15.6:3300/0 5 ==== fsmap(e 18) ==== 4792+0+0 (secure 0 0 0) 0x7fcb64027020 con 0x55ebe7932a60 2024-04-24T23:16:37.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.367+0000 7fcb757ab740 10 client.4788 fetch_fsmap finished waiting for FSMap version 18 2024-04-24T23:16:37.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.367+0000 7fcb757ab740 10 client.4788 resolve_mds: resolved name 'e' to daemon mds.e 2024-04-24T23:16:37.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.367+0000 7fcb757ab740 20 client.4788 populate_metadata read hostname 'smithi006' 2024-04-24T23:16:37.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.367+0000 7fcb757ab740 1 --2- 172.21.15.6:0/132741885 >> [v2:172.21.15.6:6836/2590421094,v1:172.21.15.6:6837/2590421094] conn(0x55ebe79a3860 0x55ebe79a5c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:37.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.367+0000 7fcb757ab740 4 client.4788 mds_command: new command op to 4203 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-24T23:16:37.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.367+0000 7fcb757ab740 1 -- 172.21.15.6:0/132741885 --> [v2:172.21.15.6:6836/2590421094,v1:172.21.15.6:6837/2590421094] -- command(tid 0: {"prefix": "status"}) -- 0x55ebe7848680 con 0x55ebe79a3860 2024-04-24T23:16:37.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.367+0000 7fcb6be00640 1 --2- 172.21.15.6:0/132741885 >> [v2:172.21.15.6:6836/2590421094,v1:172.21.15.6:6837/2590421094] conn(0x55ebe79a3860 0x55ebe79a5c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:37.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.367+0000 7fcb58c00640 10 client.4788 ms_handle_connect on v2:172.21.15.6:6836/2590421094 2024-04-24T23:16:37.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.367+0000 7fcb6be00640 1 --2- 172.21.15.6:0/132741885 >> [v2:172.21.15.6:6836/2590421094,v1:172.21.15.6:6837/2590421094] conn(0x55ebe79a3860 0x55ebe79a5c50 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=fccec14060abb399 server_cookie=733d054209560987 in_seq=0 out_seq=0 2024-04-24T23:16:37.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.368+0000 7fcb58c00640 1 -- 172.21.15.6:0/132741885 <== mds.4 v2:172.21.15.6:6836/2590421094 1 ==== command_reply(tid 0: 0 ) ==== 8+0+331 (crc 0 0 0) 0x55ebe7848680 con 0x55ebe79a3860 2024-04-24T23:16:37.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.368+0000 7fcb58c00640 10 client.4788 handle_command_reply: tid=0 2024-04-24T23:16:37.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.368+0000 7fcb757ab740 1 client.4788 shutdown 2024-04-24T23:16:37.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.368+0000 7fcb757ab740 1 -- 172.21.15.6:0/132741885 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55ebe7932a60 msgr2=0x55ebe7993370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:37.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.368+0000 7fcb757ab740 1 --2- 172.21.15.6:0/132741885 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55ebe7932a60 0x55ebe7993370 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fcb64016a10 tx=0x7fcb64016e10 comp rx=0 tx=0).stop 2024-04-24T23:16:37.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.368+0000 7fcb757ab740 1 -- 172.21.15.6:0/132741885 shutdown_connections 2024-04-24T23:16:37.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.368+0000 7fcb757ab740 1 --2- 172.21.15.6:0/132741885 >> [v2:172.21.15.6:6836/2590421094,v1:172.21.15.6:6837/2590421094] conn(0x55ebe79a3860 0x55ebe79a5c50 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:37.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.368+0000 7fcb757ab740 1 --2- 172.21.15.6:0/132741885 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x55ebe791a110 0x55ebe7992e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:37.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.368+0000 7fcb757ab740 1 --2- 172.21.15.6:0/132741885 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55ebe7932a60 0x55ebe7993370 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:37.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.368+0000 7fcb757ab740 1 --2- 172.21.15.6:0/132741885 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ebe7907060 0x55ebe7930550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:37.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.368+0000 7fcb58c00640 0 client.4788 ms_handle_reset on v2:172.21.15.6:6836/2590421094 2024-04-24T23:16:37.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.368+0000 7fcb757ab740 1 -- 172.21.15.6:0/132741885 >> 172.21.15.6:0/132741885 conn(0x55ebe78dc040 msgr2=0x55ebe78de240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:37.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.368+0000 7fcb757ab740 1 -- 172.21.15.6:0/132741885 shutdown_connections 2024-04-24T23:16:37.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.368+0000 7fcb757ab740 1 -- 172.21.15.6:0/132741885 wait complete. 2024-04-24T23:16:37.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.368+0000 7fcb757ab740 20 client.4788 trim_cache size 0 max 16384 2024-04-24T23:16:37.373 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-04-24T23:16:37.373 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_fsid": "18bef68d-5e6d-4ec5-862b-b6f206e2588d", 2024-04-24T23:16:37.373 INFO:teuthology.orchestra.run.smithi006.stdout: "whoami": 4, 2024-04-24T23:16:37.374 INFO:teuthology.orchestra.run.smithi006.stdout: "id": 4203, 2024-04-24T23:16:37.374 INFO:teuthology.orchestra.run.smithi006.stdout: "want_state": "up:active", 2024-04-24T23:16:37.374 INFO:teuthology.orchestra.run.smithi006.stdout: "state": "up:active", 2024-04-24T23:16:37.374 INFO:teuthology.orchestra.run.smithi006.stdout: "fs_name": "cephfs", 2024-04-24T23:16:37.374 INFO:teuthology.orchestra.run.smithi006.stdout: "rank_uptime": 11.082942967999999, 2024-04-24T23:16:37.374 INFO:teuthology.orchestra.run.smithi006.stdout: "mdsmap_epoch": 18, 2024-04-24T23:16:37.374 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap_epoch": 26, 2024-04-24T23:16:37.374 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap_epoch_barrier": 26, 2024-04-24T23:16:37.374 INFO:teuthology.orchestra.run.smithi006.stdout: "uptime": 66.425967722999999 2024-04-24T23:16:37.374 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-04-24T23:16:37.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.369+0000 7fcb72a00640 1 -- 172.21.15.6:0/3574306764 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fcb4806be60 msgr2=0x7fcb4806e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:37.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.369+0000 7fcb72a00640 1 --2- 172.21.15.6:0/3574306764 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fcb4806be60 0x7fcb4806e320 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fcb54007f70 tx=0x7fcb54007f00 comp rx=0 tx=0).stop 2024-04-24T23:16:37.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.369+0000 7fcb72a00640 1 -- 172.21.15.6:0/3574306764 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcb6c06d6d0 msgr2=0x7fcb6c0cb9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:37.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.369+0000 7fcb72a00640 1 --2- 172.21.15.6:0/3574306764 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcb6c06d6d0 0x7fcb6c0cb9c0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fcb5c009720 tx=0x7fcb5c0046c0 comp rx=0 tx=0).stop 2024-04-24T23:16:37.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.369+0000 7fcb72a00640 1 -- 172.21.15.6:0/3574306764 shutdown_connections 2024-04-24T23:16:37.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.369+0000 7fcb72a00640 1 --2- 172.21.15.6:0/3574306764 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fcb4806be60 0x7fcb4806e320 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:37.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.369+0000 7fcb72a00640 1 --2- 172.21.15.6:0/3574306764 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcb6c152d40 0x7fcb6c141f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:37.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.369+0000 7fcb72a00640 1 --2- 172.21.15.6:0/3574306764 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcb6c0c4770 0x7fcb6c071210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:37.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.369+0000 7fcb72a00640 1 --2- 172.21.15.6:0/3574306764 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcb6c06d6d0 0x7fcb6c0cb9c0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:37.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.369+0000 7fcb72a00640 1 -- 172.21.15.6:0/3574306764 >> 172.21.15.6:0/3574306764 conn(0x7fcb6c0c8c50 msgr2=0x7fcb6c0c31b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:37.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.369+0000 7fcb72a00640 1 -- 172.21.15.6:0/3574306764 shutdown_connections 2024-04-24T23:16:37.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.369+0000 7fcb72a00640 1 -- 172.21.15.6:0/3574306764 wait complete. 2024-04-24T23:16:37.387 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-24T23:16:37.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.568+0000 7fa361a00640 1 Processor -- start 2024-04-24T23:16:37.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.568+0000 7fa361a00640 1 -- start start 2024-04-24T23:16:37.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.570+0000 7fa361a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa35c070f30 0x7fa35c06cb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:37.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.570+0000 7fa361a00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa35c06d060 0x7fa35c15aed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:37.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.570+0000 7fa361a00640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa35c0727f0 con 0x7fa35c154c80 2024-04-24T23:16:37.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.570+0000 7fa361a00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7fa35c073490 con 0x7fa35c070f30 2024-04-24T23:16:37.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.570+0000 7fa361a00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fa35c071be0 con 0x7fa35c06d060 2024-04-24T23:16:37.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.570+0000 7fa35be00640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7fa35c154c80 0x7fa35c0c8120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.6:57846/0 (socket says 172.21.15.6:57846) 2024-04-24T23:16:37.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.570+0000 7fa35be00640 1 -- 172.21.15.6:0/1604287662 learned_addr learned my addr 172.21.15.6:0/1604287662 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:16:37.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.571+0000 7fa361000640 1 --2- 172.21.15.6:0/1604287662 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa35c06d060 0x7fa35c15aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:37.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.571+0000 7fa35b400640 1 --2- 172.21.15.6:0/1604287662 >> v2:172.21.15.6:3300/0 conn(0x7fa35c070f30 0x7fa35c06cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:37.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.571+0000 7fa35aa00640 1 -- 172.21.15.6:0/1604287662 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1420010595 0 0) 0x7fa35c0727f0 con 0x7fa35c154c80 2024-04-24T23:16:37.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.571+0000 7fa35aa00640 1 -- 172.21.15.6:0/1604287662 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa340003520 con 0x7fa35c154c80 2024-04-24T23:16:37.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.571+0000 7fa361000640 1 -- 172.21.15.6:0/1604287662 >> v2:172.21.15.6:3300/0 conn(0x7fa35c070f30 msgr2=0x7fa35c06cb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:16:37.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.571+0000 7fa361000640 1 --2- 172.21.15.6:0/1604287662 >> v2:172.21.15.6:3300/0 conn(0x7fa35c070f30 0x7fa35c06cb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:37.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.571+0000 7fa361000640 1 -- 172.21.15.6:0/1604287662 >> v1:172.21.15.190:6789/0 conn(0x7fa35c154c80 legacy=0x7fa35c0c8120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:37.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.571+0000 7fa361000640 1 -- 172.21.15.6:0/1604287662 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa35c071ec0 con 0x7fa35c06d060 2024-04-24T23:16:37.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.572+0000 7fa361000640 1 --2- 172.21.15.6:0/1604287662 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa35c06d060 0x7fa35c15aed0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fa354009d90 tx=0x7fa35402a6f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fbec30ca2f418e95 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:37.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.573+0000 7fa35aa00640 1 -- 172.21.15.6:0/1604287662 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa354007740 con 0x7fa35c06d060 2024-04-24T23:16:37.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.573+0000 7fa35aa00640 1 -- 172.21.15.6:0/1604287662 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa3540078e0 con 0x7fa35c06d060 2024-04-24T23:16:37.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.573+0000 7fa35aa00640 1 -- 172.21.15.6:0/1604287662 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa354007c90 con 0x7fa35c06d060 2024-04-24T23:16:37.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.573+0000 7fa361a00640 1 -- 172.21.15.6:0/1604287662 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa35c06d060 msgr2=0x7fa35c15aed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:37.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.573+0000 7fa361a00640 1 --2- 172.21.15.6:0/1604287662 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa35c06d060 0x7fa35c15aed0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fa354009d90 tx=0x7fa35402a6f0 comp rx=0 tx=0).stop 2024-04-24T23:16:37.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.574+0000 7fa361a00640 1 -- 172.21.15.6:0/1604287662 shutdown_connections 2024-04-24T23:16:37.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.574+0000 7fa361a00640 1 --2- 172.21.15.6:0/1604287662 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa35c06d060 0x7fa35c15aed0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:37.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.574+0000 7fa361a00640 1 --2- 172.21.15.6:0/1604287662 >> v2:172.21.15.6:3300/0 conn(0x7fa35c070f30 0x7fa35c06cb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:37.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.574+0000 7fa361a00640 1 -- 172.21.15.6:0/1604287662 >> 172.21.15.6:0/1604287662 conn(0x7fa35c0cc1e0 msgr2=0x7fa35c0cc5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:37.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.574+0000 7fa361a00640 1 -- 172.21.15.6:0/1604287662 shutdown_connections 2024-04-24T23:16:37.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.574+0000 7fa361a00640 1 -- 172.21.15.6:0/1604287662 wait complete. 2024-04-24T23:16:37.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.576+0000 7fa361a00640 1 Processor -- start 2024-04-24T23:16:37.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.576+0000 7fa361a00640 1 -- start start 2024-04-24T23:16:37.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.577+0000 7fa361a00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa35c06d060 0x7fa35c15c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:37.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.577+0000 7fa361a00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa35c070f30 0x7fa35c0c5f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:37.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.577+0000 7fa361a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa35c154c80 0x7fa35c0c51b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:37.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.577+0000 7fa361a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fa35c0718d0 con 0x7fa35c154c80 2024-04-24T23:16:37.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.577+0000 7fa361a00640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fa35c072480 con 0x7fa35c070f30 2024-04-24T23:16:37.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.577+0000 7fa361a00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fa35c0c8830 con 0x7fa35c06d060 2024-04-24T23:16:37.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.577+0000 7fa35b400640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa35c070f30 0x7fa35c0c5f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:37.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.577+0000 7fa35be00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa35c06d060 0x7fa35c15c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:37.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.577+0000 7fa361000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa35c154c80 0x7fa35c0c51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:37.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.578+0000 7fa35b400640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa35c070f30 0x7fa35c0c5f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:35252/0 (socket says 172.21.15.6:35252) 2024-04-24T23:16:37.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.578+0000 7fa35b400640 1 -- 172.21.15.6:0/3065249483 learned_addr learned my addr 172.21.15.6:0/3065249483 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:16:37.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.578+0000 7fa35b400640 1 -- 172.21.15.6:0/3065249483 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa35c06d060 msgr2=0x7fa35c15c950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:37.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.578+0000 7fa35b400640 1 --2- 172.21.15.6:0/3065249483 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa35c06d060 0x7fa35c15c950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:37.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.578+0000 7fa35b400640 1 -- 172.21.15.6:0/3065249483 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa35c154c80 msgr2=0x7fa35c0c51b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:37.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.578+0000 7fa35b400640 1 --2- 172.21.15.6:0/3065249483 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa35c154c80 0x7fa35c0c51b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:37.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.578+0000 7fa35b400640 1 -- 172.21.15.6:0/3065249483 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa35c0c3e70 con 0x7fa35c070f30 2024-04-24T23:16:37.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.578+0000 7fa35b400640 1 --2- 172.21.15.6:0/3065249483 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa35c070f30 0x7fa35c0c5f10 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fa34800e870 tx=0x7fa34800ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:37.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.578+0000 7fa353e00640 1 -- 172.21.15.6:0/3065249483 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa348017070 con 0x7fa35c070f30 2024-04-24T23:16:37.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.578+0000 7fa353e00640 1 -- 172.21.15.6:0/3065249483 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa34800c8e0 con 0x7fa35c070f30 2024-04-24T23:16:37.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.578+0000 7fa353e00640 1 -- 172.21.15.6:0/3065249483 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa34800ccd0 con 0x7fa35c070f30 2024-04-24T23:16:37.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.579+0000 7fa361a00640 1 -- 172.21.15.6:0/3065249483 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa35c0c4110 con 0x7fa35c070f30 2024-04-24T23:16:37.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.579+0000 7fa361a00640 1 -- 172.21.15.6:0/3065249483 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa35c0c46d0 con 0x7fa35c070f30 2024-04-24T23:16:37.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.581+0000 7fa353e00640 1 -- 172.21.15.6:0/3065249483 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fa34801a070 con 0x7fa35c070f30 2024-04-24T23:16:37.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.582+0000 7fa353e00640 1 --2- 172.21.15.6:0/3065249483 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fa33c06be30 0x7fa33c06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:37.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.583+0000 7fa353e00640 1 -- 172.21.15.6:0/3065249483 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6590+0+0 (secure 0 0 0) 0x7fa34808b390 con 0x7fa35c070f30 2024-04-24T23:16:37.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.583+0000 7fa35be00640 1 --2- 172.21.15.6:0/3065249483 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fa33c06be30 0x7fa33c06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:37.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.583+0000 7fa365fcc740 1 Processor -- start 2024-04-24T23:16:37.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.583+0000 7fa365fcc740 1 -- start start 2024-04-24T23:16:37.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.584+0000 7fa35be00640 1 --2- 172.21.15.6:0/3065249483 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fa33c06be30 0x7fa33c06e2f0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fa34c0011e0 tx=0x7fa34c007d30 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:37.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.584+0000 7fa365fcc740 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x560d922a3270 0x560d922beb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:37.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.584+0000 7fa365fcc740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560d92275e00 0x560d92277d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:37.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.584+0000 7fa365fcc740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560d92278250 0x560d922a3650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:37.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.584+0000 7fa365fcc740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x560d92179ed0 con 0x560d92275e00 2024-04-24T23:16:37.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.584+0000 7fa365fcc740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x560d921dcdf0 con 0x560d92278250 2024-04-24T23:16:37.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.584+0000 7fa365fcc740 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x560d921858d0 con 0x560d922a3270 2024-04-24T23:16:37.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.584+0000 7fa35b400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560d92275e00 0x560d92277d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:37.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.584+0000 7fa361000640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560d92278250 0x560d922a3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:37.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.584+0000 7fa35be00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x560d922a3270 0x560d922beb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:37.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.584+0000 7fa35b400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560d92275e00 0x560d92277d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:35528/0 (socket says 172.21.15.6:35528) 2024-04-24T23:16:37.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.584+0000 7fa35b400640 1 -- 172.21.15.6:0/4020756201 learned_addr learned my addr 172.21.15.6:0/4020756201 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:16:37.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.585+0000 7fa361000640 1 -- 172.21.15.6:0/4020756201 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x560d922a3270 msgr2=0x560d922beb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:16:37.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.585+0000 7fa361000640 1 --2- 172.21.15.6:0/4020756201 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x560d922a3270 0x560d922beb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:37.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.585+0000 7fa361000640 1 -- 172.21.15.6:0/4020756201 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560d92275e00 msgr2=0x560d92277d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:16:37.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.585+0000 7fa361000640 1 --2- 172.21.15.6:0/4020756201 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560d92275e00 0x560d92277d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:37.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.585+0000 7fa361000640 1 -- 172.21.15.6:0/4020756201 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560d91ef12d0 con 0x560d92278250 2024-04-24T23:16:37.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.585+0000 7fa35be00640 1 --2- 172.21.15.6:0/4020756201 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x560d922a3270 0x560d922beb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:16:37.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.585+0000 7fa361000640 1 --2- 172.21.15.6:0/4020756201 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560d92278250 0x560d922a3650 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fa354031000 tx=0x7fa35402e000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=73b55f18c2190729 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:37.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.586+0000 7fa350c00640 1 -- 172.21.15.6:0/4020756201 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa354003cd0 con 0x560d92278250 2024-04-24T23:16:37.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.586+0000 7fa350c00640 1 -- 172.21.15.6:0/4020756201 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa354004040 con 0x560d92278250 2024-04-24T23:16:37.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.586+0000 7fa350c00640 1 -- 172.21.15.6:0/4020756201 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa354004430 con 0x560d92278250 2024-04-24T23:16:37.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.586+0000 7fa365fcc740 1 -- 172.21.15.6:0/4020756201 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560d92278250 msgr2=0x560d922a3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:37.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.586+0000 7fa365fcc740 1 --2- 172.21.15.6:0/4020756201 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560d92278250 0x560d922a3650 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fa354031000 tx=0x7fa35402e000 comp rx=0 tx=0).stop 2024-04-24T23:16:37.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.586+0000 7fa365fcc740 1 -- 172.21.15.6:0/4020756201 shutdown_connections 2024-04-24T23:16:37.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.586+0000 7fa365fcc740 1 --2- 172.21.15.6:0/4020756201 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560d92278250 0x560d922a3650 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:37.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.586+0000 7fa365fcc740 1 --2- 172.21.15.6:0/4020756201 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560d92275e00 0x560d92277d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:37.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.586+0000 7fa365fcc740 1 --2- 172.21.15.6:0/4020756201 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x560d922a3270 0x560d922beb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:37.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.586+0000 7fa365fcc740 1 -- 172.21.15.6:0/4020756201 >> 172.21.15.6:0/4020756201 conn(0x560d92276c90 msgr2=0x560d922be160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:37.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.586+0000 7fa365fcc740 1 -- 172.21.15.6:0/4020756201 shutdown_connections 2024-04-24T23:16:37.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.587+0000 7fa365fcc740 1 -- 172.21.15.6:0/4020756201 wait complete. 2024-04-24T23:16:37.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.587+0000 7fa365fcc740 1 -- start start 2024-04-24T23:16:37.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.587+0000 7fa365fcc740 1 Processor -- start 2024-04-24T23:16:37.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.588+0000 7fa365fcc740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560d922a3270 0x560d922b5d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:37.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.588+0000 7fa365fcc740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560d922a7010 0x560d922ce670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:37.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.588+0000 7fa365fcc740 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x560d922cebb0 0x560d922a0d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:37.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.588+0000 7fa365fcc740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x560d92198d20 con 0x560d922a7010 2024-04-24T23:16:37.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.588+0000 7fa365fcc740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x560d921d8db0 con 0x560d922a3270 2024-04-24T23:16:37.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.588+0000 7fa365fcc740 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x560d92276290 con 0x560d922cebb0 2024-04-24T23:16:37.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.588+0000 7fa365fcc740 10 client.0 _ll_register_callbacks cb 0x560d921d3f10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T23:16:37.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.588+0000 7fa35b400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560d922a7010 0x560d922ce670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:37.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.589+0000 7fa35b400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560d922a7010 0x560d922ce670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:35542/0 (socket says 172.21.15.6:35542) 2024-04-24T23:16:37.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.589+0000 7fa35b400640 1 -- 172.21.15.6:0/4115259154 learned_addr learned my addr 172.21.15.6:0/4115259154 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:16:37.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.589+0000 7fa361000640 1 --2- 172.21.15.6:0/4115259154 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x560d922cebb0 0x560d922a0d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:37.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.589+0000 7fa35be00640 1 --2- 172.21.15.6:0/4115259154 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560d922a3270 0x560d922b5d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:37.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.589+0000 7fa35b400640 1 -- 172.21.15.6:0/4115259154 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x560d922cebb0 msgr2=0x560d922a0d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:16:37.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.589+0000 7fa35b400640 1 --2- 172.21.15.6:0/4115259154 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x560d922cebb0 0x560d922a0d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:37.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.589+0000 7fa35b400640 1 -- 172.21.15.6:0/4115259154 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560d922a3270 msgr2=0x560d922b5d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:16:37.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.589+0000 7fa35b400640 1 --2- 172.21.15.6:0/4115259154 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560d922a3270 0x560d922b5d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:37.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.589+0000 7fa35b400640 1 -- 172.21.15.6:0/4115259154 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560d91fbb6b0 con 0x560d922a7010 2024-04-24T23:16:37.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.589+0000 7fa35b400640 1 --2- 172.21.15.6:0/4115259154 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560d922a7010 0x560d922ce670 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fa348055740 tx=0x7fa348058c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff44eb9c8e814a5d server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:37.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.589+0000 7fa345600640 10 client.0 ms_handle_connect on v2:172.21.15.6:3300/0 2024-04-24T23:16:37.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.589+0000 7fa345600640 1 -- 172.21.15.6:0/4115259154 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa348071020 con 0x560d922a7010 2024-04-24T23:16:37.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.590+0000 7fa345600640 1 -- 172.21.15.6:0/4115259154 <== mon.0 v2:172.21.15.6:3300/0 2 ==== osd_map(26..26 src has 1..26) ==== 6590+0+0 (secure 0 0 0) 0x7fa34808b0b0 con 0x560d922a7010 2024-04-24T23:16:37.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.590+0000 7fa365fcc740 1 -- 172.21.15.6:0/4115259154 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa354004040 con 0x560d922a7010 2024-04-24T23:16:37.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.590+0000 7fa345600640 1 -- 172.21.15.6:0/4115259154 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_get_version_reply(handle=1 version=18) ==== 24+0+0 (secure 0 0 0) 0x7fa348021070 con 0x560d922a7010 2024-04-24T23:16:37.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.590+0000 7fa365fcc740 10 client.4800 fetch_fsmap learned FSMap version 18 2024-04-24T23:16:37.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.590+0000 7fa365fcc740 1 -- 172.21.15.6:0/4115259154 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,fsmap=18,monmap=2+}) -- 0x560d922b52d0 con 0x560d922a7010 2024-04-24T23:16:37.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.590+0000 7fa345600640 1 -- 172.21.15.6:0/4115259154 <== mon.0 v2:172.21.15.6:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa354004040 con 0x560d922a7010 2024-04-24T23:16:37.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.590+0000 7fa345600640 1 -- 172.21.15.6:0/4115259154 <== mon.0 v2:172.21.15.6:3300/0 5 ==== fsmap(e 18) ==== 4792+0+0 (secure 0 0 0) 0x7fa34805d170 con 0x560d922a7010 2024-04-24T23:16:37.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.590+0000 7fa365fcc740 10 client.4800 fetch_fsmap finished waiting for FSMap version 18 2024-04-24T23:16:37.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.590+0000 7fa365fcc740 10 client.4800 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-24T23:16:37.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.590+0000 7fa365fcc740 20 client.4800 populate_metadata read hostname 'smithi006' 2024-04-24T23:16:37.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.590+0000 7fa365fcc740 1 --2- 172.21.15.6:0/4115259154 >> [v2:172.21.15.190:6835/2032280158,v1:172.21.15.190:6838/2032280158] conn(0x560d922caed0 0x560d922da2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:37.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.590+0000 7fa365fcc740 4 client.4800 mds_command: new command op to 4232 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T23:16:37.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.590+0000 7fa365fcc740 1 -- 172.21.15.6:0/4115259154 --> [v2:172.21.15.190:6835/2032280158,v1:172.21.15.190:6838/2032280158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560d921ea330 con 0x560d922caed0 2024-04-24T23:16:37.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.591+0000 7fa35be00640 1 --2- 172.21.15.6:0/4115259154 >> [v2:172.21.15.190:6835/2032280158,v1:172.21.15.190:6838/2032280158] conn(0x560d922caed0 0x560d922da2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:37.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.591+0000 7fa35be00640 1 --2- 172.21.15.6:0/4115259154 >> [v2:172.21.15.190:6835/2032280158,v1:172.21.15.190:6838/2032280158] conn(0x560d922caed0 0x560d922da2f0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=15ecdb8e0ad17930 server_cookie=ee71ce222964c28c in_seq=0 out_seq=0 2024-04-24T23:16:37.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.591+0000 7fa345600640 10 client.4800 ms_handle_connect on v2:172.21.15.190:6835/2032280158 2024-04-24T23:16:37.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.592+0000 7fa345600640 1 -- 172.21.15.6:0/4115259154 <== mds.3 v2:172.21.15.190:6835/2032280158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560d921ea330 con 0x560d922caed0 2024-04-24T23:16:37.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.592+0000 7fa345600640 10 client.4800 handle_command_reply: tid=0 2024-04-24T23:16:37.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.592+0000 7fa365fcc740 1 client.4800 shutdown 2024-04-24T23:16:37.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.592+0000 7fa365fcc740 1 -- 172.21.15.6:0/4115259154 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560d922a7010 msgr2=0x560d922ce670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:37.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.592+0000 7fa365fcc740 1 --2- 172.21.15.6:0/4115259154 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560d922a7010 0x560d922ce670 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fa348055740 tx=0x7fa348058c40 comp rx=0 tx=0).stop 2024-04-24T23:16:37.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.592+0000 7fa365fcc740 1 -- 172.21.15.6:0/4115259154 shutdown_connections 2024-04-24T23:16:37.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.592+0000 7fa365fcc740 1 --2- 172.21.15.6:0/4115259154 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x560d922cebb0 0x560d922a0d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:37.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.592+0000 7fa365fcc740 1 --2- 172.21.15.6:0/4115259154 >> [v2:172.21.15.190:6835/2032280158,v1:172.21.15.190:6838/2032280158] conn(0x560d922caed0 0x560d922da2f0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:37.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.592+0000 7fa365fcc740 1 --2- 172.21.15.6:0/4115259154 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560d922a7010 0x560d922ce670 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:37.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.592+0000 7fa365fcc740 1 --2- 172.21.15.6:0/4115259154 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560d922a3270 0x560d922b5d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:37.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.592+0000 7fa345600640 0 client.4800 ms_handle_reset on v2:172.21.15.190:6835/2032280158 2024-04-24T23:16:37.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.592+0000 7fa365fcc740 1 -- 172.21.15.6:0/4115259154 >> 172.21.15.6:0/4115259154 conn(0x560d92278250 msgr2=0x560d9227a050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:37.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.592+0000 7fa365fcc740 1 -- 172.21.15.6:0/4115259154 shutdown_connections 2024-04-24T23:16:37.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.593+0000 7fa365fcc740 1 -- 172.21.15.6:0/4115259154 wait complete. 2024-04-24T23:16:37.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.593+0000 7fa365fcc740 20 client.4800 trim_cache size 0 max 16384 2024-04-24T23:16:37.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.612+0000 7fa365fcc740 1 Processor -- start 2024-04-24T23:16:37.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.612+0000 7fa365fcc740 1 -- start start 2024-04-24T23:16:37.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.613+0000 7fa365fcc740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560d922a3270 0x560d92273a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:37.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.613+0000 7fa365fcc740 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x560d922a7010 0x560d922df320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:37.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.613+0000 7fa365fcc740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560d922cebb0 0x560d922df860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:37.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.613+0000 7fa365fcc740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fa35c1f4760 con 0x560d922cebb0 2024-04-24T23:16:37.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.613+0000 7fa365fcc740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x560d922b5570 con 0x560d922a3270 2024-04-24T23:16:37.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.613+0000 7fa365fcc740 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x560d92275330 con 0x560d922a7010 2024-04-24T23:16:37.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.613+0000 7fa35be00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560d922a3270 0x560d92273a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:37.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.613+0000 7fa35b400640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x560d922a7010 0x560d922df320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:37.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.613+0000 7fa35be00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560d922a3270 0x560d92273a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:35278/0 (socket says 172.21.15.6:35278) 2024-04-24T23:16:37.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.613+0000 7fa35be00640 1 -- 172.21.15.6:0/2043993347 learned_addr learned my addr 172.21.15.6:0/2043993347 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:16:37.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.613+0000 7fa361000640 1 --2- 172.21.15.6:0/2043993347 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560d922cebb0 0x560d922df860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:37.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.613+0000 7fa35be00640 1 -- 172.21.15.6:0/2043993347 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x560d922a7010 msgr2=0x560d922df320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:16:37.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.613+0000 7fa35be00640 1 --2- 172.21.15.6:0/2043993347 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x560d922a7010 0x560d922df320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:37.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.613+0000 7fa35be00640 1 -- 172.21.15.6:0/2043993347 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560d922cebb0 msgr2=0x560d922df860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:16:37.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.613+0000 7fa35be00640 1 --2- 172.21.15.6:0/2043993347 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560d922cebb0 0x560d922df860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:37.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.613+0000 7fa35be00640 1 -- 172.21.15.6:0/2043993347 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560d922ce9e0 con 0x560d922a3270 2024-04-24T23:16:37.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.613+0000 7fa361000640 1 --2- 172.21.15.6:0/2043993347 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560d922cebb0 0x560d922df860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:16:37.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.614+0000 7fa35be00640 1 --2- 172.21.15.6:0/2043993347 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560d922a3270 0x560d92273a80 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fa34c006fd0 tx=0x7fa34c00a000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9426dcea364a3ecc server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:37.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.614+0000 7fa347e00640 1 -- 172.21.15.6:0/2043993347 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa34c001b30 con 0x560d922a3270 2024-04-24T23:16:37.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.614+0000 7fa347e00640 1 -- 172.21.15.6:0/2043993347 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa34c01b040 con 0x560d922a3270 2024-04-24T23:16:37.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.614+0000 7fa347e00640 1 -- 172.21.15.6:0/2043993347 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa34c001cd0 con 0x560d922a3270 2024-04-24T23:16:37.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.614+0000 7fa365fcc740 1 -- 172.21.15.6:0/2043993347 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560d922a3270 msgr2=0x560d92273a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:37.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.614+0000 7fa365fcc740 1 --2- 172.21.15.6:0/2043993347 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560d922a3270 0x560d92273a80 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fa34c006fd0 tx=0x7fa34c00a000 comp rx=0 tx=0).stop 2024-04-24T23:16:37.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.614+0000 7fa365fcc740 1 -- 172.21.15.6:0/2043993347 shutdown_connections 2024-04-24T23:16:37.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.614+0000 7fa365fcc740 1 --2- 172.21.15.6:0/2043993347 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x560d922a7010 0x560d922df320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:37.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.614+0000 7fa365fcc740 1 --2- 172.21.15.6:0/2043993347 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560d922cebb0 0x560d922df860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:37.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.614+0000 7fa365fcc740 1 --2- 172.21.15.6:0/2043993347 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560d922a3270 0x560d92273a80 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:37.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.615+0000 7fa365fcc740 1 -- 172.21.15.6:0/2043993347 >> 172.21.15.6:0/2043993347 conn(0x560d92278250 msgr2=0x560d922a3650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:37.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.615+0000 7fa365fcc740 1 -- 172.21.15.6:0/2043993347 shutdown_connections 2024-04-24T23:16:37.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.615+0000 7fa365fcc740 1 -- 172.21.15.6:0/2043993347 wait complete. 2024-04-24T23:16:37.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.615+0000 7fa365fcc740 1 -- start start 2024-04-24T23:16:37.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.615+0000 7fa365fcc740 1 Processor -- start 2024-04-24T23:16:37.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.616+0000 7fa365fcc740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560d922a3270 0x560d922a4ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:37.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.616+0000 7fa365fcc740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560d922a7010 0x560d9232fd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:37.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.616+0000 7fa365fcc740 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x560d922cebb0 0x560d92330270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:37.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.616+0000 7fa365fcc740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x560d922cb3a0 con 0x560d922a7010 2024-04-24T23:16:37.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.616+0000 7fa365fcc740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x560d922cad50 con 0x560d922a3270 2024-04-24T23:16:37.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.616+0000 7fa365fcc740 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x560d922cba50 con 0x560d922cebb0 2024-04-24T23:16:37.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.616+0000 7fa365fcc740 10 client.0 _ll_register_callbacks cb 0x7fa34800e7c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T23:16:37.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.617+0000 7fa35b400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560d922a7010 0x560d9232fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:37.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.617+0000 7fa35b400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560d922a7010 0x560d9232fd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:35562/0 (socket says 172.21.15.6:35562) 2024-04-24T23:16:37.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.617+0000 7fa35be00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560d922a3270 0x560d922a4ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:37.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.617+0000 7fa35b400640 1 -- 172.21.15.6:0/28408440 learned_addr learned my addr 172.21.15.6:0/28408440 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:16:37.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.617+0000 7fa361000640 1 --2- 172.21.15.6:0/28408440 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x560d922cebb0 0x560d92330270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:37.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.617+0000 7fa35be00640 1 -- 172.21.15.6:0/28408440 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x560d922cebb0 msgr2=0x560d92330270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:16:37.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.617+0000 7fa35be00640 1 --2- 172.21.15.6:0/28408440 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x560d922cebb0 0x560d92330270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:37.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.617+0000 7fa35be00640 1 -- 172.21.15.6:0/28408440 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560d922a7010 msgr2=0x560d9232fd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:16:37.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.617+0000 7fa35be00640 1 --2- 172.21.15.6:0/28408440 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560d922a7010 0x560d9232fd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:37.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.617+0000 7fa35be00640 1 -- 172.21.15.6:0/28408440 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560d92279980 con 0x560d922a3270 2024-04-24T23:16:37.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.617+0000 7fa35be00640 1 --2- 172.21.15.6:0/28408440 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560d922a3270 0x560d922a4ae0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fa34c016000 tx=0x7fa34c000ed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6c83b54d43da86cf server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:37.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.617+0000 7fa345600640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-04-24T23:16:37.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.618+0000 7fa345600640 1 -- 172.21.15.6:0/28408440 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa34c02ad60 con 0x560d922a3270 2024-04-24T23:16:37.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.618+0000 7fa365fcc740 1 -- 172.21.15.6:0/28408440 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa34c01b040 con 0x560d922a3270 2024-04-24T23:16:37.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.618+0000 7fa345600640 1 -- 172.21.15.6:0/28408440 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(26..26 src has 1..26) ==== 6590+0+0 (secure 0 0 0) 0x7fa34c006410 con 0x560d922a3270 2024-04-24T23:16:37.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.618+0000 7fa345600640 1 -- 172.21.15.6:0/28408440 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=18) ==== 24+0+0 (secure 0 0 0) 0x7fa34c006610 con 0x560d922a3270 2024-04-24T23:16:37.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.618+0000 7fa365fcc740 10 client.4942 fetch_fsmap learned FSMap version 18 2024-04-24T23:16:37.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.618+0000 7fa365fcc740 1 -- 172.21.15.6:0/28408440 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=18,monmap=2+}) -- 0x560d922c5370 con 0x560d922a3270 2024-04-24T23:16:37.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.618+0000 7fa345600640 1 -- 172.21.15.6:0/28408440 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa34c01b040 con 0x560d922a3270 2024-04-24T23:16:37.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.619+0000 7fa345600640 1 -- 172.21.15.6:0/28408440 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 18) ==== 4792+0+0 (secure 0 0 0) 0x7fa34c020040 con 0x560d922a3270 2024-04-24T23:16:37.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.619+0000 7fa365fcc740 10 client.4942 fetch_fsmap finished waiting for FSMap version 18 2024-04-24T23:16:37.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.619+0000 7fa365fcc740 10 client.4942 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-24T23:16:37.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.619+0000 7fa365fcc740 20 client.4942 populate_metadata read hostname 'smithi006' 2024-04-24T23:16:37.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.619+0000 7fa365fcc740 1 --2- 172.21.15.6:0/28408440 >> [v2:172.21.15.190:6835/2032280158,v1:172.21.15.190:6838/2032280158] conn(0x560d922a1f20 0x560d92342990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:37.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.619+0000 7fa365fcc740 4 client.4942 mds_command: new command op to 4232 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-24T23:16:37.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.619+0000 7fa365fcc740 1 -- 172.21.15.6:0/28408440 --> [v2:172.21.15.190:6835/2032280158,v1:172.21.15.190:6838/2032280158] -- command(tid 0: {"prefix": "status"}) -- 0x560d921e4680 con 0x560d922a1f20 2024-04-24T23:16:37.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.619+0000 7fa35b400640 1 --2- 172.21.15.6:0/28408440 >> [v2:172.21.15.190:6835/2032280158,v1:172.21.15.190:6838/2032280158] conn(0x560d922a1f20 0x560d92342990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:37.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.619+0000 7fa35b400640 1 --2- 172.21.15.6:0/28408440 >> [v2:172.21.15.190:6835/2032280158,v1:172.21.15.190:6838/2032280158] conn(0x560d922a1f20 0x560d92342990 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=3930d6d9060fe3b0 server_cookie=f7624cc4cfb6cfd6 in_seq=0 out_seq=0 2024-04-24T23:16:37.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.619+0000 7fa345600640 10 client.4942 ms_handle_connect on v2:172.21.15.190:6835/2032280158 2024-04-24T23:16:37.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.620+0000 7fa345600640 1 -- 172.21.15.6:0/28408440 <== mds.3 v2:172.21.15.190:6835/2032280158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+319 (crc 0 0 0) 0x560d921e4680 con 0x560d922a1f20 2024-04-24T23:16:37.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.620+0000 7fa345600640 10 client.4942 handle_command_reply: tid=0 2024-04-24T23:16:37.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.620+0000 7fa365fcc740 1 client.4942 shutdown 2024-04-24T23:16:37.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.620+0000 7fa365fcc740 1 -- 172.21.15.6:0/28408440 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560d922a3270 msgr2=0x560d922a4ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:37.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.620+0000 7fa365fcc740 1 --2- 172.21.15.6:0/28408440 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560d922a3270 0x560d922a4ae0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fa34c016000 tx=0x7fa34c000ed0 comp rx=0 tx=0).stop 2024-04-24T23:16:37.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.620+0000 7fa365fcc740 1 -- 172.21.15.6:0/28408440 shutdown_connections 2024-04-24T23:16:37.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.620+0000 7fa365fcc740 1 --2- 172.21.15.6:0/28408440 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x560d922cebb0 0x560d92330270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:37.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.620+0000 7fa365fcc740 1 --2- 172.21.15.6:0/28408440 >> [v2:172.21.15.190:6835/2032280158,v1:172.21.15.190:6838/2032280158] conn(0x560d922a1f20 0x560d92342990 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:37.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.620+0000 7fa365fcc740 1 --2- 172.21.15.6:0/28408440 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560d922a7010 0x560d9232fd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:37.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.620+0000 7fa365fcc740 1 --2- 172.21.15.6:0/28408440 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560d922a3270 0x560d922a4ae0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:37.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.620+0000 7fa345600640 0 client.4942 ms_handle_reset on v2:172.21.15.190:6835/2032280158 2024-04-24T23:16:37.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.620+0000 7fa365fcc740 1 -- 172.21.15.6:0/28408440 >> 172.21.15.6:0/28408440 conn(0x560d92278250 msgr2=0x560d92278850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:37.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.620+0000 7fa365fcc740 1 -- 172.21.15.6:0/28408440 shutdown_connections 2024-04-24T23:16:37.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.620+0000 7fa365fcc740 1 -- 172.21.15.6:0/28408440 wait complete. 2024-04-24T23:16:37.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.620+0000 7fa365fcc740 20 client.4942 trim_cache size 0 max 16384 2024-04-24T23:16:37.625 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-04-24T23:16:37.625 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_fsid": "18bef68d-5e6d-4ec5-862b-b6f206e2588d", 2024-04-24T23:16:37.625 INFO:teuthology.orchestra.run.smithi006.stdout: "whoami": 3, 2024-04-24T23:16:37.625 INFO:teuthology.orchestra.run.smithi006.stdout: "id": 4232, 2024-04-24T23:16:37.626 INFO:teuthology.orchestra.run.smithi006.stdout: "want_state": "up:active", 2024-04-24T23:16:37.626 INFO:teuthology.orchestra.run.smithi006.stdout: "state": "up:active", 2024-04-24T23:16:37.626 INFO:teuthology.orchestra.run.smithi006.stdout: "fs_name": "cephfs", 2024-04-24T23:16:37.626 INFO:teuthology.orchestra.run.smithi006.stdout: "rank_uptime": 17.363655625, 2024-04-24T23:16:37.626 INFO:teuthology.orchestra.run.smithi006.stdout: "mdsmap_epoch": 18, 2024-04-24T23:16:37.626 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap_epoch": 26, 2024-04-24T23:16:37.626 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap_epoch_barrier": 26, 2024-04-24T23:16:37.626 INFO:teuthology.orchestra.run.smithi006.stdout: "uptime": 66.746652286 2024-04-24T23:16:37.626 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-04-24T23:16:37.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.621+0000 7fa361a00640 1 -- 172.21.15.6:0/3065249483 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fa33c06be30 msgr2=0x7fa33c06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:37.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.621+0000 7fa361a00640 1 --2- 172.21.15.6:0/3065249483 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fa33c06be30 0x7fa33c06e2f0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fa34c0011e0 tx=0x7fa34c007d30 comp rx=0 tx=0).stop 2024-04-24T23:16:37.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.621+0000 7fa361a00640 1 -- 172.21.15.6:0/3065249483 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa35c070f30 msgr2=0x7fa35c0c5f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:37.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.621+0000 7fa361a00640 1 --2- 172.21.15.6:0/3065249483 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa35c070f30 0x7fa35c0c5f10 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fa34800e870 tx=0x7fa34800ed30 comp rx=0 tx=0).stop 2024-04-24T23:16:37.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.621+0000 7fa361a00640 1 -- 172.21.15.6:0/3065249483 shutdown_connections 2024-04-24T23:16:37.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.621+0000 7fa361a00640 1 --2- 172.21.15.6:0/3065249483 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fa33c06be30 0x7fa33c06e2f0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:37.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.621+0000 7fa361a00640 1 --2- 172.21.15.6:0/3065249483 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa35c154c80 0x7fa35c0c51b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:37.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.621+0000 7fa361a00640 1 --2- 172.21.15.6:0/3065249483 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa35c070f30 0x7fa35c0c5f10 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:37.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.621+0000 7fa361a00640 1 --2- 172.21.15.6:0/3065249483 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa35c06d060 0x7fa35c15c950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:37.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.621+0000 7fa361a00640 1 -- 172.21.15.6:0/3065249483 >> 172.21.15.6:0/3065249483 conn(0x7fa35c0cc1e0 msgr2=0x7fa35c0cc5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:37.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.621+0000 7fa361a00640 1 -- 172.21.15.6:0/3065249483 shutdown_connections 2024-04-24T23:16:37.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.621+0000 7fa361a00640 1 -- 172.21.15.6:0/3065249483 wait complete. 2024-04-24T23:16:37.638 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-04-24T23:16:37.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.839+0000 7f9664c00640 1 Processor -- start 2024-04-24T23:16:37.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.840+0000 7f9664c00640 1 -- start start 2024-04-24T23:16:37.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.841+0000 7f9664c00640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f9660152d40 0x7f96601592b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:37.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.842+0000 7f9664c00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f96600c4770 0x7f9660163320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:37.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.842+0000 7f9664c00640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9660153120 con 0x7f966006d6d0 2024-04-24T23:16:37.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.842+0000 7f9664c00640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f96600c8aa0 con 0x7f9660152d40 2024-04-24T23:16:37.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.842+0000 7f9664c00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f966006d230 con 0x7f96600c4770 2024-04-24T23:16:37.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.842+0000 7f965de00640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f9660152d40 0x7f96601592b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:37.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.842+0000 7f965e800640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f96600c4770 0x7f9660163320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:37.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.842+0000 7f965d400640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f966006d6d0 0x7f96600c4060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.6:57854/0 (socket says 172.21.15.6:57854) 2024-04-24T23:16:37.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.842+0000 7f965d400640 1 -- 172.21.15.6:0/816669906 learned_addr learned my addr 172.21.15.6:0/816669906 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:16:37.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.842+0000 7f965ca00640 1 -- 172.21.15.6:0/816669906 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1566140871 0 0) 0x7f9660153120 con 0x7f966006d6d0 2024-04-24T23:16:37.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.842+0000 7f965ca00640 1 -- 172.21.15.6:0/816669906 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9644003520 con 0x7f966006d6d0 2024-04-24T23:16:37.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.843+0000 7f965e800640 1 -- 172.21.15.6:0/816669906 >> v2:172.21.15.190:3300/0 conn(0x7f9660152d40 msgr2=0x7f96601592b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:16:37.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.843+0000 7f965e800640 1 --2- 172.21.15.6:0/816669906 >> v2:172.21.15.190:3300/0 conn(0x7f9660152d40 0x7f96601592b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:37.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.843+0000 7f965e800640 1 -- 172.21.15.6:0/816669906 >> v1:172.21.15.190:6789/0 conn(0x7f966006d6d0 legacy=0x7f96600c4060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:37.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.843+0000 7f965e800640 1 -- 172.21.15.6:0/816669906 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96600c63d0 con 0x7f96600c4770 2024-04-24T23:16:37.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.843+0000 7f965de00640 1 --2- 172.21.15.6:0/816669906 >> v2:172.21.15.190:3300/0 conn(0x7f9660152d40 0x7f96601592b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:16:37.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.844+0000 7f965e800640 1 --2- 172.21.15.6:0/816669906 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f96600c4770 0x7f9660163320 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f9658009d90 tx=0x7f965802a6f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e0d6dd81f17904b2 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:37.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.844+0000 7f965ca00640 1 -- 172.21.15.6:0/816669906 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9658007740 con 0x7f96600c4770 2024-04-24T23:16:37.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.844+0000 7f965ca00640 1 -- 172.21.15.6:0/816669906 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f96580078e0 con 0x7f96600c4770 2024-04-24T23:16:37.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.844+0000 7f965ca00640 1 -- 172.21.15.6:0/816669906 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9658007cb0 con 0x7f96600c4770 2024-04-24T23:16:37.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.845+0000 7f9664c00640 1 -- 172.21.15.6:0/816669906 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f96600c4770 msgr2=0x7f9660163320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:37.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.845+0000 7f9664c00640 1 --2- 172.21.15.6:0/816669906 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f96600c4770 0x7f9660163320 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f9658009d90 tx=0x7f965802a6f0 comp rx=0 tx=0).stop 2024-04-24T23:16:37.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.845+0000 7f9664c00640 1 -- 172.21.15.6:0/816669906 shutdown_connections 2024-04-24T23:16:37.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.845+0000 7f9664c00640 1 --2- 172.21.15.6:0/816669906 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f96600c4770 0x7f9660163320 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:37.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.845+0000 7f9664c00640 1 --2- 172.21.15.6:0/816669906 >> v2:172.21.15.190:3300/0 conn(0x7f9660152d40 0x7f96601592b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:37.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.845+0000 7f9664c00640 1 -- 172.21.15.6:0/816669906 >> 172.21.15.6:0/816669906 conn(0x7f96600c8c50 msgr2=0x7f96600c9060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:37.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.845+0000 7f9664c00640 1 -- 172.21.15.6:0/816669906 shutdown_connections 2024-04-24T23:16:37.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.845+0000 7f9664c00640 1 -- 172.21.15.6:0/816669906 wait complete. 2024-04-24T23:16:37.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.847+0000 7f9664c00640 1 Processor -- start 2024-04-24T23:16:37.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.847+0000 7f9664c00640 1 -- start start 2024-04-24T23:16:37.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.848+0000 7f9664c00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f966006d6d0 0x7f96601e2b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:37.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.848+0000 7f9664c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f96600c4770 0x7f9660071270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:37.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.848+0000 7f9664c00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9660152d40 0x7f96601e4ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:37.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.848+0000 7f9664c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f966006f8d0 con 0x7f96600c4770 2024-04-24T23:16:37.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.848+0000 7f9664c00640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f9660158ab0 con 0x7f966006d6d0 2024-04-24T23:16:37.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.848+0000 7f9664c00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f96600c6a50 con 0x7f9660152d40 2024-04-24T23:16:37.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.848+0000 7f965d400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f96600c4770 0x7f9660071270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:37.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.848+0000 7f965d400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f96600c4770 0x7f9660071270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:35574/0 (socket says 172.21.15.6:35574) 2024-04-24T23:16:37.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.848+0000 7f965d400640 1 -- 172.21.15.6:0/1871839078 learned_addr learned my addr 172.21.15.6:0/1871839078 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:16:37.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.848+0000 7f965de00640 1 --2- 172.21.15.6:0/1871839078 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f966006d6d0 0x7f96601e2b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:37.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.848+0000 7f965e800640 1 --2- 172.21.15.6:0/1871839078 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9660152d40 0x7f96601e4ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:37.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.849+0000 7f965de00640 1 -- 172.21.15.6:0/1871839078 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9660152d40 msgr2=0x7f96601e4ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:37.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.849+0000 7f965de00640 1 --2- 172.21.15.6:0/1871839078 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9660152d40 0x7f96601e4ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:37.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.849+0000 7f965de00640 1 -- 172.21.15.6:0/1871839078 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f96600c4770 msgr2=0x7f9660071270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:37.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.849+0000 7f965de00640 1 --2- 172.21.15.6:0/1871839078 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f96600c4770 0x7f9660071270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:37.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.849+0000 7f965de00640 1 -- 172.21.15.6:0/1871839078 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96600ca9c0 con 0x7f966006d6d0 2024-04-24T23:16:37.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.849+0000 7f965d400640 1 --2- 172.21.15.6:0/1871839078 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f96600c4770 0x7f9660071270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:16:37.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.849+0000 7f965de00640 1 --2- 172.21.15.6:0/1871839078 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f966006d6d0 0x7f96601e2b20 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f965000e8a0 tx=0x7f965000ed60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:37.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.849+0000 7f9656000640 1 -- 172.21.15.6:0/1871839078 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9650017070 con 0x7f966006d6d0 2024-04-24T23:16:37.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.849+0000 7f9656000640 1 -- 172.21.15.6:0/1871839078 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f965000c940 con 0x7f966006d6d0 2024-04-24T23:16:37.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.849+0000 7f9656000640 1 -- 172.21.15.6:0/1871839078 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f965000cd30 con 0x7f966006d6d0 2024-04-24T23:16:37.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.849+0000 7f9664c00640 1 -- 172.21.15.6:0/1871839078 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96600cac80 con 0x7f966006d6d0 2024-04-24T23:16:37.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.849+0000 7f9664c00640 1 -- 172.21.15.6:0/1871839078 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96600cb610 con 0x7f966006d6d0 2024-04-24T23:16:37.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.852+0000 7f9656000640 1 -- 172.21.15.6:0/1871839078 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f965001a070 con 0x7f966006d6d0 2024-04-24T23:16:37.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.853+0000 7f9656000640 1 --2- 172.21.15.6:0/1871839078 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f963c06be30 0x7f963c06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:37.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.853+0000 7f9656000640 1 -- 172.21.15.6:0/1871839078 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6590+0+0 (secure 0 0 0) 0x7f965008afd0 con 0x7f966006d6d0 2024-04-24T23:16:37.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.853+0000 7f965d400640 1 --2- 172.21.15.6:0/1871839078 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f963c06be30 0x7f963c06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:37.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.853+0000 7f9667a35740 1 Processor -- start 2024-04-24T23:16:37.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.854+0000 7f9667a35740 1 -- start start 2024-04-24T23:16:37.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.854+0000 7f965d400640 1 --2- 172.21.15.6:0/1871839078 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f963c06be30 0x7f963c06e2f0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f96600bfb10 tx=0x7f9648009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:37.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.854+0000 7f9667a35740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560e91122060 0x560e9113d900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:37.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.854+0000 7f9667a35740 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x560e910f4bf0 0x560e910f6b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:37.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.854+0000 7f9667a35740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560e910f7040 0x560e91122440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:37.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.855+0000 7f9667a35740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x560e90ff8ed0 con 0x560e91122060 2024-04-24T23:16:37.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.855+0000 7f9667a35740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x560e9105bdf0 con 0x560e910f7040 2024-04-24T23:16:37.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.855+0000 7f9667a35740 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x560e910048d0 con 0x560e910f4bf0 2024-04-24T23:16:37.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.855+0000 7f965de00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560e91122060 0x560e9113d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:37.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.855+0000 7f965de00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560e91122060 0x560e9113d900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:35582/0 (socket says 172.21.15.6:35582) 2024-04-24T23:16:37.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.855+0000 7f965de00640 1 -- 172.21.15.6:0/2153236061 learned_addr learned my addr 172.21.15.6:0/2153236061 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:16:37.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.855+0000 7f965d400640 1 --2- 172.21.15.6:0/2153236061 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x560e910f4bf0 0x560e910f6b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:37.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.855+0000 7f965e800640 1 --2- 172.21.15.6:0/2153236061 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560e910f7040 0x560e91122440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:37.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.855+0000 7f965de00640 1 -- 172.21.15.6:0/2153236061 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x560e910f4bf0 msgr2=0x560e910f6b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:16:37.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.855+0000 7f965de00640 1 --2- 172.21.15.6:0/2153236061 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x560e910f4bf0 0x560e910f6b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:37.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.855+0000 7f965de00640 1 -- 172.21.15.6:0/2153236061 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560e910f7040 msgr2=0x560e91122440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:16:37.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.855+0000 7f965de00640 1 --2- 172.21.15.6:0/2153236061 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560e910f7040 0x560e91122440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:37.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.855+0000 7f965de00640 1 -- 172.21.15.6:0/2153236061 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560e90d702d0 con 0x560e91122060 2024-04-24T23:16:37.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.855+0000 7f965de00640 1 --2- 172.21.15.6:0/2153236061 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560e91122060 0x560e9113d900 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f9650061520 tx=0x7f96500590a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5328f1a1fdb68c0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:37.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.856+0000 7f964e600640 1 -- 172.21.15.6:0/2153236061 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9650021040 con 0x560e91122060 2024-04-24T23:16:37.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.856+0000 7f964e600640 1 -- 172.21.15.6:0/2153236061 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9650009280 con 0x560e91122060 2024-04-24T23:16:37.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.856+0000 7f964e600640 1 -- 172.21.15.6:0/2153236061 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9650058370 con 0x560e91122060 2024-04-24T23:16:37.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.856+0000 7f9667a35740 1 -- 172.21.15.6:0/2153236061 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560e91122060 msgr2=0x560e9113d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:37.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.856+0000 7f9667a35740 1 --2- 172.21.15.6:0/2153236061 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560e91122060 0x560e9113d900 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f9650061520 tx=0x7f96500590a0 comp rx=0 tx=0).stop 2024-04-24T23:16:37.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.856+0000 7f9667a35740 1 -- 172.21.15.6:0/2153236061 shutdown_connections 2024-04-24T23:16:37.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.856+0000 7f9667a35740 1 --2- 172.21.15.6:0/2153236061 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x560e910f4bf0 0x560e910f6b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:37.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.856+0000 7f9667a35740 1 --2- 172.21.15.6:0/2153236061 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560e910f7040 0x560e91122440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:37.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.856+0000 7f9667a35740 1 --2- 172.21.15.6:0/2153236061 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560e91122060 0x560e9113d900 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:37.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.856+0000 7f9667a35740 1 -- 172.21.15.6:0/2153236061 >> 172.21.15.6:0/2153236061 conn(0x560e910f5a80 msgr2=0x560e9113cf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:37.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.857+0000 7f9667a35740 1 -- 172.21.15.6:0/2153236061 shutdown_connections 2024-04-24T23:16:37.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.857+0000 7f9667a35740 1 -- 172.21.15.6:0/2153236061 wait complete. 2024-04-24T23:16:37.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.857+0000 7f9667a35740 1 -- start start 2024-04-24T23:16:37.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.857+0000 7f9667a35740 1 Processor -- start 2024-04-24T23:16:37.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.858+0000 7f9667a35740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560e91122060 0x560e9113b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:37.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.858+0000 7f9667a35740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560e91134a40 0x560e911407c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:37.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.858+0000 7f9667a35740 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x560e9114a800 0x560e9111f9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:37.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.858+0000 7f9667a35740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x560e91017d20 con 0x560e91122060 2024-04-24T23:16:37.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.858+0000 7f9667a35740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x560e91057db0 con 0x560e91134a40 2024-04-24T23:16:37.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.858+0000 7f9667a35740 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x560e910f5080 con 0x560e9114a800 2024-04-24T23:16:37.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.859+0000 7f9667a35740 10 client.0 _ll_register_callbacks cb 0x560e91052f10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T23:16:37.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.859+0000 7f965de00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560e91122060 0x560e9113b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:37.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.859+0000 7f965d400640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560e91134a40 0x560e911407c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:37.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.859+0000 7f965de00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560e91122060 0x560e9113b730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:35588/0 (socket says 172.21.15.6:35588) 2024-04-24T23:16:37.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.859+0000 7f965de00640 1 -- 172.21.15.6:0/2410890935 learned_addr learned my addr 172.21.15.6:0/2410890935 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:16:37.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.859+0000 7f965e800640 1 --2- 172.21.15.6:0/2410890935 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x560e9114a800 0x560e9111f9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:37.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.859+0000 7f965de00640 1 -- 172.21.15.6:0/2410890935 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x560e9114a800 msgr2=0x560e9111f9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:16:37.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.859+0000 7f965de00640 1 --2- 172.21.15.6:0/2410890935 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x560e9114a800 0x560e9111f9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:37.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.859+0000 7f965de00640 1 -- 172.21.15.6:0/2410890935 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560e91134a40 msgr2=0x560e911407c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:16:37.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.859+0000 7f965de00640 1 --2- 172.21.15.6:0/2410890935 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560e91134a40 0x560e911407c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:37.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.859+0000 7f965de00640 1 -- 172.21.15.6:0/2410890935 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560e90e3a6b0 con 0x560e91122060 2024-04-24T23:16:37.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.859+0000 7f965d400640 1 --2- 172.21.15.6:0/2410890935 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560e91134a40 0x560e911407c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:16:37.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.860+0000 7f965de00640 1 --2- 172.21.15.6:0/2410890935 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560e91122060 0x560e9113b730 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f9650058bd0 tx=0x7f965000efb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=655a3c01f8cc0ec9 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:37.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.860+0000 7f9642a00640 10 client.0 ms_handle_connect on v2:172.21.15.6:3300/0 2024-04-24T23:16:37.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.860+0000 7f9642a00640 1 -- 172.21.15.6:0/2410890935 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9650073070 con 0x560e91122060 2024-04-24T23:16:37.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.860+0000 7f9642a00640 1 -- 172.21.15.6:0/2410890935 <== mon.0 v2:172.21.15.6:3300/0 2 ==== osd_map(26..26 src has 1..26) ==== 6590+0+0 (secure 0 0 0) 0x7f9650058580 con 0x560e91122060 2024-04-24T23:16:37.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.860+0000 7f9667a35740 1 -- 172.21.15.6:0/2410890935 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9650009280 con 0x560e91122060 2024-04-24T23:16:37.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.860+0000 7f9642a00640 1 -- 172.21.15.6:0/2410890935 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_get_version_reply(handle=1 version=18) ==== 24+0+0 (secure 0 0 0) 0x7f9650086030 con 0x560e91122060 2024-04-24T23:16:37.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.860+0000 7f9667a35740 10 client.4815 fetch_fsmap learned FSMap version 18 2024-04-24T23:16:37.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.860+0000 7f9667a35740 1 -- 172.21.15.6:0/2410890935 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,fsmap=18,monmap=2+}) -- 0x560e9113ac50 con 0x560e91122060 2024-04-24T23:16:37.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.860+0000 7f9642a00640 1 -- 172.21.15.6:0/2410890935 <== mon.0 v2:172.21.15.6:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9650009280 con 0x560e91122060 2024-04-24T23:16:37.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.860+0000 7f9642a00640 1 -- 172.21.15.6:0/2410890935 <== mon.0 v2:172.21.15.6:3300/0 5 ==== fsmap(e 18) ==== 4792+0+0 (secure 0 0 0) 0x7f9650024060 con 0x560e91122060 2024-04-24T23:16:37.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.861+0000 7f9667a35740 10 client.4815 fetch_fsmap finished waiting for FSMap version 18 2024-04-24T23:16:37.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.861+0000 7f9667a35740 10 client.4815 resolve_mds: resolved name 'd' to daemon mds.d 2024-04-24T23:16:37.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.861+0000 7f9667a35740 20 client.4815 populate_metadata read hostname 'smithi006' 2024-04-24T23:16:37.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.861+0000 7f9667a35740 1 --2- 172.21.15.6:0/2410890935 >> [v2:172.21.15.190:6836/1572872451,v1:172.21.15.190:6839/1572872451] conn(0x560e91156d00 0x560e911590f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:37.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.861+0000 7f9667a35740 4 client.4815 mds_command: new command op to 4235 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T23:16:37.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.861+0000 7f9667a35740 1 -- 172.21.15.6:0/2410890935 --> [v2:172.21.15.190:6836/1572872451,v1:172.21.15.190:6839/1572872451] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560e91069330 con 0x560e91156d00 2024-04-24T23:16:37.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.861+0000 7f965d400640 1 --2- 172.21.15.6:0/2410890935 >> [v2:172.21.15.190:6836/1572872451,v1:172.21.15.190:6839/1572872451] conn(0x560e91156d00 0x560e911590f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:37.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.862+0000 7f965d400640 1 --2- 172.21.15.6:0/2410890935 >> [v2:172.21.15.190:6836/1572872451,v1:172.21.15.190:6839/1572872451] conn(0x560e91156d00 0x560e911590f0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=3da0c39a14915cad server_cookie=954b86e64ddf56d0 in_seq=0 out_seq=0 2024-04-24T23:16:37.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.862+0000 7f9642a00640 10 client.4815 ms_handle_connect on v2:172.21.15.190:6836/1572872451 2024-04-24T23:16:37.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.863+0000 7f9642a00640 1 -- 172.21.15.6:0/2410890935 <== mds.2 v2:172.21.15.190:6836/1572872451 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560e91069330 con 0x560e91156d00 2024-04-24T23:16:37.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.863+0000 7f9642a00640 10 client.4815 handle_command_reply: tid=0 2024-04-24T23:16:37.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.863+0000 7f9667a35740 1 client.4815 shutdown 2024-04-24T23:16:37.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.863+0000 7f9667a35740 1 -- 172.21.15.6:0/2410890935 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560e91122060 msgr2=0x560e9113b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:37.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.863+0000 7f9667a35740 1 --2- 172.21.15.6:0/2410890935 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560e91122060 0x560e9113b730 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f9650058bd0 tx=0x7f965000efb0 comp rx=0 tx=0).stop 2024-04-24T23:16:37.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.863+0000 7f9667a35740 1 -- 172.21.15.6:0/2410890935 shutdown_connections 2024-04-24T23:16:37.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.863+0000 7f9667a35740 1 --2- 172.21.15.6:0/2410890935 >> [v2:172.21.15.190:6836/1572872451,v1:172.21.15.190:6839/1572872451] conn(0x560e91156d00 0x560e911590f0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:37.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.863+0000 7f9667a35740 1 --2- 172.21.15.6:0/2410890935 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x560e9114a800 0x560e9111f9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:37.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.863+0000 7f9667a35740 1 --2- 172.21.15.6:0/2410890935 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560e91134a40 0x560e911407c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:37.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.863+0000 7f9667a35740 1 --2- 172.21.15.6:0/2410890935 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560e91122060 0x560e9113b730 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:37.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.863+0000 7f9642a00640 0 client.4815 ms_handle_reset on v2:172.21.15.190:6836/1572872451 2024-04-24T23:16:37.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.863+0000 7f9667a35740 1 -- 172.21.15.6:0/2410890935 >> 172.21.15.6:0/2410890935 conn(0x560e910f7040 msgr2=0x560e910f8e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:37.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.863+0000 7f9667a35740 1 -- 172.21.15.6:0/2410890935 shutdown_connections 2024-04-24T23:16:37.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.863+0000 7f9667a35740 1 -- 172.21.15.6:0/2410890935 wait complete. 2024-04-24T23:16:37.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.863+0000 7f9667a35740 20 client.4815 trim_cache size 0 max 16384 2024-04-24T23:16:37.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.883+0000 7f9667a35740 1 Processor -- start 2024-04-24T23:16:37.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.883+0000 7f9667a35740 1 -- start start 2024-04-24T23:16:37.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.884+0000 7f9667a35740 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x560e91122060 0x560e9113c610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:37.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.884+0000 7f9667a35740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560e91134a40 0x560e91122440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:37.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.884+0000 7f9667a35740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560e9114a800 0x560e91122980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:37.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.884+0000 7f9667a35740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f96601fc7e0 con 0x560e9114a800 2024-04-24T23:16:37.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.884+0000 7f9667a35740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x560e910f5080 con 0x560e91134a40 2024-04-24T23:16:37.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.884+0000 7f9667a35740 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x560e91017d20 con 0x560e91122060 2024-04-24T23:16:37.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.884+0000 7f965e800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560e9114a800 0x560e91122980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:37.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.884+0000 7f965e800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560e9114a800 0x560e91122980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:35600/0 (socket says 172.21.15.6:35600) 2024-04-24T23:16:37.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.884+0000 7f965e800640 1 -- 172.21.15.6:0/3115878401 learned_addr learned my addr 172.21.15.6:0/3115878401 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:16:37.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.884+0000 7f965de00640 1 --2- 172.21.15.6:0/3115878401 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x560e91122060 0x560e9113c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:37.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.885+0000 7f965d400640 1 --2- 172.21.15.6:0/3115878401 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560e91134a40 0x560e91122440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:37.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.885+0000 7f965e800640 1 -- 172.21.15.6:0/3115878401 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x560e91122060 msgr2=0x560e9113c610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:16:37.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.885+0000 7f965e800640 1 --2- 172.21.15.6:0/3115878401 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x560e91122060 0x560e9113c610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:37.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.885+0000 7f965e800640 1 -- 172.21.15.6:0/3115878401 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560e91134a40 msgr2=0x560e91122440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:16:37.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.885+0000 7f965e800640 1 --2- 172.21.15.6:0/3115878401 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560e91134a40 0x560e91122440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:37.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.885+0000 7f965e800640 1 -- 172.21.15.6:0/3115878401 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560e91140b30 con 0x560e9114a800 2024-04-24T23:16:37.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.885+0000 7f965e800640 1 --2- 172.21.15.6:0/3115878401 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560e9114a800 0x560e91122980 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f965802ace0 tx=0x7f965802e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=488b556b44fdc2f0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:37.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.885+0000 7f964dc00640 1 -- 172.21.15.6:0/3115878401 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96580077c0 con 0x560e9114a800 2024-04-24T23:16:37.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.885+0000 7f964dc00640 1 -- 172.21.15.6:0/3115878401 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9658007960 con 0x560e9114a800 2024-04-24T23:16:37.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.885+0000 7f964dc00640 1 -- 172.21.15.6:0/3115878401 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9658007d10 con 0x560e9114a800 2024-04-24T23:16:37.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.886+0000 7f9667a35740 1 -- 172.21.15.6:0/3115878401 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560e9114a800 msgr2=0x560e91122980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:37.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.886+0000 7f9667a35740 1 --2- 172.21.15.6:0/3115878401 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560e9114a800 0x560e91122980 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f965802ace0 tx=0x7f965802e000 comp rx=0 tx=0).stop 2024-04-24T23:16:37.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.886+0000 7f9667a35740 1 -- 172.21.15.6:0/3115878401 shutdown_connections 2024-04-24T23:16:37.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.886+0000 7f9667a35740 1 --2- 172.21.15.6:0/3115878401 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560e9114a800 0x560e91122980 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:37.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.886+0000 7f9667a35740 1 --2- 172.21.15.6:0/3115878401 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560e91134a40 0x560e91122440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:37.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.886+0000 7f9667a35740 1 --2- 172.21.15.6:0/3115878401 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x560e91122060 0x560e9113c610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:37.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.886+0000 7f9667a35740 1 -- 172.21.15.6:0/3115878401 >> 172.21.15.6:0/3115878401 conn(0x560e910f7040 msgr2=0x560e910eeb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:37.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.886+0000 7f9667a35740 1 -- 172.21.15.6:0/3115878401 shutdown_connections 2024-04-24T23:16:37.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.886+0000 7f9667a35740 1 -- 172.21.15.6:0/3115878401 wait complete. 2024-04-24T23:16:37.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.887+0000 7f9667a35740 1 -- start start 2024-04-24T23:16:37.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.887+0000 7f9667a35740 1 Processor -- start 2024-04-24T23:16:37.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.888+0000 7f9667a35740 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x560e91122060 0x560e9114a100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:37.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.888+0000 7f9667a35740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560e91134a40 0x560e911b7000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:37.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.888+0000 7f9667a35740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560e9114a800 0x560e911b7540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:37.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.888+0000 7f9667a35740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x560e9113cd70 con 0x560e91134a40 2024-04-24T23:16:37.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.888+0000 7f9667a35740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x560e91057db0 con 0x560e9114a800 2024-04-24T23:16:37.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.888+0000 7f9667a35740 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x560e9114d930 con 0x560e91122060 2024-04-24T23:16:37.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.888+0000 7f9667a35740 10 client.0 _ll_register_callbacks cb 0x7f9650009060 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T23:16:37.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.888+0000 7f965e800640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560e9114a800 0x560e911b7540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:37.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.888+0000 7f965e800640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560e9114a800 0x560e911b7540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:35344/0 (socket says 172.21.15.6:35344) 2024-04-24T23:16:37.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.888+0000 7f965de00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x560e91122060 0x560e9114a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:37.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.888+0000 7f965e800640 1 -- 172.21.15.6:0/1710452265 learned_addr learned my addr 172.21.15.6:0/1710452265 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:16:37.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.888+0000 7f965d400640 1 --2- 172.21.15.6:0/1710452265 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560e91134a40 0x560e911b7000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:37.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.889+0000 7f965de00640 1 -- 172.21.15.6:0/1710452265 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560e9114a800 msgr2=0x560e911b7540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:16:37.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.889+0000 7f965de00640 1 --2- 172.21.15.6:0/1710452265 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560e9114a800 0x560e911b7540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:37.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.889+0000 7f965de00640 1 -- 172.21.15.6:0/1710452265 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560e91134a40 msgr2=0x560e911b7000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:16:37.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.889+0000 7f965de00640 1 --2- 172.21.15.6:0/1710452265 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560e91134a40 0x560e911b7000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:37.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.889+0000 7f965de00640 1 -- 172.21.15.6:0/1710452265 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({osdmap=0}) -- 0x560e9113a800 con 0x560e91122060 2024-04-24T23:16:37.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.889+0000 7f965e800640 1 --2- 172.21.15.6:0/1710452265 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560e9114a800 0x560e911b7540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:16:37.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.889+0000 7f965de00640 1 --2- 172.21.15.6:0/1710452265 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x560e91122060 0x560e9114a100 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f9650058eb0 tx=0x7f9650055240 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6cf7bdf8c5d4cb59 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:37.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.889+0000 7f9642a00640 10 client.0 ms_handle_connect on v2:172.21.15.190:3301/0 2024-04-24T23:16:37.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.889+0000 7f9642a00640 1 -- 172.21.15.6:0/1710452265 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9650069060 con 0x560e91122060 2024-04-24T23:16:37.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.889+0000 7f9642a00640 1 -- 172.21.15.6:0/1710452265 <== mon.2 v2:172.21.15.190:3301/0 2 ==== osd_map(26..26 src has 1..26) ==== 6590+0+0 (secure 0 0 0) 0x7f9650058580 con 0x560e91122060 2024-04-24T23:16:37.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.889+0000 7f9667a35740 1 -- 172.21.15.6:0/1710452265 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9658007960 con 0x560e91122060 2024-04-24T23:16:37.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.890+0000 7f9642a00640 1 -- 172.21.15.6:0/1710452265 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_get_version_reply(handle=1 version=18) ==== 24+0+0 (secure 0 0 0) 0x7f9650043e60 con 0x560e91122060 2024-04-24T23:16:37.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.890+0000 7f9667a35740 10 client.4715 fetch_fsmap learned FSMap version 18 2024-04-24T23:16:37.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.890+0000 7f9667a35740 1 -- 172.21.15.6:0/1710452265 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,fsmap=18,monmap=2+}) -- 0x560e9113a800 con 0x560e91122060 2024-04-24T23:16:37.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.890+0000 7f9642a00640 1 -- 172.21.15.6:0/1710452265 <== mon.2 v2:172.21.15.190:3301/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9658007960 con 0x560e91122060 2024-04-24T23:16:37.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.890+0000 7f9642a00640 1 -- 172.21.15.6:0/1710452265 <== mon.2 v2:172.21.15.190:3301/0 5 ==== fsmap(e 18) ==== 4792+0+0 (secure 0 0 0) 0x7f965005f420 con 0x560e91122060 2024-04-24T23:16:37.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.890+0000 7f9667a35740 10 client.4715 fetch_fsmap finished waiting for FSMap version 18 2024-04-24T23:16:37.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.890+0000 7f9667a35740 10 client.4715 resolve_mds: resolved name 'd' to daemon mds.d 2024-04-24T23:16:37.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.890+0000 7f9667a35740 20 client.4715 populate_metadata read hostname 'smithi006' 2024-04-24T23:16:37.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.890+0000 7f9667a35740 1 --2- 172.21.15.6:0/1710452265 >> [v2:172.21.15.190:6836/1572872451,v1:172.21.15.190:6839/1572872451] conn(0x560e91143310 0x560e911436f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:37.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.890+0000 7f9667a35740 4 client.4715 mds_command: new command op to 4235 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-24T23:16:37.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.890+0000 7f9667a35740 1 -- 172.21.15.6:0/1710452265 --> [v2:172.21.15.190:6836/1572872451,v1:172.21.15.190:6839/1572872451] -- command(tid 0: {"prefix": "status"}) -- 0x560e91063680 con 0x560e91143310 2024-04-24T23:16:37.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.890+0000 7f965d400640 1 --2- 172.21.15.6:0/1710452265 >> [v2:172.21.15.190:6836/1572872451,v1:172.21.15.190:6839/1572872451] conn(0x560e91143310 0x560e911436f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:37.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.891+0000 7f965d400640 1 --2- 172.21.15.6:0/1710452265 >> [v2:172.21.15.190:6836/1572872451,v1:172.21.15.190:6839/1572872451] conn(0x560e91143310 0x560e911436f0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b617e449dcbca35d server_cookie=adbf751ab2472892 in_seq=0 out_seq=0 2024-04-24T23:16:37.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.891+0000 7f9642a00640 10 client.4715 ms_handle_connect on v2:172.21.15.190:6836/1572872451 2024-04-24T23:16:37.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.891+0000 7f9642a00640 1 -- 172.21.15.6:0/1710452265 <== mds.2 v2:172.21.15.190:6836/1572872451 1 ==== command_reply(tid 0: 0 ) ==== 8+0+325 (crc 0 0 0) 0x560e91063680 con 0x560e91143310 2024-04-24T23:16:37.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.891+0000 7f9642a00640 10 client.4715 handle_command_reply: tid=0 2024-04-24T23:16:37.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.891+0000 7f9667a35740 1 client.4715 shutdown 2024-04-24T23:16:37.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.891+0000 7f9667a35740 1 -- 172.21.15.6:0/1710452265 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x560e91122060 msgr2=0x560e9114a100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:37.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.891+0000 7f9667a35740 1 --2- 172.21.15.6:0/1710452265 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x560e91122060 0x560e9114a100 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f9650058eb0 tx=0x7f9650055240 comp rx=0 tx=0).stop 2024-04-24T23:16:37.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.891+0000 7f9667a35740 1 -- 172.21.15.6:0/1710452265 shutdown_connections 2024-04-24T23:16:37.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.891+0000 7f9667a35740 1 --2- 172.21.15.6:0/1710452265 >> [v2:172.21.15.190:6836/1572872451,v1:172.21.15.190:6839/1572872451] conn(0x560e91143310 0x560e911436f0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:37.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.891+0000 7f9667a35740 1 --2- 172.21.15.6:0/1710452265 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560e9114a800 0x560e911b7540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:37.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.891+0000 7f9667a35740 1 --2- 172.21.15.6:0/1710452265 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560e91134a40 0x560e911b7000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:37.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.891+0000 7f9667a35740 1 --2- 172.21.15.6:0/1710452265 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x560e91122060 0x560e9114a100 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:37.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.891+0000 7f9642a00640 0 client.4715 ms_handle_reset on v2:172.21.15.190:6836/1572872451 2024-04-24T23:16:37.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.891+0000 7f9667a35740 1 -- 172.21.15.6:0/1710452265 >> 172.21.15.6:0/1710452265 conn(0x560e910f7040 msgr2=0x560e9111f680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:37.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.892+0000 7f9667a35740 1 -- 172.21.15.6:0/1710452265 shutdown_connections 2024-04-24T23:16:37.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.892+0000 7f9667a35740 1 -- 172.21.15.6:0/1710452265 wait complete. 2024-04-24T23:16:37.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.892+0000 7f9667a35740 20 client.4715 trim_cache size 0 max 16384 2024-04-24T23:16:37.898 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-04-24T23:16:37.898 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_fsid": "18bef68d-5e6d-4ec5-862b-b6f206e2588d", 2024-04-24T23:16:37.898 INFO:teuthology.orchestra.run.smithi006.stdout: "whoami": 2, 2024-04-24T23:16:37.898 INFO:teuthology.orchestra.run.smithi006.stdout: "id": 4235, 2024-04-24T23:16:37.898 INFO:teuthology.orchestra.run.smithi006.stdout: "want_state": "up:active", 2024-04-24T23:16:37.898 INFO:teuthology.orchestra.run.smithi006.stdout: "state": "up:active", 2024-04-24T23:16:37.898 INFO:teuthology.orchestra.run.smithi006.stdout: "fs_name": "cephfs", 2024-04-24T23:16:37.898 INFO:teuthology.orchestra.run.smithi006.stdout: "rank_uptime": 23.681953338, 2024-04-24T23:16:37.898 INFO:teuthology.orchestra.run.smithi006.stdout: "mdsmap_epoch": 18, 2024-04-24T23:16:37.898 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap_epoch": 26, 2024-04-24T23:16:37.898 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap_epoch_barrier": 26, 2024-04-24T23:16:37.898 INFO:teuthology.orchestra.run.smithi006.stdout: "uptime": 67.017171376999997 2024-04-24T23:16:37.899 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-04-24T23:16:37.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.892+0000 7f9664c00640 1 -- 172.21.15.6:0/1871839078 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f963c06be30 msgr2=0x7f963c06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:37.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.892+0000 7f9664c00640 1 --2- 172.21.15.6:0/1871839078 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f963c06be30 0x7f963c06e2f0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f96600bfb10 tx=0x7f9648009040 comp rx=0 tx=0).stop 2024-04-24T23:16:37.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.892+0000 7f9664c00640 1 -- 172.21.15.6:0/1871839078 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f966006d6d0 msgr2=0x7f96601e2b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:37.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.892+0000 7f9664c00640 1 --2- 172.21.15.6:0/1871839078 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f966006d6d0 0x7f96601e2b20 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f965000e8a0 tx=0x7f965000ed60 comp rx=0 tx=0).stop 2024-04-24T23:16:37.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.893+0000 7f9664c00640 1 -- 172.21.15.6:0/1871839078 shutdown_connections 2024-04-24T23:16:37.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.893+0000 7f9664c00640 1 --2- 172.21.15.6:0/1871839078 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f963c06be30 0x7f963c06e2f0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:37.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.893+0000 7f9664c00640 1 --2- 172.21.15.6:0/1871839078 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9660152d40 0x7f96601e4ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:37.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.893+0000 7f9664c00640 1 --2- 172.21.15.6:0/1871839078 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f96600c4770 0x7f9660071270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:37.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.893+0000 7f9664c00640 1 --2- 172.21.15.6:0/1871839078 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f966006d6d0 0x7f96601e2b20 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:37.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.893+0000 7f9664c00640 1 -- 172.21.15.6:0/1871839078 >> 172.21.15.6:0/1871839078 conn(0x7f96600c8c50 msgr2=0x7f96600c31a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:37.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.893+0000 7f9664c00640 1 -- 172.21.15.6:0/1871839078 shutdown_connections 2024-04-24T23:16:37.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:37.893+0000 7f9664c00640 1 -- 172.21.15.6:0/1871839078 wait complete. 2024-04-24T23:16:37.911 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.f status 2024-04-24T23:16:38.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.118+0000 7f8c58a00640 1 Processor -- start 2024-04-24T23:16:38.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.118+0000 7f8c58a00640 1 -- start start 2024-04-24T23:16:38.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.120+0000 7f8c58a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f8c540733e0 0x7f8c540c49e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:38.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.120+0000 7f8c58a00640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f8c540c4350 0x7f8c540c9f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:38.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.120+0000 7f8c58a00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8c540c5000 con 0x7f8c54151c00 2024-04-24T23:16:38.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.120+0000 7f8c58a00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f8c540c7340 con 0x7f8c540733e0 2024-04-24T23:16:38.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.120+0000 7f8c58a00640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f8c5415b150 con 0x7f8c540c4350 2024-04-24T23:16:38.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.120+0000 7f8c51400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f8c540733e0 0x7f8c540c49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:38.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.120+0000 7f8c51e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f8c54151c00 0x7f8c5415b5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:40272/0 (socket says 172.21.15.6:40272) 2024-04-24T23:16:38.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.120+0000 7f8c51e00640 1 -- 172.21.15.6:0/1391272238 learned_addr learned my addr 172.21.15.6:0/1391272238 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:16:38.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.120+0000 7f8c52800640 1 --2- 172.21.15.6:0/1391272238 >> v2:172.21.15.190:3300/0 conn(0x7f8c540c4350 0x7f8c540c9f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:38.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.121+0000 7f8c51400640 1 -- 172.21.15.6:0/1391272238 >> v2:172.21.15.190:3300/0 conn(0x7f8c540c4350 msgr2=0x7f8c540c9f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:16:38.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.121+0000 7f8c51400640 1 --2- 172.21.15.6:0/1391272238 >> v2:172.21.15.190:3300/0 conn(0x7f8c540c4350 0x7f8c540c9f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:38.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.121+0000 7f8c51400640 1 -- 172.21.15.6:0/1391272238 >> v1:172.21.15.6:6789/0 conn(0x7f8c54151c00 legacy=0x7f8c5415b5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:38.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.121+0000 7f8c51400640 1 -- 172.21.15.6:0/1391272238 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c54072b30 con 0x7f8c540733e0 2024-04-24T23:16:38.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.121+0000 7f8c50a00640 1 -- 172.21.15.6:0/1391272238 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 409515965 0 0) 0x7f8c540c5000 con 0x7f8c54151c00 2024-04-24T23:16:38.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.122+0000 7f8c51400640 1 --2- 172.21.15.6:0/1391272238 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8c540733e0 0x7f8c540c49e0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f8c5406f380 tx=0x7f8c3c02a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe7343ab88f66adc server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:38.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.122+0000 7f8c50a00640 1 -- 172.21.15.6:0/1391272238 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c3c037070 con 0x7f8c540733e0 2024-04-24T23:16:38.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.122+0000 7f8c50a00640 1 -- 172.21.15.6:0/1391272238 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8c3c032440 con 0x7f8c540733e0 2024-04-24T23:16:38.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.122+0000 7f8c50a00640 1 -- 172.21.15.6:0/1391272238 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c3c032810 con 0x7f8c540733e0 2024-04-24T23:16:38.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.123+0000 7f8c58a00640 1 -- 172.21.15.6:0/1391272238 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8c540733e0 msgr2=0x7f8c540c49e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:38.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.123+0000 7f8c58a00640 1 --2- 172.21.15.6:0/1391272238 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8c540733e0 0x7f8c540c49e0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f8c5406f380 tx=0x7f8c3c02a230 comp rx=0 tx=0).stop 2024-04-24T23:16:38.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.123+0000 7f8c58a00640 1 -- 172.21.15.6:0/1391272238 shutdown_connections 2024-04-24T23:16:38.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.123+0000 7f8c58a00640 1 --2- 172.21.15.6:0/1391272238 >> v2:172.21.15.190:3300/0 conn(0x7f8c540c4350 0x7f8c540c9f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:38.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.123+0000 7f8c58a00640 1 --2- 172.21.15.6:0/1391272238 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8c540733e0 0x7f8c540c49e0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:38.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.123+0000 7f8c58a00640 1 -- 172.21.15.6:0/1391272238 >> 172.21.15.6:0/1391272238 conn(0x7f8c540670e0 msgr2=0x7f8c5406c1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:38.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.123+0000 7f8c58a00640 1 -- 172.21.15.6:0/1391272238 shutdown_connections 2024-04-24T23:16:38.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.123+0000 7f8c58a00640 1 -- 172.21.15.6:0/1391272238 wait complete. 2024-04-24T23:16:38.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.125+0000 7f8c58a00640 1 Processor -- start 2024-04-24T23:16:38.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.125+0000 7f8c58a00640 1 -- start start 2024-04-24T23:16:38.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.126+0000 7f8c58a00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8c540733e0 0x7f8c541e5330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:38.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.126+0000 7f8c58a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8c540c4350 0x7f8c54146840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:38.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.126+0000 7f8c58a00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8c54151c00 0x7f8c54146d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:38.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.126+0000 7f8c58a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f8c54072830 con 0x7f8c540c4350 2024-04-24T23:16:38.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.126+0000 7f8c58a00640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f8c54157940 con 0x7f8c54151c00 2024-04-24T23:16:38.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.126+0000 7f8c58a00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f8c5415bcf0 con 0x7f8c540733e0 2024-04-24T23:16:38.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.126+0000 7f8c51400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8c540c4350 0x7f8c54146840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:38.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.126+0000 7f8c51400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8c540c4350 0x7f8c54146840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:35620/0 (socket says 172.21.15.6:35620) 2024-04-24T23:16:38.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.126+0000 7f8c51400640 1 -- 172.21.15.6:0/212475757 learned_addr learned my addr 172.21.15.6:0/212475757 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:16:38.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.126+0000 7f8c51e00640 1 --2- 172.21.15.6:0/212475757 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8c540733e0 0x7f8c541e5330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:38.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.126+0000 7f8c52800640 1 --2- 172.21.15.6:0/212475757 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8c54151c00 0x7f8c54146d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:38.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.127+0000 7f8c51400640 1 -- 172.21.15.6:0/212475757 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8c540733e0 msgr2=0x7f8c541e5330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:38.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.127+0000 7f8c51400640 1 --2- 172.21.15.6:0/212475757 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8c540733e0 0x7f8c541e5330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:38.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.127+0000 7f8c51400640 1 -- 172.21.15.6:0/212475757 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8c54151c00 msgr2=0x7f8c54146d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:38.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.127+0000 7f8c51400640 1 --2- 172.21.15.6:0/212475757 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8c54151c00 0x7f8c54146d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:38.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.127+0000 7f8c51400640 1 -- 172.21.15.6:0/212475757 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c541f0dd0 con 0x7f8c540c4350 2024-04-24T23:16:38.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.127+0000 7f8c51400640 1 --2- 172.21.15.6:0/212475757 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8c540c4350 0x7f8c54146840 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f8c3c000c00 tx=0x7f8c3c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:38.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.127+0000 7f8c4a000640 1 -- 172.21.15.6:0/212475757 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c3c037040 con 0x7f8c540c4350 2024-04-24T23:16:38.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.127+0000 7f8c4a000640 1 -- 172.21.15.6:0/212475757 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8c3c042070 con 0x7f8c540c4350 2024-04-24T23:16:38.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.127+0000 7f8c4a000640 1 -- 172.21.15.6:0/212475757 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c3c030960 con 0x7f8c540c4350 2024-04-24T23:16:38.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.127+0000 7f8c58a00640 1 -- 172.21.15.6:0/212475757 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c541f10b0 con 0x7f8c540c4350 2024-04-24T23:16:38.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.128+0000 7f8c58a00640 1 -- 172.21.15.6:0/212475757 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c541f1670 con 0x7f8c540c4350 2024-04-24T23:16:38.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.131+0000 7f8c4a000640 1 -- 172.21.15.6:0/212475757 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f8c3c030b00 con 0x7f8c540c4350 2024-04-24T23:16:38.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.131+0000 7f8c4a000640 1 --2- 172.21.15.6:0/212475757 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f8c3006be30 0x7f8c3006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:38.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.132+0000 7f8c4a000640 1 -- 172.21.15.6:0/212475757 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6590+0+0 (secure 0 0 0) 0x7f8c3c0abb00 con 0x7f8c540c4350 2024-04-24T23:16:38.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.132+0000 7f8c5b82b740 1 Processor -- start 2024-04-24T23:16:38.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.132+0000 7f8c51e00640 1 --2- 172.21.15.6:0/212475757 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f8c3006be30 0x7f8c3006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:38.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.132+0000 7f8c5b82b740 1 -- start start 2024-04-24T23:16:38.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.132+0000 7f8c51e00640 1 --2- 172.21.15.6:0/212475757 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f8c3006be30 0x7f8c3006e2f0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f8c440012e0 tx=0x7f8c44001700 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:38.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.133+0000 7f8c5b82b740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55baef87e060 0x55baef899900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:38.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.133+0000 7f8c5b82b740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55baef850bf0 0x55baef852b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:38.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.133+0000 7f8c5b82b740 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x55baef853040 0x55baef87e440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:38.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.133+0000 7f8c5b82b740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x55baef754ed0 con 0x55baef87e060 2024-04-24T23:16:38.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.133+0000 7f8c5b82b740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55baef7b7df0 con 0x55baef850bf0 2024-04-24T23:16:38.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.133+0000 7f8c5b82b740 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x55baef7608d0 con 0x55baef853040 2024-04-24T23:16:38.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.133+0000 7f8c51e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55baef87e060 0x55baef899900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:38.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.133+0000 7f8c51400640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55baef850bf0 0x55baef852b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:38.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.133+0000 7f8c51e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55baef87e060 0x55baef899900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:35636/0 (socket says 172.21.15.6:35636) 2024-04-24T23:16:38.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.133+0000 7f8c51e00640 1 -- 172.21.15.6:0/1511789876 learned_addr learned my addr 172.21.15.6:0/1511789876 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:16:38.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.133+0000 7f8c52800640 1 --2- 172.21.15.6:0/1511789876 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x55baef853040 0x55baef87e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:38.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.134+0000 7f8c51400640 1 -- 172.21.15.6:0/1511789876 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x55baef853040 msgr2=0x55baef87e440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:16:38.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.134+0000 7f8c51400640 1 --2- 172.21.15.6:0/1511789876 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x55baef853040 0x55baef87e440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:38.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.134+0000 7f8c51400640 1 -- 172.21.15.6:0/1511789876 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55baef87e060 msgr2=0x55baef899900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:16:38.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.134+0000 7f8c51400640 1 --2- 172.21.15.6:0/1511789876 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55baef87e060 0x55baef899900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:38.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.134+0000 7f8c51400640 1 -- 172.21.15.6:0/1511789876 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55baef4cc2d0 con 0x55baef850bf0 2024-04-24T23:16:38.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.134+0000 7f8c51e00640 1 --2- 172.21.15.6:0/1511789876 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55baef87e060 0x55baef899900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:16:38.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.134+0000 7f8c51400640 1 --2- 172.21.15.6:0/1511789876 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55baef850bf0 0x55baef852b00 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f8c3c079950 tx=0x7f8c3c0795c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b0f7a06fdeb07317 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:38.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.134+0000 7f8c42600640 1 -- 172.21.15.6:0/1511789876 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c3c0a8070 con 0x55baef850bf0 2024-04-24T23:16:38.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.134+0000 7f8c42600640 1 -- 172.21.15.6:0/1511789876 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8c3c07dcc0 con 0x55baef850bf0 2024-04-24T23:16:38.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.134+0000 7f8c42600640 1 -- 172.21.15.6:0/1511789876 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c3c082580 con 0x55baef850bf0 2024-04-24T23:16:38.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.135+0000 7f8c5b82b740 1 -- 172.21.15.6:0/1511789876 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55baef850bf0 msgr2=0x55baef852b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:38.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.135+0000 7f8c5b82b740 1 --2- 172.21.15.6:0/1511789876 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55baef850bf0 0x55baef852b00 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f8c3c079950 tx=0x7f8c3c0795c0 comp rx=0 tx=0).stop 2024-04-24T23:16:38.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.135+0000 7f8c5b82b740 1 -- 172.21.15.6:0/1511789876 shutdown_connections 2024-04-24T23:16:38.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.135+0000 7f8c5b82b740 1 --2- 172.21.15.6:0/1511789876 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x55baef853040 0x55baef87e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:38.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.135+0000 7f8c5b82b740 1 --2- 172.21.15.6:0/1511789876 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55baef850bf0 0x55baef852b00 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:38.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.135+0000 7f8c5b82b740 1 --2- 172.21.15.6:0/1511789876 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55baef87e060 0x55baef899900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:38.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.135+0000 7f8c5b82b740 1 -- 172.21.15.6:0/1511789876 >> 172.21.15.6:0/1511789876 conn(0x55baef851a80 msgr2=0x55baef898f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:38.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.135+0000 7f8c5b82b740 1 -- 172.21.15.6:0/1511789876 shutdown_connections 2024-04-24T23:16:38.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.135+0000 7f8c5b82b740 1 -- 172.21.15.6:0/1511789876 wait complete. 2024-04-24T23:16:38.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.136+0000 7f8c5b82b740 1 -- start start 2024-04-24T23:16:38.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.136+0000 7f8c5b82b740 1 Processor -- start 2024-04-24T23:16:38.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.137+0000 7f8c5b82b740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55baef87e060 0x55baef8558a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:38.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.137+0000 7f8c5b82b740 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x55baef8aa090 0x55baef896b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:38.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.137+0000 7f8c5b82b740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55baef876310 0x55baef8912a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:38.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.137+0000 7f8c5b82b740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x55baef773d20 con 0x55baef87e060 2024-04-24T23:16:38.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.137+0000 7f8c5b82b740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55baef7b3db0 con 0x55baef876310 2024-04-24T23:16:38.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.137+0000 7f8c5b82b740 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x55baef7b7df0 con 0x55baef8aa090 2024-04-24T23:16:38.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.137+0000 7f8c5b82b740 10 client.0 _ll_register_callbacks cb 0x55baef7aef10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T23:16:38.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.137+0000 7f8c51e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55baef87e060 0x55baef8558a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:38.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.137+0000 7f8c51e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55baef87e060 0x55baef8558a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:35646/0 (socket says 172.21.15.6:35646) 2024-04-24T23:16:38.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.137+0000 7f8c51e00640 1 -- 172.21.15.6:0/1648612086 learned_addr learned my addr 172.21.15.6:0/1648612086 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:16:38.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.137+0000 7f8c51400640 1 --2- 172.21.15.6:0/1648612086 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x55baef8aa090 0x55baef896b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:38.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.137+0000 7f8c52800640 1 --2- 172.21.15.6:0/1648612086 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55baef876310 0x55baef8912a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:38.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.137+0000 7f8c51e00640 1 -- 172.21.15.6:0/1648612086 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x55baef8aa090 msgr2=0x55baef896b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:16:38.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.137+0000 7f8c51e00640 1 --2- 172.21.15.6:0/1648612086 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x55baef8aa090 0x55baef896b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:38.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.138+0000 7f8c51e00640 1 -- 172.21.15.6:0/1648612086 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55baef876310 msgr2=0x55baef8912a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:16:38.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.138+0000 7f8c51e00640 1 --2- 172.21.15.6:0/1648612086 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55baef876310 0x55baef8912a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:38.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.138+0000 7f8c51e00640 1 -- 172.21.15.6:0/1648612086 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55baef5966b0 con 0x55baef87e060 2024-04-24T23:16:38.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.138+0000 7f8c51e00640 1 --2- 172.21.15.6:0/1648612086 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55baef87e060 0x55baef8558a0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f8c44013a20 tx=0x7f8c44001250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a458290c1e7fb73c server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:38.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.138+0000 7f8c36a00640 10 client.0 ms_handle_connect on v2:172.21.15.6:3300/0 2024-04-24T23:16:38.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.138+0000 7f8c36a00640 1 -- 172.21.15.6:0/1648612086 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c44012ae0 con 0x55baef87e060 2024-04-24T23:16:38.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.138+0000 7f8c36a00640 1 -- 172.21.15.6:0/1648612086 <== mon.0 v2:172.21.15.6:3300/0 2 ==== osd_map(26..26 src has 1..26) ==== 6590+0+0 (secure 0 0 0) 0x7f8c44012c80 con 0x55baef87e060 2024-04-24T23:16:38.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.138+0000 7f8c5b82b740 1 -- 172.21.15.6:0/1648612086 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8c3c07dcc0 con 0x55baef87e060 2024-04-24T23:16:38.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.138+0000 7f8c36a00640 1 -- 172.21.15.6:0/1648612086 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_get_version_reply(handle=1 version=18) ==== 24+0+0 (secure 0 0 0) 0x7f8c4402e050 con 0x55baef87e060 2024-04-24T23:16:38.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.138+0000 7f8c5b82b740 10 client.4836 fetch_fsmap learned FSMap version 18 2024-04-24T23:16:38.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.138+0000 7f8c5b82b740 1 -- 172.21.15.6:0/1648612086 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,fsmap=18,monmap=2+}) -- 0x55baef895d90 con 0x55baef87e060 2024-04-24T23:16:38.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.139+0000 7f8c36a00640 1 -- 172.21.15.6:0/1648612086 <== mon.0 v2:172.21.15.6:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8c3c07dcc0 con 0x55baef87e060 2024-04-24T23:16:38.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.139+0000 7f8c36a00640 1 -- 172.21.15.6:0/1648612086 <== mon.0 v2:172.21.15.6:3300/0 5 ==== fsmap(e 18) ==== 4792+0+0 (secure 0 0 0) 0x7f8c44031020 con 0x55baef87e060 2024-04-24T23:16:38.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.139+0000 7f8c5b82b740 10 client.4836 fetch_fsmap finished waiting for FSMap version 18 2024-04-24T23:16:38.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.139+0000 7f8c5b82b740 10 client.4836 resolve_mds: resolved name 'f' to daemon mds.f 2024-04-24T23:16:38.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.139+0000 7f8c5b82b740 20 client.4836 populate_metadata read hostname 'smithi006' 2024-04-24T23:16:38.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.139+0000 7f8c5b82b740 1 --2- 172.21.15.6:0/1648612086 >> [v2:172.21.15.190:6834/580315269,v1:172.21.15.190:6837/580315269] conn(0x55baef8975a0 0x55baef8b54a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:38.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.139+0000 7f8c5b82b740 4 client.4836 mds_command: new command op to 4282 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T23:16:38.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.139+0000 7f8c5b82b740 1 -- 172.21.15.6:0/1648612086 --> [v2:172.21.15.190:6834/580315269,v1:172.21.15.190:6837/580315269] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55baef7c5330 con 0x55baef8975a0 2024-04-24T23:16:38.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.139+0000 7f8c51400640 1 --2- 172.21.15.6:0/1648612086 >> [v2:172.21.15.190:6834/580315269,v1:172.21.15.190:6837/580315269] conn(0x55baef8975a0 0x55baef8b54a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:38.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.140+0000 7f8c51400640 1 --2- 172.21.15.6:0/1648612086 >> [v2:172.21.15.190:6834/580315269,v1:172.21.15.190:6837/580315269] conn(0x55baef8975a0 0x55baef8b54a0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=398fc7cdb6cee9e4 server_cookie=606f6bc5e45ce2c0 in_seq=0 out_seq=0 2024-04-24T23:16:38.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.140+0000 7f8c36a00640 10 client.4836 ms_handle_connect on v2:172.21.15.190:6834/580315269 2024-04-24T23:16:38.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.140+0000 7f8c36a00640 1 -- 172.21.15.6:0/1648612086 <== mds.1 v2:172.21.15.190:6834/580315269 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55baef7c5330 con 0x55baef8975a0 2024-04-24T23:16:38.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.140+0000 7f8c36a00640 10 client.4836 handle_command_reply: tid=0 2024-04-24T23:16:38.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.141+0000 7f8c5b82b740 1 client.4836 shutdown 2024-04-24T23:16:38.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.141+0000 7f8c5b82b740 1 -- 172.21.15.6:0/1648612086 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55baef87e060 msgr2=0x55baef8558a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:38.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.141+0000 7f8c5b82b740 1 --2- 172.21.15.6:0/1648612086 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55baef87e060 0x55baef8558a0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f8c44013a20 tx=0x7f8c44001250 comp rx=0 tx=0).stop 2024-04-24T23:16:38.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.141+0000 7f8c5b82b740 1 -- 172.21.15.6:0/1648612086 shutdown_connections 2024-04-24T23:16:38.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.141+0000 7f8c5b82b740 1 --2- 172.21.15.6:0/1648612086 >> [v2:172.21.15.190:6834/580315269,v1:172.21.15.190:6837/580315269] conn(0x55baef8975a0 0x55baef8b54a0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:38.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.141+0000 7f8c5b82b740 1 --2- 172.21.15.6:0/1648612086 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x55baef8aa090 0x55baef896b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:38.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.141+0000 7f8c5b82b740 1 --2- 172.21.15.6:0/1648612086 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55baef876310 0x55baef8912a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:38.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.141+0000 7f8c5b82b740 1 --2- 172.21.15.6:0/1648612086 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55baef87e060 0x55baef8558a0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:38.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.141+0000 7f8c36a00640 0 client.4836 ms_handle_reset on v2:172.21.15.190:6834/580315269 2024-04-24T23:16:38.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.141+0000 7f8c5b82b740 1 -- 172.21.15.6:0/1648612086 >> 172.21.15.6:0/1648612086 conn(0x55baef853040 msgr2=0x55baef87c830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:38.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.141+0000 7f8c5b82b740 1 -- 172.21.15.6:0/1648612086 shutdown_connections 2024-04-24T23:16:38.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.141+0000 7f8c5b82b740 1 -- 172.21.15.6:0/1648612086 wait complete. 2024-04-24T23:16:38.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.141+0000 7f8c5b82b740 20 client.4836 trim_cache size 0 max 16384 2024-04-24T23:16:38.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.161+0000 7f8c5b82b740 1 Processor -- start 2024-04-24T23:16:38.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.161+0000 7f8c5b82b740 1 -- start start 2024-04-24T23:16:38.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.162+0000 7f8c5b82b740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55baef876310 0x55baef8af960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:38.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.162+0000 7f8c5b82b740 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x55baef87e060 0x55baef87e440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:38.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.162+0000 7f8c5b82b740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55baef8aa090 0x55baef87e980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:38.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.162+0000 7f8c5b82b740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f8c540ccde0 con 0x55baef876310 2024-04-24T23:16:38.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.162+0000 7f8c5b82b740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55baef8a9c00 con 0x55baef8aa090 2024-04-24T23:16:38.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.162+0000 7f8c5b82b740 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x55baef8a9930 con 0x55baef87e060 2024-04-24T23:16:38.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.162+0000 7f8c51e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55baef876310 0x55baef8af960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:38.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.162+0000 7f8c51e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55baef876310 0x55baef8af960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:35660/0 (socket says 172.21.15.6:35660) 2024-04-24T23:16:38.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.162+0000 7f8c51e00640 1 -- 172.21.15.6:0/177330607 learned_addr learned my addr 172.21.15.6:0/177330607 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:16:38.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.162+0000 7f8c51400640 1 --2- 172.21.15.6:0/177330607 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x55baef87e060 0x55baef87e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:38.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.163+0000 7f8c52800640 1 --2- 172.21.15.6:0/177330607 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55baef8aa090 0x55baef87e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:38.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.163+0000 7f8c51e00640 1 -- 172.21.15.6:0/177330607 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x55baef87e060 msgr2=0x55baef87e440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:16:38.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.163+0000 7f8c51e00640 1 --2- 172.21.15.6:0/177330607 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x55baef87e060 0x55baef87e440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:38.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.163+0000 7f8c51e00640 1 -- 172.21.15.6:0/177330607 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55baef8aa090 msgr2=0x55baef87e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:16:38.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.163+0000 7f8c51e00640 1 --2- 172.21.15.6:0/177330607 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55baef8aa090 0x55baef87e980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:38.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.163+0000 7f8c51e00640 1 -- 172.21.15.6:0/177330607 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55baef855c10 con 0x55baef876310 2024-04-24T23:16:38.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.163+0000 7f8c52800640 1 --2- 172.21.15.6:0/177330607 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55baef8aa090 0x55baef87e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:16:38.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.163+0000 7f8c51e00640 1 --2- 172.21.15.6:0/177330607 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55baef876310 0x55baef8af960 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f8c44001390 tx=0x7f8c44021000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d233ca7dc39a6a87 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:38.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.163+0000 7f8c41c00640 1 -- 172.21.15.6:0/177330607 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c440136d0 con 0x55baef876310 2024-04-24T23:16:38.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.163+0000 7f8c41c00640 1 -- 172.21.15.6:0/177330607 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8c44026070 con 0x55baef876310 2024-04-24T23:16:38.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.163+0000 7f8c41c00640 1 -- 172.21.15.6:0/177330607 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c44012ce0 con 0x55baef876310 2024-04-24T23:16:38.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.164+0000 7f8c5b82b740 1 -- 172.21.15.6:0/177330607 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55baef876310 msgr2=0x55baef8af960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:38.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.164+0000 7f8c5b82b740 1 --2- 172.21.15.6:0/177330607 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55baef876310 0x55baef8af960 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f8c44001390 tx=0x7f8c44021000 comp rx=0 tx=0).stop 2024-04-24T23:16:38.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.164+0000 7f8c5b82b740 1 -- 172.21.15.6:0/177330607 shutdown_connections 2024-04-24T23:16:38.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.164+0000 7f8c5b82b740 1 --2- 172.21.15.6:0/177330607 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x55baef87e060 0x55baef87e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:38.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.164+0000 7f8c5b82b740 1 --2- 172.21.15.6:0/177330607 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55baef8aa090 0x55baef87e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:38.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.164+0000 7f8c5b82b740 1 --2- 172.21.15.6:0/177330607 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55baef876310 0x55baef8af960 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:38.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.164+0000 7f8c5b82b740 1 -- 172.21.15.6:0/177330607 >> 172.21.15.6:0/177330607 conn(0x55baef853040 msgr2=0x55baef84ab30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:38.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.164+0000 7f8c5b82b740 1 -- 172.21.15.6:0/177330607 shutdown_connections 2024-04-24T23:16:38.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.164+0000 7f8c5b82b740 1 -- 172.21.15.6:0/177330607 wait complete. 2024-04-24T23:16:38.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.165+0000 7f8c5b82b740 1 -- start start 2024-04-24T23:16:38.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.165+0000 7f8c5b82b740 1 Processor -- start 2024-04-24T23:16:38.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.166+0000 7f8c5b82b740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55baef876310 0x55baef87ce80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:38.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.166+0000 7f8c5b82b740 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x55baef87e060 0x55baef8baec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:38.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.166+0000 7f8c5b82b740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55baef8aa090 0x55baef8501a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:38.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.166+0000 7f8c5b82b740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x55baef874c80 con 0x55baef876310 2024-04-24T23:16:38.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.166+0000 7f8c5b82b740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55baef897090 con 0x55baef8aa090 2024-04-24T23:16:38.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.166+0000 7f8c5b82b740 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x55baef875300 con 0x55baef87e060 2024-04-24T23:16:38.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.166+0000 7f8c5b82b740 10 client.0 _ll_register_callbacks cb 0x7f8c44007650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T23:16:38.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.166+0000 7f8c51e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55baef876310 0x55baef87ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:38.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.166+0000 7f8c51e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55baef876310 0x55baef87ce80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:35666/0 (socket says 172.21.15.6:35666) 2024-04-24T23:16:38.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.166+0000 7f8c51e00640 1 -- 172.21.15.6:0/3515061191 learned_addr learned my addr 172.21.15.6:0/3515061191 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:16:38.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.166+0000 7f8c51400640 1 --2- 172.21.15.6:0/3515061191 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x55baef87e060 0x55baef8baec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:38.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.166+0000 7f8c52800640 1 --2- 172.21.15.6:0/3515061191 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55baef8aa090 0x55baef8501a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:38.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.167+0000 7f8c51e00640 1 -- 172.21.15.6:0/3515061191 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x55baef87e060 msgr2=0x55baef8baec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:16:38.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.167+0000 7f8c51e00640 1 --2- 172.21.15.6:0/3515061191 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x55baef87e060 0x55baef8baec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:38.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.167+0000 7f8c51e00640 1 -- 172.21.15.6:0/3515061191 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55baef8aa090 msgr2=0x55baef8501a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:16:38.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.167+0000 7f8c51e00640 1 --2- 172.21.15.6:0/3515061191 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55baef8aa090 0x55baef8501a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:38.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.167+0000 7f8c51e00640 1 -- 172.21.15.6:0/3515061191 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55baef87ebe0 con 0x55baef876310 2024-04-24T23:16:38.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.167+0000 7f8c52800640 1 --2- 172.21.15.6:0/3515061191 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55baef8aa090 0x55baef8501a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:16:38.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.167+0000 7f8c51e00640 1 --2- 172.21.15.6:0/3515061191 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55baef876310 0x55baef87ce80 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f8c4402b010 tx=0x7f8c4400a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d96376153c309231 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:38.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.167+0000 7f8c36a00640 10 client.0 ms_handle_connect on v2:172.21.15.6:3300/0 2024-04-24T23:16:38.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.167+0000 7f8c36a00640 1 -- 172.21.15.6:0/3515061191 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c44012720 con 0x55baef876310 2024-04-24T23:16:38.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.168+0000 7f8c36a00640 1 -- 172.21.15.6:0/3515061191 <== mon.0 v2:172.21.15.6:3300/0 2 ==== osd_map(26..26 src has 1..26) ==== 6590+0+0 (secure 0 0 0) 0x7f8c44023b30 con 0x55baef876310 2024-04-24T23:16:38.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.168+0000 7f8c5b82b740 1 -- 172.21.15.6:0/3515061191 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8c44026070 con 0x55baef876310 2024-04-24T23:16:38.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.168+0000 7f8c36a00640 1 -- 172.21.15.6:0/3515061191 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_get_version_reply(handle=1 version=18) ==== 24+0+0 (secure 0 0 0) 0x7f8c4404b050 con 0x55baef876310 2024-04-24T23:16:38.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.168+0000 7f8c5b82b740 10 client.4842 fetch_fsmap learned FSMap version 18 2024-04-24T23:16:38.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.168+0000 7f8c5b82b740 1 -- 172.21.15.6:0/3515061191 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,fsmap=18,monmap=2+}) -- 0x55baef84f1a0 con 0x55baef876310 2024-04-24T23:16:38.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.168+0000 7f8c36a00640 1 -- 172.21.15.6:0/3515061191 <== mon.0 v2:172.21.15.6:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8c44026070 con 0x55baef876310 2024-04-24T23:16:38.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.168+0000 7f8c36a00640 1 -- 172.21.15.6:0/3515061191 <== mon.0 v2:172.21.15.6:3300/0 5 ==== fsmap(e 18) ==== 4792+0+0 (secure 0 0 0) 0x7f8c4404e020 con 0x55baef876310 2024-04-24T23:16:38.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.168+0000 7f8c5b82b740 10 client.4842 fetch_fsmap finished waiting for FSMap version 18 2024-04-24T23:16:38.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.168+0000 7f8c5b82b740 10 client.4842 resolve_mds: resolved name 'f' to daemon mds.f 2024-04-24T23:16:38.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.168+0000 7f8c5b82b740 20 client.4842 populate_metadata read hostname 'smithi006' 2024-04-24T23:16:38.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.168+0000 7f8c5b82b740 1 --2- 172.21.15.6:0/3515061191 >> [v2:172.21.15.190:6834/580315269,v1:172.21.15.190:6837/580315269] conn(0x55baef8984f0 0x55baef91d640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:38.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.168+0000 7f8c5b82b740 4 client.4842 mds_command: new command op to 4282 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-24T23:16:38.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.168+0000 7f8c5b82b740 1 -- 172.21.15.6:0/3515061191 --> [v2:172.21.15.190:6834/580315269,v1:172.21.15.190:6837/580315269] -- command(tid 0: {"prefix": "status"}) -- 0x55baef7bf680 con 0x55baef8984f0 2024-04-24T23:16:38.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.169+0000 7f8c51400640 1 --2- 172.21.15.6:0/3515061191 >> [v2:172.21.15.190:6834/580315269,v1:172.21.15.190:6837/580315269] conn(0x55baef8984f0 0x55baef91d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:38.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.169+0000 7f8c51400640 1 --2- 172.21.15.6:0/3515061191 >> [v2:172.21.15.190:6834/580315269,v1:172.21.15.190:6837/580315269] conn(0x55baef8984f0 0x55baef91d640 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=50e742a039f27bd5 server_cookie=efb2aedcb925293e in_seq=0 out_seq=0 2024-04-24T23:16:38.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.169+0000 7f8c36a00640 10 client.4842 ms_handle_connect on v2:172.21.15.190:6834/580315269 2024-04-24T23:16:38.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.169+0000 7f8c36a00640 1 -- 172.21.15.6:0/3515061191 <== mds.1 v2:172.21.15.190:6834/580315269 1 ==== command_reply(tid 0: 0 ) ==== 8+0+331 (crc 0 0 0) 0x55baef7bf680 con 0x55baef8984f0 2024-04-24T23:16:38.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.169+0000 7f8c36a00640 10 client.4842 handle_command_reply: tid=0 2024-04-24T23:16:38.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.169+0000 7f8c5b82b740 1 client.4842 shutdown 2024-04-24T23:16:38.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.170+0000 7f8c5b82b740 1 -- 172.21.15.6:0/3515061191 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55baef876310 msgr2=0x55baef87ce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:38.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.170+0000 7f8c5b82b740 1 --2- 172.21.15.6:0/3515061191 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55baef876310 0x55baef87ce80 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f8c4402b010 tx=0x7f8c4400a000 comp rx=0 tx=0).stop 2024-04-24T23:16:38.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.170+0000 7f8c5b82b740 1 -- 172.21.15.6:0/3515061191 shutdown_connections 2024-04-24T23:16:38.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.170+0000 7f8c5b82b740 1 --2- 172.21.15.6:0/3515061191 >> [v2:172.21.15.190:6834/580315269,v1:172.21.15.190:6837/580315269] conn(0x55baef8984f0 0x55baef91d640 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:38.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.170+0000 7f8c5b82b740 1 --2- 172.21.15.6:0/3515061191 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x55baef87e060 0x55baef8baec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:38.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.170+0000 7f8c5b82b740 1 --2- 172.21.15.6:0/3515061191 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55baef8aa090 0x55baef8501a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:38.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.170+0000 7f8c5b82b740 1 --2- 172.21.15.6:0/3515061191 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55baef876310 0x55baef87ce80 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:38.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.170+0000 7f8c36a00640 0 client.4842 ms_handle_reset on v2:172.21.15.190:6834/580315269 2024-04-24T23:16:38.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.170+0000 7f8c5b82b740 1 -- 172.21.15.6:0/3515061191 >> 172.21.15.6:0/3515061191 conn(0x55baef853040 msgr2=0x55baef8785b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:38.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.170+0000 7f8c5b82b740 1 -- 172.21.15.6:0/3515061191 shutdown_connections 2024-04-24T23:16:38.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.170+0000 7f8c5b82b740 1 -- 172.21.15.6:0/3515061191 wait complete. 2024-04-24T23:16:38.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.170+0000 7f8c5b82b740 20 client.4842 trim_cache size 0 max 16384 2024-04-24T23:16:38.176 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-04-24T23:16:38.176 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_fsid": "18bef68d-5e6d-4ec5-862b-b6f206e2588d", 2024-04-24T23:16:38.176 INFO:teuthology.orchestra.run.smithi006.stdout: "whoami": 1, 2024-04-24T23:16:38.176 INFO:teuthology.orchestra.run.smithi006.stdout: "id": 4282, 2024-04-24T23:16:38.176 INFO:teuthology.orchestra.run.smithi006.stdout: "want_state": "up:active", 2024-04-24T23:16:38.176 INFO:teuthology.orchestra.run.smithi006.stdout: "state": "up:active", 2024-04-24T23:16:38.176 INFO:teuthology.orchestra.run.smithi006.stdout: "fs_name": "cephfs", 2024-04-24T23:16:38.176 INFO:teuthology.orchestra.run.smithi006.stdout: "rank_uptime": 30.019375630999999, 2024-04-24T23:16:38.176 INFO:teuthology.orchestra.run.smithi006.stdout: "mdsmap_epoch": 18, 2024-04-24T23:16:38.176 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap_epoch": 26, 2024-04-24T23:16:38.177 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap_epoch_barrier": 26, 2024-04-24T23:16:38.177 INFO:teuthology.orchestra.run.smithi006.stdout: "uptime": 67.296486841999993 2024-04-24T23:16:38.177 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-04-24T23:16:38.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.171+0000 7f8c58a00640 1 -- 172.21.15.6:0/212475757 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f8c3006be30 msgr2=0x7f8c3006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:38.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.171+0000 7f8c58a00640 1 --2- 172.21.15.6:0/212475757 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f8c3006be30 0x7f8c3006e2f0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f8c440012e0 tx=0x7f8c44001700 comp rx=0 tx=0).stop 2024-04-24T23:16:38.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.171+0000 7f8c58a00640 1 -- 172.21.15.6:0/212475757 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8c540c4350 msgr2=0x7f8c54146840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:38.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.171+0000 7f8c58a00640 1 --2- 172.21.15.6:0/212475757 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8c540c4350 0x7f8c54146840 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f8c3c000c00 tx=0x7f8c3c0046c0 comp rx=0 tx=0).stop 2024-04-24T23:16:38.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.171+0000 7f8c58a00640 1 -- 172.21.15.6:0/212475757 shutdown_connections 2024-04-24T23:16:38.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.171+0000 7f8c58a00640 1 --2- 172.21.15.6:0/212475757 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f8c3006be30 0x7f8c3006e2f0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:38.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.171+0000 7f8c58a00640 1 --2- 172.21.15.6:0/212475757 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8c54151c00 0x7f8c54146d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:38.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.171+0000 7f8c58a00640 1 --2- 172.21.15.6:0/212475757 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8c540c4350 0x7f8c54146840 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:38.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.171+0000 7f8c58a00640 1 --2- 172.21.15.6:0/212475757 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8c540733e0 0x7f8c541e5330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:38.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.171+0000 7f8c58a00640 1 -- 172.21.15.6:0/212475757 >> 172.21.15.6:0/212475757 conn(0x7f8c540670e0 msgr2=0x7f8c540cd470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:38.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.171+0000 7f8c58a00640 1 -- 172.21.15.6:0/212475757 shutdown_connections 2024-04-24T23:16:38.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.171+0000 7f8c58a00640 1 -- 172.21.15.6:0/212475757 wait complete. 2024-04-24T23:16:38.188 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-04-24T23:16:38.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.390+0000 7f0b9ec00640 1 Processor -- start 2024-04-24T23:16:38.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.390+0000 7f0b9ec00640 1 -- start start 2024-04-24T23:16:38.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.392+0000 7f0b9ec00640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f0b980b0e50 0x7f0b980c3930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:38.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.392+0000 7f0b9ec00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0b980cb5f0 con 0x7f0b9815c1b0 2024-04-24T23:16:38.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.392+0000 7f0b9ec00640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0b980cbf20 con 0x7f0b980cad80 2024-04-24T23:16:38.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.392+0000 7f0b9ec00640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f0b9815c5e0 con 0x7f0b980b0e50 2024-04-24T23:16:38.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.392+0000 7f0b97400640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f0b980cad80 0x7f0b9806ece0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.6:57864/0 (socket says 172.21.15.6:57864) 2024-04-24T23:16:38.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.392+0000 7f0b97400640 1 -- 172.21.15.6:0/3233810248 learned_addr learned my addr 172.21.15.6:0/3233810248 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:16:38.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.392+0000 7f0b97e00640 1 --2- 172.21.15.6:0/3233810248 >> v2:172.21.15.190:3300/0 conn(0x7f0b980b0e50 0x7f0b980c3930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:38.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.393+0000 7f0b96a00640 1 -- 172.21.15.6:0/3233810248 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 83954295 0 0) 0x7f0b980cbf20 con 0x7f0b980cad80 2024-04-24T23:16:38.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.393+0000 7f0b96a00640 1 -- 172.21.15.6:0/3233810248 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0b7c003520 con 0x7f0b980cad80 2024-04-24T23:16:38.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.393+0000 7f0b97e00640 1 -- 172.21.15.6:0/3233810248 >> v1:172.21.15.190:6789/0 conn(0x7f0b980cad80 legacy=0x7f0b9806ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:38.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.393+0000 7f0b97e00640 1 -- 172.21.15.6:0/3233810248 >> v1:172.21.15.6:6789/0 conn(0x7f0b9815c1b0 legacy=0x7f0b981634e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:38.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.393+0000 7f0b97e00640 1 -- 172.21.15.6:0/3233810248 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b980c5460 con 0x7f0b980b0e50 2024-04-24T23:16:38.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.394+0000 7f0b97e00640 1 --2- 172.21.15.6:0/3233810248 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0b980b0e50 0x7f0b980c3930 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f0b88009d90 tx=0x7f0b8802a780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d8042ceb9434eca4 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:38.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.394+0000 7f0b96a00640 1 -- 172.21.15.6:0/3233810248 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b88037070 con 0x7f0b980b0e50 2024-04-24T23:16:38.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.394+0000 7f0b96a00640 1 -- 172.21.15.6:0/3233810248 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b88007740 con 0x7f0b980b0e50 2024-04-24T23:16:38.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.394+0000 7f0b96a00640 1 -- 172.21.15.6:0/3233810248 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b88007b10 con 0x7f0b980b0e50 2024-04-24T23:16:38.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.395+0000 7f0b9ec00640 1 -- 172.21.15.6:0/3233810248 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0b980b0e50 msgr2=0x7f0b980c3930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:38.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.395+0000 7f0b9ec00640 1 --2- 172.21.15.6:0/3233810248 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0b980b0e50 0x7f0b980c3930 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f0b88009d90 tx=0x7f0b8802a780 comp rx=0 tx=0).stop 2024-04-24T23:16:38.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.395+0000 7f0b9ec00640 1 -- 172.21.15.6:0/3233810248 shutdown_connections 2024-04-24T23:16:38.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.395+0000 7f0b9ec00640 1 --2- 172.21.15.6:0/3233810248 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0b980b0e50 0x7f0b980c3930 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:38.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.395+0000 7f0b9ec00640 1 -- 172.21.15.6:0/3233810248 >> 172.21.15.6:0/3233810248 conn(0x7f0b9806fdc0 msgr2=0x7f0b980701d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:38.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.395+0000 7f0b9ec00640 1 -- 172.21.15.6:0/3233810248 shutdown_connections 2024-04-24T23:16:38.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.395+0000 7f0b9ec00640 1 -- 172.21.15.6:0/3233810248 wait complete. 2024-04-24T23:16:38.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.397+0000 7f0b9ec00640 1 Processor -- start 2024-04-24T23:16:38.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.397+0000 7f0b9ec00640 1 -- start start 2024-04-24T23:16:38.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.398+0000 7f0b9ec00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0b980b0e50 0x7f0b981e78f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:38.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.398+0000 7f0b9ec00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0b980cad80 0x7f0b981f16a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:38.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.398+0000 7f0b9ec00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0b9815c1b0 0x7f0b981f1be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:38.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.398+0000 7f0b9ec00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f0b980c92b0 con 0x7f0b980cad80 2024-04-24T23:16:38.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.398+0000 7f0b9ec00640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f0b980c8fd0 con 0x7f0b9815c1b0 2024-04-24T23:16:38.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.398+0000 7f0b9ec00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f0b980b1230 con 0x7f0b980b0e50 2024-04-24T23:16:38.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.398+0000 7f0b97400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0b980cad80 0x7f0b981f16a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:38.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.398+0000 7f0b97e00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0b980b0e50 0x7f0b981e78f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:38.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.398+0000 7f0b97400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0b980cad80 0x7f0b981f16a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:35680/0 (socket says 172.21.15.6:35680) 2024-04-24T23:16:38.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.398+0000 7f0b97400640 1 -- 172.21.15.6:0/2198079390 learned_addr learned my addr 172.21.15.6:0/2198079390 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:16:38.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.398+0000 7f0b9cc00640 1 --2- 172.21.15.6:0/2198079390 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0b9815c1b0 0x7f0b981f1be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:38.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.399+0000 7f0b97e00640 1 -- 172.21.15.6:0/2198079390 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0b9815c1b0 msgr2=0x7f0b981f1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:38.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.399+0000 7f0b97e00640 1 --2- 172.21.15.6:0/2198079390 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0b9815c1b0 0x7f0b981f1be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:38.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.399+0000 7f0b97e00640 1 -- 172.21.15.6:0/2198079390 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0b980cad80 msgr2=0x7f0b981f16a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:38.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.399+0000 7f0b97e00640 1 --2- 172.21.15.6:0/2198079390 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0b980cad80 0x7f0b981f16a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:38.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.399+0000 7f0b97e00640 1 -- 172.21.15.6:0/2198079390 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b980ccdf0 con 0x7f0b980b0e50 2024-04-24T23:16:38.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.399+0000 7f0b97400640 1 --2- 172.21.15.6:0/2198079390 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0b980cad80 0x7f0b981f16a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:16:38.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.399+0000 7f0b97e00640 1 --2- 172.21.15.6:0/2198079390 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0b980b0e50 0x7f0b981e78f0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f0b9806f9a0 tx=0x7f0b88004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:38.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.399+0000 7f0b8fe00640 1 -- 172.21.15.6:0/2198079390 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b88042020 con 0x7f0b980b0e50 2024-04-24T23:16:38.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.399+0000 7f0b8fe00640 1 -- 172.21.15.6:0/2198079390 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b88032870 con 0x7f0b980b0e50 2024-04-24T23:16:38.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.399+0000 7f0b9ec00640 1 -- 172.21.15.6:0/2198079390 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b980af540 con 0x7f0b980b0e50 2024-04-24T23:16:38.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.399+0000 7f0b8fe00640 1 -- 172.21.15.6:0/2198079390 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b88037040 con 0x7f0b980b0e50 2024-04-24T23:16:38.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.400+0000 7f0b9ec00640 1 -- 172.21.15.6:0/2198079390 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b980cd350 con 0x7f0b980b0e50 2024-04-24T23:16:38.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.402+0000 7f0b8fe00640 1 -- 172.21.15.6:0/2198079390 <== mon.2 v2:172.21.15.190:3301/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f0b88045020 con 0x7f0b980b0e50 2024-04-24T23:16:38.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.402+0000 7f0b8fe00640 1 --2- 172.21.15.6:0/2198079390 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f0b7406be60 0x7f0b7406e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:38.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.403+0000 7f0b8fe00640 1 -- 172.21.15.6:0/2198079390 <== mon.2 v2:172.21.15.190:3301/0 5 ==== osd_map(26..26 src has 1..26) ==== 6590+0+0 (secure 0 0 0) 0x7f0b880ac170 con 0x7f0b980b0e50 2024-04-24T23:16:38.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.403+0000 7f0b97400640 1 --2- 172.21.15.6:0/2198079390 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f0b7406be60 0x7f0b7406e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:38.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.403+0000 7f0ba1a10740 1 Processor -- start 2024-04-24T23:16:38.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.403+0000 7f0ba1a10740 1 -- start start 2024-04-24T23:16:38.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.404+0000 7f0b97400640 1 --2- 172.21.15.6:0/2198079390 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f0b7406be60 0x7f0b7406e320 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f0b98158e70 tx=0x7f0b80009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:38.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.404+0000 7f0ba1a10740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5621626ce060 0x5621626e9900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:38.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.404+0000 7f0ba1a10740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5621626a0bf0 0x5621626a2b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:38.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.404+0000 7f0ba1a10740 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x5621626a3040 0x5621626ce440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:38.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.404+0000 7f0ba1a10740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x5621625a4ed0 con 0x5621626a0bf0 2024-04-24T23:16:38.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.404+0000 7f0ba1a10740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x562162607df0 con 0x5621626ce060 2024-04-24T23:16:38.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.404+0000 7f0ba1a10740 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x5621625b08d0 con 0x5621626a3040 2024-04-24T23:16:38.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.404+0000 7f0b97e00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5621626ce060 0x5621626e9900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:38.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.404+0000 7f0b97e00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5621626ce060 0x5621626e9900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:35444/0 (socket says 172.21.15.6:35444) 2024-04-24T23:16:38.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.404+0000 7f0b97e00640 1 -- 172.21.15.6:0/274378246 learned_addr learned my addr 172.21.15.6:0/274378246 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:16:38.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.404+0000 7f0b97400640 1 --2- 172.21.15.6:0/274378246 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5621626a0bf0 0x5621626a2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:38.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.404+0000 7f0b9cc00640 1 --2- 172.21.15.6:0/274378246 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x5621626a3040 0x5621626ce440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:38.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.405+0000 7f0b97e00640 1 -- 172.21.15.6:0/274378246 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x5621626a3040 msgr2=0x5621626ce440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:16:38.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.405+0000 7f0b97e00640 1 --2- 172.21.15.6:0/274378246 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x5621626a3040 0x5621626ce440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:38.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.405+0000 7f0b97e00640 1 -- 172.21.15.6:0/274378246 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5621626a0bf0 msgr2=0x5621626a2b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:16:38.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.405+0000 7f0b97e00640 1 --2- 172.21.15.6:0/274378246 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5621626a0bf0 0x5621626a2b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:38.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.405+0000 7f0b97e00640 1 -- 172.21.15.6:0/274378246 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56216231c2d0 con 0x5621626ce060 2024-04-24T23:16:38.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.405+0000 7f0b97e00640 1 --2- 172.21.15.6:0/274378246 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5621626ce060 0x5621626e9900 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f0b880826c0 tx=0x7f0b8802a780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=135ea05385656828 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:38.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.405+0000 7f0b87e00640 1 -- 172.21.15.6:0/274378246 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b8807e3c0 con 0x5621626ce060 2024-04-24T23:16:38.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.405+0000 7f0b87e00640 1 -- 172.21.15.6:0/274378246 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b88082bf0 con 0x5621626ce060 2024-04-24T23:16:38.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.405+0000 7f0b87e00640 1 -- 172.21.15.6:0/274378246 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b88076730 con 0x5621626ce060 2024-04-24T23:16:38.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.406+0000 7f0ba1a10740 1 -- 172.21.15.6:0/274378246 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5621626ce060 msgr2=0x5621626e9900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:38.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.406+0000 7f0ba1a10740 1 --2- 172.21.15.6:0/274378246 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5621626ce060 0x5621626e9900 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f0b880826c0 tx=0x7f0b8802a780 comp rx=0 tx=0).stop 2024-04-24T23:16:38.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.406+0000 7f0ba1a10740 1 -- 172.21.15.6:0/274378246 shutdown_connections 2024-04-24T23:16:38.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.406+0000 7f0ba1a10740 1 --2- 172.21.15.6:0/274378246 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x5621626a3040 0x5621626ce440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:38.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.406+0000 7f0ba1a10740 1 --2- 172.21.15.6:0/274378246 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5621626a0bf0 0x5621626a2b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:38.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.406+0000 7f0ba1a10740 1 --2- 172.21.15.6:0/274378246 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5621626ce060 0x5621626e9900 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:38.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.406+0000 7f0ba1a10740 1 -- 172.21.15.6:0/274378246 >> 172.21.15.6:0/274378246 conn(0x5621626a1a80 msgr2=0x5621626e8f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:38.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.406+0000 7f0ba1a10740 1 -- 172.21.15.6:0/274378246 shutdown_connections 2024-04-24T23:16:38.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.406+0000 7f0ba1a10740 1 -- 172.21.15.6:0/274378246 wait complete. 2024-04-24T23:16:38.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.407+0000 7f0ba1a10740 1 -- start start 2024-04-24T23:16:38.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.407+0000 7f0ba1a10740 1 Processor -- start 2024-04-24T23:16:38.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.408+0000 7f0ba1a10740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5621626ce060 0x5621626c6830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:38.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.408+0000 7f0ba1a10740 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x5621626c7800 0x5621626e4570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:38.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.408+0000 7f0ba1a10740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5621626c8130 0x5621626f7b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:38.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.408+0000 7f0ba1a10740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x5621625c3d20 con 0x5621626c8130 2024-04-24T23:16:38.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.408+0000 7f0ba1a10740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x562162603db0 con 0x5621626ce060 2024-04-24T23:16:38.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.408+0000 7f0ba1a10740 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x5621626a1080 con 0x5621626c7800 2024-04-24T23:16:38.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.408+0000 7f0ba1a10740 10 client.0 _ll_register_callbacks cb 0x5621625fef10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T23:16:38.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.408+0000 7f0b97400640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x5621626c7800 0x5621626e4570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:38.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.408+0000 7f0b97e00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5621626ce060 0x5621626c6830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:38.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.408+0000 7f0b97400640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x5621626c7800 0x5621626e4570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.6:42932/0 (socket says 172.21.15.6:42932) 2024-04-24T23:16:38.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.408+0000 7f0b97e00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5621626ce060 0x5621626c6830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:35452/0 (socket says 172.21.15.6:35452) 2024-04-24T23:16:38.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.408+0000 7f0b97400640 1 -- 172.21.15.6:0/416851233 learned_addr learned my addr 172.21.15.6:0/416851233 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:16:38.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.408+0000 7f0b9cc00640 1 --2- 172.21.15.6:0/416851233 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5621626c8130 0x5621626f7b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:38.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.409+0000 7f0b97e00640 1 -- 172.21.15.6:0/416851233 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x5621626c7800 msgr2=0x5621626e4570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:16:38.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.409+0000 7f0b97e00640 1 --2- 172.21.15.6:0/416851233 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x5621626c7800 0x5621626e4570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:38.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.409+0000 7f0b97e00640 1 -- 172.21.15.6:0/416851233 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5621626c8130 msgr2=0x5621626f7b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:16:38.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.409+0000 7f0b97e00640 1 --2- 172.21.15.6:0/416851233 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5621626c8130 0x5621626f7b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:38.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.409+0000 7f0b97e00640 1 -- 172.21.15.6:0/416851233 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5621623e66b0 con 0x5621626ce060 2024-04-24T23:16:38.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.409+0000 7f0b97e00640 1 --2- 172.21.15.6:0/416851233 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5621626ce060 0x5621626c6830 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f0b88009d90 tx=0x7f0b88000f40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fbd043d6dcee7595 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:38.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.409+0000 7f0b84c00640 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2024-04-24T23:16:38.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.409+0000 7f0b84c00640 1 -- 172.21.15.6:0/416851233 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b8807e3c0 con 0x5621626ce060 2024-04-24T23:16:38.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.409+0000 7f0ba1a10740 1 -- 172.21.15.6:0/416851233 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0b88082bf0 con 0x5621626ce060 2024-04-24T23:16:38.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.409+0000 7f0b84c00640 1 -- 172.21.15.6:0/416851233 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(26..26 src has 1..26) ==== 6590+0+0 (secure 0 0 0) 0x7f0b88082760 con 0x5621626ce060 2024-04-24T23:16:38.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.410+0000 7f0b84c00640 1 -- 172.21.15.6:0/416851233 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=18) ==== 24+0+0 (secure 0 0 0) 0x7f0b8809f050 con 0x5621626ce060 2024-04-24T23:16:38.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.410+0000 7f0ba1a10740 10 client.4996 fetch_fsmap learned FSMap version 18 2024-04-24T23:16:38.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.410+0000 7f0ba1a10740 1 -- 172.21.15.6:0/416851233 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=18,monmap=2+}) -- 0x5621626f8040 con 0x5621626ce060 2024-04-24T23:16:38.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.410+0000 7f0b84c00640 1 -- 172.21.15.6:0/416851233 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b88082bf0 con 0x5621626ce060 2024-04-24T23:16:38.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.410+0000 7f0b84c00640 1 -- 172.21.15.6:0/416851233 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 18) ==== 4792+0+0 (secure 0 0 0) 0x7f0b880a2020 con 0x5621626ce060 2024-04-24T23:16:38.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.410+0000 7f0ba1a10740 10 client.4996 fetch_fsmap finished waiting for FSMap version 18 2024-04-24T23:16:38.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.410+0000 7f0ba1a10740 10 client.4996 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-24T23:16:38.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.410+0000 7f0ba1a10740 20 client.4996 populate_metadata read hostname 'smithi006' 2024-04-24T23:16:38.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.410+0000 7f0ba1a10740 1 --2- 172.21.15.6:0/416851233 >> [v2:172.21.15.6:6833/489092393,v1:172.21.15.6:6835/489092393] conn(0x562162702bb0 0x562162704fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:38.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.410+0000 7f0ba1a10740 4 client.4996 mds_command: new command op to 4306 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T23:16:38.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.410+0000 7f0ba1a10740 1 -- 172.21.15.6:0/416851233 --> [v2:172.21.15.6:6833/489092393,v1:172.21.15.6:6835/489092393] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562162615330 con 0x562162702bb0 2024-04-24T23:16:38.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.411+0000 7f0b97400640 1 --2- 172.21.15.6:0/416851233 >> [v2:172.21.15.6:6833/489092393,v1:172.21.15.6:6835/489092393] conn(0x562162702bb0 0x562162704fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:38.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.411+0000 7f0b84c00640 10 client.4996 ms_handle_connect on v2:172.21.15.6:6833/489092393 2024-04-24T23:16:38.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.411+0000 7f0b97400640 1 --2- 172.21.15.6:0/416851233 >> [v2:172.21.15.6:6833/489092393,v1:172.21.15.6:6835/489092393] conn(0x562162702bb0 0x562162704fa0 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d8d547f5517cdc5f server_cookie=b7bff85a5996e355 in_seq=0 out_seq=0 2024-04-24T23:16:38.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.412+0000 7f0b84c00640 1 -- 172.21.15.6:0/416851233 <== mds.0 v2:172.21.15.6:6833/489092393 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562162615330 con 0x562162702bb0 2024-04-24T23:16:38.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.412+0000 7f0b84c00640 10 client.4996 handle_command_reply: tid=0 2024-04-24T23:16:38.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.412+0000 7f0ba1a10740 1 client.4996 shutdown 2024-04-24T23:16:38.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.412+0000 7f0ba1a10740 1 -- 172.21.15.6:0/416851233 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5621626ce060 msgr2=0x5621626c6830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:38.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.412+0000 7f0ba1a10740 1 --2- 172.21.15.6:0/416851233 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5621626ce060 0x5621626c6830 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f0b88009d90 tx=0x7f0b88000f40 comp rx=0 tx=0).stop 2024-04-24T23:16:38.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.412+0000 7f0ba1a10740 1 -- 172.21.15.6:0/416851233 shutdown_connections 2024-04-24T23:16:38.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.412+0000 7f0ba1a10740 1 --2- 172.21.15.6:0/416851233 >> [v2:172.21.15.6:6833/489092393,v1:172.21.15.6:6835/489092393] conn(0x562162702bb0 0x562162704fa0 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:38.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.412+0000 7f0ba1a10740 1 --2- 172.21.15.6:0/416851233 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x5621626c7800 0x5621626e4570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:38.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.412+0000 7f0ba1a10740 1 --2- 172.21.15.6:0/416851233 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5621626c8130 0x5621626f7b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:38.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.412+0000 7f0ba1a10740 1 --2- 172.21.15.6:0/416851233 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5621626ce060 0x5621626c6830 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:38.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.412+0000 7f0b84c00640 0 client.4996 ms_handle_reset on v2:172.21.15.6:6833/489092393 2024-04-24T23:16:38.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.412+0000 7f0ba1a10740 1 -- 172.21.15.6:0/416851233 >> 172.21.15.6:0/416851233 conn(0x5621626a3040 msgr2=0x56216269e680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:38.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.412+0000 7f0ba1a10740 1 -- 172.21.15.6:0/416851233 shutdown_connections 2024-04-24T23:16:38.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.412+0000 7f0ba1a10740 1 -- 172.21.15.6:0/416851233 wait complete. 2024-04-24T23:16:38.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.413+0000 7f0ba1a10740 20 client.4996 trim_cache size 0 max 16384 2024-04-24T23:16:38.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.432+0000 7f0ba1a10740 1 Processor -- start 2024-04-24T23:16:38.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.432+0000 7f0ba1a10740 1 -- start start 2024-04-24T23:16:38.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.433+0000 7f0ba1a10740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5621626c7800 0x5621626df520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:38.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.433+0000 7f0ba1a10740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5621626c8130 0x5621626c3be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:38.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.433+0000 7f0ba1a10740 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x5621626ce060 0x5621626c4120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:38.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.433+0000 7f0ba1a10740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f0b981fc740 con 0x5621626c7800 2024-04-24T23:16:38.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.433+0000 7f0ba1a10740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x5621626a1080 con 0x5621626c8130 2024-04-24T23:16:38.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.433+0000 7f0ba1a10740 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x562162603db0 con 0x5621626ce060 2024-04-24T23:16:38.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.434+0000 7f0b9cc00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x5621626ce060 0x5621626c4120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:38.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.434+0000 7f0b97400640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5621626c8130 0x5621626c3be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:38.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.434+0000 7f0b9cc00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x5621626ce060 0x5621626c4120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.6:42948/0 (socket says 172.21.15.6:42948) 2024-04-24T23:16:38.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.434+0000 7f0b9cc00640 1 -- 172.21.15.6:0/4061519466 learned_addr learned my addr 172.21.15.6:0/4061519466 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:16:38.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.434+0000 7f0b97e00640 1 --2- 172.21.15.6:0/4061519466 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5621626c7800 0x5621626df520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:38.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.434+0000 7f0b97400640 1 -- 172.21.15.6:0/4061519466 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x5621626ce060 msgr2=0x5621626c4120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:16:38.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.434+0000 7f0b97400640 1 --2- 172.21.15.6:0/4061519466 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x5621626ce060 0x5621626c4120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:38.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.434+0000 7f0b97400640 1 -- 172.21.15.6:0/4061519466 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5621626c7800 msgr2=0x5621626df520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:16:38.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.434+0000 7f0b97400640 1 --2- 172.21.15.6:0/4061519466 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5621626c7800 0x5621626df520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:38.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.434+0000 7f0b97400640 1 -- 172.21.15.6:0/4061519466 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5621626c6ba0 con 0x5621626c8130 2024-04-24T23:16:38.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.434+0000 7f0b97400640 1 --2- 172.21.15.6:0/4061519466 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5621626c8130 0x5621626c3be0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x5621626f9ef0 tx=0x7f0b80012000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4b44d10df8dfebda server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:38.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.435+0000 7f0b87400640 1 -- 172.21.15.6:0/4061519466 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b8002ad90 con 0x5621626c8130 2024-04-24T23:16:38.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.435+0000 7f0b87400640 1 -- 172.21.15.6:0/4061519466 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b8001b040 con 0x5621626c8130 2024-04-24T23:16:38.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.435+0000 7f0b87400640 1 -- 172.21.15.6:0/4061519466 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b80016110 con 0x5621626c8130 2024-04-24T23:16:38.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.435+0000 7f0ba1a10740 1 -- 172.21.15.6:0/4061519466 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5621626c8130 msgr2=0x5621626c3be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:38.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.435+0000 7f0ba1a10740 1 --2- 172.21.15.6:0/4061519466 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5621626c8130 0x5621626c3be0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x5621626f9ef0 tx=0x7f0b80012000 comp rx=0 tx=0).stop 2024-04-24T23:16:38.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.435+0000 7f0ba1a10740 1 -- 172.21.15.6:0/4061519466 shutdown_connections 2024-04-24T23:16:38.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.435+0000 7f0ba1a10740 1 --2- 172.21.15.6:0/4061519466 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x5621626ce060 0x5621626c4120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:38.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.435+0000 7f0ba1a10740 1 --2- 172.21.15.6:0/4061519466 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5621626c8130 0x5621626c3be0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:38.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.435+0000 7f0ba1a10740 1 --2- 172.21.15.6:0/4061519466 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5621626c7800 0x5621626df520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:38.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.435+0000 7f0ba1a10740 1 -- 172.21.15.6:0/4061519466 >> 172.21.15.6:0/4061519466 conn(0x5621626a3040 msgr2=0x5621626e97b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:38.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.435+0000 7f0ba1a10740 1 -- 172.21.15.6:0/4061519466 shutdown_connections 2024-04-24T23:16:38.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.435+0000 7f0ba1a10740 1 -- 172.21.15.6:0/4061519466 wait complete. 2024-04-24T23:16:38.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.436+0000 7f0ba1a10740 1 -- start start 2024-04-24T23:16:38.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.436+0000 7f0ba1a10740 1 Processor -- start 2024-04-24T23:16:38.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.437+0000 7f0ba1a10740 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x5621626c7800 0x5621626cb6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:38.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.437+0000 7f0ba1a10740 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5621626c8130 0x56216275a080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:38.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.437+0000 7f0ba1a10740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5621626ce060 0x562162763340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:38.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.437+0000 7f0ba1a10740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x5621626f8250 con 0x5621626ce060 2024-04-24T23:16:38.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.437+0000 7f0ba1a10740 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x5621625c3d20 con 0x5621626c8130 2024-04-24T23:16:38.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.437+0000 7f0ba1a10740 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x5621626f9870 con 0x5621626c7800 2024-04-24T23:16:38.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.437+0000 7f0ba1a10740 10 client.0 _ll_register_callbacks cb 0x7f0b80017040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T23:16:38.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.437+0000 7f0b97e00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x5621626c7800 0x5621626cb6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:38.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.438+0000 7f0b9cc00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5621626ce060 0x562162763340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:38.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.438+0000 7f0b97400640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5621626c8130 0x56216275a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:38.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.438+0000 7f0b97e00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x5621626c7800 0x5621626cb6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.6:42952/0 (socket says 172.21.15.6:42952) 2024-04-24T23:16:38.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.438+0000 7f0b97e00640 1 -- 172.21.15.6:0/3095626324 learned_addr learned my addr 172.21.15.6:0/3095626324 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:16:38.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.438+0000 7f0b97e00640 1 -- 172.21.15.6:0/3095626324 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5621626c8130 msgr2=0x56216275a080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:16:38.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.438+0000 7f0b97e00640 1 --2- 172.21.15.6:0/3095626324 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5621626c8130 0x56216275a080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:38.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.438+0000 7f0b97e00640 1 -- 172.21.15.6:0/3095626324 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5621626ce060 msgr2=0x562162763340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:16:38.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.438+0000 7f0b97e00640 1 --2- 172.21.15.6:0/3095626324 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5621626ce060 0x562162763340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:38.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.438+0000 7f0b97e00640 1 -- 172.21.15.6:0/3095626324 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({osdmap=0}) -- 0x56216270adb0 con 0x5621626c7800 2024-04-24T23:16:38.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.438+0000 7f0b97400640 1 --2- 172.21.15.6:0/3095626324 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5621626c8130 0x56216275a080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:16:38.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.438+0000 7f0b9cc00640 1 --2- 172.21.15.6:0/3095626324 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5621626ce060 0x562162763340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:16:38.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.438+0000 7f0b97e00640 1 --2- 172.21.15.6:0/3095626324 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x5621626c7800 0x5621626cb6c0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f0b880a39d0 tx=0x7f0b88009750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8c531625945bf2d2 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:38.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.438+0000 7f0b84c00640 10 client.0 ms_handle_connect on v2:172.21.15.190:3301/0 2024-04-24T23:16:38.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.438+0000 7f0b84c00640 1 -- 172.21.15.6:0/3095626324 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b88009a50 con 0x5621626c7800 2024-04-24T23:16:38.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.438+0000 7f0b84c00640 1 -- 172.21.15.6:0/3095626324 <== mon.2 v2:172.21.15.190:3301/0 2 ==== osd_map(26..26 src has 1..26) ==== 6590+0+0 (secure 0 0 0) 0x7f0b88079510 con 0x5621626c7800 2024-04-24T23:16:38.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.438+0000 7f0ba1a10740 1 -- 172.21.15.6:0/3095626324 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0b8001b040 con 0x5621626c7800 2024-04-24T23:16:38.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.439+0000 7f0b84c00640 1 -- 172.21.15.6:0/3095626324 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_get_version_reply(handle=1 version=18) ==== 24+0+0 (secure 0 0 0) 0x7f0b8808b050 con 0x5621626c7800 2024-04-24T23:16:38.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.439+0000 7f0ba1a10740 10 client.4745 fetch_fsmap learned FSMap version 18 2024-04-24T23:16:38.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.439+0000 7f0ba1a10740 1 -- 172.21.15.6:0/3095626324 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,fsmap=18,monmap=2+}) -- 0x5621626ce700 con 0x5621626c7800 2024-04-24T23:16:38.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.439+0000 7f0b84c00640 1 -- 172.21.15.6:0/3095626324 <== mon.2 v2:172.21.15.190:3301/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b8001b040 con 0x5621626c7800 2024-04-24T23:16:38.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.439+0000 7f0b84c00640 1 -- 172.21.15.6:0/3095626324 <== mon.2 v2:172.21.15.190:3301/0 5 ==== fsmap(e 18) ==== 4792+0+0 (secure 0 0 0) 0x7f0b8808e020 con 0x5621626c7800 2024-04-24T23:16:38.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.439+0000 7f0ba1a10740 10 client.4745 fetch_fsmap finished waiting for FSMap version 18 2024-04-24T23:16:38.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.439+0000 7f0ba1a10740 10 client.4745 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-24T23:16:38.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.439+0000 7f0ba1a10740 20 client.4745 populate_metadata read hostname 'smithi006' 2024-04-24T23:16:38.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.439+0000 7f0ba1a10740 1 --2- 172.21.15.6:0/3095626324 >> [v2:172.21.15.6:6833/489092393,v1:172.21.15.6:6835/489092393] conn(0x5621627607e0 0x56216276c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:38.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.439+0000 7f0ba1a10740 4 client.4745 mds_command: new command op to 4306 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-24T23:16:38.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.439+0000 7f0ba1a10740 1 -- 172.21.15.6:0/3095626324 --> [v2:172.21.15.6:6833/489092393,v1:172.21.15.6:6835/489092393] -- command(tid 0: {"prefix": "status"}) -- 0x5621626e3660 con 0x5621627607e0 2024-04-24T23:16:38.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.439+0000 7f0b97400640 1 --2- 172.21.15.6:0/3095626324 >> [v2:172.21.15.6:6833/489092393,v1:172.21.15.6:6835/489092393] conn(0x5621627607e0 0x56216276c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:38.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.440+0000 7f0b97400640 1 --2- 172.21.15.6:0/3095626324 >> [v2:172.21.15.6:6833/489092393,v1:172.21.15.6:6835/489092393] conn(0x5621627607e0 0x56216276c6e0 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5ea60a19532b5f8e server_cookie=42fe2cf6ee7c6e11 in_seq=0 out_seq=0 2024-04-24T23:16:38.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.440+0000 7f0b84c00640 10 client.4745 ms_handle_connect on v2:172.21.15.6:6833/489092393 2024-04-24T23:16:38.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.440+0000 7f0b84c00640 1 -- 172.21.15.6:0/3095626324 <== mds.0 v2:172.21.15.6:6833/489092393 1 ==== command_reply(tid 0: 0 ) ==== 8+0+331 (crc 0 0 0) 0x5621626e3660 con 0x5621627607e0 2024-04-24T23:16:38.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.440+0000 7f0b84c00640 10 client.4745 handle_command_reply: tid=0 2024-04-24T23:16:38.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.440+0000 7f0ba1a10740 1 client.4745 shutdown 2024-04-24T23:16:38.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.440+0000 7f0ba1a10740 1 -- 172.21.15.6:0/3095626324 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x5621626c7800 msgr2=0x5621626cb6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:38.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.440+0000 7f0ba1a10740 1 --2- 172.21.15.6:0/3095626324 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x5621626c7800 0x5621626cb6c0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f0b880a39d0 tx=0x7f0b88009750 comp rx=0 tx=0).stop 2024-04-24T23:16:38.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.441+0000 7f0ba1a10740 1 -- 172.21.15.6:0/3095626324 shutdown_connections 2024-04-24T23:16:38.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.441+0000 7f0ba1a10740 1 --2- 172.21.15.6:0/3095626324 >> [v2:172.21.15.6:6833/489092393,v1:172.21.15.6:6835/489092393] conn(0x5621627607e0 0x56216276c6e0 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:38.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.441+0000 7f0ba1a10740 1 --2- 172.21.15.6:0/3095626324 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5621626ce060 0x562162763340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:38.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.441+0000 7f0ba1a10740 1 --2- 172.21.15.6:0/3095626324 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5621626c8130 0x56216275a080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:38.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.441+0000 7f0ba1a10740 1 --2- 172.21.15.6:0/3095626324 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x5621626c7800 0x5621626cb6c0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:38.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.441+0000 7f0b84c00640 0 client.4745 ms_handle_reset on v2:172.21.15.6:6833/489092393 2024-04-24T23:16:38.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.441+0000 7f0ba1a10740 1 -- 172.21.15.6:0/3095626324 >> 172.21.15.6:0/3095626324 conn(0x5621626a3040 msgr2=0x5621626e2050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:38.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.441+0000 7f0ba1a10740 1 -- 172.21.15.6:0/3095626324 shutdown_connections 2024-04-24T23:16:38.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.441+0000 7f0ba1a10740 1 -- 172.21.15.6:0/3095626324 wait complete. 2024-04-24T23:16:38.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.441+0000 7f0ba1a10740 20 client.4745 trim_cache size 0 max 16384 2024-04-24T23:16:38.446 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-04-24T23:16:38.446 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_fsid": "18bef68d-5e6d-4ec5-862b-b6f206e2588d", 2024-04-24T23:16:38.446 INFO:teuthology.orchestra.run.smithi006.stdout: "whoami": 0, 2024-04-24T23:16:38.446 INFO:teuthology.orchestra.run.smithi006.stdout: "id": 4306, 2024-04-24T23:16:38.446 INFO:teuthology.orchestra.run.smithi006.stdout: "want_state": "up:active", 2024-04-24T23:16:38.447 INFO:teuthology.orchestra.run.smithi006.stdout: "state": "up:active", 2024-04-24T23:16:38.447 INFO:teuthology.orchestra.run.smithi006.stdout: "fs_name": "cephfs", 2024-04-24T23:16:38.447 INFO:teuthology.orchestra.run.smithi006.stdout: "rank_uptime": 61.469115109999997, 2024-04-24T23:16:38.447 INFO:teuthology.orchestra.run.smithi006.stdout: "mdsmap_epoch": 18, 2024-04-24T23:16:38.447 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap_epoch": 26, 2024-04-24T23:16:38.447 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap_epoch_barrier": 26, 2024-04-24T23:16:38.447 INFO:teuthology.orchestra.run.smithi006.stdout: "uptime": 67.503201781000001 2024-04-24T23:16:38.447 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-04-24T23:16:38.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.442+0000 7f0b9ec00640 1 -- 172.21.15.6:0/2198079390 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f0b7406be60 msgr2=0x7f0b7406e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:38.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.442+0000 7f0b9ec00640 1 --2- 172.21.15.6:0/2198079390 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f0b7406be60 0x7f0b7406e320 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f0b98158e70 tx=0x7f0b80009040 comp rx=0 tx=0).stop 2024-04-24T23:16:38.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.442+0000 7f0b9ec00640 1 -- 172.21.15.6:0/2198079390 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0b980b0e50 msgr2=0x7f0b981e78f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:38.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.442+0000 7f0b9ec00640 1 --2- 172.21.15.6:0/2198079390 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0b980b0e50 0x7f0b981e78f0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f0b9806f9a0 tx=0x7f0b88004a40 comp rx=0 tx=0).stop 2024-04-24T23:16:38.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.442+0000 7f0b9ec00640 1 -- 172.21.15.6:0/2198079390 shutdown_connections 2024-04-24T23:16:38.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.442+0000 7f0b9ec00640 1 --2- 172.21.15.6:0/2198079390 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f0b7406be60 0x7f0b7406e320 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:38.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.442+0000 7f0b9ec00640 1 --2- 172.21.15.6:0/2198079390 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0b9815c1b0 0x7f0b981f1be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:38.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.442+0000 7f0b9ec00640 1 --2- 172.21.15.6:0/2198079390 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0b980cad80 0x7f0b981f16a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:38.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.442+0000 7f0b9ec00640 1 --2- 172.21.15.6:0/2198079390 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0b980b0e50 0x7f0b981e78f0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:38.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.442+0000 7f0b9ec00640 1 -- 172.21.15.6:0/2198079390 >> 172.21.15.6:0/2198079390 conn(0x7f0b9806fdc0 msgr2=0x7f0b980b0020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:38.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.442+0000 7f0b9ec00640 1 -- 172.21.15.6:0/2198079390 shutdown_connections 2024-04-24T23:16:38.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.442+0000 7f0b9ec00640 1 -- 172.21.15.6:0/2198079390 wait complete. 2024-04-24T23:16:38.460 INFO:teuthology.run_tasks:Running task sequential... 2024-04-24T23:16:38.469 INFO:teuthology.task.sequential:In sequential, running task sequential... 2024-04-24T23:16:38.470 INFO:teuthology.task.sequential:In sequential, running task print... 2024-04-24T23:16:38.470 INFO:teuthology.task.print:Enabling mgr modules 2024-04-24T23:16:38.470 INFO:teuthology.run_tasks:Running task kclient... 2024-04-24T23:16:38.519 INFO:tasks.kclient:Mounting kernel clients... 2024-04-24T23:16:38.519 INFO:tasks.kclient:config is {'client.0': {}} 2024-04-24T23:16:38.519 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:16:38.519 DEBUG:teuthology.orchestra.run.smithi006:> ip netns list 2024-04-24T23:16:38.576 INFO:teuthology.orchestra.run.smithi006.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2024-04-24T23:16:38.577 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:16:38.578 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-24T23:16:38.663 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:16:38.663 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip link delete ceph-brx 2024-04-24T23:16:38.774 INFO:tasks.kclient:client.0 config is {'syntax': 'v1'} 2024-04-24T23:16:38.777 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-24T23:16:38.777 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-24T23:16:38.778 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-24T23:16:38.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.959+0000 7f97faa00640 1 Processor -- start 2024-04-24T23:16:38.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.959+0000 7f97faa00640 1 -- start start 2024-04-24T23:16:38.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.961+0000 7f97faa00640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f97f4154b70 0x7f97f406f020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:38.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.961+0000 7f97faa00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f97f4072530 con 0x7f97f4071000 2024-04-24T23:16:38.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.961+0000 7f97faa00640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f97f40727d0 con 0x7f97f40c7080 2024-04-24T23:16:38.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.961+0000 7f97faa00640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f97f4072a70 con 0x7f97f4154b70 2024-04-24T23:16:38.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.961+0000 7f97f3e00640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f97f4154b70 0x7f97f406f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:38.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.962+0000 7f97f3e00640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f97f4154b70 0x7f97f406f020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:35486/0 (socket says 172.21.15.6:35486) 2024-04-24T23:16:38.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.961+0000 7f97ebe00640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f97f40c7080 0x7f97f415a9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.6:57872/0 (socket says 172.21.15.6:57872) 2024-04-24T23:16:38.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.962+0000 7f97f3e00640 1 -- 172.21.15.6:0/3705960753 learned_addr learned my addr 172.21.15.6:0/3705960753 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:16:38.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.962+0000 7f97f3e00640 1 -- 172.21.15.6:0/3705960753 >> v1:172.21.15.190:6789/0 conn(0x7f97f40c7080 legacy=0x7f97f415a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:38.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.962+0000 7f97f3e00640 1 -- 172.21.15.6:0/3705960753 >> v1:172.21.15.6:6789/0 conn(0x7f97f4071000 legacy=0x7f97f415b0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:38.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.962+0000 7f97f3400640 1 -- 172.21.15.6:0/3705960753 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1762143689 0 0) 0x7f97f40727d0 con 0x7f97f40c7080 2024-04-24T23:16:38.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.962+0000 7f97f3e00640 1 -- 172.21.15.6:0/3705960753 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97f406e540 con 0x7f97f4154b70 2024-04-24T23:16:38.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.964+0000 7f97f3e00640 1 --2- 172.21.15.6:0/3705960753 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f97f4154b70 0x7f97f406f020 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f97e40098a0 tx=0x7f97e402a320 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4560e5a3238617e1 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:38.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.964+0000 7f97f3400640 1 -- 172.21.15.6:0/3705960753 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97e4037070 con 0x7f97f4154b70 2024-04-24T23:16:38.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.964+0000 7f97f3400640 1 -- 172.21.15.6:0/3705960753 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97e4032640 con 0x7f97f4154b70 2024-04-24T23:16:38.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.964+0000 7f97f3400640 1 -- 172.21.15.6:0/3705960753 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97e4032a10 con 0x7f97f4154b70 2024-04-24T23:16:38.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.965+0000 7f97faa00640 1 -- 172.21.15.6:0/3705960753 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f97f4154b70 msgr2=0x7f97f406f020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:38.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.965+0000 7f97faa00640 1 --2- 172.21.15.6:0/3705960753 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f97f4154b70 0x7f97f406f020 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f97e40098a0 tx=0x7f97e402a320 comp rx=0 tx=0).stop 2024-04-24T23:16:38.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.965+0000 7f97faa00640 1 -- 172.21.15.6:0/3705960753 shutdown_connections 2024-04-24T23:16:38.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.965+0000 7f97faa00640 1 --2- 172.21.15.6:0/3705960753 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f97f4154b70 0x7f97f406f020 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:38.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.965+0000 7f97faa00640 1 -- 172.21.15.6:0/3705960753 >> 172.21.15.6:0/3705960753 conn(0x7f97f406b0e0 msgr2=0x7f97f406b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:38.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.965+0000 7f97faa00640 1 -- 172.21.15.6:0/3705960753 shutdown_connections 2024-04-24T23:16:38.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.965+0000 7f97faa00640 1 -- 172.21.15.6:0/3705960753 wait complete. 2024-04-24T23:16:38.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.967+0000 7f97faa00640 1 Processor -- start 2024-04-24T23:16:38.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.967+0000 7f97faa00640 1 -- start start 2024-04-24T23:16:38.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.968+0000 7f97faa00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f97f4071000 0x7f97f41f4490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:38.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.968+0000 7f97faa00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f97f40c7080 0x7f97f4157540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:38.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.968+0000 7f97faa00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f97f4154b70 0x7f97f414ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:38.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.968+0000 7f97faa00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f97f415b7f0 con 0x7f97f40c7080 2024-04-24T23:16:38.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.968+0000 7f97faa00640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f97f406eb00 con 0x7f97f4154b70 2024-04-24T23:16:38.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.968+0000 7f97faa00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f97f4154f50 con 0x7f97f4071000 2024-04-24T23:16:38.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.968+0000 7f97f3e00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f97f4071000 0x7f97f41f4490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:38.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.968+0000 7f97f8a00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f97f4154b70 0x7f97f414ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:38.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.968+0000 7f97f3e00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f97f4071000 0x7f97f41f4490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.6:42956/0 (socket says 172.21.15.6:42956) 2024-04-24T23:16:38.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.968+0000 7f97f3e00640 1 -- 172.21.15.6:0/1076006160 learned_addr learned my addr 172.21.15.6:0/1076006160 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:16:38.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.968+0000 7f97ebe00640 1 --2- 172.21.15.6:0/1076006160 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f97f40c7080 0x7f97f4157540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:38.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.969+0000 7f97f3e00640 1 -- 172.21.15.6:0/1076006160 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f97f4154b70 msgr2=0x7f97f414ac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:38.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.969+0000 7f97f3e00640 1 --2- 172.21.15.6:0/1076006160 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f97f4154b70 0x7f97f414ac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:38.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.969+0000 7f97f3e00640 1 -- 172.21.15.6:0/1076006160 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f97f40c7080 msgr2=0x7f97f4157540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:38.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.969+0000 7f97f3e00640 1 --2- 172.21.15.6:0/1076006160 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f97f40c7080 0x7f97f4157540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:38.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.969+0000 7f97f3e00640 1 -- 172.21.15.6:0/1076006160 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97f41f49d0 con 0x7f97f4071000 2024-04-24T23:16:38.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.969+0000 7f97f8a00640 1 --2- 172.21.15.6:0/1076006160 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f97f4154b70 0x7f97f414ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:16:38.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.969+0000 7f97f3e00640 1 --2- 172.21.15.6:0/1076006160 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f97f4071000 0x7f97f41f4490 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f97e4035040 tx=0x7f97e40046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:38.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.970+0000 7f97f0c00640 1 -- 172.21.15.6:0/1076006160 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97e4037040 con 0x7f97f4071000 2024-04-24T23:16:38.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.970+0000 7f97f0c00640 1 -- 172.21.15.6:0/1076006160 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97e4004230 con 0x7f97f4071000 2024-04-24T23:16:38.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.970+0000 7f97faa00640 1 -- 172.21.15.6:0/1076006160 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97f41534d0 con 0x7f97f4071000 2024-04-24T23:16:38.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.970+0000 7f97f0c00640 1 -- 172.21.15.6:0/1076006160 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97e40306f0 con 0x7f97f4071000 2024-04-24T23:16:38.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.970+0000 7f97faa00640 1 -- 172.21.15.6:0/1076006160 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f97f4153a90 con 0x7f97f4071000 2024-04-24T23:16:38.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.971+0000 7f97faa00640 1 -- 172.21.15.6:0/1076006160 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97f40767b0 con 0x7f97f4071000 2024-04-24T23:16:38.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.972+0000 7f97f0c00640 1 -- 172.21.15.6:0/1076006160 <== mon.2 v2:172.21.15.190:3301/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f97e4032500 con 0x7f97f4071000 2024-04-24T23:16:38.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.973+0000 7f97f0c00640 1 --2- 172.21.15.6:0/1076006160 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f97d006be30 0x7f97d006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:38.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.973+0000 7f97f0c00640 1 -- 172.21.15.6:0/1076006160 <== mon.2 v2:172.21.15.190:3301/0 5 ==== osd_map(26..26 src has 1..26) ==== 6590+0+0 (secure 0 0 0) 0x7f97e40abfe0 con 0x7f97f4071000 2024-04-24T23:16:38.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.974+0000 7f97ebe00640 1 --2- 172.21.15.6:0/1076006160 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f97d006be30 0x7f97d006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:38.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.974+0000 7f97ebe00640 1 --2- 172.21.15.6:0/1076006160 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f97d006be30 0x7f97d006e2f0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f97dc007f70 tx=0x7f97dc007f00 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:38.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:38.978+0000 7f97f0c00640 1 -- 172.21.15.6:0/1076006160 <== mon.2 v2:172.21.15.190:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f97e4079fa0 con 0x7f97f4071000 2024-04-24T23:16:39.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:39.227+0000 7f97faa00640 1 -- 172.21.15.6:0/1076006160 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f97f4153c90 con 0x7f97f4071000 2024-04-24T23:16:39.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:39.228+0000 7f97f0c00640 1 -- 172.21.15.6:0/1076006160 <== mon.2 v2:172.21.15.190:3301/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v18) ==== 53+0+88 (secure 0 0 0) 0x7f97e407e2a0 con 0x7f97f4071000 2024-04-24T23:16:39.229 INFO:teuthology.orchestra.run.smithi006.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-04-24T23:16:39.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:39.231+0000 7f97faa00640 1 -- 172.21.15.6:0/1076006160 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f97d006be30 msgr2=0x7f97d006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:39.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:39.231+0000 7f97faa00640 1 --2- 172.21.15.6:0/1076006160 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f97d006be30 0x7f97d006e2f0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f97dc007f70 tx=0x7f97dc007f00 comp rx=0 tx=0).stop 2024-04-24T23:16:39.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:39.231+0000 7f97faa00640 1 -- 172.21.15.6:0/1076006160 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f97f4071000 msgr2=0x7f97f41f4490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:39.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:39.231+0000 7f97faa00640 1 --2- 172.21.15.6:0/1076006160 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f97f4071000 0x7f97f41f4490 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f97e4035040 tx=0x7f97e40046c0 comp rx=0 tx=0).stop 2024-04-24T23:16:39.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:39.231+0000 7f97faa00640 1 -- 172.21.15.6:0/1076006160 shutdown_connections 2024-04-24T23:16:39.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:39.231+0000 7f97faa00640 1 --2- 172.21.15.6:0/1076006160 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f97d006be30 0x7f97d006e2f0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:39.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:39.231+0000 7f97faa00640 1 --2- 172.21.15.6:0/1076006160 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f97f4154b70 0x7f97f414ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:39.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:39.231+0000 7f97faa00640 1 --2- 172.21.15.6:0/1076006160 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f97f40c7080 0x7f97f4157540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:39.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:39.231+0000 7f97faa00640 1 --2- 172.21.15.6:0/1076006160 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f97f4071000 0x7f97f41f4490 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:39.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:39.231+0000 7f97faa00640 1 -- 172.21.15.6:0/1076006160 >> 172.21.15.6:0/1076006160 conn(0x7f97f406b0e0 msgr2=0x7f97f4152140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:39.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:39.231+0000 7f97faa00640 1 -- 172.21.15.6:0/1076006160 shutdown_connections 2024-04-24T23:16:39.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:39.231+0000 7f97faa00640 1 -- 172.21.15.6:0/1076006160 wait complete. 2024-04-24T23:16:39.251 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-24T23:16:39.251 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-24T23:16:39.251 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi006.front.sepia.ceph.com 2024-04-24T23:16:39.251 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-24T23:16:39.251 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-24T23:16:39.251 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-24T23:16:39.251 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-24T23:16:39.252 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-24T23:16:39.252 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-24T23:16:39.252 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:16:39.252 DEBUG:teuthology.orchestra.run.smithi006:> ip addr 2024-04-24T23:16:39.303 INFO:teuthology.orchestra.run.smithi006.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-24T23:16:39.304 INFO:teuthology.orchestra.run.smithi006.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-24T23:16:39.304 INFO:teuthology.orchestra.run.smithi006.stdout: inet 127.0.0.1/8 scope host lo 2024-04-24T23:16:39.304 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-04-24T23:16:39.304 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 ::1/128 scope host 2024-04-24T23:16:39.304 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-04-24T23:16:39.304 INFO:teuthology.orchestra.run.smithi006.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-24T23:16:39.304 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:6b:fe:e6 brd ff:ff:ff:ff:ff:ff 2024-04-24T23:16:39.304 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp7s0f0 2024-04-24T23:16:39.304 INFO:teuthology.orchestra.run.smithi006.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-24T23:16:39.304 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:6b:fe:e7 brd ff:ff:ff:ff:ff:ff 2024-04-24T23:16:39.304 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp7s0f1 2024-04-24T23:16:39.304 INFO:teuthology.orchestra.run.smithi006.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-24T23:16:39.304 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:bd:15:e4 brd ff:ff:ff:ff:ff:ff 2024-04-24T23:16:39.304 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp1s0f0 2024-04-24T23:16:39.305 INFO:teuthology.orchestra.run.smithi006.stdout: inet 172.21.15.6/20 brd 172.21.15.255 scope global dynamic ens1f0 2024-04-24T23:16:39.305 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft 42641sec preferred_lft 42641sec 2024-04-24T23:16:39.305 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 fe80::ec4:7aff:febd:15e4/64 scope link noprefixroute 2024-04-24T23:16:39.305 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-04-24T23:16:39.305 INFO:teuthology.orchestra.run.smithi006.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-24T23:16:39.305 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:bd:15:e5 brd ff:ff:ff:ff:ff:ff 2024-04-24T23:16:39.305 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp1s0f1 2024-04-24T23:16:39.306 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-24T23:16:39.306 DEBUG:teuthology.orchestra.run.smithi006:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-24T23:16:39.306 DEBUG:teuthology.orchestra.run.smithi006:> set -e 2024-04-24T23:16:39.306 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip link add name ceph-brx type bridge 2024-04-24T23:16:39.306 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip addr flush dev ceph-brx 2024-04-24T23:16:39.306 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip link set ceph-brx up 2024-04-24T23:16:39.306 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-24T23:16:39.306 DEBUG:teuthology.orchestra.run.smithi006:> ') 2024-04-24T23:16:39.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:39 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-24T23:16:39.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:39 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T23:16:39.561 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:16:39.561 DEBUG:teuthology.orchestra.run.smithi006:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-24T23:16:39.629 INFO:teuthology.orchestra.run.smithi006.stdout:1 2024-04-24T23:16:39.632 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:16:39.633 DEBUG:teuthology.orchestra.run.smithi006:> route 2024-04-24T23:16:39.687 INFO:teuthology.orchestra.run.smithi006.stdout:Kernel IP routing table 2024-04-24T23:16:39.687 INFO:teuthology.orchestra.run.smithi006.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-24T23:16:39.687 INFO:teuthology.orchestra.run.smithi006.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-04-24T23:16:39.687 INFO:teuthology.orchestra.run.smithi006.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-04-24T23:16:39.687 INFO:teuthology.orchestra.run.smithi006.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-24T23:16:39.688 DEBUG:teuthology.orchestra.run.smithi006:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-24T23:16:39.688 DEBUG:teuthology.orchestra.run.smithi006:> set -e 2024-04-24T23:16:39.688 DEBUG:teuthology.orchestra.run.smithi006:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-04-24T23:16:39.688 DEBUG:teuthology.orchestra.run.smithi006:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-04-24T23:16:39.688 DEBUG:teuthology.orchestra.run.smithi006:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-04-24T23:16:39.689 DEBUG:teuthology.orchestra.run.smithi006:> ') 2024-04-24T23:16:39.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:39 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-24T23:16:39.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:39 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T23:16:39.899 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:16:39.899 DEBUG:teuthology.orchestra.run.smithi006:> ip netns list 2024-04-24T23:16:39.938 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:16:39.938 DEBUG:teuthology.orchestra.run.smithi006:> ip netns list-id 2024-04-24T23:16:40.018 DEBUG:teuthology.orchestra.run.smithi006:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-24T23:16:40.018 DEBUG:teuthology.orchestra.run.smithi006:> set -e 2024-04-24T23:16:40.018 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-24T23:16:40.018 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-04-24T23:16:40.018 DEBUG:teuthology.orchestra.run.smithi006:> ') 2024-04-24T23:16:40.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:40 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-24T23:16:40.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:40 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T23:16:40.213 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-04-24T23:16:40.213 DEBUG:teuthology.orchestra.run.smithi006:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-24T23:16:40.213 DEBUG:teuthology.orchestra.run.smithi006:> set -e 2024-04-24T23:16:40.213 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-04-24T23:16:40.213 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-24T23:16:40.213 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-04-24T23:16:40.213 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-04-24T23:16:40.213 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-04-24T23:16:40.213 DEBUG:teuthology.orchestra.run.smithi006:> ') 2024-04-24T23:16:40.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:40 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-24T23:16:40.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:40 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T23:16:40.486 DEBUG:teuthology.orchestra.run.smithi006:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-24T23:16:40.486 DEBUG:teuthology.orchestra.run.smithi006:> set -e 2024-04-24T23:16:40.486 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip link set brx.0 up 2024-04-24T23:16:40.486 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip link set dev brx.0 master ceph-brx 2024-04-24T23:16:40.486 DEBUG:teuthology.orchestra.run.smithi006:> ') 2024-04-24T23:16:40.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:40 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-24T23:16:40.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:40 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T23:16:40.694 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-24T23:16:40.694 DEBUG:teuthology.orchestra.run.smithi006:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-24T23:16:40.738 INFO:teuthology.orchestra.run.smithi006.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-24T23:16:40.739 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-24T23:16:40.739 DEBUG:teuthology.orchestra.run.smithi006:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-24T23:16:40.835 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-04-24T23:16:40.836 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:16:40.836 DEBUG:teuthology.orchestra.run.smithi006:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-24T23:16:40.969 INFO:teuthology.orchestra.run.smithi006.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-24T23:16:40.970 INFO:teuthology.orchestra.run.smithi006.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-04-24T23:16:40.970 INFO:teuthology.orchestra.run.smithi006.stdout:invalid new device string format 2024-04-24T23:16:41.625 INFO:teuthology.orchestra.run.smithi006.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-24T23:16:41.625 INFO:teuthology.orchestra.run.smithi006.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-04-24T23:16:41.625 INFO:teuthology.orchestra.run.smithi006.stdout:invalid new device string format 2024-04-24T23:16:41.626 INFO:teuthology.orchestra.run.smithi006.stdout:mount.ceph: resolved to: "172.21.15.6:3300,172.21.15.190:3300,172.21.15.190:3301" 2024-04-24T23:16:41.626 INFO:teuthology.orchestra.run.smithi006.stdout:mount.ceph: switching to using v1 address with old syntax 2024-04-24T23:16:41.658 INFO:teuthology.orchestra.run.smithi006.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-24T23:16:41.658 INFO:teuthology.orchestra.run.smithi006.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-04-24T23:16:41.658 INFO:teuthology.orchestra.run.smithi006.stdout:invalid new device string format 2024-04-24T23:16:41.658 INFO:teuthology.orchestra.run.smithi006.stdout:mount.ceph: resolved to: "172.21.15.6:3300,172.21.15.190:3300,172.21.15.190:3301" 2024-04-24T23:16:41.659 INFO:teuthology.orchestra.run.smithi006.stdout:mount.ceph: switching to using v1 address with old syntax 2024-04-24T23:16:41.659 INFO:teuthology.orchestra.run.smithi006.stdout:mount.ceph: resolved to: "172.21.15.6:6789,172.21.15.190:6789,172.21.15.190:6790" 2024-04-24T23:16:41.659 INFO:teuthology.orchestra.run.smithi006.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.6:6789,172.21.15.190:6789,172.21.15.190:6790:/ 2024-04-24T23:16:41.659 INFO:teuthology.orchestra.run.smithi006.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=18bef68d-5e6d-4ec5-862b-b6f206e2588d" will pass to kernel 2024-04-24T23:16:41.662 INFO:teuthology.orchestra.run.smithi006.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-04-24T23:16:41.662 INFO:teuthology.orchestra.run.smithi006.stdout: You just mounted a file system that supports labels which does not 2024-04-24T23:16:41.662 INFO:teuthology.orchestra.run.smithi006.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-04-24T23:16:41.662 INFO:teuthology.orchestra.run.smithi006.stdout: applications will generate AVC messages and not be allowed access to 2024-04-24T23:16:41.662 INFO:teuthology.orchestra.run.smithi006.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-04-24T23:16:41.667 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-24T23:16:41.667 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:16:41.667 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-24T23:16:41.753 DEBUG:teuthology.orchestra.run.smithi006:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-24T23:16:41.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:41 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-24T23:16:42.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:42 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T23:16:42.147 INFO:teuthology.orchestra.run.smithi006.stdout:client4751 2024-04-24T23:16:42.147 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-04-24T23:16:42.147 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-24T23:16:42.205 DEBUG:teuthology.orchestra.run.smithi006:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-24T23:16:42.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:42 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-24T23:16:42.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:42 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T23:16:42.331 INFO:teuthology.orchestra.run.smithi006.stdout:client4751 2024-04-24T23:16:42.331 DEBUG:teuthology.orchestra.run.smithi006:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-24T23:16:42.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:42 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-24T23:16:42.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:42 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T23:16:42.456 INFO:teuthology.orchestra.run.smithi006.stdout:client4751 2024-04-24T23:16:42.456 DEBUG:teuthology.orchestra.run.smithi006:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/18bef68d-5e6d-4ec5-862b-b6f206e2588d.client4751/status') 2024-04-24T23:16:42.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:42 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-24T23:16:42.598 INFO:teuthology.orchestra.run.smithi006.stdout:instance: client.4751 (3)172.21.15.6:0/2646866773 2024-04-24T23:16:42.598 INFO:teuthology.orchestra.run.smithi006.stdout:blocklisted: false 2024-04-24T23:16:42.598 INFO:teuthology.orchestra.run.smithi006.stderr:0+1 records in 2024-04-24T23:16:42.598 INFO:teuthology.orchestra.run.smithi006.stderr:0+1 records out 2024-04-24T23:16:42.598 INFO:teuthology.orchestra.run.smithi006.stderr:69 bytes copied, 0.000124582 s, 554 kB/s 2024-04-24T23:16:42.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:42 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T23:16:42.612 INFO:teuthology.run_tasks:Running task mds_thrash... 2024-04-24T23:16:42.625 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T23:16:42.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:42.835+0000 7f12d5200640 1 Processor -- start 2024-04-24T23:16:42.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:42.835+0000 7f12d5200640 1 -- start start 2024-04-24T23:16:42.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:42.837+0000 7f12d5200640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f12d0151be0 0x7f12d0157650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:42.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:42.837+0000 7f12d5200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f12d0072320 0x7f12d015b610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:42.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:42.837+0000 7f12d5200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f12d0071680 con 0x7f12d00c6340 2024-04-24T23:16:42.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:42.837+0000 7f12d5200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f12d015abc0 con 0x7f12d0072320 2024-04-24T23:16:42.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:42.837+0000 7f12d5200640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f12d0071960 con 0x7f12d0151be0 2024-04-24T23:16:42.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:42.837+0000 7f12c7e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f12d0072320 0x7f12d015b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:42.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:42.837+0000 7f12c7e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f12d0072320 0x7f12d015b610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:35716/0 (socket says 172.21.15.6:35716) 2024-04-24T23:16:42.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:42.837+0000 7f12c7e00640 1 -- 172.21.15.6:0/709930749 learned_addr learned my addr 172.21.15.6:0/709930749 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:16:42.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:42.837+0000 7f12cf400640 1 --2- 172.21.15.6:0/709930749 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f12d0151be0 0x7f12d0157650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:42.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:42.838+0000 7f12c7e00640 1 -- 172.21.15.6:0/709930749 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f12d0151be0 msgr2=0x7f12d0157650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:16:42.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:42.838+0000 7f12c7e00640 1 --2- 172.21.15.6:0/709930749 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f12d0151be0 0x7f12d0157650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:42.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:42.838+0000 7f12c7e00640 1 -- 172.21.15.6:0/709930749 >> v1:172.21.15.6:6789/0 conn(0x7f12d00c6340 legacy=0x7f12d00cc600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:42.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:42.838+0000 7f12c7e00640 1 -- 172.21.15.6:0/709930749 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12d015ae10 con 0x7f12d0072320 2024-04-24T23:16:42.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:42.838+0000 7f12cea00640 1 -- 172.21.15.6:0/709930749 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3153995004 0 0) 0x7f12d0071680 con 0x7f12d00c6340 2024-04-24T23:16:42.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:42.838+0000 7f12cf400640 1 --2- 172.21.15.6:0/709930749 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f12d0151be0 0x7f12d0157650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:16:42.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:42.839+0000 7f12c7e00640 1 --2- 172.21.15.6:0/709930749 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f12d0072320 0x7f12d015b610 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f12b8009870 tx=0x7f12b802a130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62d3ce6773da0c63 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:42.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:42.839+0000 7f12cea00640 1 -- 172.21.15.6:0/709930749 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12b8037070 con 0x7f12d0072320 2024-04-24T23:16:42.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:42.839+0000 7f12cea00640 1 -- 172.21.15.6:0/709930749 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f12b8032440 con 0x7f12d0072320 2024-04-24T23:16:42.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:42.839+0000 7f12cea00640 1 -- 172.21.15.6:0/709930749 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12b8032810 con 0x7f12d0072320 2024-04-24T23:16:42.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:42.840+0000 7f12d5200640 1 -- 172.21.15.6:0/709930749 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f12d0072320 msgr2=0x7f12d015b610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:42.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:42.840+0000 7f12d5200640 1 --2- 172.21.15.6:0/709930749 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f12d0072320 0x7f12d015b610 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f12b8009870 tx=0x7f12b802a130 comp rx=0 tx=0).stop 2024-04-24T23:16:42.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:42.840+0000 7f12d5200640 1 -- 172.21.15.6:0/709930749 shutdown_connections 2024-04-24T23:16:42.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:42.840+0000 7f12d5200640 1 --2- 172.21.15.6:0/709930749 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f12d0072320 0x7f12d015b610 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:42.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:42.840+0000 7f12d5200640 1 --2- 172.21.15.6:0/709930749 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f12d0151be0 0x7f12d0157650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:42.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:42.840+0000 7f12d5200640 1 -- 172.21.15.6:0/709930749 >> 172.21.15.6:0/709930749 conn(0x7f12d006ed80 msgr2=0x7f12d006f190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:42.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:42.840+0000 7f12d5200640 1 -- 172.21.15.6:0/709930749 shutdown_connections 2024-04-24T23:16:42.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:42.840+0000 7f12d5200640 1 -- 172.21.15.6:0/709930749 wait complete. 2024-04-24T23:16:42.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:42.842+0000 7f12d5200640 1 Processor -- start 2024-04-24T23:16:42.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:42.842+0000 7f12d5200640 1 -- start start 2024-04-24T23:16:42.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:42.843+0000 7f12d5200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f12d0072320 0x7f12d01f3a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:42.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:42.843+0000 7f12d5200640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f12d00c6340 0x7f12d0146c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:42.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:42.843+0000 7f12d5200640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f12d0151be0 0x7f12d0144930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:42.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:42.843+0000 7f12d5200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f12d00c36e0 con 0x7f12d0072320 2024-04-24T23:16:42.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:42.843+0000 7f12d5200640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f12d00c3ab0 con 0x7f12d0151be0 2024-04-24T23:16:42.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:42.843+0000 7f12d5200640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f12d00c3860 con 0x7f12d00c6340 2024-04-24T23:16:42.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:42.843+0000 7f12cf400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f12d0072320 0x7f12d01f3a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:42.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:42.843+0000 7f12cf400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f12d0072320 0x7f12d01f3a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:35732/0 (socket says 172.21.15.6:35732) 2024-04-24T23:16:42.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:42.843+0000 7f12cf400640 1 -- 172.21.15.6:0/785796220 learned_addr learned my addr 172.21.15.6:0/785796220 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:16:42.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:42.843+0000 7f12cfe00640 1 --2- 172.21.15.6:0/785796220 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f12d0151be0 0x7f12d0144930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:42.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:42.843+0000 7f12c7e00640 1 --2- 172.21.15.6:0/785796220 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f12d00c6340 0x7f12d0146c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:42.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:42.844+0000 7f12cf400640 1 -- 172.21.15.6:0/785796220 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f12d00c6340 msgr2=0x7f12d0146c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:42.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:42.844+0000 7f12cf400640 1 --2- 172.21.15.6:0/785796220 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f12d00c6340 0x7f12d0146c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:42.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:42.844+0000 7f12cf400640 1 -- 172.21.15.6:0/785796220 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f12d0151be0 msgr2=0x7f12d0144930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:42.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:42.844+0000 7f12cf400640 1 --2- 172.21.15.6:0/785796220 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f12d0151be0 0x7f12d0144930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:42.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:42.844+0000 7f12cf400640 1 -- 172.21.15.6:0/785796220 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12d0147970 con 0x7f12d0072320 2024-04-24T23:16:42.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:42.844+0000 7f12c7e00640 1 --2- 172.21.15.6:0/785796220 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f12d00c6340 0x7f12d0146c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:16:42.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:42.844+0000 7f12cfe00640 1 --2- 172.21.15.6:0/785796220 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f12d0151be0 0x7f12d0144930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:16:42.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:42.844+0000 7f12cf400640 1 --2- 172.21.15.6:0/785796220 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f12d0072320 0x7f12d01f3a70 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f12c000e960 tx=0x7f12c000ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:42.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:42.844+0000 7f12c7400640 1 -- 172.21.15.6:0/785796220 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12c0017070 con 0x7f12d0072320 2024-04-24T23:16:42.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:42.844+0000 7f12c7400640 1 -- 172.21.15.6:0/785796220 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f12c000ca00 con 0x7f12d0072320 2024-04-24T23:16:42.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:42.844+0000 7f12c7400640 1 -- 172.21.15.6:0/785796220 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12c000cdd0 con 0x7f12d0072320 2024-04-24T23:16:42.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:42.844+0000 7f12d5200640 1 -- 172.21.15.6:0/785796220 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12d014c390 con 0x7f12d0072320 2024-04-24T23:16:42.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:42.845+0000 7f12d5200640 1 -- 172.21.15.6:0/785796220 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12d006ced0 con 0x7f12d0072320 2024-04-24T23:16:42.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:42.846+0000 7f12d5200640 1 -- 172.21.15.6:0/785796220 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f12d0068d50 con 0x7f12d0072320 2024-04-24T23:16:42.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:42.847+0000 7f12c7400640 1 -- 172.21.15.6:0/785796220 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f12c001a070 con 0x7f12d0072320 2024-04-24T23:16:42.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:42.848+0000 7f12c7400640 1 --2- 172.21.15.6:0/785796220 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f12ac06be30 0x7f12ac06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:42.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:42.848+0000 7f12c7400640 1 -- 172.21.15.6:0/785796220 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6590+0+0 (secure 0 0 0) 0x7f12c008b080 con 0x7f12d0072320 2024-04-24T23:16:42.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:42.848+0000 7f12c7e00640 1 --2- 172.21.15.6:0/785796220 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f12ac06be30 0x7f12ac06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:42.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:42.849+0000 7f12c7e00640 1 --2- 172.21.15.6:0/785796220 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f12ac06be30 0x7f12ac06e2f0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f12b8034000 tx=0x7f12b802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:42.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:42.853+0000 7f12c7400640 1 -- 172.21.15.6:0/785796220 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f12d0068d50 con 0x7f12d0072320 2024-04-24T23:16:43.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.056+0000 7f12d5200640 1 -- 172.21.15.6:0/785796220 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f12d0075b30 con 0x7f12d0072320 2024-04-24T23:16:43.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.057+0000 7f12c7400640 1 -- 172.21.15.6:0/785796220 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+18554 (secure 0 0 0) 0x7f12d0147970 con 0x7f12d0072320 2024-04-24T23:16:43.060 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:16:43.060 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":26,"fsid":"18bef68d-5e6d-4ec5-862b-b6f206e2588d","created":"2024-04-24T23:15:12.535072+0000","modified":"2024-04-24T23:16:18.218008+0000","last_up_change":"2024-04-24T23:15:23.773223+0000","last_in_change":"2024-04-24T23:15:14.905328+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T23:15:24.747441+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-24T23:15:34.051800+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-24T23:15:35.436870+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-24T23:15:39.034257+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"3b7a212d-d498-462e-90bf-ed493c803a09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":3022569437},{"type":"v1","addr":"172.21.15.6:6809","nonce":3022569437}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":3022569437},{"type":"v1","addr":"172.21.15.6:6811","nonce":3022569437}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":3022569437},{"type":"v1","addr":"172.21.15.6:6815","nonce":3022569437}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":3022569437},{"type":"v1","addr":"172.21.15.6:6813","nonce":3022569437}]},"public_addr":"172.21.15.6:6809/3022569437","cluster_addr":"172.21.15.6:6811/3022569437","heartbeat_back_addr":"172.21.15.6:6815/3022569437","heartbeat_front_addr":"172.21.15.6:6813/3022569437","state":["exists","up"]},{"osd":1,"uuid":"7c97f5a2-1735-4775-9f15-eb50099b19d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":3687836888},{"type":"v1","addr":"172.21.15.6:6817","nonce":3687836888}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":3687836888},{"type":"v1","addr":"172.21.15.6:6819","nonce":3687836888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":3687836888},{"type":"v1","addr":"172.21.15.6:6823","nonce":3687836888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":3687836888},{"type":"v1","addr":"172.21.15.6:6821","nonce":3687836888}]},"public_addr":"172.21.15.6:6817/3687836888","cluster_addr":"172.21.15.6:6819/3687836888","heartbeat_back_addr":"172.21.15.6:6823/3687836888","heartbeat_front_addr":"172.21.15.6:6821/3687836888","state":["exists","up"]},{"osd":2,"uuid":"d86bbc18-438c-4ec2-8018-ba7dd78f790b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":1020213724},{"type":"v1","addr":"172.21.15.6:6801","nonce":1020213724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":1020213724},{"type":"v1","addr":"172.21.15.6:6803","nonce":1020213724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":1020213724},{"type":"v1","addr":"172.21.15.6:6807","nonce":1020213724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":1020213724},{"type":"v1","addr":"172.21.15.6:6805","nonce":1020213724}]},"public_addr":"172.21.15.6:6801/1020213724","cluster_addr":"172.21.15.6:6803/1020213724","heartbeat_back_addr":"172.21.15.6:6807/1020213724","heartbeat_front_addr":"172.21.15.6:6805/1020213724","state":["exists","up"]},{"osd":3,"uuid":"3754ae16-4248-40dd-95f9-0f59c0eb5426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6824","nonce":3163883468},{"type":"v1","addr":"172.21.15.6:6825","nonce":3163883468}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":3163883468},{"type":"v1","addr":"172.21.15.6:6827","nonce":3163883468}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":3163883468},{"type":"v1","addr":"172.21.15.6:6831","nonce":3163883468}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":3163883468},{"type":"v1","addr":"172.21.15.6:6829","nonce":3163883468}]},"public_addr":"172.21.15.6:6825/3163883468","cluster_addr":"172.21.15.6:6827/3163883468","heartbeat_back_addr":"172.21.15.6:6831/3163883468","heartbeat_front_addr":"172.21.15.6:6829/3163883468","state":["exists","up"]},{"osd":4,"uuid":"8b2913a0-d006-479c-84ed-5b21e0e6c392","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":1989500934},{"type":"v1","addr":"172.21.15.190:6809","nonce":1989500934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":1989500934},{"type":"v1","addr":"172.21.15.190:6811","nonce":1989500934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":1989500934},{"type":"v1","addr":"172.21.15.190:6815","nonce":1989500934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":1989500934},{"type":"v1","addr":"172.21.15.190:6813","nonce":1989500934}]},"public_addr":"172.21.15.190:6809/1989500934","cluster_addr":"172.21.15.190:6811/1989500934","heartbeat_back_addr":"172.21.15.190:6815/1989500934","heartbeat_front_addr":"172.21.15.190:6813/1989500934","state":["exists","up"]},{"osd":5,"uuid":"499b79d0-0585-4dae-ab69-196a551793da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":3687548412},{"type":"v1","addr":"172.21.15.190:6801","nonce":3687548412}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":3687548412},{"type":"v1","addr":"172.21.15.190:6803","nonce":3687548412}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":3687548412},{"type":"v1","addr":"172.21.15.190:6807","nonce":3687548412}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":3687548412},{"type":"v1","addr":"172.21.15.190:6805","nonce":3687548412}]},"public_addr":"172.21.15.190:6801/3687548412","cluster_addr":"172.21.15.190:6803/3687548412","heartbeat_back_addr":"172.21.15.190:6807/3687548412","heartbeat_front_addr":"172.21.15.190:6805/3687548412","state":["exists","up"]},{"osd":6,"uuid":"6ba6df3b-4a6b-4a89-a890-0f79f387d4d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":870571564},{"type":"v1","addr":"172.21.15.190:6817","nonce":870571564}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":870571564},{"type":"v1","addr":"172.21.15.190:6819","nonce":870571564}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":870571564},{"type":"v1","addr":"172.21.15.190:6823","nonce":870571564}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":870571564},{"type":"v1","addr":"172.21.15.190:6821","nonce":870571564}]},"public_addr":"172.21.15.190:6817/870571564","cluster_addr":"172.21.15.190:6819/870571564","heartbeat_back_addr":"172.21.15.190:6823/870571564","heartbeat_front_addr":"172.21.15.190:6821/870571564","state":["exists","up"]},{"osd":7,"uuid":"e9668b1c-a000-4332-86e8-4efd73b02409","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":123639197},{"type":"v1","addr":"172.21.15.190:6825","nonce":123639197}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":123639197},{"type":"v1","addr":"172.21.15.190:6827","nonce":123639197}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":123639197},{"type":"v1","addr":"172.21.15.190:6831","nonce":123639197}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":123639197},{"type":"v1","addr":"172.21.15.190:6829","nonce":123639197}]},"public_addr":"172.21.15.190:6825/123639197","cluster_addr":"172.21.15.190:6827/123639197","heartbeat_back_addr":"172.21.15.190:6831/123639197","heartbeat_front_addr":"172.21.15.190:6829/123639197","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.321646+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.363873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.466963+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.531989+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:19.990707+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.043543+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.032492+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.115004+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.2f","mappings":[{"from":0,"to":7}]},{"pgid":"4.19","mappings":[{"from":7,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T23:16:43.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.060+0000 7f12d5200640 1 -- 172.21.15.6:0/785796220 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f12ac06be30 msgr2=0x7f12ac06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:43.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.060+0000 7f12d5200640 1 --2- 172.21.15.6:0/785796220 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f12ac06be30 0x7f12ac06e2f0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f12b8034000 tx=0x7f12b802e000 comp rx=0 tx=0).stop 2024-04-24T23:16:43.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.060+0000 7f12d5200640 1 -- 172.21.15.6:0/785796220 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f12d0072320 msgr2=0x7f12d01f3a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:43.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.060+0000 7f12d5200640 1 --2- 172.21.15.6:0/785796220 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f12d0072320 0x7f12d01f3a70 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f12c000e960 tx=0x7f12c000ee20 comp rx=0 tx=0).stop 2024-04-24T23:16:43.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.060+0000 7f12d5200640 1 -- 172.21.15.6:0/785796220 shutdown_connections 2024-04-24T23:16:43.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.060+0000 7f12d5200640 1 --2- 172.21.15.6:0/785796220 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f12ac06be30 0x7f12ac06e2f0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:43.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.060+0000 7f12d5200640 1 --2- 172.21.15.6:0/785796220 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f12d00c6340 0x7f12d0146c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:43.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.060+0000 7f12d5200640 1 --2- 172.21.15.6:0/785796220 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f12d0151be0 0x7f12d0144930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:43.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.060+0000 7f12d5200640 1 --2- 172.21.15.6:0/785796220 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f12d0072320 0x7f12d01f3a70 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:43.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.060+0000 7f12d5200640 1 -- 172.21.15.6:0/785796220 >> 172.21.15.6:0/785796220 conn(0x7f12d006ed80 msgr2=0x7f12d00ca960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:43.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.060+0000 7f12d5200640 1 -- 172.21.15.6:0/785796220 shutdown_connections 2024-04-24T23:16:43.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.060+0000 7f12d5200640 1 -- 172.21.15.6:0/785796220 wait complete. 2024-04-24T23:16:43.081 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T23:15:24.747441+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-24T23:15:34.051800+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-24T23:15:35.436870+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-24T23:15:39.034257+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-24T23:16:43.081 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-24T23:16:43.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.295+0000 7f8dc1e00640 1 Processor -- start 2024-04-24T23:16:43.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.295+0000 7f8dc1e00640 1 -- start start 2024-04-24T23:16:43.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.297+0000 7f8dc1e00640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f8dbc152bb0 0x7f8dbc0c6ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:43.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.297+0000 7f8dc1e00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8dbc0cca80 con 0x7f8dbc0c5e40 2024-04-24T23:16:43.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.297+0000 7f8dc1e00640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8dbc0ccd60 con 0x7f8dbc0c5a60 2024-04-24T23:16:43.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.297+0000 7f8dc1e00640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f8dbc15c440 con 0x7f8dbc152bb0 2024-04-24T23:16:43.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.297+0000 7f8dbbe00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f8dbc0c5e40 0x7f8dbc1524a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:40312/0 (socket says 172.21.15.6:40312) 2024-04-24T23:16:43.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.297+0000 7f8dbbe00640 1 -- 172.21.15.6:0/108918437 learned_addr learned my addr 172.21.15.6:0/108918437 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:16:43.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.297+0000 7f8dc1400640 1 --2- 172.21.15.6:0/108918437 >> v2:172.21.15.190:3300/0 conn(0x7f8dbc152bb0 0x7f8dbc0c6ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:43.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.298+0000 7f8dbb400640 1 -- 172.21.15.6:0/108918437 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1171729396 0 0) 0x7f8dbc0cca80 con 0x7f8dbc0c5e40 2024-04-24T23:16:43.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.298+0000 7f8dbb400640 1 -- 172.21.15.6:0/108918437 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8da4003520 con 0x7f8dbc0c5e40 2024-04-24T23:16:43.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.298+0000 7f8dc1400640 1 -- 172.21.15.6:0/108918437 >> v1:172.21.15.190:6789/0 conn(0x7f8dbc0c5a60 legacy=0x7f8dbc15bd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:43.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.298+0000 7f8dc1400640 1 -- 172.21.15.6:0/108918437 >> v1:172.21.15.6:6789/0 conn(0x7f8dbc0c5e40 legacy=0x7f8dbc1524a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:43.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.298+0000 7f8dc1400640 1 -- 172.21.15.6:0/108918437 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8dbc06eab0 con 0x7f8dbc152bb0 2024-04-24T23:16:43.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.299+0000 7f8dc1400640 1 --2- 172.21.15.6:0/108918437 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8dbc152bb0 0x7f8dbc0c6ae0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f8db4009d90 tx=0x7f8db402a6f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ca032221ff62a388 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:43.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.300+0000 7f8dbb400640 1 -- 172.21.15.6:0/108918437 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8db4007740 con 0x7f8dbc152bb0 2024-04-24T23:16:43.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.300+0000 7f8dbb400640 1 -- 172.21.15.6:0/108918437 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8db40078e0 con 0x7f8dbc152bb0 2024-04-24T23:16:43.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.300+0000 7f8dbb400640 1 -- 172.21.15.6:0/108918437 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8db4007c90 con 0x7f8dbc152bb0 2024-04-24T23:16:43.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.301+0000 7f8dc1e00640 1 -- 172.21.15.6:0/108918437 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8dbc152bb0 msgr2=0x7f8dbc0c6ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:43.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.301+0000 7f8dc1e00640 1 --2- 172.21.15.6:0/108918437 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8dbc152bb0 0x7f8dbc0c6ae0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f8db4009d90 tx=0x7f8db402a6f0 comp rx=0 tx=0).stop 2024-04-24T23:16:43.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.301+0000 7f8dc1e00640 1 -- 172.21.15.6:0/108918437 shutdown_connections 2024-04-24T23:16:43.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.301+0000 7f8dc1e00640 1 --2- 172.21.15.6:0/108918437 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8dbc152bb0 0x7f8dbc0c6ae0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:43.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.301+0000 7f8dc1e00640 1 -- 172.21.15.6:0/108918437 >> 172.21.15.6:0/108918437 conn(0x7f8dbc06cc40 msgr2=0x7f8dbc06d050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:43.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.301+0000 7f8dc1e00640 1 -- 172.21.15.6:0/108918437 shutdown_connections 2024-04-24T23:16:43.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.301+0000 7f8dc1e00640 1 -- 172.21.15.6:0/108918437 wait complete. 2024-04-24T23:16:43.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.303+0000 7f8dc1e00640 1 Processor -- start 2024-04-24T23:16:43.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.303+0000 7f8dc1e00640 1 -- start start 2024-04-24T23:16:43.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.304+0000 7f8dc1e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8dbc0c5a60 0x7f8dbc14f600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:43.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.305+0000 7f8dc1e00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8dbc0c5e40 0x7f8dbc153bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:43.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.305+0000 7f8dc1e00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8dbc152bb0 0x7f8dbc155a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:43.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.305+0000 7f8dc1e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f8dbc06f590 con 0x7f8dbc0c5a60 2024-04-24T23:16:43.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.305+0000 7f8dc1e00640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f8dbc06f070 con 0x7f8dbc152bb0 2024-04-24T23:16:43.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.305+0000 7f8dc1e00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f8dbc153720 con 0x7f8dbc0c5e40 2024-04-24T23:16:43.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.305+0000 7f8dbbe00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8dbc0c5e40 0x7f8dbc153bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:43.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.305+0000 7f8dc1400640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8dbc152bb0 0x7f8dbc155a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:43.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.305+0000 7f8dc1400640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8dbc152bb0 0x7f8dbc155a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:35514/0 (socket says 172.21.15.6:35514) 2024-04-24T23:16:43.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.305+0000 7f8dbbe00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8dbc0c5e40 0x7f8dbc153bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.6:42972/0 (socket says 172.21.15.6:42972) 2024-04-24T23:16:43.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.305+0000 7f8dc1400640 1 -- 172.21.15.6:0/1685944299 learned_addr learned my addr 172.21.15.6:0/1685944299 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:16:43.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.305+0000 7f8dc0a00640 1 --2- 172.21.15.6:0/1685944299 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8dbc0c5a60 0x7f8dbc14f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:43.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.305+0000 7f8dbbe00640 1 -- 172.21.15.6:0/1685944299 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8dbc152bb0 msgr2=0x7f8dbc155a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:43.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.305+0000 7f8dbbe00640 1 --2- 172.21.15.6:0/1685944299 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8dbc152bb0 0x7f8dbc155a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:43.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.305+0000 7f8dbbe00640 1 -- 172.21.15.6:0/1685944299 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8dbc0c5a60 msgr2=0x7f8dbc14f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:43.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.305+0000 7f8dbbe00640 1 --2- 172.21.15.6:0/1685944299 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8dbc0c5a60 0x7f8dbc14f600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:43.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.305+0000 7f8dbbe00640 1 -- 172.21.15.6:0/1685944299 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8dbc155510 con 0x7f8dbc0c5e40 2024-04-24T23:16:43.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.306+0000 7f8dc0a00640 1 --2- 172.21.15.6:0/1685944299 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8dbc0c5a60 0x7f8dbc14f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:16:43.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.306+0000 7f8dc1400640 1 --2- 172.21.15.6:0/1685944299 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8dbc152bb0 0x7f8dbc155a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:16:43.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.306+0000 7f8dbbe00640 1 --2- 172.21.15.6:0/1685944299 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8dbc0c5e40 0x7f8dbc153bb0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f8da800c400 tx=0x7f8da800c8c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:43.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.306+0000 7f8db8c00640 1 -- 172.21.15.6:0/1685944299 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8da8017070 con 0x7f8dbc0c5e40 2024-04-24T23:16:43.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.306+0000 7f8db8c00640 1 -- 172.21.15.6:0/1685944299 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8da8005ce0 con 0x7f8dbc0c5e40 2024-04-24T23:16:43.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.306+0000 7f8db8c00640 1 -- 172.21.15.6:0/1685944299 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8da80072d0 con 0x7f8dbc0c5e40 2024-04-24T23:16:43.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.306+0000 7f8dc1e00640 1 -- 172.21.15.6:0/1685944299 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8dbc153110 con 0x7f8dbc0c5e40 2024-04-24T23:16:43.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.306+0000 7f8dc1e00640 1 -- 172.21.15.6:0/1685944299 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f8dbc0cb6c0 con 0x7f8dbc0c5e40 2024-04-24T23:16:43.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.307+0000 7f8dc1e00640 1 -- 172.21.15.6:0/1685944299 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8dbc075b30 con 0x7f8dbc0c5e40 2024-04-24T23:16:43.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.308+0000 7f8db8c00640 1 -- 172.21.15.6:0/1685944299 <== mon.2 v2:172.21.15.190:3301/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f8da801a070 con 0x7f8dbc0c5e40 2024-04-24T23:16:43.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.309+0000 7f8db8c00640 1 --2- 172.21.15.6:0/1685944299 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f8d9c06be30 0x7f8d9c06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:43.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.309+0000 7f8db8c00640 1 -- 172.21.15.6:0/1685944299 <== mon.2 v2:172.21.15.190:3301/0 5 ==== osd_map(26..26 src has 1..26) ==== 6590+0+0 (secure 0 0 0) 0x7f8da808a5b0 con 0x7f8dbc0c5e40 2024-04-24T23:16:43.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.310+0000 7f8dc0a00640 1 --2- 172.21.15.6:0/1685944299 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f8d9c06be30 0x7f8d9c06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:43.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.310+0000 7f8dc0a00640 1 --2- 172.21.15.6:0/1685944299 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f8d9c06be30 0x7f8d9c06e2f0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f8db0001b70 tx=0x7f8db0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:43.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.314+0000 7f8db8c00640 1 -- 172.21.15.6:0/1685944299 <== mon.2 v2:172.21.15.190:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f8da8058470 con 0x7f8dbc0c5e40 2024-04-24T23:16:43.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.522+0000 7f8dc1e00640 1 -- 172.21.15.6:0/1685944299 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f8dbc06f710 con 0x7f8dbc0c5e40 2024-04-24T23:16:43.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.523+0000 7f8db8c00640 1 -- 172.21.15.6:0/1685944299 <== mon.2 v2:172.21.15.190:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) ==== 93+0+10 (secure 0 0 0) 0x7f8da805c770 con 0x7f8dbc0c5e40 2024-04-24T23:16:43.524 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-04-24T23:16:43.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.526+0000 7f8dc1e00640 1 -- 172.21.15.6:0/1685944299 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f8d9c06be30 msgr2=0x7f8d9c06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:43.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.526+0000 7f8dc1e00640 1 --2- 172.21.15.6:0/1685944299 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f8d9c06be30 0x7f8d9c06e2f0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f8db0001b70 tx=0x7f8db0009040 comp rx=0 tx=0).stop 2024-04-24T23:16:43.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.526+0000 7f8dc1e00640 1 -- 172.21.15.6:0/1685944299 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8dbc0c5e40 msgr2=0x7f8dbc153bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:43.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.526+0000 7f8dc1e00640 1 --2- 172.21.15.6:0/1685944299 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8dbc0c5e40 0x7f8dbc153bb0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f8da800c400 tx=0x7f8da800c8c0 comp rx=0 tx=0).stop 2024-04-24T23:16:43.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.526+0000 7f8dc1e00640 1 -- 172.21.15.6:0/1685944299 shutdown_connections 2024-04-24T23:16:43.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.526+0000 7f8dc1e00640 1 --2- 172.21.15.6:0/1685944299 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f8d9c06be30 0x7f8d9c06e2f0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:43.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.526+0000 7f8dc1e00640 1 --2- 172.21.15.6:0/1685944299 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8dbc0c5e40 0x7f8dbc153bb0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:43.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.526+0000 7f8dc1e00640 1 --2- 172.21.15.6:0/1685944299 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8dbc152bb0 0x7f8dbc155a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:43.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.526+0000 7f8dc1e00640 1 --2- 172.21.15.6:0/1685944299 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8dbc0c5a60 0x7f8dbc14f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:43.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.526+0000 7f8dc1e00640 1 -- 172.21.15.6:0/1685944299 >> 172.21.15.6:0/1685944299 conn(0x7f8dbc06cc40 msgr2=0x7f8dbc156c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:43.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.526+0000 7f8dc1e00640 1 -- 172.21.15.6:0/1685944299 shutdown_connections 2024-04-24T23:16:43.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.526+0000 7f8dc1e00640 1 -- 172.21.15.6:0/1685944299 wait complete. 2024-04-24T23:16:43.547 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-24T23:16:43.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.755+0000 7f299de00640 1 Processor -- start 2024-04-24T23:16:43.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.756+0000 7f299de00640 1 -- start start 2024-04-24T23:16:43.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.757+0000 7f299de00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f299806eb80 0x7f29980c9330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:43.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.757+0000 7f299de00640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f299815ed90 0x7f2998162a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:43.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.758+0000 7f299de00640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f299815f240 con 0x7f299806ef60 2024-04-24T23:16:43.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.758+0000 7f299de00640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f29980c8bd0 con 0x7f299815ed90 2024-04-24T23:16:43.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.758+0000 7f299de00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f299815f520 con 0x7f299806eb80 2024-04-24T23:16:43.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.758+0000 7f299ca00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f299806eb80 0x7f29980c9330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:43.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.758+0000 7f299ca00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f299806eb80 0x7f29980c9330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.6:42984/0 (socket says 172.21.15.6:42984) 2024-04-24T23:16:43.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.758+0000 7f299ca00640 1 -- 172.21.15.6:0/257612140 learned_addr learned my addr 172.21.15.6:0/257612140 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:16:43.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.758+0000 7f298fe00640 1 --2- 172.21.15.6:0/257612140 >> v2:172.21.15.190:3300/0 conn(0x7f299815ed90 0x7f2998162a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:43.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.759+0000 7f299ca00640 1 -- 172.21.15.6:0/257612140 >> v2:172.21.15.190:3300/0 conn(0x7f299815ed90 msgr2=0x7f2998162a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:16:43.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.759+0000 7f299ca00640 1 --2- 172.21.15.6:0/257612140 >> v2:172.21.15.190:3300/0 conn(0x7f299815ed90 0x7f2998162a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:43.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.759+0000 7f299ca00640 1 -- 172.21.15.6:0/257612140 >> v1:172.21.15.190:6789/0 conn(0x7f299806ef60 legacy=0x7f2998158b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:43.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.759+0000 7f299ca00640 1 -- 172.21.15.6:0/257612140 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f299815f800 con 0x7f299806eb80 2024-04-24T23:16:43.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.759+0000 7f298fe00640 1 --2- 172.21.15.6:0/257612140 >> v2:172.21.15.190:3300/0 conn(0x7f299815ed90 0x7f2998162a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:16:43.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.760+0000 7f299ca00640 1 --2- 172.21.15.6:0/257612140 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f299806eb80 0x7f29980c9330 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f2988009e50 tx=0x7f298802a7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=74bf67a2b15e1e1e server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:43.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.760+0000 7f2997e00640 1 -- 172.21.15.6:0/257612140 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2988036070 con 0x7f299806eb80 2024-04-24T23:16:43.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.760+0000 7f2997e00640 1 -- 172.21.15.6:0/257612140 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2988031600 con 0x7f299806eb80 2024-04-24T23:16:43.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.760+0000 7f2997e00640 1 -- 172.21.15.6:0/257612140 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29880319d0 con 0x7f299806eb80 2024-04-24T23:16:43.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.761+0000 7f299de00640 1 -- 172.21.15.6:0/257612140 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f299806eb80 msgr2=0x7f29980c9330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:43.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.761+0000 7f299de00640 1 --2- 172.21.15.6:0/257612140 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f299806eb80 0x7f29980c9330 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f2988009e50 tx=0x7f298802a7a0 comp rx=0 tx=0).stop 2024-04-24T23:16:43.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.761+0000 7f299de00640 1 -- 172.21.15.6:0/257612140 shutdown_connections 2024-04-24T23:16:43.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.761+0000 7f299de00640 1 --2- 172.21.15.6:0/257612140 >> v2:172.21.15.190:3300/0 conn(0x7f299815ed90 0x7f2998162a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:43.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.761+0000 7f299de00640 1 --2- 172.21.15.6:0/257612140 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f299806eb80 0x7f29980c9330 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:43.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.761+0000 7f299de00640 1 -- 172.21.15.6:0/257612140 >> 172.21.15.6:0/257612140 conn(0x7f299806cc40 msgr2=0x7f299806d050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:43.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.761+0000 7f299de00640 1 -- 172.21.15.6:0/257612140 shutdown_connections 2024-04-24T23:16:43.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.761+0000 7f299de00640 1 -- 172.21.15.6:0/257612140 wait complete. 2024-04-24T23:16:43.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.762+0000 7f299de00640 1 Processor -- start 2024-04-24T23:16:43.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.763+0000 7f299de00640 1 -- start start 2024-04-24T23:16:43.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.763+0000 7f299de00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f299806eb80 0x7f29981f4fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:43.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.763+0000 7f299de00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f299806ef60 0x7f29980cc790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:43.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.764+0000 7f299de00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f299815ed90 0x7f299815a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:43.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.764+0000 7f299de00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f2998159280 con 0x7f299806ef60 2024-04-24T23:16:43.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.764+0000 7f299de00640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f2998162520 con 0x7f299815ed90 2024-04-24T23:16:43.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.764+0000 7f299de00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f2998159400 con 0x7f299806eb80 2024-04-24T23:16:43.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.764+0000 7f298fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f299806ef60 0x7f29980cc790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:43.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.764+0000 7f298fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f299806ef60 0x7f29980cc790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:35756/0 (socket says 172.21.15.6:35756) 2024-04-24T23:16:43.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.764+0000 7f298fe00640 1 -- 172.21.15.6:0/1047544218 learned_addr learned my addr 172.21.15.6:0/1047544218 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:16:43.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.764+0000 7f299ca00640 1 --2- 172.21.15.6:0/1047544218 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f299806eb80 0x7f29981f4fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:43.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.764+0000 7f299d400640 1 --2- 172.21.15.6:0/1047544218 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f299815ed90 0x7f299815a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:43.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.764+0000 7f298fe00640 1 -- 172.21.15.6:0/1047544218 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f299806eb80 msgr2=0x7f29981f4fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:43.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.764+0000 7f298fe00640 1 --2- 172.21.15.6:0/1047544218 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f299806eb80 0x7f29981f4fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:43.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.764+0000 7f298fe00640 1 -- 172.21.15.6:0/1047544218 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f299815ed90 msgr2=0x7f299815a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:43.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.764+0000 7f298fe00640 1 --2- 172.21.15.6:0/1047544218 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f299815ed90 0x7f299815a720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:43.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.764+0000 7f298fe00640 1 -- 172.21.15.6:0/1047544218 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2998159f60 con 0x7f299806ef60 2024-04-24T23:16:43.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.765+0000 7f298fe00640 1 --2- 172.21.15.6:0/1047544218 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f299806ef60 0x7f29980cc790 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f298000e810 tx=0x7f298000ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:43.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.765+0000 7f2995600640 1 -- 172.21.15.6:0/1047544218 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f298000f040 con 0x7f299806ef60 2024-04-24T23:16:43.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.765+0000 7f2995600640 1 -- 172.21.15.6:0/1047544218 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2980004030 con 0x7f299806ef60 2024-04-24T23:16:43.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.765+0000 7f299de00640 1 -- 172.21.15.6:0/1047544218 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29980c9ee0 con 0x7f299806ef60 2024-04-24T23:16:43.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.765+0000 7f2995600640 1 -- 172.21.15.6:0/1047544218 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2980004400 con 0x7f299806ef60 2024-04-24T23:16:43.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.765+0000 7f299de00640 1 -- 172.21.15.6:0/1047544218 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29981f5900 con 0x7f299806ef60 2024-04-24T23:16:43.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.767+0000 7f299de00640 1 -- 172.21.15.6:0/1047544218 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f29980c7a60 con 0x7f299806ef60 2024-04-24T23:16:43.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.767+0000 7f2995600640 1 -- 172.21.15.6:0/1047544218 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f298000c6d0 con 0x7f299806ef60 2024-04-24T23:16:43.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.768+0000 7f2995600640 1 --2- 172.21.15.6:0/1047544218 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f297406be60 0x7f297406e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:43.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.768+0000 7f2995600640 1 -- 172.21.15.6:0/1047544218 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6590+0+0 (secure 0 0 0) 0x7f298008af00 con 0x7f299806ef60 2024-04-24T23:16:43.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.769+0000 7f299ca00640 1 --2- 172.21.15.6:0/1047544218 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f297406be60 0x7f297406e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:43.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.769+0000 7f299ca00640 1 --2- 172.21.15.6:0/1047544218 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f297406be60 0x7f297406e320 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f2988033000 tx=0x7f2988009000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:43.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.774+0000 7f2995600640 1 -- 172.21.15.6:0/1047544218 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f29980c7a60 con 0x7f299806ef60 2024-04-24T23:16:43.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.978+0000 7f299de00640 1 -- 172.21.15.6:0/1047544218 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f2998075b30 con 0x7f299806ef60 2024-04-24T23:16:43.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.979+0000 7f2995600640 1 -- 172.21.15.6:0/1047544218 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) ==== 104+0+11 (secure 0 0 0) 0x7f2980017020 con 0x7f299806ef60 2024-04-24T23:16:43.980 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 32 2024-04-24T23:16:43.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.981+0000 7f299de00640 1 -- 172.21.15.6:0/1047544218 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f297406be60 msgr2=0x7f297406e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:43.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.981+0000 7f299de00640 1 --2- 172.21.15.6:0/1047544218 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f297406be60 0x7f297406e320 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f2988033000 tx=0x7f2988009000 comp rx=0 tx=0).stop 2024-04-24T23:16:43.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.981+0000 7f299de00640 1 -- 172.21.15.6:0/1047544218 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f299806ef60 msgr2=0x7f29980cc790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:43.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.981+0000 7f299de00640 1 --2- 172.21.15.6:0/1047544218 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f299806ef60 0x7f29980cc790 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f298000e810 tx=0x7f298000ecd0 comp rx=0 tx=0).stop 2024-04-24T23:16:43.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.981+0000 7f299de00640 1 -- 172.21.15.6:0/1047544218 shutdown_connections 2024-04-24T23:16:43.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.981+0000 7f299de00640 1 --2- 172.21.15.6:0/1047544218 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f297406be60 0x7f297406e320 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:43.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.981+0000 7f299de00640 1 --2- 172.21.15.6:0/1047544218 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f299815ed90 0x7f299815a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:43.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.981+0000 7f299de00640 1 --2- 172.21.15.6:0/1047544218 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f299806ef60 0x7f29980cc790 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:43.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.981+0000 7f299de00640 1 --2- 172.21.15.6:0/1047544218 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f299806eb80 0x7f29981f4fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:43.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.981+0000 7f299de00640 1 -- 172.21.15.6:0/1047544218 >> 172.21.15.6:0/1047544218 conn(0x7f299806cc40 msgr2=0x7f299815e0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:43.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.982+0000 7f299de00640 1 -- 172.21.15.6:0/1047544218 shutdown_connections 2024-04-24T23:16:43.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:43.982+0000 7f299de00640 1 -- 172.21.15.6:0/1047544218 wait complete. 2024-04-24T23:16:44.003 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-24T23:16:44.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:44.207+0000 7fbdcf200640 1 Processor -- start 2024-04-24T23:16:44.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:44.207+0000 7fbdcf200640 1 -- start start 2024-04-24T23:16:44.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:44.209+0000 7fbdcf200640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fbdd006eb80 0x7fbdd00c9330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:44.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:44.209+0000 7fbdcf200640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbdd015ed90 0x7fbdd0162a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:44.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:44.209+0000 7fbdcf200640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbdd015f240 con 0x7fbdd006ef60 2024-04-24T23:16:44.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:44.209+0000 7fbdcf200640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7fbdd00c8bd0 con 0x7fbdd006eb80 2024-04-24T23:16:44.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:44.209+0000 7fbdcf200640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fbdd015f520 con 0x7fbdd015ed90 2024-04-24T23:16:44.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:44.209+0000 7fbdcde00640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fbdd006eb80 0x7fbdd00c9330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:44.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:44.209+0000 7fbdcd400640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbdd015ed90 0x7fbdd0162a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:44.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:44.209+0000 7fbdcde00640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fbdd006eb80 0x7fbdd00c9330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:35538/0 (socket says 172.21.15.6:35538) 2024-04-24T23:16:44.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:44.209+0000 7fbdcde00640 1 -- 172.21.15.6:0/1393410606 learned_addr learned my addr 172.21.15.6:0/1393410606 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:16:44.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:44.209+0000 7fbdce800640 1 --1- 172.21.15.6:0/1393410606 >> v1:172.21.15.190:6789/0 conn(0x7fbdd006ef60 0x7fbdd0158b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.6:57894/0 (socket says 172.21.15.6:57894) 2024-04-24T23:16:44.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:44.210+0000 7fbdcde00640 1 -- 172.21.15.6:0/1393410606 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbdd015ed90 msgr2=0x7fbdd0162a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:16:44.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:44.210+0000 7fbdcde00640 1 --2- 172.21.15.6:0/1393410606 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbdd015ed90 0x7fbdd0162a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:44.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:44.210+0000 7fbdcde00640 1 -- 172.21.15.6:0/1393410606 >> v1:172.21.15.190:6789/0 conn(0x7fbdd006ef60 legacy=0x7fbdd0158b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:44.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:44.210+0000 7fbdcde00640 1 -- 172.21.15.6:0/1393410606 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbdd015f800 con 0x7fbdd006eb80 2024-04-24T23:16:44.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:44.211+0000 7fbdcde00640 1 --2- 172.21.15.6:0/1393410606 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbdd006eb80 0x7fbdd00c9330 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fbdc0009870 tx=0x7fbdc002a2e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=45d840e859bac94d server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:44.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:44.211+0000 7fbdcca00640 1 -- 172.21.15.6:0/1393410606 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdc0037070 con 0x7fbdd006eb80 2024-04-24T23:16:44.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:44.211+0000 7fbdcca00640 1 -- 172.21.15.6:0/1393410606 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbdc0032640 con 0x7fbdd006eb80 2024-04-24T23:16:44.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:44.211+0000 7fbdcca00640 1 -- 172.21.15.6:0/1393410606 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdc0032a10 con 0x7fbdd006eb80 2024-04-24T23:16:44.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:44.212+0000 7fbdcf200640 1 -- 172.21.15.6:0/1393410606 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbdd006eb80 msgr2=0x7fbdd00c9330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:44.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:44.212+0000 7fbdcf200640 1 --2- 172.21.15.6:0/1393410606 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbdd006eb80 0x7fbdd00c9330 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fbdc0009870 tx=0x7fbdc002a2e0 comp rx=0 tx=0).stop 2024-04-24T23:16:44.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:44.212+0000 7fbdcf200640 1 -- 172.21.15.6:0/1393410606 shutdown_connections 2024-04-24T23:16:44.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:44.212+0000 7fbdcf200640 1 --2- 172.21.15.6:0/1393410606 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbdd015ed90 0x7fbdd0162a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:44.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:44.212+0000 7fbdcf200640 1 --2- 172.21.15.6:0/1393410606 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbdd006eb80 0x7fbdd00c9330 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:44.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:44.212+0000 7fbdcf200640 1 -- 172.21.15.6:0/1393410606 >> 172.21.15.6:0/1393410606 conn(0x7fbdd006cc40 msgr2=0x7fbdd006d050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:44.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:44.212+0000 7fbdcf200640 1 -- 172.21.15.6:0/1393410606 shutdown_connections 2024-04-24T23:16:44.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:44.212+0000 7fbdcf200640 1 -- 172.21.15.6:0/1393410606 wait complete. 2024-04-24T23:16:44.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:44.214+0000 7fbdcf200640 1 Processor -- start 2024-04-24T23:16:44.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:44.214+0000 7fbdcf200640 1 -- start start 2024-04-24T23:16:44.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:44.215+0000 7fbdcf200640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbdd006eb80 0x7fbdd00720c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:44.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:44.215+0000 7fbdcf200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbdd006ef60 0x7fbdd0161550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:44.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:44.215+0000 7fbdcf200640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbdd015ed90 0x7fbdd01f4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:44.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:44.215+0000 7fbdcf200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fbdd0159280 con 0x7fbdd006ef60 2024-04-24T23:16:44.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:44.215+0000 7fbdcf200640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fbdd0162520 con 0x7fbdd006eb80 2024-04-24T23:16:44.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:44.215+0000 7fbdcf200640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fbdd0159400 con 0x7fbdd015ed90 2024-04-24T23:16:44.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:44.215+0000 7fbdcde00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbdd006eb80 0x7fbdd00720c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:44.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:44.215+0000 7fbdcde00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbdd006eb80 0x7fbdd00720c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:35544/0 (socket says 172.21.15.6:35544) 2024-04-24T23:16:44.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:44.215+0000 7fbdce800640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbdd015ed90 0x7fbdd01f4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:44.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:44.216+0000 7fbdcde00640 1 -- 172.21.15.6:0/2975114950 learned_addr learned my addr 172.21.15.6:0/2975114950 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:16:44.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:44.216+0000 7fbdcd400640 1 --2- 172.21.15.6:0/2975114950 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbdd006ef60 0x7fbdd0161550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:44.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:44.216+0000 7fbdcde00640 1 -- 172.21.15.6:0/2975114950 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbdd015ed90 msgr2=0x7fbdd01f4e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:44.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:44.216+0000 7fbdcde00640 1 --2- 172.21.15.6:0/2975114950 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbdd015ed90 0x7fbdd01f4e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:44.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:44.216+0000 7fbdcde00640 1 -- 172.21.15.6:0/2975114950 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbdd006ef60 msgr2=0x7fbdd0161550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:44.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:44.216+0000 7fbdcde00640 1 --2- 172.21.15.6:0/2975114950 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbdd006ef60 0x7fbdd0161550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:44.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:44.216+0000 7fbdcde00640 1 -- 172.21.15.6:0/2975114950 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbdd00ca1f0 con 0x7fbdd006eb80 2024-04-24T23:16:44.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:44.216+0000 7fbdcde00640 1 --2- 172.21.15.6:0/2975114950 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbdd006eb80 0x7fbdd00720c0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fbdc0000c00 tx=0x7fbdc00046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:44.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:44.217+0000 7fbdc6000640 1 -- 172.21.15.6:0/2975114950 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdc0037040 con 0x7fbdd006eb80 2024-04-24T23:16:44.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:44.217+0000 7fbdc6000640 1 -- 172.21.15.6:0/2975114950 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbdc0042070 con 0x7fbdd006eb80 2024-04-24T23:16:44.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:44.217+0000 7fbdc6000640 1 -- 172.21.15.6:0/2975114950 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdc00309b0 con 0x7fbdd006eb80 2024-04-24T23:16:44.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:44.217+0000 7fbdcf200640 1 -- 172.21.15.6:0/2975114950 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbdd00c3f80 con 0x7fbdd006eb80 2024-04-24T23:16:44.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:44.217+0000 7fbdcf200640 1 -- 172.21.15.6:0/2975114950 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbdd00c4240 con 0x7fbdd006eb80 2024-04-24T23:16:44.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:44.218+0000 7fbdcf200640 1 -- 172.21.15.6:0/2975114950 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbdd0075b30 con 0x7fbdd006eb80 2024-04-24T23:16:44.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:44.219+0000 7fbdc6000640 1 -- 172.21.15.6:0/2975114950 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fbdc0030b50 con 0x7fbdd006eb80 2024-04-24T23:16:44.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:44.220+0000 7fbdc6000640 1 --2- 172.21.15.6:0/2975114950 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fbdb006be60 0x7fbdb006e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:44.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:44.220+0000 7fbdc6000640 1 -- 172.21.15.6:0/2975114950 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6590+0+0 (secure 0 0 0) 0x7fbdc00ab890 con 0x7fbdd006eb80 2024-04-24T23:16:44.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:44.220+0000 7fbdcd400640 1 --2- 172.21.15.6:0/2975114950 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fbdb006be60 0x7fbdb006e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:44.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:44.221+0000 7fbdcd400640 1 --2- 172.21.15.6:0/2975114950 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fbdb006be60 0x7fbdb006e320 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fbdd0161e10 tx=0x7fbdbc009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:44.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:44.225+0000 7fbdc6000640 1 -- 172.21.15.6:0/2975114950 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fbdc0079870 con 0x7fbdd006eb80 2024-04-24T23:16:44.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:44.430+0000 7fbdcf200640 1 -- 172.21.15.6:0/2975114950 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fbdd00c9980 con 0x7fbdd006eb80 2024-04-24T23:16:44.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:44.432+0000 7fbdc6000640 1 -- 172.21.15.6:0/2975114950 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) ==== 100+0+11 (secure 0 0 0) 0x7fbdc007db70 con 0x7fbdd006eb80 2024-04-24T23:16:44.433 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 64 2024-04-24T23:16:44.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:44.434+0000 7fbdcf200640 1 -- 172.21.15.6:0/2975114950 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fbdb006be60 msgr2=0x7fbdb006e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:44.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:44.434+0000 7fbdcf200640 1 --2- 172.21.15.6:0/2975114950 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fbdb006be60 0x7fbdb006e320 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fbdd0161e10 tx=0x7fbdbc009190 comp rx=0 tx=0).stop 2024-04-24T23:16:44.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:44.434+0000 7fbdcf200640 1 -- 172.21.15.6:0/2975114950 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbdd006eb80 msgr2=0x7fbdd00720c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:44.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:44.434+0000 7fbdcf200640 1 --2- 172.21.15.6:0/2975114950 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbdd006eb80 0x7fbdd00720c0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fbdc0000c00 tx=0x7fbdc00046c0 comp rx=0 tx=0).stop 2024-04-24T23:16:44.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:44.434+0000 7fbdcf200640 1 -- 172.21.15.6:0/2975114950 shutdown_connections 2024-04-24T23:16:44.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:44.434+0000 7fbdcf200640 1 --2- 172.21.15.6:0/2975114950 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fbdb006be60 0x7fbdb006e320 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:44.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:44.434+0000 7fbdcf200640 1 --2- 172.21.15.6:0/2975114950 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbdd015ed90 0x7fbdd01f4e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:44.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:44.435+0000 7fbdcf200640 1 --2- 172.21.15.6:0/2975114950 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbdd006ef60 0x7fbdd0161550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:44.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:44.435+0000 7fbdcf200640 1 --2- 172.21.15.6:0/2975114950 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbdd006eb80 0x7fbdd00720c0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:44.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:44.435+0000 7fbdcf200640 1 -- 172.21.15.6:0/2975114950 >> 172.21.15.6:0/2975114950 conn(0x7fbdd006cc40 msgr2=0x7fbdd015d660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:44.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:44.435+0000 7fbdcf200640 1 -- 172.21.15.6:0/2975114950 shutdown_connections 2024-04-24T23:16:44.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:44.435+0000 7fbdcf200640 1 -- 172.21.15.6:0/2975114950 wait complete. 2024-04-24T23:16:44.455 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-24T23:16:44.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:44.667+0000 7f5826e00640 1 Processor -- start 2024-04-24T23:16:44.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:44.667+0000 7f5826e00640 1 -- start start 2024-04-24T23:16:44.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:44.669+0000 7f5826e00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5820154b70 0x7f582006f020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:44.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:44.669+0000 7f5826e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f5820071000 0x7f58200c80e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:44.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:44.669+0000 7f5826e00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5820072530 con 0x7f58200c7080 2024-04-24T23:16:44.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:44.669+0000 7f5826e00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f5820072a70 con 0x7f5820071000 2024-04-24T23:16:44.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:44.669+0000 7f5826e00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f58200727d0 con 0x7f5820154b70 2024-04-24T23:16:44.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:44.670+0000 7f581fe00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5820154b70 0x7f582006f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:44.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:44.670+0000 7f5817e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f58200c7080 0x7f582015a9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:40314/0 (socket says 172.21.15.6:40314) 2024-04-24T23:16:44.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:44.670+0000 7f5817e00640 1 -- 172.21.15.6:0/1818887550 learned_addr learned my addr 172.21.15.6:0/1818887550 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:16:44.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:44.670+0000 7f5824e00640 1 --2- 172.21.15.6:0/1818887550 >> v2:172.21.15.6:3300/0 conn(0x7f5820071000 0x7f58200c80e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:44.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:44.670+0000 7f581fe00640 1 -- 172.21.15.6:0/1818887550 >> v2:172.21.15.6:3300/0 conn(0x7f5820071000 msgr2=0x7f58200c80e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:16:44.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:44.670+0000 7f581fe00640 1 --2- 172.21.15.6:0/1818887550 >> v2:172.21.15.6:3300/0 conn(0x7f5820071000 0x7f58200c80e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:44.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:44.670+0000 7f581fe00640 1 -- 172.21.15.6:0/1818887550 >> v1:172.21.15.6:6789/0 conn(0x7f58200c7080 legacy=0x7f582015a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:44.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:44.670+0000 7f581fe00640 1 -- 172.21.15.6:0/1818887550 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f582006e540 con 0x7f5820154b70 2024-04-24T23:16:44.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:44.671+0000 7f581fe00640 1 --2- 172.21.15.6:0/1818887550 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5820154b70 0x7f582006f020 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f58100097a0 tx=0x7f581002a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=54085b97114a1b71 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:44.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:44.672+0000 7f581f400640 1 -- 172.21.15.6:0/1818887550 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5810037070 con 0x7f5820154b70 2024-04-24T23:16:44.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:44.672+0000 7f581f400640 1 -- 172.21.15.6:0/1818887550 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5810032440 con 0x7f5820154b70 2024-04-24T23:16:44.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:44.672+0000 7f581f400640 1 -- 172.21.15.6:0/1818887550 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5810032810 con 0x7f5820154b70 2024-04-24T23:16:44.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:44.672+0000 7f5826e00640 1 -- 172.21.15.6:0/1818887550 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5820154b70 msgr2=0x7f582006f020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:44.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:44.672+0000 7f5826e00640 1 --2- 172.21.15.6:0/1818887550 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5820154b70 0x7f582006f020 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f58100097a0 tx=0x7f581002a180 comp rx=0 tx=0).stop 2024-04-24T23:16:44.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:44.672+0000 7f5826e00640 1 -- 172.21.15.6:0/1818887550 shutdown_connections 2024-04-24T23:16:44.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:44.672+0000 7f5826e00640 1 --2- 172.21.15.6:0/1818887550 >> v2:172.21.15.6:3300/0 conn(0x7f5820071000 0x7f58200c80e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:44.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:44.672+0000 7f5826e00640 1 --2- 172.21.15.6:0/1818887550 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5820154b70 0x7f582006f020 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:44.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:44.672+0000 7f5826e00640 1 -- 172.21.15.6:0/1818887550 >> 172.21.15.6:0/1818887550 conn(0x7f582006b0e0 msgr2=0x7f582006b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:44.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:44.673+0000 7f5826e00640 1 -- 172.21.15.6:0/1818887550 shutdown_connections 2024-04-24T23:16:44.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:44.673+0000 7f5826e00640 1 -- 172.21.15.6:0/1818887550 wait complete. 2024-04-24T23:16:44.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:44.674+0000 7f5826e00640 1 Processor -- start 2024-04-24T23:16:44.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:44.674+0000 7f5826e00640 1 -- start start 2024-04-24T23:16:44.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:44.675+0000 7f5826e00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5820071000 0x7f58200c8a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:44.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:44.675+0000 7f5826e00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f58200c7080 0x7f5820159210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:44.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:44.675+0000 7f5826e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5820154b70 0x7f582015a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:44.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:44.675+0000 7f5826e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f5820154f50 con 0x7f5820154b70 2024-04-24T23:16:44.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:44.675+0000 7f5826e00640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f582006eb00 con 0x7f58200c7080 2024-04-24T23:16:44.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:44.675+0000 7f5826e00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f58201550d0 con 0x7f5820071000 2024-04-24T23:16:44.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:44.676+0000 7f5824e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5820154b70 0x7f582015a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:44.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:44.676+0000 7f5817e00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f58200c7080 0x7f5820159210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:44.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:44.676+0000 7f5824e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5820154b70 0x7f582015a780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:35778/0 (socket says 172.21.15.6:35778) 2024-04-24T23:16:44.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:44.676+0000 7f5824e00640 1 -- 172.21.15.6:0/1242506349 learned_addr learned my addr 172.21.15.6:0/1242506349 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:16:44.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:44.676+0000 7f581fe00640 1 --2- 172.21.15.6:0/1242506349 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5820071000 0x7f58200c8a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:44.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:44.676+0000 7f5824e00640 1 -- 172.21.15.6:0/1242506349 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5820071000 msgr2=0x7f58200c8a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:44.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:44.676+0000 7f5824e00640 1 --2- 172.21.15.6:0/1242506349 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5820071000 0x7f58200c8a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:44.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:44.676+0000 7f5824e00640 1 -- 172.21.15.6:0/1242506349 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f58200c7080 msgr2=0x7f5820159210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:44.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:44.676+0000 7f5824e00640 1 --2- 172.21.15.6:0/1242506349 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f58200c7080 0x7f5820159210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:44.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:44.676+0000 7f5824e00640 1 -- 172.21.15.6:0/1242506349 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58201f1070 con 0x7f5820154b70 2024-04-24T23:16:44.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:44.676+0000 7f5824e00640 1 --2- 172.21.15.6:0/1242506349 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5820154b70 0x7f582015a780 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f5818009dd0 tx=0x7f581800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:44.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:44.677+0000 7f581cc00640 1 -- 172.21.15.6:0/1242506349 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5818018020 con 0x7f5820154b70 2024-04-24T23:16:44.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:44.677+0000 7f581cc00640 1 -- 172.21.15.6:0/1242506349 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5818014480 con 0x7f5820154b70 2024-04-24T23:16:44.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:44.677+0000 7f581cc00640 1 -- 172.21.15.6:0/1242506349 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5818014870 con 0x7f5820154b70 2024-04-24T23:16:44.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:44.677+0000 7f5826e00640 1 -- 172.21.15.6:0/1242506349 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58200c6210 con 0x7f5820154b70 2024-04-24T23:16:44.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:44.677+0000 7f5826e00640 1 -- 172.21.15.6:0/1242506349 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58200c5c50 con 0x7f5820154b70 2024-04-24T23:16:44.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:44.678+0000 7f5826e00640 1 -- 172.21.15.6:0/1242506349 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f58200767b0 con 0x7f5820154b70 2024-04-24T23:16:44.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:44.679+0000 7f581cc00640 1 -- 172.21.15.6:0/1242506349 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f5818005d10 con 0x7f5820154b70 2024-04-24T23:16:44.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:44.680+0000 7f581cc00640 1 --2- 172.21.15.6:0/1242506349 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f57fc06be30 0x7f57fc06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:44.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:44.680+0000 7f581cc00640 1 -- 172.21.15.6:0/1242506349 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6590+0+0 (secure 0 0 0) 0x7f581808bcb0 con 0x7f5820154b70 2024-04-24T23:16:44.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:44.680+0000 7f581fe00640 1 --2- 172.21.15.6:0/1242506349 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f57fc06be30 0x7f57fc06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:44.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:44.681+0000 7f581fe00640 1 --2- 172.21.15.6:0/1242506349 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f57fc06be30 0x7f57fc06e2f0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f5810034000 tx=0x7f581002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:44.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:44.684+0000 7f581cc00640 1 -- 172.21.15.6:0/1242506349 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f5818022070 con 0x7f5820154b70 2024-04-24T23:16:44.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:44.883+0000 7f5826e00640 1 -- 172.21.15.6:0/1242506349 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f5820158620 con 0x7f5820154b70 2024-04-24T23:16:44.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:44.883+0000 7f581cc00640 1 -- 172.21.15.6:0/1242506349 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) ==== 103+0+11 (secure 0 0 0) 0x7f5818010040 con 0x7f5820154b70 2024-04-24T23:16:44.885 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 32 2024-04-24T23:16:44.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:44.886+0000 7f5826e00640 1 -- 172.21.15.6:0/1242506349 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f57fc06be30 msgr2=0x7f57fc06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:44.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:44.886+0000 7f5826e00640 1 --2- 172.21.15.6:0/1242506349 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f57fc06be30 0x7f57fc06e2f0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f5810034000 tx=0x7f581002e000 comp rx=0 tx=0).stop 2024-04-24T23:16:44.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:44.886+0000 7f5826e00640 1 -- 172.21.15.6:0/1242506349 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5820154b70 msgr2=0x7f582015a780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:44.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:44.886+0000 7f5826e00640 1 --2- 172.21.15.6:0/1242506349 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5820154b70 0x7f582015a780 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f5818009dd0 tx=0x7f581800c490 comp rx=0 tx=0).stop 2024-04-24T23:16:44.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:44.886+0000 7f5826e00640 1 -- 172.21.15.6:0/1242506349 shutdown_connections 2024-04-24T23:16:44.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:44.886+0000 7f5826e00640 1 --2- 172.21.15.6:0/1242506349 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f57fc06be30 0x7f57fc06e2f0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:44.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:44.886+0000 7f5826e00640 1 --2- 172.21.15.6:0/1242506349 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5820154b70 0x7f582015a780 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:44.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:44.886+0000 7f5826e00640 1 --2- 172.21.15.6:0/1242506349 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f58200c7080 0x7f5820159210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:44.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:44.886+0000 7f5826e00640 1 --2- 172.21.15.6:0/1242506349 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5820071000 0x7f58200c8a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:44.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:44.886+0000 7f5826e00640 1 -- 172.21.15.6:0/1242506349 >> 172.21.15.6:0/1242506349 conn(0x7f582006b0e0 msgr2=0x7f58200c8190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:44.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:44.887+0000 7f5826e00640 1 -- 172.21.15.6:0/1242506349 shutdown_connections 2024-04-24T23:16:44.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:44.887+0000 7f5826e00640 1 -- 172.21.15.6:0/1242506349 wait complete. 2024-04-24T23:16:44.907 INFO:tasks.mds_thrash:mds thrasher using random seed: 1714000604 2024-04-24T23:16:44.908 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T23:16:45.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:45.122+0000 7fc464a00640 1 Processor -- start 2024-04-24T23:16:45.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:45.122+0000 7fc464a00640 1 -- start start 2024-04-24T23:16:45.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:45.124+0000 7fc464a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc46006ce30 0x7fc4600cc600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:45.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:45.124+0000 7fc464a00640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fc4600c6be0 0x7fc4600ccb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:45.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:45.124+0000 7fc464a00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc46006d390 con 0x7fc460152bd0 2024-04-24T23:16:45.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:45.124+0000 7fc464a00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7fc46006d970 con 0x7fc46006ce30 2024-04-24T23:16:45.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:45.124+0000 7fc464a00640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7fc46006dbc0 con 0x7fc4600c6be0 2024-04-24T23:16:45.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:45.124+0000 7fc45ea00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc46006ce30 0x7fc4600cc600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:45.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:45.124+0000 7fc45ea00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc46006ce30 0x7fc4600cc600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36262/0 (socket says 172.21.15.6:36262) 2024-04-24T23:16:45.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:45.124+0000 7fc45ea00640 1 -- 172.21.15.6:0/4075845841 learned_addr learned my addr 172.21.15.6:0/4075845841 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:16:45.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:45.124+0000 7fc45fe00640 1 --2- 172.21.15.6:0/4075845841 >> v2:172.21.15.190:3300/0 conn(0x7fc4600c6be0 0x7fc4600ccb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:45.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:45.125+0000 7fc45ea00640 1 -- 172.21.15.6:0/4075845841 >> v2:172.21.15.190:3300/0 conn(0x7fc4600c6be0 msgr2=0x7fc4600ccb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:16:45.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:45.125+0000 7fc45ea00640 1 --2- 172.21.15.6:0/4075845841 >> v2:172.21.15.190:3300/0 conn(0x7fc4600c6be0 0x7fc4600ccb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:45.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:45.125+0000 7fc45ea00640 1 -- 172.21.15.6:0/4075845841 >> v1:172.21.15.6:6789/0 conn(0x7fc460152bd0 legacy=0x7fc460158e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:45.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:45.125+0000 7fc45ea00640 1 -- 172.21.15.6:0/4075845841 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc46006dea0 con 0x7fc46006ce30 2024-04-24T23:16:45.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:45.126+0000 7fc45ea00640 1 --2- 172.21.15.6:0/4075845841 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc46006ce30 0x7fc4600cc600 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fc460068130 tx=0x7fc44c02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=821f53dd24faa5e6 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:45.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:45.126+0000 7fc45e000640 1 -- 172.21.15.6:0/4075845841 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc44c037070 con 0x7fc46006ce30 2024-04-24T23:16:45.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:45.126+0000 7fc45e000640 1 -- 172.21.15.6:0/4075845841 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc44c032440 con 0x7fc46006ce30 2024-04-24T23:16:45.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:45.126+0000 7fc45e000640 1 -- 172.21.15.6:0/4075845841 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc44c032810 con 0x7fc46006ce30 2024-04-24T23:16:45.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:45.127+0000 7fc464a00640 1 -- 172.21.15.6:0/4075845841 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc46006ce30 msgr2=0x7fc4600cc600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:45.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:45.127+0000 7fc464a00640 1 --2- 172.21.15.6:0/4075845841 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc46006ce30 0x7fc4600cc600 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fc460068130 tx=0x7fc44c02a270 comp rx=0 tx=0).stop 2024-04-24T23:16:45.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:45.127+0000 7fc464a00640 1 -- 172.21.15.6:0/4075845841 shutdown_connections 2024-04-24T23:16:45.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:45.127+0000 7fc464a00640 1 --2- 172.21.15.6:0/4075845841 >> v2:172.21.15.190:3300/0 conn(0x7fc4600c6be0 0x7fc4600ccb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:45.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:45.127+0000 7fc464a00640 1 --2- 172.21.15.6:0/4075845841 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc46006ce30 0x7fc4600cc600 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:45.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:45.127+0000 7fc464a00640 1 -- 172.21.15.6:0/4075845841 >> 172.21.15.6:0/4075845841 conn(0x7fc46006ed80 msgr2=0x7fc46006f190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:45.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:45.127+0000 7fc464a00640 1 -- 172.21.15.6:0/4075845841 shutdown_connections 2024-04-24T23:16:45.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:45.127+0000 7fc464a00640 1 -- 172.21.15.6:0/4075845841 wait complete. 2024-04-24T23:16:45.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:45.129+0000 7fc464a00640 1 Processor -- start 2024-04-24T23:16:45.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:45.129+0000 7fc464a00640 1 -- start start 2024-04-24T23:16:45.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:45.130+0000 7fc464a00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc46006ce30 0x7fc4600c4800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:45.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:45.130+0000 7fc464a00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc4600c6be0 0x7fc4600c4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:45.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:45.130+0000 7fc464a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc460152bd0 0x7fc4601f3eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:45.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:45.130+0000 7fc464a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fc460072650 con 0x7fc460152bd0 2024-04-24T23:16:45.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:45.130+0000 7fc464a00640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fc460071890 con 0x7fc4600c6be0 2024-04-24T23:16:45.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:45.130+0000 7fc464a00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fc4600727d0 con 0x7fc46006ce30 2024-04-24T23:16:45.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:45.131+0000 7fc45ea00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc4600c6be0 0x7fc4600c4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:45.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:45.131+0000 7fc45f400640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc46006ce30 0x7fc4600c4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:45.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:45.131+0000 7fc45fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc460152bd0 0x7fc4601f3eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:45.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:45.131+0000 7fc45ea00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc4600c6be0 0x7fc4600c4d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:36574/0 (socket says 172.21.15.6:36574) 2024-04-24T23:16:45.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:45.131+0000 7fc45ea00640 1 -- 172.21.15.6:0/3424017993 learned_addr learned my addr 172.21.15.6:0/3424017993 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:16:45.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:45.131+0000 7fc45ea00640 1 -- 172.21.15.6:0/3424017993 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc46006ce30 msgr2=0x7fc4600c4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:45.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:45.131+0000 7fc45ea00640 1 --2- 172.21.15.6:0/3424017993 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc46006ce30 0x7fc4600c4800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:45.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:45.131+0000 7fc45ea00640 1 -- 172.21.15.6:0/3424017993 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc460152bd0 msgr2=0x7fc4601f3eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:45.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:45.131+0000 7fc45ea00640 1 --2- 172.21.15.6:0/3424017993 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc460152bd0 0x7fc4601f3eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:45.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:45.131+0000 7fc45ea00640 1 -- 172.21.15.6:0/3424017993 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4600ca290 con 0x7fc4600c6be0 2024-04-24T23:16:45.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:45.131+0000 7fc45ea00640 1 --2- 172.21.15.6:0/3424017993 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc4600c6be0 0x7fc4600c4d40 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fc44c02a2b0 tx=0x7fc44c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:45.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:45.132+0000 7fc457400640 1 -- 172.21.15.6:0/3424017993 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc44c037040 con 0x7fc4600c6be0 2024-04-24T23:16:45.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:45.132+0000 7fc457400640 1 -- 172.21.15.6:0/3424017993 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc44c042070 con 0x7fc4600c6be0 2024-04-24T23:16:45.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:45.132+0000 7fc457400640 1 -- 172.21.15.6:0/3424017993 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc44c030880 con 0x7fc4600c6be0 2024-04-24T23:16:45.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:45.132+0000 7fc464a00640 1 -- 172.21.15.6:0/3424017993 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4600ca510 con 0x7fc4600c6be0 2024-04-24T23:16:45.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:45.132+0000 7fc464a00640 1 -- 172.21.15.6:0/3424017993 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4601469f0 con 0x7fc4600c6be0 2024-04-24T23:16:45.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:45.133+0000 7fc464a00640 1 -- 172.21.15.6:0/3424017993 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc4601503f0 con 0x7fc4600c6be0 2024-04-24T23:16:45.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:45.134+0000 7fc457400640 1 -- 172.21.15.6:0/3424017993 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fc44c030a20 con 0x7fc4600c6be0 2024-04-24T23:16:45.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:45.135+0000 7fc457400640 1 --2- 172.21.15.6:0/3424017993 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fc44006be30 0x7fc44006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:45.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:45.135+0000 7fc457400640 1 -- 172.21.15.6:0/3424017993 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6590+0+0 (secure 0 0 0) 0x7fc44c0abc30 con 0x7fc4600c6be0 2024-04-24T23:16:45.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:45.135+0000 7fc45f400640 1 --2- 172.21.15.6:0/3424017993 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fc44006be30 0x7fc44006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:45.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:45.136+0000 7fc45f400640 1 --2- 172.21.15.6:0/3424017993 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fc44006be30 0x7fc44006e2f0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fc450001780 tx=0x7fc450009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:45.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:45.138+0000 7fc457400640 1 -- 172.21.15.6:0/3424017993 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fc4601503f0 con 0x7fc4600c6be0 2024-04-24T23:16:45.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:45.345+0000 7fc464a00640 1 -- 172.21.15.6:0/3424017993 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc46014c900 con 0x7fc4600c6be0 2024-04-24T23:16:45.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:45.347+0000 7fc457400640 1 -- 172.21.15.6:0/3424017993 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+18554 (secure 0 0 0) 0x7fc4600ca290 con 0x7fc4600c6be0 2024-04-24T23:16:45.349 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:16:45.349 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":26,"fsid":"18bef68d-5e6d-4ec5-862b-b6f206e2588d","created":"2024-04-24T23:15:12.535072+0000","modified":"2024-04-24T23:16:18.218008+0000","last_up_change":"2024-04-24T23:15:23.773223+0000","last_in_change":"2024-04-24T23:15:14.905328+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T23:15:24.747441+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-24T23:15:34.051800+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-24T23:15:35.436870+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-24T23:15:39.034257+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"3b7a212d-d498-462e-90bf-ed493c803a09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":3022569437},{"type":"v1","addr":"172.21.15.6:6809","nonce":3022569437}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":3022569437},{"type":"v1","addr":"172.21.15.6:6811","nonce":3022569437}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":3022569437},{"type":"v1","addr":"172.21.15.6:6815","nonce":3022569437}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":3022569437},{"type":"v1","addr":"172.21.15.6:6813","nonce":3022569437}]},"public_addr":"172.21.15.6:6809/3022569437","cluster_addr":"172.21.15.6:6811/3022569437","heartbeat_back_addr":"172.21.15.6:6815/3022569437","heartbeat_front_addr":"172.21.15.6:6813/3022569437","state":["exists","up"]},{"osd":1,"uuid":"7c97f5a2-1735-4775-9f15-eb50099b19d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":3687836888},{"type":"v1","addr":"172.21.15.6:6817","nonce":3687836888}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":3687836888},{"type":"v1","addr":"172.21.15.6:6819","nonce":3687836888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":3687836888},{"type":"v1","addr":"172.21.15.6:6823","nonce":3687836888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":3687836888},{"type":"v1","addr":"172.21.15.6:6821","nonce":3687836888}]},"public_addr":"172.21.15.6:6817/3687836888","cluster_addr":"172.21.15.6:6819/3687836888","heartbeat_back_addr":"172.21.15.6:6823/3687836888","heartbeat_front_addr":"172.21.15.6:6821/3687836888","state":["exists","up"]},{"osd":2,"uuid":"d86bbc18-438c-4ec2-8018-ba7dd78f790b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":1020213724},{"type":"v1","addr":"172.21.15.6:6801","nonce":1020213724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":1020213724},{"type":"v1","addr":"172.21.15.6:6803","nonce":1020213724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":1020213724},{"type":"v1","addr":"172.21.15.6:6807","nonce":1020213724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":1020213724},{"type":"v1","addr":"172.21.15.6:6805","nonce":1020213724}]},"public_addr":"172.21.15.6:6801/1020213724","cluster_addr":"172.21.15.6:6803/1020213724","heartbeat_back_addr":"172.21.15.6:6807/1020213724","heartbeat_front_addr":"172.21.15.6:6805/1020213724","state":["exists","up"]},{"osd":3,"uuid":"3754ae16-4248-40dd-95f9-0f59c0eb5426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6824","nonce":3163883468},{"type":"v1","addr":"172.21.15.6:6825","nonce":3163883468}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":3163883468},{"type":"v1","addr":"172.21.15.6:6827","nonce":3163883468}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":3163883468},{"type":"v1","addr":"172.21.15.6:6831","nonce":3163883468}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":3163883468},{"type":"v1","addr":"172.21.15.6:6829","nonce":3163883468}]},"public_addr":"172.21.15.6:6825/3163883468","cluster_addr":"172.21.15.6:6827/3163883468","heartbeat_back_addr":"172.21.15.6:6831/3163883468","heartbeat_front_addr":"172.21.15.6:6829/3163883468","state":["exists","up"]},{"osd":4,"uuid":"8b2913a0-d006-479c-84ed-5b21e0e6c392","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":1989500934},{"type":"v1","addr":"172.21.15.190:6809","nonce":1989500934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":1989500934},{"type":"v1","addr":"172.21.15.190:6811","nonce":1989500934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":1989500934},{"type":"v1","addr":"172.21.15.190:6815","nonce":1989500934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":1989500934},{"type":"v1","addr":"172.21.15.190:6813","nonce":1989500934}]},"public_addr":"172.21.15.190:6809/1989500934","cluster_addr":"172.21.15.190:6811/1989500934","heartbeat_back_addr":"172.21.15.190:6815/1989500934","heartbeat_front_addr":"172.21.15.190:6813/1989500934","state":["exists","up"]},{"osd":5,"uuid":"499b79d0-0585-4dae-ab69-196a551793da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":3687548412},{"type":"v1","addr":"172.21.15.190:6801","nonce":3687548412}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":3687548412},{"type":"v1","addr":"172.21.15.190:6803","nonce":3687548412}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":3687548412},{"type":"v1","addr":"172.21.15.190:6807","nonce":3687548412}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":3687548412},{"type":"v1","addr":"172.21.15.190:6805","nonce":3687548412}]},"public_addr":"172.21.15.190:6801/3687548412","cluster_addr":"172.21.15.190:6803/3687548412","heartbeat_back_addr":"172.21.15.190:6807/3687548412","heartbeat_front_addr":"172.21.15.190:6805/3687548412","state":["exists","up"]},{"osd":6,"uuid":"6ba6df3b-4a6b-4a89-a890-0f79f387d4d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":870571564},{"type":"v1","addr":"172.21.15.190:6817","nonce":870571564}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":870571564},{"type":"v1","addr":"172.21.15.190:6819","nonce":870571564}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":870571564},{"type":"v1","addr":"172.21.15.190:6823","nonce":870571564}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":870571564},{"type":"v1","addr":"172.21.15.190:6821","nonce":870571564}]},"public_addr":"172.21.15.190:6817/870571564","cluster_addr":"172.21.15.190:6819/870571564","heartbeat_back_addr":"172.21.15.190:6823/870571564","heartbeat_front_addr":"172.21.15.190:6821/870571564","state":["exists","up"]},{"osd":7,"uuid":"e9668b1c-a000-4332-86e8-4efd73b02409","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":123639197},{"type":"v1","addr":"172.21.15.190:6825","nonce":123639197}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":123639197},{"type":"v1","addr":"172.21.15.190:6827","nonce":123639197}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":123639197},{"type":"v1","addr":"172.21.15.190:6831","nonce":123639197}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":123639197},{"type":"v1","addr":"172.21.15.190:6829","nonce":123639197}]},"public_addr":"172.21.15.190:6825/123639197","cluster_addr":"172.21.15.190:6827/123639197","heartbeat_back_addr":"172.21.15.190:6831/123639197","heartbeat_front_addr":"172.21.15.190:6829/123639197","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.321646+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.363873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.466963+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.531989+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:19.990707+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.043543+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.032492+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.115004+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.2f","mappings":[{"from":0,"to":7}]},{"pgid":"4.19","mappings":[{"from":7,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T23:16:45.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:45.349+0000 7fc464a00640 1 -- 172.21.15.6:0/3424017993 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fc44006be30 msgr2=0x7fc44006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:45.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:45.349+0000 7fc464a00640 1 --2- 172.21.15.6:0/3424017993 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fc44006be30 0x7fc44006e2f0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fc450001780 tx=0x7fc450009040 comp rx=0 tx=0).stop 2024-04-24T23:16:45.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:45.349+0000 7fc464a00640 1 -- 172.21.15.6:0/3424017993 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc4600c6be0 msgr2=0x7fc4600c4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:45.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:45.349+0000 7fc464a00640 1 --2- 172.21.15.6:0/3424017993 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc4600c6be0 0x7fc4600c4d40 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fc44c02a2b0 tx=0x7fc44c0046c0 comp rx=0 tx=0).stop 2024-04-24T23:16:45.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:45.349+0000 7fc464a00640 1 -- 172.21.15.6:0/3424017993 shutdown_connections 2024-04-24T23:16:45.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:45.349+0000 7fc464a00640 1 --2- 172.21.15.6:0/3424017993 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fc44006be30 0x7fc44006e2f0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:45.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:45.349+0000 7fc464a00640 1 --2- 172.21.15.6:0/3424017993 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc460152bd0 0x7fc4601f3eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:45.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:45.349+0000 7fc464a00640 1 --2- 172.21.15.6:0/3424017993 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc4600c6be0 0x7fc4600c4d40 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:45.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:45.349+0000 7fc464a00640 1 --2- 172.21.15.6:0/3424017993 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc46006ce30 0x7fc4600c4800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:45.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:45.349+0000 7fc464a00640 1 -- 172.21.15.6:0/3424017993 >> 172.21.15.6:0/3424017993 conn(0x7fc46006ed80 msgr2=0x7fc460158d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:45.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:45.350+0000 7fc464a00640 1 -- 172.21.15.6:0/3424017993 shutdown_connections 2024-04-24T23:16:45.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:45.350+0000 7fc464a00640 1 -- 172.21.15.6:0/3424017993 wait complete. 2024-04-24T23:16:45.373 INFO:tasks.mds_thrash.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T23:15:24.747441+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-24T23:15:34.051800+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-24T23:15:35.436870+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-24T23:15:39.034257+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-24T23:16:45.373 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-24T23:16:45.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:45.591+0000 7fbbf1000640 1 Processor -- start 2024-04-24T23:16:45.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:45.591+0000 7fbbf1000640 1 -- start start 2024-04-24T23:16:45.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:45.593+0000 7fbbf1000640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbbec0c9330 0x7fbbec0c6ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:45.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:45.593+0000 7fbbf1000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fbbec070e00 0x7fbbec15ab60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:45.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:45.593+0000 7fbbf1000640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbbec071a40 con 0x7fbbec154940 2024-04-24T23:16:45.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:45.593+0000 7fbbf1000640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7fbbec071d20 con 0x7fbbec070e00 2024-04-24T23:16:45.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:45.593+0000 7fbbf1000640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fbbec072000 con 0x7fbbec0c9330 2024-04-24T23:16:45.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:45.593+0000 7fbbeb400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fbbec154940 0x7fbbec06e910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:55188/0 (socket says 172.21.15.6:55188) 2024-04-24T23:16:45.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:45.593+0000 7fbbeb400640 1 -- 172.21.15.6:0/1495608181 learned_addr learned my addr 172.21.15.6:0/1495608181 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:16:45.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:45.593+0000 7fbbebe00640 1 --2- 172.21.15.6:0/1495608181 >> v2:172.21.15.6:3300/0 conn(0x7fbbec070e00 0x7fbbec15ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:45.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:45.593+0000 7fbbeaa00640 1 --2- 172.21.15.6:0/1495608181 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbbec0c9330 0x7fbbec0c6ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:45.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:45.594+0000 7fbbea000640 1 -- 172.21.15.6:0/1495608181 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1874043918 0 0) 0x7fbbec071a40 con 0x7fbbec154940 2024-04-24T23:16:45.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:45.594+0000 7fbbea000640 1 -- 172.21.15.6:0/1495608181 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbbd0003520 con 0x7fbbec154940 2024-04-24T23:16:45.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:45.594+0000 7fbbebe00640 1 -- 172.21.15.6:0/1495608181 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbbec0c9330 msgr2=0x7fbbec0c6ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:16:45.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:45.594+0000 7fbbebe00640 1 --2- 172.21.15.6:0/1495608181 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbbec0c9330 0x7fbbec0c6ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:45.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:45.594+0000 7fbbebe00640 1 -- 172.21.15.6:0/1495608181 >> v1:172.21.15.6:6789/0 conn(0x7fbbec154940 legacy=0x7fbbec06e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:45.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:45.594+0000 7fbbebe00640 1 -- 172.21.15.6:0/1495608181 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbec0722e0 con 0x7fbbec070e00 2024-04-24T23:16:45.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:45.595+0000 7fbbebe00640 1 --2- 172.21.15.6:0/1495608181 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbbec070e00 0x7fbbec15ab60 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fbbe4009d90 tx=0x7fbbe402a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b94065a480a35a43 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:45.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:45.595+0000 7fbbea000640 1 -- 172.21.15.6:0/1495608181 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbe4007740 con 0x7fbbec070e00 2024-04-24T23:16:45.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:45.595+0000 7fbbea000640 1 -- 172.21.15.6:0/1495608181 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbbe40078e0 con 0x7fbbec070e00 2024-04-24T23:16:45.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:45.595+0000 7fbbea000640 1 -- 172.21.15.6:0/1495608181 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbe4007cb0 con 0x7fbbec070e00 2024-04-24T23:16:45.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:45.596+0000 7fbbf1000640 1 -- 172.21.15.6:0/1495608181 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbbec070e00 msgr2=0x7fbbec15ab60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:45.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:45.596+0000 7fbbf1000640 1 --2- 172.21.15.6:0/1495608181 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbbec070e00 0x7fbbec15ab60 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fbbe4009d90 tx=0x7fbbe402a6f0 comp rx=0 tx=0).stop 2024-04-24T23:16:45.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:45.596+0000 7fbbf1000640 1 -- 172.21.15.6:0/1495608181 shutdown_connections 2024-04-24T23:16:45.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:45.596+0000 7fbbf1000640 1 --2- 172.21.15.6:0/1495608181 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbbec070e00 0x7fbbec15ab60 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:45.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:45.596+0000 7fbbf1000640 1 --2- 172.21.15.6:0/1495608181 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbbec0c9330 0x7fbbec0c6ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:45.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:45.596+0000 7fbbf1000640 1 -- 172.21.15.6:0/1495608181 >> 172.21.15.6:0/1495608181 conn(0x7fbbec06cc40 msgr2=0x7fbbec06d050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:45.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:45.596+0000 7fbbf1000640 1 -- 172.21.15.6:0/1495608181 shutdown_connections 2024-04-24T23:16:45.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:45.596+0000 7fbbf1000640 1 -- 172.21.15.6:0/1495608181 wait complete. 2024-04-24T23:16:45.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:45.598+0000 7fbbf1000640 1 Processor -- start 2024-04-24T23:16:45.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:45.598+0000 7fbbf1000640 1 -- start start 2024-04-24T23:16:45.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:45.599+0000 7fbbf1000640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbbec070e00 0x7fbbec15d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:45.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:45.599+0000 7fbbf1000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbbec0c9330 0x7fbbec1f0eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:45.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:45.599+0000 7fbbf1000640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbbec154940 0x7fbbec1f4500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:45.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:45.599+0000 7fbbf1000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fbbec06f020 con 0x7fbbec0c9330 2024-04-24T23:16:45.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:45.599+0000 7fbbf1000640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fbbec154d20 con 0x7fbbec154940 2024-04-24T23:16:45.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:45.599+0000 7fbbf1000640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fbbec06f1a0 con 0x7fbbec070e00 2024-04-24T23:16:45.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:45.599+0000 7fbbeb400640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbbec070e00 0x7fbbec15d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:45.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:45.599+0000 7fbbeaa00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbbec0c9330 0x7fbbec1f0eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:45.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:45.599+0000 7fbbeb400640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbbec070e00 0x7fbbec15d480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.6:59652/0 (socket says 172.21.15.6:59652) 2024-04-24T23:16:45.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:45.599+0000 7fbbeb400640 1 -- 172.21.15.6:0/4136009175 learned_addr learned my addr 172.21.15.6:0/4136009175 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:16:45.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:45.599+0000 7fbbebe00640 1 --2- 172.21.15.6:0/4136009175 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbbec154940 0x7fbbec1f4500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:45.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:45.600+0000 7fbbeb400640 1 -- 172.21.15.6:0/4136009175 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbbec154940 msgr2=0x7fbbec1f4500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:45.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:45.600+0000 7fbbeb400640 1 --2- 172.21.15.6:0/4136009175 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbbec154940 0x7fbbec1f4500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:45.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:45.600+0000 7fbbeb400640 1 -- 172.21.15.6:0/4136009175 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbbec0c9330 msgr2=0x7fbbec1f0eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:45.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:45.600+0000 7fbbeb400640 1 --2- 172.21.15.6:0/4136009175 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbbec0c9330 0x7fbbec1f0eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:45.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:45.600+0000 7fbbeb400640 1 -- 172.21.15.6:0/4136009175 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbec0c6170 con 0x7fbbec070e00 2024-04-24T23:16:45.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:45.600+0000 7fbbeb400640 1 --2- 172.21.15.6:0/4136009175 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbbec070e00 0x7fbbec15d480 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fbbdc00c3f0 tx=0x7fbbdc00c8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:45.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:45.600+0000 7fbbe3400640 1 -- 172.21.15.6:0/4136009175 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbdc017070 con 0x7fbbec070e00 2024-04-24T23:16:45.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:45.600+0000 7fbbe3400640 1 -- 172.21.15.6:0/4136009175 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbbdc005ce0 con 0x7fbbec070e00 2024-04-24T23:16:45.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:45.600+0000 7fbbe3400640 1 -- 172.21.15.6:0/4136009175 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbdc0072f0 con 0x7fbbec070e00 2024-04-24T23:16:45.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:45.601+0000 7fbbf1000640 1 -- 172.21.15.6:0/4136009175 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbbec0c6450 con 0x7fbbec070e00 2024-04-24T23:16:45.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:45.601+0000 7fbbf1000640 1 -- 172.21.15.6:0/4136009175 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fbbec1f3d00 con 0x7fbbec070e00 2024-04-24T23:16:45.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:45.602+0000 7fbbf1000640 1 -- 172.21.15.6:0/4136009175 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbbec075b30 con 0x7fbbec070e00 2024-04-24T23:16:45.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:45.603+0000 7fbbe3400640 1 -- 172.21.15.6:0/4136009175 <== mon.2 v2:172.21.15.190:3301/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fbbdc01a070 con 0x7fbbec070e00 2024-04-24T23:16:45.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:45.604+0000 7fbbe3400640 1 --2- 172.21.15.6:0/4136009175 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fbbc806be60 0x7fbbc806e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:45.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:45.604+0000 7fbbe3400640 1 -- 172.21.15.6:0/4136009175 <== mon.2 v2:172.21.15.190:3301/0 5 ==== osd_map(26..26 src has 1..26) ==== 6590+0+0 (secure 0 0 0) 0x7fbbdc08a580 con 0x7fbbec070e00 2024-04-24T23:16:45.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:45.604+0000 7fbbeaa00640 1 --2- 172.21.15.6:0/4136009175 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fbbc806be60 0x7fbbc806e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:45.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:45.605+0000 7fbbeaa00640 1 --2- 172.21.15.6:0/4136009175 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fbbc806be60 0x7fbbc806e320 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fbbec0c69b0 tx=0x7fbbd4009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:45.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:45.609+0000 7fbbe3400640 1 -- 172.21.15.6:0/4136009175 <== mon.2 v2:172.21.15.190:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fbbdc0584d0 con 0x7fbbec070e00 2024-04-24T23:16:45.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:45.809+0000 7fbbf1000640 1 -- 172.21.15.6:0/4136009175 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fbbec159810 con 0x7fbbec070e00 2024-04-24T23:16:45.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:45.810+0000 7fbbe3400640 1 -- 172.21.15.6:0/4136009175 <== mon.2 v2:172.21.15.190:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) ==== 93+0+10 (secure 0 0 0) 0x7fbbdc05c7d0 con 0x7fbbec070e00 2024-04-24T23:16:45.811 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-04-24T23:16:45.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:45.813+0000 7fbbf1000640 1 -- 172.21.15.6:0/4136009175 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fbbc806be60 msgr2=0x7fbbc806e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:45.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:45.813+0000 7fbbf1000640 1 --2- 172.21.15.6:0/4136009175 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fbbc806be60 0x7fbbc806e320 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fbbec0c69b0 tx=0x7fbbd4009190 comp rx=0 tx=0).stop 2024-04-24T23:16:45.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:45.813+0000 7fbbf1000640 1 -- 172.21.15.6:0/4136009175 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbbec070e00 msgr2=0x7fbbec15d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:45.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:45.813+0000 7fbbf1000640 1 --2- 172.21.15.6:0/4136009175 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbbec070e00 0x7fbbec15d480 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fbbdc00c3f0 tx=0x7fbbdc00c8b0 comp rx=0 tx=0).stop 2024-04-24T23:16:45.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:45.813+0000 7fbbf1000640 1 -- 172.21.15.6:0/4136009175 shutdown_connections 2024-04-24T23:16:45.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:45.813+0000 7fbbf1000640 1 --2- 172.21.15.6:0/4136009175 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fbbc806be60 0x7fbbc806e320 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:45.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:45.813+0000 7fbbf1000640 1 --2- 172.21.15.6:0/4136009175 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbbec154940 0x7fbbec1f4500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:45.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:45.813+0000 7fbbf1000640 1 --2- 172.21.15.6:0/4136009175 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbbec0c9330 0x7fbbec1f0eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:45.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:45.813+0000 7fbbf1000640 1 --2- 172.21.15.6:0/4136009175 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbbec070e00 0x7fbbec15d480 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:45.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:45.813+0000 7fbbf1000640 1 -- 172.21.15.6:0/4136009175 >> 172.21.15.6:0/4136009175 conn(0x7fbbec06cc40 msgr2=0x7fbbec0c7080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:45.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:45.813+0000 7fbbf1000640 1 -- 172.21.15.6:0/4136009175 shutdown_connections 2024-04-24T23:16:45.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:45.813+0000 7fbbf1000640 1 -- 172.21.15.6:0/4136009175 wait complete. 2024-04-24T23:16:45.833 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-24T23:16:46.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.049+0000 7f077d000640 1 Processor -- start 2024-04-24T23:16:46.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.049+0000 7f077d000640 1 -- start start 2024-04-24T23:16:46.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.051+0000 7f077d000640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f07780c5a60 0x7f07780c6ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:46.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.051+0000 7f077d000640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f07780cca80 con 0x7f07780c5e40 2024-04-24T23:16:46.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.051+0000 7f077d000640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f07780ccd60 con 0x7f07780c8570 2024-04-24T23:16:46.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.051+0000 7f077d000640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f077815c440 con 0x7f07780c5a60 2024-04-24T23:16:46.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.051+0000 7f0777400640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f07780c5a60 0x7f07780c6ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:46.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.051+0000 7f0777e00640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f07780c8570 0x7f07781524a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.6:36856/0 (socket says 172.21.15.6:36856) 2024-04-24T23:16:46.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.052+0000 7f0777e00640 1 -- 172.21.15.6:0/2494881930 learned_addr learned my addr 172.21.15.6:0/2494881930 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:16:46.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.052+0000 7f0776a00640 1 -- 172.21.15.6:0/2494881930 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2583797258 0 0) 0x7f07780ccd60 con 0x7f07780c8570 2024-04-24T23:16:46.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.052+0000 7f0776a00640 1 -- 172.21.15.6:0/2494881930 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f075c003520 con 0x7f07780c8570 2024-04-24T23:16:46.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.052+0000 7f0777400640 1 -- 172.21.15.6:0/2494881930 >> v1:172.21.15.190:6789/0 conn(0x7f07780c8570 legacy=0x7f07781524a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:46.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.052+0000 7f0777400640 1 -- 172.21.15.6:0/2494881930 >> v1:172.21.15.6:6789/0 conn(0x7f07780c5e40 legacy=0x7f077815bd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:46.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.052+0000 7f0777400640 1 -- 172.21.15.6:0/2494881930 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f077806eab0 con 0x7f07780c5a60 2024-04-24T23:16:46.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.053+0000 7f0777400640 1 --2- 172.21.15.6:0/2494881930 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f07780c5a60 0x7f07780c6ae0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f07680097d0 tx=0x7f076802a210 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4fe96faa8d9fdf5a server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:46.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.054+0000 7f0776a00640 1 -- 172.21.15.6:0/2494881930 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0768037070 con 0x7f07780c5a60 2024-04-24T23:16:46.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.054+0000 7f0776a00640 1 -- 172.21.15.6:0/2494881930 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0768032440 con 0x7f07780c5a60 2024-04-24T23:16:46.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.054+0000 7f0776a00640 1 -- 172.21.15.6:0/2494881930 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07680327f0 con 0x7f07780c5a60 2024-04-24T23:16:46.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.054+0000 7f077d000640 1 -- 172.21.15.6:0/2494881930 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f07780c5a60 msgr2=0x7f07780c6ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:46.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.054+0000 7f077d000640 1 --2- 172.21.15.6:0/2494881930 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f07780c5a60 0x7f07780c6ae0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f07680097d0 tx=0x7f076802a210 comp rx=0 tx=0).stop 2024-04-24T23:16:46.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.055+0000 7f077d000640 1 -- 172.21.15.6:0/2494881930 shutdown_connections 2024-04-24T23:16:46.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.055+0000 7f077d000640 1 --2- 172.21.15.6:0/2494881930 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f07780c5a60 0x7f07780c6ae0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:46.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.055+0000 7f077d000640 1 -- 172.21.15.6:0/2494881930 >> 172.21.15.6:0/2494881930 conn(0x7f077806cc40 msgr2=0x7f077806d050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:46.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.055+0000 7f077d000640 1 -- 172.21.15.6:0/2494881930 shutdown_connections 2024-04-24T23:16:46.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.055+0000 7f077d000640 1 -- 172.21.15.6:0/2494881930 wait complete. 2024-04-24T23:16:46.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.056+0000 7f077d000640 1 Processor -- start 2024-04-24T23:16:46.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.057+0000 7f077d000640 1 -- start start 2024-04-24T23:16:46.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.057+0000 7f077d000640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f07780c5a60 0x7f077806b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:46.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.058+0000 7f077d000640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f07780c5e40 0x7f07781f0060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:46.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.058+0000 7f077d000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f07780c8570 0x7f077814bc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:46.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.058+0000 7f077d000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f077806f590 con 0x7f07780c8570 2024-04-24T23:16:46.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.058+0000 7f077d000640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f077806f070 con 0x7f07780c5e40 2024-04-24T23:16:46.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.058+0000 7f077d000640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f0778150160 con 0x7f07780c5a60 2024-04-24T23:16:46.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.058+0000 7f0777e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f07780c8570 0x7f077814bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:46.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.058+0000 7f0777e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f07780c8570 0x7f077814bc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36298/0 (socket says 172.21.15.6:36298) 2024-04-24T23:16:46.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.058+0000 7f0777e00640 1 -- 172.21.15.6:0/2624518163 learned_addr learned my addr 172.21.15.6:0/2624518163 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:16:46.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.058+0000 7f0777400640 1 --2- 172.21.15.6:0/2624518163 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f07780c5a60 0x7f077806b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:46.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.058+0000 7f076ea00640 1 --2- 172.21.15.6:0/2624518163 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f07780c5e40 0x7f07781f0060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:46.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.058+0000 7f0777400640 1 -- 172.21.15.6:0/2624518163 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f07780c5e40 msgr2=0x7f07781f0060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:46.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.058+0000 7f0777400640 1 --2- 172.21.15.6:0/2624518163 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f07780c5e40 0x7f07781f0060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:46.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.058+0000 7f0777400640 1 -- 172.21.15.6:0/2624518163 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f07780c8570 msgr2=0x7f077814bc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:46.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.058+0000 7f0777400640 1 --2- 172.21.15.6:0/2624518163 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f07780c8570 0x7f077814bc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:46.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.058+0000 7f0777400640 1 -- 172.21.15.6:0/2624518163 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0778157620 con 0x7f07780c5a60 2024-04-24T23:16:46.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.059+0000 7f0777400640 1 --2- 172.21.15.6:0/2624518163 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f07780c5a60 0x7f077806b8f0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f0768031000 tx=0x7f07680046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:46.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.059+0000 7f076fe00640 1 -- 172.21.15.6:0/2624518163 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0768037050 con 0x7f07780c5a60 2024-04-24T23:16:46.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.059+0000 7f076fe00640 1 -- 172.21.15.6:0/2624518163 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0768038040 con 0x7f07780c5a60 2024-04-24T23:16:46.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.059+0000 7f076fe00640 1 -- 172.21.15.6:0/2624518163 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0768004430 con 0x7f07780c5a60 2024-04-24T23:16:46.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.059+0000 7f077d000640 1 -- 172.21.15.6:0/2624518163 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0778158420 con 0x7f07780c5a60 2024-04-24T23:16:46.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.059+0000 7f077d000640 1 -- 172.21.15.6:0/2624518163 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f07781599a0 con 0x7f07780c5a60 2024-04-24T23:16:46.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.060+0000 7f077d000640 1 -- 172.21.15.6:0/2624518163 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0778075b30 con 0x7f07780c5a60 2024-04-24T23:16:46.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.061+0000 7f076fe00640 1 -- 172.21.15.6:0/2624518163 <== mon.2 v2:172.21.15.190:3301/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f076803c6d0 con 0x7f07780c5a60 2024-04-24T23:16:46.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.062+0000 7f076fe00640 1 --2- 172.21.15.6:0/2624518163 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f075406be60 0x7f075406e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:46.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.062+0000 7f076fe00640 1 -- 172.21.15.6:0/2624518163 <== mon.2 v2:172.21.15.190:3301/0 5 ==== osd_map(26..26 src has 1..26) ==== 6590+0+0 (secure 0 0 0) 0x7f07680aba40 con 0x7f07780c5a60 2024-04-24T23:16:46.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.063+0000 7f076ea00640 1 --2- 172.21.15.6:0/2624518163 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f075406be60 0x7f075406e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:46.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.063+0000 7f076ea00640 1 --2- 172.21.15.6:0/2624518163 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f075406be60 0x7f075406e320 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f07780c57c0 tx=0x7f0760009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:46.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.067+0000 7f076fe00640 1 -- 172.21.15.6:0/2624518163 <== mon.2 v2:172.21.15.190:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f0768079990 con 0x7f07780c5a60 2024-04-24T23:16:46.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.264+0000 7f077d000640 1 -- 172.21.15.6:0/2624518163 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f077806f710 con 0x7f07780c5a60 2024-04-24T23:16:46.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.266+0000 7f076fe00640 1 -- 172.21.15.6:0/2624518163 <== mon.2 v2:172.21.15.190:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) ==== 104+0+11 (secure 0 0 0) 0x7f076807ee20 con 0x7f07780c5a60 2024-04-24T23:16:46.268 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 32 2024-04-24T23:16:46.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.268+0000 7f077d000640 1 -- 172.21.15.6:0/2624518163 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f075406be60 msgr2=0x7f075406e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:46.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.268+0000 7f077d000640 1 --2- 172.21.15.6:0/2624518163 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f075406be60 0x7f075406e320 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f07780c57c0 tx=0x7f0760009040 comp rx=0 tx=0).stop 2024-04-24T23:16:46.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.268+0000 7f077d000640 1 -- 172.21.15.6:0/2624518163 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f07780c5a60 msgr2=0x7f077806b8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:46.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.268+0000 7f077d000640 1 --2- 172.21.15.6:0/2624518163 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f07780c5a60 0x7f077806b8f0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f0768031000 tx=0x7f07680046c0 comp rx=0 tx=0).stop 2024-04-24T23:16:46.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.268+0000 7f077d000640 1 -- 172.21.15.6:0/2624518163 shutdown_connections 2024-04-24T23:16:46.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.268+0000 7f077d000640 1 --2- 172.21.15.6:0/2624518163 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f075406be60 0x7f075406e320 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:46.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.268+0000 7f077d000640 1 --2- 172.21.15.6:0/2624518163 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f07780c8570 0x7f077814bc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:46.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.269+0000 7f077d000640 1 --2- 172.21.15.6:0/2624518163 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f07780c5e40 0x7f07781f0060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:46.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.269+0000 7f077d000640 1 --2- 172.21.15.6:0/2624518163 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f07780c5a60 0x7f077806b8f0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:46.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.269+0000 7f077d000640 1 -- 172.21.15.6:0/2624518163 >> 172.21.15.6:0/2624518163 conn(0x7f077806cc40 msgr2=0x7f077814e220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:46.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.269+0000 7f077d000640 1 -- 172.21.15.6:0/2624518163 shutdown_connections 2024-04-24T23:16:46.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.269+0000 7f077d000640 1 -- 172.21.15.6:0/2624518163 wait complete. 2024-04-24T23:16:46.290 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-24T23:16:46.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.504+0000 7f2855c00640 1 Processor -- start 2024-04-24T23:16:46.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.504+0000 7f2855c00640 1 -- start start 2024-04-24T23:16:46.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.506+0000 7f2855c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f285006f020 0x7f2850071000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:46.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.506+0000 7f2855c00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2850072530 con 0x7f2850154b70 2024-04-24T23:16:46.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.506+0000 7f2855c00640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f28500727d0 con 0x7f28500c7080 2024-04-24T23:16:46.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.506+0000 7f2855c00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f2850072a70 con 0x7f285006f020 2024-04-24T23:16:46.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.506+0000 7f284ea00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f2850154b70 0x7f285015a9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:55200/0 (socket says 172.21.15.6:55200) 2024-04-24T23:16:46.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.506+0000 7f284ea00640 1 -- 172.21.15.6:0/3418675626 learned_addr learned my addr 172.21.15.6:0/3418675626 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:16:46.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.506+0000 7f284f400640 1 --2- 172.21.15.6:0/3418675626 >> v2:172.21.15.6:3300/0 conn(0x7f285006f020 0x7f2850071000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:46.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.507+0000 7f284e000640 1 -- 172.21.15.6:0/3418675626 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4018851781 0 0) 0x7f2850072530 con 0x7f2850154b70 2024-04-24T23:16:46.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.507+0000 7f284e000640 1 -- 172.21.15.6:0/3418675626 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2834003520 con 0x7f2850154b70 2024-04-24T23:16:46.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.507+0000 7f284e000640 1 -- 172.21.15.6:0/3418675626 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4148292499 0 0) 0x7f28500727d0 con 0x7f28500c7080 2024-04-24T23:16:46.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.507+0000 7f284f400640 1 -- 172.21.15.6:0/3418675626 >> v1:172.21.15.190:6789/0 conn(0x7f28500c7080 legacy=0x7f285015b0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:46.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.507+0000 7f284f400640 1 -- 172.21.15.6:0/3418675626 >> v1:172.21.15.6:6789/0 conn(0x7f2850154b70 legacy=0x7f285015a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:46.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.507+0000 7f284f400640 1 -- 172.21.15.6:0/3418675626 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f285006e540 con 0x7f285006f020 2024-04-24T23:16:46.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.508+0000 7f284f400640 1 --2- 172.21.15.6:0/3418675626 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f285006f020 0x7f2850071000 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f2848009dc0 tx=0x7f284802a730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7072d75b459a11b server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:46.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.508+0000 7f284e000640 1 -- 172.21.15.6:0/3418675626 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2848007740 con 0x7f285006f020 2024-04-24T23:16:46.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.508+0000 7f284e000640 1 -- 172.21.15.6:0/3418675626 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f28480078e0 con 0x7f285006f020 2024-04-24T23:16:46.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.508+0000 7f284e000640 1 -- 172.21.15.6:0/3418675626 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2848007cb0 con 0x7f285006f020 2024-04-24T23:16:46.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.509+0000 7f2855c00640 1 -- 172.21.15.6:0/3418675626 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f285006f020 msgr2=0x7f2850071000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:46.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.509+0000 7f2855c00640 1 --2- 172.21.15.6:0/3418675626 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f285006f020 0x7f2850071000 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f2848009dc0 tx=0x7f284802a730 comp rx=0 tx=0).stop 2024-04-24T23:16:46.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.509+0000 7f2855c00640 1 -- 172.21.15.6:0/3418675626 shutdown_connections 2024-04-24T23:16:46.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.509+0000 7f2855c00640 1 --2- 172.21.15.6:0/3418675626 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f285006f020 0x7f2850071000 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:46.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.509+0000 7f2855c00640 1 -- 172.21.15.6:0/3418675626 >> 172.21.15.6:0/3418675626 conn(0x7f285006b0e0 msgr2=0x7f285006b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:46.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.509+0000 7f2855c00640 1 -- 172.21.15.6:0/3418675626 shutdown_connections 2024-04-24T23:16:46.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.509+0000 7f2855c00640 1 -- 172.21.15.6:0/3418675626 wait complete. 2024-04-24T23:16:46.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.510+0000 7f2855c00640 1 Processor -- start 2024-04-24T23:16:46.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.511+0000 7f2855c00640 1 -- start start 2024-04-24T23:16:46.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.511+0000 7f2855c00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f285006f020 0x7f28500cb3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:46.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.511+0000 7f2855c00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f28500c7080 0x7f28501f4140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:46.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.511+0000 7f2855c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2850154b70 0x7f285014abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:46.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.512+0000 7f2855c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f285015b7f0 con 0x7f2850154b70 2024-04-24T23:16:46.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.512+0000 7f2855c00640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f285006eb00 con 0x7f28500c7080 2024-04-24T23:16:46.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.512+0000 7f2855c00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f2850154f50 con 0x7f285006f020 2024-04-24T23:16:46.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.512+0000 7f284f400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2850154b70 0x7f285014abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:46.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.512+0000 7f284f400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2850154b70 0x7f285014abf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36314/0 (socket says 172.21.15.6:36314) 2024-04-24T23:16:46.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.512+0000 7f284f400640 1 -- 172.21.15.6:0/3810391390 learned_addr learned my addr 172.21.15.6:0/3810391390 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:16:46.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.512+0000 7f284ea00640 1 --2- 172.21.15.6:0/3810391390 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f285006f020 0x7f28500cb3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:46.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.512+0000 7f2847e00640 1 --2- 172.21.15.6:0/3810391390 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f28500c7080 0x7f28501f4140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:46.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.512+0000 7f284f400640 1 -- 172.21.15.6:0/3810391390 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f285006f020 msgr2=0x7f28500cb3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:46.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.512+0000 7f284f400640 1 --2- 172.21.15.6:0/3810391390 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f285006f020 0x7f28500cb3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:46.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.512+0000 7f284f400640 1 -- 172.21.15.6:0/3810391390 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f28500c7080 msgr2=0x7f28501f4140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:46.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.512+0000 7f284f400640 1 --2- 172.21.15.6:0/3810391390 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f28500c7080 0x7f28501f4140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:46.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.512+0000 7f284f400640 1 -- 172.21.15.6:0/3810391390 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28501f12d0 con 0x7f2850154b70 2024-04-24T23:16:46.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.513+0000 7f284f400640 1 --2- 172.21.15.6:0/3810391390 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2850154b70 0x7f285014abf0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f2848009ee0 tx=0x7f2848004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:46.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.513+0000 7f2846a00640 1 -- 172.21.15.6:0/3810391390 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2848040070 con 0x7f2850154b70 2024-04-24T23:16:46.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.513+0000 7f2846a00640 1 -- 172.21.15.6:0/3810391390 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2848031bc0 con 0x7f2850154b70 2024-04-24T23:16:46.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.513+0000 7f2846a00640 1 -- 172.21.15.6:0/3810391390 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2848009450 con 0x7f2850154b70 2024-04-24T23:16:46.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.513+0000 7f2855c00640 1 -- 172.21.15.6:0/3810391390 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f285015bde0 con 0x7f2850154b70 2024-04-24T23:16:46.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.513+0000 7f2855c00640 1 -- 172.21.15.6:0/3810391390 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28501f3c20 con 0x7f2850154b70 2024-04-24T23:16:46.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.514+0000 7f2855c00640 1 -- 172.21.15.6:0/3810391390 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f28500767b0 con 0x7f2850154b70 2024-04-24T23:16:46.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.515+0000 7f2846a00640 1 -- 172.21.15.6:0/3810391390 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f2848039040 con 0x7f2850154b70 2024-04-24T23:16:46.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.516+0000 7f2846a00640 1 --2- 172.21.15.6:0/3810391390 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f282c06be30 0x7f282c06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:46.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.516+0000 7f2846a00640 1 -- 172.21.15.6:0/3810391390 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6590+0+0 (secure 0 0 0) 0x7f28480ac9e0 con 0x7f2850154b70 2024-04-24T23:16:46.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.517+0000 7f284ea00640 1 --2- 172.21.15.6:0/3810391390 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f282c06be30 0x7f282c06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:46.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.517+0000 7f284ea00640 1 --2- 172.21.15.6:0/3810391390 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f282c06be30 0x7f282c06e2f0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f2840005880 tx=0x7f2840009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:46.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.520+0000 7f2846a00640 1 -- 172.21.15.6:0/3810391390 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f284807b010 con 0x7f2850154b70 2024-04-24T23:16:46.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.723+0000 7f2855c00640 1 -- 172.21.15.6:0/3810391390 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f285014f160 con 0x7f2850154b70 2024-04-24T23:16:46.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.724+0000 7f2846a00640 1 -- 172.21.15.6:0/3810391390 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) ==== 100+0+11 (secure 0 0 0) 0x7f284807a910 con 0x7f2850154b70 2024-04-24T23:16:46.726 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 64 2024-04-24T23:16:46.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.727+0000 7f2855c00640 1 -- 172.21.15.6:0/3810391390 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f282c06be30 msgr2=0x7f282c06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:46.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.727+0000 7f2855c00640 1 --2- 172.21.15.6:0/3810391390 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f282c06be30 0x7f282c06e2f0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f2840005880 tx=0x7f2840009040 comp rx=0 tx=0).stop 2024-04-24T23:16:46.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.727+0000 7f2855c00640 1 -- 172.21.15.6:0/3810391390 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2850154b70 msgr2=0x7f285014abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:46.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.727+0000 7f2855c00640 1 --2- 172.21.15.6:0/3810391390 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2850154b70 0x7f285014abf0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f2848009ee0 tx=0x7f2848004a40 comp rx=0 tx=0).stop 2024-04-24T23:16:46.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.727+0000 7f2855c00640 1 -- 172.21.15.6:0/3810391390 shutdown_connections 2024-04-24T23:16:46.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.727+0000 7f2855c00640 1 --2- 172.21.15.6:0/3810391390 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f282c06be30 0x7f282c06e2f0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:46.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.727+0000 7f2855c00640 1 --2- 172.21.15.6:0/3810391390 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2850154b70 0x7f285014abf0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:46.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.727+0000 7f2855c00640 1 --2- 172.21.15.6:0/3810391390 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f28500c7080 0x7f28501f4140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:46.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.727+0000 7f2855c00640 1 --2- 172.21.15.6:0/3810391390 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f285006f020 0x7f28500cb3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:46.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.727+0000 7f2855c00640 1 -- 172.21.15.6:0/3810391390 >> 172.21.15.6:0/3810391390 conn(0x7f285006b0e0 msgr2=0x7f285006b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:46.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.727+0000 7f2855c00640 1 -- 172.21.15.6:0/3810391390 shutdown_connections 2024-04-24T23:16:46.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.727+0000 7f2855c00640 1 -- 172.21.15.6:0/3810391390 wait complete. 2024-04-24T23:16:46.747 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-24T23:16:46.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.961+0000 7fdc0fe00640 1 Processor -- start 2024-04-24T23:16:46.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.962+0000 7fdc0fe00640 1 -- start start 2024-04-24T23:16:46.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.963+0000 7fdc0fe00640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fdc1006cd50 0x7fdc1015e090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:46.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.963+0000 7fdc0fe00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdc100b35f0 0x7fdc1015e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:46.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.963+0000 7fdc0fe00640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdc100b46f0 con 0x7fdc10154f60 2024-04-24T23:16:46.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.963+0000 7fdc0fe00640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7fdc100b49d0 con 0x7fdc1006cd50 2024-04-24T23:16:46.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.963+0000 7fdc0fe00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fdc100b4cb0 con 0x7fdc100b35f0 2024-04-24T23:16:46.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.964+0000 7fdc0ea00640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7fdc10154f60 0x7fdc1006c640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.6:36884/0 (socket says 172.21.15.6:36884) 2024-04-24T23:16:46.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.964+0000 7fdc0ea00640 1 -- 172.21.15.6:0/1750278648 learned_addr learned my addr 172.21.15.6:0/1750278648 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:16:46.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.964+0000 7fdc06000640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fdc1006cd50 0x7fdc1015e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:46.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.964+0000 7fdc0f400640 1 --2- 172.21.15.6:0/1750278648 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdc100b35f0 0x7fdc1015e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:46.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.964+0000 7fdc0e000640 1 -- 172.21.15.6:0/1750278648 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3068955894 0 0) 0x7fdc100b46f0 con 0x7fdc10154f60 2024-04-24T23:16:46.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.964+0000 7fdc0e000640 1 -- 172.21.15.6:0/1750278648 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdbf4003520 con 0x7fdc10154f60 2024-04-24T23:16:46.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.964+0000 7fdc06000640 1 -- 172.21.15.6:0/1750278648 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdc100b35f0 msgr2=0x7fdc1015e5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:16:46.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.964+0000 7fdc06000640 1 --2- 172.21.15.6:0/1750278648 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdc100b35f0 0x7fdc1015e5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:46.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.964+0000 7fdc06000640 1 -- 172.21.15.6:0/1750278648 >> v1:172.21.15.190:6789/0 conn(0x7fdc10154f60 legacy=0x7fdc1006c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:46.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.964+0000 7fdc06000640 1 -- 172.21.15.6:0/1750278648 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc100b4f90 con 0x7fdc1006cd50 2024-04-24T23:16:46.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.965+0000 7fdc06000640 1 --2- 172.21.15.6:0/1750278648 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdc1006cd50 0x7fdc1015e090 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fdbf80097a0 tx=0x7fdbf802a210 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d9f9065ef041e489 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:46.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.966+0000 7fdc0e000640 1 -- 172.21.15.6:0/1750278648 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbf8037070 con 0x7fdc1006cd50 2024-04-24T23:16:46.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.966+0000 7fdc0e000640 1 -- 172.21.15.6:0/1750278648 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdbf8032440 con 0x7fdc1006cd50 2024-04-24T23:16:46.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.966+0000 7fdc0e000640 1 -- 172.21.15.6:0/1750278648 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbf8032810 con 0x7fdc1006cd50 2024-04-24T23:16:46.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.966+0000 7fdc0fe00640 1 -- 172.21.15.6:0/1750278648 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdc1006cd50 msgr2=0x7fdc1015e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:46.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.966+0000 7fdc0fe00640 1 --2- 172.21.15.6:0/1750278648 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdc1006cd50 0x7fdc1015e090 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fdbf80097a0 tx=0x7fdbf802a210 comp rx=0 tx=0).stop 2024-04-24T23:16:46.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.966+0000 7fdc0fe00640 1 -- 172.21.15.6:0/1750278648 shutdown_connections 2024-04-24T23:16:46.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.966+0000 7fdc0fe00640 1 --2- 172.21.15.6:0/1750278648 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdc100b35f0 0x7fdc1015e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:46.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.966+0000 7fdc0fe00640 1 --2- 172.21.15.6:0/1750278648 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdc1006cd50 0x7fdc1015e090 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:46.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.966+0000 7fdc0fe00640 1 -- 172.21.15.6:0/1750278648 >> 172.21.15.6:0/1750278648 conn(0x7fdc100674d0 msgr2=0x7fdc100d0320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:46.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.966+0000 7fdc0fe00640 1 -- 172.21.15.6:0/1750278648 shutdown_connections 2024-04-24T23:16:46.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.966+0000 7fdc0fe00640 1 -- 172.21.15.6:0/1750278648 wait complete. 2024-04-24T23:16:46.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.968+0000 7fdc0fe00640 1 Processor -- start 2024-04-24T23:16:46.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.968+0000 7fdc0fe00640 1 -- start start 2024-04-24T23:16:46.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.969+0000 7fdc0fe00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdc1006cd50 0x7fdc100c62c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:46.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.969+0000 7fdc0fe00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdc100b35f0 0x7fdc101e70f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:46.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.969+0000 7fdc0fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdc10154f60 0x7fdc101f3c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:46.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.969+0000 7fdc0fe00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fdc100b3240 con 0x7fdc10154f60 2024-04-24T23:16:46.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.969+0000 7fdc0fe00640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fdc1006c120 con 0x7fdc1006cd50 2024-04-24T23:16:46.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.969+0000 7fdc0fe00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fdc100b33c0 con 0x7fdc100b35f0 2024-04-24T23:16:46.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.969+0000 7fdc0ea00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdc1006cd50 0x7fdc100c62c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:46.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.969+0000 7fdc06000640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdc100b35f0 0x7fdc101e70f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:46.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.970+0000 7fdc0ea00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdc1006cd50 0x7fdc100c62c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:36622/0 (socket says 172.21.15.6:36622) 2024-04-24T23:16:46.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.970+0000 7fdc06000640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdc100b35f0 0x7fdc101e70f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.6:59722/0 (socket says 172.21.15.6:59722) 2024-04-24T23:16:46.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.970+0000 7fdc0ea00640 1 -- 172.21.15.6:0/855052511 learned_addr learned my addr 172.21.15.6:0/855052511 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:16:46.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.970+0000 7fdc0f400640 1 --2- 172.21.15.6:0/855052511 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdc10154f60 0x7fdc101f3c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:46.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.970+0000 7fdc0ea00640 1 -- 172.21.15.6:0/855052511 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdc100b35f0 msgr2=0x7fdc101e70f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:46.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.970+0000 7fdc0ea00640 1 --2- 172.21.15.6:0/855052511 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdc100b35f0 0x7fdc101e70f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:46.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.970+0000 7fdc0ea00640 1 -- 172.21.15.6:0/855052511 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdc10154f60 msgr2=0x7fdc101f3c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:46.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.970+0000 7fdc0ea00640 1 --2- 172.21.15.6:0/855052511 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdc10154f60 0x7fdc101f3c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:46.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.970+0000 7fdc0ea00640 1 -- 172.21.15.6:0/855052511 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc100c5f90 con 0x7fdc1006cd50 2024-04-24T23:16:46.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.970+0000 7fdc06000640 1 --2- 172.21.15.6:0/855052511 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdc100b35f0 0x7fdc101e70f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:16:46.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.970+0000 7fdc0ea00640 1 --2- 172.21.15.6:0/855052511 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdc1006cd50 0x7fdc100c62c0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fdc0000c410 tx=0x7fdc0000c8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:46.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.971+0000 7fdc07400640 1 -- 172.21.15.6:0/855052511 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc00017070 con 0x7fdc1006cd50 2024-04-24T23:16:46.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.971+0000 7fdc07400640 1 -- 172.21.15.6:0/855052511 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdc00005ce0 con 0x7fdc1006cd50 2024-04-24T23:16:46.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.971+0000 7fdc07400640 1 -- 172.21.15.6:0/855052511 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc000072f0 con 0x7fdc1006cd50 2024-04-24T23:16:46.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.971+0000 7fdc0fe00640 1 -- 172.21.15.6:0/855052511 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc100c6ae0 con 0x7fdc1006cd50 2024-04-24T23:16:46.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.971+0000 7fdc0fe00640 1 -- 172.21.15.6:0/855052511 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc100c54b0 con 0x7fdc1006cd50 2024-04-24T23:16:46.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.972+0000 7fdc0fe00640 1 -- 172.21.15.6:0/855052511 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc10075b30 con 0x7fdc1006cd50 2024-04-24T23:16:46.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.973+0000 7fdc07400640 1 -- 172.21.15.6:0/855052511 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fdc0001a070 con 0x7fdc1006cd50 2024-04-24T23:16:46.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.974+0000 7fdc07400640 1 --2- 172.21.15.6:0/855052511 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fdbec06be60 0x7fdbec06e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:46.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.974+0000 7fdc07400640 1 -- 172.21.15.6:0/855052511 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6590+0+0 (secure 0 0 0) 0x7fdc0008ad00 con 0x7fdc1006cd50 2024-04-24T23:16:46.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.975+0000 7fdc06000640 1 --2- 172.21.15.6:0/855052511 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fdbec06be60 0x7fdbec06e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:46.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.975+0000 7fdc06000640 1 --2- 172.21.15.6:0/855052511 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fdbec06be60 0x7fdbec06e320 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fdbf8034000 tx=0x7fdbf802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:46.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:46.978+0000 7fdc07400640 1 -- 172.21.15.6:0/855052511 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fdc00058d90 con 0x7fdc1006cd50 2024-04-24T23:16:47.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.182+0000 7fdc0fe00640 1 -- 172.21.15.6:0/855052511 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fdc10153dc0 con 0x7fdc1006cd50 2024-04-24T23:16:47.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.183+0000 7fdc07400640 1 -- 172.21.15.6:0/855052511 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) ==== 103+0+11 (secure 0 0 0) 0x7fdc0005d090 con 0x7fdc1006cd50 2024-04-24T23:16:47.185 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 32 2024-04-24T23:16:47.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.185+0000 7fdc0fe00640 1 -- 172.21.15.6:0/855052511 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fdbec06be60 msgr2=0x7fdbec06e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:47.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.185+0000 7fdc0fe00640 1 --2- 172.21.15.6:0/855052511 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fdbec06be60 0x7fdbec06e320 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fdbf8034000 tx=0x7fdbf802e000 comp rx=0 tx=0).stop 2024-04-24T23:16:47.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.185+0000 7fdc0fe00640 1 -- 172.21.15.6:0/855052511 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdc1006cd50 msgr2=0x7fdc100c62c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:47.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.185+0000 7fdc0fe00640 1 --2- 172.21.15.6:0/855052511 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdc1006cd50 0x7fdc100c62c0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fdc0000c410 tx=0x7fdc0000c8d0 comp rx=0 tx=0).stop 2024-04-24T23:16:47.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.186+0000 7fdc0fe00640 1 -- 172.21.15.6:0/855052511 shutdown_connections 2024-04-24T23:16:47.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.186+0000 7fdc0fe00640 1 --2- 172.21.15.6:0/855052511 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fdbec06be60 0x7fdbec06e320 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:47.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.186+0000 7fdc0fe00640 1 --2- 172.21.15.6:0/855052511 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdc100b35f0 0x7fdc101e70f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:47.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.186+0000 7fdc0fe00640 1 --2- 172.21.15.6:0/855052511 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdc10154f60 0x7fdc101f3c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:47.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.186+0000 7fdc0fe00640 1 --2- 172.21.15.6:0/855052511 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdc1006cd50 0x7fdc100c62c0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:47.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.186+0000 7fdc0fe00640 1 -- 172.21.15.6:0/855052511 >> 172.21.15.6:0/855052511 conn(0x7fdc100674d0 msgr2=0x7fdc100cb430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:47.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.186+0000 7fdc0fe00640 1 -- 172.21.15.6:0/855052511 shutdown_connections 2024-04-24T23:16:47.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.186+0000 7fdc0fe00640 1 -- 172.21.15.6:0/855052511 wait complete. 2024-04-24T23:16:47.207 INFO:tasks.mds_thrash:Wait for all MDSs to reach steady state... 2024-04-24T23:16:47.208 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:16:47.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.418+0000 7f6022c00640 1 Processor -- start 2024-04-24T23:16:47.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.418+0000 7f6022c00640 1 -- start start 2024-04-24T23:16:47.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.420+0000 7f6022c00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f601c154f90 0x7f601c0c7eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:47.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.420+0000 7f6022c00640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f601c0c84f0 0x7f601c15e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:47.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.420+0000 7f6022c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f601c15e900 0x7f601c072240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:47.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.420+0000 7f6022c00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f601c15ba80 con 0x7f601c15e900 2024-04-24T23:16:47.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.420+0000 7f6022c00640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f601c15ed90 con 0x7f601c0c84f0 2024-04-24T23:16:47.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.420+0000 7f6022c00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f601c15ac20 con 0x7f601c154f90 2024-04-24T23:16:47.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.420+0000 7f6020c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f601c15e900 0x7f601c072240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:47.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.420+0000 7f6020c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f601c15e900 0x7f601c072240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36334/0 (socket says 172.21.15.6:36334) 2024-04-24T23:16:47.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.420+0000 7f601be00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f601c154f90 0x7f601c0c7eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:47.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.420+0000 7f6020c00640 1 -- 172.21.15.6:0/1463775140 learned_addr learned my addr 172.21.15.6:0/1463775140 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:16:47.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.420+0000 7f601b400640 1 --2- 172.21.15.6:0/1463775140 >> v2:172.21.15.190:3300/0 conn(0x7f601c0c84f0 0x7f601c15e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:47.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.421+0000 7f601b400640 1 -- 172.21.15.6:0/1463775140 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f601c154f90 msgr2=0x7f601c0c7eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:16:47.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.421+0000 7f601b400640 1 --2- 172.21.15.6:0/1463775140 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f601c154f90 0x7f601c0c7eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:47.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.421+0000 7f601b400640 1 -- 172.21.15.6:0/1463775140 >> v2:172.21.15.6:3300/0 conn(0x7f601c15e900 msgr2=0x7f601c072240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:16:47.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.421+0000 7f601b400640 1 --2- 172.21.15.6:0/1463775140 >> v2:172.21.15.6:3300/0 conn(0x7f601c15e900 0x7f601c072240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:47.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.421+0000 7f601b400640 1 -- 172.21.15.6:0/1463775140 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f601c15af00 con 0x7f601c0c84f0 2024-04-24T23:16:47.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.421+0000 7f6020c00640 1 --2- 172.21.15.6:0/1463775140 >> v2:172.21.15.6:3300/0 conn(0x7f601c15e900 0x7f601c072240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:16:47.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.421+0000 7f601be00640 1 --2- 172.21.15.6:0/1463775140 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f601c154f90 0x7f601c0c7eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:16:47.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.422+0000 7f601b400640 1 --2- 172.21.15.6:0/1463775140 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f601c0c84f0 0x7f601c15e3c0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f601c073360 tx=0x7f600402a660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8115250a221322d7 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:47.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.422+0000 7f601aa00640 1 -- 172.21.15.6:0/1463775140 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6004007740 con 0x7f601c0c84f0 2024-04-24T23:16:47.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.422+0000 7f601aa00640 1 -- 172.21.15.6:0/1463775140 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f60040078e0 con 0x7f601c0c84f0 2024-04-24T23:16:47.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.422+0000 7f601aa00640 1 -- 172.21.15.6:0/1463775140 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6004007cb0 con 0x7f601c0c84f0 2024-04-24T23:16:47.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.423+0000 7f6022c00640 1 -- 172.21.15.6:0/1463775140 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f601c0c84f0 msgr2=0x7f601c15e3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:47.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.423+0000 7f6022c00640 1 --2- 172.21.15.6:0/1463775140 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f601c0c84f0 0x7f601c15e3c0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f601c073360 tx=0x7f600402a660 comp rx=0 tx=0).stop 2024-04-24T23:16:47.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.423+0000 7f6022c00640 1 -- 172.21.15.6:0/1463775140 shutdown_connections 2024-04-24T23:16:47.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.423+0000 7f6022c00640 1 --2- 172.21.15.6:0/1463775140 >> v2:172.21.15.6:3300/0 conn(0x7f601c15e900 0x7f601c072240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:47.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.423+0000 7f6022c00640 1 --2- 172.21.15.6:0/1463775140 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f601c0c84f0 0x7f601c15e3c0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:47.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.423+0000 7f6022c00640 1 --2- 172.21.15.6:0/1463775140 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f601c154f90 0x7f601c0c7eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:47.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.423+0000 7f6022c00640 1 -- 172.21.15.6:0/1463775140 >> 172.21.15.6:0/1463775140 conn(0x7f601c06e780 msgr2=0x7f601c06fdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:47.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.423+0000 7f6022c00640 1 -- 172.21.15.6:0/1463775140 shutdown_connections 2024-04-24T23:16:47.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.423+0000 7f6022c00640 1 -- 172.21.15.6:0/1463775140 wait complete. 2024-04-24T23:16:47.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.425+0000 7f6022c00640 1 Processor -- start 2024-04-24T23:16:47.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.425+0000 7f6022c00640 1 -- start start 2024-04-24T23:16:47.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.426+0000 7f6022c00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f601c0c84f0 0x7f601c1f13b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:47.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.426+0000 7f6022c00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f601c154f90 0x7f601c06b350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:47.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.426+0000 7f6022c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f601c15e900 0x7f601c1f3d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:47.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.426+0000 7f6022c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f601c15b7a0 con 0x7f601c15e900 2024-04-24T23:16:47.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.426+0000 7f6022c00640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f601c15b4c0 con 0x7f601c154f90 2024-04-24T23:16:47.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.426+0000 7f6022c00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f601c072780 con 0x7f601c0c84f0 2024-04-24T23:16:47.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.426+0000 7f6020c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f601c15e900 0x7f601c1f3d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:47.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.426+0000 7f6020c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f601c15e900 0x7f601c1f3d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36342/0 (socket says 172.21.15.6:36342) 2024-04-24T23:16:47.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.426+0000 7f6020c00640 1 -- 172.21.15.6:0/1604487985 learned_addr learned my addr 172.21.15.6:0/1604487985 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:16:47.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.426+0000 7f601be00640 1 --2- 172.21.15.6:0/1604487985 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f601c0c84f0 0x7f601c1f13b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:47.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.426+0000 7f601b400640 1 --2- 172.21.15.6:0/1604487985 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f601c154f90 0x7f601c06b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:47.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.427+0000 7f6020c00640 1 -- 172.21.15.6:0/1604487985 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f601c0c84f0 msgr2=0x7f601c1f13b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:47.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.427+0000 7f6020c00640 1 --2- 172.21.15.6:0/1604487985 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f601c0c84f0 0x7f601c1f13b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:47.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.427+0000 7f6020c00640 1 -- 172.21.15.6:0/1604487985 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f601c154f90 msgr2=0x7f601c06b350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:47.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.427+0000 7f6020c00640 1 --2- 172.21.15.6:0/1604487985 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f601c154f90 0x7f601c06b350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:47.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.427+0000 7f6020c00640 1 -- 172.21.15.6:0/1604487985 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f601c14aee0 con 0x7f601c15e900 2024-04-24T23:16:47.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.427+0000 7f6020c00640 1 --2- 172.21.15.6:0/1604487985 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f601c15e900 0x7f601c1f3d30 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f601400e8c0 tx=0x7f601400ed80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:47.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.427+0000 7f6013e00640 1 -- 172.21.15.6:0/1604487985 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6014017070 con 0x7f601c15e900 2024-04-24T23:16:47.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.427+0000 7f6013e00640 1 -- 172.21.15.6:0/1604487985 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6014004020 con 0x7f601c15e900 2024-04-24T23:16:47.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.427+0000 7f6022c00640 1 -- 172.21.15.6:0/1604487985 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f601c14b1a0 con 0x7f601c15e900 2024-04-24T23:16:47.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.428+0000 7f6013e00640 1 -- 172.21.15.6:0/1604487985 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60140043f0 con 0x7f601c15e900 2024-04-24T23:16:47.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.428+0000 7f6022c00640 1 -- 172.21.15.6:0/1604487985 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f601c1f08d0 con 0x7f601c15e900 2024-04-24T23:16:47.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.429+0000 7f6022c00640 1 -- 172.21.15.6:0/1604487985 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f601c0ca100 con 0x7f601c15e900 2024-04-24T23:16:47.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.430+0000 7f6013e00640 1 -- 172.21.15.6:0/1604487985 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f601400c940 con 0x7f601c15e900 2024-04-24T23:16:47.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.431+0000 7f6013e00640 1 --2- 172.21.15.6:0/1604487985 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f5ff806be60 0x7f5ff806e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:47.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.431+0000 7f6013e00640 1 -- 172.21.15.6:0/1604487985 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6590+0+0 (secure 0 0 0) 0x7f601408aff0 con 0x7f601c15e900 2024-04-24T23:16:47.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.431+0000 7f601be00640 1 --2- 172.21.15.6:0/1604487985 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f5ff806be60 0x7f5ff806e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:47.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.432+0000 7f601be00640 1 --2- 172.21.15.6:0/1604487985 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f5ff806be60 0x7f5ff806e320 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f600c009560 tx=0x7f600c009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:47.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.435+0000 7f6013e00640 1 -- 172.21.15.6:0/1604487985 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f601c0ca100 con 0x7f601c15e900 2024-04-24T23:16:47.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.673+0000 7f6022c00640 1 -- 172.21.15.6:0/1604487985 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f601c0c89f0 con 0x7f601c15e900 2024-04-24T23:16:47.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.675+0000 7f6013e00640 1 -- 172.21.15.6:0/1604487985 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) ==== 94+0+6902 (secure 0 0 0) 0x7f6014058ef0 con 0x7f601c15e900 2024-04-24T23:16:47.677 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:16:47.677 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":18,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4197,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/105267896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":105267896},{"type":"v1","addr":"172.21.15.6:6834","nonce":105267896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:16:32.310838+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4282,"mds_2":4235,"mds_3":4232,"mds_4":4203},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4203":{"gid":4203,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":17,"addr":"172.21.15.6:6837/2590421094","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":2590421094},{"type":"v1","addr":"172.21.15.6:6837","nonce":2590421094}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4232":{"gid":4232,"name":"b","rank":3,"incarnation":15,"state":"up:active","state_seq":15,"addr":"172.21.15.190:6838/2032280158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2032280158},{"type":"v1","addr":"172.21.15.190:6838","nonce":2032280158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4235":{"gid":4235,"name":"d","rank":2,"incarnation":13,"state":"up:active","state_seq":14,"addr":"172.21.15.190:6839/1572872451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":1572872451},{"type":"v1","addr":"172.21.15.190:6839","nonce":1572872451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4282":{"gid":4282,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":12,"addr":"172.21.15.190:6837/580315269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":580315269},{"type":"v1","addr":"172.21.15.190:6837","nonce":580315269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/489092393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":489092393},{"type":"v1","addr":"172.21.15.6:6835","nonce":489092393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4306,"qdb_cluster":[4203,4232,4235,4282,4306]},"id":1}]} 2024-04-24T23:16:47.678 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 18 2024-04-24T23:16:47.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.677+0000 7f6022c00640 1 -- 172.21.15.6:0/1604487985 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f5ff806be60 msgr2=0x7f5ff806e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:47.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.677+0000 7f6022c00640 1 --2- 172.21.15.6:0/1604487985 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f5ff806be60 0x7f5ff806e320 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f600c009560 tx=0x7f600c009190 comp rx=0 tx=0).stop 2024-04-24T23:16:47.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.677+0000 7f6022c00640 1 -- 172.21.15.6:0/1604487985 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f601c15e900 msgr2=0x7f601c1f3d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:47.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.677+0000 7f6022c00640 1 --2- 172.21.15.6:0/1604487985 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f601c15e900 0x7f601c1f3d30 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f601400e8c0 tx=0x7f601400ed80 comp rx=0 tx=0).stop 2024-04-24T23:16:47.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.677+0000 7f6022c00640 1 -- 172.21.15.6:0/1604487985 shutdown_connections 2024-04-24T23:16:47.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.677+0000 7f6022c00640 1 --2- 172.21.15.6:0/1604487985 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f5ff806be60 0x7f5ff806e320 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:47.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.677+0000 7f6022c00640 1 --2- 172.21.15.6:0/1604487985 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f601c15e900 0x7f601c1f3d30 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:47.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.677+0000 7f6022c00640 1 --2- 172.21.15.6:0/1604487985 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f601c154f90 0x7f601c06b350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:47.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.677+0000 7f6022c00640 1 --2- 172.21.15.6:0/1604487985 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f601c0c84f0 0x7f601c1f13b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:47.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.677+0000 7f6022c00640 1 -- 172.21.15.6:0/1604487985 >> 172.21.15.6:0/1604487985 conn(0x7f601c06e780 msgr2=0x7f601c0c6fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:47.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.678+0000 7f6022c00640 1 -- 172.21.15.6:0/1604487985 shutdown_connections 2024-04-24T23:16:47.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.678+0000 7f6022c00640 1 -- 172.21.15.6:0/1604487985 wait complete. 2024-04-24T23:16:47.698 INFO:tasks.mds_thrash:Ready to start thrashing 2024-04-24T23:16:47.698 INFO:tasks.mds_thrash.ceph_manager:waiting for clean 2024-04-24T23:16:47.699 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T23:16:47.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.902+0000 7f7645800640 1 Processor -- start 2024-04-24T23:16:47.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.902+0000 7f7645800640 1 -- start start 2024-04-24T23:16:47.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.903+0000 7f7645800640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7640152bc0 0x7f7640159390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:47.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.903+0000 7f7645800640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f764015cd20 con 0x7f76400c9330 2024-04-24T23:16:47.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.903+0000 7f7645800640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f76400c6e70 con 0x7f76400c8f50 2024-04-24T23:16:47.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.903+0000 7f7645800640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f76400c73d0 con 0x7f7640152bc0 2024-04-24T23:16:47.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.904+0000 7f763f400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f76400c9330 0x7f764015c610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:55202/0 (socket says 172.21.15.6:55202) 2024-04-24T23:16:47.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.904+0000 7f763f400640 1 -- 172.21.15.6:0/3575778320 learned_addr learned my addr 172.21.15.6:0/3575778320 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:16:47.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.904+0000 7f763ea00640 1 --2- 172.21.15.6:0/3575778320 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7640152bc0 0x7f7640159390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:47.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.904+0000 7f763d600640 1 -- 172.21.15.6:0/3575778320 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3231257986 0 0) 0x7f764015cd20 con 0x7f76400c9330 2024-04-24T23:16:47.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.904+0000 7f763d600640 1 -- 172.21.15.6:0/3575778320 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7624003520 con 0x7f76400c9330 2024-04-24T23:16:47.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.904+0000 7f763ea00640 1 -- 172.21.15.6:0/3575778320 >> v1:172.21.15.190:6789/0 conn(0x7f76400c8f50 legacy=0x7f764015bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:47.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.905+0000 7f763d600640 1 -- 172.21.15.6:0/3575778320 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3999968193 0 0) 0x7f76400c6e70 con 0x7f76400c8f50 2024-04-24T23:16:47.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.905+0000 7f763ea00640 1 -- 172.21.15.6:0/3575778320 >> v1:172.21.15.6:6789/0 conn(0x7f76400c9330 legacy=0x7f764015c610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:47.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.905+0000 7f763ea00640 1 -- 172.21.15.6:0/3575778320 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76401598d0 con 0x7f7640152bc0 2024-04-24T23:16:47.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.906+0000 7f763ea00640 1 --2- 172.21.15.6:0/3575778320 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7640152bc0 0x7f7640159390 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f76400c7550 tx=0x7f763002a160 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7caa3737237724aa server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:47.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.906+0000 7f763d600640 1 -- 172.21.15.6:0/3575778320 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7630037070 con 0x7f7640152bc0 2024-04-24T23:16:47.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.906+0000 7f763d600640 1 -- 172.21.15.6:0/3575778320 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7630032440 con 0x7f7640152bc0 2024-04-24T23:16:47.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.906+0000 7f763d600640 1 -- 172.21.15.6:0/3575778320 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76300327f0 con 0x7f7640152bc0 2024-04-24T23:16:47.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.906+0000 7f7645800640 1 -- 172.21.15.6:0/3575778320 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7640152bc0 msgr2=0x7f7640159390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:47.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.906+0000 7f7645800640 1 --2- 172.21.15.6:0/3575778320 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7640152bc0 0x7f7640159390 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f76400c7550 tx=0x7f763002a160 comp rx=0 tx=0).stop 2024-04-24T23:16:47.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.907+0000 7f7645800640 1 -- 172.21.15.6:0/3575778320 shutdown_connections 2024-04-24T23:16:47.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.907+0000 7f7645800640 1 --2- 172.21.15.6:0/3575778320 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7640152bc0 0x7f7640159390 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:47.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.907+0000 7f7645800640 1 -- 172.21.15.6:0/3575778320 >> 172.21.15.6:0/3575778320 conn(0x7f764006b0e0 msgr2=0x7f764006b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:47.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.907+0000 7f7645800640 1 -- 172.21.15.6:0/3575778320 shutdown_connections 2024-04-24T23:16:47.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.907+0000 7f7645800640 1 -- 172.21.15.6:0/3575778320 wait complete. 2024-04-24T23:16:47.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.908+0000 7f7645800640 1 Processor -- start 2024-04-24T23:16:47.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.908+0000 7f7645800640 1 -- start start 2024-04-24T23:16:47.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.909+0000 7f7645800640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f76400c8f50 0x7f76401486d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:47.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.909+0000 7f7645800640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f76400c9330 0x7f76401588e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:47.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.909+0000 7f7645800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7640152bc0 0x7f7640158e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:47.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.909+0000 7f7645800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f76400c6700 con 0x7f7640152bc0 2024-04-24T23:16:47.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.909+0000 7f7645800640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f76400c8ac0 con 0x7f76400c9330 2024-04-24T23:16:47.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.909+0000 7f7645800640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f76400c8470 con 0x7f76400c8f50 2024-04-24T23:16:47.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.910+0000 7f763f400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7640152bc0 0x7f7640158e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:47.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.910+0000 7f763f400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7640152bc0 0x7f7640158e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36358/0 (socket says 172.21.15.6:36358) 2024-04-24T23:16:47.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.910+0000 7f763f400640 1 -- 172.21.15.6:0/1173440676 learned_addr learned my addr 172.21.15.6:0/1173440676 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:16:47.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.910+0000 7f763ea00640 1 --2- 172.21.15.6:0/1173440676 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f76400c8f50 0x7f76401486d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:47.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.910+0000 7f763e000640 1 --2- 172.21.15.6:0/1173440676 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f76400c9330 0x7f76401588e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:47.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.910+0000 7f763f400640 1 -- 172.21.15.6:0/1173440676 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f76400c8f50 msgr2=0x7f76401486d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:47.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.910+0000 7f763f400640 1 --2- 172.21.15.6:0/1173440676 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f76400c8f50 0x7f76401486d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:47.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.910+0000 7f763f400640 1 -- 172.21.15.6:0/1173440676 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f76400c9330 msgr2=0x7f76401588e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:47.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.910+0000 7f763f400640 1 --2- 172.21.15.6:0/1173440676 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f76400c9330 0x7f76401588e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:47.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.910+0000 7f763f400640 1 -- 172.21.15.6:0/1173440676 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76401522a0 con 0x7f7640152bc0 2024-04-24T23:16:47.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.910+0000 7f763ea00640 1 --2- 172.21.15.6:0/1173440676 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f76400c8f50 0x7f76401486d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:16:47.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.911+0000 7f763f400640 1 --2- 172.21.15.6:0/1173440676 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7640152bc0 0x7f7640158e20 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f763800cb40 tx=0x7f763800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:47.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.911+0000 7f7636a00640 1 -- 172.21.15.6:0/1173440676 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7638017070 con 0x7f7640152bc0 2024-04-24T23:16:47.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.911+0000 7f7636a00640 1 -- 172.21.15.6:0/1173440676 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f763800aca0 con 0x7f7640152bc0 2024-04-24T23:16:47.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.911+0000 7f7636a00640 1 -- 172.21.15.6:0/1173440676 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7638012620 con 0x7f7640152bc0 2024-04-24T23:16:47.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.911+0000 7f7645800640 1 -- 172.21.15.6:0/1173440676 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76401f08c0 con 0x7f7640152bc0 2024-04-24T23:16:47.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.911+0000 7f7645800640 1 -- 172.21.15.6:0/1173440676 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76401f0e80 con 0x7f7640152bc0 2024-04-24T23:16:47.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.912+0000 7f7645800640 1 -- 172.21.15.6:0/1173440676 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7640152700 con 0x7f7640152bc0 2024-04-24T23:16:47.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.913+0000 7f7636a00640 1 -- 172.21.15.6:0/1173440676 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f763801a070 con 0x7f7640152bc0 2024-04-24T23:16:47.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.914+0000 7f7636a00640 1 --2- 172.21.15.6:0/1173440676 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f762006be60 0x7f762006e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:47.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.914+0000 7f7636a00640 1 -- 172.21.15.6:0/1173440676 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6590+0+0 (secure 0 0 0) 0x7f763808af30 con 0x7f7640152bc0 2024-04-24T23:16:47.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.914+0000 7f763ea00640 1 --2- 172.21.15.6:0/1173440676 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f762006be60 0x7f762006e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:47.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.915+0000 7f763ea00640 1 --2- 172.21.15.6:0/1173440676 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f762006be60 0x7f762006e320 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f7630034000 tx=0x7f763002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:47.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:47.919+0000 7f7636a00640 1 -- 172.21.15.6:0/1173440676 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f7638058f10 con 0x7f7640152bc0 2024-04-24T23:16:48.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:48.121+0000 7f7645800640 1 -- 172.21.15.6:0/1173440676 --> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f76400767b0 con 0x7f762006be60 2024-04-24T23:16:48.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:48.131+0000 7f7636a00640 1 -- 172.21.15.6:0/1173440676 <== mgr.4105 v2:172.21.15.190:6832/806769663 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+332657 (secure 0 0 0) 0x7f76400767b0 con 0x7f762006be60 2024-04-24T23:16:48.138 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:16:48.140 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-04-24T23:16:48.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:48.138+0000 7f7645800640 1 -- 172.21.15.6:0/1173440676 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f762006be60 msgr2=0x7f762006e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:48.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:48.138+0000 7f7645800640 1 --2- 172.21.15.6:0/1173440676 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f762006be60 0x7f762006e320 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f7630034000 tx=0x7f763002e000 comp rx=0 tx=0).stop 2024-04-24T23:16:48.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:48.138+0000 7f7645800640 1 -- 172.21.15.6:0/1173440676 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7640152bc0 msgr2=0x7f7640158e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:48.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:48.138+0000 7f7645800640 1 --2- 172.21.15.6:0/1173440676 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7640152bc0 0x7f7640158e20 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f763800cb40 tx=0x7f763800a060 comp rx=0 tx=0).stop 2024-04-24T23:16:48.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:48.139+0000 7f7645800640 1 -- 172.21.15.6:0/1173440676 shutdown_connections 2024-04-24T23:16:48.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:48.139+0000 7f7645800640 1 --2- 172.21.15.6:0/1173440676 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f762006be60 0x7f762006e320 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:48.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:48.139+0000 7f7645800640 1 --2- 172.21.15.6:0/1173440676 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7640152bc0 0x7f7640158e20 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:48.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:48.139+0000 7f7645800640 1 --2- 172.21.15.6:0/1173440676 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f76400c9330 0x7f76401588e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:48.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:48.139+0000 7f7645800640 1 --2- 172.21.15.6:0/1173440676 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f76400c8f50 0x7f76401486d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:48.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:48.139+0000 7f7645800640 1 -- 172.21.15.6:0/1173440676 >> 172.21.15.6:0/1173440676 conn(0x7f764006b0e0 msgr2=0x7f764014f570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:48.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:48.139+0000 7f7645800640 1 -- 172.21.15.6:0/1173440676 shutdown_connections 2024-04-24T23:16:48.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:48.139+0000 7f7645800640 1 -- 172.21.15.6:0/1173440676 wait complete. 2024-04-24T23:16:48.161 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":60,"stamp":"2024-04-24T23:16:46.698415+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":600890,"num_objects":100,"num_object_clones":0,"num_object_copies":200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":370,"num_write_kb":2485,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":302,"ondisk_log_size":302,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219720,"kb_used_data":3592,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755000,"statfs":{"total":773094113280,"available":772869120000,"internally_reserved":0,"allocated":3678208,"data_stored":2113124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1664,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":43,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003284"},"pg_stats":[{"pgid":"2.1f","version":"26'13","reported_seq":39,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:31.303006+0000","last_change":"2024-04-24T23:15:35.964646+0000","last_active":"2024-04-24T23:16:31.303006+0000","last_peered":"2024-04-24T23:16:31.303006+0000","last_clean":"2024-04-24T23:16:31.303006+0000","last_became_active":"2024-04-24T23:15:35.964357+0000","last_became_peered":"2024-04-24T23:15:35.964357+0000","last_unstale":"2024-04-24T23:16:31.303006+0000","last_undegraded":"2024-04-24T23:16:31.303006+0000","last_fullsized":"2024-04-24T23:16:31.303006+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:01:17.912993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.292135+0000","last_change":"2024-04-24T23:15:36.989772+0000","last_active":"2024-04-24T23:16:18.292135+0000","last_peered":"2024-04-24T23:16:18.292135+0000","last_clean":"2024-04-24T23:16:18.292135+0000","last_became_active":"2024-04-24T23:15:36.989543+0000","last_became_peered":"2024-04-24T23:15:36.989543+0000","last_unstale":"2024-04-24T23:16:18.292135+0000","last_undegraded":"2024-04-24T23:16:18.292135+0000","last_fullsized":"2024-04-24T23:16:18.292135+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:37:16.737354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.303989+0000","last_change":"2024-04-24T23:16:18.300096+0000","last_active":"2024-04-24T23:16:18.303989+0000","last_peered":"2024-04-24T23:16:18.303989+0000","last_clean":"2024-04-24T23:16:18.303989+0000","last_became_active":"2024-04-24T23:16:18.299217+0000","last_became_peered":"2024-04-24T23:16:18.299217+0000","last_unstale":"2024-04-24T23:16:18.303989+0000","last_undegraded":"2024-04-24T23:16:18.303989+0000","last_fullsized":"2024-04-24T23:16:18.303989+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:08:19.426862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4,1],"acting":[0,6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"26'2","reported_seq":28,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.292303+0000","last_change":"2024-04-24T23:15:35.946083+0000","last_active":"2024-04-24T23:16:18.292303+0000","last_peered":"2024-04-24T23:16:18.292303+0000","last_clean":"2024-04-24T23:16:18.292303+0000","last_became_active":"2024-04-24T23:15:35.945227+0000","last_became_peered":"2024-04-24T23:15:35.945227+0000","last_unstale":"2024-04-24T23:16:18.292303+0000","last_undegraded":"2024-04-24T23:16:18.292303+0000","last_fullsized":"2024-04-24T23:16:18.292303+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:05:17.727725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.292034+0000","last_change":"2024-04-24T23:15:36.998245+0000","last_active":"2024-04-24T23:16:18.292034+0000","last_peered":"2024-04-24T23:16:18.292034+0000","last_clean":"2024-04-24T23:16:18.292034+0000","last_became_active":"2024-04-24T23:15:36.998021+0000","last_became_peered":"2024-04-24T23:15:36.998021+0000","last_unstale":"2024-04-24T23:16:18.292034+0000","last_undegraded":"2024-04-24T23:16:18.292034+0000","last_fullsized":"2024-04-24T23:16:18.292034+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:03:42.359117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.293518+0000","last_change":"2024-04-24T23:15:41.008313+0000","last_active":"2024-04-24T23:16:18.293518+0000","last_peered":"2024-04-24T23:16:18.293518+0000","last_clean":"2024-04-24T23:16:18.293518+0000","last_became_active":"2024-04-24T23:15:41.007908+0000","last_became_peered":"2024-04-24T23:15:41.007908+0000","last_unstale":"2024-04-24T23:16:18.293518+0000","last_undegraded":"2024-04-24T23:16:18.293518+0000","last_fullsized":"2024-04-24T23:16:18.293518+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:02:11.902575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4,6],"acting":[2,7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1d","version":"26'12","reported_seq":38,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:31.305540+0000","last_change":"2024-04-24T23:15:35.968312+0000","last_active":"2024-04-24T23:16:31.305540+0000","last_peered":"2024-04-24T23:16:31.305540+0000","last_clean":"2024-04-24T23:16:31.305540+0000","last_became_active":"2024-04-24T23:15:35.967873+0000","last_became_peered":"2024-04-24T23:15:35.967873+0000","last_unstale":"2024-04-24T23:16:31.305540+0000","last_undegraded":"2024-04-24T23:16:31.305540+0000","last_fullsized":"2024-04-24T23:16:31.305540+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:21:40.116977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.296594+0000","last_change":"2024-04-24T23:15:36.997361+0000","last_active":"2024-04-24T23:16:18.296594+0000","last_peered":"2024-04-24T23:16:18.296594+0000","last_clean":"2024-04-24T23:16:18.296594+0000","last_became_active":"2024-04-24T23:15:36.997083+0000","last_became_peered":"2024-04-24T23:15:36.997083+0000","last_unstale":"2024-04-24T23:16:18.296594+0000","last_undegraded":"2024-04-24T23:16:18.296594+0000","last_fullsized":"2024-04-24T23:16:18.296594+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:47:59.334093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.293518+0000","last_change":"2024-04-24T23:15:41.025153+0000","last_active":"2024-04-24T23:16:18.293518+0000","last_peered":"2024-04-24T23:16:18.293518+0000","last_clean":"2024-04-24T23:16:18.293518+0000","last_became_active":"2024-04-24T23:15:41.024739+0000","last_became_peered":"2024-04-24T23:15:41.024739+0000","last_unstale":"2024-04-24T23:16:18.293518+0000","last_undegraded":"2024-04-24T23:16:18.293518+0000","last_fullsized":"2024-04-24T23:16:18.293518+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:31:00.160213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3,6],"acting":[2,5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1c","version":"26'5","reported_seq":31,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:31.301798+0000","last_change":"2024-04-24T23:15:35.967338+0000","last_active":"2024-04-24T23:16:31.301798+0000","last_peered":"2024-04-24T23:16:31.301798+0000","last_clean":"2024-04-24T23:16:31.301798+0000","last_became_active":"2024-04-24T23:15:35.966951+0000","last_became_peered":"2024-04-24T23:15:35.966951+0000","last_unstale":"2024-04-24T23:16:31.301798+0000","last_undegraded":"2024-04-24T23:16:31.301798+0000","last_fullsized":"2024-04-24T23:16:31.301798+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:08:05.248330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.230858+0000","last_change":"2024-04-24T23:15:36.998072+0000","last_active":"2024-04-24T23:16:18.230858+0000","last_peered":"2024-04-24T23:16:18.230858+0000","last_clean":"2024-04-24T23:16:18.230858+0000","last_became_active":"2024-04-24T23:15:36.997735+0000","last_became_peered":"2024-04-24T23:15:36.997735+0000","last_unstale":"2024-04-24T23:16:18.230858+0000","last_undegraded":"2024-04-24T23:16:18.230858+0000","last_fullsized":"2024-04-24T23:16:18.230858+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:22:06.869644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.294029+0000","last_change":"2024-04-24T23:15:41.008895+0000","last_active":"2024-04-24T23:16:18.294029+0000","last_peered":"2024-04-24T23:16:18.294029+0000","last_clean":"2024-04-24T23:16:18.294029+0000","last_became_active":"2024-04-24T23:15:41.006199+0000","last_became_peered":"2024-04-24T23:15:41.006199+0000","last_unstale":"2024-04-24T23:16:18.294029+0000","last_undegraded":"2024-04-24T23:16:18.294029+0000","last_fullsized":"2024-04-24T23:16:18.294029+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:13:34.583602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5,7],"acting":[2,6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.233639+0000","last_change":"2024-04-24T23:15:37.004074+0000","last_active":"2024-04-24T23:16:18.233639+0000","last_peered":"2024-04-24T23:16:18.233639+0000","last_clean":"2024-04-24T23:16:18.233639+0000","last_became_active":"2024-04-24T23:15:37.003544+0000","last_became_peered":"2024-04-24T23:15:37.003544+0000","last_unstale":"2024-04-24T23:16:18.233639+0000","last_undegraded":"2024-04-24T23:16:18.233639+0000","last_fullsized":"2024-04-24T23:16:18.233639+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:38:28.024702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"24'4","reported_seq":30,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.230075+0000","last_change":"2024-04-24T23:15:35.963000+0000","last_active":"2024-04-24T23:16:18.230075+0000","last_peered":"2024-04-24T23:16:18.230075+0000","last_clean":"2024-04-24T23:16:18.230075+0000","last_became_active":"2024-04-24T23:15:35.961960+0000","last_became_peered":"2024-04-24T23:15:35.961960+0000","last_unstale":"2024-04-24T23:16:18.230075+0000","last_undegraded":"2024-04-24T23:16:18.230075+0000","last_fullsized":"2024-04-24T23:16:18.230075+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:56:26.651193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.230595+0000","last_change":"2024-04-24T23:15:41.022481+0000","last_active":"2024-04-24T23:16:18.230595+0000","last_peered":"2024-04-24T23:16:18.230595+0000","last_clean":"2024-04-24T23:16:18.230595+0000","last_became_active":"2024-04-24T23:15:41.020034+0000","last_became_peered":"2024-04-24T23:15:41.020034+0000","last_unstale":"2024-04-24T23:16:18.230595+0000","last_undegraded":"2024-04-24T23:16:18.230595+0000","last_fullsized":"2024-04-24T23:16:18.230595+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:27:25.337503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,5],"acting":[3,1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.292564+0000","last_change":"2024-04-24T23:15:36.972020+0000","last_active":"2024-04-24T23:16:18.292564+0000","last_peered":"2024-04-24T23:16:18.292564+0000","last_clean":"2024-04-24T23:16:18.292564+0000","last_became_active":"2024-04-24T23:15:36.971583+0000","last_became_peered":"2024-04-24T23:15:36.971583+0000","last_unstale":"2024-04-24T23:16:18.292564+0000","last_undegraded":"2024-04-24T23:16:18.292564+0000","last_fullsized":"2024-04-24T23:16:18.292564+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:29:05.568608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"26'3","reported_seq":29,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:25.265282+0000","last_change":"2024-04-24T23:15:35.945916+0000","last_active":"2024-04-24T23:16:25.265282+0000","last_peered":"2024-04-24T23:16:25.265282+0000","last_clean":"2024-04-24T23:16:25.265282+0000","last_became_active":"2024-04-24T23:15:35.945610+0000","last_became_peered":"2024-04-24T23:15:35.945610+0000","last_unstale":"2024-04-24T23:16:25.265282+0000","last_undegraded":"2024-04-24T23:16:25.265282+0000","last_fullsized":"2024-04-24T23:16:25.265282+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:57:05.976074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.229068+0000","last_change":"2024-04-24T23:15:41.030165+0000","last_active":"2024-04-24T23:16:18.229068+0000","last_peered":"2024-04-24T23:16:18.229068+0000","last_clean":"2024-04-24T23:16:18.229068+0000","last_became_active":"2024-04-24T23:15:41.029685+0000","last_became_peered":"2024-04-24T23:15:41.029685+0000","last_unstale":"2024-04-24T23:16:18.229068+0000","last_undegraded":"2024-04-24T23:16:18.229068+0000","last_fullsized":"2024-04-24T23:16:18.229068+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:20:43.498190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6,1],"acting":[0,4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.223468+0000","last_change":"2024-04-24T23:15:36.965949+0000","last_active":"2024-04-24T23:16:18.223468+0000","last_peered":"2024-04-24T23:16:18.223468+0000","last_clean":"2024-04-24T23:16:18.223468+0000","last_became_active":"2024-04-24T23:15:36.965658+0000","last_became_peered":"2024-04-24T23:15:36.965658+0000","last_unstale":"2024-04-24T23:16:18.223468+0000","last_undegraded":"2024-04-24T23:16:18.223468+0000","last_fullsized":"2024-04-24T23:16:18.223468+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:38:28.024702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"26'7","reported_seq":33,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:31.301052+0000","last_change":"2024-04-24T23:15:35.972165+0000","last_active":"2024-04-24T23:16:31.301052+0000","last_peered":"2024-04-24T23:16:31.301052+0000","last_clean":"2024-04-24T23:16:31.301052+0000","last_became_active":"2024-04-24T23:15:35.971972+0000","last_became_peered":"2024-04-24T23:15:35.971972+0000","last_unstale":"2024-04-24T23:16:31.301052+0000","last_undegraded":"2024-04-24T23:16:31.301052+0000","last_fullsized":"2024-04-24T23:16:31.301052+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:11:12.088698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.292911+0000","last_change":"2024-04-24T23:15:41.019799+0000","last_active":"2024-04-24T23:16:18.292911+0000","last_peered":"2024-04-24T23:16:18.292911+0000","last_clean":"2024-04-24T23:16:18.292911+0000","last_became_active":"2024-04-24T23:15:41.019339+0000","last_became_peered":"2024-04-24T23:15:41.019339+0000","last_unstale":"2024-04-24T23:16:18.292911+0000","last_undegraded":"2024-04-24T23:16:18.292911+0000","last_fullsized":"2024-04-24T23:16:18.292911+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:59:34.601004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2,3],"acting":[7,4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.232810+0000","last_change":"2024-04-24T23:15:36.991991+0000","last_active":"2024-04-24T23:16:18.232810+0000","last_peered":"2024-04-24T23:16:18.232810+0000","last_clean":"2024-04-24T23:16:18.232810+0000","last_became_active":"2024-04-24T23:15:36.989336+0000","last_became_peered":"2024-04-24T23:15:36.989336+0000","last_unstale":"2024-04-24T23:16:18.232810+0000","last_undegraded":"2024-04-24T23:16:18.232810+0000","last_fullsized":"2024-04-24T23:16:18.232810+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:25:56.038903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.8","version":"24'2","reported_seq":28,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.228663+0000","last_change":"2024-04-24T23:15:35.965443+0000","last_active":"2024-04-24T23:16:18.228663+0000","last_peered":"2024-04-24T23:16:18.228663+0000","last_clean":"2024-04-24T23:16:18.228663+0000","last_became_active":"2024-04-24T23:15:35.965043+0000","last_became_peered":"2024-04-24T23:15:35.965043+0000","last_unstale":"2024-04-24T23:16:18.228663+0000","last_undegraded":"2024-04-24T23:16:18.228663+0000","last_fullsized":"2024-04-24T23:16:18.228663+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:07:23.320042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.294499+0000","last_change":"2024-04-24T23:15:41.025131+0000","last_active":"2024-04-24T23:16:18.294499+0000","last_peered":"2024-04-24T23:16:18.294499+0000","last_clean":"2024-04-24T23:16:18.294499+0000","last_became_active":"2024-04-24T23:15:41.024731+0000","last_became_peered":"2024-04-24T23:15:41.024731+0000","last_unstale":"2024-04-24T23:16:18.294499+0000","last_undegraded":"2024-04-24T23:16:18.294499+0000","last_fullsized":"2024-04-24T23:16:18.294499+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:28:07.952854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,7],"acting":[2,1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.224465+0000","last_change":"2024-04-24T23:15:36.965850+0000","last_active":"2024-04-24T23:16:18.224465+0000","last_peered":"2024-04-24T23:16:18.224465+0000","last_clean":"2024-04-24T23:16:18.224465+0000","last_became_active":"2024-04-24T23:15:36.965547+0000","last_became_peered":"2024-04-24T23:15:36.965547+0000","last_unstale":"2024-04-24T23:16:18.224465+0000","last_undegraded":"2024-04-24T23:16:18.224465+0000","last_fullsized":"2024-04-24T23:16:18.224465+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:06:37.125573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"26'7","reported_seq":33,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:31.305921+0000","last_change":"2024-04-24T23:15:35.971040+0000","last_active":"2024-04-24T23:16:31.305921+0000","last_peered":"2024-04-24T23:16:31.305921+0000","last_clean":"2024-04-24T23:16:31.305921+0000","last_became_active":"2024-04-24T23:15:35.970184+0000","last_became_peered":"2024-04-24T23:15:35.970184+0000","last_unstale":"2024-04-24T23:16:31.305921+0000","last_undegraded":"2024-04-24T23:16:31.305921+0000","last_fullsized":"2024-04-24T23:16:31.305921+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:45:20.913759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.295046+0000","last_change":"2024-04-24T23:15:41.025813+0000","last_active":"2024-04-24T23:16:18.295046+0000","last_peered":"2024-04-24T23:16:18.295046+0000","last_clean":"2024-04-24T23:16:18.295046+0000","last_became_active":"2024-04-24T23:15:41.025137+0000","last_became_peered":"2024-04-24T23:15:41.025137+0000","last_unstale":"2024-04-24T23:16:18.295046+0000","last_undegraded":"2024-04-24T23:16:18.295046+0000","last_fullsized":"2024-04-24T23:16:18.295046+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:35:17.784854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,3,1],"acting":[2,6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.224731+0000","last_change":"2024-04-24T23:15:36.967476+0000","last_active":"2024-04-24T23:16:18.224731+0000","last_peered":"2024-04-24T23:16:18.224731+0000","last_clean":"2024-04-24T23:16:18.224731+0000","last_became_active":"2024-04-24T23:15:36.967098+0000","last_became_peered":"2024-04-24T23:15:36.967098+0000","last_unstale":"2024-04-24T23:16:18.224731+0000","last_undegraded":"2024-04-24T23:16:18.224731+0000","last_fullsized":"2024-04-24T23:16:18.224731+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:25:56.038903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"26'7","reported_seq":33,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:25.270982+0000","last_change":"2024-04-24T23:15:35.970630+0000","last_active":"2024-04-24T23:16:25.270982+0000","last_peered":"2024-04-24T23:16:25.270982+0000","last_clean":"2024-04-24T23:16:25.270982+0000","last_became_active":"2024-04-24T23:15:35.970430+0000","last_became_peered":"2024-04-24T23:15:35.970430+0000","last_unstale":"2024-04-24T23:16:25.270982+0000","last_undegraded":"2024-04-24T23:16:25.270982+0000","last_fullsized":"2024-04-24T23:16:25.270982+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:15:59.219084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.223468+0000","last_change":"2024-04-24T23:15:41.005022+0000","last_active":"2024-04-24T23:16:18.223468+0000","last_peered":"2024-04-24T23:16:18.223468+0000","last_clean":"2024-04-24T23:16:18.223468+0000","last_became_active":"2024-04-24T23:15:41.004648+0000","last_became_peered":"2024-04-24T23:15:41.004648+0000","last_unstale":"2024-04-24T23:16:18.223468+0000","last_undegraded":"2024-04-24T23:16:18.223468+0000","last_fullsized":"2024-04-24T23:16:18.223468+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:06:01.559760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6,3],"acting":[4,1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.225003+0000","last_change":"2024-04-24T23:15:36.967053+0000","last_active":"2024-04-24T23:16:18.225003+0000","last_peered":"2024-04-24T23:16:18.225003+0000","last_clean":"2024-04-24T23:16:18.225003+0000","last_became_active":"2024-04-24T23:15:36.966746+0000","last_became_peered":"2024-04-24T23:15:36.966746+0000","last_unstale":"2024-04-24T23:16:18.225003+0000","last_undegraded":"2024-04-24T23:16:18.225003+0000","last_fullsized":"2024-04-24T23:16:18.225003+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:16:08.779860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"26'4","reported_seq":30,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:25.272679+0000","last_change":"2024-04-24T23:15:35.965890+0000","last_active":"2024-04-24T23:16:25.272679+0000","last_peered":"2024-04-24T23:16:25.272679+0000","last_clean":"2024-04-24T23:16:25.272679+0000","last_became_active":"2024-04-24T23:15:35.965631+0000","last_became_peered":"2024-04-24T23:15:35.965631+0000","last_unstale":"2024-04-24T23:16:25.272679+0000","last_undegraded":"2024-04-24T23:16:25.272679+0000","last_fullsized":"2024-04-24T23:16:25.272679+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:17:35.650731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.223982+0000","last_change":"2024-04-24T23:15:41.011405+0000","last_active":"2024-04-24T23:16:18.223982+0000","last_peered":"2024-04-24T23:16:18.223982+0000","last_clean":"2024-04-24T23:16:18.223982+0000","last_became_active":"2024-04-24T23:15:41.010982+0000","last_became_peered":"2024-04-24T23:15:41.010982+0000","last_unstale":"2024-04-24T23:16:18.223982+0000","last_undegraded":"2024-04-24T23:16:18.223982+0000","last_fullsized":"2024-04-24T23:16:18.223982+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:08:00.914296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2,7],"acting":[6,0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.294828+0000","last_change":"2024-04-24T23:15:36.975244+0000","last_active":"2024-04-24T23:16:18.294828+0000","last_peered":"2024-04-24T23:16:18.294828+0000","last_clean":"2024-04-24T23:16:18.294828+0000","last_became_active":"2024-04-24T23:15:36.974872+0000","last_became_peered":"2024-04-24T23:15:36.974872+0000","last_unstale":"2024-04-24T23:16:18.294828+0000","last_undegraded":"2024-04-24T23:16:18.294828+0000","last_fullsized":"2024-04-24T23:16:18.294828+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:45:57.018205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"26'1","reported_seq":27,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.291824+0000","last_change":"2024-04-24T23:15:35.945792+0000","last_active":"2024-04-24T23:16:18.291824+0000","last_peered":"2024-04-24T23:16:18.291824+0000","last_clean":"2024-04-24T23:16:18.291824+0000","last_became_active":"2024-04-24T23:15:35.945094+0000","last_became_peered":"2024-04-24T23:15:35.945094+0000","last_unstale":"2024-04-24T23:16:18.291824+0000","last_undegraded":"2024-04-24T23:16:18.291824+0000","last_fullsized":"2024-04-24T23:16:18.291824+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:49:12.719127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.224614+0000","last_change":"2024-04-24T23:15:41.022105+0000","last_active":"2024-04-24T23:16:18.224614+0000","last_peered":"2024-04-24T23:16:18.224614+0000","last_clean":"2024-04-24T23:16:18.224614+0000","last_became_active":"2024-04-24T23:15:41.021645+0000","last_became_peered":"2024-04-24T23:15:41.021645+0000","last_unstale":"2024-04-24T23:16:18.224614+0000","last_undegraded":"2024-04-24T23:16:18.224614+0000","last_fullsized":"2024-04-24T23:16:18.224614+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:03:52.974835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5,7],"acting":[6,3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.229269+0000","last_change":"2024-04-24T23:15:36.991239+0000","last_active":"2024-04-24T23:16:18.229269+0000","last_peered":"2024-04-24T23:16:18.229269+0000","last_clean":"2024-04-24T23:16:18.229269+0000","last_became_active":"2024-04-24T23:15:36.990963+0000","last_became_peered":"2024-04-24T23:15:36.990963+0000","last_unstale":"2024-04-24T23:16:18.229269+0000","last_undegraded":"2024-04-24T23:16:18.229269+0000","last_fullsized":"2024-04-24T23:16:18.229269+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:53:28.101334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"26'5","reported_seq":31,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:31.304824+0000","last_change":"2024-04-24T23:15:35.957931+0000","last_active":"2024-04-24T23:16:31.304824+0000","last_peered":"2024-04-24T23:16:31.304824+0000","last_clean":"2024-04-24T23:16:31.304824+0000","last_became_active":"2024-04-24T23:15:35.957432+0000","last_became_peered":"2024-04-24T23:15:35.957432+0000","last_unstale":"2024-04-24T23:16:31.304824+0000","last_undegraded":"2024-04-24T23:16:31.304824+0000","last_fullsized":"2024-04-24T23:16:31.304824+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:11:38.773425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.224009+0000","last_change":"2024-04-24T23:15:41.007857+0000","last_active":"2024-04-24T23:16:18.224009+0000","last_peered":"2024-04-24T23:16:18.224009+0000","last_clean":"2024-04-24T23:16:18.224009+0000","last_became_active":"2024-04-24T23:15:41.007449+0000","last_became_peered":"2024-04-24T23:15:41.007449+0000","last_unstale":"2024-04-24T23:16:18.224009+0000","last_undegraded":"2024-04-24T23:16:18.224009+0000","last_fullsized":"2024-04-24T23:16:18.224009+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:05:06.145440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0,3],"acting":[6,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.291576+0000","last_change":"2024-04-24T23:15:36.970444+0000","last_active":"2024-04-24T23:16:18.291576+0000","last_peered":"2024-04-24T23:16:18.291576+0000","last_clean":"2024-04-24T23:16:18.291576+0000","last_became_active":"2024-04-24T23:15:36.970159+0000","last_became_peered":"2024-04-24T23:15:36.970159+0000","last_unstale":"2024-04-24T23:16:18.291576+0000","last_undegraded":"2024-04-24T23:16:18.291576+0000","last_fullsized":"2024-04-24T23:16:18.291576+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:37:02.956224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"26'5","reported_seq":31,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:19.220734+0000","last_change":"2024-04-24T23:15:35.945461+0000","last_active":"2024-04-24T23:16:19.220734+0000","last_peered":"2024-04-24T23:16:19.220734+0000","last_clean":"2024-04-24T23:16:19.220734+0000","last_became_active":"2024-04-24T23:15:35.945185+0000","last_became_peered":"2024-04-24T23:15:35.945185+0000","last_unstale":"2024-04-24T23:16:19.220734+0000","last_undegraded":"2024-04-24T23:16:19.220734+0000","last_fullsized":"2024-04-24T23:16:19.220734+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:37:34.175539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.224370+0000","last_change":"2024-04-24T23:15:41.014584+0000","last_active":"2024-04-24T23:16:18.224370+0000","last_peered":"2024-04-24T23:16:18.224370+0000","last_clean":"2024-04-24T23:16:18.224370+0000","last_became_active":"2024-04-24T23:15:41.014137+0000","last_became_peered":"2024-04-24T23:15:41.014137+0000","last_unstale":"2024-04-24T23:16:18.224370+0000","last_undegraded":"2024-04-24T23:16:18.224370+0000","last_fullsized":"2024-04-24T23:16:18.224370+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:23:50.387635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,7],"acting":[6,0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.224277+0000","last_change":"2024-04-24T23:15:36.981783+0000","last_active":"2024-04-24T23:16:18.224277+0000","last_peered":"2024-04-24T23:16:18.224277+0000","last_clean":"2024-04-24T23:16:18.224277+0000","last_became_active":"2024-04-24T23:15:36.981502+0000","last_became_peered":"2024-04-24T23:15:36.981502+0000","last_unstale":"2024-04-24T23:16:18.224277+0000","last_undegraded":"2024-04-24T23:16:18.224277+0000","last_fullsized":"2024-04-24T23:16:18.224277+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:56:08.483410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"26'4","reported_seq":30,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:19.225767+0000","last_change":"2024-04-24T23:15:35.960519+0000","last_active":"2024-04-24T23:16:19.225767+0000","last_peered":"2024-04-24T23:16:19.225767+0000","last_clean":"2024-04-24T23:16:19.225767+0000","last_became_active":"2024-04-24T23:15:35.960180+0000","last_became_peered":"2024-04-24T23:15:35.960180+0000","last_unstale":"2024-04-24T23:16:19.225767+0000","last_undegraded":"2024-04-24T23:16:19.225767+0000","last_fullsized":"2024-04-24T23:16:19.225767+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:50:00.199120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.291646+0000","last_change":"2024-04-24T23:15:41.012876+0000","last_active":"2024-04-24T23:16:18.291646+0000","last_peered":"2024-04-24T23:16:18.291646+0000","last_clean":"2024-04-24T23:16:18.291646+0000","last_became_active":"2024-04-24T23:15:41.012493+0000","last_became_peered":"2024-04-24T23:15:41.012493+0000","last_unstale":"2024-04-24T23:16:18.291646+0000","last_undegraded":"2024-04-24T23:16:18.291646+0000","last_fullsized":"2024-04-24T23:16:18.291646+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:21:28.287059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6,1],"acting":[5,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.224197+0000","last_change":"2024-04-24T23:15:36.981449+0000","last_active":"2024-04-24T23:16:18.224197+0000","last_peered":"2024-04-24T23:16:18.224197+0000","last_clean":"2024-04-24T23:16:18.224197+0000","last_became_active":"2024-04-24T23:15:36.980837+0000","last_became_peered":"2024-04-24T23:15:36.980837+0000","last_unstale":"2024-04-24T23:16:18.224197+0000","last_undegraded":"2024-04-24T23:16:18.224197+0000","last_fullsized":"2024-04-24T23:16:18.224197+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:15:13.513341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.3","version":"26'6","reported_seq":32,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:25.276387+0000","last_change":"2024-04-24T23:15:35.952947+0000","last_active":"2024-04-24T23:16:25.276387+0000","last_peered":"2024-04-24T23:16:25.276387+0000","last_clean":"2024-04-24T23:16:25.276387+0000","last_became_active":"2024-04-24T23:15:35.952655+0000","last_became_peered":"2024-04-24T23:15:35.952655+0000","last_unstale":"2024-04-24T23:16:25.276387+0000","last_undegraded":"2024-04-24T23:16:25.276387+0000","last_fullsized":"2024-04-24T23:16:25.276387+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:50:10.939396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":182,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.296073+0000","last_change":"2024-04-24T23:15:25.826673+0000","last_active":"2024-04-24T23:16:18.296073+0000","last_peered":"2024-04-24T23:16:18.296073+0000","last_clean":"2024-04-24T23:16:18.296073+0000","last_became_active":"2024-04-24T23:15:25.820030+0000","last_became_peered":"2024-04-24T23:15:25.820030+0000","last_unstale":"2024-04-24T23:16:18.296073+0000","last_undegraded":"2024-04-24T23:16:18.296073+0000","last_fullsized":"2024-04-24T23:16:18.296073+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:24.790928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:24.790928+0000","last_clean_scrub_stamp":"2024-04-24T23:15:24.790928+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:37:24.054332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.231043+0000","last_change":"2024-04-24T23:15:41.024712+0000","last_active":"2024-04-24T23:16:18.231043+0000","last_peered":"2024-04-24T23:16:18.231043+0000","last_clean":"2024-04-24T23:16:18.231043+0000","last_became_active":"2024-04-24T23:15:41.022697+0000","last_became_peered":"2024-04-24T23:15:41.022697+0000","last_unstale":"2024-04-24T23:16:18.231043+0000","last_undegraded":"2024-04-24T23:16:18.231043+0000","last_fullsized":"2024-04-24T23:16:18.231043+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:37:08.231105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2,0],"acting":[3,1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.230951+0000","last_change":"2024-04-24T23:15:37.000489+0000","last_active":"2024-04-24T23:16:18.230951+0000","last_peered":"2024-04-24T23:16:18.230951+0000","last_clean":"2024-04-24T23:16:18.230951+0000","last_became_active":"2024-04-24T23:15:37.000179+0000","last_became_peered":"2024-04-24T23:15:37.000179+0000","last_unstale":"2024-04-24T23:16:18.230951+0000","last_undegraded":"2024-04-24T23:16:18.230951+0000","last_fullsized":"2024-04-24T23:16:18.230951+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:06:37.125573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.c","version":"26'5","reported_seq":31,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:25.270158+0000","last_change":"2024-04-24T23:15:35.966020+0000","last_active":"2024-04-24T23:16:25.270158+0000","last_peered":"2024-04-24T23:16:25.270158+0000","last_clean":"2024-04-24T23:16:25.270158+0000","last_became_active":"2024-04-24T23:15:35.965758+0000","last_became_peered":"2024-04-24T23:15:35.965758+0000","last_unstale":"2024-04-24T23:16:25.270158+0000","last_undegraded":"2024-04-24T23:16:25.270158+0000","last_fullsized":"2024-04-24T23:16:25.270158+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:39:21.233461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.294138+0000","last_change":"2024-04-24T23:15:41.024489+0000","last_active":"2024-04-24T23:16:18.294138+0000","last_peered":"2024-04-24T23:16:18.294138+0000","last_clean":"2024-04-24T23:16:18.294138+0000","last_became_active":"2024-04-24T23:15:41.024041+0000","last_became_peered":"2024-04-24T23:15:41.024041+0000","last_unstale":"2024-04-24T23:16:18.294138+0000","last_undegraded":"2024-04-24T23:16:18.294138+0000","last_fullsized":"2024-04-24T23:16:18.294138+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:27:18.197003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7,4],"acting":[2,3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.229879+0000","last_change":"2024-04-24T23:15:36.979887+0000","last_active":"2024-04-24T23:16:18.229879+0000","last_peered":"2024-04-24T23:16:18.229879+0000","last_clean":"2024-04-24T23:16:18.229879+0000","last_became_active":"2024-04-24T23:15:36.979611+0000","last_became_peered":"2024-04-24T23:15:36.979611+0000","last_unstale":"2024-04-24T23:16:18.229879+0000","last_undegraded":"2024-04-24T23:16:18.229879+0000","last_fullsized":"2024-04-24T23:16:18.229879+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:07:18.261077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"26'3","reported_seq":29,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:25.265069+0000","last_change":"2024-04-24T23:15:35.965192+0000","last_active":"2024-04-24T23:16:25.265069+0000","last_peered":"2024-04-24T23:16:25.265069+0000","last_clean":"2024-04-24T23:16:25.265069+0000","last_became_active":"2024-04-24T23:15:35.964873+0000","last_became_peered":"2024-04-24T23:15:35.964873+0000","last_unstale":"2024-04-24T23:16:25.265069+0000","last_undegraded":"2024-04-24T23:16:25.265069+0000","last_fullsized":"2024-04-24T23:16:25.265069+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:14:27.884944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.291986+0000","last_change":"2024-04-24T23:15:41.004002+0000","last_active":"2024-04-24T23:16:18.291986+0000","last_peered":"2024-04-24T23:16:18.291986+0000","last_clean":"2024-04-24T23:16:18.291986+0000","last_became_active":"2024-04-24T23:15:41.003335+0000","last_became_peered":"2024-04-24T23:15:41.003335+0000","last_unstale":"2024-04-24T23:16:18.291986+0000","last_undegraded":"2024-04-24T23:16:18.291986+0000","last_fullsized":"2024-04-24T23:16:18.291986+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:49:36.206903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1,3],"acting":[5,4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.229375+0000","last_change":"2024-04-24T23:15:37.002854+0000","last_active":"2024-04-24T23:16:18.229375+0000","last_peered":"2024-04-24T23:16:18.229375+0000","last_clean":"2024-04-24T23:16:18.229375+0000","last_became_active":"2024-04-24T23:15:37.002583+0000","last_became_peered":"2024-04-24T23:15:37.002583+0000","last_unstale":"2024-04-24T23:16:18.229375+0000","last_undegraded":"2024-04-24T23:16:18.229375+0000","last_fullsized":"2024-04-24T23:16:18.229375+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:27:04.251547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"26'5","reported_seq":31,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:31.299455+0000","last_change":"2024-04-24T23:15:35.948752+0000","last_active":"2024-04-24T23:16:31.299455+0000","last_peered":"2024-04-24T23:16:31.299455+0000","last_clean":"2024-04-24T23:16:31.299455+0000","last_became_active":"2024-04-24T23:15:35.948438+0000","last_became_peered":"2024-04-24T23:15:35.948438+0000","last_unstale":"2024-04-24T23:16:31.299455+0000","last_undegraded":"2024-04-24T23:16:31.299455+0000","last_fullsized":"2024-04-24T23:16:31.299455+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:45:56.006278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.229925+0000","last_change":"2024-04-24T23:15:41.017524+0000","last_active":"2024-04-24T23:16:18.229925+0000","last_peered":"2024-04-24T23:16:18.229925+0000","last_clean":"2024-04-24T23:16:18.229925+0000","last_became_active":"2024-04-24T23:15:41.017092+0000","last_became_peered":"2024-04-24T23:15:41.017092+0000","last_unstale":"2024-04-24T23:16:18.229925+0000","last_undegraded":"2024-04-24T23:16:18.229925+0000","last_fullsized":"2024-04-24T23:16:18.229925+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:04:53.235038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.294797+0000","last_change":"2024-04-24T23:15:36.990441+0000","last_active":"2024-04-24T23:16:18.294797+0000","last_peered":"2024-04-24T23:16:18.294797+0000","last_clean":"2024-04-24T23:16:18.294797+0000","last_became_active":"2024-04-24T23:15:36.989863+0000","last_became_peered":"2024-04-24T23:15:36.989863+0000","last_unstale":"2024-04-24T23:16:18.294797+0000","last_undegraded":"2024-04-24T23:16:18.294797+0000","last_fullsized":"2024-04-24T23:16:18.294797+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:50:07.975347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.f","version":"26'2","reported_seq":28,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.232012+0000","last_change":"2024-04-24T23:15:35.969317+0000","last_active":"2024-04-24T23:16:18.232012+0000","last_peered":"2024-04-24T23:16:18.232012+0000","last_clean":"2024-04-24T23:16:18.232012+0000","last_became_active":"2024-04-24T23:15:35.968994+0000","last_became_peered":"2024-04-24T23:15:35.968994+0000","last_unstale":"2024-04-24T23:16:18.232012+0000","last_undegraded":"2024-04-24T23:16:18.232012+0000","last_fullsized":"2024-04-24T23:16:18.232012+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:51:57.100772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.296718+0000","last_change":"2024-04-24T23:15:41.010103+0000","last_active":"2024-04-24T23:16:18.296718+0000","last_peered":"2024-04-24T23:16:18.296718+0000","last_clean":"2024-04-24T23:16:18.296718+0000","last_became_active":"2024-04-24T23:15:41.009328+0000","last_became_peered":"2024-04-24T23:15:41.009328+0000","last_unstale":"2024-04-24T23:16:18.296718+0000","last_undegraded":"2024-04-24T23:16:18.296718+0000","last_fullsized":"2024-04-24T23:16:18.296718+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:46:49.579501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,2],"acting":[1,5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.232575+0000","last_change":"2024-04-24T23:15:36.996765+0000","last_active":"2024-04-24T23:16:18.232575+0000","last_peered":"2024-04-24T23:16:18.232575+0000","last_clean":"2024-04-24T23:16:18.232575+0000","last_became_active":"2024-04-24T23:15:36.995801+0000","last_became_peered":"2024-04-24T23:15:36.995801+0000","last_unstale":"2024-04-24T23:16:18.232575+0000","last_undegraded":"2024-04-24T23:16:18.232575+0000","last_fullsized":"2024-04-24T23:16:18.232575+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:25:19.421667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"26'17","reported_seq":43,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:25.266099+0000","last_change":"2024-04-24T23:15:35.944023+0000","last_active":"2024-04-24T23:16:25.266099+0000","last_peered":"2024-04-24T23:16:25.266099+0000","last_clean":"2024-04-24T23:16:25.266099+0000","last_became_active":"2024-04-24T23:15:35.943710+0000","last_became_peered":"2024-04-24T23:15:35.943710+0000","last_unstale":"2024-04-24T23:16:25.266099+0000","last_undegraded":"2024-04-24T23:16:25.266099+0000","last_fullsized":"2024-04-24T23:16:25.266099+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:46:48.090958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.292053+0000","last_change":"2024-04-24T23:15:41.013027+0000","last_active":"2024-04-24T23:16:18.292053+0000","last_peered":"2024-04-24T23:16:18.292053+0000","last_clean":"2024-04-24T23:16:18.292053+0000","last_became_active":"2024-04-24T23:15:41.012587+0000","last_became_peered":"2024-04-24T23:15:41.012587+0000","last_unstale":"2024-04-24T23:16:18.292053+0000","last_undegraded":"2024-04-24T23:16:18.292053+0000","last_fullsized":"2024-04-24T23:16:18.292053+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:27:04.276246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1,6],"acting":[5,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.295974+0000","last_change":"2024-04-24T23:15:36.979702+0000","last_active":"2024-04-24T23:16:18.295974+0000","last_peered":"2024-04-24T23:16:18.295974+0000","last_clean":"2024-04-24T23:16:18.295974+0000","last_became_active":"2024-04-24T23:15:36.979356+0000","last_became_peered":"2024-04-24T23:15:36.979356+0000","last_unstale":"2024-04-24T23:16:18.295974+0000","last_undegraded":"2024-04-24T23:16:18.295974+0000","last_fullsized":"2024-04-24T23:16:18.295974+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:14:24.309192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.11","version":"26'5","reported_seq":31,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:25.270027+0000","last_change":"2024-04-24T23:15:35.973026+0000","last_active":"2024-04-24T23:16:25.270027+0000","last_peered":"2024-04-24T23:16:25.270027+0000","last_clean":"2024-04-24T23:16:25.270027+0000","last_became_active":"2024-04-24T23:15:35.972655+0000","last_became_peered":"2024-04-24T23:15:35.972655+0000","last_unstale":"2024-04-24T23:16:25.270027+0000","last_undegraded":"2024-04-24T23:16:25.270027+0000","last_fullsized":"2024-04-24T23:16:25.270027+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:25:34.352908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.291924+0000","last_change":"2024-04-24T23:15:41.004221+0000","last_active":"2024-04-24T23:16:18.291924+0000","last_peered":"2024-04-24T23:16:18.291924+0000","last_clean":"2024-04-24T23:16:18.291924+0000","last_became_active":"2024-04-24T23:15:41.003790+0000","last_became_peered":"2024-04-24T23:15:41.003790+0000","last_unstale":"2024-04-24T23:16:18.291924+0000","last_undegraded":"2024-04-24T23:16:18.291924+0000","last_fullsized":"2024-04-24T23:16:18.291924+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:28:58.866024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6,4],"acting":[5,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.229856+0000","last_change":"2024-04-24T23:15:36.997389+0000","last_active":"2024-04-24T23:16:18.229856+0000","last_peered":"2024-04-24T23:16:18.229856+0000","last_clean":"2024-04-24T23:16:18.229856+0000","last_became_active":"2024-04-24T23:15:36.995923+0000","last_became_peered":"2024-04-24T23:15:36.995923+0000","last_unstale":"2024-04-24T23:16:18.229856+0000","last_undegraded":"2024-04-24T23:16:18.229856+0000","last_fullsized":"2024-04-24T23:16:18.229856+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:55:47.134320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"26'5","reported_seq":31,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:25.274296+0000","last_change":"2024-04-24T23:15:35.957715+0000","last_active":"2024-04-24T23:16:25.274296+0000","last_peered":"2024-04-24T23:16:25.274296+0000","last_clean":"2024-04-24T23:16:25.274296+0000","last_became_active":"2024-04-24T23:15:35.957088+0000","last_became_peered":"2024-04-24T23:15:35.957088+0000","last_unstale":"2024-04-24T23:16:25.274296+0000","last_undegraded":"2024-04-24T23:16:25.274296+0000","last_fullsized":"2024-04-24T23:16:25.274296+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:30:34.483669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.223881+0000","last_change":"2024-04-24T23:15:41.008165+0000","last_active":"2024-04-24T23:16:18.223881+0000","last_peered":"2024-04-24T23:16:18.223881+0000","last_clean":"2024-04-24T23:16:18.223881+0000","last_became_active":"2024-04-24T23:15:41.007555+0000","last_became_peered":"2024-04-24T23:15:41.007555+0000","last_unstale":"2024-04-24T23:16:18.223881+0000","last_undegraded":"2024-04-24T23:16:18.223881+0000","last_fullsized":"2024-04-24T23:16:18.223881+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:18:08.195747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0,7],"acting":[6,4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.224736+0000","last_change":"2024-04-24T23:15:36.991382+0000","last_active":"2024-04-24T23:16:18.224736+0000","last_peered":"2024-04-24T23:16:18.224736+0000","last_clean":"2024-04-24T23:16:18.224736+0000","last_became_active":"2024-04-24T23:15:36.991115+0000","last_became_peered":"2024-04-24T23:15:36.991115+0000","last_unstale":"2024-04-24T23:16:18.224736+0000","last_undegraded":"2024-04-24T23:16:18.224736+0000","last_fullsized":"2024-04-24T23:16:18.224736+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:21:51.863947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"26'9","reported_seq":35,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:31.304710+0000","last_change":"2024-04-24T23:15:35.964245+0000","last_active":"2024-04-24T23:16:31.304710+0000","last_peered":"2024-04-24T23:16:31.304710+0000","last_clean":"2024-04-24T23:16:31.304710+0000","last_became_active":"2024-04-24T23:15:35.963907+0000","last_became_peered":"2024-04-24T23:15:35.963907+0000","last_unstale":"2024-04-24T23:16:31.304710+0000","last_undegraded":"2024-04-24T23:16:31.304710+0000","last_fullsized":"2024-04-24T23:16:31.304710+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:15:00.913899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.230757+0000","last_change":"2024-04-24T23:15:41.021981+0000","last_active":"2024-04-24T23:16:18.230757+0000","last_peered":"2024-04-24T23:16:18.230757+0000","last_clean":"2024-04-24T23:16:18.230757+0000","last_became_active":"2024-04-24T23:15:41.019193+0000","last_became_peered":"2024-04-24T23:15:41.019193+0000","last_unstale":"2024-04-24T23:16:18.230757+0000","last_undegraded":"2024-04-24T23:16:18.230757+0000","last_fullsized":"2024-04-24T23:16:18.230757+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:04:39.828502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5,6],"acting":[3,0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.232496+0000","last_change":"2024-04-24T23:15:36.986833+0000","last_active":"2024-04-24T23:16:18.232496+0000","last_peered":"2024-04-24T23:16:18.232496+0000","last_clean":"2024-04-24T23:16:18.232496+0000","last_became_active":"2024-04-24T23:15:36.986484+0000","last_became_peered":"2024-04-24T23:15:36.986484+0000","last_unstale":"2024-04-24T23:16:18.232496+0000","last_undegraded":"2024-04-24T23:16:18.232496+0000","last_fullsized":"2024-04-24T23:16:18.232496+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:12:32.245267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.14","version":"26'16","reported_seq":42,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:41.653552+0000","last_change":"2024-04-24T23:15:35.957124+0000","last_active":"2024-04-24T23:16:41.653552+0000","last_peered":"2024-04-24T23:16:41.653552+0000","last_clean":"2024-04-24T23:16:41.653552+0000","last_became_active":"2024-04-24T23:15:35.951923+0000","last_became_peered":"2024-04-24T23:15:35.951923+0000","last_unstale":"2024-04-24T23:16:41.653552+0000","last_undegraded":"2024-04-24T23:16:41.653552+0000","last_fullsized":"2024-04-24T23:16:41.653552+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:44:16.820640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4100,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.223976+0000","last_change":"2024-04-24T23:15:41.015494+0000","last_active":"2024-04-24T23:16:18.223976+0000","last_peered":"2024-04-24T23:16:18.223976+0000","last_clean":"2024-04-24T23:16:18.223976+0000","last_became_active":"2024-04-24T23:15:41.015179+0000","last_became_peered":"2024-04-24T23:15:41.015179+0000","last_unstale":"2024-04-24T23:16:18.223976+0000","last_undegraded":"2024-04-24T23:16:18.223976+0000","last_fullsized":"2024-04-24T23:16:18.223976+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:18:32.208506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5,0],"acting":[4,3,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.230201+0000","last_change":"2024-04-24T23:15:36.999334+0000","last_active":"2024-04-24T23:16:18.230201+0000","last_peered":"2024-04-24T23:16:18.230201+0000","last_clean":"2024-04-24T23:16:18.230201+0000","last_became_active":"2024-04-24T23:15:36.995470+0000","last_became_peered":"2024-04-24T23:15:36.995470+0000","last_unstale":"2024-04-24T23:16:18.230201+0000","last_undegraded":"2024-04-24T23:16:18.230201+0000","last_fullsized":"2024-04-24T23:16:18.230201+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:19:57.973402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.15","version":"26'5","reported_seq":31,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.292831+0000","last_change":"2024-04-24T23:15:35.946041+0000","last_active":"2024-04-24T23:16:18.292831+0000","last_peered":"2024-04-24T23:16:18.292831+0000","last_clean":"2024-04-24T23:16:18.292831+0000","last_became_active":"2024-04-24T23:15:35.944805+0000","last_became_peered":"2024-04-24T23:15:35.944805+0000","last_unstale":"2024-04-24T23:16:18.292831+0000","last_undegraded":"2024-04-24T23:16:18.292831+0000","last_fullsized":"2024-04-24T23:16:18.292831+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:52:19.261306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.293637+0000","last_change":"2024-04-24T23:15:41.024392+0000","last_active":"2024-04-24T23:16:18.293637+0000","last_peered":"2024-04-24T23:16:18.293637+0000","last_clean":"2024-04-24T23:16:18.293637+0000","last_became_active":"2024-04-24T23:15:41.023951+0000","last_became_peered":"2024-04-24T23:15:41.023951+0000","last_unstale":"2024-04-24T23:16:18.293637+0000","last_undegraded":"2024-04-24T23:16:18.293637+0000","last_fullsized":"2024-04-24T23:16:18.293637+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:24:17.810117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1,7],"acting":[2,3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.292208+0000","last_change":"2024-04-24T23:15:36.992243+0000","last_active":"2024-04-24T23:16:18.292208+0000","last_peered":"2024-04-24T23:16:18.292208+0000","last_clean":"2024-04-24T23:16:18.292208+0000","last_became_active":"2024-04-24T23:15:36.992002+0000","last_became_peered":"2024-04-24T23:15:36.992002+0000","last_unstale":"2024-04-24T23:16:18.292208+0000","last_undegraded":"2024-04-24T23:16:18.292208+0000","last_fullsized":"2024-04-24T23:16:18.292208+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:52:15.911009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"26'4","reported_seq":30,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:31.301673+0000","last_change":"2024-04-24T23:15:35.967408+0000","last_active":"2024-04-24T23:16:31.301673+0000","last_peered":"2024-04-24T23:16:31.301673+0000","last_clean":"2024-04-24T23:16:31.301673+0000","last_became_active":"2024-04-24T23:15:35.966646+0000","last_became_peered":"2024-04-24T23:15:35.966646+0000","last_unstale":"2024-04-24T23:16:31.301673+0000","last_undegraded":"2024-04-24T23:16:31.301673+0000","last_fullsized":"2024-04-24T23:16:31.301673+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:07:07.931773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.291779+0000","last_change":"2024-04-24T23:15:41.021697+0000","last_active":"2024-04-24T23:16:18.291779+0000","last_peered":"2024-04-24T23:16:18.291779+0000","last_clean":"2024-04-24T23:16:18.291779+0000","last_became_active":"2024-04-24T23:15:41.021281+0000","last_became_peered":"2024-04-24T23:15:41.021281+0000","last_unstale":"2024-04-24T23:16:18.291779+0000","last_undegraded":"2024-04-24T23:16:18.291779+0000","last_fullsized":"2024-04-24T23:16:18.291779+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:59:37.306593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3,2],"acting":[7,1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.294698+0000","last_change":"2024-04-24T23:15:36.990988+0000","last_active":"2024-04-24T23:16:18.294698+0000","last_peered":"2024-04-24T23:16:18.294698+0000","last_clean":"2024-04-24T23:16:18.294698+0000","last_became_active":"2024-04-24T23:15:36.990711+0000","last_became_peered":"2024-04-24T23:15:36.990711+0000","last_unstale":"2024-04-24T23:16:18.294698+0000","last_undegraded":"2024-04-24T23:16:18.294698+0000","last_fullsized":"2024-04-24T23:16:18.294698+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:33:20.663776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.17","version":"26'4","reported_seq":30,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.297684+0000","last_change":"2024-04-24T23:15:35.957838+0000","last_active":"2024-04-24T23:16:18.297684+0000","last_peered":"2024-04-24T23:16:18.297684+0000","last_clean":"2024-04-24T23:16:18.297684+0000","last_became_active":"2024-04-24T23:15:35.956870+0000","last_became_peered":"2024-04-24T23:15:35.956870+0000","last_unstale":"2024-04-24T23:16:18.297684+0000","last_undegraded":"2024-04-24T23:16:18.297684+0000","last_fullsized":"2024-04-24T23:16:18.297684+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:56:03.889693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.223850+0000","last_change":"2024-04-24T23:15:41.006098+0000","last_active":"2024-04-24T23:16:18.223850+0000","last_peered":"2024-04-24T23:16:18.223850+0000","last_clean":"2024-04-24T23:16:18.223850+0000","last_became_active":"2024-04-24T23:15:41.005578+0000","last_became_peered":"2024-04-24T23:15:41.005578+0000","last_unstale":"2024-04-24T23:16:18.223850+0000","last_undegraded":"2024-04-24T23:16:18.223850+0000","last_fullsized":"2024-04-24T23:16:18.223850+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:21:12.802112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3,7],"acting":[4,0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.224194+0000","last_change":"2024-04-24T23:15:41.022122+0000","last_active":"2024-04-24T23:16:18.224194+0000","last_peered":"2024-04-24T23:16:18.224194+0000","last_clean":"2024-04-24T23:16:18.224194+0000","last_became_active":"2024-04-24T23:15:41.021698+0000","last_became_peered":"2024-04-24T23:15:41.021698+0000","last_unstale":"2024-04-24T23:16:18.224194+0000","last_undegraded":"2024-04-24T23:16:18.224194+0000","last_fullsized":"2024-04-24T23:16:18.224194+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:13:56.730050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5,3],"acting":[4,1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.223853+0000","last_change":"2024-04-24T23:15:36.979816+0000","last_active":"2024-04-24T23:16:18.223853+0000","last_peered":"2024-04-24T23:16:18.223853+0000","last_clean":"2024-04-24T23:16:18.223853+0000","last_became_active":"2024-04-24T23:15:36.979553+0000","last_became_peered":"2024-04-24T23:15:36.979553+0000","last_unstale":"2024-04-24T23:16:18.223853+0000","last_undegraded":"2024-04-24T23:16:18.223853+0000","last_fullsized":"2024-04-24T23:16:18.223853+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:55:47.134320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.18","version":"26'3","reported_seq":29,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:25.272717+0000","last_change":"2024-04-24T23:15:35.964744+0000","last_active":"2024-04-24T23:16:25.272717+0000","last_peered":"2024-04-24T23:16:25.272717+0000","last_clean":"2024-04-24T23:16:25.272717+0000","last_became_active":"2024-04-24T23:15:35.964434+0000","last_became_peered":"2024-04-24T23:15:35.964434+0000","last_unstale":"2024-04-24T23:16:25.272717+0000","last_undegraded":"2024-04-24T23:16:25.272717+0000","last_fullsized":"2024-04-24T23:16:25.272717+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:26:32.587197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.231561+0000","last_change":"2024-04-24T23:15:41.022032+0000","last_active":"2024-04-24T23:16:18.231561+0000","last_peered":"2024-04-24T23:16:18.231561+0000","last_clean":"2024-04-24T23:16:18.231561+0000","last_became_active":"2024-04-24T23:15:41.021616+0000","last_became_peered":"2024-04-24T23:15:41.021616+0000","last_unstale":"2024-04-24T23:16:18.231561+0000","last_undegraded":"2024-04-24T23:16:18.231561+0000","last_fullsized":"2024-04-24T23:16:18.231561+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:08:59.242392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5,1],"acting":[3,2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.291442+0000","last_change":"2024-04-24T23:15:36.967198+0000","last_active":"2024-04-24T23:16:18.291442+0000","last_peered":"2024-04-24T23:16:18.291442+0000","last_clean":"2024-04-24T23:16:18.291442+0000","last_became_active":"2024-04-24T23:15:36.966910+0000","last_became_peered":"2024-04-24T23:15:36.966910+0000","last_unstale":"2024-04-24T23:16:18.291442+0000","last_undegraded":"2024-04-24T23:16:18.291442+0000","last_fullsized":"2024-04-24T23:16:18.291442+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:51:19.788437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"26'3","reported_seq":29,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:19.219459+0000","last_change":"2024-04-24T23:15:35.967169+0000","last_active":"2024-04-24T23:16:19.219459+0000","last_peered":"2024-04-24T23:16:19.219459+0000","last_clean":"2024-04-24T23:16:19.219459+0000","last_became_active":"2024-04-24T23:15:35.966939+0000","last_became_peered":"2024-04-24T23:15:35.966939+0000","last_unstale":"2024-04-24T23:16:19.219459+0000","last_undegraded":"2024-04-24T23:16:19.219459+0000","last_fullsized":"2024-04-24T23:16:19.219459+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:38:55.460569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"26'7","reported_seq":33,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:25.275257+0000","last_change":"2024-04-24T23:15:35.971236+0000","last_active":"2024-04-24T23:16:25.275257+0000","last_peered":"2024-04-24T23:16:25.275257+0000","last_clean":"2024-04-24T23:16:25.275257+0000","last_became_active":"2024-04-24T23:15:35.970908+0000","last_became_peered":"2024-04-24T23:15:35.970908+0000","last_unstale":"2024-04-24T23:16:25.275257+0000","last_undegraded":"2024-04-24T23:16:25.275257+0000","last_fullsized":"2024-04-24T23:16:25.275257+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:14:15.548024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.223928+0000","last_change":"2024-04-24T23:15:36.988156+0000","last_active":"2024-04-24T23:16:18.223928+0000","last_peered":"2024-04-24T23:16:18.223928+0000","last_clean":"2024-04-24T23:16:18.223928+0000","last_became_active":"2024-04-24T23:15:36.987896+0000","last_became_peered":"2024-04-24T23:15:36.987896+0000","last_unstale":"2024-04-24T23:16:18.223928+0000","last_undegraded":"2024-04-24T23:16:18.223928+0000","last_fullsized":"2024-04-24T23:16:18.223928+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:22:06.869644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.224383+0000","last_change":"2024-04-24T23:15:40.997559+0000","last_active":"2024-04-24T23:16:18.224383+0000","last_peered":"2024-04-24T23:16:18.224383+0000","last_clean":"2024-04-24T23:16:18.224383+0000","last_became_active":"2024-04-24T23:15:40.997102+0000","last_became_peered":"2024-04-24T23:15:40.997102+0000","last_unstale":"2024-04-24T23:16:18.224383+0000","last_undegraded":"2024-04-24T23:16:18.224383+0000","last_fullsized":"2024-04-24T23:16:18.224383+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:25:02.046761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,7],"acting":[4,6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"24'5","reported_seq":31,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.292037+0000","last_change":"2024-04-24T23:15:35.944842+0000","last_active":"2024-04-24T23:16:18.292037+0000","last_peered":"2024-04-24T23:16:18.292037+0000","last_clean":"2024-04-24T23:16:18.292037+0000","last_became_active":"2024-04-24T23:15:35.944563+0000","last_became_peered":"2024-04-24T23:15:35.944563+0000","last_unstale":"2024-04-24T23:16:18.292037+0000","last_undegraded":"2024-04-24T23:16:18.292037+0000","last_fullsized":"2024-04-24T23:16:18.292037+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:40:58.933384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.296474+0000","last_change":"2024-04-24T23:15:36.981103+0000","last_active":"2024-04-24T23:16:18.296474+0000","last_peered":"2024-04-24T23:16:18.296474+0000","last_clean":"2024-04-24T23:16:18.296474+0000","last_became_active":"2024-04-24T23:15:36.979637+0000","last_became_peered":"2024-04-24T23:15:36.979637+0000","last_unstale":"2024-04-24T23:16:18.296474+0000","last_undegraded":"2024-04-24T23:16:18.296474+0000","last_fullsized":"2024-04-24T23:16:18.296474+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:09:39.750272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.292643+0000","last_change":"2024-04-24T23:15:41.017087+0000","last_active":"2024-04-24T23:16:18.292643+0000","last_peered":"2024-04-24T23:16:18.292643+0000","last_clean":"2024-04-24T23:16:18.292643+0000","last_became_active":"2024-04-24T23:15:41.016696+0000","last_became_peered":"2024-04-24T23:15:41.016696+0000","last_unstale":"2024-04-24T23:16:18.292643+0000","last_undegraded":"2024-04-24T23:16:18.292643+0000","last_fullsized":"2024-04-24T23:16:18.292643+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:05:57.823447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1,0],"acting":[7,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.293973+0000","last_change":"2024-04-24T23:15:36.970211+0000","last_active":"2024-04-24T23:16:18.293973+0000","last_peered":"2024-04-24T23:16:18.293973+0000","last_clean":"2024-04-24T23:16:18.293973+0000","last_became_active":"2024-04-24T23:15:36.969090+0000","last_became_peered":"2024-04-24T23:15:36.969090+0000","last_unstale":"2024-04-24T23:16:18.293973+0000","last_undegraded":"2024-04-24T23:16:18.293973+0000","last_fullsized":"2024-04-24T23:16:18.293973+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:16:00.231012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.291726+0000","last_change":"2024-04-24T23:15:36.982528+0000","last_active":"2024-04-24T23:16:18.291726+0000","last_peered":"2024-04-24T23:16:18.291726+0000","last_clean":"2024-04-24T23:16:18.291726+0000","last_became_active":"2024-04-24T23:15:36.982299+0000","last_became_peered":"2024-04-24T23:15:36.982299+0000","last_unstale":"2024-04-24T23:16:18.291726+0000","last_undegraded":"2024-04-24T23:16:18.291726+0000","last_fullsized":"2024-04-24T23:16:18.291726+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:02:17.732324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.233323+0000","last_change":"2024-04-24T23:15:36.989899+0000","last_active":"2024-04-24T23:16:18.233323+0000","last_peered":"2024-04-24T23:16:18.233323+0000","last_clean":"2024-04-24T23:16:18.233323+0000","last_became_active":"2024-04-24T23:15:36.989557+0000","last_became_peered":"2024-04-24T23:15:36.989557+0000","last_unstale":"2024-04-24T23:16:18.233323+0000","last_undegraded":"2024-04-24T23:16:18.233323+0000","last_fullsized":"2024-04-24T23:16:18.233323+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:15:13.513341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.291922+0000","last_change":"2024-04-24T23:15:36.969628+0000","last_active":"2024-04-24T23:16:18.291922+0000","last_peered":"2024-04-24T23:16:18.291922+0000","last_clean":"2024-04-24T23:16:18.291922+0000","last_became_active":"2024-04-24T23:15:36.969327+0000","last_became_peered":"2024-04-24T23:15:36.969327+0000","last_unstale":"2024-04-24T23:16:18.291922+0000","last_undegraded":"2024-04-24T23:16:18.291922+0000","last_fullsized":"2024-04-24T23:16:18.291922+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:08:47.021999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.224974+0000","last_change":"2024-04-24T23:15:36.986343+0000","last_active":"2024-04-24T23:16:18.224974+0000","last_peered":"2024-04-24T23:16:18.224974+0000","last_clean":"2024-04-24T23:16:18.224974+0000","last_became_active":"2024-04-24T23:15:36.986026+0000","last_became_peered":"2024-04-24T23:15:36.986026+0000","last_unstale":"2024-04-24T23:16:18.224974+0000","last_undegraded":"2024-04-24T23:16:18.224974+0000","last_fullsized":"2024-04-24T23:16:18.224974+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:10:00.347826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.224857+0000","last_change":"2024-04-24T23:15:36.968538+0000","last_active":"2024-04-24T23:16:18.224857+0000","last_peered":"2024-04-24T23:16:18.224857+0000","last_clean":"2024-04-24T23:16:18.224857+0000","last_became_active":"2024-04-24T23:15:36.968263+0000","last_became_peered":"2024-04-24T23:15:36.968263+0000","last_unstale":"2024-04-24T23:16:18.224857+0000","last_undegraded":"2024-04-24T23:16:18.224857+0000","last_fullsized":"2024-04-24T23:16:18.224857+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:07:43.731738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.294600+0000","last_change":"2024-04-24T23:15:36.991008+0000","last_active":"2024-04-24T23:16:18.294600+0000","last_peered":"2024-04-24T23:16:18.294600+0000","last_clean":"2024-04-24T23:16:18.294600+0000","last_became_active":"2024-04-24T23:15:36.990764+0000","last_became_peered":"2024-04-24T23:15:36.990764+0000","last_unstale":"2024-04-24T23:16:18.294600+0000","last_undegraded":"2024-04-24T23:16:18.294600+0000","last_fullsized":"2024-04-24T23:16:18.294600+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:50:21.865233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.291911+0000","last_change":"2024-04-24T23:15:36.993550+0000","last_active":"2024-04-24T23:16:18.291911+0000","last_peered":"2024-04-24T23:16:18.291911+0000","last_clean":"2024-04-24T23:16:18.291911+0000","last_became_active":"2024-04-24T23:15:36.993310+0000","last_became_peered":"2024-04-24T23:15:36.993310+0000","last_unstale":"2024-04-24T23:16:18.291911+0000","last_undegraded":"2024-04-24T23:16:18.291911+0000","last_fullsized":"2024-04-24T23:16:18.291911+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:21:20.617940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.230900+0000","last_change":"2024-04-24T23:15:36.987810+0000","last_active":"2024-04-24T23:16:18.230900+0000","last_peered":"2024-04-24T23:16:18.230900+0000","last_clean":"2024-04-24T23:16:18.230900+0000","last_became_active":"2024-04-24T23:15:36.987540+0000","last_became_peered":"2024-04-24T23:15:36.987540+0000","last_unstale":"2024-04-24T23:16:18.230900+0000","last_undegraded":"2024-04-24T23:16:18.230900+0000","last_fullsized":"2024-04-24T23:16:18.230900+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:49:56.176021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.291530+0000","last_change":"2024-04-24T23:15:36.981187+0000","last_active":"2024-04-24T23:16:18.291530+0000","last_peered":"2024-04-24T23:16:18.291530+0000","last_clean":"2024-04-24T23:16:18.291530+0000","last_became_active":"2024-04-24T23:15:36.980838+0000","last_became_peered":"2024-04-24T23:15:36.980838+0000","last_unstale":"2024-04-24T23:16:18.291530+0000","last_undegraded":"2024-04-24T23:16:18.291530+0000","last_fullsized":"2024-04-24T23:16:18.291530+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:57:41.178367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.296378+0000","last_change":"2024-04-24T23:15:36.999238+0000","last_active":"2024-04-24T23:16:18.296378+0000","last_peered":"2024-04-24T23:16:18.296378+0000","last_clean":"2024-04-24T23:16:18.296378+0000","last_became_active":"2024-04-24T23:15:36.996245+0000","last_became_peered":"2024-04-24T23:15:36.996245+0000","last_unstale":"2024-04-24T23:16:18.296378+0000","last_undegraded":"2024-04-24T23:16:18.296378+0000","last_fullsized":"2024-04-24T23:16:18.296378+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:21:20.443640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.223856+0000","last_change":"2024-04-24T23:15:36.981219+0000","last_active":"2024-04-24T23:16:18.223856+0000","last_peered":"2024-04-24T23:16:18.223856+0000","last_clean":"2024-04-24T23:16:18.223856+0000","last_became_active":"2024-04-24T23:15:36.980911+0000","last_became_peered":"2024-04-24T23:15:36.980911+0000","last_unstale":"2024-04-24T23:16:18.223856+0000","last_undegraded":"2024-04-24T23:16:18.223856+0000","last_fullsized":"2024-04-24T23:16:18.223856+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:19:57.973402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.229789+0000","last_change":"2024-04-24T23:15:36.991418+0000","last_active":"2024-04-24T23:16:18.229789+0000","last_peered":"2024-04-24T23:16:18.229789+0000","last_clean":"2024-04-24T23:16:18.229789+0000","last_became_active":"2024-04-24T23:15:36.989792+0000","last_became_peered":"2024-04-24T23:15:36.989792+0000","last_unstale":"2024-04-24T23:16:18.229789+0000","last_undegraded":"2024-04-24T23:16:18.229789+0000","last_fullsized":"2024-04-24T23:16:18.229789+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:02:53.601311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.231144+0000","last_change":"2024-04-24T23:15:36.987810+0000","last_active":"2024-04-24T23:16:18.231144+0000","last_peered":"2024-04-24T23:16:18.231144+0000","last_clean":"2024-04-24T23:16:18.231144+0000","last_became_active":"2024-04-24T23:15:36.987494+0000","last_became_peered":"2024-04-24T23:15:36.987494+0000","last_unstale":"2024-04-24T23:16:18.231144+0000","last_undegraded":"2024-04-24T23:16:18.231144+0000","last_fullsized":"2024-04-24T23:16:18.231144+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:43:21.141172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.292884+0000","last_change":"2024-04-24T23:15:36.986576+0000","last_active":"2024-04-24T23:16:18.292884+0000","last_peered":"2024-04-24T23:16:18.292884+0000","last_clean":"2024-04-24T23:16:18.292884+0000","last_became_active":"2024-04-24T23:15:36.986272+0000","last_became_peered":"2024-04-24T23:15:36.986272+0000","last_unstale":"2024-04-24T23:16:18.292884+0000","last_undegraded":"2024-04-24T23:16:18.292884+0000","last_fullsized":"2024-04-24T23:16:18.292884+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:55:08.894945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":29,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.294151+0000","last_change":"2024-04-24T23:16:18.294151+0000","last_active":"2024-04-24T23:16:18.294151+0000","last_peered":"2024-04-24T23:16:18.294151+0000","last_clean":"2024-04-24T23:16:18.294151+0000","last_became_active":"2024-04-24T23:16:18.293579+0000","last_became_peered":"2024-04-24T23:16:18.293579+0000","last_unstale":"2024-04-24T23:16:18.294151+0000","last_undegraded":"2024-04-24T23:16:18.294151+0000","last_fullsized":"2024-04-24T23:16:18.294151+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:22:20.603555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.230543+0000","last_change":"2024-04-24T23:15:36.996521+0000","last_active":"2024-04-24T23:16:18.230543+0000","last_peered":"2024-04-24T23:16:18.230543+0000","last_clean":"2024-04-24T23:16:18.230543+0000","last_became_active":"2024-04-24T23:15:36.995507+0000","last_became_peered":"2024-04-24T23:15:36.995507+0000","last_unstale":"2024-04-24T23:16:18.230543+0000","last_undegraded":"2024-04-24T23:16:18.230543+0000","last_fullsized":"2024-04-24T23:16:18.230543+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:58:50.153471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.291317+0000","last_change":"2024-04-24T23:15:36.984994+0000","last_active":"2024-04-24T23:16:18.291317+0000","last_peered":"2024-04-24T23:16:18.291317+0000","last_clean":"2024-04-24T23:16:18.291317+0000","last_became_active":"2024-04-24T23:15:36.984741+0000","last_became_peered":"2024-04-24T23:15:36.984741+0000","last_unstale":"2024-04-24T23:16:18.291317+0000","last_undegraded":"2024-04-24T23:16:18.291317+0000","last_fullsized":"2024-04-24T23:16:18.291317+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:45:23.429836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.224730+0000","last_change":"2024-04-24T23:15:36.966165+0000","last_active":"2024-04-24T23:16:18.224730+0000","last_peered":"2024-04-24T23:16:18.224730+0000","last_clean":"2024-04-24T23:16:18.224730+0000","last_became_active":"2024-04-24T23:15:36.965877+0000","last_became_peered":"2024-04-24T23:15:36.965877+0000","last_unstale":"2024-04-24T23:16:18.224730+0000","last_undegraded":"2024-04-24T23:16:18.224730+0000","last_fullsized":"2024-04-24T23:16:18.224730+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:30:12.390978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.224249+0000","last_change":"2024-04-24T23:15:36.990667+0000","last_active":"2024-04-24T23:16:18.224249+0000","last_peered":"2024-04-24T23:16:18.224249+0000","last_clean":"2024-04-24T23:16:18.224249+0000","last_became_active":"2024-04-24T23:15:36.990383+0000","last_became_peered":"2024-04-24T23:15:36.990383+0000","last_unstale":"2024-04-24T23:16:18.224249+0000","last_undegraded":"2024-04-24T23:16:18.224249+0000","last_fullsized":"2024-04-24T23:16:18.224249+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:36:43.850046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.229311+0000","last_change":"2024-04-24T23:15:36.991686+0000","last_active":"2024-04-24T23:16:18.229311+0000","last_peered":"2024-04-24T23:16:18.229311+0000","last_clean":"2024-04-24T23:16:18.229311+0000","last_became_active":"2024-04-24T23:15:36.990155+0000","last_became_peered":"2024-04-24T23:15:36.990155+0000","last_unstale":"2024-04-24T23:16:18.229311+0000","last_undegraded":"2024-04-24T23:16:18.229311+0000","last_fullsized":"2024-04-24T23:16:18.229311+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:00:52.424019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.232389+0000","last_change":"2024-04-24T23:15:36.992958+0000","last_active":"2024-04-24T23:16:18.232389+0000","last_peered":"2024-04-24T23:16:18.232389+0000","last_clean":"2024-04-24T23:16:18.232389+0000","last_became_active":"2024-04-24T23:15:36.992608+0000","last_became_peered":"2024-04-24T23:15:36.992608+0000","last_unstale":"2024-04-24T23:16:18.232389+0000","last_undegraded":"2024-04-24T23:16:18.232389+0000","last_fullsized":"2024-04-24T23:16:18.232389+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:15:30.416763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.232332+0000","last_change":"2024-04-24T23:15:36.992943+0000","last_active":"2024-04-24T23:16:18.232332+0000","last_peered":"2024-04-24T23:16:18.232332+0000","last_clean":"2024-04-24T23:16:18.232332+0000","last_became_active":"2024-04-24T23:15:36.992567+0000","last_became_peered":"2024-04-24T23:15:36.992567+0000","last_unstale":"2024-04-24T23:16:18.232332+0000","last_undegraded":"2024-04-24T23:16:18.232332+0000","last_fullsized":"2024-04-24T23:16:18.232332+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:50:43.209232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.297503+0000","last_change":"2024-04-24T23:15:36.999697+0000","last_active":"2024-04-24T23:16:18.297503+0000","last_peered":"2024-04-24T23:16:18.297503+0000","last_clean":"2024-04-24T23:16:18.297503+0000","last_became_active":"2024-04-24T23:15:36.999413+0000","last_became_peered":"2024-04-24T23:15:36.999413+0000","last_unstale":"2024-04-24T23:16:18.297503+0000","last_undegraded":"2024-04-24T23:16:18.297503+0000","last_fullsized":"2024-04-24T23:16:18.297503+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:25:42.856471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.291309+0000","last_change":"2024-04-24T23:15:36.986742+0000","last_active":"2024-04-24T23:16:18.291309+0000","last_peered":"2024-04-24T23:16:18.291309+0000","last_clean":"2024-04-24T23:16:18.291309+0000","last_became_active":"2024-04-24T23:15:36.986013+0000","last_became_peered":"2024-04-24T23:15:36.986013+0000","last_unstale":"2024-04-24T23:16:18.291309+0000","last_undegraded":"2024-04-24T23:16:18.291309+0000","last_fullsized":"2024-04-24T23:16:18.291309+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:16:15.203151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.234253+0000","last_change":"2024-04-24T23:15:36.984336+0000","last_active":"2024-04-24T23:16:18.234253+0000","last_peered":"2024-04-24T23:16:18.234253+0000","last_clean":"2024-04-24T23:16:18.234253+0000","last_became_active":"2024-04-24T23:15:36.983966+0000","last_became_peered":"2024-04-24T23:15:36.983966+0000","last_unstale":"2024-04-24T23:16:18.234253+0000","last_undegraded":"2024-04-24T23:16:18.234253+0000","last_fullsized":"2024-04-24T23:16:18.234253+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:54:16.319608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.294392+0000","last_change":"2024-04-24T23:15:36.979079+0000","last_active":"2024-04-24T23:16:18.294392+0000","last_peered":"2024-04-24T23:16:18.294392+0000","last_clean":"2024-04-24T23:16:18.294392+0000","last_became_active":"2024-04-24T23:15:36.978706+0000","last_became_peered":"2024-04-24T23:15:36.978706+0000","last_unstale":"2024-04-24T23:16:18.294392+0000","last_undegraded":"2024-04-24T23:16:18.294392+0000","last_fullsized":"2024-04-24T23:16:18.294392+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:53:39.888150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.229376+0000","last_change":"2024-04-24T23:15:36.986008+0000","last_active":"2024-04-24T23:16:18.229376+0000","last_peered":"2024-04-24T23:16:18.229376+0000","last_clean":"2024-04-24T23:16:18.229376+0000","last_became_active":"2024-04-24T23:15:36.985552+0000","last_became_peered":"2024-04-24T23:15:36.985552+0000","last_unstale":"2024-04-24T23:16:18.229376+0000","last_undegraded":"2024-04-24T23:16:18.229376+0000","last_fullsized":"2024-04-24T23:16:18.229376+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:36:50.827503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.293789+0000","last_change":"2024-04-24T23:15:36.978673+0000","last_active":"2024-04-24T23:16:18.293789+0000","last_peered":"2024-04-24T23:16:18.293789+0000","last_clean":"2024-04-24T23:16:18.293789+0000","last_became_active":"2024-04-24T23:15:36.972744+0000","last_became_peered":"2024-04-24T23:15:36.972744+0000","last_unstale":"2024-04-24T23:16:18.293789+0000","last_undegraded":"2024-04-24T23:16:18.293789+0000","last_fullsized":"2024-04-24T23:16:18.293789+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:28:57.028825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.291690+0000","last_change":"2024-04-24T23:15:36.970238+0000","last_active":"2024-04-24T23:16:18.291690+0000","last_peered":"2024-04-24T23:16:18.291690+0000","last_clean":"2024-04-24T23:16:18.291690+0000","last_became_active":"2024-04-24T23:15:36.969751+0000","last_became_peered":"2024-04-24T23:15:36.969751+0000","last_unstale":"2024-04-24T23:16:18.291690+0000","last_undegraded":"2024-04-24T23:16:18.291690+0000","last_fullsized":"2024-04-24T23:16:18.291690+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:19:56.456801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.291790+0000","last_change":"2024-04-24T23:15:36.968825+0000","last_active":"2024-04-24T23:16:18.291790+0000","last_peered":"2024-04-24T23:16:18.291790+0000","last_clean":"2024-04-24T23:16:18.291790+0000","last_became_active":"2024-04-24T23:15:36.968543+0000","last_became_peered":"2024-04-24T23:15:36.968543+0000","last_unstale":"2024-04-24T23:16:18.291790+0000","last_undegraded":"2024-04-24T23:16:18.291790+0000","last_fullsized":"2024-04-24T23:16:18.291790+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:36:48.795062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.229059+0000","last_change":"2024-04-24T23:15:36.992800+0000","last_active":"2024-04-24T23:16:18.229059+0000","last_peered":"2024-04-24T23:16:18.229059+0000","last_clean":"2024-04-24T23:16:18.229059+0000","last_became_active":"2024-04-24T23:15:36.992476+0000","last_became_peered":"2024-04-24T23:15:36.992476+0000","last_unstale":"2024-04-24T23:16:18.229059+0000","last_undegraded":"2024-04-24T23:16:18.229059+0000","last_fullsized":"2024-04-24T23:16:18.229059+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:01:21.103559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":10522,"num_objects":98,"num_object_clones":0,"num_object_copies":196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":211,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":21044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":185,"ondisk_log_size":185,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574868,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27324,"kb_used_data":308,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344516,"statfs":{"total":96636764160,"available":96608784384,"internally_reserved":0,"allocated":315392,"data_stored":114972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574868,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":292,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":299008,"data_stored":116298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574868,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27892,"kb_used_data":876,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343948,"statfs":{"total":96636764160,"available":96608202752,"internally_reserved":0,"allocated":897024,"data_stored":705112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574869,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27344,"kb_used_data":328,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344496,"statfs":{"total":96636764160,"available":96608763904,"internally_reserved":0,"allocated":335872,"data_stored":119856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574869,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27332,"kb_used_data":316,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344508,"statfs":{"total":96636764160,"available":96608776192,"internally_reserved":0,"allocated":323584,"data_stored":116640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574868,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":292,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":299008,"data_stored":118086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574867,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27888,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343952,"statfs":{"total":96636764160,"available":96608206848,"internally_reserved":0,"allocated":892928,"data_stored":705700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574868,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27324,"kb_used_data":308,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344516,"statfs":{"total":96636764160,"available":96608784384,"internally_reserved":0,"allocated":315392,"data_stored":116460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2542,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":5938,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1054,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T23:16:48.166 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T23:16:48.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:48.419+0000 7f6df7200640 1 Processor -- start 2024-04-24T23:16:48.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:48.419+0000 7f6df7200640 1 -- start start 2024-04-24T23:16:48.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:48.421+0000 7f6df7200640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6df00cd390 0x7f6df00cacf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:48.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:48.421+0000 7f6df7200640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f6df01548a0 0x7f6df0154cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:48.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:48.421+0000 7f6df7200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6df006be70 con 0x7f6df00ccfb0 2024-04-24T23:16:48.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:48.421+0000 7f6df7200640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f6df006ccd0 con 0x7f6df01548a0 2024-04-24T23:16:48.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:48.421+0000 7f6df7200640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f6df00cb230 con 0x7f6df00cd390 2024-04-24T23:16:48.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:48.421+0000 7f6df5400640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6df00cd390 0x7f6df00cacf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:48.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:48.421+0000 7f6df6800640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f6df01548a0 0x7f6df0154cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:48.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:48.421+0000 7f6df5400640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6df00cd390 0x7f6df00cacf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.6:59774/0 (socket says 172.21.15.6:59774) 2024-04-24T23:16:48.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:48.421+0000 7f6df5400640 1 -- 172.21.15.6:0/3516055005 learned_addr learned my addr 172.21.15.6:0/3516055005 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:16:48.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:48.422+0000 7f6df5400640 1 -- 172.21.15.6:0/3516055005 >> v2:172.21.15.190:3300/0 conn(0x7f6df01548a0 msgr2=0x7f6df0154cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:16:48.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:48.422+0000 7f6df5400640 1 --2- 172.21.15.6:0/3516055005 >> v2:172.21.15.190:3300/0 conn(0x7f6df01548a0 0x7f6df0154cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:48.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:48.422+0000 7f6df5400640 1 -- 172.21.15.6:0/3516055005 >> v1:172.21.15.6:6789/0 conn(0x7f6df00ccfb0 legacy=0x7f6df015b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:48.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:48.422+0000 7f6df5400640 1 -- 172.21.15.6:0/3516055005 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6df006c150 con 0x7f6df00cd390 2024-04-24T23:16:48.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:48.423+0000 7f6df5400640 1 --2- 172.21.15.6:0/3516055005 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6df00cd390 0x7f6df00cacf0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f6de0009870 tx=0x7f6de002a300 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5101e41ae2471a7f server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:48.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:48.423+0000 7f6df4a00640 1 -- 172.21.15.6:0/3516055005 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6de0037070 con 0x7f6df00cd390 2024-04-24T23:16:48.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:48.423+0000 7f6df4a00640 1 -- 172.21.15.6:0/3516055005 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6de002ae50 con 0x7f6df00cd390 2024-04-24T23:16:48.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:48.423+0000 7f6df4a00640 1 -- 172.21.15.6:0/3516055005 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6de0032860 con 0x7f6df00cd390 2024-04-24T23:16:48.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:48.424+0000 7f6df7200640 1 -- 172.21.15.6:0/3516055005 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6df00cd390 msgr2=0x7f6df00cacf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:48.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:48.424+0000 7f6df7200640 1 --2- 172.21.15.6:0/3516055005 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6df00cd390 0x7f6df00cacf0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f6de0009870 tx=0x7f6de002a300 comp rx=0 tx=0).stop 2024-04-24T23:16:48.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:48.424+0000 7f6df7200640 1 -- 172.21.15.6:0/3516055005 shutdown_connections 2024-04-24T23:16:48.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:48.424+0000 7f6df7200640 1 --2- 172.21.15.6:0/3516055005 >> v2:172.21.15.190:3300/0 conn(0x7f6df01548a0 0x7f6df0154cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:48.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:48.424+0000 7f6df7200640 1 --2- 172.21.15.6:0/3516055005 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6df00cd390 0x7f6df00cacf0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:48.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:48.424+0000 7f6df7200640 1 -- 172.21.15.6:0/3516055005 >> 172.21.15.6:0/3516055005 conn(0x7f6df006d9f0 msgr2=0x7f6df0066ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:48.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:48.424+0000 7f6df7200640 1 -- 172.21.15.6:0/3516055005 shutdown_connections 2024-04-24T23:16:48.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:48.424+0000 7f6df7200640 1 -- 172.21.15.6:0/3516055005 wait complete. 2024-04-24T23:16:48.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:48.426+0000 7f6df7200640 1 Processor -- start 2024-04-24T23:16:48.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:48.426+0000 7f6df7200640 1 -- start start 2024-04-24T23:16:48.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:48.427+0000 7f6df7200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6df00ccfb0 0x7f6df00c76b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:48.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:48.427+0000 7f6df7200640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6df00cd390 0x7f6df00c9ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:48.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:48.427+0000 7f6df7200640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6df01548a0 0x7f6df00b4fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:48.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:48.427+0000 7f6df7200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f6df006c9f0 con 0x7f6df00ccfb0 2024-04-24T23:16:48.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:48.427+0000 7f6df7200640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f6df006c710 con 0x7f6df00cd390 2024-04-24T23:16:48.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:48.427+0000 7f6df7200640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f6df0067bb0 con 0x7f6df01548a0 2024-04-24T23:16:48.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:48.427+0000 7f6df5e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6df00ccfb0 0x7f6df00c76b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:48.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:48.427+0000 7f6df6800640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6df01548a0 0x7f6df00b4fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:48.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:48.427+0000 7f6df6800640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6df01548a0 0x7f6df00b4fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.6:59776/0 (socket says 172.21.15.6:59776) 2024-04-24T23:16:48.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:48.427+0000 7f6df6800640 1 -- 172.21.15.6:0/450700970 learned_addr learned my addr 172.21.15.6:0/450700970 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:16:48.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:48.427+0000 7f6df5400640 1 --2- 172.21.15.6:0/450700970 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6df00cd390 0x7f6df00c9ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:48.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:48.428+0000 7f6df6800640 1 -- 172.21.15.6:0/450700970 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6df00cd390 msgr2=0x7f6df00c9ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:48.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:48.428+0000 7f6df6800640 1 --2- 172.21.15.6:0/450700970 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6df00cd390 0x7f6df00c9ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:48.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:48.428+0000 7f6df6800640 1 -- 172.21.15.6:0/450700970 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6df00ccfb0 msgr2=0x7f6df00c76b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:48.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:48.428+0000 7f6df6800640 1 --2- 172.21.15.6:0/450700970 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6df00ccfb0 0x7f6df00c76b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:48.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:48.428+0000 7f6df6800640 1 -- 172.21.15.6:0/450700970 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6df015bbf0 con 0x7f6df01548a0 2024-04-24T23:16:48.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:48.428+0000 7f6df6800640 1 --2- 172.21.15.6:0/450700970 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6df01548a0 0x7f6df00b4fe0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f6dec00e7f0 tx=0x7f6dec00ebf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:48.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:48.428+0000 7f6de6000640 1 -- 172.21.15.6:0/450700970 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dec017070 con 0x7f6df01548a0 2024-04-24T23:16:48.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:48.428+0000 7f6de6000640 1 -- 172.21.15.6:0/450700970 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6dec00ee40 con 0x7f6df01548a0 2024-04-24T23:16:48.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:48.428+0000 7f6de6000640 1 -- 172.21.15.6:0/450700970 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dec00cb80 con 0x7f6df01548a0 2024-04-24T23:16:48.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:48.428+0000 7f6df7200640 1 -- 172.21.15.6:0/450700970 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6df00c7d40 con 0x7f6df01548a0 2024-04-24T23:16:48.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:48.428+0000 7f6df7200640 1 -- 172.21.15.6:0/450700970 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f6df01f4260 con 0x7f6df01548a0 2024-04-24T23:16:48.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:48.430+0000 7f6df7200640 1 -- 172.21.15.6:0/450700970 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6df0075b30 con 0x7f6df01548a0 2024-04-24T23:16:48.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:48.431+0000 7f6de6000640 1 -- 172.21.15.6:0/450700970 <== mon.2 v2:172.21.15.190:3301/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f6dec01a070 con 0x7f6df01548a0 2024-04-24T23:16:48.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:48.431+0000 7f6de6000640 1 --2- 172.21.15.6:0/450700970 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f6dd406be30 0x7f6dd406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:48.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:48.432+0000 7f6de6000640 1 -- 172.21.15.6:0/450700970 <== mon.2 v2:172.21.15.190:3301/0 5 ==== osd_map(26..26 src has 1..26) ==== 6590+0+0 (secure 0 0 0) 0x7f6dec08b3d0 con 0x7f6df01548a0 2024-04-24T23:16:48.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:48.432+0000 7f6df5e00640 1 --2- 172.21.15.6:0/450700970 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f6dd406be30 0x7f6dd406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:48.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:48.432+0000 7f6df5e00640 1 --2- 172.21.15.6:0/450700970 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f6dd406be30 0x7f6dd406e2f0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f6de8001f10 tx=0x7f6de8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:48.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:48.436+0000 7f6de6000640 1 -- 172.21.15.6:0/450700970 <== mon.2 v2:172.21.15.190:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f6dec059320 con 0x7f6df01548a0 2024-04-24T23:16:48.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:48.634+0000 7f6df7200640 1 -- 172.21.15.6:0/450700970 --> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f6df01f3d30 con 0x7f6dd406be30 2024-04-24T23:16:48.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:48.644+0000 7f6de6000640 1 -- 172.21.15.6:0/450700970 <== mgr.4105 v2:172.21.15.190:6832/806769663 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+332657 (secure 0 0 0) 0x7f6df01f3d30 con 0x7f6dd406be30 2024-04-24T23:16:48.651 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:16:48.653 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-04-24T23:16:48.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:48.652+0000 7f6df7200640 1 -- 172.21.15.6:0/450700970 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f6dd406be30 msgr2=0x7f6dd406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:48.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:48.652+0000 7f6df7200640 1 --2- 172.21.15.6:0/450700970 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f6dd406be30 0x7f6dd406e2f0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f6de8001f10 tx=0x7f6de8009040 comp rx=0 tx=0).stop 2024-04-24T23:16:48.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:48.652+0000 7f6df7200640 1 -- 172.21.15.6:0/450700970 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6df01548a0 msgr2=0x7f6df00b4fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:48.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:48.652+0000 7f6df7200640 1 --2- 172.21.15.6:0/450700970 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6df01548a0 0x7f6df00b4fe0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f6dec00e7f0 tx=0x7f6dec00ebf0 comp rx=0 tx=0).stop 2024-04-24T23:16:48.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:48.652+0000 7f6df7200640 1 -- 172.21.15.6:0/450700970 shutdown_connections 2024-04-24T23:16:48.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:48.652+0000 7f6df7200640 1 --2- 172.21.15.6:0/450700970 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f6dd406be30 0x7f6dd406e2f0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:48.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:48.652+0000 7f6df7200640 1 --2- 172.21.15.6:0/450700970 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6df01548a0 0x7f6df00b4fe0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:48.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:48.652+0000 7f6df7200640 1 --2- 172.21.15.6:0/450700970 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6df00cd390 0x7f6df00c9ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:48.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:48.652+0000 7f6df7200640 1 --2- 172.21.15.6:0/450700970 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6df00ccfb0 0x7f6df00c76b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:48.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:48.652+0000 7f6df7200640 1 -- 172.21.15.6:0/450700970 >> 172.21.15.6:0/450700970 conn(0x7f6df006d9f0 msgr2=0x7f6df0150b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:48.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:48.652+0000 7f6df7200640 1 -- 172.21.15.6:0/450700970 shutdown_connections 2024-04-24T23:16:48.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:48.652+0000 7f6df7200640 1 -- 172.21.15.6:0/450700970 wait complete. 2024-04-24T23:16:48.675 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":60,"stamp":"2024-04-24T23:16:46.698415+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":600890,"num_objects":100,"num_object_clones":0,"num_object_copies":200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":370,"num_write_kb":2485,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":302,"ondisk_log_size":302,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219720,"kb_used_data":3592,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755000,"statfs":{"total":773094113280,"available":772869120000,"internally_reserved":0,"allocated":3678208,"data_stored":2113124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1664,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":43,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003284"},"pg_stats":[{"pgid":"2.1f","version":"26'13","reported_seq":39,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:31.303006+0000","last_change":"2024-04-24T23:15:35.964646+0000","last_active":"2024-04-24T23:16:31.303006+0000","last_peered":"2024-04-24T23:16:31.303006+0000","last_clean":"2024-04-24T23:16:31.303006+0000","last_became_active":"2024-04-24T23:15:35.964357+0000","last_became_peered":"2024-04-24T23:15:35.964357+0000","last_unstale":"2024-04-24T23:16:31.303006+0000","last_undegraded":"2024-04-24T23:16:31.303006+0000","last_fullsized":"2024-04-24T23:16:31.303006+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:01:17.912993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.292135+0000","last_change":"2024-04-24T23:15:36.989772+0000","last_active":"2024-04-24T23:16:18.292135+0000","last_peered":"2024-04-24T23:16:18.292135+0000","last_clean":"2024-04-24T23:16:18.292135+0000","last_became_active":"2024-04-24T23:15:36.989543+0000","last_became_peered":"2024-04-24T23:15:36.989543+0000","last_unstale":"2024-04-24T23:16:18.292135+0000","last_undegraded":"2024-04-24T23:16:18.292135+0000","last_fullsized":"2024-04-24T23:16:18.292135+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:37:16.737354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.303989+0000","last_change":"2024-04-24T23:16:18.300096+0000","last_active":"2024-04-24T23:16:18.303989+0000","last_peered":"2024-04-24T23:16:18.303989+0000","last_clean":"2024-04-24T23:16:18.303989+0000","last_became_active":"2024-04-24T23:16:18.299217+0000","last_became_peered":"2024-04-24T23:16:18.299217+0000","last_unstale":"2024-04-24T23:16:18.303989+0000","last_undegraded":"2024-04-24T23:16:18.303989+0000","last_fullsized":"2024-04-24T23:16:18.303989+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:08:19.426862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4,1],"acting":[0,6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"26'2","reported_seq":28,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.292303+0000","last_change":"2024-04-24T23:15:35.946083+0000","last_active":"2024-04-24T23:16:18.292303+0000","last_peered":"2024-04-24T23:16:18.292303+0000","last_clean":"2024-04-24T23:16:18.292303+0000","last_became_active":"2024-04-24T23:15:35.945227+0000","last_became_peered":"2024-04-24T23:15:35.945227+0000","last_unstale":"2024-04-24T23:16:18.292303+0000","last_undegraded":"2024-04-24T23:16:18.292303+0000","last_fullsized":"2024-04-24T23:16:18.292303+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:05:17.727725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.292034+0000","last_change":"2024-04-24T23:15:36.998245+0000","last_active":"2024-04-24T23:16:18.292034+0000","last_peered":"2024-04-24T23:16:18.292034+0000","last_clean":"2024-04-24T23:16:18.292034+0000","last_became_active":"2024-04-24T23:15:36.998021+0000","last_became_peered":"2024-04-24T23:15:36.998021+0000","last_unstale":"2024-04-24T23:16:18.292034+0000","last_undegraded":"2024-04-24T23:16:18.292034+0000","last_fullsized":"2024-04-24T23:16:18.292034+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:03:42.359117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.293518+0000","last_change":"2024-04-24T23:15:41.008313+0000","last_active":"2024-04-24T23:16:18.293518+0000","last_peered":"2024-04-24T23:16:18.293518+0000","last_clean":"2024-04-24T23:16:18.293518+0000","last_became_active":"2024-04-24T23:15:41.007908+0000","last_became_peered":"2024-04-24T23:15:41.007908+0000","last_unstale":"2024-04-24T23:16:18.293518+0000","last_undegraded":"2024-04-24T23:16:18.293518+0000","last_fullsized":"2024-04-24T23:16:18.293518+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:02:11.902575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4,6],"acting":[2,7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1d","version":"26'12","reported_seq":38,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:31.305540+0000","last_change":"2024-04-24T23:15:35.968312+0000","last_active":"2024-04-24T23:16:31.305540+0000","last_peered":"2024-04-24T23:16:31.305540+0000","last_clean":"2024-04-24T23:16:31.305540+0000","last_became_active":"2024-04-24T23:15:35.967873+0000","last_became_peered":"2024-04-24T23:15:35.967873+0000","last_unstale":"2024-04-24T23:16:31.305540+0000","last_undegraded":"2024-04-24T23:16:31.305540+0000","last_fullsized":"2024-04-24T23:16:31.305540+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:21:40.116977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.296594+0000","last_change":"2024-04-24T23:15:36.997361+0000","last_active":"2024-04-24T23:16:18.296594+0000","last_peered":"2024-04-24T23:16:18.296594+0000","last_clean":"2024-04-24T23:16:18.296594+0000","last_became_active":"2024-04-24T23:15:36.997083+0000","last_became_peered":"2024-04-24T23:15:36.997083+0000","last_unstale":"2024-04-24T23:16:18.296594+0000","last_undegraded":"2024-04-24T23:16:18.296594+0000","last_fullsized":"2024-04-24T23:16:18.296594+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:47:59.334093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.293518+0000","last_change":"2024-04-24T23:15:41.025153+0000","last_active":"2024-04-24T23:16:18.293518+0000","last_peered":"2024-04-24T23:16:18.293518+0000","last_clean":"2024-04-24T23:16:18.293518+0000","last_became_active":"2024-04-24T23:15:41.024739+0000","last_became_peered":"2024-04-24T23:15:41.024739+0000","last_unstale":"2024-04-24T23:16:18.293518+0000","last_undegraded":"2024-04-24T23:16:18.293518+0000","last_fullsized":"2024-04-24T23:16:18.293518+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:31:00.160213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3,6],"acting":[2,5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1c","version":"26'5","reported_seq":31,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:31.301798+0000","last_change":"2024-04-24T23:15:35.967338+0000","last_active":"2024-04-24T23:16:31.301798+0000","last_peered":"2024-04-24T23:16:31.301798+0000","last_clean":"2024-04-24T23:16:31.301798+0000","last_became_active":"2024-04-24T23:15:35.966951+0000","last_became_peered":"2024-04-24T23:15:35.966951+0000","last_unstale":"2024-04-24T23:16:31.301798+0000","last_undegraded":"2024-04-24T23:16:31.301798+0000","last_fullsized":"2024-04-24T23:16:31.301798+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:08:05.248330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.230858+0000","last_change":"2024-04-24T23:15:36.998072+0000","last_active":"2024-04-24T23:16:18.230858+0000","last_peered":"2024-04-24T23:16:18.230858+0000","last_clean":"2024-04-24T23:16:18.230858+0000","last_became_active":"2024-04-24T23:15:36.997735+0000","last_became_peered":"2024-04-24T23:15:36.997735+0000","last_unstale":"2024-04-24T23:16:18.230858+0000","last_undegraded":"2024-04-24T23:16:18.230858+0000","last_fullsized":"2024-04-24T23:16:18.230858+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:22:06.869644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.294029+0000","last_change":"2024-04-24T23:15:41.008895+0000","last_active":"2024-04-24T23:16:18.294029+0000","last_peered":"2024-04-24T23:16:18.294029+0000","last_clean":"2024-04-24T23:16:18.294029+0000","last_became_active":"2024-04-24T23:15:41.006199+0000","last_became_peered":"2024-04-24T23:15:41.006199+0000","last_unstale":"2024-04-24T23:16:18.294029+0000","last_undegraded":"2024-04-24T23:16:18.294029+0000","last_fullsized":"2024-04-24T23:16:18.294029+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:13:34.583602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5,7],"acting":[2,6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.233639+0000","last_change":"2024-04-24T23:15:37.004074+0000","last_active":"2024-04-24T23:16:18.233639+0000","last_peered":"2024-04-24T23:16:18.233639+0000","last_clean":"2024-04-24T23:16:18.233639+0000","last_became_active":"2024-04-24T23:15:37.003544+0000","last_became_peered":"2024-04-24T23:15:37.003544+0000","last_unstale":"2024-04-24T23:16:18.233639+0000","last_undegraded":"2024-04-24T23:16:18.233639+0000","last_fullsized":"2024-04-24T23:16:18.233639+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:38:28.024702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"24'4","reported_seq":30,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.230075+0000","last_change":"2024-04-24T23:15:35.963000+0000","last_active":"2024-04-24T23:16:18.230075+0000","last_peered":"2024-04-24T23:16:18.230075+0000","last_clean":"2024-04-24T23:16:18.230075+0000","last_became_active":"2024-04-24T23:15:35.961960+0000","last_became_peered":"2024-04-24T23:15:35.961960+0000","last_unstale":"2024-04-24T23:16:18.230075+0000","last_undegraded":"2024-04-24T23:16:18.230075+0000","last_fullsized":"2024-04-24T23:16:18.230075+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:56:26.651193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.230595+0000","last_change":"2024-04-24T23:15:41.022481+0000","last_active":"2024-04-24T23:16:18.230595+0000","last_peered":"2024-04-24T23:16:18.230595+0000","last_clean":"2024-04-24T23:16:18.230595+0000","last_became_active":"2024-04-24T23:15:41.020034+0000","last_became_peered":"2024-04-24T23:15:41.020034+0000","last_unstale":"2024-04-24T23:16:18.230595+0000","last_undegraded":"2024-04-24T23:16:18.230595+0000","last_fullsized":"2024-04-24T23:16:18.230595+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:27:25.337503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,5],"acting":[3,1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.292564+0000","last_change":"2024-04-24T23:15:36.972020+0000","last_active":"2024-04-24T23:16:18.292564+0000","last_peered":"2024-04-24T23:16:18.292564+0000","last_clean":"2024-04-24T23:16:18.292564+0000","last_became_active":"2024-04-24T23:15:36.971583+0000","last_became_peered":"2024-04-24T23:15:36.971583+0000","last_unstale":"2024-04-24T23:16:18.292564+0000","last_undegraded":"2024-04-24T23:16:18.292564+0000","last_fullsized":"2024-04-24T23:16:18.292564+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:29:05.568608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"26'3","reported_seq":29,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:25.265282+0000","last_change":"2024-04-24T23:15:35.945916+0000","last_active":"2024-04-24T23:16:25.265282+0000","last_peered":"2024-04-24T23:16:25.265282+0000","last_clean":"2024-04-24T23:16:25.265282+0000","last_became_active":"2024-04-24T23:15:35.945610+0000","last_became_peered":"2024-04-24T23:15:35.945610+0000","last_unstale":"2024-04-24T23:16:25.265282+0000","last_undegraded":"2024-04-24T23:16:25.265282+0000","last_fullsized":"2024-04-24T23:16:25.265282+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:57:05.976074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.229068+0000","last_change":"2024-04-24T23:15:41.030165+0000","last_active":"2024-04-24T23:16:18.229068+0000","last_peered":"2024-04-24T23:16:18.229068+0000","last_clean":"2024-04-24T23:16:18.229068+0000","last_became_active":"2024-04-24T23:15:41.029685+0000","last_became_peered":"2024-04-24T23:15:41.029685+0000","last_unstale":"2024-04-24T23:16:18.229068+0000","last_undegraded":"2024-04-24T23:16:18.229068+0000","last_fullsized":"2024-04-24T23:16:18.229068+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:20:43.498190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6,1],"acting":[0,4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.223468+0000","last_change":"2024-04-24T23:15:36.965949+0000","last_active":"2024-04-24T23:16:18.223468+0000","last_peered":"2024-04-24T23:16:18.223468+0000","last_clean":"2024-04-24T23:16:18.223468+0000","last_became_active":"2024-04-24T23:15:36.965658+0000","last_became_peered":"2024-04-24T23:15:36.965658+0000","last_unstale":"2024-04-24T23:16:18.223468+0000","last_undegraded":"2024-04-24T23:16:18.223468+0000","last_fullsized":"2024-04-24T23:16:18.223468+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:38:28.024702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"26'7","reported_seq":33,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:31.301052+0000","last_change":"2024-04-24T23:15:35.972165+0000","last_active":"2024-04-24T23:16:31.301052+0000","last_peered":"2024-04-24T23:16:31.301052+0000","last_clean":"2024-04-24T23:16:31.301052+0000","last_became_active":"2024-04-24T23:15:35.971972+0000","last_became_peered":"2024-04-24T23:15:35.971972+0000","last_unstale":"2024-04-24T23:16:31.301052+0000","last_undegraded":"2024-04-24T23:16:31.301052+0000","last_fullsized":"2024-04-24T23:16:31.301052+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:11:12.088698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.292911+0000","last_change":"2024-04-24T23:15:41.019799+0000","last_active":"2024-04-24T23:16:18.292911+0000","last_peered":"2024-04-24T23:16:18.292911+0000","last_clean":"2024-04-24T23:16:18.292911+0000","last_became_active":"2024-04-24T23:15:41.019339+0000","last_became_peered":"2024-04-24T23:15:41.019339+0000","last_unstale":"2024-04-24T23:16:18.292911+0000","last_undegraded":"2024-04-24T23:16:18.292911+0000","last_fullsized":"2024-04-24T23:16:18.292911+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:59:34.601004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2,3],"acting":[7,4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.232810+0000","last_change":"2024-04-24T23:15:36.991991+0000","last_active":"2024-04-24T23:16:18.232810+0000","last_peered":"2024-04-24T23:16:18.232810+0000","last_clean":"2024-04-24T23:16:18.232810+0000","last_became_active":"2024-04-24T23:15:36.989336+0000","last_became_peered":"2024-04-24T23:15:36.989336+0000","last_unstale":"2024-04-24T23:16:18.232810+0000","last_undegraded":"2024-04-24T23:16:18.232810+0000","last_fullsized":"2024-04-24T23:16:18.232810+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:25:56.038903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.8","version":"24'2","reported_seq":28,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.228663+0000","last_change":"2024-04-24T23:15:35.965443+0000","last_active":"2024-04-24T23:16:18.228663+0000","last_peered":"2024-04-24T23:16:18.228663+0000","last_clean":"2024-04-24T23:16:18.228663+0000","last_became_active":"2024-04-24T23:15:35.965043+0000","last_became_peered":"2024-04-24T23:15:35.965043+0000","last_unstale":"2024-04-24T23:16:18.228663+0000","last_undegraded":"2024-04-24T23:16:18.228663+0000","last_fullsized":"2024-04-24T23:16:18.228663+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:07:23.320042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.294499+0000","last_change":"2024-04-24T23:15:41.025131+0000","last_active":"2024-04-24T23:16:18.294499+0000","last_peered":"2024-04-24T23:16:18.294499+0000","last_clean":"2024-04-24T23:16:18.294499+0000","last_became_active":"2024-04-24T23:15:41.024731+0000","last_became_peered":"2024-04-24T23:15:41.024731+0000","last_unstale":"2024-04-24T23:16:18.294499+0000","last_undegraded":"2024-04-24T23:16:18.294499+0000","last_fullsized":"2024-04-24T23:16:18.294499+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:28:07.952854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,7],"acting":[2,1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.224465+0000","last_change":"2024-04-24T23:15:36.965850+0000","last_active":"2024-04-24T23:16:18.224465+0000","last_peered":"2024-04-24T23:16:18.224465+0000","last_clean":"2024-04-24T23:16:18.224465+0000","last_became_active":"2024-04-24T23:15:36.965547+0000","last_became_peered":"2024-04-24T23:15:36.965547+0000","last_unstale":"2024-04-24T23:16:18.224465+0000","last_undegraded":"2024-04-24T23:16:18.224465+0000","last_fullsized":"2024-04-24T23:16:18.224465+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:06:37.125573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"26'7","reported_seq":33,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:31.305921+0000","last_change":"2024-04-24T23:15:35.971040+0000","last_active":"2024-04-24T23:16:31.305921+0000","last_peered":"2024-04-24T23:16:31.305921+0000","last_clean":"2024-04-24T23:16:31.305921+0000","last_became_active":"2024-04-24T23:15:35.970184+0000","last_became_peered":"2024-04-24T23:15:35.970184+0000","last_unstale":"2024-04-24T23:16:31.305921+0000","last_undegraded":"2024-04-24T23:16:31.305921+0000","last_fullsized":"2024-04-24T23:16:31.305921+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:45:20.913759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.295046+0000","last_change":"2024-04-24T23:15:41.025813+0000","last_active":"2024-04-24T23:16:18.295046+0000","last_peered":"2024-04-24T23:16:18.295046+0000","last_clean":"2024-04-24T23:16:18.295046+0000","last_became_active":"2024-04-24T23:15:41.025137+0000","last_became_peered":"2024-04-24T23:15:41.025137+0000","last_unstale":"2024-04-24T23:16:18.295046+0000","last_undegraded":"2024-04-24T23:16:18.295046+0000","last_fullsized":"2024-04-24T23:16:18.295046+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:35:17.784854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,3,1],"acting":[2,6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.224731+0000","last_change":"2024-04-24T23:15:36.967476+0000","last_active":"2024-04-24T23:16:18.224731+0000","last_peered":"2024-04-24T23:16:18.224731+0000","last_clean":"2024-04-24T23:16:18.224731+0000","last_became_active":"2024-04-24T23:15:36.967098+0000","last_became_peered":"2024-04-24T23:15:36.967098+0000","last_unstale":"2024-04-24T23:16:18.224731+0000","last_undegraded":"2024-04-24T23:16:18.224731+0000","last_fullsized":"2024-04-24T23:16:18.224731+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:25:56.038903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"26'7","reported_seq":33,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:25.270982+0000","last_change":"2024-04-24T23:15:35.970630+0000","last_active":"2024-04-24T23:16:25.270982+0000","last_peered":"2024-04-24T23:16:25.270982+0000","last_clean":"2024-04-24T23:16:25.270982+0000","last_became_active":"2024-04-24T23:15:35.970430+0000","last_became_peered":"2024-04-24T23:15:35.970430+0000","last_unstale":"2024-04-24T23:16:25.270982+0000","last_undegraded":"2024-04-24T23:16:25.270982+0000","last_fullsized":"2024-04-24T23:16:25.270982+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:15:59.219084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.223468+0000","last_change":"2024-04-24T23:15:41.005022+0000","last_active":"2024-04-24T23:16:18.223468+0000","last_peered":"2024-04-24T23:16:18.223468+0000","last_clean":"2024-04-24T23:16:18.223468+0000","last_became_active":"2024-04-24T23:15:41.004648+0000","last_became_peered":"2024-04-24T23:15:41.004648+0000","last_unstale":"2024-04-24T23:16:18.223468+0000","last_undegraded":"2024-04-24T23:16:18.223468+0000","last_fullsized":"2024-04-24T23:16:18.223468+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:06:01.559760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6,3],"acting":[4,1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.225003+0000","last_change":"2024-04-24T23:15:36.967053+0000","last_active":"2024-04-24T23:16:18.225003+0000","last_peered":"2024-04-24T23:16:18.225003+0000","last_clean":"2024-04-24T23:16:18.225003+0000","last_became_active":"2024-04-24T23:15:36.966746+0000","last_became_peered":"2024-04-24T23:15:36.966746+0000","last_unstale":"2024-04-24T23:16:18.225003+0000","last_undegraded":"2024-04-24T23:16:18.225003+0000","last_fullsized":"2024-04-24T23:16:18.225003+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:16:08.779860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"26'4","reported_seq":30,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:25.272679+0000","last_change":"2024-04-24T23:15:35.965890+0000","last_active":"2024-04-24T23:16:25.272679+0000","last_peered":"2024-04-24T23:16:25.272679+0000","last_clean":"2024-04-24T23:16:25.272679+0000","last_became_active":"2024-04-24T23:15:35.965631+0000","last_became_peered":"2024-04-24T23:15:35.965631+0000","last_unstale":"2024-04-24T23:16:25.272679+0000","last_undegraded":"2024-04-24T23:16:25.272679+0000","last_fullsized":"2024-04-24T23:16:25.272679+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:17:35.650731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.223982+0000","last_change":"2024-04-24T23:15:41.011405+0000","last_active":"2024-04-24T23:16:18.223982+0000","last_peered":"2024-04-24T23:16:18.223982+0000","last_clean":"2024-04-24T23:16:18.223982+0000","last_became_active":"2024-04-24T23:15:41.010982+0000","last_became_peered":"2024-04-24T23:15:41.010982+0000","last_unstale":"2024-04-24T23:16:18.223982+0000","last_undegraded":"2024-04-24T23:16:18.223982+0000","last_fullsized":"2024-04-24T23:16:18.223982+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:08:00.914296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2,7],"acting":[6,0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.294828+0000","last_change":"2024-04-24T23:15:36.975244+0000","last_active":"2024-04-24T23:16:18.294828+0000","last_peered":"2024-04-24T23:16:18.294828+0000","last_clean":"2024-04-24T23:16:18.294828+0000","last_became_active":"2024-04-24T23:15:36.974872+0000","last_became_peered":"2024-04-24T23:15:36.974872+0000","last_unstale":"2024-04-24T23:16:18.294828+0000","last_undegraded":"2024-04-24T23:16:18.294828+0000","last_fullsized":"2024-04-24T23:16:18.294828+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:45:57.018205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"26'1","reported_seq":27,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.291824+0000","last_change":"2024-04-24T23:15:35.945792+0000","last_active":"2024-04-24T23:16:18.291824+0000","last_peered":"2024-04-24T23:16:18.291824+0000","last_clean":"2024-04-24T23:16:18.291824+0000","last_became_active":"2024-04-24T23:15:35.945094+0000","last_became_peered":"2024-04-24T23:15:35.945094+0000","last_unstale":"2024-04-24T23:16:18.291824+0000","last_undegraded":"2024-04-24T23:16:18.291824+0000","last_fullsized":"2024-04-24T23:16:18.291824+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:49:12.719127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.224614+0000","last_change":"2024-04-24T23:15:41.022105+0000","last_active":"2024-04-24T23:16:18.224614+0000","last_peered":"2024-04-24T23:16:18.224614+0000","last_clean":"2024-04-24T23:16:18.224614+0000","last_became_active":"2024-04-24T23:15:41.021645+0000","last_became_peered":"2024-04-24T23:15:41.021645+0000","last_unstale":"2024-04-24T23:16:18.224614+0000","last_undegraded":"2024-04-24T23:16:18.224614+0000","last_fullsized":"2024-04-24T23:16:18.224614+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:03:52.974835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5,7],"acting":[6,3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.229269+0000","last_change":"2024-04-24T23:15:36.991239+0000","last_active":"2024-04-24T23:16:18.229269+0000","last_peered":"2024-04-24T23:16:18.229269+0000","last_clean":"2024-04-24T23:16:18.229269+0000","last_became_active":"2024-04-24T23:15:36.990963+0000","last_became_peered":"2024-04-24T23:15:36.990963+0000","last_unstale":"2024-04-24T23:16:18.229269+0000","last_undegraded":"2024-04-24T23:16:18.229269+0000","last_fullsized":"2024-04-24T23:16:18.229269+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:53:28.101334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"26'5","reported_seq":31,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:31.304824+0000","last_change":"2024-04-24T23:15:35.957931+0000","last_active":"2024-04-24T23:16:31.304824+0000","last_peered":"2024-04-24T23:16:31.304824+0000","last_clean":"2024-04-24T23:16:31.304824+0000","last_became_active":"2024-04-24T23:15:35.957432+0000","last_became_peered":"2024-04-24T23:15:35.957432+0000","last_unstale":"2024-04-24T23:16:31.304824+0000","last_undegraded":"2024-04-24T23:16:31.304824+0000","last_fullsized":"2024-04-24T23:16:31.304824+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:11:38.773425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.224009+0000","last_change":"2024-04-24T23:15:41.007857+0000","last_active":"2024-04-24T23:16:18.224009+0000","last_peered":"2024-04-24T23:16:18.224009+0000","last_clean":"2024-04-24T23:16:18.224009+0000","last_became_active":"2024-04-24T23:15:41.007449+0000","last_became_peered":"2024-04-24T23:15:41.007449+0000","last_unstale":"2024-04-24T23:16:18.224009+0000","last_undegraded":"2024-04-24T23:16:18.224009+0000","last_fullsized":"2024-04-24T23:16:18.224009+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:05:06.145440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0,3],"acting":[6,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.291576+0000","last_change":"2024-04-24T23:15:36.970444+0000","last_active":"2024-04-24T23:16:18.291576+0000","last_peered":"2024-04-24T23:16:18.291576+0000","last_clean":"2024-04-24T23:16:18.291576+0000","last_became_active":"2024-04-24T23:15:36.970159+0000","last_became_peered":"2024-04-24T23:15:36.970159+0000","last_unstale":"2024-04-24T23:16:18.291576+0000","last_undegraded":"2024-04-24T23:16:18.291576+0000","last_fullsized":"2024-04-24T23:16:18.291576+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:37:02.956224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"26'5","reported_seq":31,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:19.220734+0000","last_change":"2024-04-24T23:15:35.945461+0000","last_active":"2024-04-24T23:16:19.220734+0000","last_peered":"2024-04-24T23:16:19.220734+0000","last_clean":"2024-04-24T23:16:19.220734+0000","last_became_active":"2024-04-24T23:15:35.945185+0000","last_became_peered":"2024-04-24T23:15:35.945185+0000","last_unstale":"2024-04-24T23:16:19.220734+0000","last_undegraded":"2024-04-24T23:16:19.220734+0000","last_fullsized":"2024-04-24T23:16:19.220734+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:37:34.175539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.224370+0000","last_change":"2024-04-24T23:15:41.014584+0000","last_active":"2024-04-24T23:16:18.224370+0000","last_peered":"2024-04-24T23:16:18.224370+0000","last_clean":"2024-04-24T23:16:18.224370+0000","last_became_active":"2024-04-24T23:15:41.014137+0000","last_became_peered":"2024-04-24T23:15:41.014137+0000","last_unstale":"2024-04-24T23:16:18.224370+0000","last_undegraded":"2024-04-24T23:16:18.224370+0000","last_fullsized":"2024-04-24T23:16:18.224370+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:23:50.387635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,7],"acting":[6,0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.224277+0000","last_change":"2024-04-24T23:15:36.981783+0000","last_active":"2024-04-24T23:16:18.224277+0000","last_peered":"2024-04-24T23:16:18.224277+0000","last_clean":"2024-04-24T23:16:18.224277+0000","last_became_active":"2024-04-24T23:15:36.981502+0000","last_became_peered":"2024-04-24T23:15:36.981502+0000","last_unstale":"2024-04-24T23:16:18.224277+0000","last_undegraded":"2024-04-24T23:16:18.224277+0000","last_fullsized":"2024-04-24T23:16:18.224277+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:56:08.483410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"26'4","reported_seq":30,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:19.225767+0000","last_change":"2024-04-24T23:15:35.960519+0000","last_active":"2024-04-24T23:16:19.225767+0000","last_peered":"2024-04-24T23:16:19.225767+0000","last_clean":"2024-04-24T23:16:19.225767+0000","last_became_active":"2024-04-24T23:15:35.960180+0000","last_became_peered":"2024-04-24T23:15:35.960180+0000","last_unstale":"2024-04-24T23:16:19.225767+0000","last_undegraded":"2024-04-24T23:16:19.225767+0000","last_fullsized":"2024-04-24T23:16:19.225767+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:50:00.199120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.291646+0000","last_change":"2024-04-24T23:15:41.012876+0000","last_active":"2024-04-24T23:16:18.291646+0000","last_peered":"2024-04-24T23:16:18.291646+0000","last_clean":"2024-04-24T23:16:18.291646+0000","last_became_active":"2024-04-24T23:15:41.012493+0000","last_became_peered":"2024-04-24T23:15:41.012493+0000","last_unstale":"2024-04-24T23:16:18.291646+0000","last_undegraded":"2024-04-24T23:16:18.291646+0000","last_fullsized":"2024-04-24T23:16:18.291646+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:21:28.287059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6,1],"acting":[5,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.224197+0000","last_change":"2024-04-24T23:15:36.981449+0000","last_active":"2024-04-24T23:16:18.224197+0000","last_peered":"2024-04-24T23:16:18.224197+0000","last_clean":"2024-04-24T23:16:18.224197+0000","last_became_active":"2024-04-24T23:15:36.980837+0000","last_became_peered":"2024-04-24T23:15:36.980837+0000","last_unstale":"2024-04-24T23:16:18.224197+0000","last_undegraded":"2024-04-24T23:16:18.224197+0000","last_fullsized":"2024-04-24T23:16:18.224197+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:15:13.513341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.3","version":"26'6","reported_seq":32,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:25.276387+0000","last_change":"2024-04-24T23:15:35.952947+0000","last_active":"2024-04-24T23:16:25.276387+0000","last_peered":"2024-04-24T23:16:25.276387+0000","last_clean":"2024-04-24T23:16:25.276387+0000","last_became_active":"2024-04-24T23:15:35.952655+0000","last_became_peered":"2024-04-24T23:15:35.952655+0000","last_unstale":"2024-04-24T23:16:25.276387+0000","last_undegraded":"2024-04-24T23:16:25.276387+0000","last_fullsized":"2024-04-24T23:16:25.276387+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:50:10.939396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":182,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.296073+0000","last_change":"2024-04-24T23:15:25.826673+0000","last_active":"2024-04-24T23:16:18.296073+0000","last_peered":"2024-04-24T23:16:18.296073+0000","last_clean":"2024-04-24T23:16:18.296073+0000","last_became_active":"2024-04-24T23:15:25.820030+0000","last_became_peered":"2024-04-24T23:15:25.820030+0000","last_unstale":"2024-04-24T23:16:18.296073+0000","last_undegraded":"2024-04-24T23:16:18.296073+0000","last_fullsized":"2024-04-24T23:16:18.296073+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:24.790928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:24.790928+0000","last_clean_scrub_stamp":"2024-04-24T23:15:24.790928+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:37:24.054332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.231043+0000","last_change":"2024-04-24T23:15:41.024712+0000","last_active":"2024-04-24T23:16:18.231043+0000","last_peered":"2024-04-24T23:16:18.231043+0000","last_clean":"2024-04-24T23:16:18.231043+0000","last_became_active":"2024-04-24T23:15:41.022697+0000","last_became_peered":"2024-04-24T23:15:41.022697+0000","last_unstale":"2024-04-24T23:16:18.231043+0000","last_undegraded":"2024-04-24T23:16:18.231043+0000","last_fullsized":"2024-04-24T23:16:18.231043+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:37:08.231105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2,0],"acting":[3,1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.230951+0000","last_change":"2024-04-24T23:15:37.000489+0000","last_active":"2024-04-24T23:16:18.230951+0000","last_peered":"2024-04-24T23:16:18.230951+0000","last_clean":"2024-04-24T23:16:18.230951+0000","last_became_active":"2024-04-24T23:15:37.000179+0000","last_became_peered":"2024-04-24T23:15:37.000179+0000","last_unstale":"2024-04-24T23:16:18.230951+0000","last_undegraded":"2024-04-24T23:16:18.230951+0000","last_fullsized":"2024-04-24T23:16:18.230951+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:06:37.125573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.c","version":"26'5","reported_seq":31,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:25.270158+0000","last_change":"2024-04-24T23:15:35.966020+0000","last_active":"2024-04-24T23:16:25.270158+0000","last_peered":"2024-04-24T23:16:25.270158+0000","last_clean":"2024-04-24T23:16:25.270158+0000","last_became_active":"2024-04-24T23:15:35.965758+0000","last_became_peered":"2024-04-24T23:15:35.965758+0000","last_unstale":"2024-04-24T23:16:25.270158+0000","last_undegraded":"2024-04-24T23:16:25.270158+0000","last_fullsized":"2024-04-24T23:16:25.270158+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:39:21.233461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.294138+0000","last_change":"2024-04-24T23:15:41.024489+0000","last_active":"2024-04-24T23:16:18.294138+0000","last_peered":"2024-04-24T23:16:18.294138+0000","last_clean":"2024-04-24T23:16:18.294138+0000","last_became_active":"2024-04-24T23:15:41.024041+0000","last_became_peered":"2024-04-24T23:15:41.024041+0000","last_unstale":"2024-04-24T23:16:18.294138+0000","last_undegraded":"2024-04-24T23:16:18.294138+0000","last_fullsized":"2024-04-24T23:16:18.294138+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:27:18.197003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7,4],"acting":[2,3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.229879+0000","last_change":"2024-04-24T23:15:36.979887+0000","last_active":"2024-04-24T23:16:18.229879+0000","last_peered":"2024-04-24T23:16:18.229879+0000","last_clean":"2024-04-24T23:16:18.229879+0000","last_became_active":"2024-04-24T23:15:36.979611+0000","last_became_peered":"2024-04-24T23:15:36.979611+0000","last_unstale":"2024-04-24T23:16:18.229879+0000","last_undegraded":"2024-04-24T23:16:18.229879+0000","last_fullsized":"2024-04-24T23:16:18.229879+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:07:18.261077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"26'3","reported_seq":29,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:25.265069+0000","last_change":"2024-04-24T23:15:35.965192+0000","last_active":"2024-04-24T23:16:25.265069+0000","last_peered":"2024-04-24T23:16:25.265069+0000","last_clean":"2024-04-24T23:16:25.265069+0000","last_became_active":"2024-04-24T23:15:35.964873+0000","last_became_peered":"2024-04-24T23:15:35.964873+0000","last_unstale":"2024-04-24T23:16:25.265069+0000","last_undegraded":"2024-04-24T23:16:25.265069+0000","last_fullsized":"2024-04-24T23:16:25.265069+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:14:27.884944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.291986+0000","last_change":"2024-04-24T23:15:41.004002+0000","last_active":"2024-04-24T23:16:18.291986+0000","last_peered":"2024-04-24T23:16:18.291986+0000","last_clean":"2024-04-24T23:16:18.291986+0000","last_became_active":"2024-04-24T23:15:41.003335+0000","last_became_peered":"2024-04-24T23:15:41.003335+0000","last_unstale":"2024-04-24T23:16:18.291986+0000","last_undegraded":"2024-04-24T23:16:18.291986+0000","last_fullsized":"2024-04-24T23:16:18.291986+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:49:36.206903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1,3],"acting":[5,4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.229375+0000","last_change":"2024-04-24T23:15:37.002854+0000","last_active":"2024-04-24T23:16:18.229375+0000","last_peered":"2024-04-24T23:16:18.229375+0000","last_clean":"2024-04-24T23:16:18.229375+0000","last_became_active":"2024-04-24T23:15:37.002583+0000","last_became_peered":"2024-04-24T23:15:37.002583+0000","last_unstale":"2024-04-24T23:16:18.229375+0000","last_undegraded":"2024-04-24T23:16:18.229375+0000","last_fullsized":"2024-04-24T23:16:18.229375+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:27:04.251547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"26'5","reported_seq":31,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:31.299455+0000","last_change":"2024-04-24T23:15:35.948752+0000","last_active":"2024-04-24T23:16:31.299455+0000","last_peered":"2024-04-24T23:16:31.299455+0000","last_clean":"2024-04-24T23:16:31.299455+0000","last_became_active":"2024-04-24T23:15:35.948438+0000","last_became_peered":"2024-04-24T23:15:35.948438+0000","last_unstale":"2024-04-24T23:16:31.299455+0000","last_undegraded":"2024-04-24T23:16:31.299455+0000","last_fullsized":"2024-04-24T23:16:31.299455+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:45:56.006278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.229925+0000","last_change":"2024-04-24T23:15:41.017524+0000","last_active":"2024-04-24T23:16:18.229925+0000","last_peered":"2024-04-24T23:16:18.229925+0000","last_clean":"2024-04-24T23:16:18.229925+0000","last_became_active":"2024-04-24T23:15:41.017092+0000","last_became_peered":"2024-04-24T23:15:41.017092+0000","last_unstale":"2024-04-24T23:16:18.229925+0000","last_undegraded":"2024-04-24T23:16:18.229925+0000","last_fullsized":"2024-04-24T23:16:18.229925+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:04:53.235038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.294797+0000","last_change":"2024-04-24T23:15:36.990441+0000","last_active":"2024-04-24T23:16:18.294797+0000","last_peered":"2024-04-24T23:16:18.294797+0000","last_clean":"2024-04-24T23:16:18.294797+0000","last_became_active":"2024-04-24T23:15:36.989863+0000","last_became_peered":"2024-04-24T23:15:36.989863+0000","last_unstale":"2024-04-24T23:16:18.294797+0000","last_undegraded":"2024-04-24T23:16:18.294797+0000","last_fullsized":"2024-04-24T23:16:18.294797+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:50:07.975347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.f","version":"26'2","reported_seq":28,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.232012+0000","last_change":"2024-04-24T23:15:35.969317+0000","last_active":"2024-04-24T23:16:18.232012+0000","last_peered":"2024-04-24T23:16:18.232012+0000","last_clean":"2024-04-24T23:16:18.232012+0000","last_became_active":"2024-04-24T23:15:35.968994+0000","last_became_peered":"2024-04-24T23:15:35.968994+0000","last_unstale":"2024-04-24T23:16:18.232012+0000","last_undegraded":"2024-04-24T23:16:18.232012+0000","last_fullsized":"2024-04-24T23:16:18.232012+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:51:57.100772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.296718+0000","last_change":"2024-04-24T23:15:41.010103+0000","last_active":"2024-04-24T23:16:18.296718+0000","last_peered":"2024-04-24T23:16:18.296718+0000","last_clean":"2024-04-24T23:16:18.296718+0000","last_became_active":"2024-04-24T23:15:41.009328+0000","last_became_peered":"2024-04-24T23:15:41.009328+0000","last_unstale":"2024-04-24T23:16:18.296718+0000","last_undegraded":"2024-04-24T23:16:18.296718+0000","last_fullsized":"2024-04-24T23:16:18.296718+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:46:49.579501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,2],"acting":[1,5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.232575+0000","last_change":"2024-04-24T23:15:36.996765+0000","last_active":"2024-04-24T23:16:18.232575+0000","last_peered":"2024-04-24T23:16:18.232575+0000","last_clean":"2024-04-24T23:16:18.232575+0000","last_became_active":"2024-04-24T23:15:36.995801+0000","last_became_peered":"2024-04-24T23:15:36.995801+0000","last_unstale":"2024-04-24T23:16:18.232575+0000","last_undegraded":"2024-04-24T23:16:18.232575+0000","last_fullsized":"2024-04-24T23:16:18.232575+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:25:19.421667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"26'17","reported_seq":43,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:25.266099+0000","last_change":"2024-04-24T23:15:35.944023+0000","last_active":"2024-04-24T23:16:25.266099+0000","last_peered":"2024-04-24T23:16:25.266099+0000","last_clean":"2024-04-24T23:16:25.266099+0000","last_became_active":"2024-04-24T23:15:35.943710+0000","last_became_peered":"2024-04-24T23:15:35.943710+0000","last_unstale":"2024-04-24T23:16:25.266099+0000","last_undegraded":"2024-04-24T23:16:25.266099+0000","last_fullsized":"2024-04-24T23:16:25.266099+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:46:48.090958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.292053+0000","last_change":"2024-04-24T23:15:41.013027+0000","last_active":"2024-04-24T23:16:18.292053+0000","last_peered":"2024-04-24T23:16:18.292053+0000","last_clean":"2024-04-24T23:16:18.292053+0000","last_became_active":"2024-04-24T23:15:41.012587+0000","last_became_peered":"2024-04-24T23:15:41.012587+0000","last_unstale":"2024-04-24T23:16:18.292053+0000","last_undegraded":"2024-04-24T23:16:18.292053+0000","last_fullsized":"2024-04-24T23:16:18.292053+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:27:04.276246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1,6],"acting":[5,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.295974+0000","last_change":"2024-04-24T23:15:36.979702+0000","last_active":"2024-04-24T23:16:18.295974+0000","last_peered":"2024-04-24T23:16:18.295974+0000","last_clean":"2024-04-24T23:16:18.295974+0000","last_became_active":"2024-04-24T23:15:36.979356+0000","last_became_peered":"2024-04-24T23:15:36.979356+0000","last_unstale":"2024-04-24T23:16:18.295974+0000","last_undegraded":"2024-04-24T23:16:18.295974+0000","last_fullsized":"2024-04-24T23:16:18.295974+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:14:24.309192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.11","version":"26'5","reported_seq":31,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:25.270027+0000","last_change":"2024-04-24T23:15:35.973026+0000","last_active":"2024-04-24T23:16:25.270027+0000","last_peered":"2024-04-24T23:16:25.270027+0000","last_clean":"2024-04-24T23:16:25.270027+0000","last_became_active":"2024-04-24T23:15:35.972655+0000","last_became_peered":"2024-04-24T23:15:35.972655+0000","last_unstale":"2024-04-24T23:16:25.270027+0000","last_undegraded":"2024-04-24T23:16:25.270027+0000","last_fullsized":"2024-04-24T23:16:25.270027+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:25:34.352908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.291924+0000","last_change":"2024-04-24T23:15:41.004221+0000","last_active":"2024-04-24T23:16:18.291924+0000","last_peered":"2024-04-24T23:16:18.291924+0000","last_clean":"2024-04-24T23:16:18.291924+0000","last_became_active":"2024-04-24T23:15:41.003790+0000","last_became_peered":"2024-04-24T23:15:41.003790+0000","last_unstale":"2024-04-24T23:16:18.291924+0000","last_undegraded":"2024-04-24T23:16:18.291924+0000","last_fullsized":"2024-04-24T23:16:18.291924+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:28:58.866024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6,4],"acting":[5,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.229856+0000","last_change":"2024-04-24T23:15:36.997389+0000","last_active":"2024-04-24T23:16:18.229856+0000","last_peered":"2024-04-24T23:16:18.229856+0000","last_clean":"2024-04-24T23:16:18.229856+0000","last_became_active":"2024-04-24T23:15:36.995923+0000","last_became_peered":"2024-04-24T23:15:36.995923+0000","last_unstale":"2024-04-24T23:16:18.229856+0000","last_undegraded":"2024-04-24T23:16:18.229856+0000","last_fullsized":"2024-04-24T23:16:18.229856+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:55:47.134320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"26'5","reported_seq":31,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:25.274296+0000","last_change":"2024-04-24T23:15:35.957715+0000","last_active":"2024-04-24T23:16:25.274296+0000","last_peered":"2024-04-24T23:16:25.274296+0000","last_clean":"2024-04-24T23:16:25.274296+0000","last_became_active":"2024-04-24T23:15:35.957088+0000","last_became_peered":"2024-04-24T23:15:35.957088+0000","last_unstale":"2024-04-24T23:16:25.274296+0000","last_undegraded":"2024-04-24T23:16:25.274296+0000","last_fullsized":"2024-04-24T23:16:25.274296+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:30:34.483669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.223881+0000","last_change":"2024-04-24T23:15:41.008165+0000","last_active":"2024-04-24T23:16:18.223881+0000","last_peered":"2024-04-24T23:16:18.223881+0000","last_clean":"2024-04-24T23:16:18.223881+0000","last_became_active":"2024-04-24T23:15:41.007555+0000","last_became_peered":"2024-04-24T23:15:41.007555+0000","last_unstale":"2024-04-24T23:16:18.223881+0000","last_undegraded":"2024-04-24T23:16:18.223881+0000","last_fullsized":"2024-04-24T23:16:18.223881+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:18:08.195747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0,7],"acting":[6,4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.224736+0000","last_change":"2024-04-24T23:15:36.991382+0000","last_active":"2024-04-24T23:16:18.224736+0000","last_peered":"2024-04-24T23:16:18.224736+0000","last_clean":"2024-04-24T23:16:18.224736+0000","last_became_active":"2024-04-24T23:15:36.991115+0000","last_became_peered":"2024-04-24T23:15:36.991115+0000","last_unstale":"2024-04-24T23:16:18.224736+0000","last_undegraded":"2024-04-24T23:16:18.224736+0000","last_fullsized":"2024-04-24T23:16:18.224736+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:21:51.863947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"26'9","reported_seq":35,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:31.304710+0000","last_change":"2024-04-24T23:15:35.964245+0000","last_active":"2024-04-24T23:16:31.304710+0000","last_peered":"2024-04-24T23:16:31.304710+0000","last_clean":"2024-04-24T23:16:31.304710+0000","last_became_active":"2024-04-24T23:15:35.963907+0000","last_became_peered":"2024-04-24T23:15:35.963907+0000","last_unstale":"2024-04-24T23:16:31.304710+0000","last_undegraded":"2024-04-24T23:16:31.304710+0000","last_fullsized":"2024-04-24T23:16:31.304710+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:15:00.913899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.230757+0000","last_change":"2024-04-24T23:15:41.021981+0000","last_active":"2024-04-24T23:16:18.230757+0000","last_peered":"2024-04-24T23:16:18.230757+0000","last_clean":"2024-04-24T23:16:18.230757+0000","last_became_active":"2024-04-24T23:15:41.019193+0000","last_became_peered":"2024-04-24T23:15:41.019193+0000","last_unstale":"2024-04-24T23:16:18.230757+0000","last_undegraded":"2024-04-24T23:16:18.230757+0000","last_fullsized":"2024-04-24T23:16:18.230757+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:04:39.828502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5,6],"acting":[3,0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.232496+0000","last_change":"2024-04-24T23:15:36.986833+0000","last_active":"2024-04-24T23:16:18.232496+0000","last_peered":"2024-04-24T23:16:18.232496+0000","last_clean":"2024-04-24T23:16:18.232496+0000","last_became_active":"2024-04-24T23:15:36.986484+0000","last_became_peered":"2024-04-24T23:15:36.986484+0000","last_unstale":"2024-04-24T23:16:18.232496+0000","last_undegraded":"2024-04-24T23:16:18.232496+0000","last_fullsized":"2024-04-24T23:16:18.232496+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:12:32.245267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.14","version":"26'16","reported_seq":42,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:41.653552+0000","last_change":"2024-04-24T23:15:35.957124+0000","last_active":"2024-04-24T23:16:41.653552+0000","last_peered":"2024-04-24T23:16:41.653552+0000","last_clean":"2024-04-24T23:16:41.653552+0000","last_became_active":"2024-04-24T23:15:35.951923+0000","last_became_peered":"2024-04-24T23:15:35.951923+0000","last_unstale":"2024-04-24T23:16:41.653552+0000","last_undegraded":"2024-04-24T23:16:41.653552+0000","last_fullsized":"2024-04-24T23:16:41.653552+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:44:16.820640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4100,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.223976+0000","last_change":"2024-04-24T23:15:41.015494+0000","last_active":"2024-04-24T23:16:18.223976+0000","last_peered":"2024-04-24T23:16:18.223976+0000","last_clean":"2024-04-24T23:16:18.223976+0000","last_became_active":"2024-04-24T23:15:41.015179+0000","last_became_peered":"2024-04-24T23:15:41.015179+0000","last_unstale":"2024-04-24T23:16:18.223976+0000","last_undegraded":"2024-04-24T23:16:18.223976+0000","last_fullsized":"2024-04-24T23:16:18.223976+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:18:32.208506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5,0],"acting":[4,3,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.230201+0000","last_change":"2024-04-24T23:15:36.999334+0000","last_active":"2024-04-24T23:16:18.230201+0000","last_peered":"2024-04-24T23:16:18.230201+0000","last_clean":"2024-04-24T23:16:18.230201+0000","last_became_active":"2024-04-24T23:15:36.995470+0000","last_became_peered":"2024-04-24T23:15:36.995470+0000","last_unstale":"2024-04-24T23:16:18.230201+0000","last_undegraded":"2024-04-24T23:16:18.230201+0000","last_fullsized":"2024-04-24T23:16:18.230201+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:19:57.973402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.15","version":"26'5","reported_seq":31,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.292831+0000","last_change":"2024-04-24T23:15:35.946041+0000","last_active":"2024-04-24T23:16:18.292831+0000","last_peered":"2024-04-24T23:16:18.292831+0000","last_clean":"2024-04-24T23:16:18.292831+0000","last_became_active":"2024-04-24T23:15:35.944805+0000","last_became_peered":"2024-04-24T23:15:35.944805+0000","last_unstale":"2024-04-24T23:16:18.292831+0000","last_undegraded":"2024-04-24T23:16:18.292831+0000","last_fullsized":"2024-04-24T23:16:18.292831+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:52:19.261306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.293637+0000","last_change":"2024-04-24T23:15:41.024392+0000","last_active":"2024-04-24T23:16:18.293637+0000","last_peered":"2024-04-24T23:16:18.293637+0000","last_clean":"2024-04-24T23:16:18.293637+0000","last_became_active":"2024-04-24T23:15:41.023951+0000","last_became_peered":"2024-04-24T23:15:41.023951+0000","last_unstale":"2024-04-24T23:16:18.293637+0000","last_undegraded":"2024-04-24T23:16:18.293637+0000","last_fullsized":"2024-04-24T23:16:18.293637+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:24:17.810117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1,7],"acting":[2,3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.292208+0000","last_change":"2024-04-24T23:15:36.992243+0000","last_active":"2024-04-24T23:16:18.292208+0000","last_peered":"2024-04-24T23:16:18.292208+0000","last_clean":"2024-04-24T23:16:18.292208+0000","last_became_active":"2024-04-24T23:15:36.992002+0000","last_became_peered":"2024-04-24T23:15:36.992002+0000","last_unstale":"2024-04-24T23:16:18.292208+0000","last_undegraded":"2024-04-24T23:16:18.292208+0000","last_fullsized":"2024-04-24T23:16:18.292208+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:52:15.911009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"26'4","reported_seq":30,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:31.301673+0000","last_change":"2024-04-24T23:15:35.967408+0000","last_active":"2024-04-24T23:16:31.301673+0000","last_peered":"2024-04-24T23:16:31.301673+0000","last_clean":"2024-04-24T23:16:31.301673+0000","last_became_active":"2024-04-24T23:15:35.966646+0000","last_became_peered":"2024-04-24T23:15:35.966646+0000","last_unstale":"2024-04-24T23:16:31.301673+0000","last_undegraded":"2024-04-24T23:16:31.301673+0000","last_fullsized":"2024-04-24T23:16:31.301673+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:07:07.931773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.291779+0000","last_change":"2024-04-24T23:15:41.021697+0000","last_active":"2024-04-24T23:16:18.291779+0000","last_peered":"2024-04-24T23:16:18.291779+0000","last_clean":"2024-04-24T23:16:18.291779+0000","last_became_active":"2024-04-24T23:15:41.021281+0000","last_became_peered":"2024-04-24T23:15:41.021281+0000","last_unstale":"2024-04-24T23:16:18.291779+0000","last_undegraded":"2024-04-24T23:16:18.291779+0000","last_fullsized":"2024-04-24T23:16:18.291779+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:59:37.306593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3,2],"acting":[7,1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.294698+0000","last_change":"2024-04-24T23:15:36.990988+0000","last_active":"2024-04-24T23:16:18.294698+0000","last_peered":"2024-04-24T23:16:18.294698+0000","last_clean":"2024-04-24T23:16:18.294698+0000","last_became_active":"2024-04-24T23:15:36.990711+0000","last_became_peered":"2024-04-24T23:15:36.990711+0000","last_unstale":"2024-04-24T23:16:18.294698+0000","last_undegraded":"2024-04-24T23:16:18.294698+0000","last_fullsized":"2024-04-24T23:16:18.294698+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:33:20.663776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.17","version":"26'4","reported_seq":30,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.297684+0000","last_change":"2024-04-24T23:15:35.957838+0000","last_active":"2024-04-24T23:16:18.297684+0000","last_peered":"2024-04-24T23:16:18.297684+0000","last_clean":"2024-04-24T23:16:18.297684+0000","last_became_active":"2024-04-24T23:15:35.956870+0000","last_became_peered":"2024-04-24T23:15:35.956870+0000","last_unstale":"2024-04-24T23:16:18.297684+0000","last_undegraded":"2024-04-24T23:16:18.297684+0000","last_fullsized":"2024-04-24T23:16:18.297684+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:56:03.889693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.223850+0000","last_change":"2024-04-24T23:15:41.006098+0000","last_active":"2024-04-24T23:16:18.223850+0000","last_peered":"2024-04-24T23:16:18.223850+0000","last_clean":"2024-04-24T23:16:18.223850+0000","last_became_active":"2024-04-24T23:15:41.005578+0000","last_became_peered":"2024-04-24T23:15:41.005578+0000","last_unstale":"2024-04-24T23:16:18.223850+0000","last_undegraded":"2024-04-24T23:16:18.223850+0000","last_fullsized":"2024-04-24T23:16:18.223850+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:21:12.802112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3,7],"acting":[4,0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.224194+0000","last_change":"2024-04-24T23:15:41.022122+0000","last_active":"2024-04-24T23:16:18.224194+0000","last_peered":"2024-04-24T23:16:18.224194+0000","last_clean":"2024-04-24T23:16:18.224194+0000","last_became_active":"2024-04-24T23:15:41.021698+0000","last_became_peered":"2024-04-24T23:15:41.021698+0000","last_unstale":"2024-04-24T23:16:18.224194+0000","last_undegraded":"2024-04-24T23:16:18.224194+0000","last_fullsized":"2024-04-24T23:16:18.224194+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:13:56.730050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5,3],"acting":[4,1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.223853+0000","last_change":"2024-04-24T23:15:36.979816+0000","last_active":"2024-04-24T23:16:18.223853+0000","last_peered":"2024-04-24T23:16:18.223853+0000","last_clean":"2024-04-24T23:16:18.223853+0000","last_became_active":"2024-04-24T23:15:36.979553+0000","last_became_peered":"2024-04-24T23:15:36.979553+0000","last_unstale":"2024-04-24T23:16:18.223853+0000","last_undegraded":"2024-04-24T23:16:18.223853+0000","last_fullsized":"2024-04-24T23:16:18.223853+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:55:47.134320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.18","version":"26'3","reported_seq":29,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:25.272717+0000","last_change":"2024-04-24T23:15:35.964744+0000","last_active":"2024-04-24T23:16:25.272717+0000","last_peered":"2024-04-24T23:16:25.272717+0000","last_clean":"2024-04-24T23:16:25.272717+0000","last_became_active":"2024-04-24T23:15:35.964434+0000","last_became_peered":"2024-04-24T23:15:35.964434+0000","last_unstale":"2024-04-24T23:16:25.272717+0000","last_undegraded":"2024-04-24T23:16:25.272717+0000","last_fullsized":"2024-04-24T23:16:25.272717+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:26:32.587197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.231561+0000","last_change":"2024-04-24T23:15:41.022032+0000","last_active":"2024-04-24T23:16:18.231561+0000","last_peered":"2024-04-24T23:16:18.231561+0000","last_clean":"2024-04-24T23:16:18.231561+0000","last_became_active":"2024-04-24T23:15:41.021616+0000","last_became_peered":"2024-04-24T23:15:41.021616+0000","last_unstale":"2024-04-24T23:16:18.231561+0000","last_undegraded":"2024-04-24T23:16:18.231561+0000","last_fullsized":"2024-04-24T23:16:18.231561+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:08:59.242392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5,1],"acting":[3,2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.291442+0000","last_change":"2024-04-24T23:15:36.967198+0000","last_active":"2024-04-24T23:16:18.291442+0000","last_peered":"2024-04-24T23:16:18.291442+0000","last_clean":"2024-04-24T23:16:18.291442+0000","last_became_active":"2024-04-24T23:15:36.966910+0000","last_became_peered":"2024-04-24T23:15:36.966910+0000","last_unstale":"2024-04-24T23:16:18.291442+0000","last_undegraded":"2024-04-24T23:16:18.291442+0000","last_fullsized":"2024-04-24T23:16:18.291442+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:51:19.788437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"26'3","reported_seq":29,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:19.219459+0000","last_change":"2024-04-24T23:15:35.967169+0000","last_active":"2024-04-24T23:16:19.219459+0000","last_peered":"2024-04-24T23:16:19.219459+0000","last_clean":"2024-04-24T23:16:19.219459+0000","last_became_active":"2024-04-24T23:15:35.966939+0000","last_became_peered":"2024-04-24T23:15:35.966939+0000","last_unstale":"2024-04-24T23:16:19.219459+0000","last_undegraded":"2024-04-24T23:16:19.219459+0000","last_fullsized":"2024-04-24T23:16:19.219459+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:38:55.460569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"26'7","reported_seq":33,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:25.275257+0000","last_change":"2024-04-24T23:15:35.971236+0000","last_active":"2024-04-24T23:16:25.275257+0000","last_peered":"2024-04-24T23:16:25.275257+0000","last_clean":"2024-04-24T23:16:25.275257+0000","last_became_active":"2024-04-24T23:15:35.970908+0000","last_became_peered":"2024-04-24T23:15:35.970908+0000","last_unstale":"2024-04-24T23:16:25.275257+0000","last_undegraded":"2024-04-24T23:16:25.275257+0000","last_fullsized":"2024-04-24T23:16:25.275257+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:14:15.548024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.223928+0000","last_change":"2024-04-24T23:15:36.988156+0000","last_active":"2024-04-24T23:16:18.223928+0000","last_peered":"2024-04-24T23:16:18.223928+0000","last_clean":"2024-04-24T23:16:18.223928+0000","last_became_active":"2024-04-24T23:15:36.987896+0000","last_became_peered":"2024-04-24T23:15:36.987896+0000","last_unstale":"2024-04-24T23:16:18.223928+0000","last_undegraded":"2024-04-24T23:16:18.223928+0000","last_fullsized":"2024-04-24T23:16:18.223928+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:22:06.869644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.224383+0000","last_change":"2024-04-24T23:15:40.997559+0000","last_active":"2024-04-24T23:16:18.224383+0000","last_peered":"2024-04-24T23:16:18.224383+0000","last_clean":"2024-04-24T23:16:18.224383+0000","last_became_active":"2024-04-24T23:15:40.997102+0000","last_became_peered":"2024-04-24T23:15:40.997102+0000","last_unstale":"2024-04-24T23:16:18.224383+0000","last_undegraded":"2024-04-24T23:16:18.224383+0000","last_fullsized":"2024-04-24T23:16:18.224383+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:25:02.046761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,7],"acting":[4,6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"24'5","reported_seq":31,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.292037+0000","last_change":"2024-04-24T23:15:35.944842+0000","last_active":"2024-04-24T23:16:18.292037+0000","last_peered":"2024-04-24T23:16:18.292037+0000","last_clean":"2024-04-24T23:16:18.292037+0000","last_became_active":"2024-04-24T23:15:35.944563+0000","last_became_peered":"2024-04-24T23:15:35.944563+0000","last_unstale":"2024-04-24T23:16:18.292037+0000","last_undegraded":"2024-04-24T23:16:18.292037+0000","last_fullsized":"2024-04-24T23:16:18.292037+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:40:58.933384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.296474+0000","last_change":"2024-04-24T23:15:36.981103+0000","last_active":"2024-04-24T23:16:18.296474+0000","last_peered":"2024-04-24T23:16:18.296474+0000","last_clean":"2024-04-24T23:16:18.296474+0000","last_became_active":"2024-04-24T23:15:36.979637+0000","last_became_peered":"2024-04-24T23:15:36.979637+0000","last_unstale":"2024-04-24T23:16:18.296474+0000","last_undegraded":"2024-04-24T23:16:18.296474+0000","last_fullsized":"2024-04-24T23:16:18.296474+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:09:39.750272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.292643+0000","last_change":"2024-04-24T23:15:41.017087+0000","last_active":"2024-04-24T23:16:18.292643+0000","last_peered":"2024-04-24T23:16:18.292643+0000","last_clean":"2024-04-24T23:16:18.292643+0000","last_became_active":"2024-04-24T23:15:41.016696+0000","last_became_peered":"2024-04-24T23:15:41.016696+0000","last_unstale":"2024-04-24T23:16:18.292643+0000","last_undegraded":"2024-04-24T23:16:18.292643+0000","last_fullsized":"2024-04-24T23:16:18.292643+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:05:57.823447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1,0],"acting":[7,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.293973+0000","last_change":"2024-04-24T23:15:36.970211+0000","last_active":"2024-04-24T23:16:18.293973+0000","last_peered":"2024-04-24T23:16:18.293973+0000","last_clean":"2024-04-24T23:16:18.293973+0000","last_became_active":"2024-04-24T23:15:36.969090+0000","last_became_peered":"2024-04-24T23:15:36.969090+0000","last_unstale":"2024-04-24T23:16:18.293973+0000","last_undegraded":"2024-04-24T23:16:18.293973+0000","last_fullsized":"2024-04-24T23:16:18.293973+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:16:00.231012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.291726+0000","last_change":"2024-04-24T23:15:36.982528+0000","last_active":"2024-04-24T23:16:18.291726+0000","last_peered":"2024-04-24T23:16:18.291726+0000","last_clean":"2024-04-24T23:16:18.291726+0000","last_became_active":"2024-04-24T23:15:36.982299+0000","last_became_peered":"2024-04-24T23:15:36.982299+0000","last_unstale":"2024-04-24T23:16:18.291726+0000","last_undegraded":"2024-04-24T23:16:18.291726+0000","last_fullsized":"2024-04-24T23:16:18.291726+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:02:17.732324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.233323+0000","last_change":"2024-04-24T23:15:36.989899+0000","last_active":"2024-04-24T23:16:18.233323+0000","last_peered":"2024-04-24T23:16:18.233323+0000","last_clean":"2024-04-24T23:16:18.233323+0000","last_became_active":"2024-04-24T23:15:36.989557+0000","last_became_peered":"2024-04-24T23:15:36.989557+0000","last_unstale":"2024-04-24T23:16:18.233323+0000","last_undegraded":"2024-04-24T23:16:18.233323+0000","last_fullsized":"2024-04-24T23:16:18.233323+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:15:13.513341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.291922+0000","last_change":"2024-04-24T23:15:36.969628+0000","last_active":"2024-04-24T23:16:18.291922+0000","last_peered":"2024-04-24T23:16:18.291922+0000","last_clean":"2024-04-24T23:16:18.291922+0000","last_became_active":"2024-04-24T23:15:36.969327+0000","last_became_peered":"2024-04-24T23:15:36.969327+0000","last_unstale":"2024-04-24T23:16:18.291922+0000","last_undegraded":"2024-04-24T23:16:18.291922+0000","last_fullsized":"2024-04-24T23:16:18.291922+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:08:47.021999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.224974+0000","last_change":"2024-04-24T23:15:36.986343+0000","last_active":"2024-04-24T23:16:18.224974+0000","last_peered":"2024-04-24T23:16:18.224974+0000","last_clean":"2024-04-24T23:16:18.224974+0000","last_became_active":"2024-04-24T23:15:36.986026+0000","last_became_peered":"2024-04-24T23:15:36.986026+0000","last_unstale":"2024-04-24T23:16:18.224974+0000","last_undegraded":"2024-04-24T23:16:18.224974+0000","last_fullsized":"2024-04-24T23:16:18.224974+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:10:00.347826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.224857+0000","last_change":"2024-04-24T23:15:36.968538+0000","last_active":"2024-04-24T23:16:18.224857+0000","last_peered":"2024-04-24T23:16:18.224857+0000","last_clean":"2024-04-24T23:16:18.224857+0000","last_became_active":"2024-04-24T23:15:36.968263+0000","last_became_peered":"2024-04-24T23:15:36.968263+0000","last_unstale":"2024-04-24T23:16:18.224857+0000","last_undegraded":"2024-04-24T23:16:18.224857+0000","last_fullsized":"2024-04-24T23:16:18.224857+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:07:43.731738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.294600+0000","last_change":"2024-04-24T23:15:36.991008+0000","last_active":"2024-04-24T23:16:18.294600+0000","last_peered":"2024-04-24T23:16:18.294600+0000","last_clean":"2024-04-24T23:16:18.294600+0000","last_became_active":"2024-04-24T23:15:36.990764+0000","last_became_peered":"2024-04-24T23:15:36.990764+0000","last_unstale":"2024-04-24T23:16:18.294600+0000","last_undegraded":"2024-04-24T23:16:18.294600+0000","last_fullsized":"2024-04-24T23:16:18.294600+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:50:21.865233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.291911+0000","last_change":"2024-04-24T23:15:36.993550+0000","last_active":"2024-04-24T23:16:18.291911+0000","last_peered":"2024-04-24T23:16:18.291911+0000","last_clean":"2024-04-24T23:16:18.291911+0000","last_became_active":"2024-04-24T23:15:36.993310+0000","last_became_peered":"2024-04-24T23:15:36.993310+0000","last_unstale":"2024-04-24T23:16:18.291911+0000","last_undegraded":"2024-04-24T23:16:18.291911+0000","last_fullsized":"2024-04-24T23:16:18.291911+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:21:20.617940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.230900+0000","last_change":"2024-04-24T23:15:36.987810+0000","last_active":"2024-04-24T23:16:18.230900+0000","last_peered":"2024-04-24T23:16:18.230900+0000","last_clean":"2024-04-24T23:16:18.230900+0000","last_became_active":"2024-04-24T23:15:36.987540+0000","last_became_peered":"2024-04-24T23:15:36.987540+0000","last_unstale":"2024-04-24T23:16:18.230900+0000","last_undegraded":"2024-04-24T23:16:18.230900+0000","last_fullsized":"2024-04-24T23:16:18.230900+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:49:56.176021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.291530+0000","last_change":"2024-04-24T23:15:36.981187+0000","last_active":"2024-04-24T23:16:18.291530+0000","last_peered":"2024-04-24T23:16:18.291530+0000","last_clean":"2024-04-24T23:16:18.291530+0000","last_became_active":"2024-04-24T23:15:36.980838+0000","last_became_peered":"2024-04-24T23:15:36.980838+0000","last_unstale":"2024-04-24T23:16:18.291530+0000","last_undegraded":"2024-04-24T23:16:18.291530+0000","last_fullsized":"2024-04-24T23:16:18.291530+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:57:41.178367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.296378+0000","last_change":"2024-04-24T23:15:36.999238+0000","last_active":"2024-04-24T23:16:18.296378+0000","last_peered":"2024-04-24T23:16:18.296378+0000","last_clean":"2024-04-24T23:16:18.296378+0000","last_became_active":"2024-04-24T23:15:36.996245+0000","last_became_peered":"2024-04-24T23:15:36.996245+0000","last_unstale":"2024-04-24T23:16:18.296378+0000","last_undegraded":"2024-04-24T23:16:18.296378+0000","last_fullsized":"2024-04-24T23:16:18.296378+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:21:20.443640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.223856+0000","last_change":"2024-04-24T23:15:36.981219+0000","last_active":"2024-04-24T23:16:18.223856+0000","last_peered":"2024-04-24T23:16:18.223856+0000","last_clean":"2024-04-24T23:16:18.223856+0000","last_became_active":"2024-04-24T23:15:36.980911+0000","last_became_peered":"2024-04-24T23:15:36.980911+0000","last_unstale":"2024-04-24T23:16:18.223856+0000","last_undegraded":"2024-04-24T23:16:18.223856+0000","last_fullsized":"2024-04-24T23:16:18.223856+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:19:57.973402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.229789+0000","last_change":"2024-04-24T23:15:36.991418+0000","last_active":"2024-04-24T23:16:18.229789+0000","last_peered":"2024-04-24T23:16:18.229789+0000","last_clean":"2024-04-24T23:16:18.229789+0000","last_became_active":"2024-04-24T23:15:36.989792+0000","last_became_peered":"2024-04-24T23:15:36.989792+0000","last_unstale":"2024-04-24T23:16:18.229789+0000","last_undegraded":"2024-04-24T23:16:18.229789+0000","last_fullsized":"2024-04-24T23:16:18.229789+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:02:53.601311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.231144+0000","last_change":"2024-04-24T23:15:36.987810+0000","last_active":"2024-04-24T23:16:18.231144+0000","last_peered":"2024-04-24T23:16:18.231144+0000","last_clean":"2024-04-24T23:16:18.231144+0000","last_became_active":"2024-04-24T23:15:36.987494+0000","last_became_peered":"2024-04-24T23:15:36.987494+0000","last_unstale":"2024-04-24T23:16:18.231144+0000","last_undegraded":"2024-04-24T23:16:18.231144+0000","last_fullsized":"2024-04-24T23:16:18.231144+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:43:21.141172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.292884+0000","last_change":"2024-04-24T23:15:36.986576+0000","last_active":"2024-04-24T23:16:18.292884+0000","last_peered":"2024-04-24T23:16:18.292884+0000","last_clean":"2024-04-24T23:16:18.292884+0000","last_became_active":"2024-04-24T23:15:36.986272+0000","last_became_peered":"2024-04-24T23:15:36.986272+0000","last_unstale":"2024-04-24T23:16:18.292884+0000","last_undegraded":"2024-04-24T23:16:18.292884+0000","last_fullsized":"2024-04-24T23:16:18.292884+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:55:08.894945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":29,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.294151+0000","last_change":"2024-04-24T23:16:18.294151+0000","last_active":"2024-04-24T23:16:18.294151+0000","last_peered":"2024-04-24T23:16:18.294151+0000","last_clean":"2024-04-24T23:16:18.294151+0000","last_became_active":"2024-04-24T23:16:18.293579+0000","last_became_peered":"2024-04-24T23:16:18.293579+0000","last_unstale":"2024-04-24T23:16:18.294151+0000","last_undegraded":"2024-04-24T23:16:18.294151+0000","last_fullsized":"2024-04-24T23:16:18.294151+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:22:20.603555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.230543+0000","last_change":"2024-04-24T23:15:36.996521+0000","last_active":"2024-04-24T23:16:18.230543+0000","last_peered":"2024-04-24T23:16:18.230543+0000","last_clean":"2024-04-24T23:16:18.230543+0000","last_became_active":"2024-04-24T23:15:36.995507+0000","last_became_peered":"2024-04-24T23:15:36.995507+0000","last_unstale":"2024-04-24T23:16:18.230543+0000","last_undegraded":"2024-04-24T23:16:18.230543+0000","last_fullsized":"2024-04-24T23:16:18.230543+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:58:50.153471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.291317+0000","last_change":"2024-04-24T23:15:36.984994+0000","last_active":"2024-04-24T23:16:18.291317+0000","last_peered":"2024-04-24T23:16:18.291317+0000","last_clean":"2024-04-24T23:16:18.291317+0000","last_became_active":"2024-04-24T23:15:36.984741+0000","last_became_peered":"2024-04-24T23:15:36.984741+0000","last_unstale":"2024-04-24T23:16:18.291317+0000","last_undegraded":"2024-04-24T23:16:18.291317+0000","last_fullsized":"2024-04-24T23:16:18.291317+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:45:23.429836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.224730+0000","last_change":"2024-04-24T23:15:36.966165+0000","last_active":"2024-04-24T23:16:18.224730+0000","last_peered":"2024-04-24T23:16:18.224730+0000","last_clean":"2024-04-24T23:16:18.224730+0000","last_became_active":"2024-04-24T23:15:36.965877+0000","last_became_peered":"2024-04-24T23:15:36.965877+0000","last_unstale":"2024-04-24T23:16:18.224730+0000","last_undegraded":"2024-04-24T23:16:18.224730+0000","last_fullsized":"2024-04-24T23:16:18.224730+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:30:12.390978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.224249+0000","last_change":"2024-04-24T23:15:36.990667+0000","last_active":"2024-04-24T23:16:18.224249+0000","last_peered":"2024-04-24T23:16:18.224249+0000","last_clean":"2024-04-24T23:16:18.224249+0000","last_became_active":"2024-04-24T23:15:36.990383+0000","last_became_peered":"2024-04-24T23:15:36.990383+0000","last_unstale":"2024-04-24T23:16:18.224249+0000","last_undegraded":"2024-04-24T23:16:18.224249+0000","last_fullsized":"2024-04-24T23:16:18.224249+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:36:43.850046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.229311+0000","last_change":"2024-04-24T23:15:36.991686+0000","last_active":"2024-04-24T23:16:18.229311+0000","last_peered":"2024-04-24T23:16:18.229311+0000","last_clean":"2024-04-24T23:16:18.229311+0000","last_became_active":"2024-04-24T23:15:36.990155+0000","last_became_peered":"2024-04-24T23:15:36.990155+0000","last_unstale":"2024-04-24T23:16:18.229311+0000","last_undegraded":"2024-04-24T23:16:18.229311+0000","last_fullsized":"2024-04-24T23:16:18.229311+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:00:52.424019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.232389+0000","last_change":"2024-04-24T23:15:36.992958+0000","last_active":"2024-04-24T23:16:18.232389+0000","last_peered":"2024-04-24T23:16:18.232389+0000","last_clean":"2024-04-24T23:16:18.232389+0000","last_became_active":"2024-04-24T23:15:36.992608+0000","last_became_peered":"2024-04-24T23:15:36.992608+0000","last_unstale":"2024-04-24T23:16:18.232389+0000","last_undegraded":"2024-04-24T23:16:18.232389+0000","last_fullsized":"2024-04-24T23:16:18.232389+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:15:30.416763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.232332+0000","last_change":"2024-04-24T23:15:36.992943+0000","last_active":"2024-04-24T23:16:18.232332+0000","last_peered":"2024-04-24T23:16:18.232332+0000","last_clean":"2024-04-24T23:16:18.232332+0000","last_became_active":"2024-04-24T23:15:36.992567+0000","last_became_peered":"2024-04-24T23:15:36.992567+0000","last_unstale":"2024-04-24T23:16:18.232332+0000","last_undegraded":"2024-04-24T23:16:18.232332+0000","last_fullsized":"2024-04-24T23:16:18.232332+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:50:43.209232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.297503+0000","last_change":"2024-04-24T23:15:36.999697+0000","last_active":"2024-04-24T23:16:18.297503+0000","last_peered":"2024-04-24T23:16:18.297503+0000","last_clean":"2024-04-24T23:16:18.297503+0000","last_became_active":"2024-04-24T23:15:36.999413+0000","last_became_peered":"2024-04-24T23:15:36.999413+0000","last_unstale":"2024-04-24T23:16:18.297503+0000","last_undegraded":"2024-04-24T23:16:18.297503+0000","last_fullsized":"2024-04-24T23:16:18.297503+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:25:42.856471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.291309+0000","last_change":"2024-04-24T23:15:36.986742+0000","last_active":"2024-04-24T23:16:18.291309+0000","last_peered":"2024-04-24T23:16:18.291309+0000","last_clean":"2024-04-24T23:16:18.291309+0000","last_became_active":"2024-04-24T23:15:36.986013+0000","last_became_peered":"2024-04-24T23:15:36.986013+0000","last_unstale":"2024-04-24T23:16:18.291309+0000","last_undegraded":"2024-04-24T23:16:18.291309+0000","last_fullsized":"2024-04-24T23:16:18.291309+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:16:15.203151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.234253+0000","last_change":"2024-04-24T23:15:36.984336+0000","last_active":"2024-04-24T23:16:18.234253+0000","last_peered":"2024-04-24T23:16:18.234253+0000","last_clean":"2024-04-24T23:16:18.234253+0000","last_became_active":"2024-04-24T23:15:36.983966+0000","last_became_peered":"2024-04-24T23:15:36.983966+0000","last_unstale":"2024-04-24T23:16:18.234253+0000","last_undegraded":"2024-04-24T23:16:18.234253+0000","last_fullsized":"2024-04-24T23:16:18.234253+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:54:16.319608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.294392+0000","last_change":"2024-04-24T23:15:36.979079+0000","last_active":"2024-04-24T23:16:18.294392+0000","last_peered":"2024-04-24T23:16:18.294392+0000","last_clean":"2024-04-24T23:16:18.294392+0000","last_became_active":"2024-04-24T23:15:36.978706+0000","last_became_peered":"2024-04-24T23:15:36.978706+0000","last_unstale":"2024-04-24T23:16:18.294392+0000","last_undegraded":"2024-04-24T23:16:18.294392+0000","last_fullsized":"2024-04-24T23:16:18.294392+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:53:39.888150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.229376+0000","last_change":"2024-04-24T23:15:36.986008+0000","last_active":"2024-04-24T23:16:18.229376+0000","last_peered":"2024-04-24T23:16:18.229376+0000","last_clean":"2024-04-24T23:16:18.229376+0000","last_became_active":"2024-04-24T23:15:36.985552+0000","last_became_peered":"2024-04-24T23:15:36.985552+0000","last_unstale":"2024-04-24T23:16:18.229376+0000","last_undegraded":"2024-04-24T23:16:18.229376+0000","last_fullsized":"2024-04-24T23:16:18.229376+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:36:50.827503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.293789+0000","last_change":"2024-04-24T23:15:36.978673+0000","last_active":"2024-04-24T23:16:18.293789+0000","last_peered":"2024-04-24T23:16:18.293789+0000","last_clean":"2024-04-24T23:16:18.293789+0000","last_became_active":"2024-04-24T23:15:36.972744+0000","last_became_peered":"2024-04-24T23:15:36.972744+0000","last_unstale":"2024-04-24T23:16:18.293789+0000","last_undegraded":"2024-04-24T23:16:18.293789+0000","last_fullsized":"2024-04-24T23:16:18.293789+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:28:57.028825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.291690+0000","last_change":"2024-04-24T23:15:36.970238+0000","last_active":"2024-04-24T23:16:18.291690+0000","last_peered":"2024-04-24T23:16:18.291690+0000","last_clean":"2024-04-24T23:16:18.291690+0000","last_became_active":"2024-04-24T23:15:36.969751+0000","last_became_peered":"2024-04-24T23:15:36.969751+0000","last_unstale":"2024-04-24T23:16:18.291690+0000","last_undegraded":"2024-04-24T23:16:18.291690+0000","last_fullsized":"2024-04-24T23:16:18.291690+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:19:56.456801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.291790+0000","last_change":"2024-04-24T23:15:36.968825+0000","last_active":"2024-04-24T23:16:18.291790+0000","last_peered":"2024-04-24T23:16:18.291790+0000","last_clean":"2024-04-24T23:16:18.291790+0000","last_became_active":"2024-04-24T23:15:36.968543+0000","last_became_peered":"2024-04-24T23:15:36.968543+0000","last_unstale":"2024-04-24T23:16:18.291790+0000","last_undegraded":"2024-04-24T23:16:18.291790+0000","last_fullsized":"2024-04-24T23:16:18.291790+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:36:48.795062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.229059+0000","last_change":"2024-04-24T23:15:36.992800+0000","last_active":"2024-04-24T23:16:18.229059+0000","last_peered":"2024-04-24T23:16:18.229059+0000","last_clean":"2024-04-24T23:16:18.229059+0000","last_became_active":"2024-04-24T23:15:36.992476+0000","last_became_peered":"2024-04-24T23:15:36.992476+0000","last_unstale":"2024-04-24T23:16:18.229059+0000","last_undegraded":"2024-04-24T23:16:18.229059+0000","last_fullsized":"2024-04-24T23:16:18.229059+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:01:21.103559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":10522,"num_objects":98,"num_object_clones":0,"num_object_copies":196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":211,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":21044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":185,"ondisk_log_size":185,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574868,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27324,"kb_used_data":308,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344516,"statfs":{"total":96636764160,"available":96608784384,"internally_reserved":0,"allocated":315392,"data_stored":114972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574868,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":292,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":299008,"data_stored":116298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574868,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27892,"kb_used_data":876,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343948,"statfs":{"total":96636764160,"available":96608202752,"internally_reserved":0,"allocated":897024,"data_stored":705112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574869,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27344,"kb_used_data":328,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344496,"statfs":{"total":96636764160,"available":96608763904,"internally_reserved":0,"allocated":335872,"data_stored":119856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574869,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27332,"kb_used_data":316,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344508,"statfs":{"total":96636764160,"available":96608776192,"internally_reserved":0,"allocated":323584,"data_stored":116640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574868,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":292,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":299008,"data_stored":118086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574867,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27888,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343952,"statfs":{"total":96636764160,"available":96608206848,"internally_reserved":0,"allocated":892928,"data_stored":705700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574868,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27324,"kb_used_data":308,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344516,"statfs":{"total":96636764160,"available":96608784384,"internally_reserved":0,"allocated":315392,"data_stored":116460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2542,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":5938,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1054,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T23:16:48.680 INFO:tasks.mds_thrash.ceph_manager:clean! 2024-04-24T23:16:48.680 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T23:16:48.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:48.927+0000 7f395be00640 1 Processor -- start 2024-04-24T23:16:48.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:48.927+0000 7f395be00640 1 -- start start 2024-04-24T23:16:48.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:48.928+0000 7f395be00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f395c0cb090 0x7f395c0d0820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:48.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:48.928+0000 7f395be00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f395c155170 con 0x7f395c0d0d60 2024-04-24T23:16:48.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:48.928+0000 7f395be00640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f395c0cc9a0 con 0x7f395c154d90 2024-04-24T23:16:48.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:48.928+0000 7f395be00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f395c15ad60 con 0x7f395c0cb090 2024-04-24T23:16:48.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:48.929+0000 7f395a000640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f395c0cb090 0x7f395c0d0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:48.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:48.929+0000 7f395aa00640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f395c154d90 0x7f395c0cbe80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.6:36916/0 (socket says 172.21.15.6:36916) 2024-04-24T23:16:48.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:48.929+0000 7f395aa00640 1 -- 172.21.15.6:0/3916668777 learned_addr learned my addr 172.21.15.6:0/3916668777 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:16:48.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:48.929+0000 7f3959600640 1 -- 172.21.15.6:0/3916668777 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2889238499 0 0) 0x7f395c0cc9a0 con 0x7f395c154d90 2024-04-24T23:16:48.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:48.929+0000 7f3959600640 1 -- 172.21.15.6:0/3916668777 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3940003520 con 0x7f395c154d90 2024-04-24T23:16:48.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:48.929+0000 7f395a000640 1 -- 172.21.15.6:0/3916668777 >> v1:172.21.15.190:6789/0 conn(0x7f395c154d90 legacy=0x7f395c0cbe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:48.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:48.929+0000 7f395a000640 1 -- 172.21.15.6:0/3916668777 >> v1:172.21.15.6:6789/0 conn(0x7f395c0d0d60 legacy=0x7f395c15e000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:48.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:48.929+0000 7f395a000640 1 -- 172.21.15.6:0/3916668777 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f395c15b040 con 0x7f395c0cb090 2024-04-24T23:16:48.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:48.930+0000 7f395a000640 1 --2- 172.21.15.6:0/3916668777 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f395c0cb090 0x7f395c0d0820 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f3948009dc0 tx=0x7f394802a760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d4061758e7b8b4b3 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:48.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:48.931+0000 7f3959600640 1 -- 172.21.15.6:0/3916668777 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3948007740 con 0x7f395c0cb090 2024-04-24T23:16:48.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:48.931+0000 7f3959600640 1 -- 172.21.15.6:0/3916668777 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f39480078e0 con 0x7f395c0cb090 2024-04-24T23:16:48.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:48.931+0000 7f3959600640 1 -- 172.21.15.6:0/3916668777 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3948007cb0 con 0x7f395c0cb090 2024-04-24T23:16:48.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:48.931+0000 7f395be00640 1 -- 172.21.15.6:0/3916668777 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f395c0cb090 msgr2=0x7f395c0d0820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:48.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:48.931+0000 7f395be00640 1 --2- 172.21.15.6:0/3916668777 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f395c0cb090 0x7f395c0d0820 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f3948009dc0 tx=0x7f394802a760 comp rx=0 tx=0).stop 2024-04-24T23:16:48.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:48.931+0000 7f395be00640 1 -- 172.21.15.6:0/3916668777 shutdown_connections 2024-04-24T23:16:48.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:48.931+0000 7f395be00640 1 --2- 172.21.15.6:0/3916668777 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f395c0cb090 0x7f395c0d0820 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:48.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:48.931+0000 7f395be00640 1 -- 172.21.15.6:0/3916668777 >> 172.21.15.6:0/3916668777 conn(0x7f395c06be10 msgr2=0x7f395c06c1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:48.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:48.931+0000 7f395be00640 1 -- 172.21.15.6:0/3916668777 shutdown_connections 2024-04-24T23:16:48.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:48.931+0000 7f395be00640 1 -- 172.21.15.6:0/3916668777 wait complete. 2024-04-24T23:16:48.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:48.933+0000 7f395be00640 1 Processor -- start 2024-04-24T23:16:48.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:48.933+0000 7f395be00640 1 -- start start 2024-04-24T23:16:48.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:48.934+0000 7f395be00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f395c0cb090 0x7f395c15a190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:48.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:48.934+0000 7f395be00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f395c0d0d60 0x7f395c1f3fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:48.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:48.934+0000 7f395be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f395c154d90 0x7f395c158d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:48.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:48.934+0000 7f395be00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f395c15aa30 con 0x7f395c154d90 2024-04-24T23:16:48.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:48.934+0000 7f395be00640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f395c15b600 con 0x7f395c0d0d60 2024-04-24T23:16:48.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:48.934+0000 7f395be00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f395c0cc590 con 0x7f395c0cb090 2024-04-24T23:16:48.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:48.934+0000 7f395aa00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f395c0cb090 0x7f395c15a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:48.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:48.934+0000 7f395aa00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f395c0cb090 0x7f395c15a190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.6:59786/0 (socket says 172.21.15.6:59786) 2024-04-24T23:16:48.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:48.934+0000 7f395aa00640 1 -- 172.21.15.6:0/282327901 learned_addr learned my addr 172.21.15.6:0/282327901 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:16:48.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:48.935+0000 7f395a000640 1 --2- 172.21.15.6:0/282327901 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f395c0d0d60 0x7f395c1f3fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:48.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:48.935+0000 7f395b400640 1 --2- 172.21.15.6:0/282327901 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f395c154d90 0x7f395c158d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:48.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:48.935+0000 7f395aa00640 1 -- 172.21.15.6:0/282327901 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f395c0d0d60 msgr2=0x7f395c1f3fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:48.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:48.935+0000 7f395aa00640 1 --2- 172.21.15.6:0/282327901 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f395c0d0d60 0x7f395c1f3fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:48.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:48.935+0000 7f395aa00640 1 -- 172.21.15.6:0/282327901 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f395c154d90 msgr2=0x7f395c158d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:48.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:48.935+0000 7f395aa00640 1 --2- 172.21.15.6:0/282327901 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f395c154d90 0x7f395c158d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:48.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:48.935+0000 7f395aa00640 1 -- 172.21.15.6:0/282327901 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f395c15a840 con 0x7f395c0cb090 2024-04-24T23:16:48.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:48.935+0000 7f395aa00640 1 --2- 172.21.15.6:0/282327901 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f395c0cb090 0x7f395c15a190 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f394c00c440 tx=0x7f394c00c900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:48.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:48.936+0000 7f3952a00640 1 -- 172.21.15.6:0/282327901 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f394c017070 con 0x7f395c0cb090 2024-04-24T23:16:48.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:48.936+0000 7f3952a00640 1 -- 172.21.15.6:0/282327901 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f394c005ce0 con 0x7f395c0cb090 2024-04-24T23:16:48.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:48.936+0000 7f395be00640 1 -- 172.21.15.6:0/282327901 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f395c1524b0 con 0x7f395c0cb090 2024-04-24T23:16:48.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:48.936+0000 7f3952a00640 1 -- 172.21.15.6:0/282327901 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f394c0072b0 con 0x7f395c0cb090 2024-04-24T23:16:48.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:48.936+0000 7f395be00640 1 -- 172.21.15.6:0/282327901 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f395c1f37b0 con 0x7f395c0cb090 2024-04-24T23:16:48.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:48.937+0000 7f395be00640 1 -- 172.21.15.6:0/282327901 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f395c075b30 con 0x7f395c0cb090 2024-04-24T23:16:48.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:48.938+0000 7f3952a00640 1 -- 172.21.15.6:0/282327901 <== mon.2 v2:172.21.15.190:3301/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f394c01a070 con 0x7f395c0cb090 2024-04-24T23:16:48.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:48.939+0000 7f3952a00640 1 --2- 172.21.15.6:0/282327901 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f393c06be30 0x7f393c06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:48.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:48.939+0000 7f3952a00640 1 -- 172.21.15.6:0/282327901 <== mon.2 v2:172.21.15.190:3301/0 5 ==== osd_map(26..26 src has 1..26) ==== 6590+0+0 (secure 0 0 0) 0x7f394c08a4d0 con 0x7f395c0cb090 2024-04-24T23:16:48.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:48.939+0000 7f395a000640 1 --2- 172.21.15.6:0/282327901 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f393c06be30 0x7f393c06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:48.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:48.940+0000 7f395a000640 1 --2- 172.21.15.6:0/282327901 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f393c06be30 0x7f393c06e2f0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f3948034000 tx=0x7f394802f000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:48.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:48.943+0000 7f3952a00640 1 -- 172.21.15.6:0/282327901 <== mon.2 v2:172.21.15.190:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f394c0584e0 con 0x7f395c0cb090 2024-04-24T23:16:49.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:49.151+0000 7f395be00640 1 -- 172.21.15.6:0/282327901 --> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f395c074ec0 con 0x7f393c06be30 2024-04-24T23:16:49.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:49.161+0000 7f3952a00640 1 -- 172.21.15.6:0/282327901 <== mgr.4105 v2:172.21.15.190:6832/806769663 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+332650 (secure 0 0 0) 0x7f395c074ec0 con 0x7f393c06be30 2024-04-24T23:16:49.168 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:16:49.170 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-04-24T23:16:49.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:49.168+0000 7f395be00640 1 -- 172.21.15.6:0/282327901 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f393c06be30 msgr2=0x7f393c06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:49.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:49.168+0000 7f395be00640 1 --2- 172.21.15.6:0/282327901 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f393c06be30 0x7f393c06e2f0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f3948034000 tx=0x7f394802f000 comp rx=0 tx=0).stop 2024-04-24T23:16:49.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:49.168+0000 7f395be00640 1 -- 172.21.15.6:0/282327901 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f395c0cb090 msgr2=0x7f395c15a190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:49.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:49.168+0000 7f395be00640 1 --2- 172.21.15.6:0/282327901 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f395c0cb090 0x7f395c15a190 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f394c00c440 tx=0x7f394c00c900 comp rx=0 tx=0).stop 2024-04-24T23:16:49.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:49.168+0000 7f395be00640 1 -- 172.21.15.6:0/282327901 shutdown_connections 2024-04-24T23:16:49.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:49.168+0000 7f395be00640 1 --2- 172.21.15.6:0/282327901 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f393c06be30 0x7f393c06e2f0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:49.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:49.168+0000 7f395be00640 1 --2- 172.21.15.6:0/282327901 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f395c154d90 0x7f395c158d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:49.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:49.168+0000 7f395be00640 1 --2- 172.21.15.6:0/282327901 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f395c0d0d60 0x7f395c1f3fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:49.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:49.168+0000 7f395be00640 1 --2- 172.21.15.6:0/282327901 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f395c0cb090 0x7f395c15a190 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:49.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:49.168+0000 7f395be00640 1 -- 172.21.15.6:0/282327901 >> 172.21.15.6:0/282327901 conn(0x7f395c06be10 msgr2=0x7f395c06c1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:49.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:49.168+0000 7f395be00640 1 -- 172.21.15.6:0/282327901 shutdown_connections 2024-04-24T23:16:49.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:49.168+0000 7f395be00640 1 -- 172.21.15.6:0/282327901 wait complete. 2024-04-24T23:16:49.189 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":61,"stamp":"2024-04-24T23:16:48.699017+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":600890,"num_objects":100,"num_object_clones":0,"num_object_copies":200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":370,"num_write_kb":2485,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":302,"ondisk_log_size":302,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219720,"kb_used_data":3592,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755000,"statfs":{"total":773094113280,"available":772869120000,"internally_reserved":0,"allocated":3678208,"data_stored":2113124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":258,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003499"},"pg_stats":[{"pgid":"2.1f","version":"26'13","reported_seq":39,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:31.303006+0000","last_change":"2024-04-24T23:15:35.964646+0000","last_active":"2024-04-24T23:16:31.303006+0000","last_peered":"2024-04-24T23:16:31.303006+0000","last_clean":"2024-04-24T23:16:31.303006+0000","last_became_active":"2024-04-24T23:15:35.964357+0000","last_became_peered":"2024-04-24T23:15:35.964357+0000","last_unstale":"2024-04-24T23:16:31.303006+0000","last_undegraded":"2024-04-24T23:16:31.303006+0000","last_fullsized":"2024-04-24T23:16:31.303006+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:01:17.912993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.292135+0000","last_change":"2024-04-24T23:15:36.989772+0000","last_active":"2024-04-24T23:16:18.292135+0000","last_peered":"2024-04-24T23:16:18.292135+0000","last_clean":"2024-04-24T23:16:18.292135+0000","last_became_active":"2024-04-24T23:15:36.989543+0000","last_became_peered":"2024-04-24T23:15:36.989543+0000","last_unstale":"2024-04-24T23:16:18.292135+0000","last_undegraded":"2024-04-24T23:16:18.292135+0000","last_fullsized":"2024-04-24T23:16:18.292135+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:37:16.737354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.303989+0000","last_change":"2024-04-24T23:16:18.300096+0000","last_active":"2024-04-24T23:16:18.303989+0000","last_peered":"2024-04-24T23:16:18.303989+0000","last_clean":"2024-04-24T23:16:18.303989+0000","last_became_active":"2024-04-24T23:16:18.299217+0000","last_became_peered":"2024-04-24T23:16:18.299217+0000","last_unstale":"2024-04-24T23:16:18.303989+0000","last_undegraded":"2024-04-24T23:16:18.303989+0000","last_fullsized":"2024-04-24T23:16:18.303989+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:08:19.426862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4,1],"acting":[0,6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"26'2","reported_seq":28,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.292303+0000","last_change":"2024-04-24T23:15:35.946083+0000","last_active":"2024-04-24T23:16:18.292303+0000","last_peered":"2024-04-24T23:16:18.292303+0000","last_clean":"2024-04-24T23:16:18.292303+0000","last_became_active":"2024-04-24T23:15:35.945227+0000","last_became_peered":"2024-04-24T23:15:35.945227+0000","last_unstale":"2024-04-24T23:16:18.292303+0000","last_undegraded":"2024-04-24T23:16:18.292303+0000","last_fullsized":"2024-04-24T23:16:18.292303+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:05:17.727725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.292034+0000","last_change":"2024-04-24T23:15:36.998245+0000","last_active":"2024-04-24T23:16:18.292034+0000","last_peered":"2024-04-24T23:16:18.292034+0000","last_clean":"2024-04-24T23:16:18.292034+0000","last_became_active":"2024-04-24T23:15:36.998021+0000","last_became_peered":"2024-04-24T23:15:36.998021+0000","last_unstale":"2024-04-24T23:16:18.292034+0000","last_undegraded":"2024-04-24T23:16:18.292034+0000","last_fullsized":"2024-04-24T23:16:18.292034+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:03:42.359117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.293518+0000","last_change":"2024-04-24T23:15:41.008313+0000","last_active":"2024-04-24T23:16:18.293518+0000","last_peered":"2024-04-24T23:16:18.293518+0000","last_clean":"2024-04-24T23:16:18.293518+0000","last_became_active":"2024-04-24T23:15:41.007908+0000","last_became_peered":"2024-04-24T23:15:41.007908+0000","last_unstale":"2024-04-24T23:16:18.293518+0000","last_undegraded":"2024-04-24T23:16:18.293518+0000","last_fullsized":"2024-04-24T23:16:18.293518+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:02:11.902575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4,6],"acting":[2,7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1d","version":"26'12","reported_seq":38,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:31.305540+0000","last_change":"2024-04-24T23:15:35.968312+0000","last_active":"2024-04-24T23:16:31.305540+0000","last_peered":"2024-04-24T23:16:31.305540+0000","last_clean":"2024-04-24T23:16:31.305540+0000","last_became_active":"2024-04-24T23:15:35.967873+0000","last_became_peered":"2024-04-24T23:15:35.967873+0000","last_unstale":"2024-04-24T23:16:31.305540+0000","last_undegraded":"2024-04-24T23:16:31.305540+0000","last_fullsized":"2024-04-24T23:16:31.305540+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:21:40.116977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.296594+0000","last_change":"2024-04-24T23:15:36.997361+0000","last_active":"2024-04-24T23:16:18.296594+0000","last_peered":"2024-04-24T23:16:18.296594+0000","last_clean":"2024-04-24T23:16:18.296594+0000","last_became_active":"2024-04-24T23:15:36.997083+0000","last_became_peered":"2024-04-24T23:15:36.997083+0000","last_unstale":"2024-04-24T23:16:18.296594+0000","last_undegraded":"2024-04-24T23:16:18.296594+0000","last_fullsized":"2024-04-24T23:16:18.296594+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:47:59.334093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.293518+0000","last_change":"2024-04-24T23:15:41.025153+0000","last_active":"2024-04-24T23:16:18.293518+0000","last_peered":"2024-04-24T23:16:18.293518+0000","last_clean":"2024-04-24T23:16:18.293518+0000","last_became_active":"2024-04-24T23:15:41.024739+0000","last_became_peered":"2024-04-24T23:15:41.024739+0000","last_unstale":"2024-04-24T23:16:18.293518+0000","last_undegraded":"2024-04-24T23:16:18.293518+0000","last_fullsized":"2024-04-24T23:16:18.293518+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:31:00.160213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3,6],"acting":[2,5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1c","version":"26'5","reported_seq":31,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:31.301798+0000","last_change":"2024-04-24T23:15:35.967338+0000","last_active":"2024-04-24T23:16:31.301798+0000","last_peered":"2024-04-24T23:16:31.301798+0000","last_clean":"2024-04-24T23:16:31.301798+0000","last_became_active":"2024-04-24T23:15:35.966951+0000","last_became_peered":"2024-04-24T23:15:35.966951+0000","last_unstale":"2024-04-24T23:16:31.301798+0000","last_undegraded":"2024-04-24T23:16:31.301798+0000","last_fullsized":"2024-04-24T23:16:31.301798+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:08:05.248330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.230858+0000","last_change":"2024-04-24T23:15:36.998072+0000","last_active":"2024-04-24T23:16:18.230858+0000","last_peered":"2024-04-24T23:16:18.230858+0000","last_clean":"2024-04-24T23:16:18.230858+0000","last_became_active":"2024-04-24T23:15:36.997735+0000","last_became_peered":"2024-04-24T23:15:36.997735+0000","last_unstale":"2024-04-24T23:16:18.230858+0000","last_undegraded":"2024-04-24T23:16:18.230858+0000","last_fullsized":"2024-04-24T23:16:18.230858+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:22:06.869644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.294029+0000","last_change":"2024-04-24T23:15:41.008895+0000","last_active":"2024-04-24T23:16:18.294029+0000","last_peered":"2024-04-24T23:16:18.294029+0000","last_clean":"2024-04-24T23:16:18.294029+0000","last_became_active":"2024-04-24T23:15:41.006199+0000","last_became_peered":"2024-04-24T23:15:41.006199+0000","last_unstale":"2024-04-24T23:16:18.294029+0000","last_undegraded":"2024-04-24T23:16:18.294029+0000","last_fullsized":"2024-04-24T23:16:18.294029+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:13:34.583602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5,7],"acting":[2,6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.233639+0000","last_change":"2024-04-24T23:15:37.004074+0000","last_active":"2024-04-24T23:16:18.233639+0000","last_peered":"2024-04-24T23:16:18.233639+0000","last_clean":"2024-04-24T23:16:18.233639+0000","last_became_active":"2024-04-24T23:15:37.003544+0000","last_became_peered":"2024-04-24T23:15:37.003544+0000","last_unstale":"2024-04-24T23:16:18.233639+0000","last_undegraded":"2024-04-24T23:16:18.233639+0000","last_fullsized":"2024-04-24T23:16:18.233639+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:38:28.024702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"24'4","reported_seq":30,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.230075+0000","last_change":"2024-04-24T23:15:35.963000+0000","last_active":"2024-04-24T23:16:18.230075+0000","last_peered":"2024-04-24T23:16:18.230075+0000","last_clean":"2024-04-24T23:16:18.230075+0000","last_became_active":"2024-04-24T23:15:35.961960+0000","last_became_peered":"2024-04-24T23:15:35.961960+0000","last_unstale":"2024-04-24T23:16:18.230075+0000","last_undegraded":"2024-04-24T23:16:18.230075+0000","last_fullsized":"2024-04-24T23:16:18.230075+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:56:26.651193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.230595+0000","last_change":"2024-04-24T23:15:41.022481+0000","last_active":"2024-04-24T23:16:18.230595+0000","last_peered":"2024-04-24T23:16:18.230595+0000","last_clean":"2024-04-24T23:16:18.230595+0000","last_became_active":"2024-04-24T23:15:41.020034+0000","last_became_peered":"2024-04-24T23:15:41.020034+0000","last_unstale":"2024-04-24T23:16:18.230595+0000","last_undegraded":"2024-04-24T23:16:18.230595+0000","last_fullsized":"2024-04-24T23:16:18.230595+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:27:25.337503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,5],"acting":[3,1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.292564+0000","last_change":"2024-04-24T23:15:36.972020+0000","last_active":"2024-04-24T23:16:18.292564+0000","last_peered":"2024-04-24T23:16:18.292564+0000","last_clean":"2024-04-24T23:16:18.292564+0000","last_became_active":"2024-04-24T23:15:36.971583+0000","last_became_peered":"2024-04-24T23:15:36.971583+0000","last_unstale":"2024-04-24T23:16:18.292564+0000","last_undegraded":"2024-04-24T23:16:18.292564+0000","last_fullsized":"2024-04-24T23:16:18.292564+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:29:05.568608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"26'3","reported_seq":29,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:25.265282+0000","last_change":"2024-04-24T23:15:35.945916+0000","last_active":"2024-04-24T23:16:25.265282+0000","last_peered":"2024-04-24T23:16:25.265282+0000","last_clean":"2024-04-24T23:16:25.265282+0000","last_became_active":"2024-04-24T23:15:35.945610+0000","last_became_peered":"2024-04-24T23:15:35.945610+0000","last_unstale":"2024-04-24T23:16:25.265282+0000","last_undegraded":"2024-04-24T23:16:25.265282+0000","last_fullsized":"2024-04-24T23:16:25.265282+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:57:05.976074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.229068+0000","last_change":"2024-04-24T23:15:41.030165+0000","last_active":"2024-04-24T23:16:18.229068+0000","last_peered":"2024-04-24T23:16:18.229068+0000","last_clean":"2024-04-24T23:16:18.229068+0000","last_became_active":"2024-04-24T23:15:41.029685+0000","last_became_peered":"2024-04-24T23:15:41.029685+0000","last_unstale":"2024-04-24T23:16:18.229068+0000","last_undegraded":"2024-04-24T23:16:18.229068+0000","last_fullsized":"2024-04-24T23:16:18.229068+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:20:43.498190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6,1],"acting":[0,4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.223468+0000","last_change":"2024-04-24T23:15:36.965949+0000","last_active":"2024-04-24T23:16:18.223468+0000","last_peered":"2024-04-24T23:16:18.223468+0000","last_clean":"2024-04-24T23:16:18.223468+0000","last_became_active":"2024-04-24T23:15:36.965658+0000","last_became_peered":"2024-04-24T23:15:36.965658+0000","last_unstale":"2024-04-24T23:16:18.223468+0000","last_undegraded":"2024-04-24T23:16:18.223468+0000","last_fullsized":"2024-04-24T23:16:18.223468+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:38:28.024702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"26'7","reported_seq":33,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:31.301052+0000","last_change":"2024-04-24T23:15:35.972165+0000","last_active":"2024-04-24T23:16:31.301052+0000","last_peered":"2024-04-24T23:16:31.301052+0000","last_clean":"2024-04-24T23:16:31.301052+0000","last_became_active":"2024-04-24T23:15:35.971972+0000","last_became_peered":"2024-04-24T23:15:35.971972+0000","last_unstale":"2024-04-24T23:16:31.301052+0000","last_undegraded":"2024-04-24T23:16:31.301052+0000","last_fullsized":"2024-04-24T23:16:31.301052+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:11:12.088698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.292911+0000","last_change":"2024-04-24T23:15:41.019799+0000","last_active":"2024-04-24T23:16:18.292911+0000","last_peered":"2024-04-24T23:16:18.292911+0000","last_clean":"2024-04-24T23:16:18.292911+0000","last_became_active":"2024-04-24T23:15:41.019339+0000","last_became_peered":"2024-04-24T23:15:41.019339+0000","last_unstale":"2024-04-24T23:16:18.292911+0000","last_undegraded":"2024-04-24T23:16:18.292911+0000","last_fullsized":"2024-04-24T23:16:18.292911+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:59:34.601004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2,3],"acting":[7,4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.232810+0000","last_change":"2024-04-24T23:15:36.991991+0000","last_active":"2024-04-24T23:16:18.232810+0000","last_peered":"2024-04-24T23:16:18.232810+0000","last_clean":"2024-04-24T23:16:18.232810+0000","last_became_active":"2024-04-24T23:15:36.989336+0000","last_became_peered":"2024-04-24T23:15:36.989336+0000","last_unstale":"2024-04-24T23:16:18.232810+0000","last_undegraded":"2024-04-24T23:16:18.232810+0000","last_fullsized":"2024-04-24T23:16:18.232810+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:25:56.038903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.8","version":"24'2","reported_seq":28,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.228663+0000","last_change":"2024-04-24T23:15:35.965443+0000","last_active":"2024-04-24T23:16:18.228663+0000","last_peered":"2024-04-24T23:16:18.228663+0000","last_clean":"2024-04-24T23:16:18.228663+0000","last_became_active":"2024-04-24T23:15:35.965043+0000","last_became_peered":"2024-04-24T23:15:35.965043+0000","last_unstale":"2024-04-24T23:16:18.228663+0000","last_undegraded":"2024-04-24T23:16:18.228663+0000","last_fullsized":"2024-04-24T23:16:18.228663+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:07:23.320042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.294499+0000","last_change":"2024-04-24T23:15:41.025131+0000","last_active":"2024-04-24T23:16:18.294499+0000","last_peered":"2024-04-24T23:16:18.294499+0000","last_clean":"2024-04-24T23:16:18.294499+0000","last_became_active":"2024-04-24T23:15:41.024731+0000","last_became_peered":"2024-04-24T23:15:41.024731+0000","last_unstale":"2024-04-24T23:16:18.294499+0000","last_undegraded":"2024-04-24T23:16:18.294499+0000","last_fullsized":"2024-04-24T23:16:18.294499+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:28:07.952854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,7],"acting":[2,1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.224465+0000","last_change":"2024-04-24T23:15:36.965850+0000","last_active":"2024-04-24T23:16:18.224465+0000","last_peered":"2024-04-24T23:16:18.224465+0000","last_clean":"2024-04-24T23:16:18.224465+0000","last_became_active":"2024-04-24T23:15:36.965547+0000","last_became_peered":"2024-04-24T23:15:36.965547+0000","last_unstale":"2024-04-24T23:16:18.224465+0000","last_undegraded":"2024-04-24T23:16:18.224465+0000","last_fullsized":"2024-04-24T23:16:18.224465+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:06:37.125573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"26'7","reported_seq":33,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:31.305921+0000","last_change":"2024-04-24T23:15:35.971040+0000","last_active":"2024-04-24T23:16:31.305921+0000","last_peered":"2024-04-24T23:16:31.305921+0000","last_clean":"2024-04-24T23:16:31.305921+0000","last_became_active":"2024-04-24T23:15:35.970184+0000","last_became_peered":"2024-04-24T23:15:35.970184+0000","last_unstale":"2024-04-24T23:16:31.305921+0000","last_undegraded":"2024-04-24T23:16:31.305921+0000","last_fullsized":"2024-04-24T23:16:31.305921+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:45:20.913759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.295046+0000","last_change":"2024-04-24T23:15:41.025813+0000","last_active":"2024-04-24T23:16:18.295046+0000","last_peered":"2024-04-24T23:16:18.295046+0000","last_clean":"2024-04-24T23:16:18.295046+0000","last_became_active":"2024-04-24T23:15:41.025137+0000","last_became_peered":"2024-04-24T23:15:41.025137+0000","last_unstale":"2024-04-24T23:16:18.295046+0000","last_undegraded":"2024-04-24T23:16:18.295046+0000","last_fullsized":"2024-04-24T23:16:18.295046+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:35:17.784854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,3,1],"acting":[2,6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.224731+0000","last_change":"2024-04-24T23:15:36.967476+0000","last_active":"2024-04-24T23:16:18.224731+0000","last_peered":"2024-04-24T23:16:18.224731+0000","last_clean":"2024-04-24T23:16:18.224731+0000","last_became_active":"2024-04-24T23:15:36.967098+0000","last_became_peered":"2024-04-24T23:15:36.967098+0000","last_unstale":"2024-04-24T23:16:18.224731+0000","last_undegraded":"2024-04-24T23:16:18.224731+0000","last_fullsized":"2024-04-24T23:16:18.224731+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:25:56.038903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"26'7","reported_seq":33,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:25.270982+0000","last_change":"2024-04-24T23:15:35.970630+0000","last_active":"2024-04-24T23:16:25.270982+0000","last_peered":"2024-04-24T23:16:25.270982+0000","last_clean":"2024-04-24T23:16:25.270982+0000","last_became_active":"2024-04-24T23:15:35.970430+0000","last_became_peered":"2024-04-24T23:15:35.970430+0000","last_unstale":"2024-04-24T23:16:25.270982+0000","last_undegraded":"2024-04-24T23:16:25.270982+0000","last_fullsized":"2024-04-24T23:16:25.270982+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:15:59.219084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.223468+0000","last_change":"2024-04-24T23:15:41.005022+0000","last_active":"2024-04-24T23:16:18.223468+0000","last_peered":"2024-04-24T23:16:18.223468+0000","last_clean":"2024-04-24T23:16:18.223468+0000","last_became_active":"2024-04-24T23:15:41.004648+0000","last_became_peered":"2024-04-24T23:15:41.004648+0000","last_unstale":"2024-04-24T23:16:18.223468+0000","last_undegraded":"2024-04-24T23:16:18.223468+0000","last_fullsized":"2024-04-24T23:16:18.223468+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:06:01.559760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6,3],"acting":[4,1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.225003+0000","last_change":"2024-04-24T23:15:36.967053+0000","last_active":"2024-04-24T23:16:18.225003+0000","last_peered":"2024-04-24T23:16:18.225003+0000","last_clean":"2024-04-24T23:16:18.225003+0000","last_became_active":"2024-04-24T23:15:36.966746+0000","last_became_peered":"2024-04-24T23:15:36.966746+0000","last_unstale":"2024-04-24T23:16:18.225003+0000","last_undegraded":"2024-04-24T23:16:18.225003+0000","last_fullsized":"2024-04-24T23:16:18.225003+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:16:08.779860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"26'4","reported_seq":30,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:25.272679+0000","last_change":"2024-04-24T23:15:35.965890+0000","last_active":"2024-04-24T23:16:25.272679+0000","last_peered":"2024-04-24T23:16:25.272679+0000","last_clean":"2024-04-24T23:16:25.272679+0000","last_became_active":"2024-04-24T23:15:35.965631+0000","last_became_peered":"2024-04-24T23:15:35.965631+0000","last_unstale":"2024-04-24T23:16:25.272679+0000","last_undegraded":"2024-04-24T23:16:25.272679+0000","last_fullsized":"2024-04-24T23:16:25.272679+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:17:35.650731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.223982+0000","last_change":"2024-04-24T23:15:41.011405+0000","last_active":"2024-04-24T23:16:18.223982+0000","last_peered":"2024-04-24T23:16:18.223982+0000","last_clean":"2024-04-24T23:16:18.223982+0000","last_became_active":"2024-04-24T23:15:41.010982+0000","last_became_peered":"2024-04-24T23:15:41.010982+0000","last_unstale":"2024-04-24T23:16:18.223982+0000","last_undegraded":"2024-04-24T23:16:18.223982+0000","last_fullsized":"2024-04-24T23:16:18.223982+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:08:00.914296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2,7],"acting":[6,0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.294828+0000","last_change":"2024-04-24T23:15:36.975244+0000","last_active":"2024-04-24T23:16:18.294828+0000","last_peered":"2024-04-24T23:16:18.294828+0000","last_clean":"2024-04-24T23:16:18.294828+0000","last_became_active":"2024-04-24T23:15:36.974872+0000","last_became_peered":"2024-04-24T23:15:36.974872+0000","last_unstale":"2024-04-24T23:16:18.294828+0000","last_undegraded":"2024-04-24T23:16:18.294828+0000","last_fullsized":"2024-04-24T23:16:18.294828+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:45:57.018205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"26'1","reported_seq":27,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.291824+0000","last_change":"2024-04-24T23:15:35.945792+0000","last_active":"2024-04-24T23:16:18.291824+0000","last_peered":"2024-04-24T23:16:18.291824+0000","last_clean":"2024-04-24T23:16:18.291824+0000","last_became_active":"2024-04-24T23:15:35.945094+0000","last_became_peered":"2024-04-24T23:15:35.945094+0000","last_unstale":"2024-04-24T23:16:18.291824+0000","last_undegraded":"2024-04-24T23:16:18.291824+0000","last_fullsized":"2024-04-24T23:16:18.291824+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:49:12.719127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.224614+0000","last_change":"2024-04-24T23:15:41.022105+0000","last_active":"2024-04-24T23:16:18.224614+0000","last_peered":"2024-04-24T23:16:18.224614+0000","last_clean":"2024-04-24T23:16:18.224614+0000","last_became_active":"2024-04-24T23:15:41.021645+0000","last_became_peered":"2024-04-24T23:15:41.021645+0000","last_unstale":"2024-04-24T23:16:18.224614+0000","last_undegraded":"2024-04-24T23:16:18.224614+0000","last_fullsized":"2024-04-24T23:16:18.224614+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:03:52.974835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5,7],"acting":[6,3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.229269+0000","last_change":"2024-04-24T23:15:36.991239+0000","last_active":"2024-04-24T23:16:18.229269+0000","last_peered":"2024-04-24T23:16:18.229269+0000","last_clean":"2024-04-24T23:16:18.229269+0000","last_became_active":"2024-04-24T23:15:36.990963+0000","last_became_peered":"2024-04-24T23:15:36.990963+0000","last_unstale":"2024-04-24T23:16:18.229269+0000","last_undegraded":"2024-04-24T23:16:18.229269+0000","last_fullsized":"2024-04-24T23:16:18.229269+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:53:28.101334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"26'5","reported_seq":31,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:31.304824+0000","last_change":"2024-04-24T23:15:35.957931+0000","last_active":"2024-04-24T23:16:31.304824+0000","last_peered":"2024-04-24T23:16:31.304824+0000","last_clean":"2024-04-24T23:16:31.304824+0000","last_became_active":"2024-04-24T23:15:35.957432+0000","last_became_peered":"2024-04-24T23:15:35.957432+0000","last_unstale":"2024-04-24T23:16:31.304824+0000","last_undegraded":"2024-04-24T23:16:31.304824+0000","last_fullsized":"2024-04-24T23:16:31.304824+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:11:38.773425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.224009+0000","last_change":"2024-04-24T23:15:41.007857+0000","last_active":"2024-04-24T23:16:18.224009+0000","last_peered":"2024-04-24T23:16:18.224009+0000","last_clean":"2024-04-24T23:16:18.224009+0000","last_became_active":"2024-04-24T23:15:41.007449+0000","last_became_peered":"2024-04-24T23:15:41.007449+0000","last_unstale":"2024-04-24T23:16:18.224009+0000","last_undegraded":"2024-04-24T23:16:18.224009+0000","last_fullsized":"2024-04-24T23:16:18.224009+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:05:06.145440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0,3],"acting":[6,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.291576+0000","last_change":"2024-04-24T23:15:36.970444+0000","last_active":"2024-04-24T23:16:18.291576+0000","last_peered":"2024-04-24T23:16:18.291576+0000","last_clean":"2024-04-24T23:16:18.291576+0000","last_became_active":"2024-04-24T23:15:36.970159+0000","last_became_peered":"2024-04-24T23:15:36.970159+0000","last_unstale":"2024-04-24T23:16:18.291576+0000","last_undegraded":"2024-04-24T23:16:18.291576+0000","last_fullsized":"2024-04-24T23:16:18.291576+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:37:02.956224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"26'5","reported_seq":31,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:19.220734+0000","last_change":"2024-04-24T23:15:35.945461+0000","last_active":"2024-04-24T23:16:19.220734+0000","last_peered":"2024-04-24T23:16:19.220734+0000","last_clean":"2024-04-24T23:16:19.220734+0000","last_became_active":"2024-04-24T23:15:35.945185+0000","last_became_peered":"2024-04-24T23:15:35.945185+0000","last_unstale":"2024-04-24T23:16:19.220734+0000","last_undegraded":"2024-04-24T23:16:19.220734+0000","last_fullsized":"2024-04-24T23:16:19.220734+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:37:34.175539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.224370+0000","last_change":"2024-04-24T23:15:41.014584+0000","last_active":"2024-04-24T23:16:18.224370+0000","last_peered":"2024-04-24T23:16:18.224370+0000","last_clean":"2024-04-24T23:16:18.224370+0000","last_became_active":"2024-04-24T23:15:41.014137+0000","last_became_peered":"2024-04-24T23:15:41.014137+0000","last_unstale":"2024-04-24T23:16:18.224370+0000","last_undegraded":"2024-04-24T23:16:18.224370+0000","last_fullsized":"2024-04-24T23:16:18.224370+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:23:50.387635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,7],"acting":[6,0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.224277+0000","last_change":"2024-04-24T23:15:36.981783+0000","last_active":"2024-04-24T23:16:18.224277+0000","last_peered":"2024-04-24T23:16:18.224277+0000","last_clean":"2024-04-24T23:16:18.224277+0000","last_became_active":"2024-04-24T23:15:36.981502+0000","last_became_peered":"2024-04-24T23:15:36.981502+0000","last_unstale":"2024-04-24T23:16:18.224277+0000","last_undegraded":"2024-04-24T23:16:18.224277+0000","last_fullsized":"2024-04-24T23:16:18.224277+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:56:08.483410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"26'4","reported_seq":30,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:19.225767+0000","last_change":"2024-04-24T23:15:35.960519+0000","last_active":"2024-04-24T23:16:19.225767+0000","last_peered":"2024-04-24T23:16:19.225767+0000","last_clean":"2024-04-24T23:16:19.225767+0000","last_became_active":"2024-04-24T23:15:35.960180+0000","last_became_peered":"2024-04-24T23:15:35.960180+0000","last_unstale":"2024-04-24T23:16:19.225767+0000","last_undegraded":"2024-04-24T23:16:19.225767+0000","last_fullsized":"2024-04-24T23:16:19.225767+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:50:00.199120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.291646+0000","last_change":"2024-04-24T23:15:41.012876+0000","last_active":"2024-04-24T23:16:18.291646+0000","last_peered":"2024-04-24T23:16:18.291646+0000","last_clean":"2024-04-24T23:16:18.291646+0000","last_became_active":"2024-04-24T23:15:41.012493+0000","last_became_peered":"2024-04-24T23:15:41.012493+0000","last_unstale":"2024-04-24T23:16:18.291646+0000","last_undegraded":"2024-04-24T23:16:18.291646+0000","last_fullsized":"2024-04-24T23:16:18.291646+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:21:28.287059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6,1],"acting":[5,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.224197+0000","last_change":"2024-04-24T23:15:36.981449+0000","last_active":"2024-04-24T23:16:18.224197+0000","last_peered":"2024-04-24T23:16:18.224197+0000","last_clean":"2024-04-24T23:16:18.224197+0000","last_became_active":"2024-04-24T23:15:36.980837+0000","last_became_peered":"2024-04-24T23:15:36.980837+0000","last_unstale":"2024-04-24T23:16:18.224197+0000","last_undegraded":"2024-04-24T23:16:18.224197+0000","last_fullsized":"2024-04-24T23:16:18.224197+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:15:13.513341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.3","version":"26'6","reported_seq":32,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:25.276387+0000","last_change":"2024-04-24T23:15:35.952947+0000","last_active":"2024-04-24T23:16:25.276387+0000","last_peered":"2024-04-24T23:16:25.276387+0000","last_clean":"2024-04-24T23:16:25.276387+0000","last_became_active":"2024-04-24T23:15:35.952655+0000","last_became_peered":"2024-04-24T23:15:35.952655+0000","last_unstale":"2024-04-24T23:16:25.276387+0000","last_undegraded":"2024-04-24T23:16:25.276387+0000","last_fullsized":"2024-04-24T23:16:25.276387+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:50:10.939396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":182,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.296073+0000","last_change":"2024-04-24T23:15:25.826673+0000","last_active":"2024-04-24T23:16:18.296073+0000","last_peered":"2024-04-24T23:16:18.296073+0000","last_clean":"2024-04-24T23:16:18.296073+0000","last_became_active":"2024-04-24T23:15:25.820030+0000","last_became_peered":"2024-04-24T23:15:25.820030+0000","last_unstale":"2024-04-24T23:16:18.296073+0000","last_undegraded":"2024-04-24T23:16:18.296073+0000","last_fullsized":"2024-04-24T23:16:18.296073+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:24.790928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:24.790928+0000","last_clean_scrub_stamp":"2024-04-24T23:15:24.790928+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:37:24.054332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.231043+0000","last_change":"2024-04-24T23:15:41.024712+0000","last_active":"2024-04-24T23:16:18.231043+0000","last_peered":"2024-04-24T23:16:18.231043+0000","last_clean":"2024-04-24T23:16:18.231043+0000","last_became_active":"2024-04-24T23:15:41.022697+0000","last_became_peered":"2024-04-24T23:15:41.022697+0000","last_unstale":"2024-04-24T23:16:18.231043+0000","last_undegraded":"2024-04-24T23:16:18.231043+0000","last_fullsized":"2024-04-24T23:16:18.231043+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:37:08.231105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2,0],"acting":[3,1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.230951+0000","last_change":"2024-04-24T23:15:37.000489+0000","last_active":"2024-04-24T23:16:18.230951+0000","last_peered":"2024-04-24T23:16:18.230951+0000","last_clean":"2024-04-24T23:16:18.230951+0000","last_became_active":"2024-04-24T23:15:37.000179+0000","last_became_peered":"2024-04-24T23:15:37.000179+0000","last_unstale":"2024-04-24T23:16:18.230951+0000","last_undegraded":"2024-04-24T23:16:18.230951+0000","last_fullsized":"2024-04-24T23:16:18.230951+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:06:37.125573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.c","version":"26'5","reported_seq":31,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:25.270158+0000","last_change":"2024-04-24T23:15:35.966020+0000","last_active":"2024-04-24T23:16:25.270158+0000","last_peered":"2024-04-24T23:16:25.270158+0000","last_clean":"2024-04-24T23:16:25.270158+0000","last_became_active":"2024-04-24T23:15:35.965758+0000","last_became_peered":"2024-04-24T23:15:35.965758+0000","last_unstale":"2024-04-24T23:16:25.270158+0000","last_undegraded":"2024-04-24T23:16:25.270158+0000","last_fullsized":"2024-04-24T23:16:25.270158+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:39:21.233461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.294138+0000","last_change":"2024-04-24T23:15:41.024489+0000","last_active":"2024-04-24T23:16:18.294138+0000","last_peered":"2024-04-24T23:16:18.294138+0000","last_clean":"2024-04-24T23:16:18.294138+0000","last_became_active":"2024-04-24T23:15:41.024041+0000","last_became_peered":"2024-04-24T23:15:41.024041+0000","last_unstale":"2024-04-24T23:16:18.294138+0000","last_undegraded":"2024-04-24T23:16:18.294138+0000","last_fullsized":"2024-04-24T23:16:18.294138+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:27:18.197003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7,4],"acting":[2,3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.229879+0000","last_change":"2024-04-24T23:15:36.979887+0000","last_active":"2024-04-24T23:16:18.229879+0000","last_peered":"2024-04-24T23:16:18.229879+0000","last_clean":"2024-04-24T23:16:18.229879+0000","last_became_active":"2024-04-24T23:15:36.979611+0000","last_became_peered":"2024-04-24T23:15:36.979611+0000","last_unstale":"2024-04-24T23:16:18.229879+0000","last_undegraded":"2024-04-24T23:16:18.229879+0000","last_fullsized":"2024-04-24T23:16:18.229879+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:07:18.261077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"26'3","reported_seq":29,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:25.265069+0000","last_change":"2024-04-24T23:15:35.965192+0000","last_active":"2024-04-24T23:16:25.265069+0000","last_peered":"2024-04-24T23:16:25.265069+0000","last_clean":"2024-04-24T23:16:25.265069+0000","last_became_active":"2024-04-24T23:15:35.964873+0000","last_became_peered":"2024-04-24T23:15:35.964873+0000","last_unstale":"2024-04-24T23:16:25.265069+0000","last_undegraded":"2024-04-24T23:16:25.265069+0000","last_fullsized":"2024-04-24T23:16:25.265069+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:14:27.884944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.291986+0000","last_change":"2024-04-24T23:15:41.004002+0000","last_active":"2024-04-24T23:16:18.291986+0000","last_peered":"2024-04-24T23:16:18.291986+0000","last_clean":"2024-04-24T23:16:18.291986+0000","last_became_active":"2024-04-24T23:15:41.003335+0000","last_became_peered":"2024-04-24T23:15:41.003335+0000","last_unstale":"2024-04-24T23:16:18.291986+0000","last_undegraded":"2024-04-24T23:16:18.291986+0000","last_fullsized":"2024-04-24T23:16:18.291986+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:49:36.206903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1,3],"acting":[5,4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.229375+0000","last_change":"2024-04-24T23:15:37.002854+0000","last_active":"2024-04-24T23:16:18.229375+0000","last_peered":"2024-04-24T23:16:18.229375+0000","last_clean":"2024-04-24T23:16:18.229375+0000","last_became_active":"2024-04-24T23:15:37.002583+0000","last_became_peered":"2024-04-24T23:15:37.002583+0000","last_unstale":"2024-04-24T23:16:18.229375+0000","last_undegraded":"2024-04-24T23:16:18.229375+0000","last_fullsized":"2024-04-24T23:16:18.229375+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:27:04.251547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"26'5","reported_seq":31,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:31.299455+0000","last_change":"2024-04-24T23:15:35.948752+0000","last_active":"2024-04-24T23:16:31.299455+0000","last_peered":"2024-04-24T23:16:31.299455+0000","last_clean":"2024-04-24T23:16:31.299455+0000","last_became_active":"2024-04-24T23:15:35.948438+0000","last_became_peered":"2024-04-24T23:15:35.948438+0000","last_unstale":"2024-04-24T23:16:31.299455+0000","last_undegraded":"2024-04-24T23:16:31.299455+0000","last_fullsized":"2024-04-24T23:16:31.299455+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:45:56.006278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.229925+0000","last_change":"2024-04-24T23:15:41.017524+0000","last_active":"2024-04-24T23:16:18.229925+0000","last_peered":"2024-04-24T23:16:18.229925+0000","last_clean":"2024-04-24T23:16:18.229925+0000","last_became_active":"2024-04-24T23:15:41.017092+0000","last_became_peered":"2024-04-24T23:15:41.017092+0000","last_unstale":"2024-04-24T23:16:18.229925+0000","last_undegraded":"2024-04-24T23:16:18.229925+0000","last_fullsized":"2024-04-24T23:16:18.229925+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:04:53.235038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.294797+0000","last_change":"2024-04-24T23:15:36.990441+0000","last_active":"2024-04-24T23:16:18.294797+0000","last_peered":"2024-04-24T23:16:18.294797+0000","last_clean":"2024-04-24T23:16:18.294797+0000","last_became_active":"2024-04-24T23:15:36.989863+0000","last_became_peered":"2024-04-24T23:15:36.989863+0000","last_unstale":"2024-04-24T23:16:18.294797+0000","last_undegraded":"2024-04-24T23:16:18.294797+0000","last_fullsized":"2024-04-24T23:16:18.294797+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:50:07.975347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.f","version":"26'2","reported_seq":28,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.232012+0000","last_change":"2024-04-24T23:15:35.969317+0000","last_active":"2024-04-24T23:16:18.232012+0000","last_peered":"2024-04-24T23:16:18.232012+0000","last_clean":"2024-04-24T23:16:18.232012+0000","last_became_active":"2024-04-24T23:15:35.968994+0000","last_became_peered":"2024-04-24T23:15:35.968994+0000","last_unstale":"2024-04-24T23:16:18.232012+0000","last_undegraded":"2024-04-24T23:16:18.232012+0000","last_fullsized":"2024-04-24T23:16:18.232012+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:51:57.100772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.296718+0000","last_change":"2024-04-24T23:15:41.010103+0000","last_active":"2024-04-24T23:16:18.296718+0000","last_peered":"2024-04-24T23:16:18.296718+0000","last_clean":"2024-04-24T23:16:18.296718+0000","last_became_active":"2024-04-24T23:15:41.009328+0000","last_became_peered":"2024-04-24T23:15:41.009328+0000","last_unstale":"2024-04-24T23:16:18.296718+0000","last_undegraded":"2024-04-24T23:16:18.296718+0000","last_fullsized":"2024-04-24T23:16:18.296718+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:46:49.579501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,2],"acting":[1,5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.232575+0000","last_change":"2024-04-24T23:15:36.996765+0000","last_active":"2024-04-24T23:16:18.232575+0000","last_peered":"2024-04-24T23:16:18.232575+0000","last_clean":"2024-04-24T23:16:18.232575+0000","last_became_active":"2024-04-24T23:15:36.995801+0000","last_became_peered":"2024-04-24T23:15:36.995801+0000","last_unstale":"2024-04-24T23:16:18.232575+0000","last_undegraded":"2024-04-24T23:16:18.232575+0000","last_fullsized":"2024-04-24T23:16:18.232575+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:25:19.421667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"26'17","reported_seq":43,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:25.266099+0000","last_change":"2024-04-24T23:15:35.944023+0000","last_active":"2024-04-24T23:16:25.266099+0000","last_peered":"2024-04-24T23:16:25.266099+0000","last_clean":"2024-04-24T23:16:25.266099+0000","last_became_active":"2024-04-24T23:15:35.943710+0000","last_became_peered":"2024-04-24T23:15:35.943710+0000","last_unstale":"2024-04-24T23:16:25.266099+0000","last_undegraded":"2024-04-24T23:16:25.266099+0000","last_fullsized":"2024-04-24T23:16:25.266099+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:46:48.090958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.292053+0000","last_change":"2024-04-24T23:15:41.013027+0000","last_active":"2024-04-24T23:16:18.292053+0000","last_peered":"2024-04-24T23:16:18.292053+0000","last_clean":"2024-04-24T23:16:18.292053+0000","last_became_active":"2024-04-24T23:15:41.012587+0000","last_became_peered":"2024-04-24T23:15:41.012587+0000","last_unstale":"2024-04-24T23:16:18.292053+0000","last_undegraded":"2024-04-24T23:16:18.292053+0000","last_fullsized":"2024-04-24T23:16:18.292053+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:27:04.276246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1,6],"acting":[5,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.295974+0000","last_change":"2024-04-24T23:15:36.979702+0000","last_active":"2024-04-24T23:16:18.295974+0000","last_peered":"2024-04-24T23:16:18.295974+0000","last_clean":"2024-04-24T23:16:18.295974+0000","last_became_active":"2024-04-24T23:15:36.979356+0000","last_became_peered":"2024-04-24T23:15:36.979356+0000","last_unstale":"2024-04-24T23:16:18.295974+0000","last_undegraded":"2024-04-24T23:16:18.295974+0000","last_fullsized":"2024-04-24T23:16:18.295974+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:14:24.309192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.11","version":"26'5","reported_seq":31,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:25.270027+0000","last_change":"2024-04-24T23:15:35.973026+0000","last_active":"2024-04-24T23:16:25.270027+0000","last_peered":"2024-04-24T23:16:25.270027+0000","last_clean":"2024-04-24T23:16:25.270027+0000","last_became_active":"2024-04-24T23:15:35.972655+0000","last_became_peered":"2024-04-24T23:15:35.972655+0000","last_unstale":"2024-04-24T23:16:25.270027+0000","last_undegraded":"2024-04-24T23:16:25.270027+0000","last_fullsized":"2024-04-24T23:16:25.270027+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:25:34.352908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.291924+0000","last_change":"2024-04-24T23:15:41.004221+0000","last_active":"2024-04-24T23:16:18.291924+0000","last_peered":"2024-04-24T23:16:18.291924+0000","last_clean":"2024-04-24T23:16:18.291924+0000","last_became_active":"2024-04-24T23:15:41.003790+0000","last_became_peered":"2024-04-24T23:15:41.003790+0000","last_unstale":"2024-04-24T23:16:18.291924+0000","last_undegraded":"2024-04-24T23:16:18.291924+0000","last_fullsized":"2024-04-24T23:16:18.291924+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:28:58.866024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6,4],"acting":[5,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.229856+0000","last_change":"2024-04-24T23:15:36.997389+0000","last_active":"2024-04-24T23:16:18.229856+0000","last_peered":"2024-04-24T23:16:18.229856+0000","last_clean":"2024-04-24T23:16:18.229856+0000","last_became_active":"2024-04-24T23:15:36.995923+0000","last_became_peered":"2024-04-24T23:15:36.995923+0000","last_unstale":"2024-04-24T23:16:18.229856+0000","last_undegraded":"2024-04-24T23:16:18.229856+0000","last_fullsized":"2024-04-24T23:16:18.229856+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:55:47.134320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"26'5","reported_seq":31,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:25.274296+0000","last_change":"2024-04-24T23:15:35.957715+0000","last_active":"2024-04-24T23:16:25.274296+0000","last_peered":"2024-04-24T23:16:25.274296+0000","last_clean":"2024-04-24T23:16:25.274296+0000","last_became_active":"2024-04-24T23:15:35.957088+0000","last_became_peered":"2024-04-24T23:15:35.957088+0000","last_unstale":"2024-04-24T23:16:25.274296+0000","last_undegraded":"2024-04-24T23:16:25.274296+0000","last_fullsized":"2024-04-24T23:16:25.274296+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:30:34.483669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.223881+0000","last_change":"2024-04-24T23:15:41.008165+0000","last_active":"2024-04-24T23:16:18.223881+0000","last_peered":"2024-04-24T23:16:18.223881+0000","last_clean":"2024-04-24T23:16:18.223881+0000","last_became_active":"2024-04-24T23:15:41.007555+0000","last_became_peered":"2024-04-24T23:15:41.007555+0000","last_unstale":"2024-04-24T23:16:18.223881+0000","last_undegraded":"2024-04-24T23:16:18.223881+0000","last_fullsized":"2024-04-24T23:16:18.223881+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:18:08.195747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0,7],"acting":[6,4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.224736+0000","last_change":"2024-04-24T23:15:36.991382+0000","last_active":"2024-04-24T23:16:18.224736+0000","last_peered":"2024-04-24T23:16:18.224736+0000","last_clean":"2024-04-24T23:16:18.224736+0000","last_became_active":"2024-04-24T23:15:36.991115+0000","last_became_peered":"2024-04-24T23:15:36.991115+0000","last_unstale":"2024-04-24T23:16:18.224736+0000","last_undegraded":"2024-04-24T23:16:18.224736+0000","last_fullsized":"2024-04-24T23:16:18.224736+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:21:51.863947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"26'9","reported_seq":35,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:31.304710+0000","last_change":"2024-04-24T23:15:35.964245+0000","last_active":"2024-04-24T23:16:31.304710+0000","last_peered":"2024-04-24T23:16:31.304710+0000","last_clean":"2024-04-24T23:16:31.304710+0000","last_became_active":"2024-04-24T23:15:35.963907+0000","last_became_peered":"2024-04-24T23:15:35.963907+0000","last_unstale":"2024-04-24T23:16:31.304710+0000","last_undegraded":"2024-04-24T23:16:31.304710+0000","last_fullsized":"2024-04-24T23:16:31.304710+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:15:00.913899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.230757+0000","last_change":"2024-04-24T23:15:41.021981+0000","last_active":"2024-04-24T23:16:18.230757+0000","last_peered":"2024-04-24T23:16:18.230757+0000","last_clean":"2024-04-24T23:16:18.230757+0000","last_became_active":"2024-04-24T23:15:41.019193+0000","last_became_peered":"2024-04-24T23:15:41.019193+0000","last_unstale":"2024-04-24T23:16:18.230757+0000","last_undegraded":"2024-04-24T23:16:18.230757+0000","last_fullsized":"2024-04-24T23:16:18.230757+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:04:39.828502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5,6],"acting":[3,0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.232496+0000","last_change":"2024-04-24T23:15:36.986833+0000","last_active":"2024-04-24T23:16:18.232496+0000","last_peered":"2024-04-24T23:16:18.232496+0000","last_clean":"2024-04-24T23:16:18.232496+0000","last_became_active":"2024-04-24T23:15:36.986484+0000","last_became_peered":"2024-04-24T23:15:36.986484+0000","last_unstale":"2024-04-24T23:16:18.232496+0000","last_undegraded":"2024-04-24T23:16:18.232496+0000","last_fullsized":"2024-04-24T23:16:18.232496+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:12:32.245267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.14","version":"26'16","reported_seq":42,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:41.653552+0000","last_change":"2024-04-24T23:15:35.957124+0000","last_active":"2024-04-24T23:16:41.653552+0000","last_peered":"2024-04-24T23:16:41.653552+0000","last_clean":"2024-04-24T23:16:41.653552+0000","last_became_active":"2024-04-24T23:15:35.951923+0000","last_became_peered":"2024-04-24T23:15:35.951923+0000","last_unstale":"2024-04-24T23:16:41.653552+0000","last_undegraded":"2024-04-24T23:16:41.653552+0000","last_fullsized":"2024-04-24T23:16:41.653552+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:44:16.820640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4100,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.223976+0000","last_change":"2024-04-24T23:15:41.015494+0000","last_active":"2024-04-24T23:16:18.223976+0000","last_peered":"2024-04-24T23:16:18.223976+0000","last_clean":"2024-04-24T23:16:18.223976+0000","last_became_active":"2024-04-24T23:15:41.015179+0000","last_became_peered":"2024-04-24T23:15:41.015179+0000","last_unstale":"2024-04-24T23:16:18.223976+0000","last_undegraded":"2024-04-24T23:16:18.223976+0000","last_fullsized":"2024-04-24T23:16:18.223976+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:18:32.208506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5,0],"acting":[4,3,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.230201+0000","last_change":"2024-04-24T23:15:36.999334+0000","last_active":"2024-04-24T23:16:18.230201+0000","last_peered":"2024-04-24T23:16:18.230201+0000","last_clean":"2024-04-24T23:16:18.230201+0000","last_became_active":"2024-04-24T23:15:36.995470+0000","last_became_peered":"2024-04-24T23:15:36.995470+0000","last_unstale":"2024-04-24T23:16:18.230201+0000","last_undegraded":"2024-04-24T23:16:18.230201+0000","last_fullsized":"2024-04-24T23:16:18.230201+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:19:57.973402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.15","version":"26'5","reported_seq":31,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.292831+0000","last_change":"2024-04-24T23:15:35.946041+0000","last_active":"2024-04-24T23:16:18.292831+0000","last_peered":"2024-04-24T23:16:18.292831+0000","last_clean":"2024-04-24T23:16:18.292831+0000","last_became_active":"2024-04-24T23:15:35.944805+0000","last_became_peered":"2024-04-24T23:15:35.944805+0000","last_unstale":"2024-04-24T23:16:18.292831+0000","last_undegraded":"2024-04-24T23:16:18.292831+0000","last_fullsized":"2024-04-24T23:16:18.292831+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:52:19.261306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.293637+0000","last_change":"2024-04-24T23:15:41.024392+0000","last_active":"2024-04-24T23:16:18.293637+0000","last_peered":"2024-04-24T23:16:18.293637+0000","last_clean":"2024-04-24T23:16:18.293637+0000","last_became_active":"2024-04-24T23:15:41.023951+0000","last_became_peered":"2024-04-24T23:15:41.023951+0000","last_unstale":"2024-04-24T23:16:18.293637+0000","last_undegraded":"2024-04-24T23:16:18.293637+0000","last_fullsized":"2024-04-24T23:16:18.293637+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:24:17.810117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1,7],"acting":[2,3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.292208+0000","last_change":"2024-04-24T23:15:36.992243+0000","last_active":"2024-04-24T23:16:18.292208+0000","last_peered":"2024-04-24T23:16:18.292208+0000","last_clean":"2024-04-24T23:16:18.292208+0000","last_became_active":"2024-04-24T23:15:36.992002+0000","last_became_peered":"2024-04-24T23:15:36.992002+0000","last_unstale":"2024-04-24T23:16:18.292208+0000","last_undegraded":"2024-04-24T23:16:18.292208+0000","last_fullsized":"2024-04-24T23:16:18.292208+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:52:15.911009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"26'4","reported_seq":30,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:31.301673+0000","last_change":"2024-04-24T23:15:35.967408+0000","last_active":"2024-04-24T23:16:31.301673+0000","last_peered":"2024-04-24T23:16:31.301673+0000","last_clean":"2024-04-24T23:16:31.301673+0000","last_became_active":"2024-04-24T23:15:35.966646+0000","last_became_peered":"2024-04-24T23:15:35.966646+0000","last_unstale":"2024-04-24T23:16:31.301673+0000","last_undegraded":"2024-04-24T23:16:31.301673+0000","last_fullsized":"2024-04-24T23:16:31.301673+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:07:07.931773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.291779+0000","last_change":"2024-04-24T23:15:41.021697+0000","last_active":"2024-04-24T23:16:18.291779+0000","last_peered":"2024-04-24T23:16:18.291779+0000","last_clean":"2024-04-24T23:16:18.291779+0000","last_became_active":"2024-04-24T23:15:41.021281+0000","last_became_peered":"2024-04-24T23:15:41.021281+0000","last_unstale":"2024-04-24T23:16:18.291779+0000","last_undegraded":"2024-04-24T23:16:18.291779+0000","last_fullsized":"2024-04-24T23:16:18.291779+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:59:37.306593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3,2],"acting":[7,1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.294698+0000","last_change":"2024-04-24T23:15:36.990988+0000","last_active":"2024-04-24T23:16:18.294698+0000","last_peered":"2024-04-24T23:16:18.294698+0000","last_clean":"2024-04-24T23:16:18.294698+0000","last_became_active":"2024-04-24T23:15:36.990711+0000","last_became_peered":"2024-04-24T23:15:36.990711+0000","last_unstale":"2024-04-24T23:16:18.294698+0000","last_undegraded":"2024-04-24T23:16:18.294698+0000","last_fullsized":"2024-04-24T23:16:18.294698+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:33:20.663776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.17","version":"26'4","reported_seq":30,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.297684+0000","last_change":"2024-04-24T23:15:35.957838+0000","last_active":"2024-04-24T23:16:18.297684+0000","last_peered":"2024-04-24T23:16:18.297684+0000","last_clean":"2024-04-24T23:16:18.297684+0000","last_became_active":"2024-04-24T23:15:35.956870+0000","last_became_peered":"2024-04-24T23:15:35.956870+0000","last_unstale":"2024-04-24T23:16:18.297684+0000","last_undegraded":"2024-04-24T23:16:18.297684+0000","last_fullsized":"2024-04-24T23:16:18.297684+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:56:03.889693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.223850+0000","last_change":"2024-04-24T23:15:41.006098+0000","last_active":"2024-04-24T23:16:18.223850+0000","last_peered":"2024-04-24T23:16:18.223850+0000","last_clean":"2024-04-24T23:16:18.223850+0000","last_became_active":"2024-04-24T23:15:41.005578+0000","last_became_peered":"2024-04-24T23:15:41.005578+0000","last_unstale":"2024-04-24T23:16:18.223850+0000","last_undegraded":"2024-04-24T23:16:18.223850+0000","last_fullsized":"2024-04-24T23:16:18.223850+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:21:12.802112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3,7],"acting":[4,0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.224194+0000","last_change":"2024-04-24T23:15:41.022122+0000","last_active":"2024-04-24T23:16:18.224194+0000","last_peered":"2024-04-24T23:16:18.224194+0000","last_clean":"2024-04-24T23:16:18.224194+0000","last_became_active":"2024-04-24T23:15:41.021698+0000","last_became_peered":"2024-04-24T23:15:41.021698+0000","last_unstale":"2024-04-24T23:16:18.224194+0000","last_undegraded":"2024-04-24T23:16:18.224194+0000","last_fullsized":"2024-04-24T23:16:18.224194+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:13:56.730050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5,3],"acting":[4,1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.223853+0000","last_change":"2024-04-24T23:15:36.979816+0000","last_active":"2024-04-24T23:16:18.223853+0000","last_peered":"2024-04-24T23:16:18.223853+0000","last_clean":"2024-04-24T23:16:18.223853+0000","last_became_active":"2024-04-24T23:15:36.979553+0000","last_became_peered":"2024-04-24T23:15:36.979553+0000","last_unstale":"2024-04-24T23:16:18.223853+0000","last_undegraded":"2024-04-24T23:16:18.223853+0000","last_fullsized":"2024-04-24T23:16:18.223853+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:55:47.134320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.18","version":"26'3","reported_seq":29,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:25.272717+0000","last_change":"2024-04-24T23:15:35.964744+0000","last_active":"2024-04-24T23:16:25.272717+0000","last_peered":"2024-04-24T23:16:25.272717+0000","last_clean":"2024-04-24T23:16:25.272717+0000","last_became_active":"2024-04-24T23:15:35.964434+0000","last_became_peered":"2024-04-24T23:15:35.964434+0000","last_unstale":"2024-04-24T23:16:25.272717+0000","last_undegraded":"2024-04-24T23:16:25.272717+0000","last_fullsized":"2024-04-24T23:16:25.272717+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:26:32.587197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.231561+0000","last_change":"2024-04-24T23:15:41.022032+0000","last_active":"2024-04-24T23:16:18.231561+0000","last_peered":"2024-04-24T23:16:18.231561+0000","last_clean":"2024-04-24T23:16:18.231561+0000","last_became_active":"2024-04-24T23:15:41.021616+0000","last_became_peered":"2024-04-24T23:15:41.021616+0000","last_unstale":"2024-04-24T23:16:18.231561+0000","last_undegraded":"2024-04-24T23:16:18.231561+0000","last_fullsized":"2024-04-24T23:16:18.231561+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:08:59.242392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5,1],"acting":[3,2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.291442+0000","last_change":"2024-04-24T23:15:36.967198+0000","last_active":"2024-04-24T23:16:18.291442+0000","last_peered":"2024-04-24T23:16:18.291442+0000","last_clean":"2024-04-24T23:16:18.291442+0000","last_became_active":"2024-04-24T23:15:36.966910+0000","last_became_peered":"2024-04-24T23:15:36.966910+0000","last_unstale":"2024-04-24T23:16:18.291442+0000","last_undegraded":"2024-04-24T23:16:18.291442+0000","last_fullsized":"2024-04-24T23:16:18.291442+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:51:19.788437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"26'3","reported_seq":29,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:19.219459+0000","last_change":"2024-04-24T23:15:35.967169+0000","last_active":"2024-04-24T23:16:19.219459+0000","last_peered":"2024-04-24T23:16:19.219459+0000","last_clean":"2024-04-24T23:16:19.219459+0000","last_became_active":"2024-04-24T23:15:35.966939+0000","last_became_peered":"2024-04-24T23:15:35.966939+0000","last_unstale":"2024-04-24T23:16:19.219459+0000","last_undegraded":"2024-04-24T23:16:19.219459+0000","last_fullsized":"2024-04-24T23:16:19.219459+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:38:55.460569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"26'7","reported_seq":33,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:25.275257+0000","last_change":"2024-04-24T23:15:35.971236+0000","last_active":"2024-04-24T23:16:25.275257+0000","last_peered":"2024-04-24T23:16:25.275257+0000","last_clean":"2024-04-24T23:16:25.275257+0000","last_became_active":"2024-04-24T23:15:35.970908+0000","last_became_peered":"2024-04-24T23:15:35.970908+0000","last_unstale":"2024-04-24T23:16:25.275257+0000","last_undegraded":"2024-04-24T23:16:25.275257+0000","last_fullsized":"2024-04-24T23:16:25.275257+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:14:15.548024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.223928+0000","last_change":"2024-04-24T23:15:36.988156+0000","last_active":"2024-04-24T23:16:18.223928+0000","last_peered":"2024-04-24T23:16:18.223928+0000","last_clean":"2024-04-24T23:16:18.223928+0000","last_became_active":"2024-04-24T23:15:36.987896+0000","last_became_peered":"2024-04-24T23:15:36.987896+0000","last_unstale":"2024-04-24T23:16:18.223928+0000","last_undegraded":"2024-04-24T23:16:18.223928+0000","last_fullsized":"2024-04-24T23:16:18.223928+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:22:06.869644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.224383+0000","last_change":"2024-04-24T23:15:40.997559+0000","last_active":"2024-04-24T23:16:18.224383+0000","last_peered":"2024-04-24T23:16:18.224383+0000","last_clean":"2024-04-24T23:16:18.224383+0000","last_became_active":"2024-04-24T23:15:40.997102+0000","last_became_peered":"2024-04-24T23:15:40.997102+0000","last_unstale":"2024-04-24T23:16:18.224383+0000","last_undegraded":"2024-04-24T23:16:18.224383+0000","last_fullsized":"2024-04-24T23:16:18.224383+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:25:02.046761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,7],"acting":[4,6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"24'5","reported_seq":31,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.292037+0000","last_change":"2024-04-24T23:15:35.944842+0000","last_active":"2024-04-24T23:16:18.292037+0000","last_peered":"2024-04-24T23:16:18.292037+0000","last_clean":"2024-04-24T23:16:18.292037+0000","last_became_active":"2024-04-24T23:15:35.944563+0000","last_became_peered":"2024-04-24T23:15:35.944563+0000","last_unstale":"2024-04-24T23:16:18.292037+0000","last_undegraded":"2024-04-24T23:16:18.292037+0000","last_fullsized":"2024-04-24T23:16:18.292037+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:40:58.933384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.296474+0000","last_change":"2024-04-24T23:15:36.981103+0000","last_active":"2024-04-24T23:16:18.296474+0000","last_peered":"2024-04-24T23:16:18.296474+0000","last_clean":"2024-04-24T23:16:18.296474+0000","last_became_active":"2024-04-24T23:15:36.979637+0000","last_became_peered":"2024-04-24T23:15:36.979637+0000","last_unstale":"2024-04-24T23:16:18.296474+0000","last_undegraded":"2024-04-24T23:16:18.296474+0000","last_fullsized":"2024-04-24T23:16:18.296474+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:09:39.750272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.292643+0000","last_change":"2024-04-24T23:15:41.017087+0000","last_active":"2024-04-24T23:16:18.292643+0000","last_peered":"2024-04-24T23:16:18.292643+0000","last_clean":"2024-04-24T23:16:18.292643+0000","last_became_active":"2024-04-24T23:15:41.016696+0000","last_became_peered":"2024-04-24T23:15:41.016696+0000","last_unstale":"2024-04-24T23:16:18.292643+0000","last_undegraded":"2024-04-24T23:16:18.292643+0000","last_fullsized":"2024-04-24T23:16:18.292643+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:05:57.823447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1,0],"acting":[7,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.293973+0000","last_change":"2024-04-24T23:15:36.970211+0000","last_active":"2024-04-24T23:16:18.293973+0000","last_peered":"2024-04-24T23:16:18.293973+0000","last_clean":"2024-04-24T23:16:18.293973+0000","last_became_active":"2024-04-24T23:15:36.969090+0000","last_became_peered":"2024-04-24T23:15:36.969090+0000","last_unstale":"2024-04-24T23:16:18.293973+0000","last_undegraded":"2024-04-24T23:16:18.293973+0000","last_fullsized":"2024-04-24T23:16:18.293973+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:16:00.231012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.291726+0000","last_change":"2024-04-24T23:15:36.982528+0000","last_active":"2024-04-24T23:16:18.291726+0000","last_peered":"2024-04-24T23:16:18.291726+0000","last_clean":"2024-04-24T23:16:18.291726+0000","last_became_active":"2024-04-24T23:15:36.982299+0000","last_became_peered":"2024-04-24T23:15:36.982299+0000","last_unstale":"2024-04-24T23:16:18.291726+0000","last_undegraded":"2024-04-24T23:16:18.291726+0000","last_fullsized":"2024-04-24T23:16:18.291726+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:02:17.732324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.233323+0000","last_change":"2024-04-24T23:15:36.989899+0000","last_active":"2024-04-24T23:16:18.233323+0000","last_peered":"2024-04-24T23:16:18.233323+0000","last_clean":"2024-04-24T23:16:18.233323+0000","last_became_active":"2024-04-24T23:15:36.989557+0000","last_became_peered":"2024-04-24T23:15:36.989557+0000","last_unstale":"2024-04-24T23:16:18.233323+0000","last_undegraded":"2024-04-24T23:16:18.233323+0000","last_fullsized":"2024-04-24T23:16:18.233323+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:15:13.513341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.291922+0000","last_change":"2024-04-24T23:15:36.969628+0000","last_active":"2024-04-24T23:16:18.291922+0000","last_peered":"2024-04-24T23:16:18.291922+0000","last_clean":"2024-04-24T23:16:18.291922+0000","last_became_active":"2024-04-24T23:15:36.969327+0000","last_became_peered":"2024-04-24T23:15:36.969327+0000","last_unstale":"2024-04-24T23:16:18.291922+0000","last_undegraded":"2024-04-24T23:16:18.291922+0000","last_fullsized":"2024-04-24T23:16:18.291922+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:08:47.021999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.224974+0000","last_change":"2024-04-24T23:15:36.986343+0000","last_active":"2024-04-24T23:16:18.224974+0000","last_peered":"2024-04-24T23:16:18.224974+0000","last_clean":"2024-04-24T23:16:18.224974+0000","last_became_active":"2024-04-24T23:15:36.986026+0000","last_became_peered":"2024-04-24T23:15:36.986026+0000","last_unstale":"2024-04-24T23:16:18.224974+0000","last_undegraded":"2024-04-24T23:16:18.224974+0000","last_fullsized":"2024-04-24T23:16:18.224974+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:10:00.347826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.224857+0000","last_change":"2024-04-24T23:15:36.968538+0000","last_active":"2024-04-24T23:16:18.224857+0000","last_peered":"2024-04-24T23:16:18.224857+0000","last_clean":"2024-04-24T23:16:18.224857+0000","last_became_active":"2024-04-24T23:15:36.968263+0000","last_became_peered":"2024-04-24T23:15:36.968263+0000","last_unstale":"2024-04-24T23:16:18.224857+0000","last_undegraded":"2024-04-24T23:16:18.224857+0000","last_fullsized":"2024-04-24T23:16:18.224857+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:07:43.731738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.294600+0000","last_change":"2024-04-24T23:15:36.991008+0000","last_active":"2024-04-24T23:16:18.294600+0000","last_peered":"2024-04-24T23:16:18.294600+0000","last_clean":"2024-04-24T23:16:18.294600+0000","last_became_active":"2024-04-24T23:15:36.990764+0000","last_became_peered":"2024-04-24T23:15:36.990764+0000","last_unstale":"2024-04-24T23:16:18.294600+0000","last_undegraded":"2024-04-24T23:16:18.294600+0000","last_fullsized":"2024-04-24T23:16:18.294600+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:50:21.865233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.291911+0000","last_change":"2024-04-24T23:15:36.993550+0000","last_active":"2024-04-24T23:16:18.291911+0000","last_peered":"2024-04-24T23:16:18.291911+0000","last_clean":"2024-04-24T23:16:18.291911+0000","last_became_active":"2024-04-24T23:15:36.993310+0000","last_became_peered":"2024-04-24T23:15:36.993310+0000","last_unstale":"2024-04-24T23:16:18.291911+0000","last_undegraded":"2024-04-24T23:16:18.291911+0000","last_fullsized":"2024-04-24T23:16:18.291911+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:21:20.617940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.230900+0000","last_change":"2024-04-24T23:15:36.987810+0000","last_active":"2024-04-24T23:16:18.230900+0000","last_peered":"2024-04-24T23:16:18.230900+0000","last_clean":"2024-04-24T23:16:18.230900+0000","last_became_active":"2024-04-24T23:15:36.987540+0000","last_became_peered":"2024-04-24T23:15:36.987540+0000","last_unstale":"2024-04-24T23:16:18.230900+0000","last_undegraded":"2024-04-24T23:16:18.230900+0000","last_fullsized":"2024-04-24T23:16:18.230900+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:49:56.176021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.291530+0000","last_change":"2024-04-24T23:15:36.981187+0000","last_active":"2024-04-24T23:16:18.291530+0000","last_peered":"2024-04-24T23:16:18.291530+0000","last_clean":"2024-04-24T23:16:18.291530+0000","last_became_active":"2024-04-24T23:15:36.980838+0000","last_became_peered":"2024-04-24T23:15:36.980838+0000","last_unstale":"2024-04-24T23:16:18.291530+0000","last_undegraded":"2024-04-24T23:16:18.291530+0000","last_fullsized":"2024-04-24T23:16:18.291530+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:57:41.178367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.296378+0000","last_change":"2024-04-24T23:15:36.999238+0000","last_active":"2024-04-24T23:16:18.296378+0000","last_peered":"2024-04-24T23:16:18.296378+0000","last_clean":"2024-04-24T23:16:18.296378+0000","last_became_active":"2024-04-24T23:15:36.996245+0000","last_became_peered":"2024-04-24T23:15:36.996245+0000","last_unstale":"2024-04-24T23:16:18.296378+0000","last_undegraded":"2024-04-24T23:16:18.296378+0000","last_fullsized":"2024-04-24T23:16:18.296378+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:21:20.443640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.223856+0000","last_change":"2024-04-24T23:15:36.981219+0000","last_active":"2024-04-24T23:16:18.223856+0000","last_peered":"2024-04-24T23:16:18.223856+0000","last_clean":"2024-04-24T23:16:18.223856+0000","last_became_active":"2024-04-24T23:15:36.980911+0000","last_became_peered":"2024-04-24T23:15:36.980911+0000","last_unstale":"2024-04-24T23:16:18.223856+0000","last_undegraded":"2024-04-24T23:16:18.223856+0000","last_fullsized":"2024-04-24T23:16:18.223856+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:19:57.973402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.229789+0000","last_change":"2024-04-24T23:15:36.991418+0000","last_active":"2024-04-24T23:16:18.229789+0000","last_peered":"2024-04-24T23:16:18.229789+0000","last_clean":"2024-04-24T23:16:18.229789+0000","last_became_active":"2024-04-24T23:15:36.989792+0000","last_became_peered":"2024-04-24T23:15:36.989792+0000","last_unstale":"2024-04-24T23:16:18.229789+0000","last_undegraded":"2024-04-24T23:16:18.229789+0000","last_fullsized":"2024-04-24T23:16:18.229789+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:02:53.601311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.231144+0000","last_change":"2024-04-24T23:15:36.987810+0000","last_active":"2024-04-24T23:16:18.231144+0000","last_peered":"2024-04-24T23:16:18.231144+0000","last_clean":"2024-04-24T23:16:18.231144+0000","last_became_active":"2024-04-24T23:15:36.987494+0000","last_became_peered":"2024-04-24T23:15:36.987494+0000","last_unstale":"2024-04-24T23:16:18.231144+0000","last_undegraded":"2024-04-24T23:16:18.231144+0000","last_fullsized":"2024-04-24T23:16:18.231144+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:43:21.141172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.292884+0000","last_change":"2024-04-24T23:15:36.986576+0000","last_active":"2024-04-24T23:16:18.292884+0000","last_peered":"2024-04-24T23:16:18.292884+0000","last_clean":"2024-04-24T23:16:18.292884+0000","last_became_active":"2024-04-24T23:15:36.986272+0000","last_became_peered":"2024-04-24T23:15:36.986272+0000","last_unstale":"2024-04-24T23:16:18.292884+0000","last_undegraded":"2024-04-24T23:16:18.292884+0000","last_fullsized":"2024-04-24T23:16:18.292884+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:55:08.894945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":29,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.294151+0000","last_change":"2024-04-24T23:16:18.294151+0000","last_active":"2024-04-24T23:16:18.294151+0000","last_peered":"2024-04-24T23:16:18.294151+0000","last_clean":"2024-04-24T23:16:18.294151+0000","last_became_active":"2024-04-24T23:16:18.293579+0000","last_became_peered":"2024-04-24T23:16:18.293579+0000","last_unstale":"2024-04-24T23:16:18.294151+0000","last_undegraded":"2024-04-24T23:16:18.294151+0000","last_fullsized":"2024-04-24T23:16:18.294151+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:22:20.603555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.230543+0000","last_change":"2024-04-24T23:15:36.996521+0000","last_active":"2024-04-24T23:16:18.230543+0000","last_peered":"2024-04-24T23:16:18.230543+0000","last_clean":"2024-04-24T23:16:18.230543+0000","last_became_active":"2024-04-24T23:15:36.995507+0000","last_became_peered":"2024-04-24T23:15:36.995507+0000","last_unstale":"2024-04-24T23:16:18.230543+0000","last_undegraded":"2024-04-24T23:16:18.230543+0000","last_fullsized":"2024-04-24T23:16:18.230543+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:58:50.153471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.291317+0000","last_change":"2024-04-24T23:15:36.984994+0000","last_active":"2024-04-24T23:16:18.291317+0000","last_peered":"2024-04-24T23:16:18.291317+0000","last_clean":"2024-04-24T23:16:18.291317+0000","last_became_active":"2024-04-24T23:15:36.984741+0000","last_became_peered":"2024-04-24T23:15:36.984741+0000","last_unstale":"2024-04-24T23:16:18.291317+0000","last_undegraded":"2024-04-24T23:16:18.291317+0000","last_fullsized":"2024-04-24T23:16:18.291317+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:45:23.429836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.224730+0000","last_change":"2024-04-24T23:15:36.966165+0000","last_active":"2024-04-24T23:16:18.224730+0000","last_peered":"2024-04-24T23:16:18.224730+0000","last_clean":"2024-04-24T23:16:18.224730+0000","last_became_active":"2024-04-24T23:15:36.965877+0000","last_became_peered":"2024-04-24T23:15:36.965877+0000","last_unstale":"2024-04-24T23:16:18.224730+0000","last_undegraded":"2024-04-24T23:16:18.224730+0000","last_fullsized":"2024-04-24T23:16:18.224730+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:30:12.390978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.224249+0000","last_change":"2024-04-24T23:15:36.990667+0000","last_active":"2024-04-24T23:16:18.224249+0000","last_peered":"2024-04-24T23:16:18.224249+0000","last_clean":"2024-04-24T23:16:18.224249+0000","last_became_active":"2024-04-24T23:15:36.990383+0000","last_became_peered":"2024-04-24T23:15:36.990383+0000","last_unstale":"2024-04-24T23:16:18.224249+0000","last_undegraded":"2024-04-24T23:16:18.224249+0000","last_fullsized":"2024-04-24T23:16:18.224249+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:36:43.850046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.229311+0000","last_change":"2024-04-24T23:15:36.991686+0000","last_active":"2024-04-24T23:16:18.229311+0000","last_peered":"2024-04-24T23:16:18.229311+0000","last_clean":"2024-04-24T23:16:18.229311+0000","last_became_active":"2024-04-24T23:15:36.990155+0000","last_became_peered":"2024-04-24T23:15:36.990155+0000","last_unstale":"2024-04-24T23:16:18.229311+0000","last_undegraded":"2024-04-24T23:16:18.229311+0000","last_fullsized":"2024-04-24T23:16:18.229311+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:00:52.424019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.232389+0000","last_change":"2024-04-24T23:15:36.992958+0000","last_active":"2024-04-24T23:16:18.232389+0000","last_peered":"2024-04-24T23:16:18.232389+0000","last_clean":"2024-04-24T23:16:18.232389+0000","last_became_active":"2024-04-24T23:15:36.992608+0000","last_became_peered":"2024-04-24T23:15:36.992608+0000","last_unstale":"2024-04-24T23:16:18.232389+0000","last_undegraded":"2024-04-24T23:16:18.232389+0000","last_fullsized":"2024-04-24T23:16:18.232389+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:15:30.416763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.232332+0000","last_change":"2024-04-24T23:15:36.992943+0000","last_active":"2024-04-24T23:16:18.232332+0000","last_peered":"2024-04-24T23:16:18.232332+0000","last_clean":"2024-04-24T23:16:18.232332+0000","last_became_active":"2024-04-24T23:15:36.992567+0000","last_became_peered":"2024-04-24T23:15:36.992567+0000","last_unstale":"2024-04-24T23:16:18.232332+0000","last_undegraded":"2024-04-24T23:16:18.232332+0000","last_fullsized":"2024-04-24T23:16:18.232332+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:50:43.209232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.297503+0000","last_change":"2024-04-24T23:15:36.999697+0000","last_active":"2024-04-24T23:16:18.297503+0000","last_peered":"2024-04-24T23:16:18.297503+0000","last_clean":"2024-04-24T23:16:18.297503+0000","last_became_active":"2024-04-24T23:15:36.999413+0000","last_became_peered":"2024-04-24T23:15:36.999413+0000","last_unstale":"2024-04-24T23:16:18.297503+0000","last_undegraded":"2024-04-24T23:16:18.297503+0000","last_fullsized":"2024-04-24T23:16:18.297503+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:25:42.856471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.291309+0000","last_change":"2024-04-24T23:15:36.986742+0000","last_active":"2024-04-24T23:16:18.291309+0000","last_peered":"2024-04-24T23:16:18.291309+0000","last_clean":"2024-04-24T23:16:18.291309+0000","last_became_active":"2024-04-24T23:15:36.986013+0000","last_became_peered":"2024-04-24T23:15:36.986013+0000","last_unstale":"2024-04-24T23:16:18.291309+0000","last_undegraded":"2024-04-24T23:16:18.291309+0000","last_fullsized":"2024-04-24T23:16:18.291309+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:16:15.203151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.234253+0000","last_change":"2024-04-24T23:15:36.984336+0000","last_active":"2024-04-24T23:16:18.234253+0000","last_peered":"2024-04-24T23:16:18.234253+0000","last_clean":"2024-04-24T23:16:18.234253+0000","last_became_active":"2024-04-24T23:15:36.983966+0000","last_became_peered":"2024-04-24T23:15:36.983966+0000","last_unstale":"2024-04-24T23:16:18.234253+0000","last_undegraded":"2024-04-24T23:16:18.234253+0000","last_fullsized":"2024-04-24T23:16:18.234253+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:54:16.319608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.294392+0000","last_change":"2024-04-24T23:15:36.979079+0000","last_active":"2024-04-24T23:16:18.294392+0000","last_peered":"2024-04-24T23:16:18.294392+0000","last_clean":"2024-04-24T23:16:18.294392+0000","last_became_active":"2024-04-24T23:15:36.978706+0000","last_became_peered":"2024-04-24T23:15:36.978706+0000","last_unstale":"2024-04-24T23:16:18.294392+0000","last_undegraded":"2024-04-24T23:16:18.294392+0000","last_fullsized":"2024-04-24T23:16:18.294392+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:53:39.888150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.229376+0000","last_change":"2024-04-24T23:15:36.986008+0000","last_active":"2024-04-24T23:16:18.229376+0000","last_peered":"2024-04-24T23:16:18.229376+0000","last_clean":"2024-04-24T23:16:18.229376+0000","last_became_active":"2024-04-24T23:15:36.985552+0000","last_became_peered":"2024-04-24T23:15:36.985552+0000","last_unstale":"2024-04-24T23:16:18.229376+0000","last_undegraded":"2024-04-24T23:16:18.229376+0000","last_fullsized":"2024-04-24T23:16:18.229376+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:36:50.827503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.293789+0000","last_change":"2024-04-24T23:15:36.978673+0000","last_active":"2024-04-24T23:16:18.293789+0000","last_peered":"2024-04-24T23:16:18.293789+0000","last_clean":"2024-04-24T23:16:18.293789+0000","last_became_active":"2024-04-24T23:15:36.972744+0000","last_became_peered":"2024-04-24T23:15:36.972744+0000","last_unstale":"2024-04-24T23:16:18.293789+0000","last_undegraded":"2024-04-24T23:16:18.293789+0000","last_fullsized":"2024-04-24T23:16:18.293789+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:28:57.028825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.291690+0000","last_change":"2024-04-24T23:15:36.970238+0000","last_active":"2024-04-24T23:16:18.291690+0000","last_peered":"2024-04-24T23:16:18.291690+0000","last_clean":"2024-04-24T23:16:18.291690+0000","last_became_active":"2024-04-24T23:15:36.969751+0000","last_became_peered":"2024-04-24T23:15:36.969751+0000","last_unstale":"2024-04-24T23:16:18.291690+0000","last_undegraded":"2024-04-24T23:16:18.291690+0000","last_fullsized":"2024-04-24T23:16:18.291690+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:19:56.456801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.291790+0000","last_change":"2024-04-24T23:15:36.968825+0000","last_active":"2024-04-24T23:16:18.291790+0000","last_peered":"2024-04-24T23:16:18.291790+0000","last_clean":"2024-04-24T23:16:18.291790+0000","last_became_active":"2024-04-24T23:15:36.968543+0000","last_became_peered":"2024-04-24T23:15:36.968543+0000","last_unstale":"2024-04-24T23:16:18.291790+0000","last_undegraded":"2024-04-24T23:16:18.291790+0000","last_fullsized":"2024-04-24T23:16:18.291790+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:36:48.795062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-24T23:16:18.229059+0000","last_change":"2024-04-24T23:15:36.992800+0000","last_active":"2024-04-24T23:16:18.229059+0000","last_peered":"2024-04-24T23:16:18.229059+0000","last_clean":"2024-04-24T23:16:18.229059+0000","last_became_active":"2024-04-24T23:15:36.992476+0000","last_became_peered":"2024-04-24T23:15:36.992476+0000","last_unstale":"2024-04-24T23:16:18.229059+0000","last_undegraded":"2024-04-24T23:16:18.229059+0000","last_fullsized":"2024-04-24T23:16:18.229059+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:01:21.103559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":10522,"num_objects":98,"num_object_clones":0,"num_object_copies":196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":211,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":21044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":185,"ondisk_log_size":185,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574868,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27324,"kb_used_data":308,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344516,"statfs":{"total":96636764160,"available":96608784384,"internally_reserved":0,"allocated":315392,"data_stored":114972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574868,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":292,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":299008,"data_stored":116298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574868,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27892,"kb_used_data":876,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343948,"statfs":{"total":96636764160,"available":96608202752,"internally_reserved":0,"allocated":897024,"data_stored":705112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574869,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27344,"kb_used_data":328,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344496,"statfs":{"total":96636764160,"available":96608763904,"internally_reserved":0,"allocated":335872,"data_stored":119856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574869,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27332,"kb_used_data":316,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344508,"statfs":{"total":96636764160,"available":96608776192,"internally_reserved":0,"allocated":323584,"data_stored":116640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574868,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":292,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":299008,"data_stored":118086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574867,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27888,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343952,"statfs":{"total":96636764160,"available":96608206848,"internally_reserved":0,"allocated":892928,"data_stored":705700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574868,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27324,"kb_used_data":308,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344516,"statfs":{"total":96636764160,"available":96608784384,"internally_reserved":0,"allocated":315392,"data_stored":116460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2542,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":5938,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1054,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T23:16:49.197 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T23:16:49.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:49.443+0000 7f3eea600640 1 Processor -- start 2024-04-24T23:16:49.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:49.444+0000 7f3eea600640 1 -- start start 2024-04-24T23:16:49.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:49.445+0000 7f3eea600640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f3ee40c6340 0x7f3ee4157650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:49.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:49.445+0000 7f3eea600640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3ee4071680 con 0x7f3ee4072320 2024-04-24T23:16:49.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:49.446+0000 7f3eea600640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3ee4071960 con 0x7f3ee4151be0 2024-04-24T23:16:49.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:49.446+0000 7f3eea600640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f3ee415abc0 con 0x7f3ee40c6340 2024-04-24T23:16:49.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:49.446+0000 7f3ee9c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f3ee40c6340 0x7f3ee4157650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:49.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:49.446+0000 7f3ee3e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f3ee4072320 0x7f3ee415ff20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:55228/0 (socket says 172.21.15.6:55228) 2024-04-24T23:16:49.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:49.446+0000 7f3ee3e00640 1 -- 172.21.15.6:0/2840477249 learned_addr learned my addr 172.21.15.6:0/2840477249 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:16:49.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:49.446+0000 7f3ee9c00640 1 -- 172.21.15.6:0/2840477249 >> v1:172.21.15.190:6789/0 conn(0x7f3ee4151be0 legacy=0x7f3ee40cc600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:49.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:49.446+0000 7f3ee9c00640 1 -- 172.21.15.6:0/2840477249 >> v1:172.21.15.6:6789/0 conn(0x7f3ee4072320 legacy=0x7f3ee415ff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:49.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:49.446+0000 7f3ee3400640 1 -- 172.21.15.6:0/2840477249 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3376294793 0 0) 0x7f3ee4071960 con 0x7f3ee4151be0 2024-04-24T23:16:49.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:49.446+0000 7f3ee9c00640 1 -- 172.21.15.6:0/2840477249 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ee415ae10 con 0x7f3ee40c6340 2024-04-24T23:16:49.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:49.448+0000 7f3ee9c00640 1 --2- 172.21.15.6:0/2840477249 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3ee40c6340 0x7f3ee4157650 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f3ee415ad40 tx=0x7f3edc02a4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=395c298d3de8298a server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:49.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:49.448+0000 7f3ee3400640 1 -- 172.21.15.6:0/2840477249 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3edc007740 con 0x7f3ee40c6340 2024-04-24T23:16:49.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:49.448+0000 7f3ee3400640 1 -- 172.21.15.6:0/2840477249 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3edc0078e0 con 0x7f3ee40c6340 2024-04-24T23:16:49.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:49.448+0000 7f3ee3400640 1 -- 172.21.15.6:0/2840477249 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3edc007cb0 con 0x7f3ee40c6340 2024-04-24T23:16:49.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:49.448+0000 7f3eea600640 1 -- 172.21.15.6:0/2840477249 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3ee40c6340 msgr2=0x7f3ee4157650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:49.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:49.448+0000 7f3eea600640 1 --2- 172.21.15.6:0/2840477249 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3ee40c6340 0x7f3ee4157650 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f3ee415ad40 tx=0x7f3edc02a4d0 comp rx=0 tx=0).stop 2024-04-24T23:16:49.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:49.449+0000 7f3eea600640 1 -- 172.21.15.6:0/2840477249 shutdown_connections 2024-04-24T23:16:49.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:49.449+0000 7f3eea600640 1 --2- 172.21.15.6:0/2840477249 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3ee40c6340 0x7f3ee4157650 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:49.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:49.449+0000 7f3eea600640 1 -- 172.21.15.6:0/2840477249 >> 172.21.15.6:0/2840477249 conn(0x7f3ee406ed80 msgr2=0x7f3ee406f190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:49.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:49.449+0000 7f3eea600640 1 -- 172.21.15.6:0/2840477249 shutdown_connections 2024-04-24T23:16:49.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:49.449+0000 7f3eea600640 1 -- 172.21.15.6:0/2840477249 wait complete. 2024-04-24T23:16:49.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:49.450+0000 7f3eea600640 1 Processor -- start 2024-04-24T23:16:49.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:49.451+0000 7f3eea600640 1 -- start start 2024-04-24T23:16:49.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:49.451+0000 7f3eea600640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3ee4072320 0x7f3ee4144dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:49.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:49.451+0000 7f3eea600640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3ee40c6340 0x7f3ee4149840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:49.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:49.451+0000 7f3eea600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3ee4151be0 0x7f3ee41f0410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:49.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:49.451+0000 7f3eea600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f3ee40c36e0 con 0x7f3ee4151be0 2024-04-24T23:16:49.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:49.451+0000 7f3eea600640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f3ee40c3ab0 con 0x7f3ee40c6340 2024-04-24T23:16:49.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:49.451+0000 7f3eea600640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f3ee40c3860 con 0x7f3ee4072320 2024-04-24T23:16:49.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:49.452+0000 7f3ee9200640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3ee4072320 0x7f3ee4144dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:49.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:49.452+0000 7f3ee3e00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3ee40c6340 0x7f3ee4149840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:49.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:49.452+0000 7f3ee9200640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3ee4072320 0x7f3ee4144dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.6:59788/0 (socket says 172.21.15.6:59788) 2024-04-24T23:16:49.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:49.452+0000 7f3ee9200640 1 -- 172.21.15.6:0/3402101128 learned_addr learned my addr 172.21.15.6:0/3402101128 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:16:49.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:49.452+0000 7f3ee9c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3ee4151be0 0x7f3ee41f0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:49.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:49.452+0000 7f3ee9200640 1 -- 172.21.15.6:0/3402101128 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3ee40c6340 msgr2=0x7f3ee4149840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:49.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:49.452+0000 7f3ee9200640 1 --2- 172.21.15.6:0/3402101128 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3ee40c6340 0x7f3ee4149840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:49.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:49.452+0000 7f3ee9200640 1 -- 172.21.15.6:0/3402101128 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3ee4151be0 msgr2=0x7f3ee41f0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:49.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:49.452+0000 7f3ee9200640 1 --2- 172.21.15.6:0/3402101128 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3ee4151be0 0x7f3ee41f0410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:49.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:49.452+0000 7f3ee9200640 1 -- 172.21.15.6:0/3402101128 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ee414be80 con 0x7f3ee4072320 2024-04-24T23:16:49.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:49.453+0000 7f3ee9c00640 1 --2- 172.21.15.6:0/3402101128 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3ee4151be0 0x7f3ee41f0410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:16:49.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:49.453+0000 7f3ee9200640 1 --2- 172.21.15.6:0/3402101128 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3ee4072320 0x7f3ee4144dd0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f3ed800c360 tx=0x7f3ed800c820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:49.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:49.453+0000 7f3ee0c00640 1 -- 172.21.15.6:0/3402101128 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ed8017070 con 0x7f3ee4072320 2024-04-24T23:16:49.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:49.453+0000 7f3ee0c00640 1 -- 172.21.15.6:0/3402101128 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ed80035f0 con 0x7f3ee4072320 2024-04-24T23:16:49.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:49.453+0000 7f3ee0c00640 1 -- 172.21.15.6:0/3402101128 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ed80039e0 con 0x7f3ee4072320 2024-04-24T23:16:49.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:49.453+0000 7f3eea600640 1 -- 172.21.15.6:0/3402101128 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ee414b5e0 con 0x7f3ee4072320 2024-04-24T23:16:49.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:49.453+0000 7f3eea600640 1 -- 172.21.15.6:0/3402101128 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ee414bba0 con 0x7f3ee4072320 2024-04-24T23:16:49.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:49.454+0000 7f3eea600640 1 -- 172.21.15.6:0/3402101128 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ee4075b30 con 0x7f3ee4072320 2024-04-24T23:16:49.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:49.455+0000 7f3ee0c00640 1 -- 172.21.15.6:0/3402101128 <== mon.2 v2:172.21.15.190:3301/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f3ed801a070 con 0x7f3ee4072320 2024-04-24T23:16:49.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:49.456+0000 7f3ee0c00640 1 --2- 172.21.15.6:0/3402101128 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f3ec406be30 0x7f3ec406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:49.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:49.456+0000 7f3ee0c00640 1 -- 172.21.15.6:0/3402101128 <== mon.2 v2:172.21.15.190:3301/0 5 ==== osd_map(26..26 src has 1..26) ==== 6590+0+0 (secure 0 0 0) 0x7f3ed808a620 con 0x7f3ee4072320 2024-04-24T23:16:49.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:49.456+0000 7f3ee3e00640 1 --2- 172.21.15.6:0/3402101128 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f3ec406be30 0x7f3ec406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:49.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:49.457+0000 7f3ee3e00640 1 --2- 172.21.15.6:0/3402101128 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f3ec406be30 0x7f3ec406e2f0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f3ee40c4a40 tx=0x7f3ed0007e20 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:49.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:49.461+0000 7f3ee0c00640 1 -- 172.21.15.6:0/3402101128 <== mon.2 v2:172.21.15.190:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f3ed80584e0 con 0x7f3ee4072320 2024-04-24T23:16:49.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:49.665+0000 7f3eea600640 1 -- 172.21.15.6:0/3402101128 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3ee41453b0 con 0x7f3ee4072320 2024-04-24T23:16:49.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:49.666+0000 7f3ee0c00640 1 -- 172.21.15.6:0/3402101128 <== mon.2 v2:172.21.15.190:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+18554 (secure 0 0 0) 0x7f3ed805c7e0 con 0x7f3ee4072320 2024-04-24T23:16:49.669 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:16:49.669 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":26,"fsid":"18bef68d-5e6d-4ec5-862b-b6f206e2588d","created":"2024-04-24T23:15:12.535072+0000","modified":"2024-04-24T23:16:18.218008+0000","last_up_change":"2024-04-24T23:15:23.773223+0000","last_in_change":"2024-04-24T23:15:14.905328+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T23:15:24.747441+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-24T23:15:34.051800+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-24T23:15:35.436870+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-24T23:15:39.034257+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"3b7a212d-d498-462e-90bf-ed493c803a09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":3022569437},{"type":"v1","addr":"172.21.15.6:6809","nonce":3022569437}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":3022569437},{"type":"v1","addr":"172.21.15.6:6811","nonce":3022569437}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":3022569437},{"type":"v1","addr":"172.21.15.6:6815","nonce":3022569437}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":3022569437},{"type":"v1","addr":"172.21.15.6:6813","nonce":3022569437}]},"public_addr":"172.21.15.6:6809/3022569437","cluster_addr":"172.21.15.6:6811/3022569437","heartbeat_back_addr":"172.21.15.6:6815/3022569437","heartbeat_front_addr":"172.21.15.6:6813/3022569437","state":["exists","up"]},{"osd":1,"uuid":"7c97f5a2-1735-4775-9f15-eb50099b19d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":3687836888},{"type":"v1","addr":"172.21.15.6:6817","nonce":3687836888}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":3687836888},{"type":"v1","addr":"172.21.15.6:6819","nonce":3687836888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":3687836888},{"type":"v1","addr":"172.21.15.6:6823","nonce":3687836888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":3687836888},{"type":"v1","addr":"172.21.15.6:6821","nonce":3687836888}]},"public_addr":"172.21.15.6:6817/3687836888","cluster_addr":"172.21.15.6:6819/3687836888","heartbeat_back_addr":"172.21.15.6:6823/3687836888","heartbeat_front_addr":"172.21.15.6:6821/3687836888","state":["exists","up"]},{"osd":2,"uuid":"d86bbc18-438c-4ec2-8018-ba7dd78f790b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":1020213724},{"type":"v1","addr":"172.21.15.6:6801","nonce":1020213724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":1020213724},{"type":"v1","addr":"172.21.15.6:6803","nonce":1020213724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":1020213724},{"type":"v1","addr":"172.21.15.6:6807","nonce":1020213724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":1020213724},{"type":"v1","addr":"172.21.15.6:6805","nonce":1020213724}]},"public_addr":"172.21.15.6:6801/1020213724","cluster_addr":"172.21.15.6:6803/1020213724","heartbeat_back_addr":"172.21.15.6:6807/1020213724","heartbeat_front_addr":"172.21.15.6:6805/1020213724","state":["exists","up"]},{"osd":3,"uuid":"3754ae16-4248-40dd-95f9-0f59c0eb5426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6824","nonce":3163883468},{"type":"v1","addr":"172.21.15.6:6825","nonce":3163883468}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":3163883468},{"type":"v1","addr":"172.21.15.6:6827","nonce":3163883468}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":3163883468},{"type":"v1","addr":"172.21.15.6:6831","nonce":3163883468}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":3163883468},{"type":"v1","addr":"172.21.15.6:6829","nonce":3163883468}]},"public_addr":"172.21.15.6:6825/3163883468","cluster_addr":"172.21.15.6:6827/3163883468","heartbeat_back_addr":"172.21.15.6:6831/3163883468","heartbeat_front_addr":"172.21.15.6:6829/3163883468","state":["exists","up"]},{"osd":4,"uuid":"8b2913a0-d006-479c-84ed-5b21e0e6c392","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":1989500934},{"type":"v1","addr":"172.21.15.190:6809","nonce":1989500934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":1989500934},{"type":"v1","addr":"172.21.15.190:6811","nonce":1989500934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":1989500934},{"type":"v1","addr":"172.21.15.190:6815","nonce":1989500934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":1989500934},{"type":"v1","addr":"172.21.15.190:6813","nonce":1989500934}]},"public_addr":"172.21.15.190:6809/1989500934","cluster_addr":"172.21.15.190:6811/1989500934","heartbeat_back_addr":"172.21.15.190:6815/1989500934","heartbeat_front_addr":"172.21.15.190:6813/1989500934","state":["exists","up"]},{"osd":5,"uuid":"499b79d0-0585-4dae-ab69-196a551793da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":3687548412},{"type":"v1","addr":"172.21.15.190:6801","nonce":3687548412}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":3687548412},{"type":"v1","addr":"172.21.15.190:6803","nonce":3687548412}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":3687548412},{"type":"v1","addr":"172.21.15.190:6807","nonce":3687548412}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":3687548412},{"type":"v1","addr":"172.21.15.190:6805","nonce":3687548412}]},"public_addr":"172.21.15.190:6801/3687548412","cluster_addr":"172.21.15.190:6803/3687548412","heartbeat_back_addr":"172.21.15.190:6807/3687548412","heartbeat_front_addr":"172.21.15.190:6805/3687548412","state":["exists","up"]},{"osd":6,"uuid":"6ba6df3b-4a6b-4a89-a890-0f79f387d4d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":870571564},{"type":"v1","addr":"172.21.15.190:6817","nonce":870571564}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":870571564},{"type":"v1","addr":"172.21.15.190:6819","nonce":870571564}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":870571564},{"type":"v1","addr":"172.21.15.190:6823","nonce":870571564}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":870571564},{"type":"v1","addr":"172.21.15.190:6821","nonce":870571564}]},"public_addr":"172.21.15.190:6817/870571564","cluster_addr":"172.21.15.190:6819/870571564","heartbeat_back_addr":"172.21.15.190:6823/870571564","heartbeat_front_addr":"172.21.15.190:6821/870571564","state":["exists","up"]},{"osd":7,"uuid":"e9668b1c-a000-4332-86e8-4efd73b02409","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":123639197},{"type":"v1","addr":"172.21.15.190:6825","nonce":123639197}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":123639197},{"type":"v1","addr":"172.21.15.190:6827","nonce":123639197}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":123639197},{"type":"v1","addr":"172.21.15.190:6831","nonce":123639197}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":123639197},{"type":"v1","addr":"172.21.15.190:6829","nonce":123639197}]},"public_addr":"172.21.15.190:6825/123639197","cluster_addr":"172.21.15.190:6827/123639197","heartbeat_back_addr":"172.21.15.190:6831/123639197","heartbeat_front_addr":"172.21.15.190:6829/123639197","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.321646+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.363873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.466963+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.531989+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:19.990707+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.043543+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.032492+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.115004+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.2f","mappings":[{"from":0,"to":7}]},{"pgid":"4.19","mappings":[{"from":7,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T23:16:49.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:49.670+0000 7f3eea600640 1 -- 172.21.15.6:0/3402101128 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f3ec406be30 msgr2=0x7f3ec406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:49.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:49.670+0000 7f3eea600640 1 --2- 172.21.15.6:0/3402101128 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f3ec406be30 0x7f3ec406e2f0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f3ee40c4a40 tx=0x7f3ed0007e20 comp rx=0 tx=0).stop 2024-04-24T23:16:49.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:49.670+0000 7f3eea600640 1 -- 172.21.15.6:0/3402101128 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3ee4072320 msgr2=0x7f3ee4144dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:49.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:49.670+0000 7f3eea600640 1 --2- 172.21.15.6:0/3402101128 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3ee4072320 0x7f3ee4144dd0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f3ed800c360 tx=0x7f3ed800c820 comp rx=0 tx=0).stop 2024-04-24T23:16:49.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:49.670+0000 7f3eea600640 1 -- 172.21.15.6:0/3402101128 shutdown_connections 2024-04-24T23:16:49.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:49.670+0000 7f3eea600640 1 --2- 172.21.15.6:0/3402101128 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f3ec406be30 0x7f3ec406e2f0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:49.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:49.670+0000 7f3eea600640 1 --2- 172.21.15.6:0/3402101128 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3ee4151be0 0x7f3ee41f0410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:49.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:49.670+0000 7f3eea600640 1 --2- 172.21.15.6:0/3402101128 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3ee40c6340 0x7f3ee4149840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:49.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:49.670+0000 7f3eea600640 1 --2- 172.21.15.6:0/3402101128 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3ee4072320 0x7f3ee4144dd0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:49.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:49.670+0000 7f3eea600640 1 -- 172.21.15.6:0/3402101128 >> 172.21.15.6:0/3402101128 conn(0x7f3ee406ed80 msgr2=0x7f3ee4150de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:49.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:49.670+0000 7f3eea600640 1 -- 172.21.15.6:0/3402101128 shutdown_connections 2024-04-24T23:16:49.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:49.670+0000 7f3eea600640 1 -- 172.21.15.6:0/3402101128 wait complete. 2024-04-24T23:16:49.694 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T23:15:24.747441+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-24T23:15:34.051800+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-24T23:15:35.436870+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-24T23:15:39.034257+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-24T23:16:49.694 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-24T23:16:49.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:49.892+0000 7f2d1a200640 1 Processor -- start 2024-04-24T23:16:49.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:49.892+0000 7f2d1a200640 1 -- start start 2024-04-24T23:16:49.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:49.894+0000 7f2d1a200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f2d140c5e40 0x7f2d140c6ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:49.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:49.894+0000 7f2d1a200640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2d140c8570 0x7f2d141524a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:49.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:49.894+0000 7f2d1a200640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2d140cca80 con 0x7f2d140c5a60 2024-04-24T23:16:49.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:49.894+0000 7f2d1a200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f2d1415c440 con 0x7f2d140c5e40 2024-04-24T23:16:49.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:49.894+0000 7f2d1a200640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f2d140ccd60 con 0x7f2d140c8570 2024-04-24T23:16:49.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:49.894+0000 7f2d13e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f2d140c5e40 0x7f2d140c6ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:49.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:49.894+0000 7f2d13e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f2d140c5e40 0x7f2d140c6ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36398/0 (socket says 172.21.15.6:36398) 2024-04-24T23:16:49.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:49.894+0000 7f2d13e00640 1 -- 172.21.15.6:0/1226076136 learned_addr learned my addr 172.21.15.6:0/1226076136 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:16:49.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:49.894+0000 7f2d19800640 1 --2- 172.21.15.6:0/1226076136 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2d140c8570 0x7f2d141524a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:49.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:49.894+0000 7f2d13e00640 1 -- 172.21.15.6:0/1226076136 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2d140c8570 msgr2=0x7f2d141524a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:16:49.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:49.894+0000 7f2d13e00640 1 --2- 172.21.15.6:0/1226076136 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2d140c8570 0x7f2d141524a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:49.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:49.894+0000 7f2d13e00640 1 -- 172.21.15.6:0/1226076136 >> v1:172.21.15.190:6789/0 conn(0x7f2d140c5a60 legacy=0x7f2d1415bd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:49.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:49.895+0000 7f2d13e00640 1 -- 172.21.15.6:0/1226076136 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d1406eab0 con 0x7f2d140c5e40 2024-04-24T23:16:49.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:49.896+0000 7f2d13e00640 1 --2- 172.21.15.6:0/1226076136 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2d140c5e40 0x7f2d140c6ae0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f2d00009870 tx=0x7f2d0002a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61a31109aadc75a4 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:49.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:49.896+0000 7f2d13400640 1 -- 172.21.15.6:0/1226076136 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d00037070 con 0x7f2d140c5e40 2024-04-24T23:16:49.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:49.896+0000 7f2d13400640 1 -- 172.21.15.6:0/1226076136 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d00032440 con 0x7f2d140c5e40 2024-04-24T23:16:49.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:49.896+0000 7f2d13400640 1 -- 172.21.15.6:0/1226076136 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d00032810 con 0x7f2d140c5e40 2024-04-24T23:16:49.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:49.896+0000 7f2d1a200640 1 -- 172.21.15.6:0/1226076136 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2d140c5e40 msgr2=0x7f2d140c6ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:49.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:49.896+0000 7f2d1a200640 1 --2- 172.21.15.6:0/1226076136 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2d140c5e40 0x7f2d140c6ae0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f2d00009870 tx=0x7f2d0002a1c0 comp rx=0 tx=0).stop 2024-04-24T23:16:49.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:49.897+0000 7f2d1a200640 1 -- 172.21.15.6:0/1226076136 shutdown_connections 2024-04-24T23:16:49.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:49.897+0000 7f2d1a200640 1 --2- 172.21.15.6:0/1226076136 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2d140c8570 0x7f2d141524a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:49.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:49.897+0000 7f2d1a200640 1 --2- 172.21.15.6:0/1226076136 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2d140c5e40 0x7f2d140c6ae0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:49.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:49.897+0000 7f2d1a200640 1 -- 172.21.15.6:0/1226076136 >> 172.21.15.6:0/1226076136 conn(0x7f2d1406cc40 msgr2=0x7f2d1406d050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:49.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:49.897+0000 7f2d1a200640 1 -- 172.21.15.6:0/1226076136 shutdown_connections 2024-04-24T23:16:49.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:49.897+0000 7f2d1a200640 1 -- 172.21.15.6:0/1226076136 wait complete. 2024-04-24T23:16:49.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:49.898+0000 7f2d1a200640 1 Processor -- start 2024-04-24T23:16:49.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:49.899+0000 7f2d1a200640 1 -- start start 2024-04-24T23:16:49.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:49.900+0000 7f2d1a200640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2d140c5a60 0x7f2d141e2060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:49.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:49.900+0000 7f2d1a200640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2d140c5e40 0x7f2d1414e060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:49.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:49.900+0000 7f2d1a200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2d140c8570 0x7f2d140c31d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:49.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:49.900+0000 7f2d1a200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f2d1406f590 con 0x7f2d140c8570 2024-04-24T23:16:49.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:49.900+0000 7f2d1a200640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f2d1406f070 con 0x7f2d140c5a60 2024-04-24T23:16:49.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:49.900+0000 7f2d1a200640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f2d140c5640 con 0x7f2d140c5e40 2024-04-24T23:16:49.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:49.900+0000 7f2d19800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2d140c8570 0x7f2d140c31d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:49.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:49.900+0000 7f2d18e00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2d140c5a60 0x7f2d141e2060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:49.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:49.900+0000 7f2d13e00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2d140c5e40 0x7f2d1414e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:49.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:49.900+0000 7f2d19800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2d140c8570 0x7f2d140c31d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36414/0 (socket says 172.21.15.6:36414) 2024-04-24T23:16:49.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:49.900+0000 7f2d19800640 1 -- 172.21.15.6:0/3148775468 learned_addr learned my addr 172.21.15.6:0/3148775468 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:16:49.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:49.900+0000 7f2d13e00640 1 -- 172.21.15.6:0/3148775468 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2d140c5a60 msgr2=0x7f2d141e2060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:49.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:49.900+0000 7f2d13e00640 1 --2- 172.21.15.6:0/3148775468 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2d140c5a60 0x7f2d141e2060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:49.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:49.900+0000 7f2d13e00640 1 -- 172.21.15.6:0/3148775468 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2d140c8570 msgr2=0x7f2d140c31d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:49.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:49.900+0000 7f2d13e00640 1 --2- 172.21.15.6:0/3148775468 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2d140c8570 0x7f2d140c31d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:49.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:49.900+0000 7f2d13e00640 1 -- 172.21.15.6:0/3148775468 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d141e3ae0 con 0x7f2d140c5e40 2024-04-24T23:16:49.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:49.901+0000 7f2d13e00640 1 --2- 172.21.15.6:0/3148775468 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2d140c5e40 0x7f2d1414e060 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f2d00031000 tx=0x7f2d000046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:49.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:49.901+0000 7f2d10c00640 1 -- 172.21.15.6:0/3148775468 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d00041020 con 0x7f2d140c5e40 2024-04-24T23:16:49.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:49.901+0000 7f2d10c00640 1 -- 172.21.15.6:0/3148775468 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d00030660 con 0x7f2d140c5e40 2024-04-24T23:16:49.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:49.901+0000 7f2d1a200640 1 -- 172.21.15.6:0/3148775468 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d141e28b0 con 0x7f2d140c5e40 2024-04-24T23:16:49.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:49.901+0000 7f2d10c00640 1 -- 172.21.15.6:0/3148775468 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d00037040 con 0x7f2d140c5e40 2024-04-24T23:16:49.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:49.901+0000 7f2d1a200640 1 -- 172.21.15.6:0/3148775468 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d140c4b60 con 0x7f2d140c5e40 2024-04-24T23:16:49.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:49.902+0000 7f2d1a200640 1 -- 172.21.15.6:0/3148775468 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d14075b30 con 0x7f2d140c5e40 2024-04-24T23:16:49.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:49.903+0000 7f2d10c00640 1 -- 172.21.15.6:0/3148775468 <== mon.2 v2:172.21.15.190:3301/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f2d00044020 con 0x7f2d140c5e40 2024-04-24T23:16:49.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:49.904+0000 7f2d10c00640 1 --2- 172.21.15.6:0/3148775468 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f2cf406be30 0x7f2cf406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:49.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:49.905+0000 7f2d10c00640 1 -- 172.21.15.6:0/3148775468 <== mon.2 v2:172.21.15.190:3301/0 5 ==== osd_map(26..26 src has 1..26) ==== 6590+0+0 (secure 0 0 0) 0x7f2d000ab810 con 0x7f2d140c5e40 2024-04-24T23:16:49.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:49.905+0000 7f2d18e00640 1 --2- 172.21.15.6:0/3148775468 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f2cf406be30 0x7f2cf406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:49.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:49.906+0000 7f2d18e00640 1 --2- 172.21.15.6:0/3148775468 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f2cf406be30 0x7f2cf406e2f0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f2d08001780 tx=0x7f2d08009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:49.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:49.909+0000 7f2d10c00640 1 -- 172.21.15.6:0/3148775468 <== mon.2 v2:172.21.15.190:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f2d0004b030 con 0x7f2d140c5e40 2024-04-24T23:16:50.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.112+0000 7f2d1a200640 1 -- 172.21.15.6:0/3148775468 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f2d1414d5f0 con 0x7f2d140c5e40 2024-04-24T23:16:50.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.113+0000 7f2d10c00640 1 -- 172.21.15.6:0/3148775468 <== mon.2 v2:172.21.15.190:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) ==== 93+0+10 (secure 0 0 0) 0x7f2d000796d0 con 0x7f2d140c5e40 2024-04-24T23:16:50.116 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-04-24T23:16:50.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.116+0000 7f2d1a200640 1 -- 172.21.15.6:0/3148775468 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f2cf406be30 msgr2=0x7f2cf406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:50.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.116+0000 7f2d1a200640 1 --2- 172.21.15.6:0/3148775468 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f2cf406be30 0x7f2cf406e2f0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f2d08001780 tx=0x7f2d08009040 comp rx=0 tx=0).stop 2024-04-24T23:16:50.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.116+0000 7f2d1a200640 1 -- 172.21.15.6:0/3148775468 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2d140c5e40 msgr2=0x7f2d1414e060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:50.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.116+0000 7f2d1a200640 1 --2- 172.21.15.6:0/3148775468 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2d140c5e40 0x7f2d1414e060 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f2d00031000 tx=0x7f2d000046c0 comp rx=0 tx=0).stop 2024-04-24T23:16:50.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.116+0000 7f2d1a200640 1 -- 172.21.15.6:0/3148775468 shutdown_connections 2024-04-24T23:16:50.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.116+0000 7f2d1a200640 1 --2- 172.21.15.6:0/3148775468 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f2cf406be30 0x7f2cf406e2f0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:50.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.116+0000 7f2d1a200640 1 --2- 172.21.15.6:0/3148775468 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2d140c5e40 0x7f2d1414e060 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:50.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.116+0000 7f2d1a200640 1 --2- 172.21.15.6:0/3148775468 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2d140c8570 0x7f2d140c31d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:50.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.116+0000 7f2d1a200640 1 --2- 172.21.15.6:0/3148775468 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2d140c5a60 0x7f2d141e2060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:50.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.116+0000 7f2d1a200640 1 -- 172.21.15.6:0/3148775468 >> 172.21.15.6:0/3148775468 conn(0x7f2d1406cc40 msgr2=0x7f2d141566b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:50.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.116+0000 7f2d1a200640 1 -- 172.21.15.6:0/3148775468 shutdown_connections 2024-04-24T23:16:50.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.116+0000 7f2d1a200640 1 -- 172.21.15.6:0/3148775468 wait complete. 2024-04-24T23:16:50.137 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-24T23:16:50.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.349+0000 7ff347200640 1 Processor -- start 2024-04-24T23:16:50.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.349+0000 7ff347200640 1 -- start start 2024-04-24T23:16:50.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.351+0000 7ff347200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ff3400c5e40 0x7ff3400c6ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:50.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.351+0000 7ff347200640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff3400c8570 0x7ff3401524a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:50.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.351+0000 7ff347200640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff3400cca80 con 0x7ff3400c5a60 2024-04-24T23:16:50.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.351+0000 7ff347200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7ff34015c440 con 0x7ff3400c5e40 2024-04-24T23:16:50.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.351+0000 7ff347200640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7ff3400ccd60 con 0x7ff3400c8570 2024-04-24T23:16:50.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.351+0000 7ff345400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ff3400c5e40 0x7ff3400c6ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:50.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.351+0000 7ff345e00640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7ff3400c5a60 0x7ff34015bd30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.6:36942/0 (socket says 172.21.15.6:36942) 2024-04-24T23:16:50.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.352+0000 7ff345e00640 1 -- 172.21.15.6:0/2798924636 learned_addr learned my addr 172.21.15.6:0/2798924636 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:16:50.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.352+0000 7ff346800640 1 --2- 172.21.15.6:0/2798924636 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff3400c8570 0x7ff3401524a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:50.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.352+0000 7ff344a00640 1 -- 172.21.15.6:0/2798924636 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2178642249 0 0) 0x7ff3400cca80 con 0x7ff3400c5a60 2024-04-24T23:16:50.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.352+0000 7ff345400640 1 -- 172.21.15.6:0/2798924636 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff3400c8570 msgr2=0x7ff3401524a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:16:50.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.352+0000 7ff345400640 1 --2- 172.21.15.6:0/2798924636 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff3400c8570 0x7ff3401524a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:50.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.352+0000 7ff345400640 1 -- 172.21.15.6:0/2798924636 >> v1:172.21.15.190:6789/0 conn(0x7ff3400c5a60 legacy=0x7ff34015bd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:50.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.352+0000 7ff345400640 1 -- 172.21.15.6:0/2798924636 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff34006eab0 con 0x7ff3400c5e40 2024-04-24T23:16:50.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.353+0000 7ff345400640 1 --2- 172.21.15.6:0/2798924636 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff3400c5e40 0x7ff3400c6ae0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7ff3300097a0 tx=0x7ff33002a100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e35d846993cdfe5a server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:50.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.354+0000 7ff344a00640 1 -- 172.21.15.6:0/2798924636 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff330037070 con 0x7ff3400c5e40 2024-04-24T23:16:50.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.354+0000 7ff344a00640 1 -- 172.21.15.6:0/2798924636 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff33002ae50 con 0x7ff3400c5e40 2024-04-24T23:16:50.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.354+0000 7ff344a00640 1 -- 172.21.15.6:0/2798924636 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff330032660 con 0x7ff3400c5e40 2024-04-24T23:16:50.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.354+0000 7ff347200640 1 -- 172.21.15.6:0/2798924636 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff3400c5e40 msgr2=0x7ff3400c6ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:50.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.354+0000 7ff347200640 1 --2- 172.21.15.6:0/2798924636 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff3400c5e40 0x7ff3400c6ae0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7ff3300097a0 tx=0x7ff33002a100 comp rx=0 tx=0).stop 2024-04-24T23:16:50.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.354+0000 7ff347200640 1 -- 172.21.15.6:0/2798924636 shutdown_connections 2024-04-24T23:16:50.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.354+0000 7ff347200640 1 --2- 172.21.15.6:0/2798924636 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff3400c8570 0x7ff3401524a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:50.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.354+0000 7ff347200640 1 --2- 172.21.15.6:0/2798924636 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff3400c5e40 0x7ff3400c6ae0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:50.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.354+0000 7ff347200640 1 -- 172.21.15.6:0/2798924636 >> 172.21.15.6:0/2798924636 conn(0x7ff34006cc40 msgr2=0x7ff34006d050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:50.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.354+0000 7ff347200640 1 -- 172.21.15.6:0/2798924636 shutdown_connections 2024-04-24T23:16:50.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.354+0000 7ff347200640 1 -- 172.21.15.6:0/2798924636 wait complete. 2024-04-24T23:16:50.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.356+0000 7ff347200640 1 Processor -- start 2024-04-24T23:16:50.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.356+0000 7ff347200640 1 -- start start 2024-04-24T23:16:50.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.357+0000 7ff347200640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff3400c5a60 0x7ff3400cbe10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:50.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.357+0000 7ff347200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff3400c5e40 0x7ff3400cb2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:50.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.357+0000 7ff347200640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff3400c8570 0x7ff340146930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:50.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.357+0000 7ff347200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7ff34006f590 con 0x7ff3400c5e40 2024-04-24T23:16:50.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.357+0000 7ff347200640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7ff34006f070 con 0x7ff3400c5a60 2024-04-24T23:16:50.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.357+0000 7ff347200640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7ff3401effb0 con 0x7ff3400c8570 2024-04-24T23:16:50.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.358+0000 7ff345400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff3400c5e40 0x7ff3400cb2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:50.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.358+0000 7ff345e00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff3400c5a60 0x7ff3400cbe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:50.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.358+0000 7ff345400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff3400c5e40 0x7ff3400cb2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36436/0 (socket says 172.21.15.6:36436) 2024-04-24T23:16:50.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.358+0000 7ff345400640 1 -- 172.21.15.6:0/3878187081 learned_addr learned my addr 172.21.15.6:0/3878187081 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:16:50.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.358+0000 7ff346800640 1 --2- 172.21.15.6:0/3878187081 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff3400c8570 0x7ff340146930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:50.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.358+0000 7ff346800640 1 -- 172.21.15.6:0/3878187081 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff3400c5a60 msgr2=0x7ff3400cbe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:50.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.358+0000 7ff346800640 1 --2- 172.21.15.6:0/3878187081 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff3400c5a60 0x7ff3400cbe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:50.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.358+0000 7ff346800640 1 -- 172.21.15.6:0/3878187081 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff3400c5e40 msgr2=0x7ff3400cb2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:50.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.358+0000 7ff346800640 1 --2- 172.21.15.6:0/3878187081 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff3400c5e40 0x7ff3400cb2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:50.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.358+0000 7ff346800640 1 -- 172.21.15.6:0/3878187081 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3401f0290 con 0x7ff3400c8570 2024-04-24T23:16:50.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.358+0000 7ff345e00640 1 --2- 172.21.15.6:0/3878187081 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff3400c5a60 0x7ff3400cbe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:16:50.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.359+0000 7ff346800640 1 --2- 172.21.15.6:0/3878187081 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff3400c8570 0x7ff340146930 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7ff33c00ebb0 tx=0x7ff33c00c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:50.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.359+0000 7ff33a000640 1 -- 172.21.15.6:0/3878187081 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff33c018020 con 0x7ff3400c8570 2024-04-24T23:16:50.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.359+0000 7ff33a000640 1 -- 172.21.15.6:0/3878187081 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff33c00cd40 con 0x7ff3400c8570 2024-04-24T23:16:50.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.359+0000 7ff33a000640 1 -- 172.21.15.6:0/3878187081 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff33c0146c0 con 0x7ff3400c8570 2024-04-24T23:16:50.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.359+0000 7ff347200640 1 -- 172.21.15.6:0/3878187081 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3401f0570 con 0x7ff3400c8570 2024-04-24T23:16:50.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.359+0000 7ff347200640 1 -- 172.21.15.6:0/3878187081 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3401f0b30 con 0x7ff3400c8570 2024-04-24T23:16:50.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.360+0000 7ff347200640 1 -- 172.21.15.6:0/3878187081 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff3401f0d70 con 0x7ff3400c8570 2024-04-24T23:16:50.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.361+0000 7ff33a000640 1 -- 172.21.15.6:0/3878187081 <== mon.2 v2:172.21.15.190:3301/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7ff33c01b020 con 0x7ff3400c8570 2024-04-24T23:16:50.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.362+0000 7ff33a000640 1 --2- 172.21.15.6:0/3878187081 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7ff32406be30 0x7ff32406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:50.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.362+0000 7ff33a000640 1 -- 172.21.15.6:0/3878187081 <== mon.2 v2:172.21.15.190:3301/0 5 ==== osd_map(26..26 src has 1..26) ==== 6590+0+0 (secure 0 0 0) 0x7ff33c08be30 con 0x7ff3400c8570 2024-04-24T23:16:50.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.363+0000 7ff345e00640 1 --2- 172.21.15.6:0/3878187081 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7ff32406be30 0x7ff32406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:50.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.363+0000 7ff345e00640 1 --2- 172.21.15.6:0/3878187081 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7ff32406be30 0x7ff32406e2f0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7ff334000f60 tx=0x7ff334007d50 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:50.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.367+0000 7ff33a000640 1 -- 172.21.15.6:0/3878187081 <== mon.2 v2:172.21.15.190:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7ff33c022020 con 0x7ff3400c8570 2024-04-24T23:16:50.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.570+0000 7ff347200640 1 -- 172.21.15.6:0/3878187081 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7ff340075b30 con 0x7ff3400c8570 2024-04-24T23:16:50.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.571+0000 7ff33a000640 1 -- 172.21.15.6:0/3878187081 <== mon.2 v2:172.21.15.190:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) ==== 104+0+11 (secure 0 0 0) 0x7ff33c010040 con 0x7ff3400c8570 2024-04-24T23:16:50.574 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 32 2024-04-24T23:16:50.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.574+0000 7ff347200640 1 -- 172.21.15.6:0/3878187081 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7ff32406be30 msgr2=0x7ff32406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:50.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.574+0000 7ff347200640 1 --2- 172.21.15.6:0/3878187081 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7ff32406be30 0x7ff32406e2f0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7ff334000f60 tx=0x7ff334007d50 comp rx=0 tx=0).stop 2024-04-24T23:16:50.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.574+0000 7ff347200640 1 -- 172.21.15.6:0/3878187081 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff3400c8570 msgr2=0x7ff340146930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:50.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.574+0000 7ff347200640 1 --2- 172.21.15.6:0/3878187081 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff3400c8570 0x7ff340146930 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7ff33c00ebb0 tx=0x7ff33c00c490 comp rx=0 tx=0).stop 2024-04-24T23:16:50.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.574+0000 7ff347200640 1 -- 172.21.15.6:0/3878187081 shutdown_connections 2024-04-24T23:16:50.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.574+0000 7ff347200640 1 --2- 172.21.15.6:0/3878187081 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7ff32406be30 0x7ff32406e2f0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:50.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.574+0000 7ff347200640 1 --2- 172.21.15.6:0/3878187081 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff3400c8570 0x7ff340146930 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:50.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.574+0000 7ff347200640 1 --2- 172.21.15.6:0/3878187081 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff3400c5e40 0x7ff3400cb2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:50.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.574+0000 7ff347200640 1 --2- 172.21.15.6:0/3878187081 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff3400c5a60 0x7ff3400cbe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:50.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.574+0000 7ff347200640 1 -- 172.21.15.6:0/3878187081 >> 172.21.15.6:0/3878187081 conn(0x7ff34006cc40 msgr2=0x7ff340157bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:50.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.575+0000 7ff347200640 1 -- 172.21.15.6:0/3878187081 shutdown_connections 2024-04-24T23:16:50.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.575+0000 7ff347200640 1 -- 172.21.15.6:0/3878187081 wait complete. 2024-04-24T23:16:50.596 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-24T23:16:50.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.809+0000 7fb3bca00640 1 Processor -- start 2024-04-24T23:16:50.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.809+0000 7fb3bca00640 1 -- start start 2024-04-24T23:16:50.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.811+0000 7fb3bca00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb3b80c5a60 0x7fb3b80c6ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:50.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.811+0000 7fb3bca00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fb3b80c8570 0x7fb3b81524a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:50.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.811+0000 7fb3bca00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb3b80cca80 con 0x7fb3b80c5e40 2024-04-24T23:16:50.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.811+0000 7fb3bca00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7fb3b815c440 con 0x7fb3b80c8570 2024-04-24T23:16:50.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.811+0000 7fb3bca00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fb3b80ccd60 con 0x7fb3b80c5a60 2024-04-24T23:16:50.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.811+0000 7fb3b6800640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb3b80c5a60 0x7fb3b80c6ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:50.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.811+0000 7fb3b5e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fb3b80c5e40 0x7fb3b815bd30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:55230/0 (socket says 172.21.15.6:55230) 2024-04-24T23:16:50.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.811+0000 7fb3b5e00640 1 -- 172.21.15.6:0/3655883264 learned_addr learned my addr 172.21.15.6:0/3655883264 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:16:50.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.811+0000 7fb3b7200640 1 --2- 172.21.15.6:0/3655883264 >> v2:172.21.15.6:3300/0 conn(0x7fb3b80c8570 0x7fb3b81524a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:50.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.812+0000 7fb3b5400640 1 -- 172.21.15.6:0/3655883264 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 145672228 0 0) 0x7fb3b80cca80 con 0x7fb3b80c5e40 2024-04-24T23:16:50.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.812+0000 7fb3b5400640 1 -- 172.21.15.6:0/3655883264 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb39c003520 con 0x7fb3b80c5e40 2024-04-24T23:16:50.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.812+0000 7fb3b7200640 1 -- 172.21.15.6:0/3655883264 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb3b80c5a60 msgr2=0x7fb3b80c6ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:16:50.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.812+0000 7fb3b7200640 1 --2- 172.21.15.6:0/3655883264 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb3b80c5a60 0x7fb3b80c6ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:50.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.812+0000 7fb3b7200640 1 -- 172.21.15.6:0/3655883264 >> v1:172.21.15.6:6789/0 conn(0x7fb3b80c5e40 legacy=0x7fb3b815bd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:50.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.812+0000 7fb3b7200640 1 -- 172.21.15.6:0/3655883264 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3b806eab0 con 0x7fb3b80c8570 2024-04-24T23:16:50.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.812+0000 7fb3b6800640 1 --2- 172.21.15.6:0/3655883264 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb3b80c5a60 0x7fb3b80c6ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:16:50.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.813+0000 7fb3b7200640 1 --2- 172.21.15.6:0/3655883264 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb3b80c8570 0x7fb3b81524a0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fb3b0009d90 tx=0x7fb3b002a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea30b81aa4589 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:50.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.813+0000 7fb3b5400640 1 -- 172.21.15.6:0/3655883264 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3b0007740 con 0x7fb3b80c8570 2024-04-24T23:16:50.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.813+0000 7fb3b5400640 1 -- 172.21.15.6:0/3655883264 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb3b00078e0 con 0x7fb3b80c8570 2024-04-24T23:16:50.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.813+0000 7fb3b5400640 1 -- 172.21.15.6:0/3655883264 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3b0007cb0 con 0x7fb3b80c8570 2024-04-24T23:16:50.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.814+0000 7fb3bca00640 1 -- 172.21.15.6:0/3655883264 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb3b80c8570 msgr2=0x7fb3b81524a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:50.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.814+0000 7fb3bca00640 1 --2- 172.21.15.6:0/3655883264 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb3b80c8570 0x7fb3b81524a0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fb3b0009d90 tx=0x7fb3b002a660 comp rx=0 tx=0).stop 2024-04-24T23:16:50.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.814+0000 7fb3bca00640 1 -- 172.21.15.6:0/3655883264 shutdown_connections 2024-04-24T23:16:50.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.814+0000 7fb3bca00640 1 --2- 172.21.15.6:0/3655883264 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb3b80c8570 0x7fb3b81524a0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:50.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.814+0000 7fb3bca00640 1 --2- 172.21.15.6:0/3655883264 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb3b80c5a60 0x7fb3b80c6ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:50.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.814+0000 7fb3bca00640 1 -- 172.21.15.6:0/3655883264 >> 172.21.15.6:0/3655883264 conn(0x7fb3b806cc40 msgr2=0x7fb3b806d050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:50.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.814+0000 7fb3bca00640 1 -- 172.21.15.6:0/3655883264 shutdown_connections 2024-04-24T23:16:50.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.814+0000 7fb3bca00640 1 -- 172.21.15.6:0/3655883264 wait complete. 2024-04-24T23:16:50.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.816+0000 7fb3bca00640 1 Processor -- start 2024-04-24T23:16:50.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.816+0000 7fb3bca00640 1 -- start start 2024-04-24T23:16:50.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.817+0000 7fb3bca00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb3b80c5a60 0x7fb3b81553a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:50.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.817+0000 7fb3bca00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb3b80c5e40 0x7fb3b8155ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:50.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.817+0000 7fb3bca00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb3b80c8570 0x7fb3b81585a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:50.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.817+0000 7fb3bca00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fb3b806f590 con 0x7fb3b80c5e40 2024-04-24T23:16:50.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.817+0000 7fb3bca00640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fb3b806f070 con 0x7fb3b80c5a60 2024-04-24T23:16:50.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.817+0000 7fb3bca00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fb3b81558e0 con 0x7fb3b80c8570 2024-04-24T23:16:50.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.817+0000 7fb3b5e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb3b80c5e40 0x7fb3b8155ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:50.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.817+0000 7fb3b5e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb3b80c5e40 0x7fb3b8155ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36464/0 (socket says 172.21.15.6:36464) 2024-04-24T23:16:50.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.817+0000 7fb3b5e00640 1 -- 172.21.15.6:0/720123764 learned_addr learned my addr 172.21.15.6:0/720123764 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:16:50.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.817+0000 7fb3b7200640 1 --2- 172.21.15.6:0/720123764 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb3b80c8570 0x7fb3b81585a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:50.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.817+0000 7fb3b6800640 1 --2- 172.21.15.6:0/720123764 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb3b80c5a60 0x7fb3b81553a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:50.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.818+0000 7fb3b7200640 1 -- 172.21.15.6:0/720123764 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb3b80c5a60 msgr2=0x7fb3b81553a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:50.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.818+0000 7fb3b7200640 1 --2- 172.21.15.6:0/720123764 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb3b80c5a60 0x7fb3b81553a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:50.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.818+0000 7fb3b7200640 1 -- 172.21.15.6:0/720123764 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb3b80c5e40 msgr2=0x7fb3b8155ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:50.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.818+0000 7fb3b7200640 1 --2- 172.21.15.6:0/720123764 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb3b80c5e40 0x7fb3b8155ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:50.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.818+0000 7fb3b7200640 1 -- 172.21.15.6:0/720123764 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3b81f1410 con 0x7fb3b80c8570 2024-04-24T23:16:50.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.818+0000 7fb3b7200640 1 --2- 172.21.15.6:0/720123764 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb3b80c8570 0x7fb3b81585a0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fb3b0031000 tx=0x7fb3b0004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:50.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.818+0000 7fb3aea00640 1 -- 172.21.15.6:0/720123764 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3b003f070 con 0x7fb3b80c8570 2024-04-24T23:16:50.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.819+0000 7fb3aea00640 1 -- 172.21.15.6:0/720123764 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb3b0031c00 con 0x7fb3b80c8570 2024-04-24T23:16:50.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.819+0000 7fb3aea00640 1 -- 172.21.15.6:0/720123764 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3b0009450 con 0x7fb3b80c8570 2024-04-24T23:16:50.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.819+0000 7fb3bca00640 1 -- 172.21.15.6:0/720123764 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3b81f0890 con 0x7fb3b80c8570 2024-04-24T23:16:50.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.819+0000 7fb3bca00640 1 -- 172.21.15.6:0/720123764 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3b81f0e50 con 0x7fb3b80c8570 2024-04-24T23:16:50.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.820+0000 7fb3bca00640 1 -- 172.21.15.6:0/720123764 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb3b8075b30 con 0x7fb3b80c8570 2024-04-24T23:16:50.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.821+0000 7fb3aea00640 1 -- 172.21.15.6:0/720123764 <== mon.2 v2:172.21.15.190:3301/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fb3b0007790 con 0x7fb3b80c8570 2024-04-24T23:16:50.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.822+0000 7fb3aea00640 1 --2- 172.21.15.6:0/720123764 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fb39806be30 0x7fb39806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:50.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.822+0000 7fb3aea00640 1 -- 172.21.15.6:0/720123764 <== mon.2 v2:172.21.15.190:3301/0 5 ==== osd_map(26..26 src has 1..26) ==== 6590+0+0 (secure 0 0 0) 0x7fb3b00ac740 con 0x7fb3b80c8570 2024-04-24T23:16:50.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.822+0000 7fb3b6800640 1 --2- 172.21.15.6:0/720123764 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fb39806be30 0x7fb39806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:50.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.823+0000 7fb3b6800640 1 --2- 172.21.15.6:0/720123764 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fb39806be30 0x7fb39806e2f0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fb3a8009560 tx=0x7fb3a8009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:50.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:50.826+0000 7fb3aea00640 1 -- 172.21.15.6:0/720123764 <== mon.2 v2:172.21.15.190:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fb3b007a840 con 0x7fb3b80c8570 2024-04-24T23:16:51.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.028+0000 7fb3bca00640 1 -- 172.21.15.6:0/720123764 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fb3b8071ba0 con 0x7fb3b80c8570 2024-04-24T23:16:51.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.029+0000 7fb3aea00640 1 -- 172.21.15.6:0/720123764 <== mon.2 v2:172.21.15.190:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) ==== 100+0+11 (secure 0 0 0) 0x7fb3b007eb40 con 0x7fb3b80c8570 2024-04-24T23:16:51.031 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 64 2024-04-24T23:16:51.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.031+0000 7fb3bca00640 1 -- 172.21.15.6:0/720123764 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fb39806be30 msgr2=0x7fb39806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:51.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.031+0000 7fb3bca00640 1 --2- 172.21.15.6:0/720123764 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fb39806be30 0x7fb39806e2f0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fb3a8009560 tx=0x7fb3a8009190 comp rx=0 tx=0).stop 2024-04-24T23:16:51.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.031+0000 7fb3bca00640 1 -- 172.21.15.6:0/720123764 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb3b80c8570 msgr2=0x7fb3b81585a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:51.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.031+0000 7fb3bca00640 1 --2- 172.21.15.6:0/720123764 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb3b80c8570 0x7fb3b81585a0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fb3b0031000 tx=0x7fb3b0004a40 comp rx=0 tx=0).stop 2024-04-24T23:16:51.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.031+0000 7fb3bca00640 1 -- 172.21.15.6:0/720123764 shutdown_connections 2024-04-24T23:16:51.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.031+0000 7fb3bca00640 1 --2- 172.21.15.6:0/720123764 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fb39806be30 0x7fb39806e2f0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:51.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.031+0000 7fb3bca00640 1 --2- 172.21.15.6:0/720123764 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb3b80c8570 0x7fb3b81585a0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:51.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.031+0000 7fb3bca00640 1 --2- 172.21.15.6:0/720123764 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb3b80c5e40 0x7fb3b8155ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:51.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.031+0000 7fb3bca00640 1 --2- 172.21.15.6:0/720123764 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb3b80c5a60 0x7fb3b81553a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:51.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.031+0000 7fb3bca00640 1 -- 172.21.15.6:0/720123764 >> 172.21.15.6:0/720123764 conn(0x7fb3b806cc40 msgr2=0x7fb3b8151770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:51.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.031+0000 7fb3bca00640 1 -- 172.21.15.6:0/720123764 shutdown_connections 2024-04-24T23:16:51.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.032+0000 7fb3bca00640 1 -- 172.21.15.6:0/720123764 wait complete. 2024-04-24T23:16:51.053 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-24T23:16:51.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.263+0000 7fc623200640 1 Processor -- start 2024-04-24T23:16:51.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.263+0000 7fc623200640 1 -- start start 2024-04-24T23:16:51.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.265+0000 7fc623200640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc62406eb80 0x7fc6240c9330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:51.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.265+0000 7fc623200640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fc62406ef60 0x7fc624158b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:51.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.265+0000 7fc623200640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc62415f240 con 0x7fc62415ed90 2024-04-24T23:16:51.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.265+0000 7fc623200640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7fc6240c8bd0 con 0x7fc62406ef60 2024-04-24T23:16:51.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.265+0000 7fc623200640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fc62415f520 con 0x7fc62406eb80 2024-04-24T23:16:51.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.265+0000 7fc621e00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc62406eb80 0x7fc6240c9330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:51.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.265+0000 7fc622800640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fc62406ef60 0x7fc624158b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:51.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.265+0000 7fc621e00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc62406eb80 0x7fc6240c9330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.6:59842/0 (socket says 172.21.15.6:59842) 2024-04-24T23:16:51.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.265+0000 7fc621e00640 1 -- 172.21.15.6:0/2080306615 learned_addr learned my addr 172.21.15.6:0/2080306615 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:16:51.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.265+0000 7fc619400640 1 --1- 172.21.15.6:0/2080306615 >> v1:172.21.15.190:6789/0 conn(0x7fc62415ed90 0x7fc624162a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.6:36954/0 (socket says 172.21.15.6:36954) 2024-04-24T23:16:51.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.266+0000 7fc621e00640 1 -- 172.21.15.6:0/2080306615 >> v2:172.21.15.190:3300/0 conn(0x7fc62406ef60 msgr2=0x7fc624158b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:16:51.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.266+0000 7fc621e00640 1 --2- 172.21.15.6:0/2080306615 >> v2:172.21.15.190:3300/0 conn(0x7fc62406ef60 0x7fc624158b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:51.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.266+0000 7fc621e00640 1 -- 172.21.15.6:0/2080306615 >> v1:172.21.15.190:6789/0 conn(0x7fc62415ed90 legacy=0x7fc624162a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:51.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.266+0000 7fc621e00640 1 -- 172.21.15.6:0/2080306615 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc62415f800 con 0x7fc62406eb80 2024-04-24T23:16:51.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.267+0000 7fc621e00640 1 --2- 172.21.15.6:0/2080306615 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc62406eb80 0x7fc6240c9330 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fc614009870 tx=0x7fc61402a300 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8763dc79bcb23e0a server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:51.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.267+0000 7fc621400640 1 -- 172.21.15.6:0/2080306615 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc614037070 con 0x7fc62406eb80 2024-04-24T23:16:51.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.267+0000 7fc621400640 1 -- 172.21.15.6:0/2080306615 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc61402ae50 con 0x7fc62406eb80 2024-04-24T23:16:51.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.267+0000 7fc621400640 1 -- 172.21.15.6:0/2080306615 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc614032860 con 0x7fc62406eb80 2024-04-24T23:16:51.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.268+0000 7fc623200640 1 -- 172.21.15.6:0/2080306615 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc62406eb80 msgr2=0x7fc6240c9330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:51.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.268+0000 7fc623200640 1 --2- 172.21.15.6:0/2080306615 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc62406eb80 0x7fc6240c9330 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fc614009870 tx=0x7fc61402a300 comp rx=0 tx=0).stop 2024-04-24T23:16:51.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.268+0000 7fc623200640 1 -- 172.21.15.6:0/2080306615 shutdown_connections 2024-04-24T23:16:51.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.268+0000 7fc623200640 1 --2- 172.21.15.6:0/2080306615 >> v2:172.21.15.190:3300/0 conn(0x7fc62406ef60 0x7fc624158b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:51.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.268+0000 7fc623200640 1 --2- 172.21.15.6:0/2080306615 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc62406eb80 0x7fc6240c9330 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:51.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.268+0000 7fc623200640 1 -- 172.21.15.6:0/2080306615 >> 172.21.15.6:0/2080306615 conn(0x7fc62406cc40 msgr2=0x7fc62406d050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:51.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.268+0000 7fc623200640 1 -- 172.21.15.6:0/2080306615 shutdown_connections 2024-04-24T23:16:51.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.268+0000 7fc623200640 1 -- 172.21.15.6:0/2080306615 wait complete. 2024-04-24T23:16:51.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.270+0000 7fc623200640 1 Processor -- start 2024-04-24T23:16:51.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.270+0000 7fc623200640 1 -- start start 2024-04-24T23:16:51.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.272+0000 7fc623200640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc62406eb80 0x7fc6241540b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:51.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.272+0000 7fc623200640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc62406ef60 0x7fc624161930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:51.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.272+0000 7fc623200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc62415ed90 0x7fc62414ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:51.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.272+0000 7fc623200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fc624163150 con 0x7fc62415ed90 2024-04-24T23:16:51.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.272+0000 7fc623200640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fc624162520 con 0x7fc62406eb80 2024-04-24T23:16:51.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.272+0000 7fc623200640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fc6241632d0 con 0x7fc62406ef60 2024-04-24T23:16:51.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.272+0000 7fc621e00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc62406eb80 0x7fc6241540b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:51.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.272+0000 7fc622800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc62415ed90 0x7fc62414ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:51.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.272+0000 7fc621e00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc62406eb80 0x7fc6241540b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:36726/0 (socket says 172.21.15.6:36726) 2024-04-24T23:16:51.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.272+0000 7fc621e00640 1 -- 172.21.15.6:0/3918552025 learned_addr learned my addr 172.21.15.6:0/3918552025 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:16:51.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.272+0000 7fc619400640 1 --2- 172.21.15.6:0/3918552025 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc62406ef60 0x7fc624161930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:51.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.273+0000 7fc621e00640 1 -- 172.21.15.6:0/3918552025 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc62406ef60 msgr2=0x7fc624161930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:51.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.273+0000 7fc621e00640 1 --2- 172.21.15.6:0/3918552025 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc62406ef60 0x7fc624161930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:51.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.273+0000 7fc621e00640 1 -- 172.21.15.6:0/3918552025 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc62415ed90 msgr2=0x7fc62414ea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:51.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.273+0000 7fc621e00640 1 --2- 172.21.15.6:0/3918552025 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc62415ed90 0x7fc62414ea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:51.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.273+0000 7fc621e00640 1 -- 172.21.15.6:0/3918552025 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc624155360 con 0x7fc62406eb80 2024-04-24T23:16:51.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.273+0000 7fc622800640 1 --2- 172.21.15.6:0/3918552025 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc62415ed90 0x7fc62414ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:16:51.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.273+0000 7fc621e00640 1 --2- 172.21.15.6:0/3918552025 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc62406eb80 0x7fc6241540b0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fc614000c00 tx=0x7fc6140046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:51.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.273+0000 7fc61aa00640 1 -- 172.21.15.6:0/3918552025 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc614041070 con 0x7fc62406eb80 2024-04-24T23:16:51.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.273+0000 7fc61aa00640 1 -- 172.21.15.6:0/3918552025 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc61403c660 con 0x7fc62406eb80 2024-04-24T23:16:51.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.273+0000 7fc623200640 1 -- 172.21.15.6:0/3918552025 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc624154970 con 0x7fc62406eb80 2024-04-24T23:16:51.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.273+0000 7fc61aa00640 1 -- 172.21.15.6:0/3918552025 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc614037040 con 0x7fc62406eb80 2024-04-24T23:16:51.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.274+0000 7fc623200640 1 -- 172.21.15.6:0/3918552025 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6241f8840 con 0x7fc62406eb80 2024-04-24T23:16:51.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.275+0000 7fc623200640 1 -- 172.21.15.6:0/3918552025 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc624075b30 con 0x7fc62406eb80 2024-04-24T23:16:51.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.276+0000 7fc61aa00640 1 -- 172.21.15.6:0/3918552025 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fc614044070 con 0x7fc62406eb80 2024-04-24T23:16:51.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.276+0000 7fc61aa00640 1 --2- 172.21.15.6:0/3918552025 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fc60006be30 0x7fc60006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:51.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.277+0000 7fc61aa00640 1 -- 172.21.15.6:0/3918552025 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6590+0+0 (secure 0 0 0) 0x7fc6140ab930 con 0x7fc62406eb80 2024-04-24T23:16:51.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.277+0000 7fc619400640 1 --2- 172.21.15.6:0/3918552025 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fc60006be30 0x7fc60006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:51.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.278+0000 7fc619400640 1 --2- 172.21.15.6:0/3918552025 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fc60006be30 0x7fc60006e2f0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fc624155850 tx=0x7fc60c000ed0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:51.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.282+0000 7fc61aa00640 1 -- 172.21.15.6:0/3918552025 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fc6140799a0 con 0x7fc62406eb80 2024-04-24T23:16:51.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.486+0000 7fc623200640 1 -- 172.21.15.6:0/3918552025 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fc624067b30 con 0x7fc62406eb80 2024-04-24T23:16:51.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.487+0000 7fc61aa00640 1 -- 172.21.15.6:0/3918552025 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) ==== 103+0+11 (secure 0 0 0) 0x7fc61407dca0 con 0x7fc62406eb80 2024-04-24T23:16:51.490 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 32 2024-04-24T23:16:51.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.490+0000 7fc623200640 1 -- 172.21.15.6:0/3918552025 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fc60006be30 msgr2=0x7fc60006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:51.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.490+0000 7fc623200640 1 --2- 172.21.15.6:0/3918552025 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fc60006be30 0x7fc60006e2f0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fc624155850 tx=0x7fc60c000ed0 comp rx=0 tx=0).stop 2024-04-24T23:16:51.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.490+0000 7fc623200640 1 -- 172.21.15.6:0/3918552025 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc62406eb80 msgr2=0x7fc6241540b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:51.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.490+0000 7fc623200640 1 --2- 172.21.15.6:0/3918552025 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc62406eb80 0x7fc6241540b0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fc614000c00 tx=0x7fc6140046c0 comp rx=0 tx=0).stop 2024-04-24T23:16:51.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.491+0000 7fc623200640 1 -- 172.21.15.6:0/3918552025 shutdown_connections 2024-04-24T23:16:51.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.491+0000 7fc623200640 1 --2- 172.21.15.6:0/3918552025 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fc60006be30 0x7fc60006e2f0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:51.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.491+0000 7fc623200640 1 --2- 172.21.15.6:0/3918552025 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc62406ef60 0x7fc624161930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:51.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.491+0000 7fc623200640 1 --2- 172.21.15.6:0/3918552025 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc62415ed90 0x7fc62414ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:51.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.491+0000 7fc623200640 1 --2- 172.21.15.6:0/3918552025 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc62406eb80 0x7fc6241540b0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:51.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.491+0000 7fc623200640 1 -- 172.21.15.6:0/3918552025 >> 172.21.15.6:0/3918552025 conn(0x7fc62406cc40 msgr2=0x7fc624158940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:51.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.491+0000 7fc623200640 1 -- 172.21.15.6:0/3918552025 shutdown_connections 2024-04-24T23:16:51.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.491+0000 7fc623200640 1 -- 172.21.15.6:0/3918552025 wait complete. 2024-04-24T23:16:51.513 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:16:51.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.723+0000 7f5ca7200640 1 Processor -- start 2024-04-24T23:16:51.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.723+0000 7f5ca7200640 1 -- start start 2024-04-24T23:16:51.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.725+0000 7f5ca7200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f5ca815aa30 0x7f5ca80cb0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:51.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.726+0000 7f5ca7200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5ca815b9d0 con 0x7f5ca8154b90 2024-04-24T23:16:51.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.726+0000 7f5ca7200640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5ca815b130 con 0x7f5ca80cb5f0 2024-04-24T23:16:51.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.726+0000 7f5ca7200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f5ca815b410 con 0x7f5ca815aa30 2024-04-24T23:16:51.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.726+0000 7f5ca6800640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f5ca80cb5f0 0x7f5ca80c65a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.6:36960/0 (socket says 172.21.15.6:36960) 2024-04-24T23:16:51.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.726+0000 7f5ca6800640 1 -- 172.21.15.6:0/1599346870 learned_addr learned my addr 172.21.15.6:0/1599346870 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:16:51.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.726+0000 7f5ca5400640 1 --2- 172.21.15.6:0/1599346870 >> v2:172.21.15.6:3300/0 conn(0x7f5ca815aa30 0x7f5ca80cb0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:51.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.726+0000 7f5ca4a00640 1 -- 172.21.15.6:0/1599346870 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3259340522 0 0) 0x7f5ca815b130 con 0x7f5ca80cb5f0 2024-04-24T23:16:51.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.726+0000 7f5ca4a00640 1 -- 172.21.15.6:0/1599346870 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5c8c003520 con 0x7f5ca80cb5f0 2024-04-24T23:16:51.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.727+0000 7f5ca5400640 1 -- 172.21.15.6:0/1599346870 >> v1:172.21.15.190:6789/0 conn(0x7f5ca80cb5f0 legacy=0x7f5ca80c65a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:51.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.727+0000 7f5ca4a00640 1 -- 172.21.15.6:0/1599346870 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1226231695 0 0) 0x7f5c8c003520 con 0x7f5ca80cb5f0 2024-04-24T23:16:51.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.727+0000 7f5ca5400640 1 -- 172.21.15.6:0/1599346870 >> v1:172.21.15.6:6789/0 conn(0x7f5ca8154b90 legacy=0x7f5ca80c52d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:51.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.727+0000 7f5ca5400640 1 -- 172.21.15.6:0/1599346870 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ca815b6f0 con 0x7f5ca815aa30 2024-04-24T23:16:51.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.728+0000 7f5ca5400640 1 --2- 172.21.15.6:0/1599346870 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5ca815aa30 0x7f5ca80cb0b0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f5c940097a0 tx=0x7f5c9402a160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a996997c9615cea2 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:51.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.728+0000 7f5ca4a00640 1 -- 172.21.15.6:0/1599346870 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c94037070 con 0x7f5ca815aa30 2024-04-24T23:16:51.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.728+0000 7f5ca4a00640 1 -- 172.21.15.6:0/1599346870 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5c9402abf0 con 0x7f5ca815aa30 2024-04-24T23:16:51.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.728+0000 7f5ca4a00640 1 -- 172.21.15.6:0/1599346870 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ca00019f0 con 0x7f5ca815aa30 2024-04-24T23:16:51.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.729+0000 7f5ca7200640 1 -- 172.21.15.6:0/1599346870 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5ca815aa30 msgr2=0x7f5ca80cb0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:51.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.729+0000 7f5ca7200640 1 --2- 172.21.15.6:0/1599346870 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5ca815aa30 0x7f5ca80cb0b0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f5c940097a0 tx=0x7f5c9402a160 comp rx=0 tx=0).stop 2024-04-24T23:16:51.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.729+0000 7f5ca7200640 1 -- 172.21.15.6:0/1599346870 shutdown_connections 2024-04-24T23:16:51.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.729+0000 7f5ca7200640 1 --2- 172.21.15.6:0/1599346870 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5ca815aa30 0x7f5ca80cb0b0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:51.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.729+0000 7f5ca7200640 1 -- 172.21.15.6:0/1599346870 >> 172.21.15.6:0/1599346870 conn(0x7f5ca806cd70 msgr2=0x7f5ca806d870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:51.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.729+0000 7f5ca7200640 1 -- 172.21.15.6:0/1599346870 shutdown_connections 2024-04-24T23:16:51.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.729+0000 7f5ca7200640 1 -- 172.21.15.6:0/1599346870 wait complete. 2024-04-24T23:16:51.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.730+0000 7f5ca7200640 1 Processor -- start 2024-04-24T23:16:51.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.731+0000 7f5ca7200640 1 -- start start 2024-04-24T23:16:51.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.731+0000 7f5ca7200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5ca80cb5f0 0x7f5ca80ca100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:51.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.732+0000 7f5ca7200640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5ca8154b90 0x7f5ca81f3a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:51.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.732+0000 7f5ca7200640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5ca815aa30 0x7f5ca814f480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:51.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.732+0000 7f5ca7200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f5ca815ae30 con 0x7f5ca80cb5f0 2024-04-24T23:16:51.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.732+0000 7f5ca7200640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f5ca80c6080 con 0x7f5ca8154b90 2024-04-24T23:16:51.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.732+0000 7f5ca7200640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f5ca80c59e0 con 0x7f5ca815aa30 2024-04-24T23:16:51.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.732+0000 7f5ca5e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5ca80cb5f0 0x7f5ca80ca100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:51.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.732+0000 7f5ca5e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5ca80cb5f0 0x7f5ca80ca100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36492/0 (socket says 172.21.15.6:36492) 2024-04-24T23:16:51.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.732+0000 7f5ca5e00640 1 -- 172.21.15.6:0/3739095361 learned_addr learned my addr 172.21.15.6:0/3739095361 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:16:51.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.732+0000 7f5ca6800640 1 --2- 172.21.15.6:0/3739095361 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5ca815aa30 0x7f5ca814f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:51.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.732+0000 7f5ca5400640 1 --2- 172.21.15.6:0/3739095361 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5ca8154b90 0x7f5ca81f3a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:51.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.732+0000 7f5ca5e00640 1 -- 172.21.15.6:0/3739095361 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5ca815aa30 msgr2=0x7f5ca814f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:51.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.732+0000 7f5ca5e00640 1 --2- 172.21.15.6:0/3739095361 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5ca815aa30 0x7f5ca814f480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:51.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.732+0000 7f5ca5e00640 1 -- 172.21.15.6:0/3739095361 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5ca8154b90 msgr2=0x7f5ca81f3a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:51.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.732+0000 7f5ca5e00640 1 --2- 172.21.15.6:0/3739095361 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5ca8154b90 0x7f5ca81f3a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:51.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.732+0000 7f5ca5e00640 1 -- 172.21.15.6:0/3739095361 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ca81f43b0 con 0x7f5ca80cb5f0 2024-04-24T23:16:51.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.733+0000 7f5ca5e00640 1 --2- 172.21.15.6:0/3739095361 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5ca80cb5f0 0x7f5ca80ca100 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f5c9800c5b0 tx=0x7f5c9800ca70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:51.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.733+0000 7f5c9e000640 1 -- 172.21.15.6:0/3739095361 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c98017070 con 0x7f5ca80cb5f0 2024-04-24T23:16:51.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.733+0000 7f5c9e000640 1 -- 172.21.15.6:0/3739095361 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5c98005480 con 0x7f5ca80cb5f0 2024-04-24T23:16:51.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.733+0000 7f5c9e000640 1 -- 172.21.15.6:0/3739095361 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c98003830 con 0x7f5ca80cb5f0 2024-04-24T23:16:51.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.733+0000 7f5ca7200640 1 -- 172.21.15.6:0/3739095361 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ca81f4690 con 0x7f5ca80cb5f0 2024-04-24T23:16:51.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.733+0000 7f5ca7200640 1 -- 172.21.15.6:0/3739095361 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ca81515d0 con 0x7f5ca80cb5f0 2024-04-24T23:16:51.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.734+0000 7f5ca7200640 1 -- 172.21.15.6:0/3739095361 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ca8075b30 con 0x7f5ca80cb5f0 2024-04-24T23:16:51.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.735+0000 7f5c9e000640 1 -- 172.21.15.6:0/3739095361 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f5c980039d0 con 0x7f5ca80cb5f0 2024-04-24T23:16:51.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.736+0000 7f5c9e000640 1 --2- 172.21.15.6:0/3739095361 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f5c8806be30 0x7f5c8806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:51.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.736+0000 7f5c9e000640 1 -- 172.21.15.6:0/3739095361 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6590+0+0 (secure 0 0 0) 0x7f5c9808a4c0 con 0x7f5ca80cb5f0 2024-04-24T23:16:51.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.737+0000 7f5ca5400640 1 --2- 172.21.15.6:0/3739095361 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f5c8806be30 0x7f5c8806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:51.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.737+0000 7f5ca5400640 1 --2- 172.21.15.6:0/3739095361 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f5c8806be30 0x7f5c8806e2f0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f5c94034000 tx=0x7f5c9402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:51.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.741+0000 7f5c9e000640 1 -- 172.21.15.6:0/3739095361 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f5c98058410 con 0x7f5ca80cb5f0 2024-04-24T23:16:51.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.985+0000 7f5ca7200640 1 -- 172.21.15.6:0/3739095361 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5ca80cbc60 con 0x7f5ca80cb5f0 2024-04-24T23:16:51.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.986+0000 7f5c9e000640 1 -- 172.21.15.6:0/3739095361 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) ==== 94+0+6902 (secure 0 0 0) 0x7f5c9805c710 con 0x7f5ca80cb5f0 2024-04-24T23:16:51.989 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:16:51.989 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":18,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4197,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/105267896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":105267896},{"type":"v1","addr":"172.21.15.6:6834","nonce":105267896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:16:32.310838+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4282,"mds_2":4235,"mds_3":4232,"mds_4":4203},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4203":{"gid":4203,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":17,"addr":"172.21.15.6:6837/2590421094","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":2590421094},{"type":"v1","addr":"172.21.15.6:6837","nonce":2590421094}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4232":{"gid":4232,"name":"b","rank":3,"incarnation":15,"state":"up:active","state_seq":15,"addr":"172.21.15.190:6838/2032280158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2032280158},{"type":"v1","addr":"172.21.15.190:6838","nonce":2032280158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4235":{"gid":4235,"name":"d","rank":2,"incarnation":13,"state":"up:active","state_seq":14,"addr":"172.21.15.190:6839/1572872451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":1572872451},{"type":"v1","addr":"172.21.15.190:6839","nonce":1572872451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4282":{"gid":4282,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":12,"addr":"172.21.15.190:6837/580315269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":580315269},{"type":"v1","addr":"172.21.15.190:6837","nonce":580315269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/489092393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":489092393},{"type":"v1","addr":"172.21.15.6:6835","nonce":489092393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4306,"qdb_cluster":[4203,4232,4235,4282,4306]},"id":1}]} 2024-04-24T23:16:51.989 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 18 2024-04-24T23:16:51.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.989+0000 7f5ca7200640 1 -- 172.21.15.6:0/3739095361 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f5c8806be30 msgr2=0x7f5c8806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:51.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.989+0000 7f5ca7200640 1 --2- 172.21.15.6:0/3739095361 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f5c8806be30 0x7f5c8806e2f0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f5c94034000 tx=0x7f5c9402e000 comp rx=0 tx=0).stop 2024-04-24T23:16:51.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.989+0000 7f5ca7200640 1 -- 172.21.15.6:0/3739095361 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5ca80cb5f0 msgr2=0x7f5ca80ca100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:51.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.989+0000 7f5ca7200640 1 --2- 172.21.15.6:0/3739095361 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5ca80cb5f0 0x7f5ca80ca100 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f5c9800c5b0 tx=0x7f5c9800ca70 comp rx=0 tx=0).stop 2024-04-24T23:16:51.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.989+0000 7f5ca7200640 1 -- 172.21.15.6:0/3739095361 shutdown_connections 2024-04-24T23:16:51.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.989+0000 7f5ca7200640 1 --2- 172.21.15.6:0/3739095361 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f5c8806be30 0x7f5c8806e2f0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:51.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.989+0000 7f5ca7200640 1 --2- 172.21.15.6:0/3739095361 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5ca815aa30 0x7f5ca814f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:51.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.989+0000 7f5ca7200640 1 --2- 172.21.15.6:0/3739095361 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5ca8154b90 0x7f5ca81f3a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:51.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.989+0000 7f5ca7200640 1 --2- 172.21.15.6:0/3739095361 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5ca80cb5f0 0x7f5ca80ca100 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:51.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.989+0000 7f5ca7200640 1 -- 172.21.15.6:0/3739095361 >> 172.21.15.6:0/3739095361 conn(0x7f5ca806cd70 msgr2=0x7f5ca806d870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:51.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.989+0000 7f5ca7200640 1 -- 172.21.15.6:0/3739095361 shutdown_connections 2024-04-24T23:16:51.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:51.989+0000 7f5ca7200640 1 -- 172.21.15.6:0/3739095361 wait complete. 2024-04-24T23:16:52.009 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T23:16:52.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:52.220+0000 7f2435a00640 1 Processor -- start 2024-04-24T23:16:52.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:52.220+0000 7f2435a00640 1 -- start start 2024-04-24T23:16:52.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:52.222+0000 7f2435a00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2430154d90 0x7f24300cbe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:52.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:52.222+0000 7f2435a00640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f24300cc3c0 0x7f243015e000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:52.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:52.222+0000 7f2435a00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2430155170 con 0x7f24300cb090 2024-04-24T23:16:52.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:52.222+0000 7f2435a00640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f24300cc9a0 con 0x7f24300cc3c0 2024-04-24T23:16:52.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:52.222+0000 7f2435a00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f243015ad60 con 0x7f2430154d90 2024-04-24T23:16:52.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:52.222+0000 7f242fe00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2430154d90 0x7f24300cbe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:52.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:52.222+0000 7f242f400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f24300cb090 0x7f24300d0820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:55260/0 (socket says 172.21.15.6:55260) 2024-04-24T23:16:52.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:52.222+0000 7f242f400640 1 -- 172.21.15.6:0/1061961009 learned_addr learned my addr 172.21.15.6:0/1061961009 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:16:52.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:52.222+0000 7f2435000640 1 --2- 172.21.15.6:0/1061961009 >> v2:172.21.15.190:3300/0 conn(0x7f24300cc3c0 0x7f243015e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:52.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:52.223+0000 7f242fe00640 1 -- 172.21.15.6:0/1061961009 >> v2:172.21.15.190:3300/0 conn(0x7f24300cc3c0 msgr2=0x7f243015e000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:16:52.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:52.223+0000 7f242fe00640 1 --2- 172.21.15.6:0/1061961009 >> v2:172.21.15.190:3300/0 conn(0x7f24300cc3c0 0x7f243015e000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:52.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:52.223+0000 7f242fe00640 1 -- 172.21.15.6:0/1061961009 >> v1:172.21.15.6:6789/0 conn(0x7f24300cb090 legacy=0x7f24300d0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:52.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:52.223+0000 7f242fe00640 1 -- 172.21.15.6:0/1061961009 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f243015b040 con 0x7f2430154d90 2024-04-24T23:16:52.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:52.224+0000 7f242fe00640 1 --2- 172.21.15.6:0/1061961009 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2430154d90 0x7f24300cbe80 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f24200097a0 tx=0x7f242002a210 comp rx=0 tx=0).ready entity=mon.2 client_cookie=68dc01a8e473a9a8 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:52.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:52.224+0000 7f242ea00640 1 -- 172.21.15.6:0/1061961009 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2420037070 con 0x7f2430154d90 2024-04-24T23:16:52.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:52.224+0000 7f242ea00640 1 -- 172.21.15.6:0/1061961009 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2420032440 con 0x7f2430154d90 2024-04-24T23:16:52.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:52.224+0000 7f242ea00640 1 -- 172.21.15.6:0/1061961009 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2420032810 con 0x7f2430154d90 2024-04-24T23:16:52.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:52.225+0000 7f2435a00640 1 -- 172.21.15.6:0/1061961009 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2430154d90 msgr2=0x7f24300cbe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:52.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:52.225+0000 7f2435a00640 1 --2- 172.21.15.6:0/1061961009 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2430154d90 0x7f24300cbe80 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f24200097a0 tx=0x7f242002a210 comp rx=0 tx=0).stop 2024-04-24T23:16:52.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:52.225+0000 7f2435a00640 1 -- 172.21.15.6:0/1061961009 shutdown_connections 2024-04-24T23:16:52.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:52.225+0000 7f2435a00640 1 --2- 172.21.15.6:0/1061961009 >> v2:172.21.15.190:3300/0 conn(0x7f24300cc3c0 0x7f243015e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:52.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:52.225+0000 7f2435a00640 1 --2- 172.21.15.6:0/1061961009 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2430154d90 0x7f24300cbe80 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:52.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:52.225+0000 7f2435a00640 1 -- 172.21.15.6:0/1061961009 >> 172.21.15.6:0/1061961009 conn(0x7f243006be10 msgr2=0x7f243006c1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:52.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:52.225+0000 7f2435a00640 1 -- 172.21.15.6:0/1061961009 shutdown_connections 2024-04-24T23:16:52.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:52.225+0000 7f2435a00640 1 -- 172.21.15.6:0/1061961009 wait complete. 2024-04-24T23:16:52.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:52.227+0000 7f2435a00640 1 Processor -- start 2024-04-24T23:16:52.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:52.227+0000 7f2435a00640 1 -- start start 2024-04-24T23:16:52.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:52.228+0000 7f2435a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f24300cb090 0x7f2430159b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:52.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:52.228+0000 7f2435a00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f24300cc3c0 0x7f24301e47a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:52.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:52.228+0000 7f2435a00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2430154d90 0x7f24301e4ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:52.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:52.228+0000 7f2435a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f243015aa30 con 0x7f24300cb090 2024-04-24T23:16:52.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:52.228+0000 7f2435a00640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f243015b600 con 0x7f2430154d90 2024-04-24T23:16:52.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:52.228+0000 7f2435a00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f24300d0f30 con 0x7f24300cc3c0 2024-04-24T23:16:52.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:52.228+0000 7f242fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f24300cb090 0x7f2430159b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:52.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:52.228+0000 7f242fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f24300cb090 0x7f2430159b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36506/0 (socket says 172.21.15.6:36506) 2024-04-24T23:16:52.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:52.228+0000 7f242fe00640 1 -- 172.21.15.6:0/3882552435 learned_addr learned my addr 172.21.15.6:0/3882552435 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:16:52.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:52.228+0000 7f2435000640 1 --2- 172.21.15.6:0/3882552435 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2430154d90 0x7f24301e4ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:52.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:52.228+0000 7f242f400640 1 --2- 172.21.15.6:0/3882552435 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f24300cc3c0 0x7f24301e47a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:52.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:52.229+0000 7f242fe00640 1 -- 172.21.15.6:0/3882552435 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f24300cc3c0 msgr2=0x7f24301e47a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:52.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:52.229+0000 7f242fe00640 1 --2- 172.21.15.6:0/3882552435 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f24300cc3c0 0x7f24301e47a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:52.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:52.229+0000 7f242fe00640 1 -- 172.21.15.6:0/3882552435 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2430154d90 msgr2=0x7f24301e4ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:52.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:52.229+0000 7f242fe00640 1 --2- 172.21.15.6:0/3882552435 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2430154d90 0x7f24301e4ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:52.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:52.229+0000 7f242fe00640 1 -- 172.21.15.6:0/3882552435 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f243015eb70 con 0x7f24300cb090 2024-04-24T23:16:52.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:52.229+0000 7f242fe00640 1 --2- 172.21.15.6:0/3882552435 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f24300cb090 0x7f2430159b60 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f24200098c0 tx=0x7f24200046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:52.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:52.229+0000 7f2427e00640 1 -- 172.21.15.6:0/3882552435 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2420037040 con 0x7f24300cb090 2024-04-24T23:16:52.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:52.229+0000 7f2427e00640 1 -- 172.21.15.6:0/3882552435 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2420030500 con 0x7f24300cb090 2024-04-24T23:16:52.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:52.229+0000 7f2427e00640 1 -- 172.21.15.6:0/3882552435 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24200308d0 con 0x7f24300cb090 2024-04-24T23:16:52.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:52.229+0000 7f2435a00640 1 -- 172.21.15.6:0/3882552435 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24301e3ed0 con 0x7f24300cb090 2024-04-24T23:16:52.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:52.230+0000 7f2435a00640 1 -- 172.21.15.6:0/3882552435 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24301e6d30 con 0x7f24300cb090 2024-04-24T23:16:52.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:52.231+0000 7f2435a00640 1 -- 172.21.15.6:0/3882552435 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2430075b30 con 0x7f24300cb090 2024-04-24T23:16:52.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:52.232+0000 7f2427e00640 1 -- 172.21.15.6:0/3882552435 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f2420044020 con 0x7f24300cb090 2024-04-24T23:16:52.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:52.233+0000 7f2427e00640 1 --2- 172.21.15.6:0/3882552435 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f240c06be60 0x7f240c06e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:16:52.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:52.233+0000 7f2427e00640 1 -- 172.21.15.6:0/3882552435 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6590+0+0 (secure 0 0 0) 0x7f24200ab300 con 0x7f24300cb090 2024-04-24T23:16:52.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:52.233+0000 7f242f400640 1 --2- 172.21.15.6:0/3882552435 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f240c06be60 0x7f240c06e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:16:52.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:52.234+0000 7f242f400640 1 --2- 172.21.15.6:0/3882552435 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f240c06be60 0x7f240c06e320 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f24300b61c0 tx=0x7f2418001990 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:16:52.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:52.237+0000 7f2427e00640 1 -- 172.21.15.6:0/3882552435 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f24200792e0 con 0x7f24300cb090 2024-04-24T23:16:52.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:52.442+0000 7f2435a00640 1 -- 172.21.15.6:0/3882552435 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f24300cbc60 con 0x7f24300cb090 2024-04-24T23:16:52.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:52.445+0000 7f2427e00640 1 -- 172.21.15.6:0/3882552435 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+18554 (secure 0 0 0) 0x7f242007d5e0 con 0x7f24300cb090 2024-04-24T23:16:52.448 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:16:52.448 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":26,"fsid":"18bef68d-5e6d-4ec5-862b-b6f206e2588d","created":"2024-04-24T23:15:12.535072+0000","modified":"2024-04-24T23:16:18.218008+0000","last_up_change":"2024-04-24T23:15:23.773223+0000","last_in_change":"2024-04-24T23:15:14.905328+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T23:15:24.747441+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-24T23:15:34.051800+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-24T23:15:35.436870+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-24T23:15:39.034257+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"3b7a212d-d498-462e-90bf-ed493c803a09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":3022569437},{"type":"v1","addr":"172.21.15.6:6809","nonce":3022569437}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":3022569437},{"type":"v1","addr":"172.21.15.6:6811","nonce":3022569437}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":3022569437},{"type":"v1","addr":"172.21.15.6:6815","nonce":3022569437}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":3022569437},{"type":"v1","addr":"172.21.15.6:6813","nonce":3022569437}]},"public_addr":"172.21.15.6:6809/3022569437","cluster_addr":"172.21.15.6:6811/3022569437","heartbeat_back_addr":"172.21.15.6:6815/3022569437","heartbeat_front_addr":"172.21.15.6:6813/3022569437","state":["exists","up"]},{"osd":1,"uuid":"7c97f5a2-1735-4775-9f15-eb50099b19d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":3687836888},{"type":"v1","addr":"172.21.15.6:6817","nonce":3687836888}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":3687836888},{"type":"v1","addr":"172.21.15.6:6819","nonce":3687836888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":3687836888},{"type":"v1","addr":"172.21.15.6:6823","nonce":3687836888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":3687836888},{"type":"v1","addr":"172.21.15.6:6821","nonce":3687836888}]},"public_addr":"172.21.15.6:6817/3687836888","cluster_addr":"172.21.15.6:6819/3687836888","heartbeat_back_addr":"172.21.15.6:6823/3687836888","heartbeat_front_addr":"172.21.15.6:6821/3687836888","state":["exists","up"]},{"osd":2,"uuid":"d86bbc18-438c-4ec2-8018-ba7dd78f790b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":1020213724},{"type":"v1","addr":"172.21.15.6:6801","nonce":1020213724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":1020213724},{"type":"v1","addr":"172.21.15.6:6803","nonce":1020213724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":1020213724},{"type":"v1","addr":"172.21.15.6:6807","nonce":1020213724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":1020213724},{"type":"v1","addr":"172.21.15.6:6805","nonce":1020213724}]},"public_addr":"172.21.15.6:6801/1020213724","cluster_addr":"172.21.15.6:6803/1020213724","heartbeat_back_addr":"172.21.15.6:6807/1020213724","heartbeat_front_addr":"172.21.15.6:6805/1020213724","state":["exists","up"]},{"osd":3,"uuid":"3754ae16-4248-40dd-95f9-0f59c0eb5426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6824","nonce":3163883468},{"type":"v1","addr":"172.21.15.6:6825","nonce":3163883468}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":3163883468},{"type":"v1","addr":"172.21.15.6:6827","nonce":3163883468}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":3163883468},{"type":"v1","addr":"172.21.15.6:6831","nonce":3163883468}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":3163883468},{"type":"v1","addr":"172.21.15.6:6829","nonce":3163883468}]},"public_addr":"172.21.15.6:6825/3163883468","cluster_addr":"172.21.15.6:6827/3163883468","heartbeat_back_addr":"172.21.15.6:6831/3163883468","heartbeat_front_addr":"172.21.15.6:6829/3163883468","state":["exists","up"]},{"osd":4,"uuid":"8b2913a0-d006-479c-84ed-5b21e0e6c392","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":1989500934},{"type":"v1","addr":"172.21.15.190:6809","nonce":1989500934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":1989500934},{"type":"v1","addr":"172.21.15.190:6811","nonce":1989500934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":1989500934},{"type":"v1","addr":"172.21.15.190:6815","nonce":1989500934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":1989500934},{"type":"v1","addr":"172.21.15.190:6813","nonce":1989500934}]},"public_addr":"172.21.15.190:6809/1989500934","cluster_addr":"172.21.15.190:6811/1989500934","heartbeat_back_addr":"172.21.15.190:6815/1989500934","heartbeat_front_addr":"172.21.15.190:6813/1989500934","state":["exists","up"]},{"osd":5,"uuid":"499b79d0-0585-4dae-ab69-196a551793da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":3687548412},{"type":"v1","addr":"172.21.15.190:6801","nonce":3687548412}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":3687548412},{"type":"v1","addr":"172.21.15.190:6803","nonce":3687548412}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":3687548412},{"type":"v1","addr":"172.21.15.190:6807","nonce":3687548412}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":3687548412},{"type":"v1","addr":"172.21.15.190:6805","nonce":3687548412}]},"public_addr":"172.21.15.190:6801/3687548412","cluster_addr":"172.21.15.190:6803/3687548412","heartbeat_back_addr":"172.21.15.190:6807/3687548412","heartbeat_front_addr":"172.21.15.190:6805/3687548412","state":["exists","up"]},{"osd":6,"uuid":"6ba6df3b-4a6b-4a89-a890-0f79f387d4d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":870571564},{"type":"v1","addr":"172.21.15.190:6817","nonce":870571564}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":870571564},{"type":"v1","addr":"172.21.15.190:6819","nonce":870571564}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":870571564},{"type":"v1","addr":"172.21.15.190:6823","nonce":870571564}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":870571564},{"type":"v1","addr":"172.21.15.190:6821","nonce":870571564}]},"public_addr":"172.21.15.190:6817/870571564","cluster_addr":"172.21.15.190:6819/870571564","heartbeat_back_addr":"172.21.15.190:6823/870571564","heartbeat_front_addr":"172.21.15.190:6821/870571564","state":["exists","up"]},{"osd":7,"uuid":"e9668b1c-a000-4332-86e8-4efd73b02409","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":123639197},{"type":"v1","addr":"172.21.15.190:6825","nonce":123639197}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":123639197},{"type":"v1","addr":"172.21.15.190:6827","nonce":123639197}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":123639197},{"type":"v1","addr":"172.21.15.190:6831","nonce":123639197}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":123639197},{"type":"v1","addr":"172.21.15.190:6829","nonce":123639197}]},"public_addr":"172.21.15.190:6825/123639197","cluster_addr":"172.21.15.190:6827/123639197","heartbeat_back_addr":"172.21.15.190:6831/123639197","heartbeat_front_addr":"172.21.15.190:6829/123639197","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.321646+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.363873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.466963+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.531989+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:19.990707+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.043543+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.032492+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.115004+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.2f","mappings":[{"from":0,"to":7}]},{"pgid":"4.19","mappings":[{"from":7,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T23:16:52.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:52.448+0000 7f2435a00640 1 -- 172.21.15.6:0/3882552435 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f240c06be60 msgr2=0x7f240c06e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:52.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:52.448+0000 7f2435a00640 1 --2- 172.21.15.6:0/3882552435 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f240c06be60 0x7f240c06e320 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f24300b61c0 tx=0x7f2418001990 comp rx=0 tx=0).stop 2024-04-24T23:16:52.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:52.448+0000 7f2435a00640 1 -- 172.21.15.6:0/3882552435 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f24300cb090 msgr2=0x7f2430159b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:16:52.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:52.448+0000 7f2435a00640 1 --2- 172.21.15.6:0/3882552435 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f24300cb090 0x7f2430159b60 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f24200098c0 tx=0x7f24200046c0 comp rx=0 tx=0).stop 2024-04-24T23:16:52.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:52.449+0000 7f2435a00640 1 -- 172.21.15.6:0/3882552435 shutdown_connections 2024-04-24T23:16:52.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:52.449+0000 7f2435a00640 1 --2- 172.21.15.6:0/3882552435 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f240c06be60 0x7f240c06e320 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:52.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:52.449+0000 7f2435a00640 1 --2- 172.21.15.6:0/3882552435 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f24300cc3c0 0x7f24301e47a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:52.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:52.449+0000 7f2435a00640 1 --2- 172.21.15.6:0/3882552435 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2430154d90 0x7f24301e4ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:52.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:52.449+0000 7f2435a00640 1 --2- 172.21.15.6:0/3882552435 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f24300cb090 0x7f2430159b60 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:16:52.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:52.449+0000 7f2435a00640 1 -- 172.21.15.6:0/3882552435 >> 172.21.15.6:0/3882552435 conn(0x7f243006be10 msgr2=0x7f243014f4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:16:52.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:52.449+0000 7f2435a00640 1 -- 172.21.15.6:0/3882552435 shutdown_connections 2024-04-24T23:16:52.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:16:52.449+0000 7f2435a00640 1 -- 172.21.15.6:0/3882552435 wait complete. 2024-04-24T23:16:52.473 DEBUG:tasks.mds_thrash:Yielding 2024-04-24T23:16:52.473 INFO:teuthology.run_tasks:Running task workunit... 2024-04-24T23:16:52.512 INFO:tasks.workunit:Pulling workunits from ref d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-24T23:16:52.512 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-04-24T23:16:52.512 INFO:tasks.workunit:timeout=3h 2024-04-24T23:16:52.512 INFO:tasks.workunit:cleanup=True 2024-04-24T23:16:52.513 DEBUG:teuthology.orchestra.run.smithi006:> stat -- /home/ubuntu/cephtest/mnt.0 2024-04-24T23:16:52.513 INFO:tasks.mds_thrash.fs.[cephfs]:starting mds_do_thrash for fs cephfs 2024-04-24T23:16:52.513 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 79 secs before thrashing 2024-04-24T23:16:52.569 INFO:teuthology.orchestra.run.smithi006.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-04-24T23:16:52.569 INFO:teuthology.orchestra.run.smithi006.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-04-24T23:16:52.569 INFO:teuthology.orchestra.run.smithi006.stdout:Device: 28h/40d Inode: 1 Links: 2 2024-04-24T23:16:52.569 INFO:teuthology.orchestra.run.smithi006.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-24T23:16:52.569 INFO:teuthology.orchestra.run.smithi006.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-04-24T23:16:52.569 INFO:teuthology.orchestra.run.smithi006.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2024-04-24T23:16:52.569 INFO:teuthology.orchestra.run.smithi006.stdout:Modify: 2024-04-24 23:15:41.975097714 +0000 2024-04-24T23:16:52.569 INFO:teuthology.orchestra.run.smithi006.stdout:Change: 2024-04-24 23:15:53.275505451 +0000 2024-04-24T23:16:52.569 INFO:teuthology.orchestra.run.smithi006.stdout: Birth: 2024-04-24 23:15:41.975097714 +0000 2024-04-24T23:16:52.569 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-04-24T23:16:52.570 DEBUG:teuthology.orchestra.run.smithi006:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-04-24T23:16:52.666 DEBUG:teuthology.orchestra.run.smithi006:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-24T23:16:52.725 INFO:tasks.workunit.client.0.smithi006.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-04-24T23:17:02.628 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-24T23:17:02.636 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-24T23:17:32.746 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-24T23:17:32.754 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-24T23:18:02.854 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-24T23:18:02.860 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-24T23:18:09.678 INFO:tasks.workunit.client.0.smithi006.stderr:Updating files: 60% (7571/12492) Updating files: 61% (7621/12492) Updating files: 62% (7746/12492) Updating files: 63% (7870/12492) Updating files: 64% (7995/12492) Updating files: 65% (8120/12492) Updating files: 66% (8245/12492) Updating files: 67% (8370/12492) Updating files: 68% (8495/12492) Updating files: 69% (8620/12492) Updating files: 70% (8745/12492) Updating files: 71% (8870/12492) Updating files: 72% (8995/12492) Updating files: 73% (9120/12492) Updating files: 74% (9245/12492) Updating files: 75% (9369/12492) Updating files: 76% (9494/12492) Updating files: 77% (9619/12492) Updating files: 78% (9744/12492) Updating files: 79% (9869/12492) Updating files: 80% (9994/12492) Updating files: 81% (10119/12492) Updating files: 82% (10244/12492) Updating files: 83% (10369/12492) Updating files: 84% (10494/12492) Updating files: 85% (10619/12492) Updating files: 86% (10744/12492) Updating files: 87% (10869/12492) Updating files: 88% (10993/12492) Updating files: 89% (11118/12492) Updating files: 90% (11243/12492) Updating files: 91% (11368/12492) Updating files: 92% (11493/12492) Updating files: 93% (11618/12492) Updating files: 94% (11743/12492) Updating files: 95% (11868/12492) Updating files: 96% (11993/12492) Updating files: 96% (12064/12492) Updating files: 97% (12118/12492) Updating files: 98% (12243/12492) Updating files: 99% (12368/12492) Updating files: 100% (12492/12492) Updating files: 100% (12492/12492), done. 2024-04-24T23:18:10.413 INFO:tasks.workunit.client.0.smithi006.stderr:Note: switching to 'd2c45d977142e024a59b68a8f1dce8f0b9840551'. 2024-04-24T23:18:10.413 INFO:tasks.workunit.client.0.smithi006.stderr: 2024-04-24T23:18:10.413 INFO:tasks.workunit.client.0.smithi006.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-04-24T23:18:10.413 INFO:tasks.workunit.client.0.smithi006.stderr:changes and commit them, and you can discard any commits you make in this 2024-04-24T23:18:10.413 INFO:tasks.workunit.client.0.smithi006.stderr:state without impacting any branches by switching back to a branch. 2024-04-24T23:18:10.413 INFO:tasks.workunit.client.0.smithi006.stderr: 2024-04-24T23:18:10.413 INFO:tasks.workunit.client.0.smithi006.stderr:If you want to create a new branch to retain commits you create, you may 2024-04-24T23:18:10.413 INFO:tasks.workunit.client.0.smithi006.stderr:do so (now or later) by using -c with the switch command. Example: 2024-04-24T23:18:10.413 INFO:tasks.workunit.client.0.smithi006.stderr: 2024-04-24T23:18:10.413 INFO:tasks.workunit.client.0.smithi006.stderr: git switch -c 2024-04-24T23:18:10.413 INFO:tasks.workunit.client.0.smithi006.stderr: 2024-04-24T23:18:10.414 INFO:tasks.workunit.client.0.smithi006.stderr:Or undo this operation with: 2024-04-24T23:18:10.414 INFO:tasks.workunit.client.0.smithi006.stderr: 2024-04-24T23:18:10.414 INFO:tasks.workunit.client.0.smithi006.stderr: git switch - 2024-04-24T23:18:10.414 INFO:tasks.workunit.client.0.smithi006.stderr: 2024-04-24T23:18:10.414 INFO:tasks.workunit.client.0.smithi006.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-04-24T23:18:10.414 INFO:tasks.workunit.client.0.smithi006.stderr: 2024-04-24T23:18:10.414 INFO:tasks.workunit.client.0.smithi006.stderr:HEAD is now at d2c45d97714 Merge pull request #56528 from adk3798/squid-mypy-bump 2024-04-24T23:18:10.421 DEBUG:teuthology.orchestra.run.smithi006:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-04-24T23:18:10.482 INFO:tasks.workunit.client.0.smithi006.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-04-24T23:18:10.489 INFO:tasks.workunit.client.0.smithi006.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-24T23:18:10.489 INFO:tasks.workunit.client.0.smithi006.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-04-24T23:18:11.317 INFO:tasks.workunit.client.0.smithi006.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-04-24T23:18:11.424 INFO:tasks.workunit.client.0.smithi006.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-04-24T23:18:11.513 INFO:tasks.workunit.client.0.smithi006.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-24T23:18:11.515 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:18:11.520 INFO:tasks.workunit.client.0.smithi006.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-24T23:18:11.520 INFO:tasks.workunit.client.0.smithi006.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-04-24T23:18:11.600 INFO:tasks.workunit.client.0.smithi006.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-24T23:18:11.608 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-04-24T23:18:11.608 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-04-24T23:18:11.648 INFO:tasks.workunit:Running workunits matching suites/ffsb.sh on client.0... 2024-04-24T23:18:11.649 INFO:tasks.workunit:Running workunit suites/ffsb.sh... 2024-04-24T23:18:11.649 DEBUG:teuthology.orchestra.run.smithi006:workunit test suites/ffsb.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d2c45d977142e024a59b68a8f1dce8f0b9840551 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh 2024-04-24T23:18:11.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:11.704+0000 7fe797a00640 1 Processor -- start 2024-04-24T23:18:11.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:11.704+0000 7fe797a00640 1 -- start start 2024-04-24T23:18:11.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:11.706+0000 7fe797a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fe7900c9330 0x7fe79006eab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:18:11.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:11.706+0000 7fe797a00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe7900c6e70 con 0x7fe7900c8f50 2024-04-24T23:18:11.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:11.706+0000 7fe797a00640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe7900c7110 con 0x7fe790152bc0 2024-04-24T23:18:11.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:11.706+0000 7fe797a00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7fe7900c73d0 con 0x7fe7900c9330 2024-04-24T23:18:11.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:11.706+0000 7fe78fe00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fe7900c8f50 0x7fe79006e3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:40830/0 (socket says 172.21.15.6:40830) 2024-04-24T23:18:11.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:11.706+0000 7fe78fe00640 1 -- 172.21.15.6:0/3372748225 learned_addr learned my addr 172.21.15.6:0/3372748225 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:18:11.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:11.706+0000 7fe795a00640 1 --2- 172.21.15.6:0/3372748225 >> v2:172.21.15.6:3300/0 conn(0x7fe7900c9330 0x7fe79006eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:18:11.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:11.707+0000 7fe78f400640 1 -- 172.21.15.6:0/3372748225 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 384124332 0 0) 0x7fe7900c6e70 con 0x7fe7900c8f50 2024-04-24T23:18:11.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:11.707+0000 7fe795a00640 1 -- 172.21.15.6:0/3372748225 >> v1:172.21.15.190:6789/0 conn(0x7fe790152bc0 legacy=0x7fe790159390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:18:11.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:11.707+0000 7fe795a00640 1 -- 172.21.15.6:0/3372748225 >> v1:172.21.15.6:6789/0 conn(0x7fe7900c8f50 legacy=0x7fe79006e3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:18:11.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:11.707+0000 7fe795a00640 1 -- 172.21.15.6:0/3372748225 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7900c6700 con 0x7fe7900c9330 2024-04-24T23:18:11.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:11.708+0000 7fe795a00640 1 --2- 172.21.15.6:0/3372748225 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe7900c9330 0x7fe79006eab0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fe788009d90 tx=0x7fe78802a580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ee919fb97ae8cbd server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:18:11.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:11.708+0000 7fe78f400640 1 -- 172.21.15.6:0/3372748225 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe788007740 con 0x7fe7900c9330 2024-04-24T23:18:11.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:11.708+0000 7fe78f400640 1 -- 172.21.15.6:0/3372748225 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe7880078e0 con 0x7fe7900c9330 2024-04-24T23:18:11.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:11.708+0000 7fe78f400640 1 -- 172.21.15.6:0/3372748225 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe788007cb0 con 0x7fe7900c9330 2024-04-24T23:18:11.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:11.709+0000 7fe797a00640 1 -- 172.21.15.6:0/3372748225 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe7900c9330 msgr2=0x7fe79006eab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:18:11.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:11.709+0000 7fe797a00640 1 --2- 172.21.15.6:0/3372748225 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe7900c9330 0x7fe79006eab0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fe788009d90 tx=0x7fe78802a580 comp rx=0 tx=0).stop 2024-04-24T23:18:11.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:11.709+0000 7fe797a00640 1 -- 172.21.15.6:0/3372748225 shutdown_connections 2024-04-24T23:18:11.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:11.709+0000 7fe797a00640 1 --2- 172.21.15.6:0/3372748225 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe7900c9330 0x7fe79006eab0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:11.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:11.709+0000 7fe797a00640 1 -- 172.21.15.6:0/3372748225 >> 172.21.15.6:0/3372748225 conn(0x7fe79006b0e0 msgr2=0x7fe79006b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:18:11.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:11.709+0000 7fe797a00640 1 -- 172.21.15.6:0/3372748225 shutdown_connections 2024-04-24T23:18:11.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:11.709+0000 7fe797a00640 1 -- 172.21.15.6:0/3372748225 wait complete. 2024-04-24T23:18:11.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:11.711+0000 7fe797a00640 1 Processor -- start 2024-04-24T23:18:11.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:11.711+0000 7fe797a00640 1 -- start start 2024-04-24T23:18:11.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:11.712+0000 7fe797a00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe7900c8f50 0x7fe790146d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:18:11.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:11.712+0000 7fe797a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe7900c9330 0x7fe790071970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:18:11.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:11.712+0000 7fe797a00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe790152bc0 0x7fe790071eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:18:11.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:11.712+0000 7fe797a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fe7900c8750 con 0x7fe7900c9330 2024-04-24T23:18:11.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:11.712+0000 7fe797a00640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fe7900c8ac0 con 0x7fe7900c8f50 2024-04-24T23:18:11.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:11.712+0000 7fe797a00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fe790152fa0 con 0x7fe790152bc0 2024-04-24T23:18:11.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:11.712+0000 7fe78fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe7900c9330 0x7fe790071970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:18:11.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:11.712+0000 7fe795000640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe7900c8f50 0x7fe790146d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:18:11.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:11.712+0000 7fe78fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe7900c9330 0x7fe790071970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:59688/0 (socket says 172.21.15.6:59688) 2024-04-24T23:18:11.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:11.712+0000 7fe78fe00640 1 -- 172.21.15.6:0/1551859252 learned_addr learned my addr 172.21.15.6:0/1551859252 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:18:11.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:11.712+0000 7fe795000640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe7900c8f50 0x7fe790146d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:34726/0 (socket says 172.21.15.6:34726) 2024-04-24T23:18:11.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:11.712+0000 7fe795a00640 1 --2- 172.21.15.6:0/1551859252 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe790152bc0 0x7fe790071eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:18:11.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:11.713+0000 7fe795000640 1 -- 172.21.15.6:0/1551859252 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe790152bc0 msgr2=0x7fe790071eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:18:11.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:11.713+0000 7fe795000640 1 --2- 172.21.15.6:0/1551859252 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe790152bc0 0x7fe790071eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:11.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:11.713+0000 7fe795000640 1 -- 172.21.15.6:0/1551859252 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe7900c9330 msgr2=0x7fe790071970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:18:11.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:11.713+0000 7fe795000640 1 --2- 172.21.15.6:0/1551859252 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe7900c9330 0x7fe790071970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:11.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:11.713+0000 7fe795000640 1 -- 172.21.15.6:0/1551859252 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7901f1300 con 0x7fe7900c8f50 2024-04-24T23:18:11.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:11.713+0000 7fe78fe00640 1 --2- 172.21.15.6:0/1551859252 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe7900c9330 0x7fe790071970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:18:11.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:11.713+0000 7fe795000640 1 --2- 172.21.15.6:0/1551859252 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe7900c8f50 0x7fe790146d50 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fe78400c4a0 tx=0x7fe78400c960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:18:11.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:11.713+0000 7fe78cc00640 1 -- 172.21.15.6:0/1551859252 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe784017070 con 0x7fe7900c8f50 2024-04-24T23:18:11.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:11.714+0000 7fe78cc00640 1 -- 172.21.15.6:0/1551859252 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe784005480 con 0x7fe7900c8f50 2024-04-24T23:18:11.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:11.714+0000 7fe78cc00640 1 -- 172.21.15.6:0/1551859252 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe784003870 con 0x7fe7900c8f50 2024-04-24T23:18:11.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:11.714+0000 7fe797a00640 1 -- 172.21.15.6:0/1551859252 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe790147290 con 0x7fe7900c8f50 2024-04-24T23:18:11.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:11.714+0000 7fe797a00640 1 -- 172.21.15.6:0/1551859252 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe79015a2b0 con 0x7fe7900c8f50 2024-04-24T23:18:11.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:11.715+0000 7fe797a00640 1 -- 172.21.15.6:0/1551859252 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe790156580 con 0x7fe7900c8f50 2024-04-24T23:18:11.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:11.716+0000 7fe78cc00640 1 -- 172.21.15.6:0/1551859252 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fe78401a070 con 0x7fe7900c8f50 2024-04-24T23:18:11.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:11.717+0000 7fe78cc00640 1 --2- 172.21.15.6:0/1551859252 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fe77006be60 0x7fe77006e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:18:11.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:11.717+0000 7fe78cc00640 1 -- 172.21.15.6:0/1551859252 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6590+0+0 (secure 0 0 0) 0x7fe78408a570 con 0x7fe7900c8f50 2024-04-24T23:18:11.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:11.717+0000 7fe78fe00640 1 --2- 172.21.15.6:0/1551859252 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fe77006be60 0x7fe77006e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:18:11.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:11.718+0000 7fe78fe00640 1 --2- 172.21.15.6:0/1551859252 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fe77006be60 0x7fe77006e320 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fe79006b830 tx=0x7fe77c007a60 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:18:11.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:11.722+0000 7fe78cc00640 1 -- 172.21.15.6:0/1551859252 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fe790156580 con 0x7fe7900c8f50 2024-04-24T23:18:11.749 INFO:tasks.workunit.client.0.smithi006.stderr:++ dirname /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh 2024-04-24T23:18:11.751 INFO:tasks.workunit.client.0.smithi006.stderr:+ mydir=/home/ubuntu/cephtest/clone.client.0/qa/workunits/suites 2024-04-24T23:18:11.752 INFO:tasks.workunit.client.0.smithi006.stderr:+ trap -- retry EXIT 2024-04-24T23:18:11.752 INFO:tasks.workunit.client.0.smithi006.stderr:+ rm -rf ffsb 2024-04-24T23:18:11.754 INFO:tasks.workunit.client.0.smithi006.stderr:+ timeout 1800 git clone https://git.ceph.com/ffsb.git --depth 1 2024-04-24T23:18:11.760 INFO:tasks.workunit.client.0.smithi006.stderr:Cloning into 'ffsb'... 2024-04-24T23:18:12.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:12.013+0000 7fe797a00640 1 -- 172.21.15.6:0/1551859252 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe79014cc40 con 0x7fe7900c8f50 2024-04-24T23:18:12.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:12.014+0000 7fe78cc00640 1 -- 172.21.15.6:0/1551859252 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) ==== 94+0+6902 (secure 0 0 0) 0x7fe79014cc40 con 0x7fe7900c8f50 2024-04-24T23:18:12.016 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:18:12.017 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":18,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4197,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/105267896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":105267896},{"type":"v1","addr":"172.21.15.6:6834","nonce":105267896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:16:32.310838+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4282,"mds_2":4235,"mds_3":4232,"mds_4":4203},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4203":{"gid":4203,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":17,"addr":"172.21.15.6:6837/2590421094","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":2590421094},{"type":"v1","addr":"172.21.15.6:6837","nonce":2590421094}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4232":{"gid":4232,"name":"b","rank":3,"incarnation":15,"state":"up:active","state_seq":15,"addr":"172.21.15.190:6838/2032280158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2032280158},{"type":"v1","addr":"172.21.15.190:6838","nonce":2032280158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4235":{"gid":4235,"name":"d","rank":2,"incarnation":13,"state":"up:active","state_seq":14,"addr":"172.21.15.190:6839/1572872451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":1572872451},{"type":"v1","addr":"172.21.15.190:6839","nonce":1572872451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4282":{"gid":4282,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":12,"addr":"172.21.15.190:6837/580315269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":580315269},{"type":"v1","addr":"172.21.15.190:6837","nonce":580315269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/489092393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":489092393},{"type":"v1","addr":"172.21.15.6:6835","nonce":489092393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4306,"qdb_cluster":[4203,4232,4235,4282,4306]},"id":1}]} 2024-04-24T23:18:12.017 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 18 2024-04-24T23:18:12.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:12.016+0000 7fe797a00640 1 -- 172.21.15.6:0/1551859252 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fe77006be60 msgr2=0x7fe77006e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:18:12.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:12.016+0000 7fe797a00640 1 --2- 172.21.15.6:0/1551859252 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fe77006be60 0x7fe77006e320 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fe79006b830 tx=0x7fe77c007a60 comp rx=0 tx=0).stop 2024-04-24T23:18:12.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:12.016+0000 7fe797a00640 1 -- 172.21.15.6:0/1551859252 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe7900c8f50 msgr2=0x7fe790146d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:18:12.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:12.016+0000 7fe797a00640 1 --2- 172.21.15.6:0/1551859252 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe7900c8f50 0x7fe790146d50 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fe78400c4a0 tx=0x7fe78400c960 comp rx=0 tx=0).stop 2024-04-24T23:18:12.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:12.016+0000 7fe797a00640 1 -- 172.21.15.6:0/1551859252 shutdown_connections 2024-04-24T23:18:12.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:12.016+0000 7fe797a00640 1 --2- 172.21.15.6:0/1551859252 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fe77006be60 0x7fe77006e320 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:12.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:12.016+0000 7fe797a00640 1 --2- 172.21.15.6:0/1551859252 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe790152bc0 0x7fe790071eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:12.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:12.016+0000 7fe797a00640 1 --2- 172.21.15.6:0/1551859252 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe7900c9330 0x7fe790071970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:12.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:12.016+0000 7fe797a00640 1 --2- 172.21.15.6:0/1551859252 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe7900c8f50 0x7fe790146d50 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:12.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:12.016+0000 7fe797a00640 1 -- 172.21.15.6:0/1551859252 >> 172.21.15.6:0/1551859252 conn(0x7fe79006b0e0 msgr2=0x7fe7900ccab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:18:12.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:12.017+0000 7fe797a00640 1 -- 172.21.15.6:0/1551859252 shutdown_connections 2024-04-24T23:18:12.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:12.017+0000 7fe797a00640 1 -- 172.21.15.6:0/1551859252 wait complete. 2024-04-24T23:18:12.039 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.e (rank=4) 2024-04-24T23:18:12.040 DEBUG:tasks.ceph.mds.e:waiting for process to exit 2024-04-24T23:18:12.040 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T23:18:12.104 INFO:tasks.ceph.mds.e:Stopped 2024-04-24T23:18:12.105 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:18:12.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:12.334+0000 7f2633200640 1 Processor -- start 2024-04-24T23:18:12.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:12.335+0000 7f2633200640 1 -- start start 2024-04-24T23:18:12.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:12.336+0000 7f2633200640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f263415b5d0 0x7f2634166400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:18:12.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:12.336+0000 7f2633200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f263415ba40 con 0x7f2634154d70 2024-04-24T23:18:12.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:12.336+0000 7f2633200640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f263406cc20 con 0x7f26340c6b10 2024-04-24T23:18:12.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:12.336+0000 7f2633200640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f263406c9f0 con 0x7f263415b5d0 2024-04-24T23:18:12.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:12.336+0000 7f2631e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f2634154d70 0x7f26340c7890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:40836/0 (socket says 172.21.15.6:40836) 2024-04-24T23:18:12.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:12.336+0000 7f2631e00640 1 -- 172.21.15.6:0/1160997360 learned_addr learned my addr 172.21.15.6:0/1160997360 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:18:12.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:12.337+0000 7f2632800640 1 --2- 172.21.15.6:0/1160997360 >> v2:172.21.15.190:3300/0 conn(0x7f263415b5d0 0x7f2634166400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:18:12.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:12.337+0000 7f2631400640 1 -- 172.21.15.6:0/1160997360 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3747139145 0 0) 0x7f263415ba40 con 0x7f2634154d70 2024-04-24T23:18:12.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:12.337+0000 7f2631400640 1 -- 172.21.15.6:0/1160997360 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2618003520 con 0x7f2634154d70 2024-04-24T23:18:12.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:12.337+0000 7f2632800640 1 -- 172.21.15.6:0/1160997360 >> v1:172.21.15.190:6789/0 conn(0x7f26340c6b10 legacy=0x7f26340c85f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:18:12.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:12.337+0000 7f2632800640 1 -- 172.21.15.6:0/1160997360 >> v1:172.21.15.6:6789/0 conn(0x7f2634154d70 legacy=0x7f26340c7890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:18:12.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:12.337+0000 7f2632800640 1 -- 172.21.15.6:0/1160997360 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f263406cf00 con 0x7f263415b5d0 2024-04-24T23:18:12.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:12.338+0000 7f2632800640 1 --2- 172.21.15.6:0/1160997360 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f263415b5d0 0x7f2634166400 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f262c0097a0 tx=0x7f262c02a2c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b677acbdfbfff147 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:18:12.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:12.338+0000 7f2631400640 1 -- 172.21.15.6:0/1160997360 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f262c037070 con 0x7f263415b5d0 2024-04-24T23:18:12.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:12.339+0000 7f2631400640 1 -- 172.21.15.6:0/1160997360 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f262c032640 con 0x7f263415b5d0 2024-04-24T23:18:12.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:12.339+0000 7f2631400640 1 -- 172.21.15.6:0/1160997360 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f262c0329f0 con 0x7f263415b5d0 2024-04-24T23:18:12.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:12.339+0000 7f2633200640 1 -- 172.21.15.6:0/1160997360 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f263415b5d0 msgr2=0x7f2634166400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:18:12.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:12.339+0000 7f2633200640 1 --2- 172.21.15.6:0/1160997360 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f263415b5d0 0x7f2634166400 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f262c0097a0 tx=0x7f262c02a2c0 comp rx=0 tx=0).stop 2024-04-24T23:18:12.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:12.339+0000 7f2633200640 1 -- 172.21.15.6:0/1160997360 shutdown_connections 2024-04-24T23:18:12.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:12.339+0000 7f2633200640 1 --2- 172.21.15.6:0/1160997360 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f263415b5d0 0x7f2634166400 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:12.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:12.339+0000 7f2633200640 1 -- 172.21.15.6:0/1160997360 >> 172.21.15.6:0/1160997360 conn(0x7f26340c9320 msgr2=0x7f26340c9700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:18:12.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:12.339+0000 7f2633200640 1 -- 172.21.15.6:0/1160997360 shutdown_connections 2024-04-24T23:18:12.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:12.339+0000 7f2633200640 1 -- 172.21.15.6:0/1160997360 wait complete. 2024-04-24T23:18:12.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:12.341+0000 7f2633200640 1 Processor -- start 2024-04-24T23:18:12.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:12.341+0000 7f2633200640 1 -- start start 2024-04-24T23:18:12.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:12.342+0000 7f2633200640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f26340c6b10 0x7f26341f0e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:18:12.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:12.342+0000 7f2633200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2634154d70 0x7f26341e70b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:18:12.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:12.342+0000 7f2633200640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f263415b5d0 0x7f26340c4df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:18:12.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:12.342+0000 7f2633200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f263406d4c0 con 0x7f2634154d70 2024-04-24T23:18:12.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:12.342+0000 7f2633200640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f26340c7120 con 0x7f26340c6b10 2024-04-24T23:18:12.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:12.342+0000 7f2633200640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f26340c7370 con 0x7f263415b5d0 2024-04-24T23:18:12.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:12.342+0000 7f2631e00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f26340c6b10 0x7f26341f0e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:18:12.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:12.342+0000 7f2631e00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f26340c6b10 0x7f26341f0e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:34742/0 (socket says 172.21.15.6:34742) 2024-04-24T23:18:12.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:12.342+0000 7f2631e00640 1 -- 172.21.15.6:0/2155143764 learned_addr learned my addr 172.21.15.6:0/2155143764 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:18:12.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:12.342+0000 7f2632800640 1 --2- 172.21.15.6:0/2155143764 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f263415b5d0 0x7f26340c4df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:18:12.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:12.343+0000 7f2629400640 1 --2- 172.21.15.6:0/2155143764 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2634154d70 0x7f26341e70b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:18:12.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:12.343+0000 7f2631e00640 1 -- 172.21.15.6:0/2155143764 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f263415b5d0 msgr2=0x7f26340c4df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:18:12.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:12.343+0000 7f2631e00640 1 --2- 172.21.15.6:0/2155143764 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f263415b5d0 0x7f26340c4df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:12.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:12.343+0000 7f2631e00640 1 -- 172.21.15.6:0/2155143764 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2634154d70 msgr2=0x7f26341e70b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:18:12.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:12.343+0000 7f2631e00640 1 --2- 172.21.15.6:0/2155143764 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2634154d70 0x7f26341e70b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:12.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:12.343+0000 7f2631e00640 1 -- 172.21.15.6:0/2155143764 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26340c5ca0 con 0x7f26340c6b10 2024-04-24T23:18:12.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:12.343+0000 7f2631e00640 1 --2- 172.21.15.6:0/2155143764 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f26340c6b10 0x7f26341f0e20 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f262400cbc0 tx=0x7f262400a430 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:18:12.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:12.343+0000 7f262aa00640 1 -- 172.21.15.6:0/2155143764 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2624017070 con 0x7f26340c6b10 2024-04-24T23:18:12.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:12.343+0000 7f262aa00640 1 -- 172.21.15.6:0/2155143764 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f262400ad40 con 0x7f26340c6b10 2024-04-24T23:18:12.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:12.343+0000 7f262aa00640 1 -- 172.21.15.6:0/2155143764 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2624012600 con 0x7f26340c6b10 2024-04-24T23:18:12.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:12.343+0000 7f2633200640 1 -- 172.21.15.6:0/2155143764 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26340c5f40 con 0x7f26340c6b10 2024-04-24T23:18:12.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:12.344+0000 7f2633200640 1 -- 172.21.15.6:0/2155143764 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26340c45f0 con 0x7f26340c6b10 2024-04-24T23:18:12.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:12.344+0000 7f2633200640 1 -- 172.21.15.6:0/2155143764 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2634075b30 con 0x7f26340c6b10 2024-04-24T23:18:12.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:12.346+0000 7f262aa00640 1 -- 172.21.15.6:0/2155143764 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f262401a070 con 0x7f26340c6b10 2024-04-24T23:18:12.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:12.347+0000 7f262aa00640 1 --2- 172.21.15.6:0/2155143764 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f261006be30 0x7f261006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:18:12.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:12.347+0000 7f262aa00640 1 -- 172.21.15.6:0/2155143764 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6590+0+0 (secure 0 0 0) 0x7f262408b390 con 0x7f26340c6b10 2024-04-24T23:18:12.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:12.347+0000 7f2629400640 1 --2- 172.21.15.6:0/2155143764 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f261006be30 0x7f261006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:18:12.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:12.348+0000 7f2629400640 1 --2- 172.21.15.6:0/2155143764 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f261006be30 0x7f261006e2f0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f26340c3a10 tx=0x7f261c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:18:12.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:12.351+0000 7f262aa00640 1 -- 172.21.15.6:0/2155143764 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f2624059360 con 0x7f26340c6b10 2024-04-24T23:18:12.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:12.597+0000 7f2633200640 1 -- 172.21.15.6:0/2155143764 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f263406da60 con 0x7f26340c6b10 2024-04-24T23:18:12.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:12.598+0000 7f262aa00640 1 -- 172.21.15.6:0/2155143764 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) ==== 94+0+6902 (secure 0 0 0) 0x7f262405d660 con 0x7f26340c6b10 2024-04-24T23:18:12.601 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:18:12.601 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":18,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4197,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/105267896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":105267896},{"type":"v1","addr":"172.21.15.6:6834","nonce":105267896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:16:32.310838+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4282,"mds_2":4235,"mds_3":4232,"mds_4":4203},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4203":{"gid":4203,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":17,"addr":"172.21.15.6:6837/2590421094","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":2590421094},{"type":"v1","addr":"172.21.15.6:6837","nonce":2590421094}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4232":{"gid":4232,"name":"b","rank":3,"incarnation":15,"state":"up:active","state_seq":15,"addr":"172.21.15.190:6838/2032280158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2032280158},{"type":"v1","addr":"172.21.15.190:6838","nonce":2032280158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4235":{"gid":4235,"name":"d","rank":2,"incarnation":13,"state":"up:active","state_seq":14,"addr":"172.21.15.190:6839/1572872451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":1572872451},{"type":"v1","addr":"172.21.15.190:6839","nonce":1572872451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4282":{"gid":4282,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":12,"addr":"172.21.15.190:6837/580315269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":580315269},{"type":"v1","addr":"172.21.15.190:6837","nonce":580315269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/489092393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":489092393},{"type":"v1","addr":"172.21.15.6:6835","nonce":489092393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4306,"qdb_cluster":[4203,4232,4235,4282,4306]},"id":1}]} 2024-04-24T23:18:12.601 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 18 2024-04-24T23:18:12.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:12.601+0000 7f2633200640 1 -- 172.21.15.6:0/2155143764 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f261006be30 msgr2=0x7f261006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:18:12.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:12.601+0000 7f2633200640 1 --2- 172.21.15.6:0/2155143764 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f261006be30 0x7f261006e2f0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f26340c3a10 tx=0x7f261c009040 comp rx=0 tx=0).stop 2024-04-24T23:18:12.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:12.601+0000 7f2633200640 1 -- 172.21.15.6:0/2155143764 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f26340c6b10 msgr2=0x7f26341f0e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:18:12.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:12.601+0000 7f2633200640 1 --2- 172.21.15.6:0/2155143764 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f26340c6b10 0x7f26341f0e20 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f262400cbc0 tx=0x7f262400a430 comp rx=0 tx=0).stop 2024-04-24T23:18:12.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:12.601+0000 7f2633200640 1 -- 172.21.15.6:0/2155143764 shutdown_connections 2024-04-24T23:18:12.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:12.601+0000 7f2633200640 1 --2- 172.21.15.6:0/2155143764 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f261006be30 0x7f261006e2f0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:12.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:12.601+0000 7f2633200640 1 --2- 172.21.15.6:0/2155143764 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f263415b5d0 0x7f26340c4df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:12.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:12.601+0000 7f2633200640 1 --2- 172.21.15.6:0/2155143764 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2634154d70 0x7f26341e70b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:12.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:12.601+0000 7f2633200640 1 --2- 172.21.15.6:0/2155143764 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f26340c6b10 0x7f26341f0e20 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:12.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:12.601+0000 7f2633200640 1 -- 172.21.15.6:0/2155143764 >> 172.21.15.6:0/2155143764 conn(0x7f26340c9320 msgr2=0x7f263414fab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:18:12.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:12.601+0000 7f2633200640 1 -- 172.21.15.6:0/2155143764 shutdown_connections 2024-04-24T23:18:12.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:12.601+0000 7f2633200640 1 -- 172.21.15.6:0/2155143764 wait complete. 2024-04-24T23:18:12.622 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2024-04-24T23:18:14.623 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:18:14.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:14.846+0000 7f52cec00640 1 Processor -- start 2024-04-24T23:18:14.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:14.846+0000 7f52cec00640 1 -- start start 2024-04-24T23:18:14.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:14.847+0000 7f52cec00640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f52c8152c00 0x7f52c81589f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:18:14.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:14.847+0000 7f52cec00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f52c80c97f0 0x7f52c8163350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:18:14.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:14.847+0000 7f52cec00640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f52c80c7ca0 con 0x7f52c80cde30 2024-04-24T23:18:14.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:14.847+0000 7f52cec00640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f52c815c740 con 0x7f52c8152c00 2024-04-24T23:18:14.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:14.847+0000 7f52cec00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f52c8158f30 con 0x7f52c80c97f0 2024-04-24T23:18:14.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:14.848+0000 7f52bfe00640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f52c80cde30 0x7f52c80c69a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.6:40780/0 (socket says 172.21.15.6:40780) 2024-04-24T23:18:14.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:14.848+0000 7f52bfe00640 1 -- 172.21.15.6:0/318125646 learned_addr learned my addr 172.21.15.6:0/318125646 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:18:14.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:14.848+0000 7f52c7e00640 1 --2- 172.21.15.6:0/318125646 >> v2:172.21.15.190:3300/0 conn(0x7f52c8152c00 0x7f52c81589f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:18:14.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:14.848+0000 7f52ccc00640 1 --2- 172.21.15.6:0/318125646 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f52c80c97f0 0x7f52c8163350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:18:14.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:14.848+0000 7f52c7400640 1 -- 172.21.15.6:0/318125646 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3990981713 0 0) 0x7f52c80c7ca0 con 0x7f52c80cde30 2024-04-24T23:18:14.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:14.848+0000 7f52c7400640 1 -- 172.21.15.6:0/318125646 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f52ac003520 con 0x7f52c80cde30 2024-04-24T23:18:14.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:14.848+0000 7f52c7e00640 1 -- 172.21.15.6:0/318125646 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f52c80c97f0 msgr2=0x7f52c8163350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:18:14.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:14.848+0000 7f52c7e00640 1 --2- 172.21.15.6:0/318125646 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f52c80c97f0 0x7f52c8163350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:14.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:14.849+0000 7f52c7e00640 1 -- 172.21.15.6:0/318125646 >> v1:172.21.15.190:6789/0 conn(0x7f52c80cde30 legacy=0x7f52c80c69a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:18:14.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:14.849+0000 7f52c7400640 1 -- 172.21.15.6:0/318125646 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1240708503 0 0) 0x7f52ac003520 con 0x7f52c80cde30 2024-04-24T23:18:14.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:14.849+0000 7f52c7e00640 1 -- 172.21.15.6:0/318125646 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52c80c7f60 con 0x7f52c8152c00 2024-04-24T23:18:14.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:14.850+0000 7f52c7e00640 1 --2- 172.21.15.6:0/318125646 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f52c8152c00 0x7f52c81589f0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f52b80097a0 tx=0x7f52b802a110 comp rx=0 tx=0).ready entity=mon.1 client_cookie=65891a36020ce601 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:18:14.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:14.850+0000 7f52c7400640 1 -- 172.21.15.6:0/318125646 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52b8037070 con 0x7f52c8152c00 2024-04-24T23:18:14.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:14.850+0000 7f52c7400640 1 -- 172.21.15.6:0/318125646 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f52b802aba0 con 0x7f52c8152c00 2024-04-24T23:18:14.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:14.850+0000 7f52c7400640 1 -- 172.21.15.6:0/318125646 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52b0001690 con 0x7f52c8152c00 2024-04-24T23:18:14.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:14.851+0000 7f52cec00640 1 -- 172.21.15.6:0/318125646 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f52c8152c00 msgr2=0x7f52c81589f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:18:14.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:14.851+0000 7f52cec00640 1 --2- 172.21.15.6:0/318125646 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f52c8152c00 0x7f52c81589f0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f52b80097a0 tx=0x7f52b802a110 comp rx=0 tx=0).stop 2024-04-24T23:18:14.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:14.851+0000 7f52cec00640 1 -- 172.21.15.6:0/318125646 shutdown_connections 2024-04-24T23:18:14.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:14.851+0000 7f52cec00640 1 --2- 172.21.15.6:0/318125646 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f52c80c97f0 0x7f52c8163350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:14.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:14.851+0000 7f52cec00640 1 --2- 172.21.15.6:0/318125646 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f52c8152c00 0x7f52c81589f0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:14.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:14.851+0000 7f52cec00640 1 -- 172.21.15.6:0/318125646 >> 172.21.15.6:0/318125646 conn(0x7f52c80cd290 msgr2=0x7f52c80cd670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:18:14.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:14.852+0000 7f52cec00640 1 -- 172.21.15.6:0/318125646 shutdown_connections 2024-04-24T23:18:14.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:14.852+0000 7f52cec00640 1 -- 172.21.15.6:0/318125646 wait complete. 2024-04-24T23:18:14.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:14.855+0000 7f52cec00640 1 Processor -- start 2024-04-24T23:18:14.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:14.855+0000 7f52cec00640 1 -- start start 2024-04-24T23:18:14.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:14.856+0000 7f52cec00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f52c80c97f0 0x7f52c8156ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:18:14.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:14.856+0000 7f52cec00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f52c80cde30 0x7f52c8159c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:18:14.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:14.857+0000 7f52cec00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f52c8152c00 0x7f52c815a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:18:14.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:14.857+0000 7f52cec00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f52c80c8240 con 0x7f52c80cde30 2024-04-24T23:18:14.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:14.857+0000 7f52cec00640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f52c815c3d0 con 0x7f52c80c97f0 2024-04-24T23:18:14.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:14.857+0000 7f52cec00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f52c8073170 con 0x7f52c8152c00 2024-04-24T23:18:14.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:14.857+0000 7f52bfe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f52c80cde30 0x7f52c8159c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:18:14.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:14.857+0000 7f52c7e00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f52c80c97f0 0x7f52c8156ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:18:14.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:14.857+0000 7f52c7e00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f52c80c97f0 0x7f52c8156ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:46436/0 (socket says 172.21.15.6:46436) 2024-04-24T23:18:14.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:14.857+0000 7f52c7e00640 1 -- 172.21.15.6:0/3577404058 learned_addr learned my addr 172.21.15.6:0/3577404058 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:18:14.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:14.857+0000 7f52ccc00640 1 --2- 172.21.15.6:0/3577404058 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f52c8152c00 0x7f52c815a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:18:14.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:14.857+0000 7f52c7e00640 1 -- 172.21.15.6:0/3577404058 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f52c8152c00 msgr2=0x7f52c815a1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:18:14.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:14.857+0000 7f52c7e00640 1 --2- 172.21.15.6:0/3577404058 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f52c8152c00 0x7f52c815a1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:14.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:14.857+0000 7f52c7e00640 1 -- 172.21.15.6:0/3577404058 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f52c80cde30 msgr2=0x7f52c8159c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:18:14.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:14.857+0000 7f52c7e00640 1 --2- 172.21.15.6:0/3577404058 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f52c80cde30 0x7f52c8159c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:14.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:14.857+0000 7f52c7e00640 1 -- 172.21.15.6:0/3577404058 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52c80cc730 con 0x7f52c80c97f0 2024-04-24T23:18:14.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:14.857+0000 7f52bfe00640 1 --2- 172.21.15.6:0/3577404058 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f52c80cde30 0x7f52c8159c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:18:14.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:14.858+0000 7f52c7e00640 1 --2- 172.21.15.6:0/3577404058 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f52c80c97f0 0x7f52c8156ad0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f52b8035040 tx=0x7f52b80046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:18:14.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:14.858+0000 7f52c4c00640 1 -- 172.21.15.6:0/3577404058 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52b8041020 con 0x7f52c80c97f0 2024-04-24T23:18:14.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:14.858+0000 7f52c4c00640 1 -- 172.21.15.6:0/3577404058 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f52b803d620 con 0x7f52c80c97f0 2024-04-24T23:18:14.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:14.858+0000 7f52c4c00640 1 -- 172.21.15.6:0/3577404058 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52b8037040 con 0x7f52c80c97f0 2024-04-24T23:18:14.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:14.858+0000 7f52cec00640 1 -- 172.21.15.6:0/3577404058 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52c815b910 con 0x7f52c80c97f0 2024-04-24T23:18:14.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:14.858+0000 7f52cec00640 1 -- 172.21.15.6:0/3577404058 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52c80c4f10 con 0x7f52c80c97f0 2024-04-24T23:18:14.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:14.860+0000 7f52cec00640 1 -- 172.21.15.6:0/3577404058 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f52c8068430 con 0x7f52c80c97f0 2024-04-24T23:18:14.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:14.861+0000 7f52c4c00640 1 -- 172.21.15.6:0/3577404058 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f52b8044020 con 0x7f52c80c97f0 2024-04-24T23:18:14.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:14.862+0000 7f52c4c00640 1 --2- 172.21.15.6:0/3577404058 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f52a406be80 0x7f52a406e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:18:14.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:14.863+0000 7f52c4c00640 1 -- 172.21.15.6:0/3577404058 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6590+0+0 (secure 0 0 0) 0x7f52b80ab660 con 0x7f52c80c97f0 2024-04-24T23:18:14.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:14.863+0000 7f52bfe00640 1 --2- 172.21.15.6:0/3577404058 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f52a406be80 0x7f52a406e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:18:14.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:14.864+0000 7f52bfe00640 1 --2- 172.21.15.6:0/3577404058 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f52a406be80 0x7f52a406e340 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f52b0001800 tx=0x7f52b0001e00 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:18:14.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:14.867+0000 7f52c4c00640 1 -- 172.21.15.6:0/3577404058 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f52b804b030 con 0x7f52c80c97f0 2024-04-24T23:18:15.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:15.152+0000 7f52cec00640 1 -- 172.21.15.6:0/3577404058 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f52c80c4cc0 con 0x7f52c80c97f0 2024-04-24T23:18:15.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:15.153+0000 7f52c4c00640 1 -- 172.21.15.6:0/3577404058 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) ==== 94+0+6902 (secure 0 0 0) 0x7f52c80cc730 con 0x7f52c80c97f0 2024-04-24T23:18:15.156 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:18:15.156 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":18,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4197,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/105267896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":105267896},{"type":"v1","addr":"172.21.15.6:6834","nonce":105267896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:16:32.310838+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4282,"mds_2":4235,"mds_3":4232,"mds_4":4203},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4203":{"gid":4203,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":17,"addr":"172.21.15.6:6837/2590421094","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":2590421094},{"type":"v1","addr":"172.21.15.6:6837","nonce":2590421094}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4232":{"gid":4232,"name":"b","rank":3,"incarnation":15,"state":"up:active","state_seq":15,"addr":"172.21.15.190:6838/2032280158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2032280158},{"type":"v1","addr":"172.21.15.190:6838","nonce":2032280158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4235":{"gid":4235,"name":"d","rank":2,"incarnation":13,"state":"up:active","state_seq":14,"addr":"172.21.15.190:6839/1572872451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":1572872451},{"type":"v1","addr":"172.21.15.190:6839","nonce":1572872451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4282":{"gid":4282,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":12,"addr":"172.21.15.190:6837/580315269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":580315269},{"type":"v1","addr":"172.21.15.190:6837","nonce":580315269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/489092393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":489092393},{"type":"v1","addr":"172.21.15.6:6835","nonce":489092393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4306,"qdb_cluster":[4203,4232,4235,4282,4306]},"id":1}]} 2024-04-24T23:18:15.156 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 18 2024-04-24T23:18:15.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:15.156+0000 7f52cec00640 1 -- 172.21.15.6:0/3577404058 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f52a406be80 msgr2=0x7f52a406e340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:18:15.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:15.156+0000 7f52cec00640 1 --2- 172.21.15.6:0/3577404058 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f52a406be80 0x7f52a406e340 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f52b0001800 tx=0x7f52b0001e00 comp rx=0 tx=0).stop 2024-04-24T23:18:15.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:15.156+0000 7f52cec00640 1 -- 172.21.15.6:0/3577404058 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f52c80c97f0 msgr2=0x7f52c8156ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:18:15.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:15.156+0000 7f52cec00640 1 --2- 172.21.15.6:0/3577404058 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f52c80c97f0 0x7f52c8156ad0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f52b8035040 tx=0x7f52b80046c0 comp rx=0 tx=0).stop 2024-04-24T23:18:15.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:15.156+0000 7f52cec00640 1 -- 172.21.15.6:0/3577404058 shutdown_connections 2024-04-24T23:18:15.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:15.156+0000 7f52cec00640 1 --2- 172.21.15.6:0/3577404058 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f52a406be80 0x7f52a406e340 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:15.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:15.156+0000 7f52cec00640 1 --2- 172.21.15.6:0/3577404058 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f52c8152c00 0x7f52c815a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:15.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:15.156+0000 7f52cec00640 1 --2- 172.21.15.6:0/3577404058 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f52c80cde30 0x7f52c8159c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:15.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:15.156+0000 7f52cec00640 1 --2- 172.21.15.6:0/3577404058 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f52c80c97f0 0x7f52c8156ad0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:15.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:15.156+0000 7f52cec00640 1 -- 172.21.15.6:0/3577404058 >> 172.21.15.6:0/3577404058 conn(0x7f52c80cd290 msgr2=0x7f52c8151c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:18:15.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:15.157+0000 7f52cec00640 1 -- 172.21.15.6:0/3577404058 shutdown_connections 2024-04-24T23:18:15.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:15.157+0000 7f52cec00640 1 -- 172.21.15.6:0/3577404058 wait complete. 2024-04-24T23:18:15.179 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2024-04-24T23:18:15.831 INFO:tasks.workunit.client.0.smithi006.stderr:+ trap - EXIT 2024-04-24T23:18:15.831 INFO:tasks.workunit.client.0.smithi006.stderr:+ cd ffsb 2024-04-24T23:18:15.831 INFO:tasks.workunit.client.0.smithi006.stderr:+ ./configure 2024-04-24T23:18:16.241 INFO:tasks.workunit.client.0.smithi006.stdout:checking for a BSD-compatible install... /usr/bin/install -c 2024-04-24T23:18:17.180 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:18:17.257 INFO:tasks.workunit.client.0.smithi006.stdout:checking whether build environment is sane... yes 2024-04-24T23:18:17.280 INFO:tasks.workunit.client.0.smithi006.stdout:checking for a thread-safe mkdir -p... /usr/bin/mkdir -p 2024-04-24T23:18:17.280 INFO:tasks.workunit.client.0.smithi006.stdout:checking for gawk... gawk 2024-04-24T23:18:17.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:17.384+0000 7f806a000640 1 Processor -- start 2024-04-24T23:18:17.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:17.385+0000 7f806a000640 1 -- start start 2024-04-24T23:18:17.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:17.386+0000 7f806a000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f8064151860 0x7f8064158060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:18:17.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:17.386+0000 7f806a000640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8064158a50 con 0x7f80640c3d90 2024-04-24T23:18:17.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:17.386+0000 7f806a000640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f80640c41e0 con 0x7f80640c9ef0 2024-04-24T23:18:17.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:17.386+0000 7f806a000640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f8064066ea0 con 0x7f8064151860 2024-04-24T23:18:17.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:17.387+0000 7f8069600640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f80640c3d90 0x7f80641594f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:40680/0 (socket says 172.21.15.6:40680) 2024-04-24T23:18:17.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:17.387+0000 7f8068c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f8064151860 0x7f8064158060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:18:17.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:17.387+0000 7f8069600640 1 -- 172.21.15.6:0/415286807 learned_addr learned my addr 172.21.15.6:0/415286807 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:18:17.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:17.387+0000 7f8068c00640 1 -- 172.21.15.6:0/415286807 >> v1:172.21.15.190:6789/0 conn(0x7f80640c9ef0 legacy=0x7f80640c5740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:18:17.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:17.387+0000 7f8068c00640 1 -- 172.21.15.6:0/415286807 >> v1:172.21.15.6:6789/0 conn(0x7f80640c3d90 legacy=0x7f80641594f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:18:17.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:17.387+0000 7f8063400640 1 -- 172.21.15.6:0/415286807 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3272313514 0 0) 0x7f80640c41e0 con 0x7f80640c9ef0 2024-04-24T23:18:17.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:17.387+0000 7f8068c00640 1 -- 172.21.15.6:0/415286807 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80640670f0 con 0x7f8064151860 2024-04-24T23:18:17.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:17.388+0000 7f8068c00640 1 --2- 172.21.15.6:0/415286807 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8064151860 0x7f8064158060 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f80640c43b0 tx=0x7f805402a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b85b4770fff443f5 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:18:17.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:17.389+0000 7f8063400640 1 -- 172.21.15.6:0/415286807 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8054007740 con 0x7f8064151860 2024-04-24T23:18:17.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:17.389+0000 7f8063400640 1 -- 172.21.15.6:0/415286807 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f80540078e0 con 0x7f8064151860 2024-04-24T23:18:17.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:17.389+0000 7f8063400640 1 -- 172.21.15.6:0/415286807 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8054007cb0 con 0x7f8064151860 2024-04-24T23:18:17.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:17.389+0000 7f806a000640 1 -- 172.21.15.6:0/415286807 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8064151860 msgr2=0x7f8064158060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:18:17.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:17.389+0000 7f806a000640 1 --2- 172.21.15.6:0/415286807 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8064151860 0x7f8064158060 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f80640c43b0 tx=0x7f805402a710 comp rx=0 tx=0).stop 2024-04-24T23:18:17.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:17.389+0000 7f806a000640 1 -- 172.21.15.6:0/415286807 shutdown_connections 2024-04-24T23:18:17.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:17.389+0000 7f806a000640 1 --2- 172.21.15.6:0/415286807 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8064151860 0x7f8064158060 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:17.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:17.389+0000 7f806a000640 1 -- 172.21.15.6:0/415286807 >> 172.21.15.6:0/415286807 conn(0x7f80640c9390 msgr2=0x7f80640c97a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:18:17.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:17.389+0000 7f806a000640 1 -- 172.21.15.6:0/415286807 shutdown_connections 2024-04-24T23:18:17.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:17.389+0000 7f806a000640 1 -- 172.21.15.6:0/415286807 wait complete. 2024-04-24T23:18:17.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:17.391+0000 7f806a000640 1 Processor -- start 2024-04-24T23:18:17.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:17.391+0000 7f806a000640 1 -- start start 2024-04-24T23:18:17.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:17.392+0000 7f806a000640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f80640c3d90 0x7f80640c8240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:18:17.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:17.392+0000 7f806a000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f80640c9ef0 0x7f80640c2ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:18:17.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:17.392+0000 7f806a000640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8064151860 0x7f8064152ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:18:17.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:17.392+0000 7f806a000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f8064151c40 con 0x7f80640c9ef0 2024-04-24T23:18:17.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:17.392+0000 7f806a000640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f8064158cf0 con 0x7f8064151860 2024-04-24T23:18:17.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:17.392+0000 7f806a000640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f8064154a50 con 0x7f80640c3d90 2024-04-24T23:18:17.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:17.392+0000 7f8063e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f80640c9ef0 0x7f80640c2ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:18:17.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:17.393+0000 7f8068c00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f80640c3d90 0x7f80640c8240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:18:17.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:17.393+0000 7f8069600640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8064151860 0x7f8064152ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:18:17.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:17.393+0000 7f8068c00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f80640c3d90 0x7f80640c8240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.6:40022/0 (socket says 172.21.15.6:40022) 2024-04-24T23:18:17.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:17.393+0000 7f8068c00640 1 -- 172.21.15.6:0/2828975405 learned_addr learned my addr 172.21.15.6:0/2828975405 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:18:17.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:17.393+0000 7f8069600640 1 -- 172.21.15.6:0/2828975405 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f80640c3d90 msgr2=0x7f80640c8240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:18:17.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:17.393+0000 7f8069600640 1 --2- 172.21.15.6:0/2828975405 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f80640c3d90 0x7f80640c8240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:17.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:17.393+0000 7f8069600640 1 -- 172.21.15.6:0/2828975405 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f80640c9ef0 msgr2=0x7f80640c2ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:18:17.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:17.393+0000 7f8069600640 1 --2- 172.21.15.6:0/2828975405 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f80640c9ef0 0x7f80640c2ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:17.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:17.393+0000 7f8069600640 1 -- 172.21.15.6:0/2828975405 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80641f08e0 con 0x7f8064151860 2024-04-24T23:18:17.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:17.393+0000 7f8068c00640 1 --2- 172.21.15.6:0/2828975405 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f80640c3d90 0x7f80640c8240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:18:17.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:17.393+0000 7f8063e00640 1 --2- 172.21.15.6:0/2828975405 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f80640c9ef0 0x7f80640c2ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:18:17.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:17.393+0000 7f8069600640 1 --2- 172.21.15.6:0/2828975405 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8064151860 0x7f8064152ef0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f805c00c3f0 tx=0x7f805c00c8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:18:17.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:17.394+0000 7f8060c00640 1 -- 172.21.15.6:0/2828975405 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f805c017070 con 0x7f8064151860 2024-04-24T23:18:17.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:17.394+0000 7f8060c00640 1 -- 172.21.15.6:0/2828975405 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f805c005ce0 con 0x7f8064151860 2024-04-24T23:18:17.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:17.394+0000 7f806a000640 1 -- 172.21.15.6:0/2828975405 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80641f0bc0 con 0x7f8064151860 2024-04-24T23:18:17.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:17.394+0000 7f8060c00640 1 -- 172.21.15.6:0/2828975405 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f805c0072d0 con 0x7f8064151860 2024-04-24T23:18:17.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:17.394+0000 7f806a000640 1 -- 172.21.15.6:0/2828975405 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80641f1180 con 0x7f8064151860 2024-04-24T23:18:17.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:17.395+0000 7f806a000640 1 -- 172.21.15.6:0/2828975405 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8064075b30 con 0x7f8064151860 2024-04-24T23:18:17.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:17.396+0000 7f8060c00640 1 -- 172.21.15.6:0/2828975405 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f805c01a070 con 0x7f8064151860 2024-04-24T23:18:17.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:17.397+0000 7f8060c00640 1 --2- 172.21.15.6:0/2828975405 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f804006be30 0x7f804006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:18:17.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:17.397+0000 7f8060c00640 1 -- 172.21.15.6:0/2828975405 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6590+0+0 (secure 0 0 0) 0x7f805c08a510 con 0x7f8064151860 2024-04-24T23:18:17.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:17.397+0000 7f8068c00640 1 --2- 172.21.15.6:0/2828975405 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f804006be30 0x7f804006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:18:17.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:17.398+0000 7f8068c00640 1 --2- 172.21.15.6:0/2828975405 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f804006be30 0x7f804006e2f0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f8054009bd0 tx=0x7f805402f000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:18:17.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:17.401+0000 7f8060c00640 1 -- 172.21.15.6:0/2828975405 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f805c0583d0 con 0x7f8064151860 2024-04-24T23:18:17.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:17.647+0000 7f806a000640 1 -- 172.21.15.6:0/2828975405 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8064074f90 con 0x7f8064151860 2024-04-24T23:18:17.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:17.648+0000 7f8060c00640 1 -- 172.21.15.6:0/2828975405 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) ==== 94+0+6902 (secure 0 0 0) 0x7f805c05c6d0 con 0x7f8064151860 2024-04-24T23:18:17.651 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:18:17.651 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":18,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4197,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/105267896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":105267896},{"type":"v1","addr":"172.21.15.6:6834","nonce":105267896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:16:32.310838+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4282,"mds_2":4235,"mds_3":4232,"mds_4":4203},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4203":{"gid":4203,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":17,"addr":"172.21.15.6:6837/2590421094","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":2590421094},{"type":"v1","addr":"172.21.15.6:6837","nonce":2590421094}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4232":{"gid":4232,"name":"b","rank":3,"incarnation":15,"state":"up:active","state_seq":15,"addr":"172.21.15.190:6838/2032280158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2032280158},{"type":"v1","addr":"172.21.15.190:6838","nonce":2032280158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4235":{"gid":4235,"name":"d","rank":2,"incarnation":13,"state":"up:active","state_seq":14,"addr":"172.21.15.190:6839/1572872451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":1572872451},{"type":"v1","addr":"172.21.15.190:6839","nonce":1572872451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4282":{"gid":4282,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":12,"addr":"172.21.15.190:6837/580315269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":580315269},{"type":"v1","addr":"172.21.15.190:6837","nonce":580315269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/489092393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":489092393},{"type":"v1","addr":"172.21.15.6:6835","nonce":489092393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4306,"qdb_cluster":[4203,4232,4235,4282,4306]},"id":1}]} 2024-04-24T23:18:17.651 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 18 2024-04-24T23:18:17.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:17.652+0000 7f806a000640 1 -- 172.21.15.6:0/2828975405 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f804006be30 msgr2=0x7f804006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:18:17.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:17.652+0000 7f806a000640 1 --2- 172.21.15.6:0/2828975405 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f804006be30 0x7f804006e2f0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f8054009bd0 tx=0x7f805402f000 comp rx=0 tx=0).stop 2024-04-24T23:18:17.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:17.652+0000 7f806a000640 1 -- 172.21.15.6:0/2828975405 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8064151860 msgr2=0x7f8064152ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:18:17.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:17.652+0000 7f806a000640 1 --2- 172.21.15.6:0/2828975405 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8064151860 0x7f8064152ef0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f805c00c3f0 tx=0x7f805c00c8b0 comp rx=0 tx=0).stop 2024-04-24T23:18:17.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:17.652+0000 7f806a000640 1 -- 172.21.15.6:0/2828975405 shutdown_connections 2024-04-24T23:18:17.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:17.652+0000 7f806a000640 1 --2- 172.21.15.6:0/2828975405 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f804006be30 0x7f804006e2f0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:17.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:17.652+0000 7f806a000640 1 --2- 172.21.15.6:0/2828975405 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8064151860 0x7f8064152ef0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:17.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:17.652+0000 7f806a000640 1 --2- 172.21.15.6:0/2828975405 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f80640c9ef0 0x7f80640c2ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:17.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:17.652+0000 7f806a000640 1 --2- 172.21.15.6:0/2828975405 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f80640c3d90 0x7f80640c8240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:17.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:17.652+0000 7f806a000640 1 -- 172.21.15.6:0/2828975405 >> 172.21.15.6:0/2828975405 conn(0x7f80640c9390 msgr2=0x7f8064071cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:18:17.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:17.652+0000 7f806a000640 1 -- 172.21.15.6:0/2828975405 shutdown_connections 2024-04-24T23:18:17.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:17.652+0000 7f806a000640 1 -- 172.21.15.6:0/2828975405 wait complete. 2024-04-24T23:18:17.673 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2024-04-24T23:18:17.847 INFO:tasks.workunit.client.0.smithi006.stdout:checking whether make sets $(MAKE)... yes 2024-04-24T23:18:17.953 INFO:tasks.workunit.client.0.smithi006.stdout:checking build system type... x86_64-unknown-linux-gnu 2024-04-24T23:18:17.954 INFO:tasks.workunit.client.0.smithi006.stdout:checking host system type... x86_64-unknown-linux-gnu 2024-04-24T23:18:17.974 INFO:tasks.workunit.client.0.smithi006.stdout:checking for style of include used by make... GNU 2024-04-24T23:18:17.979 INFO:tasks.workunit.client.0.smithi006.stdout:checking for gcc... gcc 2024-04-24T23:18:18.109 INFO:tasks.workunit.client.0.smithi006.stdout:checking for C compiler default output file name... a.out 2024-04-24T23:18:18.116 INFO:tasks.workunit.client.0.smithi006.stdout:checking whether the C compiler works... yes 2024-04-24T23:18:18.121 INFO:tasks.workunit.client.0.smithi006.stdout:checking whether we are cross compiling... no 2024-04-24T23:18:18.205 INFO:tasks.workunit.client.0.smithi006.stdout:checking for suffix of executables... 2024-04-24T23:18:18.289 INFO:tasks.workunit.client.0.smithi006.stdout:checking for suffix of object files... o 2024-04-24T23:18:18.374 INFO:tasks.workunit.client.0.smithi006.stdout:checking whether we are using the GNU C compiler... yes 2024-04-24T23:18:18.465 INFO:tasks.workunit.client.0.smithi006.stdout:checking whether gcc accepts -g... yes 2024-04-24T23:18:18.600 INFO:tasks.workunit.client.0.smithi006.stdout:checking for gcc option to accept ISO C89... none needed 2024-04-24T23:18:18.840 INFO:tasks.workunit.client.0.smithi006.stdout:checking dependency style of gcc... gcc3 2024-04-24T23:18:19.002 INFO:tasks.workunit.client.0.smithi006.stdout:checking how to run the C preprocessor... gcc -E 2024-04-24T23:18:19.118 INFO:tasks.workunit.client.0.smithi006.stdout:checking for grep that handles long lines and -e... /usr/bin/grep 2024-04-24T23:18:19.122 INFO:tasks.workunit.client.0.smithi006.stdout:checking for egrep... /usr/bin/grep -E 2024-04-24T23:18:19.151 INFO:tasks.workunit.client.0.smithi006.stdout:checking for AIX... no 2024-04-24T23:18:19.180 INFO:tasks.workunit.client.0.smithi006.stdout:checking for gcc... (cached) gcc 2024-04-24T23:18:19.210 INFO:tasks.workunit.client.0.smithi006.stdout:checking whether we are using the GNU C compiler... (cached) yes 2024-04-24T23:18:19.213 INFO:tasks.workunit.client.0.smithi006.stdout:checking whether gcc accepts -g... (cached) yes 2024-04-24T23:18:19.213 INFO:tasks.workunit.client.0.smithi006.stdout:checking for gcc option to accept ISO C89... (cached) none needed 2024-04-24T23:18:19.213 INFO:tasks.workunit.client.0.smithi006.stdout:checking dependency style of gcc... (cached) gcc3 2024-04-24T23:18:19.338 INFO:tasks.workunit.client.0.smithi006.stdout:checking for main in -lm... yes 2024-04-24T23:18:19.463 INFO:tasks.workunit.client.0.smithi006.stdout:checking for main in -lpthread... yes 2024-04-24T23:18:19.674 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:18:19.810 INFO:tasks.workunit.client.0.smithi006.stdout:checking for ANSI C header files... yes 2024-04-24T23:18:19.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:19.866+0000 7f9b52800640 1 Processor -- start 2024-04-24T23:18:19.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:19.866+0000 7f9b52800640 1 -- start start 2024-04-24T23:18:19.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:19.867+0000 7f9b52800640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f9b4c154b70 0x7f9b4c06f020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:18:19.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:19.867+0000 7f9b52800640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9b4c0c7080 0x7f9b4c071000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:18:19.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:19.867+0000 7f9b52800640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9b4c0c8570 con 0x7f9b4c0c80e0 2024-04-24T23:18:19.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:19.867+0000 7f9b52800640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f9b4c072a70 con 0x7f9b4c154b70 2024-04-24T23:18:19.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:19.867+0000 7f9b52800640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f9b4c072530 con 0x7f9b4c0c7080 2024-04-24T23:18:19.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:19.868+0000 7f9b4aa00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9b4c0c7080 0x7f9b4c071000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:18:19.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:19.868+0000 7f9b4b400640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f9b4c154b70 0x7f9b4c06f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:18:19.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:19.868+0000 7f9b4aa00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9b4c0c7080 0x7f9b4c071000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.6:40036/0 (socket says 172.21.15.6:40036) 2024-04-24T23:18:19.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:19.868+0000 7f9b4aa00640 1 -- 172.21.15.6:0/2571321501 learned_addr learned my addr 172.21.15.6:0/2571321501 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:18:19.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:19.868+0000 7f9b4be00640 1 --1- 172.21.15.6:0/2571321501 >> v1:172.21.15.190:6789/0 conn(0x7f9b4c0c80e0 0x7f9b4c15a9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.6:40796/0 (socket says 172.21.15.6:40796) 2024-04-24T23:18:19.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:19.868+0000 7f9b4aa00640 1 -- 172.21.15.6:0/2571321501 >> v2:172.21.15.190:3300/0 conn(0x7f9b4c154b70 msgr2=0x7f9b4c06f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:18:19.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:19.868+0000 7f9b4aa00640 1 --2- 172.21.15.6:0/2571321501 >> v2:172.21.15.190:3300/0 conn(0x7f9b4c154b70 0x7f9b4c06f020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:19.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:19.868+0000 7f9b4aa00640 1 -- 172.21.15.6:0/2571321501 >> v1:172.21.15.190:6789/0 conn(0x7f9b4c0c80e0 legacy=0x7f9b4c15a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:18:19.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:19.868+0000 7f9b4aa00640 1 -- 172.21.15.6:0/2571321501 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b4c0727d0 con 0x7f9b4c0c7080 2024-04-24T23:18:19.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:19.869+0000 7f9b4aa00640 1 --2- 172.21.15.6:0/2571321501 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9b4c0c7080 0x7f9b4c071000 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f9b38009870 tx=0x7f9b3802a250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9b8d313cb704ddbc server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:18:19.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:19.869+0000 7f9b4a000640 1 -- 172.21.15.6:0/2571321501 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b38037070 con 0x7f9b4c0c7080 2024-04-24T23:18:19.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:19.869+0000 7f9b4a000640 1 -- 172.21.15.6:0/2571321501 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9b38032440 con 0x7f9b4c0c7080 2024-04-24T23:18:19.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:19.869+0000 7f9b4a000640 1 -- 172.21.15.6:0/2571321501 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b38032810 con 0x7f9b4c0c7080 2024-04-24T23:18:19.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:19.870+0000 7f9b52800640 1 -- 172.21.15.6:0/2571321501 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9b4c0c7080 msgr2=0x7f9b4c071000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:18:19.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:19.870+0000 7f9b52800640 1 --2- 172.21.15.6:0/2571321501 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9b4c0c7080 0x7f9b4c071000 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f9b38009870 tx=0x7f9b3802a250 comp rx=0 tx=0).stop 2024-04-24T23:18:19.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:19.870+0000 7f9b52800640 1 -- 172.21.15.6:0/2571321501 shutdown_connections 2024-04-24T23:18:19.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:19.870+0000 7f9b52800640 1 --2- 172.21.15.6:0/2571321501 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9b4c0c7080 0x7f9b4c071000 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:19.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:19.870+0000 7f9b52800640 1 --2- 172.21.15.6:0/2571321501 >> v2:172.21.15.190:3300/0 conn(0x7f9b4c154b70 0x7f9b4c06f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:19.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:19.870+0000 7f9b52800640 1 -- 172.21.15.6:0/2571321501 >> 172.21.15.6:0/2571321501 conn(0x7f9b4c06b0e0 msgr2=0x7f9b4c06b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:18:19.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:19.870+0000 7f9b52800640 1 -- 172.21.15.6:0/2571321501 shutdown_connections 2024-04-24T23:18:19.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:19.870+0000 7f9b52800640 1 -- 172.21.15.6:0/2571321501 wait complete. 2024-04-24T23:18:19.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:19.872+0000 7f9b52800640 1 Processor -- start 2024-04-24T23:18:19.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:19.872+0000 7f9b52800640 1 -- start start 2024-04-24T23:18:19.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:19.873+0000 7f9b52800640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9b4c0c7080 0x7f9b4c1f0870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:18:19.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:19.873+0000 7f9b52800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9b4c0c80e0 0x7f9b4c1f47c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:18:19.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:19.873+0000 7f9b52800640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9b4c154b70 0x7f9b4c151720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:18:19.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:19.873+0000 7f9b52800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f9b4c06eb00 con 0x7f9b4c0c80e0 2024-04-24T23:18:19.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:19.873+0000 7f9b52800640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f9b4c06e820 con 0x7f9b4c0c7080 2024-04-24T23:18:19.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:19.873+0000 7f9b52800640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f9b4c154f50 con 0x7f9b4c154b70 2024-04-24T23:18:19.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:19.874+0000 7f9b4aa00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9b4c0c80e0 0x7f9b4c1f47c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:18:19.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:19.874+0000 7f9b4aa00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9b4c0c80e0 0x7f9b4c1f47c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57924/0 (socket says 172.21.15.6:57924) 2024-04-24T23:18:19.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:19.874+0000 7f9b4aa00640 1 -- 172.21.15.6:0/1495699867 learned_addr learned my addr 172.21.15.6:0/1495699867 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:18:19.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:19.874+0000 7f9b4be00640 1 --2- 172.21.15.6:0/1495699867 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9b4c154b70 0x7f9b4c151720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:18:19.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:19.874+0000 7f9b4b400640 1 --2- 172.21.15.6:0/1495699867 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9b4c0c7080 0x7f9b4c1f0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:18:19.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:19.874+0000 7f9b4aa00640 1 -- 172.21.15.6:0/1495699867 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9b4c154b70 msgr2=0x7f9b4c151720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:18:19.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:19.874+0000 7f9b4aa00640 1 --2- 172.21.15.6:0/1495699867 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9b4c154b70 0x7f9b4c151720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:19.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:19.874+0000 7f9b4aa00640 1 -- 172.21.15.6:0/1495699867 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9b4c0c7080 msgr2=0x7f9b4c1f0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:18:19.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:19.874+0000 7f9b4aa00640 1 --2- 172.21.15.6:0/1495699867 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9b4c0c7080 0x7f9b4c1f0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:19.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:19.874+0000 7f9b4aa00640 1 -- 172.21.15.6:0/1495699867 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b4c0c5350 con 0x7f9b4c0c80e0 2024-04-24T23:18:19.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:19.874+0000 7f9b4be00640 1 --2- 172.21.15.6:0/1495699867 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9b4c154b70 0x7f9b4c151720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:18:19.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:19.874+0000 7f9b4b400640 1 --2- 172.21.15.6:0/1495699867 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9b4c0c7080 0x7f9b4c1f0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:18:19.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:19.875+0000 7f9b4aa00640 1 --2- 172.21.15.6:0/1495699867 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9b4c0c80e0 0x7f9b4c1f47c0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f9b3802a290 tx=0x7f9b380046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:18:19.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:19.875+0000 7f9b43400640 1 -- 172.21.15.6:0/1495699867 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b38037040 con 0x7f9b4c0c80e0 2024-04-24T23:18:19.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:19.875+0000 7f9b43400640 1 -- 172.21.15.6:0/1495699867 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9b38042070 con 0x7f9b4c0c80e0 2024-04-24T23:18:19.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:19.875+0000 7f9b43400640 1 -- 172.21.15.6:0/1495699867 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b38030880 con 0x7f9b4c0c80e0 2024-04-24T23:18:19.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:19.875+0000 7f9b52800640 1 -- 172.21.15.6:0/1495699867 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b4c0c5080 con 0x7f9b4c0c80e0 2024-04-24T23:18:19.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:19.875+0000 7f9b52800640 1 -- 172.21.15.6:0/1495699867 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b4c0c58f0 con 0x7f9b4c0c80e0 2024-04-24T23:18:19.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:19.876+0000 7f9b52800640 1 -- 172.21.15.6:0/1495699867 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9b4c068e00 con 0x7f9b4c0c80e0 2024-04-24T23:18:19.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:19.878+0000 7f9b43400640 1 -- 172.21.15.6:0/1495699867 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f9b38030a20 con 0x7f9b4c0c80e0 2024-04-24T23:18:19.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:19.879+0000 7f9b43400640 1 --2- 172.21.15.6:0/1495699867 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f9b2c06be60 0x7f9b2c06e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:18:19.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:19.879+0000 7f9b43400640 1 -- 172.21.15.6:0/1495699867 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6590+0+0 (secure 0 0 0) 0x7f9b380ab940 con 0x7f9b4c0c80e0 2024-04-24T23:18:19.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:19.879+0000 7f9b4b400640 1 --2- 172.21.15.6:0/1495699867 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f9b2c06be60 0x7f9b2c06e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:18:19.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:19.880+0000 7f9b4b400640 1 --2- 172.21.15.6:0/1495699867 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f9b2c06be60 0x7f9b2c06e320 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f9b3c009e90 tx=0x7f9b3c009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:18:19.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:19.885+0000 7f9b43400640 1 -- 172.21.15.6:0/1495699867 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f9b4c068e00 con 0x7f9b4c0c80e0 2024-04-24T23:18:19.908 INFO:tasks.workunit.client.0.smithi006.stdout:checking for sys/wait.h that is POSIX.1 compatible... yes 2024-04-24T23:18:20.011 INFO:tasks.workunit.client.0.smithi006.stdout:checking for sys/types.h... yes 2024-04-24T23:18:20.111 INFO:tasks.workunit.client.0.smithi006.stdout:checking for sys/stat.h... yes 2024-04-24T23:18:20.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:20.148+0000 7f9b52800640 1 -- 172.21.15.6:0/1495699867 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9b4c0767b0 con 0x7f9b4c0c80e0 2024-04-24T23:18:20.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:20.149+0000 7f9b43400640 1 -- 172.21.15.6:0/1495699867 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) ==== 94+0+6902 (secure 0 0 0) 0x7f9b380798a0 con 0x7f9b4c0c80e0 2024-04-24T23:18:20.151 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:18:20.151 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":18,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4197,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/105267896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":105267896},{"type":"v1","addr":"172.21.15.6:6834","nonce":105267896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:16:32.310838+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4282,"mds_2":4235,"mds_3":4232,"mds_4":4203},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4203":{"gid":4203,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":17,"addr":"172.21.15.6:6837/2590421094","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":2590421094},{"type":"v1","addr":"172.21.15.6:6837","nonce":2590421094}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4232":{"gid":4232,"name":"b","rank":3,"incarnation":15,"state":"up:active","state_seq":15,"addr":"172.21.15.190:6838/2032280158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2032280158},{"type":"v1","addr":"172.21.15.190:6838","nonce":2032280158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4235":{"gid":4235,"name":"d","rank":2,"incarnation":13,"state":"up:active","state_seq":14,"addr":"172.21.15.190:6839/1572872451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":1572872451},{"type":"v1","addr":"172.21.15.190:6839","nonce":1572872451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4282":{"gid":4282,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":12,"addr":"172.21.15.190:6837/580315269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":580315269},{"type":"v1","addr":"172.21.15.190:6837","nonce":580315269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/489092393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":489092393},{"type":"v1","addr":"172.21.15.6:6835","nonce":489092393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4306,"qdb_cluster":[4203,4232,4235,4282,4306]},"id":1}]} 2024-04-24T23:18:20.152 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 18 2024-04-24T23:18:20.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:20.151+0000 7f9b52800640 1 -- 172.21.15.6:0/1495699867 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f9b2c06be60 msgr2=0x7f9b2c06e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:18:20.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:20.151+0000 7f9b52800640 1 --2- 172.21.15.6:0/1495699867 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f9b2c06be60 0x7f9b2c06e320 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f9b3c009e90 tx=0x7f9b3c009190 comp rx=0 tx=0).stop 2024-04-24T23:18:20.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:20.152+0000 7f9b52800640 1 -- 172.21.15.6:0/1495699867 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9b4c0c80e0 msgr2=0x7f9b4c1f47c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:18:20.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:20.152+0000 7f9b52800640 1 --2- 172.21.15.6:0/1495699867 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9b4c0c80e0 0x7f9b4c1f47c0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f9b3802a290 tx=0x7f9b380046c0 comp rx=0 tx=0).stop 2024-04-24T23:18:20.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:20.152+0000 7f9b52800640 1 -- 172.21.15.6:0/1495699867 shutdown_connections 2024-04-24T23:18:20.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:20.152+0000 7f9b52800640 1 --2- 172.21.15.6:0/1495699867 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f9b2c06be60 0x7f9b2c06e320 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:20.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:20.152+0000 7f9b52800640 1 --2- 172.21.15.6:0/1495699867 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9b4c154b70 0x7f9b4c151720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:20.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:20.152+0000 7f9b52800640 1 --2- 172.21.15.6:0/1495699867 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9b4c0c80e0 0x7f9b4c1f47c0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:20.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:20.152+0000 7f9b52800640 1 --2- 172.21.15.6:0/1495699867 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9b4c0c7080 0x7f9b4c1f0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:20.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:20.152+0000 7f9b52800640 1 -- 172.21.15.6:0/1495699867 >> 172.21.15.6:0/1495699867 conn(0x7f9b4c06b0e0 msgr2=0x7f9b4c153060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:18:20.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:20.152+0000 7f9b52800640 1 -- 172.21.15.6:0/1495699867 shutdown_connections 2024-04-24T23:18:20.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:20.152+0000 7f9b52800640 1 -- 172.21.15.6:0/1495699867 wait complete. 2024-04-24T23:18:20.171 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2024-04-24T23:18:20.208 INFO:tasks.workunit.client.0.smithi006.stdout:checking for stdlib.h... yes 2024-04-24T23:18:20.590 INFO:tasks.workunit.client.0.smithi006.stdout:checking for string.h... yes 2024-04-24T23:18:20.990 INFO:tasks.workunit.client.0.smithi006.stdout:checking for memory.h... yes 2024-04-24T23:18:21.117 INFO:tasks.workunit.client.0.smithi006.stdout:checking for strings.h... yes 2024-04-24T23:18:21.243 INFO:tasks.workunit.client.0.smithi006.stdout:checking for inttypes.h... yes 2024-04-24T23:18:21.374 INFO:tasks.workunit.client.0.smithi006.stdout:checking for stdint.h... yes 2024-04-24T23:18:21.492 INFO:tasks.workunit.client.0.smithi006.stdout:checking for unistd.h... yes 2024-04-24T23:18:21.624 INFO:tasks.workunit.client.0.smithi006.stdout:checking pthread.h usability... yes 2024-04-24T23:18:21.700 INFO:tasks.workunit.client.0.smithi006.stdout:checking pthread.h presence... yes 2024-04-24T23:18:21.702 INFO:tasks.workunit.client.0.smithi006.stdout:checking for pthread.h... yes 2024-04-24T23:18:21.837 INFO:tasks.workunit.client.0.smithi006.stdout:checking fcntl.h usability... yes 2024-04-24T23:18:21.903 INFO:tasks.workunit.client.0.smithi006.stdout:checking fcntl.h presence... yes 2024-04-24T23:18:21.905 INFO:tasks.workunit.client.0.smithi006.stdout:checking for fcntl.h... yes 2024-04-24T23:18:22.037 INFO:tasks.workunit.client.0.smithi006.stdout:checking limits.h usability... yes 2024-04-24T23:18:22.172 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:18:22.383 INFO:tasks.workunit.client.0.smithi006.stdout:checking limits.h presence... yes 2024-04-24T23:18:22.385 INFO:tasks.workunit.client.0.smithi006.stdout:checking for limits.h... yes 2024-04-24T23:18:22.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:22.387+0000 7f6163e00640 1 Processor -- start 2024-04-24T23:18:22.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:22.387+0000 7f6163e00640 1 -- start start 2024-04-24T23:18:22.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:22.389+0000 7f6163e00640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f6164154c40 0x7f61640c6710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:18:22.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:22.389+0000 7f6163e00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f616415ef20 con 0x7f616415e020 2024-04-24T23:18:22.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:22.389+0000 7f6163e00640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f616415ec40 con 0x7f61640c53d0 2024-04-24T23:18:22.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:22.389+0000 7f6163e00640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f61640c5950 con 0x7f6164154c40 2024-04-24T23:18:22.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:22.390+0000 7f6162a00640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f6164154c40 0x7f61640c6710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:18:22.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:22.390+0000 7f615a000640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f61640c53d0 0x7f61640cb0b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.6:40804/0 (socket says 172.21.15.6:40804) 2024-04-24T23:18:22.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:22.390+0000 7f615a000640 1 -- 172.21.15.6:0/1571191068 learned_addr learned my addr 172.21.15.6:0/1571191068 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:18:22.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:22.390+0000 7f6162a00640 1 -- 172.21.15.6:0/1571191068 >> v1:172.21.15.190:6789/0 conn(0x7f61640c53d0 legacy=0x7f61640cb0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:18:22.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:22.390+0000 7f6162a00640 1 -- 172.21.15.6:0/1571191068 >> v1:172.21.15.6:6789/0 conn(0x7f616415e020 legacy=0x7f616415e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:18:22.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:22.390+0000 7f6162000640 1 -- 172.21.15.6:0/1571191068 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1045042513 0 0) 0x7f616415ec40 con 0x7f61640c53d0 2024-04-24T23:18:22.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:22.390+0000 7f6162a00640 1 -- 172.21.15.6:0/1571191068 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61640c5c30 con 0x7f6164154c40 2024-04-24T23:18:22.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:22.391+0000 7f6162a00640 1 --2- 172.21.15.6:0/1571191068 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6164154c40 0x7f61640c6710 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f6154009d90 tx=0x7f615402a680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4def191ce0f86fec server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:18:22.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:22.392+0000 7f6162000640 1 -- 172.21.15.6:0/1571191068 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6154007740 con 0x7f6164154c40 2024-04-24T23:18:22.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:22.392+0000 7f6162000640 1 -- 172.21.15.6:0/1571191068 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f61540078e0 con 0x7f6164154c40 2024-04-24T23:18:22.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:22.392+0000 7f6162000640 1 -- 172.21.15.6:0/1571191068 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6154007cb0 con 0x7f6164154c40 2024-04-24T23:18:22.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:22.393+0000 7f6163e00640 1 -- 172.21.15.6:0/1571191068 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6164154c40 msgr2=0x7f61640c6710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:18:22.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:22.393+0000 7f6163e00640 1 --2- 172.21.15.6:0/1571191068 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6164154c40 0x7f61640c6710 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f6154009d90 tx=0x7f615402a680 comp rx=0 tx=0).stop 2024-04-24T23:18:22.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:22.393+0000 7f6163e00640 1 -- 172.21.15.6:0/1571191068 shutdown_connections 2024-04-24T23:18:22.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:22.393+0000 7f6163e00640 1 --2- 172.21.15.6:0/1571191068 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6164154c40 0x7f61640c6710 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:22.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:22.393+0000 7f6163e00640 1 -- 172.21.15.6:0/1571191068 >> 172.21.15.6:0/1571191068 conn(0x7f616406cd70 msgr2=0x7f616406d870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:18:22.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:22.393+0000 7f6163e00640 1 -- 172.21.15.6:0/1571191068 shutdown_connections 2024-04-24T23:18:22.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:22.393+0000 7f6163e00640 1 -- 172.21.15.6:0/1571191068 wait complete. 2024-04-24T23:18:22.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:22.395+0000 7f6163e00640 1 Processor -- start 2024-04-24T23:18:22.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:22.395+0000 7f6163e00640 1 -- start start 2024-04-24T23:18:22.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:22.397+0000 7f6163e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f61640c53d0 0x7f61641e69f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:18:22.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:22.397+0000 7f6163e00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6164154c40 0x7f6164156ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:18:22.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:22.397+0000 7f6163e00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f616415e020 0x7f6164157430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:18:22.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:22.397+0000 7f6163e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f61640cb7c0 con 0x7f61640c53d0 2024-04-24T23:18:22.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:22.397+0000 7f6163e00640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f61640cc470 con 0x7f616415e020 2024-04-24T23:18:22.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:22.397+0000 7f6163e00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f61640cb940 con 0x7f6164154c40 2024-04-24T23:18:22.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:22.397+0000 7f6162a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f61640c53d0 0x7f61641e69f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:18:22.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:22.397+0000 7f6162a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f61640c53d0 0x7f61641e69f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57936/0 (socket says 172.21.15.6:57936) 2024-04-24T23:18:22.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:22.397+0000 7f6162a00640 1 -- 172.21.15.6:0/3409396263 learned_addr learned my addr 172.21.15.6:0/3409396263 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:18:22.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:22.397+0000 7f6163400640 1 --2- 172.21.15.6:0/3409396263 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f616415e020 0x7f6164157430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:18:22.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:22.397+0000 7f615a000640 1 --2- 172.21.15.6:0/3409396263 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6164154c40 0x7f6164156ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:18:22.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:22.398+0000 7f6163400640 1 -- 172.21.15.6:0/3409396263 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6164154c40 msgr2=0x7f6164156ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:18:22.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:22.398+0000 7f6163400640 1 --2- 172.21.15.6:0/3409396263 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6164154c40 0x7f6164156ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:22.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:22.398+0000 7f6163400640 1 -- 172.21.15.6:0/3409396263 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f61640c53d0 msgr2=0x7f61641e69f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:18:22.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:22.398+0000 7f6163400640 1 --2- 172.21.15.6:0/3409396263 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f61640c53d0 0x7f61641e69f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:22.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:22.398+0000 7f6163400640 1 -- 172.21.15.6:0/3409396263 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61641f1150 con 0x7f616415e020 2024-04-24T23:18:22.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:22.398+0000 7f6163400640 1 --2- 172.21.15.6:0/3409396263 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f616415e020 0x7f6164157430 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f615c00c480 tx=0x7f615c00c940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:18:22.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:22.399+0000 7f615b400640 1 -- 172.21.15.6:0/3409396263 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f615c017070 con 0x7f616415e020 2024-04-24T23:18:22.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:22.399+0000 7f615b400640 1 -- 172.21.15.6:0/3409396263 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f615c005480 con 0x7f616415e020 2024-04-24T23:18:22.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:22.399+0000 7f615b400640 1 -- 172.21.15.6:0/3409396263 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f615c003830 con 0x7f616415e020 2024-04-24T23:18:22.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:22.399+0000 7f6163e00640 1 -- 172.21.15.6:0/3409396263 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61641f1420 con 0x7f616415e020 2024-04-24T23:18:22.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:22.399+0000 7f6163e00640 1 -- 172.21.15.6:0/3409396263 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6164156070 con 0x7f616415e020 2024-04-24T23:18:22.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:22.401+0000 7f6163e00640 1 -- 172.21.15.6:0/3409396263 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f616415ec40 con 0x7f616415e020 2024-04-24T23:18:22.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:22.401+0000 7f615b400640 1 -- 172.21.15.6:0/3409396263 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f615c01a070 con 0x7f616415e020 2024-04-24T23:18:22.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:22.402+0000 7f615b400640 1 --2- 172.21.15.6:0/3409396263 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f614006be30 0x7f614006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:18:22.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:22.402+0000 7f615b400640 1 -- 172.21.15.6:0/3409396263 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6590+0+0 (secure 0 0 0) 0x7f615c08a540 con 0x7f616415e020 2024-04-24T23:18:22.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:22.403+0000 7f6162a00640 1 --2- 172.21.15.6:0/3409396263 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f614006be30 0x7f614006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:18:22.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:22.403+0000 7f6162a00640 1 --2- 172.21.15.6:0/3409396263 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f614006be30 0x7f614006e2f0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f6154009bd0 tx=0x7f615402f000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:18:22.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:22.407+0000 7f615b400640 1 -- 172.21.15.6:0/3409396263 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f616415ec40 con 0x7f616415e020 2024-04-24T23:18:22.414 INFO:tasks.workunit.client.0.smithi006.stdout:checking for stdint.h... (cached) yes 2024-04-24T23:18:22.506 INFO:tasks.workunit.client.0.smithi006.stdout:checking sys/time.h usability... yes 2024-04-24T23:18:22.550 INFO:tasks.workunit.client.0.smithi006.stdout:checking sys/time.h presence... yes 2024-04-24T23:18:22.551 INFO:tasks.workunit.client.0.smithi006.stdout:checking for sys/time.h... yes 2024-04-24T23:18:22.586 INFO:tasks.workunit.client.0.smithi006.stdout:checking for unistd.h... (cached) yes 2024-04-24T23:18:22.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:22.666+0000 7f6163e00640 1 -- 172.21.15.6:0/3409396263 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f616406d360 con 0x7f616415e020 2024-04-24T23:18:22.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:22.667+0000 7f615b400640 1 -- 172.21.15.6:0/3409396263 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) ==== 94+0+6902 (secure 0 0 0) 0x7f616406d360 con 0x7f616415e020 2024-04-24T23:18:22.670 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:18:22.670 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":18,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4197,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/105267896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":105267896},{"type":"v1","addr":"172.21.15.6:6834","nonce":105267896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:16:32.310838+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4282,"mds_2":4235,"mds_3":4232,"mds_4":4203},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4203":{"gid":4203,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":17,"addr":"172.21.15.6:6837/2590421094","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":2590421094},{"type":"v1","addr":"172.21.15.6:6837","nonce":2590421094}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4232":{"gid":4232,"name":"b","rank":3,"incarnation":15,"state":"up:active","state_seq":15,"addr":"172.21.15.190:6838/2032280158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2032280158},{"type":"v1","addr":"172.21.15.190:6838","nonce":2032280158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4235":{"gid":4235,"name":"d","rank":2,"incarnation":13,"state":"up:active","state_seq":14,"addr":"172.21.15.190:6839/1572872451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":1572872451},{"type":"v1","addr":"172.21.15.190:6839","nonce":1572872451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4282":{"gid":4282,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":12,"addr":"172.21.15.190:6837/580315269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":580315269},{"type":"v1","addr":"172.21.15.190:6837","nonce":580315269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/489092393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":489092393},{"type":"v1","addr":"172.21.15.6:6835","nonce":489092393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4306,"qdb_cluster":[4203,4232,4235,4282,4306]},"id":1}]} 2024-04-24T23:18:22.670 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 18 2024-04-24T23:18:22.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:22.670+0000 7f6163e00640 1 -- 172.21.15.6:0/3409396263 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f614006be30 msgr2=0x7f614006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:18:22.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:22.670+0000 7f6163e00640 1 --2- 172.21.15.6:0/3409396263 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f614006be30 0x7f614006e2f0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f6154009bd0 tx=0x7f615402f000 comp rx=0 tx=0).stop 2024-04-24T23:18:22.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:22.670+0000 7f6163e00640 1 -- 172.21.15.6:0/3409396263 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f616415e020 msgr2=0x7f6164157430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:18:22.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:22.670+0000 7f6163e00640 1 --2- 172.21.15.6:0/3409396263 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f616415e020 0x7f6164157430 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f615c00c480 tx=0x7f615c00c940 comp rx=0 tx=0).stop 2024-04-24T23:18:22.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:22.670+0000 7f6163e00640 1 -- 172.21.15.6:0/3409396263 shutdown_connections 2024-04-24T23:18:22.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:22.670+0000 7f6163e00640 1 --2- 172.21.15.6:0/3409396263 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f614006be30 0x7f614006e2f0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:22.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:22.670+0000 7f6163e00640 1 --2- 172.21.15.6:0/3409396263 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6164154c40 0x7f6164156ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:22.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:22.670+0000 7f6163e00640 1 --2- 172.21.15.6:0/3409396263 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f616415e020 0x7f6164157430 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:22.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:22.670+0000 7f6163e00640 1 --2- 172.21.15.6:0/3409396263 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f61640c53d0 0x7f61641e69f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:22.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:22.670+0000 7f6163e00640 1 -- 172.21.15.6:0/3409396263 >> 172.21.15.6:0/3409396263 conn(0x7f616406cd70 msgr2=0x7f61640cd650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:18:22.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:22.670+0000 7f6163e00640 1 -- 172.21.15.6:0/3409396263 shutdown_connections 2024-04-24T23:18:22.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:22.671+0000 7f6163e00640 1 -- 172.21.15.6:0/3409396263 wait complete. 2024-04-24T23:18:22.676 INFO:tasks.workunit.client.0.smithi006.stdout:checking sys/vfs.h usability... yes 2024-04-24T23:18:22.693 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2024-04-24T23:18:22.727 INFO:tasks.workunit.client.0.smithi006.stdout:checking sys/vfs.h presence... yes 2024-04-24T23:18:22.730 INFO:tasks.workunit.client.0.smithi006.stdout:checking for sys/vfs.h... yes 2024-04-24T23:18:22.848 INFO:tasks.workunit.client.0.smithi006.stdout:checking sys/limits.h usability... no 2024-04-24T23:18:22.927 INFO:tasks.workunit.client.0.smithi006.stdout:checking sys/limits.h presence... no 2024-04-24T23:18:22.930 INFO:tasks.workunit.client.0.smithi006.stdout:checking for sys/limits.h... no 2024-04-24T23:18:23.533 INFO:tasks.workunit.client.0.smithi006.stdout:checking for an ANSI C-conforming const... yes 2024-04-24T23:18:23.649 INFO:tasks.workunit.client.0.smithi006.stdout:checking whether time.h and sys/time.h may both be included... yes 2024-04-24T23:18:23.654 INFO:tasks.workunit.client.0.smithi006.stdout:checking for function prototypes... yes 2024-04-24T23:18:23.745 INFO:tasks.workunit.client.0.smithi006.stdout:checking whether setvbuf arguments are reversed... no 2024-04-24T23:18:23.911 INFO:tasks.workunit.client.0.smithi006.stdout:checking for system... yes 2024-04-24T23:18:24.061 INFO:tasks.workunit.client.0.smithi006.stdout:checking for gettimeofday... yes 2024-04-24T23:18:24.202 INFO:tasks.workunit.client.0.smithi006.stdout:checking for mkdir... yes 2024-04-24T23:18:24.357 INFO:tasks.workunit.client.0.smithi006.stdout:checking for strerror... yes 2024-04-24T23:18:24.509 INFO:tasks.workunit.client.0.smithi006.stdout:checking for open64... yes 2024-04-24T23:18:24.650 INFO:tasks.workunit.client.0.smithi006.stdout:checking for stat64... yes 2024-04-24T23:18:24.693 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:18:24.789 INFO:tasks.workunit.client.0.smithi006.stdout:checking for fseeko64... yes 2024-04-24T23:18:24.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:24.883+0000 7f8a39e00640 1 Processor -- start 2024-04-24T23:18:24.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:24.883+0000 7f8a39e00640 1 -- start start 2024-04-24T23:18:24.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:24.884+0000 7f8a39e00640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f8a340ca520 0x7f8a340cb810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:18:24.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:24.885+0000 7f8a39e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f8a3415c1b0 0x7f8a340cd400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:18:24.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:24.885+0000 7f8a39e00640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8a340ca900 con 0x7f8a3415e2b0 2024-04-24T23:18:24.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:24.885+0000 7f8a39e00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f8a340ccf70 con 0x7f8a3415c1b0 2024-04-24T23:18:24.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:24.885+0000 7f8a39e00640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f8a340cc360 con 0x7f8a340ca520 2024-04-24T23:18:24.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:24.885+0000 7f8a32000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f8a3415c1b0 0x7f8a340cd400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:18:24.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:24.885+0000 7f8a32000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f8a3415c1b0 0x7f8a340cd400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:56114/0 (socket says 172.21.15.6:56114) 2024-04-24T23:18:24.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:24.885+0000 7f8a32000640 1 -- 172.21.15.6:0/1031179668 learned_addr learned my addr 172.21.15.6:0/1031179668 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:18:24.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:24.885+0000 7f8a32a00640 1 --2- 172.21.15.6:0/1031179668 >> v2:172.21.15.190:3300/0 conn(0x7f8a340ca520 0x7f8a340cb810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:18:24.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:24.885+0000 7f8a32000640 1 -- 172.21.15.6:0/1031179668 >> v2:172.21.15.190:3300/0 conn(0x7f8a340ca520 msgr2=0x7f8a340cb810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:18:24.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:24.885+0000 7f8a32000640 1 --2- 172.21.15.6:0/1031179668 >> v2:172.21.15.190:3300/0 conn(0x7f8a340ca520 0x7f8a340cb810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:24.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:24.886+0000 7f8a32000640 1 -- 172.21.15.6:0/1031179668 >> v1:172.21.15.190:6789/0 conn(0x7f8a3415e2b0 legacy=0x7f8a3415e740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:18:24.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:24.886+0000 7f8a32000640 1 -- 172.21.15.6:0/1031179668 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a340cc640 con 0x7f8a3415c1b0 2024-04-24T23:18:24.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:24.886+0000 7f8a32a00640 1 --2- 172.21.15.6:0/1031179668 >> v2:172.21.15.190:3300/0 conn(0x7f8a340ca520 0x7f8a340cb810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:18:24.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:24.887+0000 7f8a32000640 1 --2- 172.21.15.6:0/1031179668 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8a3415c1b0 0x7f8a340cd400 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f8a3415be70 tx=0x7f8a2002a300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1733fa90ce3efa5e server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:18:24.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:24.887+0000 7f8a31600640 1 -- 172.21.15.6:0/1031179668 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a20037070 con 0x7f8a3415c1b0 2024-04-24T23:18:24.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:24.887+0000 7f8a31600640 1 -- 172.21.15.6:0/1031179668 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a2002ae50 con 0x7f8a3415c1b0 2024-04-24T23:18:24.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:24.887+0000 7f8a31600640 1 -- 172.21.15.6:0/1031179668 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a20032860 con 0x7f8a3415c1b0 2024-04-24T23:18:24.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:24.888+0000 7f8a39e00640 1 -- 172.21.15.6:0/1031179668 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8a3415c1b0 msgr2=0x7f8a340cd400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:18:24.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:24.888+0000 7f8a39e00640 1 --2- 172.21.15.6:0/1031179668 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8a3415c1b0 0x7f8a340cd400 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f8a3415be70 tx=0x7f8a2002a300 comp rx=0 tx=0).stop 2024-04-24T23:18:24.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:24.888+0000 7f8a39e00640 1 -- 172.21.15.6:0/1031179668 shutdown_connections 2024-04-24T23:18:24.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:24.888+0000 7f8a39e00640 1 --2- 172.21.15.6:0/1031179668 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8a3415c1b0 0x7f8a340cd400 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:24.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:24.888+0000 7f8a39e00640 1 --2- 172.21.15.6:0/1031179668 >> v2:172.21.15.190:3300/0 conn(0x7f8a340ca520 0x7f8a340cb810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:24.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:24.888+0000 7f8a39e00640 1 -- 172.21.15.6:0/1031179668 >> 172.21.15.6:0/1031179668 conn(0x7f8a3406c0f0 msgr2=0x7f8a34066ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:18:24.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:24.888+0000 7f8a39e00640 1 -- 172.21.15.6:0/1031179668 shutdown_connections 2024-04-24T23:18:24.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:24.888+0000 7f8a39e00640 1 -- 172.21.15.6:0/1031179668 wait complete. 2024-04-24T23:18:24.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:24.890+0000 7f8a39e00640 1 Processor -- start 2024-04-24T23:18:24.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:24.890+0000 7f8a39e00640 1 -- start start 2024-04-24T23:18:24.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:24.891+0000 7f8a39e00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8a340ca520 0x7f8a341e4080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:18:24.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:24.891+0000 7f8a39e00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8a3415c1b0 0x7f8a34151ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:18:24.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:24.891+0000 7f8a39e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8a3415e2b0 0x7f8a341e6f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:18:24.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:24.891+0000 7f8a39e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f8a340cd940 con 0x7f8a3415e2b0 2024-04-24T23:18:24.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:24.891+0000 7f8a39e00640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f8a340ccc00 con 0x7f8a3415c1b0 2024-04-24T23:18:24.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:24.891+0000 7f8a39e00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f8a34067890 con 0x7f8a340ca520 2024-04-24T23:18:24.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:24.891+0000 7f8a32000640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8a3415c1b0 0x7f8a34151ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:18:24.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:24.891+0000 7f8a32a00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8a340ca520 0x7f8a341e4080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:18:24.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:24.891+0000 7f8a32000640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8a3415c1b0 0x7f8a34151ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:60214/0 (socket says 172.21.15.6:60214) 2024-04-24T23:18:24.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:24.891+0000 7f8a32000640 1 -- 172.21.15.6:0/2438506821 learned_addr learned my addr 172.21.15.6:0/2438506821 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:18:24.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:24.891+0000 7f8a33400640 1 --2- 172.21.15.6:0/2438506821 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8a3415e2b0 0x7f8a341e6f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:18:24.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:24.892+0000 7f8a32000640 1 -- 172.21.15.6:0/2438506821 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8a340ca520 msgr2=0x7f8a341e4080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:18:24.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:24.892+0000 7f8a32000640 1 --2- 172.21.15.6:0/2438506821 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8a340ca520 0x7f8a341e4080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:24.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:24.892+0000 7f8a32000640 1 -- 172.21.15.6:0/2438506821 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8a3415e2b0 msgr2=0x7f8a341e6f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:18:24.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:24.892+0000 7f8a32000640 1 --2- 172.21.15.6:0/2438506821 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8a3415e2b0 0x7f8a341e6f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:24.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:24.892+0000 7f8a32000640 1 -- 172.21.15.6:0/2438506821 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a341e6770 con 0x7f8a3415c1b0 2024-04-24T23:18:24.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:24.892+0000 7f8a32000640 1 --2- 172.21.15.6:0/2438506821 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8a3415c1b0 0x7f8a34151ba0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f8a2002a340 tx=0x7f8a200046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:18:24.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:24.893+0000 7f8a2b400640 1 -- 172.21.15.6:0/2438506821 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a20041070 con 0x7f8a3415c1b0 2024-04-24T23:18:24.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:24.893+0000 7f8a2b400640 1 -- 172.21.15.6:0/2438506821 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a2003c670 con 0x7f8a3415c1b0 2024-04-24T23:18:24.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:24.893+0000 7f8a2b400640 1 -- 172.21.15.6:0/2438506821 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a20037040 con 0x7f8a3415c1b0 2024-04-24T23:18:24.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:24.893+0000 7f8a39e00640 1 -- 172.21.15.6:0/2438506821 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a341e4730 con 0x7f8a3415c1b0 2024-04-24T23:18:24.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:24.893+0000 7f8a39e00640 1 -- 172.21.15.6:0/2438506821 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a341e4cf0 con 0x7f8a3415c1b0 2024-04-24T23:18:24.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:24.894+0000 7f8a39e00640 1 -- 172.21.15.6:0/2438506821 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a340767b0 con 0x7f8a3415c1b0 2024-04-24T23:18:24.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:24.895+0000 7f8a2b400640 1 -- 172.21.15.6:0/2438506821 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f8a20044070 con 0x7f8a3415c1b0 2024-04-24T23:18:24.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:24.896+0000 7f8a2b400640 1 --2- 172.21.15.6:0/2438506821 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f8a1406be60 0x7f8a1406e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:18:24.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:24.896+0000 7f8a2b400640 1 -- 172.21.15.6:0/2438506821 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6590+0+0 (secure 0 0 0) 0x7f8a200abaf0 con 0x7f8a3415c1b0 2024-04-24T23:18:24.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:24.897+0000 7f8a32a00640 1 --2- 172.21.15.6:0/2438506821 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f8a1406be60 0x7f8a1406e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:18:24.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:24.897+0000 7f8a32a00640 1 --2- 172.21.15.6:0/2438506821 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f8a1406be60 0x7f8a1406e320 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f8a24006fd0 tx=0x7f8a24008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:18:24.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:24.901+0000 7f8a2b400640 1 -- 172.21.15.6:0/2438506821 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f8a20079b60 con 0x7f8a3415c1b0 2024-04-24T23:18:24.916 INFO:tasks.workunit.client.0.smithi006.stdout:checking for lrand48_r... yes 2024-04-24T23:18:25.063 INFO:tasks.workunit.client.0.smithi006.stdout:checking for srand48_r... yes 2024-04-24T23:18:25.115 INFO:tasks.workunit.client.0.smithi006.stdout:configure: creating ./config.status 2024-04-24T23:18:25.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:25.152+0000 7f8a39e00640 1 -- 172.21.15.6:0/2438506821 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8a3415e020 con 0x7f8a3415c1b0 2024-04-24T23:18:25.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:25.154+0000 7f8a2b400640 1 -- 172.21.15.6:0/2438506821 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) ==== 94+0+6902 (secure 0 0 0) 0x7f8a2007de60 con 0x7f8a3415c1b0 2024-04-24T23:18:25.156 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:18:25.156 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":18,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4197,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/105267896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":105267896},{"type":"v1","addr":"172.21.15.6:6834","nonce":105267896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:16:32.310838+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4282,"mds_2":4235,"mds_3":4232,"mds_4":4203},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4203":{"gid":4203,"name":"e","rank":4,"incarnation":17,"state":"up:active","state_seq":17,"addr":"172.21.15.6:6837/2590421094","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":2590421094},{"type":"v1","addr":"172.21.15.6:6837","nonce":2590421094}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4232":{"gid":4232,"name":"b","rank":3,"incarnation":15,"state":"up:active","state_seq":15,"addr":"172.21.15.190:6838/2032280158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2032280158},{"type":"v1","addr":"172.21.15.190:6838","nonce":2032280158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4235":{"gid":4235,"name":"d","rank":2,"incarnation":13,"state":"up:active","state_seq":14,"addr":"172.21.15.190:6839/1572872451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":1572872451},{"type":"v1","addr":"172.21.15.190:6839","nonce":1572872451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4282":{"gid":4282,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":12,"addr":"172.21.15.190:6837/580315269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":580315269},{"type":"v1","addr":"172.21.15.190:6837","nonce":580315269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/489092393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":489092393},{"type":"v1","addr":"172.21.15.6:6835","nonce":489092393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4306,"qdb_cluster":[4203,4232,4235,4282,4306]},"id":1}]} 2024-04-24T23:18:25.156 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 18 2024-04-24T23:18:25.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:25.156+0000 7f8a39e00640 1 -- 172.21.15.6:0/2438506821 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f8a1406be60 msgr2=0x7f8a1406e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:18:25.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:25.156+0000 7f8a39e00640 1 --2- 172.21.15.6:0/2438506821 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f8a1406be60 0x7f8a1406e320 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f8a24006fd0 tx=0x7f8a24008040 comp rx=0 tx=0).stop 2024-04-24T23:18:25.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:25.156+0000 7f8a39e00640 1 -- 172.21.15.6:0/2438506821 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8a3415c1b0 msgr2=0x7f8a34151ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:18:25.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:25.156+0000 7f8a39e00640 1 --2- 172.21.15.6:0/2438506821 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8a3415c1b0 0x7f8a34151ba0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f8a2002a340 tx=0x7f8a200046c0 comp rx=0 tx=0).stop 2024-04-24T23:18:25.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:25.156+0000 7f8a39e00640 1 -- 172.21.15.6:0/2438506821 shutdown_connections 2024-04-24T23:18:25.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:25.156+0000 7f8a39e00640 1 --2- 172.21.15.6:0/2438506821 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f8a1406be60 0x7f8a1406e320 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:25.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:25.156+0000 7f8a39e00640 1 --2- 172.21.15.6:0/2438506821 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8a3415e2b0 0x7f8a341e6f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:25.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:25.156+0000 7f8a39e00640 1 --2- 172.21.15.6:0/2438506821 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8a3415c1b0 0x7f8a34151ba0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:25.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:25.156+0000 7f8a39e00640 1 --2- 172.21.15.6:0/2438506821 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8a340ca520 0x7f8a341e4080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:25.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:25.156+0000 7f8a39e00640 1 -- 172.21.15.6:0/2438506821 >> 172.21.15.6:0/2438506821 conn(0x7f8a3406c0f0 msgr2=0x7f8a340ce600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:18:25.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:25.157+0000 7f8a39e00640 1 -- 172.21.15.6:0/2438506821 shutdown_connections 2024-04-24T23:18:25.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:25.157+0000 7f8a39e00640 1 -- 172.21.15.6:0/2438506821 wait complete. 2024-04-24T23:18:25.177 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2024-04-24T23:18:25.401 INFO:tasks.workunit.client.0.smithi006.stdout:config.status: creating Makefile 2024-04-24T23:18:25.440 INFO:tasks.workunit.client.0.smithi006.stdout:config.status: creating config.h 2024-04-24T23:18:25.490 INFO:tasks.workunit.client.0.smithi006.stdout:config.status: executing depfiles commands 2024-04-24T23:18:25.919 INFO:tasks.workunit.client.0.smithi006.stdout:config.status: executing default commands 2024-04-24T23:18:26.041 INFO:tasks.workunit.client.0.smithi006.stderr:+ make 2024-04-24T23:18:26.064 INFO:tasks.workunit.client.0.smithi006.stdout:cd . && /bin/sh /home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing --run aclocal-1.10 2024-04-24T23:18:26.074 INFO:tasks.workunit.client.0.smithi006.stderr:/home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing: line 46: aclocal-1.10: command not found 2024-04-24T23:18:26.077 INFO:tasks.workunit.client.0.smithi006.stderr:WARNING: `aclocal-1.10' is missing on your system. You should only need it if 2024-04-24T23:18:26.077 INFO:tasks.workunit.client.0.smithi006.stderr: you modified `acinclude.m4' or `configure.in'. You might want 2024-04-24T23:18:26.077 INFO:tasks.workunit.client.0.smithi006.stderr: to install the `Automake' and `Perl' packages. Grab them from 2024-04-24T23:18:26.077 INFO:tasks.workunit.client.0.smithi006.stderr: any GNU archive site. 2024-04-24T23:18:26.083 INFO:tasks.workunit.client.0.smithi006.stdout: cd . && /bin/sh /home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing --run automake-1.10 --gnu 2024-04-24T23:18:26.088 INFO:tasks.workunit.client.0.smithi006.stderr:/home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing: line 46: automake-1.10: command not found 2024-04-24T23:18:26.091 INFO:tasks.workunit.client.0.smithi006.stderr:WARNING: `automake-1.10' is missing on your system. You should only need it if 2024-04-24T23:18:26.091 INFO:tasks.workunit.client.0.smithi006.stderr: you modified `Makefile.am', `acinclude.m4' or `configure.in'. 2024-04-24T23:18:26.091 INFO:tasks.workunit.client.0.smithi006.stderr: You might want to install the `Automake' and `Perl' packages. 2024-04-24T23:18:26.091 INFO:tasks.workunit.client.0.smithi006.stderr: Grab them from any GNU archive site. 2024-04-24T23:18:26.914 INFO:tasks.workunit.client.0.smithi006.stdout:cd . && /bin/sh /home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing --run autoconf 2024-04-24T23:18:27.177 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:18:27.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:27.418+0000 7fa5a8800640 1 Processor -- start 2024-04-24T23:18:27.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:27.418+0000 7fa5a8800640 1 -- start start 2024-04-24T23:18:27.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:27.420+0000 7fa5a8800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa5a0071530 0x7fa5a00c8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:18:27.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:27.420+0000 7fa5a8800640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fa5a00c9230 0x7fa5a0157950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:18:27.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:27.420+0000 7fa5a8800640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5a00c8390 con 0x7fa5a0151bb0 2024-04-24T23:18:27.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:27.420+0000 7fa5a8800640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7fa5a0071d10 con 0x7fa5a0071530 2024-04-24T23:18:27.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:27.420+0000 7fa5a8800640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7fa5a0071ff0 con 0x7fa5a00c9230 2024-04-24T23:18:27.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:27.420+0000 7fa5a5e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fa5a0151bb0 0x7fa5a00c37f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:51156/0 (socket says 172.21.15.6:51156) 2024-04-24T23:18:27.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:27.420+0000 7fa5a5e00640 1 -- 172.21.15.6:0/2875547149 learned_addr learned my addr 172.21.15.6:0/2875547149 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:18:27.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:27.421+0000 7fa5a6800640 1 --2- 172.21.15.6:0/2875547149 >> v2:172.21.15.190:3300/0 conn(0x7fa5a00c9230 0x7fa5a0157950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:18:27.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:27.421+0000 7fa5a5400640 1 --2- 172.21.15.6:0/2875547149 >> v2:172.21.15.6:3300/0 conn(0x7fa5a0071530 0x7fa5a00c8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:18:27.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:27.421+0000 7fa5a6800640 1 -- 172.21.15.6:0/2875547149 >> v2:172.21.15.6:3300/0 conn(0x7fa5a0071530 msgr2=0x7fa5a00c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:18:27.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:27.421+0000 7fa5a6800640 1 --2- 172.21.15.6:0/2875547149 >> v2:172.21.15.6:3300/0 conn(0x7fa5a0071530 0x7fa5a00c8cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:27.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:27.421+0000 7fa5a6800640 1 -- 172.21.15.6:0/2875547149 >> v1:172.21.15.6:6789/0 conn(0x7fa5a0151bb0 legacy=0x7fa5a00c37f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:18:27.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:27.421+0000 7fa5a4a00640 1 -- 172.21.15.6:0/2875547149 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 829664577 0 0) 0x7fa5a00c8390 con 0x7fa5a0151bb0 2024-04-24T23:18:27.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:27.421+0000 7fa5a6800640 1 -- 172.21.15.6:0/2875547149 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5a00722d0 con 0x7fa5a00c9230 2024-04-24T23:18:27.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:27.421+0000 7fa5a5400640 1 --2- 172.21.15.6:0/2875547149 >> v2:172.21.15.6:3300/0 conn(0x7fa5a0071530 0x7fa5a00c8cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:18:27.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:27.423+0000 7fa5a6800640 1 --2- 172.21.15.6:0/2875547149 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa5a00c9230 0x7fa5a0157950 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fa5a01553b0 tx=0x7fa59c02a710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=404ed2b0ec9a8114 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:18:27.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:27.423+0000 7fa5a4a00640 1 -- 172.21.15.6:0/2875547149 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa59c007740 con 0x7fa5a00c9230 2024-04-24T23:18:27.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:27.423+0000 7fa5a4a00640 1 -- 172.21.15.6:0/2875547149 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa59c0078e0 con 0x7fa5a00c9230 2024-04-24T23:18:27.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:27.423+0000 7fa5a4a00640 1 -- 172.21.15.6:0/2875547149 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa59c007cb0 con 0x7fa5a00c9230 2024-04-24T23:18:27.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:27.424+0000 7fa5a8800640 1 -- 172.21.15.6:0/2875547149 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa5a00c9230 msgr2=0x7fa5a0157950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:18:27.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:27.424+0000 7fa5a8800640 1 --2- 172.21.15.6:0/2875547149 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa5a00c9230 0x7fa5a0157950 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fa5a01553b0 tx=0x7fa59c02a710 comp rx=0 tx=0).stop 2024-04-24T23:18:27.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:27.424+0000 7fa5a8800640 1 -- 172.21.15.6:0/2875547149 shutdown_connections 2024-04-24T23:18:27.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:27.424+0000 7fa5a8800640 1 --2- 172.21.15.6:0/2875547149 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa5a00c9230 0x7fa5a0157950 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:27.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:27.424+0000 7fa5a8800640 1 --2- 172.21.15.6:0/2875547149 >> v2:172.21.15.6:3300/0 conn(0x7fa5a0071530 0x7fa5a00c8cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:27.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:27.424+0000 7fa5a8800640 1 -- 172.21.15.6:0/2875547149 >> 172.21.15.6:0/2875547149 conn(0x7fa5a006b0e0 msgr2=0x7fa5a006b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:18:27.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:27.424+0000 7fa5a8800640 1 -- 172.21.15.6:0/2875547149 shutdown_connections 2024-04-24T23:18:27.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:27.424+0000 7fa5a8800640 1 -- 172.21.15.6:0/2875547149 wait complete. 2024-04-24T23:18:27.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:27.426+0000 7fa5a8800640 1 Processor -- start 2024-04-24T23:18:27.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:27.426+0000 7fa5a8800640 1 -- start start 2024-04-24T23:18:27.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:27.427+0000 7fa5a8800640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa5a0071530 0x7fa5a0150180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:18:27.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:27.427+0000 7fa5a8800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa5a00c9230 0x7fa5a01562c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:18:27.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:27.427+0000 7fa5a8800640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa5a0151bb0 0x7fa5a01f0910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:18:27.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:27.427+0000 7fa5a8800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fa5a00c80f0 con 0x7fa5a00c9230 2024-04-24T23:18:27.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:27.427+0000 7fa5a8800640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fa5a00c7f70 con 0x7fa5a0071530 2024-04-24T23:18:27.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:27.427+0000 7fa5a8800640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fa5a0157e90 con 0x7fa5a0151bb0 2024-04-24T23:18:27.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:27.427+0000 7fa5a5400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa5a00c9230 0x7fa5a01562c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:18:27.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:27.427+0000 7fa5a5400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa5a00c9230 0x7fa5a01562c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:56144/0 (socket says 172.21.15.6:56144) 2024-04-24T23:18:27.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:27.427+0000 7fa5a6800640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa5a0151bb0 0x7fa5a01f0910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:18:27.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:27.427+0000 7fa5a5400640 1 -- 172.21.15.6:0/4062606727 learned_addr learned my addr 172.21.15.6:0/4062606727 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:18:27.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:27.427+0000 7fa5a5e00640 1 --2- 172.21.15.6:0/4062606727 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa5a0071530 0x7fa5a0150180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:18:27.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:27.428+0000 7fa5a5400640 1 -- 172.21.15.6:0/4062606727 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa5a0151bb0 msgr2=0x7fa5a01f0910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:18:27.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:27.428+0000 7fa5a5400640 1 --2- 172.21.15.6:0/4062606727 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa5a0151bb0 0x7fa5a01f0910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:27.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:27.428+0000 7fa5a5400640 1 -- 172.21.15.6:0/4062606727 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa5a0071530 msgr2=0x7fa5a0150180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:18:27.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:27.428+0000 7fa5a5400640 1 --2- 172.21.15.6:0/4062606727 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa5a0071530 0x7fa5a0150180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:27.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:27.428+0000 7fa5a5400640 1 -- 172.21.15.6:0/4062606727 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5a0156c90 con 0x7fa5a00c9230 2024-04-24T23:18:27.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:27.428+0000 7fa5a5400640 1 --2- 172.21.15.6:0/4062606727 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa5a00c9230 0x7fa5a01562c0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fa59000ebe0 tx=0x7fa59000c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:18:27.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:27.428+0000 7fa59a000640 1 -- 172.21.15.6:0/4062606727 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa590017070 con 0x7fa5a00c9230 2024-04-24T23:18:27.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:27.428+0000 7fa59a000640 1 -- 172.21.15.6:0/4062606727 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa59000cc30 con 0x7fa5a00c9230 2024-04-24T23:18:27.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:27.428+0000 7fa5a8800640 1 -- 172.21.15.6:0/4062606727 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5a0156f70 con 0x7fa5a00c9230 2024-04-24T23:18:27.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:27.428+0000 7fa59a000640 1 -- 172.21.15.6:0/4062606727 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa590013470 con 0x7fa5a00c9230 2024-04-24T23:18:27.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:27.429+0000 7fa5a8800640 1 -- 172.21.15.6:0/4062606727 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5a01f0110 con 0x7fa5a00c9230 2024-04-24T23:18:27.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:27.430+0000 7fa5a8800640 1 -- 172.21.15.6:0/4062606727 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa5a014f180 con 0x7fa5a00c9230 2024-04-24T23:18:27.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:27.431+0000 7fa59a000640 1 -- 172.21.15.6:0/4062606727 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fa59001a070 con 0x7fa5a00c9230 2024-04-24T23:18:27.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:27.432+0000 7fa59a000640 1 --2- 172.21.15.6:0/4062606727 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fa58406be30 0x7fa58406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:18:27.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:27.432+0000 7fa59a000640 1 -- 172.21.15.6:0/4062606727 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6590+0+0 (secure 0 0 0) 0x7fa59008b530 con 0x7fa5a00c9230 2024-04-24T23:18:27.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:27.433+0000 7fa5a5e00640 1 --2- 172.21.15.6:0/4062606727 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fa58406be30 0x7fa58406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:18:27.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:27.433+0000 7fa5a5e00640 1 --2- 172.21.15.6:0/4062606727 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fa58406be30 0x7fa58406e2f0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fa594007fd0 tx=0x7fa594007de0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:18:27.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:27.437+0000 7fa59a000640 1 -- 172.21.15.6:0/4062606727 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fa5900593f0 con 0x7fa5a00c9230 2024-04-24T23:18:27.455 INFO:tasks.workunit.client.0.smithi006.stderr:aclocal.m4:16: warning: this file was generated for autoconf 2.61. 2024-04-24T23:18:27.455 INFO:tasks.workunit.client.0.smithi006.stderr:You have another version of autoconf. It may work, but is not guaranteed to. 2024-04-24T23:18:27.455 INFO:tasks.workunit.client.0.smithi006.stderr:If you have problems, you may need to regenerate the build system entirely. 2024-04-24T23:18:27.455 INFO:tasks.workunit.client.0.smithi006.stderr:To do so, use the procedure documented by the package, typically `autoreconf'. 2024-04-24T23:18:27.617 INFO:tasks.workunit.client.0.smithi006.stdout:/bin/sh ./config.status --recheck 2024-04-24T23:18:27.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:27.727+0000 7fa5a8800640 1 -- 172.21.15.6:0/4062606727 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa5a006f970 con 0x7fa5a00c9230 2024-04-24T23:18:27.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:27.732+0000 7fa59a000640 1 -- 172.21.15.6:0/4062606727 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) ==== 94+0+6077 (secure 0 0 0) 0x7fa59005d6f0 con 0x7fa5a00c9230 2024-04-24T23:18:27.734 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:18:27.738 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:18:27.556689+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4282,"mds_2":4235,"mds_3":4232,"mds_4":4197},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4197":{"gid":4197,"name":"a","rank":4,"incarnation":19,"state":"up:replay","state_seq":1,"addr":"172.21.15.6:6834/105267896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":105267896},{"type":"v1","addr":"172.21.15.6:6834","nonce":105267896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4232":{"gid":4232,"name":"b","rank":3,"incarnation":15,"state":"up:active","state_seq":15,"addr":"172.21.15.190:6838/2032280158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2032280158},{"type":"v1","addr":"172.21.15.190:6838","nonce":2032280158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4235":{"gid":4235,"name":"d","rank":2,"incarnation":13,"state":"up:active","state_seq":14,"addr":"172.21.15.190:6839/1572872451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":1572872451},{"type":"v1","addr":"172.21.15.190:6839","nonce":1572872451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4282":{"gid":4282,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":12,"addr":"172.21.15.190:6837/580315269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":580315269},{"type":"v1","addr":"172.21.15.190:6837","nonce":580315269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/489092393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":489092393},{"type":"v1","addr":"172.21.15.6:6835","nonce":489092393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4306,"qdb_cluster":[4232,4235,4282,4306]},"id":1}]} 2024-04-24T23:18:27.738 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 19 2024-04-24T23:18:27.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:27.737+0000 7fa5a8800640 1 -- 172.21.15.6:0/4062606727 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fa58406be30 msgr2=0x7fa58406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:18:27.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:27.737+0000 7fa5a8800640 1 --2- 172.21.15.6:0/4062606727 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fa58406be30 0x7fa58406e2f0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fa594007fd0 tx=0x7fa594007de0 comp rx=0 tx=0).stop 2024-04-24T23:18:27.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:27.737+0000 7fa5a8800640 1 -- 172.21.15.6:0/4062606727 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa5a00c9230 msgr2=0x7fa5a01562c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:18:27.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:27.737+0000 7fa5a8800640 1 --2- 172.21.15.6:0/4062606727 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa5a00c9230 0x7fa5a01562c0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fa59000ebe0 tx=0x7fa59000c520 comp rx=0 tx=0).stop 2024-04-24T23:18:27.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:27.738+0000 7fa5a8800640 1 -- 172.21.15.6:0/4062606727 shutdown_connections 2024-04-24T23:18:27.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:27.738+0000 7fa5a8800640 1 --2- 172.21.15.6:0/4062606727 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fa58406be30 0x7fa58406e2f0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:27.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:27.738+0000 7fa5a8800640 1 --2- 172.21.15.6:0/4062606727 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa5a0151bb0 0x7fa5a01f0910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:27.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:27.738+0000 7fa5a8800640 1 --2- 172.21.15.6:0/4062606727 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa5a00c9230 0x7fa5a01562c0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:27.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:27.738+0000 7fa5a8800640 1 --2- 172.21.15.6:0/4062606727 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa5a0071530 0x7fa5a0150180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:27.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:27.738+0000 7fa5a8800640 1 -- 172.21.15.6:0/4062606727 >> 172.21.15.6:0/4062606727 conn(0x7fa5a006b0e0 msgr2=0x7fa5a00c5cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:18:27.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:27.738+0000 7fa5a8800640 1 -- 172.21.15.6:0/4062606727 shutdown_connections 2024-04-24T23:18:27.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:27.738+0000 7fa5a8800640 1 -- 172.21.15.6:0/4062606727 wait complete. 2024-04-24T23:18:27.780 INFO:tasks.mds_thrash.fs.[cephfs]:mds.e down, removed from mdsmap 2024-04-24T23:18:27.780 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2024-04-24T23:18:27.780 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:18:27.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:27.952+0000 7f4613200640 1 Processor -- start 2024-04-24T23:18:27.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:27.952+0000 7f4613200640 1 -- start start 2024-04-24T23:18:27.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:27.954+0000 7f4613200640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f461406ce30 0x7f46140cc600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:18:27.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:27.954+0000 7f4613200640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f46140c6be0 0x7f46140ccb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:18:27.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:27.954+0000 7f4613200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f461406d390 con 0x7f4614152bd0 2024-04-24T23:18:27.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:27.954+0000 7f4613200640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f461406d970 con 0x7f461406ce30 2024-04-24T23:18:27.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:27.954+0000 7f4613200640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f461406dbc0 con 0x7f46140c6be0 2024-04-24T23:18:27.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:27.954+0000 7f4611e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f4614152bd0 0x7f4614158e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:51162/0 (socket says 172.21.15.6:51162) 2024-04-24T23:18:27.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:27.954+0000 7f4611e00640 1 -- 172.21.15.6:0/2184944322 learned_addr learned my addr 172.21.15.6:0/2184944322 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:18:27.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:27.954+0000 7f4609400640 1 --2- 172.21.15.6:0/2184944322 >> v2:172.21.15.190:3300/0 conn(0x7f461406ce30 0x7f46140cc600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:18:27.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:27.954+0000 7f4612800640 1 --2- 172.21.15.6:0/2184944322 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f46140c6be0 0x7f46140ccb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:18:27.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:27.955+0000 7f4609400640 1 -- 172.21.15.6:0/2184944322 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f46140c6be0 msgr2=0x7f46140ccb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:18:27.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:27.955+0000 7f4609400640 1 --2- 172.21.15.6:0/2184944322 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f46140c6be0 0x7f46140ccb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:27.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:27.955+0000 7f4609400640 1 -- 172.21.15.6:0/2184944322 >> v1:172.21.15.6:6789/0 conn(0x7f4614152bd0 legacy=0x7f4614158e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:18:27.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:27.955+0000 7f4611400640 1 -- 172.21.15.6:0/2184944322 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3691907060 0 0) 0x7f461406d390 con 0x7f4614152bd0 2024-04-24T23:18:27.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:27.955+0000 7f4609400640 1 -- 172.21.15.6:0/2184944322 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f461406dea0 con 0x7f461406ce30 2024-04-24T23:18:27.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:27.956+0000 7f4609400640 1 --2- 172.21.15.6:0/2184944322 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f461406ce30 0x7f46140cc600 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f45fc0097a0 tx=0x7f45fc02a100 comp rx=0 tx=0).ready entity=mon.1 client_cookie=330b62091475a942 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:18:27.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:27.956+0000 7f4611400640 1 -- 172.21.15.6:0/2184944322 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45fc037070 con 0x7f461406ce30 2024-04-24T23:18:27.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:27.956+0000 7f4611400640 1 -- 172.21.15.6:0/2184944322 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f45fc02ae50 con 0x7f461406ce30 2024-04-24T23:18:27.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:27.956+0000 7f4611400640 1 -- 172.21.15.6:0/2184944322 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45fc032660 con 0x7f461406ce30 2024-04-24T23:18:27.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:27.957+0000 7f4613200640 1 -- 172.21.15.6:0/2184944322 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f461406ce30 msgr2=0x7f46140cc600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:18:27.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:27.957+0000 7f4613200640 1 --2- 172.21.15.6:0/2184944322 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f461406ce30 0x7f46140cc600 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f45fc0097a0 tx=0x7f45fc02a100 comp rx=0 tx=0).stop 2024-04-24T23:18:27.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:27.957+0000 7f4613200640 1 -- 172.21.15.6:0/2184944322 shutdown_connections 2024-04-24T23:18:27.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:27.957+0000 7f4613200640 1 --2- 172.21.15.6:0/2184944322 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f46140c6be0 0x7f46140ccb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:27.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:27.957+0000 7f4613200640 1 --2- 172.21.15.6:0/2184944322 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f461406ce30 0x7f46140cc600 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:27.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:27.957+0000 7f4613200640 1 -- 172.21.15.6:0/2184944322 >> 172.21.15.6:0/2184944322 conn(0x7f461406ed80 msgr2=0x7f461406f190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:18:27.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:27.957+0000 7f4613200640 1 -- 172.21.15.6:0/2184944322 shutdown_connections 2024-04-24T23:18:27.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:27.957+0000 7f4613200640 1 -- 172.21.15.6:0/2184944322 wait complete. 2024-04-24T23:18:27.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:27.959+0000 7f4613200640 1 Processor -- start 2024-04-24T23:18:27.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:27.959+0000 7f4613200640 1 -- start start 2024-04-24T23:18:27.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:27.960+0000 7f4613200640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f461406ce30 0x7f46140c4cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:18:27.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:27.960+0000 7f4613200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f46140c6be0 0x7f46141e1f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:18:27.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:27.960+0000 7f4613200640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4614152bd0 0x7f46141e2460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:18:27.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:27.960+0000 7f4613200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f4614072650 con 0x7f46140c6be0 2024-04-24T23:18:27.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:27.960+0000 7f4613200640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f4614071890 con 0x7f461406ce30 2024-04-24T23:18:27.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:27.960+0000 7f4613200640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f46140727d0 con 0x7f4614152bd0 2024-04-24T23:18:27.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:27.960+0000 7f4611e00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f461406ce30 0x7f46140c4cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:18:27.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:27.960+0000 7f4612800640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4614152bd0 0x7f46141e2460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:18:27.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:27.960+0000 7f4612800640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4614152bd0 0x7f46141e2460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.6:44484/0 (socket says 172.21.15.6:44484) 2024-04-24T23:18:27.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:27.960+0000 7f4612800640 1 -- 172.21.15.6:0/2155370806 learned_addr learned my addr 172.21.15.6:0/2155370806 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:18:27.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:27.961+0000 7f4609400640 1 --2- 172.21.15.6:0/2155370806 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f46140c6be0 0x7f46141e1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:18:27.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:27.961+0000 7f4611e00640 1 -- 172.21.15.6:0/2155370806 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4614152bd0 msgr2=0x7f46141e2460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:18:27.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:27.961+0000 7f4611e00640 1 --2- 172.21.15.6:0/2155370806 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4614152bd0 0x7f46141e2460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:27.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:27.961+0000 7f4611e00640 1 -- 172.21.15.6:0/2155370806 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f46140c6be0 msgr2=0x7f46141e1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:18:27.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:27.961+0000 7f4611e00640 1 --2- 172.21.15.6:0/2155370806 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f46140c6be0 0x7f46141e1f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:27.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:27.961+0000 7f4611e00640 1 -- 172.21.15.6:0/2155370806 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4614158750 con 0x7f461406ce30 2024-04-24T23:18:27.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:27.961+0000 7f4611e00640 1 --2- 172.21.15.6:0/2155370806 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f461406ce30 0x7f46140c4cb0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f460400c7e0 tx=0x7f460400cca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:18:27.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:27.961+0000 7f460aa00640 1 -- 172.21.15.6:0/2155370806 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4604018070 con 0x7f461406ce30 2024-04-24T23:18:27.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:27.961+0000 7f460aa00640 1 -- 172.21.15.6:0/2155370806 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f460400a640 con 0x7f461406ce30 2024-04-24T23:18:27.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:27.961+0000 7f4613200640 1 -- 172.21.15.6:0/2155370806 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4614158490 con 0x7f461406ce30 2024-04-24T23:18:27.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:27.961+0000 7f460aa00640 1 -- 172.21.15.6:0/2155370806 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f460400aa30 con 0x7f461406ce30 2024-04-24T23:18:27.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:27.962+0000 7f4613200640 1 -- 172.21.15.6:0/2155370806 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46141f1bf0 con 0x7f461406ce30 2024-04-24T23:18:27.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:27.963+0000 7f4613200640 1 -- 172.21.15.6:0/2155370806 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f461406d390 con 0x7f461406ce30 2024-04-24T23:18:27.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:27.964+0000 7f460aa00640 1 -- 172.21.15.6:0/2155370806 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f460401b070 con 0x7f461406ce30 2024-04-24T23:18:27.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:27.964+0000 7f460aa00640 1 --2- 172.21.15.6:0/2155370806 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f45f006be30 0x7f45f006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:18:27.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:27.965+0000 7f460aa00640 1 -- 172.21.15.6:0/2155370806 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6676+0+0 (secure 0 0 0) 0x7f460408a770 con 0x7f461406ce30 2024-04-24T23:18:27.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:27.965+0000 7f4609400640 1 --2- 172.21.15.6:0/2155370806 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f45f006be30 0x7f45f006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:18:27.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:27.965+0000 7f4609400640 1 --2- 172.21.15.6:0/2155370806 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f45f006be30 0x7f45f006e2f0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f45fc034000 tx=0x7f45fc02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:18:27.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:27.969+0000 7f460aa00640 1 -- 172.21.15.6:0/2155370806 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f461406d390 con 0x7f461406ce30 2024-04-24T23:18:28.136 INFO:tasks.workunit.client.0.smithi006.stdout:running CONFIG_SHELL=/bin/sh /bin/sh ./configure --no-create --no-recursion 2024-04-24T23:18:28.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:28.222+0000 7f4613200640 1 -- 172.21.15.6:0/2155370806 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4614075b30 con 0x7f461406ce30 2024-04-24T23:18:28.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:28.223+0000 7f460aa00640 1 -- 172.21.15.6:0/2155370806 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) ==== 94+0+6077 (secure 0 0 0) 0x7f4604058660 con 0x7f461406ce30 2024-04-24T23:18:28.225 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:18:28.226 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:18:27.556689+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4282,"mds_2":4235,"mds_3":4232,"mds_4":4197},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4197":{"gid":4197,"name":"a","rank":4,"incarnation":19,"state":"up:replay","state_seq":1,"addr":"172.21.15.6:6834/105267896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":105267896},{"type":"v1","addr":"172.21.15.6:6834","nonce":105267896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4232":{"gid":4232,"name":"b","rank":3,"incarnation":15,"state":"up:active","state_seq":15,"addr":"172.21.15.190:6838/2032280158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2032280158},{"type":"v1","addr":"172.21.15.190:6838","nonce":2032280158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4235":{"gid":4235,"name":"d","rank":2,"incarnation":13,"state":"up:active","state_seq":14,"addr":"172.21.15.190:6839/1572872451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":1572872451},{"type":"v1","addr":"172.21.15.190:6839","nonce":1572872451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4282":{"gid":4282,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":12,"addr":"172.21.15.190:6837/580315269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":580315269},{"type":"v1","addr":"172.21.15.190:6837","nonce":580315269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/489092393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":489092393},{"type":"v1","addr":"172.21.15.6:6835","nonce":489092393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4306,"qdb_cluster":[4232,4235,4282,4306]},"id":1}]} 2024-04-24T23:18:28.226 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 19 2024-04-24T23:18:28.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:28.225+0000 7f4613200640 1 -- 172.21.15.6:0/2155370806 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f45f006be30 msgr2=0x7f45f006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:18:28.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:28.225+0000 7f4613200640 1 --2- 172.21.15.6:0/2155370806 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f45f006be30 0x7f45f006e2f0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f45fc034000 tx=0x7f45fc02e000 comp rx=0 tx=0).stop 2024-04-24T23:18:28.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:28.225+0000 7f4613200640 1 -- 172.21.15.6:0/2155370806 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f461406ce30 msgr2=0x7f46140c4cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:18:28.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:28.225+0000 7f4613200640 1 --2- 172.21.15.6:0/2155370806 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f461406ce30 0x7f46140c4cb0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f460400c7e0 tx=0x7f460400cca0 comp rx=0 tx=0).stop 2024-04-24T23:18:28.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:28.226+0000 7f4613200640 1 -- 172.21.15.6:0/2155370806 shutdown_connections 2024-04-24T23:18:28.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:28.226+0000 7f4613200640 1 --2- 172.21.15.6:0/2155370806 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f45f006be30 0x7f45f006e2f0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:28.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:28.226+0000 7f4613200640 1 --2- 172.21.15.6:0/2155370806 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4614152bd0 0x7f46141e2460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:28.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:28.226+0000 7f4613200640 1 --2- 172.21.15.6:0/2155370806 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f46140c6be0 0x7f46141e1f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:28.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:28.226+0000 7f4613200640 1 --2- 172.21.15.6:0/2155370806 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f461406ce30 0x7f46140c4cb0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:28.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:28.226+0000 7f4613200640 1 -- 172.21.15.6:0/2155370806 >> 172.21.15.6:0/2155370806 conn(0x7f461406ed80 msgr2=0x7f46140cc600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:18:28.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:28.226+0000 7f4613200640 1 -- 172.21.15.6:0/2155370806 shutdown_connections 2024-04-24T23:18:28.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:28.226+0000 7f4613200640 1 -- 172.21.15.6:0/2155370806 wait complete. 2024-04-24T23:18:28.248 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 19, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-24T23:15:36.400716+0000", "damaged": [], "data_pools": [ 3, 4 ], "enabled": true, "epoch": 19, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4197": { "addr": "172.21.15.6:6834/105267896", "addrs": { "addrvec": [ { "addr": "172.21.15.6:6832", "nonce": 105267896, "type": "v2" }, { "addr": "172.21.15.6:6834", "nonce": 105267896, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4197, "incarnation": 19, "join_fscid": -1, "name": "a", "rank": 4, "state": "up:replay", "state_seq": 1 }, "gid_4232": { "addr": "172.21.15.190:6838/2032280158", "addrs": { "addrvec": [ { "addr": "172.21.15.190:6835", "nonce": 2032280158, "type": "v2" }, { "addr": "172.21.15.190:6838", "nonce": 2032280158, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4232, "incarnation": 15, "join_fscid": -1, "name": "b", "rank": 3, "state": "up:active", "state_seq": 15 }, "gid_4235": { "addr": "172.21.15.190:6839/1572872451", "addrs": { "addrvec": [ { "addr": "172.21.15.190:6836", "nonce": 1572872451, "type": "v2" }, { "addr": "172.21.15.190:6839", "nonce": 1572872451, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4235, "incarnation": 13, "join_fscid": -1, "name": "d", "rank": 2, "state": "up:active", "state_seq": 14 }, "gid_4282": { "addr": "172.21.15.190:6837/580315269", "addrs": { "addrvec": [ { "addr": "172.21.15.190:6834", "nonce": 580315269, "type": "v2" }, { "addr": "172.21.15.190:6837", "nonce": 580315269, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4282, "incarnation": 9, "join_fscid": -1, "name": "f", "rank": 1, "state": "up:active", "state_seq": 12 }, "gid_4306": { "addr": "172.21.15.6:6835/489092393", "addrs": { "addrvec": [ { "addr": "172.21.15.6:6833", "nonce": 489092393, "type": "v2" }, { "addr": "172.21.15.6:6835", "nonce": 489092393, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4306, "incarnation": 4, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 4 } }, "last_failure": 0, "last_failure_osd_epoch": 27, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-04-24T23:18:27.556689+0000", "qdb_cluster": [ 4232, 4235, 4282, 4306 ], "qdb_leader": 4306, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4306, "mds_1": 4282, "mds_2": 4235, "mds_3": 4232, "mds_4": 4197 } } } ], "standbys": [] } 2024-04-24T23:18:28.384 INFO:tasks.workunit.client.0.smithi006.stdout:checking for a BSD-compatible install... /usr/bin/install -c 2024-04-24T23:18:29.406 INFO:tasks.workunit.client.0.smithi006.stdout:checking whether build environment is sane... yes 2024-04-24T23:18:29.425 INFO:tasks.workunit.client.0.smithi006.stdout:checking for a thread-safe mkdir -p... /usr/bin/mkdir -p 2024-04-24T23:18:29.426 INFO:tasks.workunit.client.0.smithi006.stdout:checking for gawk... gawk 2024-04-24T23:18:29.464 INFO:tasks.workunit.client.0.smithi006.stdout:checking whether make sets $(MAKE)... yes 2024-04-24T23:18:29.566 INFO:tasks.workunit.client.0.smithi006.stdout:checking build system type... x86_64-unknown-linux-gnu 2024-04-24T23:18:29.566 INFO:tasks.workunit.client.0.smithi006.stdout:checking host system type... x86_64-unknown-linux-gnu 2024-04-24T23:18:29.589 INFO:tasks.workunit.client.0.smithi006.stdout:checking for style of include used by make... GNU 2024-04-24T23:18:29.863 INFO:tasks.workunit.client.0.smithi006.stdout:checking for gcc... gcc 2024-04-24T23:18:30.249 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:18:30.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:30.468+0000 7f4bca000640 1 Processor -- start 2024-04-24T23:18:30.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:30.468+0000 7f4bca000640 1 -- start start 2024-04-24T23:18:30.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:30.470+0000 7f4bca000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f4bc4154d70 0x7f4bc415b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:18:30.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:30.470+0000 7f4bca000640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4bc406cc20 con 0x7f4bc40c85f0 2024-04-24T23:18:30.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:30.470+0000 7f4bca000640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4bc406cf00 con 0x7f4bc40c6b10 2024-04-24T23:18:30.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:30.470+0000 7f4bca000640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f4bc406c9f0 con 0x7f4bc4154d70 2024-04-24T23:18:30.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:30.470+0000 7f4bc8c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f4bc4154d70 0x7f4bc415b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:18:30.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:30.470+0000 7f4bc8c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f4bc4154d70 0x7f4bc415b5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:56162/0 (socket says 172.21.15.6:56162) 2024-04-24T23:18:30.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:30.470+0000 7f4bc8c00640 1 -- 172.21.15.6:0/1416508559 learned_addr learned my addr 172.21.15.6:0/1416508559 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:18:30.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:30.471+0000 7f4bc8c00640 1 -- 172.21.15.6:0/1416508559 >> v1:172.21.15.190:6789/0 conn(0x7f4bc40c6b10 legacy=0x7f4bc40c7890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:18:30.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:30.471+0000 7f4bc8c00640 1 -- 172.21.15.6:0/1416508559 >> v1:172.21.15.6:6789/0 conn(0x7f4bc40c85f0 legacy=0x7f4bc41653f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:18:30.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:30.471+0000 7f4bc3400640 1 -- 172.21.15.6:0/1416508559 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1107121270 0 0) 0x7f4bc406cf00 con 0x7f4bc40c6b10 2024-04-24T23:18:30.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:30.471+0000 7f4bc8c00640 1 -- 172.21.15.6:0/1416508559 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bc406d1e0 con 0x7f4bc4154d70 2024-04-24T23:18:30.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:30.472+0000 7f4bc8c00640 1 --2- 172.21.15.6:0/1416508559 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4bc4154d70 0x7f4bc415b5d0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f4bb8009870 tx=0x7f4bb802a3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c84c2ffbe29730ca server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:18:30.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:30.472+0000 7f4bc3400640 1 -- 172.21.15.6:0/1416508559 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bb8037070 con 0x7f4bc4154d70 2024-04-24T23:18:30.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:30.472+0000 7f4bc3400640 1 -- 172.21.15.6:0/1416508559 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4bb8032640 con 0x7f4bc4154d70 2024-04-24T23:18:30.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:30.472+0000 7f4bc3400640 1 -- 172.21.15.6:0/1416508559 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bb8032a10 con 0x7f4bc4154d70 2024-04-24T23:18:30.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:30.473+0000 7f4bca000640 1 -- 172.21.15.6:0/1416508559 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4bc4154d70 msgr2=0x7f4bc415b5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:18:30.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:30.473+0000 7f4bca000640 1 --2- 172.21.15.6:0/1416508559 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4bc4154d70 0x7f4bc415b5d0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f4bb8009870 tx=0x7f4bb802a3c0 comp rx=0 tx=0).stop 2024-04-24T23:18:30.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:30.473+0000 7f4bca000640 1 -- 172.21.15.6:0/1416508559 shutdown_connections 2024-04-24T23:18:30.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:30.473+0000 7f4bca000640 1 --2- 172.21.15.6:0/1416508559 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4bc4154d70 0x7f4bc415b5d0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:30.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:30.473+0000 7f4bca000640 1 -- 172.21.15.6:0/1416508559 >> 172.21.15.6:0/1416508559 conn(0x7f4bc40c9320 msgr2=0x7f4bc40c9700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:18:30.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:30.473+0000 7f4bca000640 1 -- 172.21.15.6:0/1416508559 shutdown_connections 2024-04-24T23:18:30.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:30.473+0000 7f4bca000640 1 -- 172.21.15.6:0/1416508559 wait complete. 2024-04-24T23:18:30.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:30.475+0000 7f4bca000640 1 Processor -- start 2024-04-24T23:18:30.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:30.475+0000 7f4bca000640 1 -- start start 2024-04-24T23:18:30.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:30.476+0000 7f4bca000640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4bc40c6b10 0x7f4bc414f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:18:30.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:30.476+0000 7f4bca000640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4bc40c85f0 0x7f4bc414fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:18:30.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:30.476+0000 7f4bca000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4bc4154d70 0x7f4bc4159750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:18:30.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:30.476+0000 7f4bca000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f4bc40c7370 con 0x7f4bc4154d70 2024-04-24T23:18:30.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:30.476+0000 7f4bca000640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f4bc40c7120 con 0x7f4bc40c6b10 2024-04-24T23:18:30.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:30.476+0000 7f4bca000640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f4bc406dc80 con 0x7f4bc40c85f0 2024-04-24T23:18:30.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:30.476+0000 7f4bc8c00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4bc40c6b10 0x7f4bc414f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:18:30.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:30.476+0000 7f4bc8c00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4bc40c6b10 0x7f4bc414f6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:60256/0 (socket says 172.21.15.6:60256) 2024-04-24T23:18:30.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:30.476+0000 7f4bc8c00640 1 -- 172.21.15.6:0/772260714 learned_addr learned my addr 172.21.15.6:0/772260714 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:18:30.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:30.476+0000 7f4bc3e00640 1 --2- 172.21.15.6:0/772260714 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4bc40c85f0 0x7f4bc414fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:18:30.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:30.476+0000 7f4bc9600640 1 --2- 172.21.15.6:0/772260714 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4bc4154d70 0x7f4bc4159750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:18:30.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:30.477+0000 7f4bc8c00640 1 -- 172.21.15.6:0/772260714 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4bc40c85f0 msgr2=0x7f4bc414fc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:18:30.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:30.477+0000 7f4bc8c00640 1 --2- 172.21.15.6:0/772260714 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4bc40c85f0 0x7f4bc414fc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:30.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:30.477+0000 7f4bc8c00640 1 -- 172.21.15.6:0/772260714 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4bc4154d70 msgr2=0x7f4bc4159750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:18:30.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:30.477+0000 7f4bc8c00640 1 --2- 172.21.15.6:0/772260714 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4bc4154d70 0x7f4bc4159750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:30.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:30.477+0000 7f4bc8c00640 1 -- 172.21.15.6:0/772260714 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bc4155d70 con 0x7f4bc40c6b10 2024-04-24T23:18:30.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:30.477+0000 7f4bc8c00640 1 --2- 172.21.15.6:0/772260714 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4bc40c6b10 0x7f4bc414f6d0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f4bb8009870 tx=0x7f4bb80046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:18:30.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:30.477+0000 7f4bc0c00640 1 -- 172.21.15.6:0/772260714 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bb8037040 con 0x7f4bc40c6b10 2024-04-24T23:18:30.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:30.477+0000 7f4bc0c00640 1 -- 172.21.15.6:0/772260714 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4bb8004230 con 0x7f4bc40c6b10 2024-04-24T23:18:30.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:30.477+0000 7f4bc0c00640 1 -- 172.21.15.6:0/772260714 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bb80306f0 con 0x7f4bc40c6b10 2024-04-24T23:18:30.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:30.477+0000 7f4bca000640 1 -- 172.21.15.6:0/772260714 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4bc4156030 con 0x7f4bc40c6b10 2024-04-24T23:18:30.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:30.478+0000 7f4bca000640 1 -- 172.21.15.6:0/772260714 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4bc41565f0 con 0x7f4bc40c6b10 2024-04-24T23:18:30.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:30.479+0000 7f4bca000640 1 -- 172.21.15.6:0/772260714 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4bc4075b30 con 0x7f4bc40c6b10 2024-04-24T23:18:30.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:30.480+0000 7f4bc0c00640 1 -- 172.21.15.6:0/772260714 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f4bb8032500 con 0x7f4bc40c6b10 2024-04-24T23:18:30.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:30.481+0000 7f4bc0c00640 1 --2- 172.21.15.6:0/772260714 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f4ba406be30 0x7f4ba406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:18:30.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:30.481+0000 7f4bc0c00640 1 -- 172.21.15.6:0/772260714 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6676+0+0 (secure 0 0 0) 0x7f4bb80ac1d0 con 0x7f4bc40c6b10 2024-04-24T23:18:30.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:30.481+0000 7f4bc3e00640 1 --2- 172.21.15.6:0/772260714 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f4ba406be30 0x7f4ba406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:18:30.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:30.482+0000 7f4bc3e00640 1 --2- 172.21.15.6:0/772260714 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f4ba406be30 0x7f4ba406e2f0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f4bc4150f60 tx=0x7f4bb0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:18:30.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:30.485+0000 7f4bc0c00640 1 -- 172.21.15.6:0/772260714 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f4bb807a1e0 con 0x7f4bc40c6b10 2024-04-24T23:18:30.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:30.733+0000 7f4bca000640 1 -- 172.21.15.6:0/772260714 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4bc414eaa0 con 0x7f4bc40c6b10 2024-04-24T23:18:30.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:30.735+0000 7f4bc0c00640 1 -- 172.21.15.6:0/772260714 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) ==== 94+0+6077 (secure 0 0 0) 0x7f4bb807e4e0 con 0x7f4bc40c6b10 2024-04-24T23:18:30.737 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:18:30.737 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:18:27.556689+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4282,"mds_2":4235,"mds_3":4232,"mds_4":4197},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4197":{"gid":4197,"name":"a","rank":4,"incarnation":19,"state":"up:replay","state_seq":1,"addr":"172.21.15.6:6834/105267896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":105267896},{"type":"v1","addr":"172.21.15.6:6834","nonce":105267896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4232":{"gid":4232,"name":"b","rank":3,"incarnation":15,"state":"up:active","state_seq":15,"addr":"172.21.15.190:6838/2032280158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2032280158},{"type":"v1","addr":"172.21.15.190:6838","nonce":2032280158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4235":{"gid":4235,"name":"d","rank":2,"incarnation":13,"state":"up:active","state_seq":14,"addr":"172.21.15.190:6839/1572872451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":1572872451},{"type":"v1","addr":"172.21.15.190:6839","nonce":1572872451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4282":{"gid":4282,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":12,"addr":"172.21.15.190:6837/580315269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":580315269},{"type":"v1","addr":"172.21.15.190:6837","nonce":580315269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/489092393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":489092393},{"type":"v1","addr":"172.21.15.6:6835","nonce":489092393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4306,"qdb_cluster":[4232,4235,4282,4306]},"id":1}]} 2024-04-24T23:18:30.738 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 19 2024-04-24T23:18:30.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:30.738+0000 7f4bca000640 1 -- 172.21.15.6:0/772260714 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f4ba406be30 msgr2=0x7f4ba406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:18:30.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:30.738+0000 7f4bca000640 1 --2- 172.21.15.6:0/772260714 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f4ba406be30 0x7f4ba406e2f0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f4bc4150f60 tx=0x7f4bb0009040 comp rx=0 tx=0).stop 2024-04-24T23:18:30.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:30.738+0000 7f4bca000640 1 -- 172.21.15.6:0/772260714 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4bc40c6b10 msgr2=0x7f4bc414f6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:18:30.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:30.738+0000 7f4bca000640 1 --2- 172.21.15.6:0/772260714 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4bc40c6b10 0x7f4bc414f6d0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f4bb8009870 tx=0x7f4bb80046c0 comp rx=0 tx=0).stop 2024-04-24T23:18:30.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:30.738+0000 7f4bca000640 1 -- 172.21.15.6:0/772260714 shutdown_connections 2024-04-24T23:18:30.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:30.738+0000 7f4bca000640 1 --2- 172.21.15.6:0/772260714 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f4ba406be30 0x7f4ba406e2f0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:30.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:30.738+0000 7f4bca000640 1 --2- 172.21.15.6:0/772260714 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4bc40c85f0 0x7f4bc414fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:30.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:30.738+0000 7f4bca000640 1 --2- 172.21.15.6:0/772260714 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4bc4154d70 0x7f4bc4159750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:30.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:30.738+0000 7f4bca000640 1 --2- 172.21.15.6:0/772260714 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4bc40c6b10 0x7f4bc414f6d0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:30.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:30.738+0000 7f4bca000640 1 -- 172.21.15.6:0/772260714 >> 172.21.15.6:0/772260714 conn(0x7f4bc40c9320 msgr2=0x7f4bc4072400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:18:30.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:30.738+0000 7f4bca000640 1 -- 172.21.15.6:0/772260714 shutdown_connections 2024-04-24T23:18:30.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:30.738+0000 7f4bca000640 1 -- 172.21.15.6:0/772260714 wait complete. 2024-04-24T23:18:30.760 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-24T23:18:30.957 INFO:tasks.workunit.client.0.smithi006.stdout:checking whether the C compiler works... yes 2024-04-24T23:18:30.957 INFO:tasks.workunit.client.0.smithi006.stdout:checking for C compiler default output file name... a.out 2024-04-24T23:18:31.047 INFO:tasks.workunit.client.0.smithi006.stdout:checking for suffix of executables... 2024-04-24T23:18:31.142 INFO:tasks.workunit.client.0.smithi006.stdout:checking whether we are cross compiling... no 2024-04-24T23:18:31.229 INFO:tasks.workunit.client.0.smithi006.stdout:checking for suffix of object files... o 2024-04-24T23:18:31.291 INFO:tasks.workunit.client.0.smithi006.stdout:checking whether we are using the GNU C compiler... yes 2024-04-24T23:18:31.346 INFO:tasks.workunit.client.0.smithi006.stdout:checking whether gcc accepts -g... yes 2024-04-24T23:18:31.493 INFO:tasks.workunit.client.0.smithi006.stdout:checking for gcc option to accept ISO C89... none needed 2024-04-24T23:18:31.730 INFO:tasks.workunit.client.0.smithi006.stdout:checking dependency style of gcc... gcc3 2024-04-24T23:18:31.838 INFO:tasks.workunit.client.0.smithi006.stdout:checking how to run the C preprocessor... gcc -E 2024-04-24T23:18:31.974 INFO:tasks.workunit.client.0.smithi006.stdout:checking for grep that handles long lines and -e... /usr/bin/grep 2024-04-24T23:18:31.981 INFO:tasks.workunit.client.0.smithi006.stdout:checking for egrep... /usr/bin/grep -E 2024-04-24T23:18:32.253 INFO:tasks.workunit.client.0.smithi006.stdout:checking for ANSI C header files... yes 2024-04-24T23:18:32.344 INFO:tasks.workunit.client.0.smithi006.stdout:checking for sys/types.h... yes 2024-04-24T23:18:32.417 INFO:tasks.workunit.client.0.smithi006.stdout:checking for sys/stat.h... yes 2024-04-24T23:18:32.503 INFO:tasks.workunit.client.0.smithi006.stdout:checking for stdlib.h... yes 2024-04-24T23:18:32.761 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:18:32.969 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-24T23:18:32.976 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-24T23:18:32.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:32.974+0000 7f29e6400640 1 Processor -- start 2024-04-24T23:18:32.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:32.974+0000 7f29e6400640 1 -- start start 2024-04-24T23:18:32.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:32.976+0000 7f29e6400640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f29e00cb090 0x7f29e00d0820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:18:32.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:32.976+0000 7f29e6400640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f29e00d0d60 0x7f29e015e000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:18:32.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:32.976+0000 7f29e6400640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f29e0155170 con 0x7f29e0154d90 2024-04-24T23:18:32.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:32.976+0000 7f29e6400640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f29e00cc9a0 con 0x7f29e00d0d60 2024-04-24T23:18:32.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:32.976+0000 7f29e6400640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f29e015ad60 con 0x7f29e00cb090 2024-04-24T23:18:32.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:32.976+0000 7f29dfe00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f29e00cb090 0x7f29e00d0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:18:32.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:32.976+0000 7f29e5a00640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f29e00d0d60 0x7f29e015e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:18:32.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:32.976+0000 7f29e5000640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f29e0154d90 0x7f29e00cbe80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:51178/0 (socket says 172.21.15.6:51178) 2024-04-24T23:18:32.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:32.976+0000 7f29e5000640 1 -- 172.21.15.6:0/2582698674 learned_addr learned my addr 172.21.15.6:0/2582698674 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:18:32.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:32.977+0000 7f29dfe00640 1 -- 172.21.15.6:0/2582698674 >> v2:172.21.15.190:3300/0 conn(0x7f29e00d0d60 msgr2=0x7f29e015e000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:18:32.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:32.977+0000 7f29dfe00640 1 --2- 172.21.15.6:0/2582698674 >> v2:172.21.15.190:3300/0 conn(0x7f29e00d0d60 0x7f29e015e000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:32.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:32.977+0000 7f29dfe00640 1 -- 172.21.15.6:0/2582698674 >> v1:172.21.15.6:6789/0 conn(0x7f29e0154d90 legacy=0x7f29e00cbe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:18:32.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:32.977+0000 7f29dfe00640 1 -- 172.21.15.6:0/2582698674 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29e015b040 con 0x7f29e00cb090 2024-04-24T23:18:32.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:32.978+0000 7f29dfe00640 1 --2- 172.21.15.6:0/2582698674 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f29e00cb090 0x7f29e00d0820 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f29cc0097a0 tx=0x7f29cc02a210 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6627e797f30d6af8 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:18:32.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:32.978+0000 7f29df400640 1 -- 172.21.15.6:0/2582698674 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29cc037070 con 0x7f29e00cb090 2024-04-24T23:18:32.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:32.978+0000 7f29df400640 1 -- 172.21.15.6:0/2582698674 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f29cc032440 con 0x7f29e00cb090 2024-04-24T23:18:32.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:32.978+0000 7f29df400640 1 -- 172.21.15.6:0/2582698674 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29cc032810 con 0x7f29e00cb090 2024-04-24T23:18:32.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:32.979+0000 7f29e6400640 1 -- 172.21.15.6:0/2582698674 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f29e00cb090 msgr2=0x7f29e00d0820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:18:32.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:32.979+0000 7f29e6400640 1 --2- 172.21.15.6:0/2582698674 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f29e00cb090 0x7f29e00d0820 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f29cc0097a0 tx=0x7f29cc02a210 comp rx=0 tx=0).stop 2024-04-24T23:18:32.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:32.979+0000 7f29e6400640 1 -- 172.21.15.6:0/2582698674 shutdown_connections 2024-04-24T23:18:32.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:32.979+0000 7f29e6400640 1 --2- 172.21.15.6:0/2582698674 >> v2:172.21.15.190:3300/0 conn(0x7f29e00d0d60 0x7f29e015e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:32.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:32.979+0000 7f29e6400640 1 --2- 172.21.15.6:0/2582698674 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f29e00cb090 0x7f29e00d0820 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:32.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:32.979+0000 7f29e6400640 1 -- 172.21.15.6:0/2582698674 >> 172.21.15.6:0/2582698674 conn(0x7f29e006be10 msgr2=0x7f29e006c1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:18:32.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:32.979+0000 7f29e6400640 1 -- 172.21.15.6:0/2582698674 shutdown_connections 2024-04-24T23:18:32.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:32.979+0000 7f29e6400640 1 -- 172.21.15.6:0/2582698674 wait complete. 2024-04-24T23:18:32.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:32.981+0000 7f29e6400640 1 Processor -- start 2024-04-24T23:18:32.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:32.981+0000 7f29e6400640 1 -- start start 2024-04-24T23:18:32.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:32.982+0000 7f29e6400640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f29e00cb090 0x7f29e014c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:18:32.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:32.982+0000 7f29e6400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f29e00d0d60 0x7f29e014a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:18:32.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:32.982+0000 7f29e6400640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f29e0154d90 0x7f29e014aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:18:32.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:32.982+0000 7f29e6400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f29e015aa30 con 0x7f29e00d0d60 2024-04-24T23:18:32.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:32.982+0000 7f29e6400640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f29e015b600 con 0x7f29e00cb090 2024-04-24T23:18:32.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:32.982+0000 7f29e6400640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f29e00cc590 con 0x7f29e0154d90 2024-04-24T23:18:32.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:32.982+0000 7f29e5000640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f29e00cb090 0x7f29e014c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:18:32.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:32.982+0000 7f29dfe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f29e00d0d60 0x7f29e014a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:18:32.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:32.982+0000 7f29e5a00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f29e0154d90 0x7f29e014aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:18:32.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:32.982+0000 7f29e5000640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f29e00cb090 0x7f29e014c7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:60282/0 (socket says 172.21.15.6:60282) 2024-04-24T23:18:32.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:32.982+0000 7f29e5000640 1 -- 172.21.15.6:0/310100653 learned_addr learned my addr 172.21.15.6:0/310100653 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:18:32.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:32.983+0000 7f29e5a00640 1 -- 172.21.15.6:0/310100653 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f29e00cb090 msgr2=0x7f29e014c7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:18:32.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:32.983+0000 7f29e5a00640 1 --2- 172.21.15.6:0/310100653 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f29e00cb090 0x7f29e014c7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:32.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:32.983+0000 7f29e5a00640 1 -- 172.21.15.6:0/310100653 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f29e00d0d60 msgr2=0x7f29e014a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:18:32.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:32.983+0000 7f29e5a00640 1 --2- 172.21.15.6:0/310100653 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f29e00d0d60 0x7f29e014a550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:32.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:32.983+0000 7f29e5a00640 1 -- 172.21.15.6:0/310100653 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29e01f07a0 con 0x7f29e0154d90 2024-04-24T23:18:32.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:32.983+0000 7f29e5000640 1 --2- 172.21.15.6:0/310100653 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f29e00cb090 0x7f29e014c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:18:32.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:32.983+0000 7f29dfe00640 1 --2- 172.21.15.6:0/310100653 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f29e00d0d60 0x7f29e014a550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:18:32.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:32.983+0000 7f29e5a00640 1 --2- 172.21.15.6:0/310100653 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f29e0154d90 0x7f29e014aa90 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f29d800efc0 tx=0x7f29d800c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:18:32.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:32.983+0000 7f29dcc00640 1 -- 172.21.15.6:0/310100653 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29d8018020 con 0x7f29e0154d90 2024-04-24T23:18:32.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:32.983+0000 7f29dcc00640 1 -- 172.21.15.6:0/310100653 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f29d8014480 con 0x7f29e0154d90 2024-04-24T23:18:32.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:32.983+0000 7f29dcc00640 1 -- 172.21.15.6:0/310100653 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29d8014870 con 0x7f29e0154d90 2024-04-24T23:18:32.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:32.984+0000 7f29e6400640 1 -- 172.21.15.6:0/310100653 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29e01f0a40 con 0x7f29e0154d90 2024-04-24T23:18:32.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:32.984+0000 7f29e6400640 1 -- 172.21.15.6:0/310100653 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f29e01f1000 con 0x7f29e0154d90 2024-04-24T23:18:32.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:32.985+0000 7f29e6400640 1 -- 172.21.15.6:0/310100653 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f29e0067c20 con 0x7f29e0154d90 2024-04-24T23:18:32.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:32.986+0000 7f29dcc00640 1 -- 172.21.15.6:0/310100653 <== mon.2 v2:172.21.15.190:3301/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f29d8005d10 con 0x7f29e0154d90 2024-04-24T23:18:32.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:32.987+0000 7f29dcc00640 1 --2- 172.21.15.6:0/310100653 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f29c006be30 0x7f29c006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:18:32.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:32.987+0000 7f29dcc00640 1 -- 172.21.15.6:0/310100653 <== mon.2 v2:172.21.15.190:3301/0 5 ==== osd_map(27..27 src has 1..27) ==== 6676+0+0 (secure 0 0 0) 0x7f29d808bf90 con 0x7f29e0154d90 2024-04-24T23:18:32.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:32.988+0000 7f29e5000640 1 --2- 172.21.15.6:0/310100653 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f29c006be30 0x7f29c006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:18:32.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:32.989+0000 7f29e5000640 1 --2- 172.21.15.6:0/310100653 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f29c006be30 0x7f29c006e2f0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f29d4004e60 tx=0x7f29d4007f10 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:18:32.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:32.993+0000 7f29dcc00640 1 -- 172.21.15.6:0/310100653 <== mon.2 v2:172.21.15.190:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f29e0067c20 con 0x7f29e0154d90 2024-04-24T23:18:33.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:33.237+0000 7f29e6400640 1 -- 172.21.15.6:0/310100653 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f29e0075b30 con 0x7f29e0154d90 2024-04-24T23:18:33.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:33.239+0000 7f29dcc00640 1 -- 172.21.15.6:0/310100653 <== mon.2 v2:172.21.15.190:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+6079 (secure 0 0 0) 0x7f29d8022030 con 0x7f29e0154d90 2024-04-24T23:18:33.241 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:18:33.241 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:18:32.573171+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4282,"mds_2":4235,"mds_3":4232,"mds_4":4197},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4197":{"gid":4197,"name":"a","rank":4,"incarnation":19,"state":"up:resolve","state_seq":47,"addr":"172.21.15.6:6834/105267896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":105267896},{"type":"v1","addr":"172.21.15.6:6834","nonce":105267896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4232":{"gid":4232,"name":"b","rank":3,"incarnation":15,"state":"up:active","state_seq":15,"addr":"172.21.15.190:6838/2032280158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2032280158},{"type":"v1","addr":"172.21.15.190:6838","nonce":2032280158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4235":{"gid":4235,"name":"d","rank":2,"incarnation":13,"state":"up:active","state_seq":14,"addr":"172.21.15.190:6839/1572872451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":1572872451},{"type":"v1","addr":"172.21.15.190:6839","nonce":1572872451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4282":{"gid":4282,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":12,"addr":"172.21.15.190:6837/580315269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":580315269},{"type":"v1","addr":"172.21.15.190:6837","nonce":580315269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/489092393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":489092393},{"type":"v1","addr":"172.21.15.6:6835","nonce":489092393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4306,"qdb_cluster":[4306,4282,4235,4232]},"id":1}]} 2024-04-24T23:18:33.241 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 20 2024-04-24T23:18:33.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:33.241+0000 7f29e6400640 1 -- 172.21.15.6:0/310100653 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f29c006be30 msgr2=0x7f29c006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:18:33.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:33.241+0000 7f29e6400640 1 --2- 172.21.15.6:0/310100653 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f29c006be30 0x7f29c006e2f0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f29d4004e60 tx=0x7f29d4007f10 comp rx=0 tx=0).stop 2024-04-24T23:18:33.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:33.241+0000 7f29e6400640 1 -- 172.21.15.6:0/310100653 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f29e0154d90 msgr2=0x7f29e014aa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:18:33.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:33.241+0000 7f29e6400640 1 --2- 172.21.15.6:0/310100653 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f29e0154d90 0x7f29e014aa90 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f29d800efc0 tx=0x7f29d800c490 comp rx=0 tx=0).stop 2024-04-24T23:18:33.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:33.242+0000 7f29e6400640 1 -- 172.21.15.6:0/310100653 shutdown_connections 2024-04-24T23:18:33.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:33.242+0000 7f29e6400640 1 --2- 172.21.15.6:0/310100653 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f29c006be30 0x7f29c006e2f0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:33.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:33.242+0000 7f29e6400640 1 --2- 172.21.15.6:0/310100653 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f29e0154d90 0x7f29e014aa90 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:33.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:33.242+0000 7f29e6400640 1 --2- 172.21.15.6:0/310100653 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f29e00d0d60 0x7f29e014a550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:33.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:33.242+0000 7f29e6400640 1 --2- 172.21.15.6:0/310100653 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f29e00cb090 0x7f29e014c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:33.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:33.242+0000 7f29e6400640 1 -- 172.21.15.6:0/310100653 >> 172.21.15.6:0/310100653 conn(0x7f29e006be10 msgr2=0x7f29e00cbe80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:18:33.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:33.242+0000 7f29e6400640 1 -- 172.21.15.6:0/310100653 shutdown_connections 2024-04-24T23:18:33.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:33.242+0000 7f29e6400640 1 -- 172.21.15.6:0/310100653 wait complete. 2024-04-24T23:18:33.264 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-24T23:18:33.363 INFO:tasks.workunit.client.0.smithi006.stdout:checking for string.h... yes 2024-04-24T23:18:33.433 INFO:tasks.workunit.client.0.smithi006.stdout:checking for memory.h... yes 2024-04-24T23:18:33.513 INFO:tasks.workunit.client.0.smithi006.stdout:checking for strings.h... yes 2024-04-24T23:18:33.600 INFO:tasks.workunit.client.0.smithi006.stdout:checking for inttypes.h... yes 2024-04-24T23:18:33.691 INFO:tasks.workunit.client.0.smithi006.stdout:checking for stdint.h... yes 2024-04-24T23:18:33.791 INFO:tasks.workunit.client.0.smithi006.stdout:checking for unistd.h... yes 2024-04-24T23:18:33.891 INFO:tasks.workunit.client.0.smithi006.stdout:checking minix/config.h usability... no 2024-04-24T23:18:33.955 INFO:tasks.workunit.client.0.smithi006.stdout:checking minix/config.h presence... no 2024-04-24T23:18:33.955 INFO:tasks.workunit.client.0.smithi006.stdout:checking for minix/config.h... no 2024-04-24T23:18:34.179 INFO:tasks.workunit.client.0.smithi006.stdout:checking whether it is safe to define __EXTENSIONS__... yes 2024-04-24T23:18:34.180 INFO:tasks.workunit.client.0.smithi006.stdout:checking for gcc... (cached) gcc 2024-04-24T23:18:34.271 INFO:tasks.workunit.client.0.smithi006.stdout:checking whether we are using the GNU C compiler... (cached) yes 2024-04-24T23:18:34.271 INFO:tasks.workunit.client.0.smithi006.stdout:checking whether gcc accepts -g... (cached) yes 2024-04-24T23:18:34.271 INFO:tasks.workunit.client.0.smithi006.stdout:checking for gcc option to accept ISO C89... (cached) none needed 2024-04-24T23:18:34.272 INFO:tasks.workunit.client.0.smithi006.stdout:checking dependency style of gcc... (cached) gcc3 2024-04-24T23:18:34.381 INFO:tasks.workunit.client.0.smithi006.stdout:checking for main in -lm... yes 2024-04-24T23:18:34.478 INFO:tasks.workunit.client.0.smithi006.stdout:checking for main in -lpthread... yes 2024-04-24T23:18:34.480 INFO:tasks.workunit.client.0.smithi006.stdout:checking for ANSI C header files... (cached) yes 2024-04-24T23:18:34.553 INFO:tasks.workunit.client.0.smithi006.stdout:checking for sys/wait.h that is POSIX.1 compatible... yes 2024-04-24T23:18:34.726 INFO:tasks.workunit.client.0.smithi006.stdout:checking pthread.h usability... yes 2024-04-24T23:18:34.789 INFO:tasks.workunit.client.0.smithi006.stdout:checking pthread.h presence... yes 2024-04-24T23:18:34.789 INFO:tasks.workunit.client.0.smithi006.stdout:checking for pthread.h... yes 2024-04-24T23:18:34.906 INFO:tasks.workunit.client.0.smithi006.stdout:checking fcntl.h usability... yes 2024-04-24T23:18:34.960 INFO:tasks.workunit.client.0.smithi006.stdout:checking fcntl.h presence... yes 2024-04-24T23:18:34.960 INFO:tasks.workunit.client.0.smithi006.stdout:checking for fcntl.h... yes 2024-04-24T23:18:35.070 INFO:tasks.workunit.client.0.smithi006.stdout:checking limits.h usability... yes 2024-04-24T23:18:35.113 INFO:tasks.workunit.client.0.smithi006.stdout:checking limits.h presence... yes 2024-04-24T23:18:35.114 INFO:tasks.workunit.client.0.smithi006.stdout:checking for limits.h... yes 2024-04-24T23:18:35.148 INFO:tasks.workunit.client.0.smithi006.stdout:checking for stdint.h... (cached) yes 2024-04-24T23:18:35.241 INFO:tasks.workunit.client.0.smithi006.stdout:checking sys/time.h usability... yes 2024-04-24T23:18:35.265 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:18:35.283 INFO:tasks.workunit.client.0.smithi006.stdout:checking sys/time.h presence... yes 2024-04-24T23:18:35.283 INFO:tasks.workunit.client.0.smithi006.stdout:checking for sys/time.h... yes 2024-04-24T23:18:35.308 INFO:tasks.workunit.client.0.smithi006.stdout:checking for unistd.h... (cached) yes 2024-04-24T23:18:35.383 INFO:tasks.workunit.client.0.smithi006.stdout:checking sys/vfs.h usability... yes 2024-04-24T23:18:35.417 INFO:tasks.workunit.client.0.smithi006.stdout:checking sys/vfs.h presence... yes 2024-04-24T23:18:35.417 INFO:tasks.workunit.client.0.smithi006.stdout:checking for sys/vfs.h... yes 2024-04-24T23:18:35.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:35.451+0000 7f39a1000640 1 Processor -- start 2024-04-24T23:18:35.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:35.451+0000 7f39a1000640 1 -- start start 2024-04-24T23:18:35.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:35.453+0000 7f39a1000640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f399c152bc0 0x7f399c159390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:18:35.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:35.453+0000 7f39a1000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f399c0c9330 0x7f399c15c610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:18:35.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:35.453+0000 7f39a1000640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f399c0c6e70 con 0x7f399c0c8f50 2024-04-24T23:18:35.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:35.453+0000 7f39a1000640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f399c0c73d0 con 0x7f399c0c9330 2024-04-24T23:18:35.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:35.453+0000 7f39a1000640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f399c0c8ac0 con 0x7f399c152bc0 2024-04-24T23:18:35.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:35.453+0000 7f399a800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f399c0c9330 0x7f399c15c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:18:35.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:35.453+0000 7f3999e00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f399c152bc0 0x7f399c159390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:18:35.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:35.453+0000 7f3999400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f399c0c8f50 0x7f399c15bf00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:47176/0 (socket says 172.21.15.6:47176) 2024-04-24T23:18:35.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:35.453+0000 7f3999400640 1 -- 172.21.15.6:0/1041731974 learned_addr learned my addr 172.21.15.6:0/1041731974 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:18:35.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:35.454+0000 7f399a800640 1 -- 172.21.15.6:0/1041731974 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f399c152bc0 msgr2=0x7f399c159390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:18:35.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:35.454+0000 7f399a800640 1 --2- 172.21.15.6:0/1041731974 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f399c152bc0 0x7f399c159390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:35.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:35.454+0000 7f399a800640 1 -- 172.21.15.6:0/1041731974 >> v1:172.21.15.6:6789/0 conn(0x7f399c0c8f50 legacy=0x7f399c15bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:18:35.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:35.454+0000 7f399a800640 1 -- 172.21.15.6:0/1041731974 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f399c1598d0 con 0x7f399c0c9330 2024-04-24T23:18:35.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:35.454+0000 7f3999e00640 1 --2- 172.21.15.6:0/1041731974 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f399c152bc0 0x7f399c159390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:18:35.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:35.454+0000 7f3998a00640 1 -- 172.21.15.6:0/1041731974 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1553774898 0 0) 0x7f399c0c6e70 con 0x7f399c0c8f50 2024-04-24T23:18:35.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:35.455+0000 7f399a800640 1 --2- 172.21.15.6:0/1041731974 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f399c0c9330 0x7f399c15c610 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f3994009d90 tx=0x7f399402a5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb3718c4af9d79b5 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:18:35.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:35.455+0000 7f3998a00640 1 -- 172.21.15.6:0/1041731974 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3994031600 con 0x7f399c0c9330 2024-04-24T23:18:35.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:35.455+0000 7f3998a00640 1 -- 172.21.15.6:0/1041731974 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f39940317a0 con 0x7f399c0c9330 2024-04-24T23:18:35.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:35.455+0000 7f3998a00640 1 -- 172.21.15.6:0/1041731974 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3994031b70 con 0x7f399c0c9330 2024-04-24T23:18:35.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:35.456+0000 7f39a1000640 1 -- 172.21.15.6:0/1041731974 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f399c0c9330 msgr2=0x7f399c15c610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:18:35.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:35.456+0000 7f39a1000640 1 --2- 172.21.15.6:0/1041731974 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f399c0c9330 0x7f399c15c610 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f3994009d90 tx=0x7f399402a5d0 comp rx=0 tx=0).stop 2024-04-24T23:18:35.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:35.456+0000 7f39a1000640 1 -- 172.21.15.6:0/1041731974 shutdown_connections 2024-04-24T23:18:35.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:35.456+0000 7f39a1000640 1 --2- 172.21.15.6:0/1041731974 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f399c0c9330 0x7f399c15c610 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:35.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:35.456+0000 7f39a1000640 1 --2- 172.21.15.6:0/1041731974 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f399c152bc0 0x7f399c159390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:35.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:35.456+0000 7f39a1000640 1 -- 172.21.15.6:0/1041731974 >> 172.21.15.6:0/1041731974 conn(0x7f399c06b0e0 msgr2=0x7f399c06b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:18:35.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:35.456+0000 7f39a1000640 1 -- 172.21.15.6:0/1041731974 shutdown_connections 2024-04-24T23:18:35.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:35.456+0000 7f39a1000640 1 -- 172.21.15.6:0/1041731974 wait complete. 2024-04-24T23:18:35.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:35.458+0000 7f39a1000640 1 Processor -- start 2024-04-24T23:18:35.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:35.458+0000 7f39a1000640 1 -- start start 2024-04-24T23:18:35.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:35.459+0000 7f39a1000640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f399c0c8f50 0x7f399c14f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:18:35.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:35.459+0000 7f39a1000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f399c0c9330 0x7f399c14fb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:18:35.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:35.459+0000 7f39a1000640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f399c152bc0 0x7f399c15a610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:18:35.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:35.459+0000 7f39a1000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f399c0c8470 con 0x7f399c0c9330 2024-04-24T23:18:35.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:35.459+0000 7f39a1000640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f399c0c6700 con 0x7f399c0c8f50 2024-04-24T23:18:35.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:35.459+0000 7f39a1000640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f399c0c8750 con 0x7f399c152bc0 2024-04-24T23:18:35.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:35.459+0000 7f3999400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f399c0c9330 0x7f399c14fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:18:35.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:35.459+0000 7f3999400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f399c0c9330 0x7f399c14fb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45844/0 (socket says 172.21.15.6:45844) 2024-04-24T23:18:35.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:35.459+0000 7f399a800640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f399c152bc0 0x7f399c15a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:18:35.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:35.459+0000 7f3999400640 1 -- 172.21.15.6:0/2886061985 learned_addr learned my addr 172.21.15.6:0/2886061985 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:18:35.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:35.459+0000 7f3999e00640 1 --2- 172.21.15.6:0/2886061985 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f399c0c8f50 0x7f399c14f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:18:35.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:35.460+0000 7f399a800640 1 -- 172.21.15.6:0/2886061985 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f399c0c8f50 msgr2=0x7f399c14f5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:18:35.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:35.460+0000 7f399a800640 1 --2- 172.21.15.6:0/2886061985 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f399c0c8f50 0x7f399c14f5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:35.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:35.460+0000 7f399a800640 1 -- 172.21.15.6:0/2886061985 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f399c0c9330 msgr2=0x7f399c14fb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:18:35.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:35.460+0000 7f399a800640 1 --2- 172.21.15.6:0/2886061985 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f399c0c9330 0x7f399c14fb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:35.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:35.460+0000 7f399a800640 1 -- 172.21.15.6:0/2886061985 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f399c159b70 con 0x7f399c152bc0 2024-04-24T23:18:35.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:35.460+0000 7f3999400640 1 --2- 172.21.15.6:0/2886061985 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f399c0c9330 0x7f399c14fb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:18:35.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:35.460+0000 7f399a800640 1 --2- 172.21.15.6:0/2886061985 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f399c152bc0 0x7f399c15a610 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f399c06c0c0 tx=0x7f3994004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:18:35.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:35.461+0000 7f3992000640 1 -- 172.21.15.6:0/2886061985 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3994009450 con 0x7f399c152bc0 2024-04-24T23:18:35.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:35.461+0000 7f3992000640 1 -- 172.21.15.6:0/2886061985 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3994030ba0 con 0x7f399c152bc0 2024-04-24T23:18:35.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:35.461+0000 7f3992000640 1 -- 172.21.15.6:0/2886061985 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f399403c420 con 0x7f399c152bc0 2024-04-24T23:18:35.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:35.461+0000 7f39a1000640 1 -- 172.21.15.6:0/2886061985 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f399c1eef10 con 0x7f399c152bc0 2024-04-24T23:18:35.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:35.461+0000 7f39a1000640 1 -- 172.21.15.6:0/2886061985 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f399c15a0f0 con 0x7f399c152bc0 2024-04-24T23:18:35.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:35.462+0000 7f39a1000640 1 -- 172.21.15.6:0/2886061985 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f399c0767b0 con 0x7f399c152bc0 2024-04-24T23:18:35.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:35.463+0000 7f3992000640 1 -- 172.21.15.6:0/2886061985 <== mon.2 v2:172.21.15.190:3301/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f3994039040 con 0x7f399c152bc0 2024-04-24T23:18:35.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:35.464+0000 7f3992000640 1 --2- 172.21.15.6:0/2886061985 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f397c06be80 0x7f397c06e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:18:35.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:35.464+0000 7f3992000640 1 -- 172.21.15.6:0/2886061985 <== mon.2 v2:172.21.15.190:3301/0 5 ==== osd_map(27..27 src has 1..27) ==== 6676+0+0 (secure 0 0 0) 0x7f39940ac6e0 con 0x7f399c152bc0 2024-04-24T23:18:35.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:35.465+0000 7f3999e00640 1 --2- 172.21.15.6:0/2886061985 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f397c06be80 0x7f397c06e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:18:35.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:35.466+0000 7f3999e00640 1 --2- 172.21.15.6:0/2886061985 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f397c06be80 0x7f397c06e340 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f398c009560 tx=0x7f398c009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:18:35.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:35.469+0000 7f3992000640 1 -- 172.21.15.6:0/2886061985 <== mon.2 v2:172.21.15.190:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f3994040030 con 0x7f399c152bc0 2024-04-24T23:18:35.518 INFO:tasks.workunit.client.0.smithi006.stdout:checking sys/limits.h usability... no 2024-04-24T23:18:35.580 INFO:tasks.workunit.client.0.smithi006.stdout:checking sys/limits.h presence... no 2024-04-24T23:18:35.580 INFO:tasks.workunit.client.0.smithi006.stdout:checking for sys/limits.h... no 2024-04-24T23:18:35.646 INFO:tasks.workunit.client.0.smithi006.stdout:checking for an ANSI C-conforming const... yes 2024-04-24T23:18:35.708 INFO:tasks.workunit.client.0.smithi006.stdout:checking whether time.h and sys/time.h may both be included... yes 2024-04-24T23:18:35.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:35.747+0000 7f39a1000640 1 -- 172.21.15.6:0/2886061985 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f399c0c75e0 con 0x7f399c152bc0 2024-04-24T23:18:35.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:35.748+0000 7f3992000640 1 -- 172.21.15.6:0/2886061985 <== mon.2 v2:172.21.15.190:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+6079 (secure 0 0 0) 0x7f399407a5c0 con 0x7f399c152bc0 2024-04-24T23:18:35.751 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:18:35.751 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:18:32.573171+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4282,"mds_2":4235,"mds_3":4232,"mds_4":4197},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4197":{"gid":4197,"name":"a","rank":4,"incarnation":19,"state":"up:resolve","state_seq":47,"addr":"172.21.15.6:6834/105267896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":105267896},{"type":"v1","addr":"172.21.15.6:6834","nonce":105267896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4232":{"gid":4232,"name":"b","rank":3,"incarnation":15,"state":"up:active","state_seq":15,"addr":"172.21.15.190:6838/2032280158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2032280158},{"type":"v1","addr":"172.21.15.190:6838","nonce":2032280158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4235":{"gid":4235,"name":"d","rank":2,"incarnation":13,"state":"up:active","state_seq":14,"addr":"172.21.15.190:6839/1572872451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":1572872451},{"type":"v1","addr":"172.21.15.190:6839","nonce":1572872451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4282":{"gid":4282,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":12,"addr":"172.21.15.190:6837/580315269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":580315269},{"type":"v1","addr":"172.21.15.190:6837","nonce":580315269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/489092393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":489092393},{"type":"v1","addr":"172.21.15.6:6835","nonce":489092393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4306,"qdb_cluster":[4306,4282,4235,4232]},"id":1}]} 2024-04-24T23:18:35.751 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 20 2024-04-24T23:18:35.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:35.751+0000 7f39a1000640 1 -- 172.21.15.6:0/2886061985 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f397c06be80 msgr2=0x7f397c06e340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:18:35.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:35.751+0000 7f39a1000640 1 --2- 172.21.15.6:0/2886061985 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f397c06be80 0x7f397c06e340 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f398c009560 tx=0x7f398c009190 comp rx=0 tx=0).stop 2024-04-24T23:18:35.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:35.751+0000 7f39a1000640 1 -- 172.21.15.6:0/2886061985 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f399c152bc0 msgr2=0x7f399c15a610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:18:35.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:35.751+0000 7f39a1000640 1 --2- 172.21.15.6:0/2886061985 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f399c152bc0 0x7f399c15a610 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f399c06c0c0 tx=0x7f3994004a40 comp rx=0 tx=0).stop 2024-04-24T23:18:35.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:35.751+0000 7f39a1000640 1 -- 172.21.15.6:0/2886061985 shutdown_connections 2024-04-24T23:18:35.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:35.751+0000 7f39a1000640 1 --2- 172.21.15.6:0/2886061985 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f397c06be80 0x7f397c06e340 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:35.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:35.751+0000 7f39a1000640 1 --2- 172.21.15.6:0/2886061985 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f399c152bc0 0x7f399c15a610 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:35.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:35.751+0000 7f39a1000640 1 --2- 172.21.15.6:0/2886061985 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f399c0c9330 0x7f399c14fb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:35.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:35.751+0000 7f39a1000640 1 --2- 172.21.15.6:0/2886061985 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f399c0c8f50 0x7f399c14f5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:35.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:35.751+0000 7f39a1000640 1 -- 172.21.15.6:0/2886061985 >> 172.21.15.6:0/2886061985 conn(0x7f399c06b0e0 msgr2=0x7f399c06e130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:18:35.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:35.751+0000 7f39a1000640 1 -- 172.21.15.6:0/2886061985 shutdown_connections 2024-04-24T23:18:35.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:35.751+0000 7f39a1000640 1 -- 172.21.15.6:0/2886061985 wait complete. 2024-04-24T23:18:35.771 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-24T23:18:35.827 INFO:tasks.workunit.client.0.smithi006.stdout:checking for system... yes 2024-04-24T23:18:35.955 INFO:tasks.workunit.client.0.smithi006.stdout:checking for gettimeofday... yes 2024-04-24T23:18:36.081 INFO:tasks.workunit.client.0.smithi006.stdout:checking for mkdir... yes 2024-04-24T23:18:36.474 INFO:tasks.workunit.client.0.smithi006.stdout:checking for strerror... yes 2024-04-24T23:18:36.606 INFO:tasks.workunit.client.0.smithi006.stdout:checking for open64... yes 2024-04-24T23:18:36.986 INFO:tasks.workunit.client.0.smithi006.stdout:checking for stat64... yes 2024-04-24T23:18:37.772 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:18:37.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:37.989+0000 7fe4cf200640 1 Processor -- start 2024-04-24T23:18:37.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:37.989+0000 7fe4cf200640 1 -- start start 2024-04-24T23:18:37.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:37.991+0000 7fe4cf200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fe4c8152b30 0x7fe4c806f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:18:37.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:37.991+0000 7fe4cf200640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe4c80c50b0 0x7fe4c806fb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:18:37.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:37.991+0000 7fe4cf200640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe4c80ca4e0 con 0x7fe4c80c9ff0 2024-04-24T23:18:37.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:37.991+0000 7fe4cf200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7fe4c80c3bd0 con 0x7fe4c8152b30 2024-04-24T23:18:37.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:37.991+0000 7fe4cf200640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fe4c80c3740 con 0x7fe4c80c50b0 2024-04-24T23:18:37.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:37.992+0000 7fe4ce800640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe4c80c50b0 0x7fe4c806fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:18:37.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:37.992+0000 7fe4cd400640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7fe4c80c9ff0 0x7fe4c80caaa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.6:54456/0 (socket says 172.21.15.6:54456) 2024-04-24T23:18:37.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:37.992+0000 7fe4cd400640 1 -- 172.21.15.6:0/1879901895 learned_addr learned my addr 172.21.15.6:0/1879901895 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:18:37.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:37.992+0000 7fe4cde00640 1 --2- 172.21.15.6:0/1879901895 >> v2:172.21.15.6:3300/0 conn(0x7fe4c8152b30 0x7fe4c806f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:18:37.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:37.992+0000 7fe4ce800640 1 -- 172.21.15.6:0/1879901895 >> v2:172.21.15.6:3300/0 conn(0x7fe4c8152b30 msgr2=0x7fe4c806f5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:18:37.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:37.992+0000 7fe4ce800640 1 --2- 172.21.15.6:0/1879901895 >> v2:172.21.15.6:3300/0 conn(0x7fe4c8152b30 0x7fe4c806f5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:37.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:37.992+0000 7fe4ce800640 1 -- 172.21.15.6:0/1879901895 >> v1:172.21.15.190:6789/0 conn(0x7fe4c80c9ff0 legacy=0x7fe4c80caaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:18:37.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:37.992+0000 7fe4ce800640 1 -- 172.21.15.6:0/1879901895 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4c80ca7c0 con 0x7fe4c80c50b0 2024-04-24T23:18:37.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:37.992+0000 7fe4cca00640 1 -- 172.21.15.6:0/1879901895 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3459675295 0 0) 0x7fe4c80ca4e0 con 0x7fe4c80c9ff0 2024-04-24T23:18:37.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:37.993+0000 7fe4ce800640 1 --2- 172.21.15.6:0/1879901895 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe4c80c50b0 0x7fe4c806fb30 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fe4c4009d90 tx=0x7fe4c402a6f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e419e39e2ef961c8 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:18:37.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:37.994+0000 7fe4cca00640 1 -- 172.21.15.6:0/1879901895 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4c4007740 con 0x7fe4c80c50b0 2024-04-24T23:18:37.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:37.994+0000 7fe4cca00640 1 -- 172.21.15.6:0/1879901895 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe4c40078e0 con 0x7fe4c80c50b0 2024-04-24T23:18:37.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:37.994+0000 7fe4cca00640 1 -- 172.21.15.6:0/1879901895 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4c4007cb0 con 0x7fe4c80c50b0 2024-04-24T23:18:37.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:37.994+0000 7fe4cf200640 1 -- 172.21.15.6:0/1879901895 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe4c80c50b0 msgr2=0x7fe4c806fb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:18:37.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:37.994+0000 7fe4cf200640 1 --2- 172.21.15.6:0/1879901895 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe4c80c50b0 0x7fe4c806fb30 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fe4c4009d90 tx=0x7fe4c402a6f0 comp rx=0 tx=0).stop 2024-04-24T23:18:37.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:37.994+0000 7fe4cf200640 1 -- 172.21.15.6:0/1879901895 shutdown_connections 2024-04-24T23:18:37.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:37.994+0000 7fe4cf200640 1 --2- 172.21.15.6:0/1879901895 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe4c80c50b0 0x7fe4c806fb30 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:37.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:37.994+0000 7fe4cf200640 1 --2- 172.21.15.6:0/1879901895 >> v2:172.21.15.6:3300/0 conn(0x7fe4c8152b30 0x7fe4c806f5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:37.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:37.994+0000 7fe4cf200640 1 -- 172.21.15.6:0/1879901895 >> 172.21.15.6:0/1879901895 conn(0x7fe4c806b500 msgr2=0x7fe4c8066ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:18:37.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:37.995+0000 7fe4cf200640 1 -- 172.21.15.6:0/1879901895 shutdown_connections 2024-04-24T23:18:37.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:37.995+0000 7fe4cf200640 1 -- 172.21.15.6:0/1879901895 wait complete. 2024-04-24T23:18:37.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:37.997+0000 7fe4cf200640 1 Processor -- start 2024-04-24T23:18:37.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:37.997+0000 7fe4cf200640 1 -- start start 2024-04-24T23:18:38.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:37.998+0000 7fe4cf200640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe4c80c50b0 0x7fe4c81580c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:18:38.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:37.998+0000 7fe4cf200640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe4c80c9ff0 0x7fe4c80b2870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:18:38.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:37.998+0000 7fe4cf200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe4c8152b30 0x7fe4c81559d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:18:38.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:37.998+0000 7fe4cf200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fe4c80c90e0 con 0x7fe4c8152b30 2024-04-24T23:18:38.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:37.998+0000 7fe4cf200640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fe4c8152f10 con 0x7fe4c80c9ff0 2024-04-24T23:18:38.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:37.998+0000 7fe4cf200640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fe4c80c8d10 con 0x7fe4c80c50b0 2024-04-24T23:18:38.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:37.998+0000 7fe4cde00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe4c80c50b0 0x7fe4c81580c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:18:38.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:37.999+0000 7fe4cde00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe4c80c50b0 0x7fe4c81580c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.6:36422/0 (socket says 172.21.15.6:36422) 2024-04-24T23:18:38.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:37.999+0000 7fe4cd400640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe4c80c9ff0 0x7fe4c80b2870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:18:38.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:37.999+0000 7fe4cde00640 1 -- 172.21.15.6:0/3195571673 learned_addr learned my addr 172.21.15.6:0/3195571673 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:18:38.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:37.999+0000 7fe4ce800640 1 --2- 172.21.15.6:0/3195571673 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe4c8152b30 0x7fe4c81559d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:18:38.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:37.999+0000 7fe4cde00640 1 -- 172.21.15.6:0/3195571673 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe4c80c9ff0 msgr2=0x7fe4c80b2870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:18:38.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:37.999+0000 7fe4cde00640 1 --2- 172.21.15.6:0/3195571673 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe4c80c9ff0 0x7fe4c80b2870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:38.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:37.999+0000 7fe4cde00640 1 -- 172.21.15.6:0/3195571673 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe4c8152b30 msgr2=0x7fe4c81559d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:18:38.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:37.999+0000 7fe4cde00640 1 --2- 172.21.15.6:0/3195571673 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe4c8152b30 0x7fe4c81559d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:38.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:37.999+0000 7fe4cde00640 1 -- 172.21.15.6:0/3195571673 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4c81595f0 con 0x7fe4c80c50b0 2024-04-24T23:18:38.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:37.999+0000 7fe4ce800640 1 --2- 172.21.15.6:0/3195571673 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe4c8152b30 0x7fe4c81559d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:18:38.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:37.999+0000 7fe4cde00640 1 --2- 172.21.15.6:0/3195571673 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe4c80c50b0 0x7fe4c81580c0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fe4bc00e8a0 tx=0x7fe4bc00ed60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:18:38.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:37.999+0000 7fe4c2000640 1 -- 172.21.15.6:0/3195571673 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4bc017070 con 0x7fe4c80c50b0 2024-04-24T23:18:38.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:38.000+0000 7fe4c2000640 1 -- 172.21.15.6:0/3195571673 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe4bc00c940 con 0x7fe4c80c50b0 2024-04-24T23:18:38.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:38.000+0000 7fe4c2000640 1 -- 172.21.15.6:0/3195571673 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4bc00cd30 con 0x7fe4c80c50b0 2024-04-24T23:18:38.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:38.000+0000 7fe4cf200640 1 -- 172.21.15.6:0/3195571673 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4c8158b30 con 0x7fe4c80c50b0 2024-04-24T23:18:38.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:38.000+0000 7fe4cf200640 1 -- 172.21.15.6:0/3195571673 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4c8157370 con 0x7fe4c80c50b0 2024-04-24T23:18:38.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:38.001+0000 7fe4cf200640 1 -- 172.21.15.6:0/3195571673 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe4c8075b30 con 0x7fe4c80c50b0 2024-04-24T23:18:38.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:38.002+0000 7fe4c2000640 1 -- 172.21.15.6:0/3195571673 <== mon.2 v2:172.21.15.190:3301/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fe4bc01a070 con 0x7fe4c80c50b0 2024-04-24T23:18:38.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:38.003+0000 7fe4c2000640 1 --2- 172.21.15.6:0/3195571673 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fe4ac06be60 0x7fe4ac06e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:18:38.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:38.003+0000 7fe4c2000640 1 -- 172.21.15.6:0/3195571673 <== mon.2 v2:172.21.15.190:3301/0 5 ==== osd_map(27..27 src has 1..27) ==== 6676+0+0 (secure 0 0 0) 0x7fe4bc08b120 con 0x7fe4c80c50b0 2024-04-24T23:18:38.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:38.003+0000 7fe4cd400640 1 --2- 172.21.15.6:0/3195571673 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fe4ac06be60 0x7fe4ac06e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:18:38.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:38.004+0000 7fe4cd400640 1 --2- 172.21.15.6:0/3195571673 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fe4ac06be60 0x7fe4ac06e320 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fe4c80ca5f0 tx=0x7fe4b8007d50 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:18:38.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:38.008+0000 7fe4c2000640 1 -- 172.21.15.6:0/3195571673 <== mon.2 v2:172.21.15.190:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fe4bc058f80 con 0x7fe4c80c50b0 2024-04-24T23:18:38.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:38.257+0000 7fe4cf200640 1 -- 172.21.15.6:0/3195571673 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe4c8070180 con 0x7fe4c80c50b0 2024-04-24T23:18:38.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:38.258+0000 7fe4c2000640 1 -- 172.21.15.6:0/3195571673 <== mon.2 v2:172.21.15.190:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+6081 (secure 0 0 0) 0x7fe4bc05d280 con 0x7fe4c80c50b0 2024-04-24T23:18:38.260 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:18:38.260 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":21,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:18:37.773499+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4282,"mds_2":4235,"mds_3":4232,"mds_4":4197},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4197":{"gid":4197,"name":"a","rank":4,"incarnation":19,"state":"up:reconnect","state_seq":49,"addr":"172.21.15.6:6834/105267896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":105267896},{"type":"v1","addr":"172.21.15.6:6834","nonce":105267896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4232":{"gid":4232,"name":"b","rank":3,"incarnation":15,"state":"up:active","state_seq":15,"addr":"172.21.15.190:6838/2032280158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2032280158},{"type":"v1","addr":"172.21.15.190:6838","nonce":2032280158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4235":{"gid":4235,"name":"d","rank":2,"incarnation":13,"state":"up:active","state_seq":14,"addr":"172.21.15.190:6839/1572872451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":1572872451},{"type":"v1","addr":"172.21.15.190:6839","nonce":1572872451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4282":{"gid":4282,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":12,"addr":"172.21.15.190:6837/580315269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":580315269},{"type":"v1","addr":"172.21.15.190:6837","nonce":580315269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/489092393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":489092393},{"type":"v1","addr":"172.21.15.6:6835","nonce":489092393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4306,"qdb_cluster":[4232,4235,4282,4306]},"id":1}]} 2024-04-24T23:18:38.261 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 21 2024-04-24T23:18:38.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:38.261+0000 7fe4cf200640 1 -- 172.21.15.6:0/3195571673 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fe4ac06be60 msgr2=0x7fe4ac06e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:18:38.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:38.261+0000 7fe4cf200640 1 --2- 172.21.15.6:0/3195571673 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fe4ac06be60 0x7fe4ac06e320 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fe4c80ca5f0 tx=0x7fe4b8007d50 comp rx=0 tx=0).stop 2024-04-24T23:18:38.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:38.261+0000 7fe4cf200640 1 -- 172.21.15.6:0/3195571673 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe4c80c50b0 msgr2=0x7fe4c81580c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:18:38.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:38.261+0000 7fe4cf200640 1 --2- 172.21.15.6:0/3195571673 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe4c80c50b0 0x7fe4c81580c0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fe4bc00e8a0 tx=0x7fe4bc00ed60 comp rx=0 tx=0).stop 2024-04-24T23:18:38.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:38.261+0000 7fe4cf200640 1 -- 172.21.15.6:0/3195571673 shutdown_connections 2024-04-24T23:18:38.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:38.261+0000 7fe4cf200640 1 --2- 172.21.15.6:0/3195571673 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fe4ac06be60 0x7fe4ac06e320 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:38.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:38.261+0000 7fe4cf200640 1 --2- 172.21.15.6:0/3195571673 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe4c8152b30 0x7fe4c81559d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:38.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:38.261+0000 7fe4cf200640 1 --2- 172.21.15.6:0/3195571673 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe4c80c9ff0 0x7fe4c80b2870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:38.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:38.261+0000 7fe4cf200640 1 --2- 172.21.15.6:0/3195571673 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe4c80c50b0 0x7fe4c81580c0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:38.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:38.261+0000 7fe4cf200640 1 -- 172.21.15.6:0/3195571673 >> 172.21.15.6:0/3195571673 conn(0x7fe4c806b500 msgr2=0x7fe4c80cc110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:18:38.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:38.261+0000 7fe4cf200640 1 -- 172.21.15.6:0/3195571673 shutdown_connections 2024-04-24T23:18:38.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:38.261+0000 7fe4cf200640 1 -- 172.21.15.6:0/3195571673 wait complete. 2024-04-24T23:18:38.283 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-24T23:18:38.332 INFO:tasks.workunit.client.0.smithi006.stdout:checking for fseeko64... yes 2024-04-24T23:18:38.902 INFO:tasks.workunit.client.0.smithi006.stdout:checking for lrand48_r... yes 2024-04-24T23:18:39.297 INFO:tasks.workunit.client.0.smithi006.stdout:checking for srand48_r... yes 2024-04-24T23:18:39.353 INFO:tasks.workunit.client.0.smithi006.stdout:configure: creating ./config.status 2024-04-24T23:18:39.816 INFO:tasks.workunit.client.0.smithi006.stdout: /bin/sh ./config.status 2024-04-24T23:18:39.955 INFO:tasks.workunit.client.0.smithi006.stdout:config.status: creating Makefile 2024-04-24T23:18:39.991 INFO:tasks.workunit.client.0.smithi006.stdout:config.status: creating config.h 2024-04-24T23:18:40.043 INFO:tasks.workunit.client.0.smithi006.stdout:config.status: executing depfiles commands 2024-04-24T23:18:40.083 INFO:tasks.workunit.client.0.smithi006.stdout:config.status: executing default commands 2024-04-24T23:18:40.132 INFO:tasks.workunit.client.0.smithi006.stdout:cd . && /bin/sh /home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing --run autoheader 2024-04-24T23:18:40.284 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:18:40.399 INFO:tasks.workunit.client.0.smithi006.stdout:rm -f stamp-h1 2024-04-24T23:18:40.403 INFO:tasks.workunit.client.0.smithi006.stdout:touch config.h.in 2024-04-24T23:18:40.407 INFO:tasks.workunit.client.0.smithi006.stdout:cd . && /bin/sh ./config.status config.h 2024-04-24T23:18:40.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:40.469+0000 7ffb83000640 1 Processor -- start 2024-04-24T23:18:40.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:40.469+0000 7ffb83000640 1 -- start start 2024-04-24T23:18:40.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:40.471+0000 7ffb83000640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ffb7c167180 0x7ffb7c0c7dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:18:40.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:40.471+0000 7ffb83000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ffb7c16abf0 0x7ffb7c16b080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:18:40.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:40.471+0000 7ffb83000640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffb7c072970 con 0x7ffb7c072530 2024-04-24T23:18:40.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:40.471+0000 7ffb83000640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7ffb7c168570 con 0x7ffb7c16abf0 2024-04-24T23:18:40.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:40.471+0000 7ffb83000640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7ffb7c0720a0 con 0x7ffb7c167180 2024-04-24T23:18:40.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:40.471+0000 7ffb7b400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7ffb7c072530 0x7ffb7c06edb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:47188/0 (socket says 172.21.15.6:47188) 2024-04-24T23:18:40.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:40.471+0000 7ffb7b400640 1 -- 172.21.15.6:0/1468606598 learned_addr learned my addr 172.21.15.6:0/1468606598 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:18:40.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:40.471+0000 7ffb7be00640 1 --2- 172.21.15.6:0/1468606598 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ffb7c167180 0x7ffb7c0c7dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:18:40.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:40.471+0000 7ffb81000640 1 --2- 172.21.15.6:0/1468606598 >> v2:172.21.15.6:3300/0 conn(0x7ffb7c16abf0 0x7ffb7c16b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:18:40.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:40.472+0000 7ffb7aa00640 1 -- 172.21.15.6:0/1468606598 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3740498441 0 0) 0x7ffb68001110 con 0x7ffb7c072530 2024-04-24T23:18:40.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:40.473+0000 7ffb7aa00640 1 -- 172.21.15.6:0/1468606598 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffb60003500 con 0x7ffb7c072530 2024-04-24T23:18:40.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:40.473+0000 7ffb7aa00640 1 -- 172.21.15.6:0/1468606598 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1588722343 0 0) 0x7ffb60003500 con 0x7ffb7c072530 2024-04-24T23:18:40.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:40.473+0000 7ffb81000640 1 -- 172.21.15.6:0/1468606598 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ffb7c167180 msgr2=0x7ffb7c0c7dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:18:40.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:40.473+0000 7ffb81000640 1 --2- 172.21.15.6:0/1468606598 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ffb7c167180 0x7ffb7c0c7dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:40.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:40.473+0000 7ffb81000640 1 -- 172.21.15.6:0/1468606598 >> v1:172.21.15.6:6789/0 conn(0x7ffb7c072530 legacy=0x7ffb7c06edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:18:40.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:40.473+0000 7ffb81000640 1 -- 172.21.15.6:0/1468606598 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb7c0c74c0 con 0x7ffb7c16abf0 2024-04-24T23:18:40.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:40.474+0000 7ffb7be00640 1 --2- 172.21.15.6:0/1468606598 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ffb7c167180 0x7ffb7c0c7dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:18:40.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:40.475+0000 7ffb81000640 1 --2- 172.21.15.6:0/1468606598 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffb7c16abf0 0x7ffb7c16b080 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7ffb7400f370 tx=0x7ffb7402ea90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aecf7b89b433d2bf server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:18:40.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:40.475+0000 7ffb7aa00640 1 -- 172.21.15.6:0/1468606598 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb74037020 con 0x7ffb7c16abf0 2024-04-24T23:18:40.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:40.475+0000 7ffb7aa00640 1 -- 172.21.15.6:0/1468606598 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffb7400d2e0 con 0x7ffb7c16abf0 2024-04-24T23:18:40.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:40.475+0000 7ffb7aa00640 1 -- 172.21.15.6:0/1468606598 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb68001840 con 0x7ffb7c16abf0 2024-04-24T23:18:40.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:40.476+0000 7ffb83000640 1 -- 172.21.15.6:0/1468606598 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffb7c16abf0 msgr2=0x7ffb7c16b080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:18:40.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:40.476+0000 7ffb83000640 1 --2- 172.21.15.6:0/1468606598 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffb7c16abf0 0x7ffb7c16b080 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7ffb7400f370 tx=0x7ffb7402ea90 comp rx=0 tx=0).stop 2024-04-24T23:18:40.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:40.476+0000 7ffb83000640 1 -- 172.21.15.6:0/1468606598 shutdown_connections 2024-04-24T23:18:40.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:40.476+0000 7ffb83000640 1 --2- 172.21.15.6:0/1468606598 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffb7c16abf0 0x7ffb7c16b080 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:40.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:40.476+0000 7ffb83000640 1 --2- 172.21.15.6:0/1468606598 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ffb7c167180 0x7ffb7c0c7dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:40.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:40.476+0000 7ffb83000640 1 -- 172.21.15.6:0/1468606598 >> 172.21.15.6:0/1468606598 conn(0x7ffb7c06b0e0 msgr2=0x7ffb7c06b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:18:40.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:40.476+0000 7ffb83000640 1 -- 172.21.15.6:0/1468606598 shutdown_connections 2024-04-24T23:18:40.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:40.476+0000 7ffb83000640 1 -- 172.21.15.6:0/1468606598 wait complete. 2024-04-24T23:18:40.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:40.478+0000 7ffb83000640 1 Processor -- start 2024-04-24T23:18:40.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:40.478+0000 7ffb83000640 1 -- start start 2024-04-24T23:18:40.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:40.479+0000 7ffb83000640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ffb7c072530 0x7ffb7c163ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:18:40.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:40.479+0000 7ffb83000640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ffb7c167180 0x7ffb7c163010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:18:40.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:40.479+0000 7ffb83000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffb7c16abf0 0x7ffb7c0c8550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:18:40.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:40.479+0000 7ffb83000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7ffb7c071a50 con 0x7ffb7c16abf0 2024-04-24T23:18:40.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:40.479+0000 7ffb83000640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7ffb7c071770 con 0x7ffb7c167180 2024-04-24T23:18:40.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:40.479+0000 7ffb83000640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7ffb7c071d30 con 0x7ffb7c072530 2024-04-24T23:18:40.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:40.479+0000 7ffb7be00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ffb7c072530 0x7ffb7c163ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:18:40.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:40.479+0000 7ffb7b400640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ffb7c167180 0x7ffb7c163010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:18:40.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:40.479+0000 7ffb7be00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ffb7c072530 0x7ffb7c163ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.6:36442/0 (socket says 172.21.15.6:36442) 2024-04-24T23:18:40.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:40.479+0000 7ffb81000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffb7c16abf0 0x7ffb7c0c8550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:18:40.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:40.479+0000 7ffb7be00640 1 -- 172.21.15.6:0/2369723341 learned_addr learned my addr 172.21.15.6:0/2369723341 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:18:40.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:40.480+0000 7ffb7be00640 1 -- 172.21.15.6:0/2369723341 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ffb7c167180 msgr2=0x7ffb7c163010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:18:40.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:40.480+0000 7ffb7be00640 1 --2- 172.21.15.6:0/2369723341 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ffb7c167180 0x7ffb7c163010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:40.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:40.480+0000 7ffb7be00640 1 -- 172.21.15.6:0/2369723341 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffb7c16abf0 msgr2=0x7ffb7c0c8550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:18:40.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:40.480+0000 7ffb7be00640 1 --2- 172.21.15.6:0/2369723341 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffb7c16abf0 0x7ffb7c0c8550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:40.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:40.480+0000 7ffb7be00640 1 -- 172.21.15.6:0/2369723341 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb7c163550 con 0x7ffb7c072530 2024-04-24T23:18:40.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:40.480+0000 7ffb7b400640 1 --2- 172.21.15.6:0/2369723341 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ffb7c167180 0x7ffb7c163010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:18:40.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:40.480+0000 7ffb7be00640 1 --2- 172.21.15.6:0/2369723341 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ffb7c072530 0x7ffb7c163ce0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7ffb6c00ba50 tx=0x7ffb6c00e4f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:18:40.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:40.480+0000 7ffb73e00640 1 -- 172.21.15.6:0/2369723341 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb6c004030 con 0x7ffb7c072530 2024-04-24T23:18:40.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:40.481+0000 7ffb73e00640 1 -- 172.21.15.6:0/2369723341 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffb6c0041d0 con 0x7ffb7c072530 2024-04-24T23:18:40.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:40.481+0000 7ffb73e00640 1 -- 172.21.15.6:0/2369723341 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb6c003610 con 0x7ffb7c072530 2024-04-24T23:18:40.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:40.481+0000 7ffb83000640 1 -- 172.21.15.6:0/2369723341 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb7c0c4450 con 0x7ffb7c072530 2024-04-24T23:18:40.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:40.481+0000 7ffb83000640 1 -- 172.21.15.6:0/2369723341 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb7c0c4a10 con 0x7ffb7c072530 2024-04-24T23:18:40.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:40.482+0000 7ffb83000640 1 -- 172.21.15.6:0/2369723341 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffb7c0c41e0 con 0x7ffb7c072530 2024-04-24T23:18:40.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:40.484+0000 7ffb73e00640 1 -- 172.21.15.6:0/2369723341 <== mon.2 v2:172.21.15.190:3301/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7ffb6c0037b0 con 0x7ffb7c072530 2024-04-24T23:18:40.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:40.485+0000 7ffb73e00640 1 --2- 172.21.15.6:0/2369723341 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7ffb5c06be60 0x7ffb5c06e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:18:40.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:40.485+0000 7ffb73e00640 1 -- 172.21.15.6:0/2369723341 <== mon.2 v2:172.21.15.190:3301/0 5 ==== osd_map(27..27 src has 1..27) ==== 6676+0+0 (secure 0 0 0) 0x7ffb6c08bdb0 con 0x7ffb7c072530 2024-04-24T23:18:40.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:40.485+0000 7ffb7b400640 1 --2- 172.21.15.6:0/2369723341 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7ffb5c06be60 0x7ffb5c06e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:18:40.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:40.486+0000 7ffb7b400640 1 --2- 172.21.15.6:0/2369723341 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7ffb5c06be60 0x7ffb5c06e320 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7ffb680019b0 tx=0x7ffb680012d0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:18:40.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:40.489+0000 7ffb73e00640 1 -- 172.21.15.6:0/2369723341 <== mon.2 v2:172.21.15.190:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7ffb6c090070 con 0x7ffb7c072530 2024-04-24T23:18:40.500 INFO:tasks.workunit.client.0.smithi006.stdout:config.status: creating config.h 2024-04-24T23:18:40.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:40.731+0000 7ffb83000640 1 -- 172.21.15.6:0/2369723341 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ffb7c06f2e0 con 0x7ffb7c072530 2024-04-24T23:18:40.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:40.732+0000 7ffb73e00640 1 -- 172.21.15.6:0/2369723341 <== mon.2 v2:172.21.15.190:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+6081 (secure 0 0 0) 0x7ffb7c06f2e0 con 0x7ffb7c072530 2024-04-24T23:18:40.734 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:18:40.734 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":21,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:18:37.773499+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4282,"mds_2":4235,"mds_3":4232,"mds_4":4197},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4197":{"gid":4197,"name":"a","rank":4,"incarnation":19,"state":"up:reconnect","state_seq":49,"addr":"172.21.15.6:6834/105267896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":105267896},{"type":"v1","addr":"172.21.15.6:6834","nonce":105267896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4232":{"gid":4232,"name":"b","rank":3,"incarnation":15,"state":"up:active","state_seq":15,"addr":"172.21.15.190:6838/2032280158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2032280158},{"type":"v1","addr":"172.21.15.190:6838","nonce":2032280158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4235":{"gid":4235,"name":"d","rank":2,"incarnation":13,"state":"up:active","state_seq":14,"addr":"172.21.15.190:6839/1572872451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":1572872451},{"type":"v1","addr":"172.21.15.190:6839","nonce":1572872451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4282":{"gid":4282,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":12,"addr":"172.21.15.190:6837/580315269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":580315269},{"type":"v1","addr":"172.21.15.190:6837","nonce":580315269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/489092393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":489092393},{"type":"v1","addr":"172.21.15.6:6835","nonce":489092393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4306,"qdb_cluster":[4232,4235,4282,4306]},"id":1}]} 2024-04-24T23:18:40.735 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 21 2024-04-24T23:18:40.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:40.735+0000 7ffb83000640 1 -- 172.21.15.6:0/2369723341 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7ffb5c06be60 msgr2=0x7ffb5c06e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:18:40.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:40.735+0000 7ffb83000640 1 --2- 172.21.15.6:0/2369723341 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7ffb5c06be60 0x7ffb5c06e320 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7ffb680019b0 tx=0x7ffb680012d0 comp rx=0 tx=0).stop 2024-04-24T23:18:40.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:40.735+0000 7ffb83000640 1 -- 172.21.15.6:0/2369723341 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ffb7c072530 msgr2=0x7ffb7c163ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:18:40.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:40.735+0000 7ffb83000640 1 --2- 172.21.15.6:0/2369723341 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ffb7c072530 0x7ffb7c163ce0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7ffb6c00ba50 tx=0x7ffb6c00e4f0 comp rx=0 tx=0).stop 2024-04-24T23:18:40.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:40.735+0000 7ffb83000640 1 -- 172.21.15.6:0/2369723341 shutdown_connections 2024-04-24T23:18:40.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:40.735+0000 7ffb83000640 1 --2- 172.21.15.6:0/2369723341 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7ffb5c06be60 0x7ffb5c06e320 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:40.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:40.735+0000 7ffb83000640 1 --2- 172.21.15.6:0/2369723341 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffb7c16abf0 0x7ffb7c0c8550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:40.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:40.735+0000 7ffb83000640 1 --2- 172.21.15.6:0/2369723341 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ffb7c167180 0x7ffb7c163010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:40.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:40.735+0000 7ffb83000640 1 --2- 172.21.15.6:0/2369723341 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ffb7c072530 0x7ffb7c163ce0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:40.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:40.735+0000 7ffb83000640 1 -- 172.21.15.6:0/2369723341 >> 172.21.15.6:0/2369723341 conn(0x7ffb7c06b0e0 msgr2=0x7ffb7c0cbab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:18:40.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:40.735+0000 7ffb83000640 1 -- 172.21.15.6:0/2369723341 shutdown_connections 2024-04-24T23:18:40.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:40.735+0000 7ffb83000640 1 -- 172.21.15.6:0/2369723341 wait complete. 2024-04-24T23:18:40.758 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 21, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-24T23:15:36.400716+0000", "damaged": [], "data_pools": [ 3, 4 ], "enabled": true, "epoch": 21, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4197": { "addr": "172.21.15.6:6834/105267896", "addrs": { "addrvec": [ { "addr": "172.21.15.6:6832", "nonce": 105267896, "type": "v2" }, { "addr": "172.21.15.6:6834", "nonce": 105267896, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4197, "incarnation": 19, "join_fscid": -1, "name": "a", "rank": 4, "state": "up:reconnect", "state_seq": 49 }, "gid_4232": { "addr": "172.21.15.190:6838/2032280158", "addrs": { "addrvec": [ { "addr": "172.21.15.190:6835", "nonce": 2032280158, "type": "v2" }, { "addr": "172.21.15.190:6838", "nonce": 2032280158, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4232, "incarnation": 15, "join_fscid": -1, "name": "b", "rank": 3, "state": "up:active", "state_seq": 15 }, "gid_4235": { "addr": "172.21.15.190:6839/1572872451", "addrs": { "addrvec": [ { "addr": "172.21.15.190:6836", "nonce": 1572872451, "type": "v2" }, { "addr": "172.21.15.190:6839", "nonce": 1572872451, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4235, "incarnation": 13, "join_fscid": -1, "name": "d", "rank": 2, "state": "up:active", "state_seq": 14 }, "gid_4282": { "addr": "172.21.15.190:6837/580315269", "addrs": { "addrvec": [ { "addr": "172.21.15.190:6834", "nonce": 580315269, "type": "v2" }, { "addr": "172.21.15.190:6837", "nonce": 580315269, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4282, "incarnation": 9, "join_fscid": -1, "name": "f", "rank": 1, "state": "up:active", "state_seq": 12 }, "gid_4306": { "addr": "172.21.15.6:6835/489092393", "addrs": { "addrvec": [ { "addr": "172.21.15.6:6833", "nonce": 489092393, "type": "v2" }, { "addr": "172.21.15.6:6835", "nonce": 489092393, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4306, "incarnation": 4, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 4 } }, "last_failure": 0, "last_failure_osd_epoch": 27, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-04-24T23:18:37.773499+0000", "qdb_cluster": [ 4232, 4235, 4282, 4306 ], "qdb_leader": 4306, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4306, "mds_1": 4282, "mds_2": 4235, "mds_3": 4232, "mds_4": 4197 } } } ], "standbys": [] } 2024-04-24T23:18:41.520 INFO:tasks.workunit.client.0.smithi006.stdout:make all-am 2024-04-24T23:18:41.537 INFO:tasks.workunit.client.0.smithi006.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb' 2024-04-24T23:18:41.538 INFO:tasks.workunit.client.0.smithi006.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT fileops.o -MD -MP -MF .deps/fileops.Tpo -c -o fileops.o fileops.c 2024-04-24T23:18:41.834 INFO:tasks.workunit.client.0.smithi006.stdout:mv -f .deps/fileops.Tpo .deps/fileops.Po 2024-04-24T23:18:41.843 INFO:tasks.workunit.client.0.smithi006.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT rand.o -MD -MP -MF .deps/rand.Tpo -c -o rand.o rand.c 2024-04-24T23:18:41.983 INFO:tasks.workunit.client.0.smithi006.stdout:mv -f .deps/rand.Tpo .deps/rand.Po 2024-04-24T23:18:41.989 INFO:tasks.workunit.client.0.smithi006.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT main.o -MD -MP -MF .deps/main.Tpo -c -o main.o main.c 2024-04-24T23:18:42.131 INFO:tasks.workunit.client.0.smithi006.stdout:mv -f .deps/main.Tpo .deps/main.Po 2024-04-24T23:18:42.139 INFO:tasks.workunit.client.0.smithi006.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT fh.o -MD -MP -MF .deps/fh.Tpo -c -o fh.o fh.c 2024-04-24T23:18:42.280 INFO:tasks.workunit.client.0.smithi006.stdout:mv -f .deps/fh.Tpo .deps/fh.Po 2024-04-24T23:18:42.287 INFO:tasks.workunit.client.0.smithi006.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT filelist.o -MD -MP -MF .deps/filelist.Tpo -c -o filelist.o filelist.c 2024-04-24T23:18:42.470 INFO:tasks.workunit.client.0.smithi006.stdout:mv -f .deps/filelist.Tpo .deps/filelist.Po 2024-04-24T23:18:42.478 INFO:tasks.workunit.client.0.smithi006.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT metaops.o -MD -MP -MF .deps/metaops.Tpo -c -o metaops.o metaops.c 2024-04-24T23:18:42.571 INFO:tasks.workunit.client.0.smithi006.stdout:mv -f .deps/metaops.Tpo .deps/metaops.Po 2024-04-24T23:18:42.577 INFO:tasks.workunit.client.0.smithi006.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT rwlock.o -MD -MP -MF .deps/rwlock.Tpo -c -o rwlock.o rwlock.c 2024-04-24T23:18:42.662 INFO:tasks.workunit.client.0.smithi006.stdout:mv -f .deps/rwlock.Tpo .deps/rwlock.Po 2024-04-24T23:18:42.665 INFO:tasks.workunit.client.0.smithi006.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT cirlist.o -MD -MP -MF .deps/cirlist.Tpo -c -o cirlist.o cirlist.c 2024-04-24T23:18:42.754 INFO:tasks.workunit.client.0.smithi006.stdout:mv -f .deps/cirlist.Tpo .deps/cirlist.Po 2024-04-24T23:18:42.759 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:18:42.768 INFO:tasks.workunit.client.0.smithi006.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT rbt.o -MD -MP -MF .deps/rbt.Tpo -c -o rbt.o rbt.c 2024-04-24T23:18:42.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:42.960+0000 7efead400640 1 Processor -- start 2024-04-24T23:18:42.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:42.960+0000 7efead400640 1 -- start start 2024-04-24T23:18:42.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:42.962+0000 7efead400640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efea8071530 0x7efea80c8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:18:42.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:42.962+0000 7efead400640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efea80c8390 con 0x7efea80c9230 2024-04-24T23:18:42.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:42.962+0000 7efead400640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efea8071d10 con 0x7efea8151bb0 2024-04-24T23:18:42.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:42.962+0000 7efead400640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7efea8071ff0 con 0x7efea8071530 2024-04-24T23:18:42.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:42.962+0000 7efea6800640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7efea8151bb0 0x7efea80c37f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.6:54466/0 (socket says 172.21.15.6:54466) 2024-04-24T23:18:42.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:42.962+0000 7efea6800640 1 -- 172.21.15.6:0/3124594316 learned_addr learned my addr 172.21.15.6:0/3124594316 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:18:42.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:42.962+0000 7efe9fe00640 1 --2- 172.21.15.6:0/3124594316 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efea8071530 0x7efea80c8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:18:42.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:42.962+0000 7efea5e00640 1 -- 172.21.15.6:0/3124594316 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4057227946 0 0) 0x7efea8071d10 con 0x7efea8151bb0 2024-04-24T23:18:42.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:42.962+0000 7efea5e00640 1 -- 172.21.15.6:0/3124594316 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efe8c003520 con 0x7efea8151bb0 2024-04-24T23:18:42.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:42.963+0000 7efea5e00640 1 -- 172.21.15.6:0/3124594316 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1891864106 0 0) 0x7efea80c8390 con 0x7efea80c9230 2024-04-24T23:18:42.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:42.963+0000 7efe9fe00640 1 -- 172.21.15.6:0/3124594316 >> v1:172.21.15.190:6789/0 conn(0x7efea8151bb0 legacy=0x7efea80c37f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:18:42.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:42.963+0000 7efe9fe00640 1 -- 172.21.15.6:0/3124594316 >> v1:172.21.15.6:6789/0 conn(0x7efea80c9230 legacy=0x7efea8157950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:18:42.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:42.963+0000 7efe9fe00640 1 -- 172.21.15.6:0/3124594316 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efea80722d0 con 0x7efea8071530 2024-04-24T23:18:42.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:42.964+0000 7efe9fe00640 1 --2- 172.21.15.6:0/3124594316 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efea8071530 0x7efea80c8cf0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7efe900097d0 tx=0x7efe9002a290 comp rx=0 tx=0).ready entity=mon.2 client_cookie=781391d5b94a6d46 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:18:42.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:42.964+0000 7efea5e00640 1 -- 172.21.15.6:0/3124594316 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe90037070 con 0x7efea8071530 2024-04-24T23:18:42.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:42.964+0000 7efea5e00640 1 -- 172.21.15.6:0/3124594316 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efe90032440 con 0x7efea8071530 2024-04-24T23:18:42.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:42.964+0000 7efea5e00640 1 -- 172.21.15.6:0/3124594316 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe900327f0 con 0x7efea8071530 2024-04-24T23:18:42.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:42.965+0000 7efead400640 1 -- 172.21.15.6:0/3124594316 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efea8071530 msgr2=0x7efea80c8cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:18:42.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:42.965+0000 7efead400640 1 --2- 172.21.15.6:0/3124594316 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efea8071530 0x7efea80c8cf0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7efe900097d0 tx=0x7efe9002a290 comp rx=0 tx=0).stop 2024-04-24T23:18:42.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:42.965+0000 7efead400640 1 -- 172.21.15.6:0/3124594316 shutdown_connections 2024-04-24T23:18:42.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:42.965+0000 7efead400640 1 --2- 172.21.15.6:0/3124594316 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efea8071530 0x7efea80c8cf0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:42.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:42.965+0000 7efead400640 1 -- 172.21.15.6:0/3124594316 >> 172.21.15.6:0/3124594316 conn(0x7efea806b0e0 msgr2=0x7efea806b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:18:42.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:42.965+0000 7efead400640 1 -- 172.21.15.6:0/3124594316 shutdown_connections 2024-04-24T23:18:42.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:42.965+0000 7efead400640 1 -- 172.21.15.6:0/3124594316 wait complete. 2024-04-24T23:18:42.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:42.967+0000 7efead400640 1 Processor -- start 2024-04-24T23:18:42.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:42.967+0000 7efead400640 1 -- start start 2024-04-24T23:18:42.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:42.968+0000 7efead400640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efea8071530 0x7efea814a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:18:42.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:42.968+0000 7efead400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efea80c9230 0x7efea8148890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:18:42.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:42.968+0000 7efead400640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efea8151bb0 0x7efea81559c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:18:42.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:42.968+0000 7efead400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7efea80c80f0 con 0x7efea80c9230 2024-04-24T23:18:42.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:42.968+0000 7efead400640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7efea80c7f70 con 0x7efea8151bb0 2024-04-24T23:18:42.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:42.968+0000 7efead400640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7efea8158060 con 0x7efea8071530 2024-04-24T23:18:42.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:42.968+0000 7efe9fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efea80c9230 0x7efea8148890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:18:42.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:42.968+0000 7efea6800640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efea8071530 0x7efea814a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:18:42.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:42.968+0000 7efe9fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efea80c9230 0x7efea8148890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45896/0 (socket says 172.21.15.6:45896) 2024-04-24T23:18:42.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:42.968+0000 7efe9fe00640 1 -- 172.21.15.6:0/1150771274 learned_addr learned my addr 172.21.15.6:0/1150771274 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:18:42.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:42.968+0000 7efea7200640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efea8151bb0 0x7efea81559c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:18:42.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:42.969+0000 7efe9fe00640 1 -- 172.21.15.6:0/1150771274 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efea8071530 msgr2=0x7efea814a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:18:42.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:42.969+0000 7efe9fe00640 1 --2- 172.21.15.6:0/1150771274 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efea8071530 0x7efea814a760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:42.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:42.969+0000 7efe9fe00640 1 -- 172.21.15.6:0/1150771274 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efea8151bb0 msgr2=0x7efea81559c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:18:42.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:42.969+0000 7efe9fe00640 1 --2- 172.21.15.6:0/1150771274 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efea8151bb0 0x7efea81559c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:42.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:42.969+0000 7efe9fe00640 1 -- 172.21.15.6:0/1150771274 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efea81f1460 con 0x7efea80c9230 2024-04-24T23:18:42.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:42.969+0000 7efea6800640 1 --2- 172.21.15.6:0/1150771274 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efea8071530 0x7efea814a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:18:42.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:42.969+0000 7efe9fe00640 1 --2- 172.21.15.6:0/1150771274 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efea80c9230 0x7efea8148890 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7efe900098f0 tx=0x7efe900046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:18:42.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:42.969+0000 7efe9ea00640 1 -- 172.21.15.6:0/1150771274 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe90037040 con 0x7efea80c9230 2024-04-24T23:18:42.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:42.969+0000 7efe9ea00640 1 -- 172.21.15.6:0/1150771274 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efe90004230 con 0x7efea80c9230 2024-04-24T23:18:42.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:42.969+0000 7efe9ea00640 1 -- 172.21.15.6:0/1150771274 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe90003610 con 0x7efea80c9230 2024-04-24T23:18:42.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:42.969+0000 7efead400640 1 -- 172.21.15.6:0/1150771274 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efea81f08e0 con 0x7efea80c9230 2024-04-24T23:18:42.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:42.969+0000 7efead400640 1 -- 172.21.15.6:0/1150771274 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efea81f0ea0 con 0x7efea80c9230 2024-04-24T23:18:42.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:42.971+0000 7efe97e00640 1 -- 172.21.15.6:0/1150771274 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efea80767b0 con 0x7efea80c9230 2024-04-24T23:18:42.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:42.973+0000 7efe9ea00640 1 -- 172.21.15.6:0/1150771274 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7efe900037b0 con 0x7efea80c9230 2024-04-24T23:18:42.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:42.974+0000 7efe9ea00640 1 --2- 172.21.15.6:0/1150771274 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7efe8406be60 0x7efe8406e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:18:42.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:42.974+0000 7efe9ea00640 1 -- 172.21.15.6:0/1150771274 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6676+0+0 (secure 0 0 0) 0x7efe900abde0 con 0x7efea80c9230 2024-04-24T23:18:42.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:42.974+0000 7efea6800640 1 --2- 172.21.15.6:0/1150771274 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7efe8406be60 0x7efe8406e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:18:42.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:42.975+0000 7efea6800640 1 --2- 172.21.15.6:0/1150771274 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7efe8406be60 0x7efe8406e320 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7efe980011e0 tx=0x7efe98007d30 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:18:42.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:42.979+0000 7efe9ea00640 1 -- 172.21.15.6:0/1150771274 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7efe90079d60 con 0x7efea80c9230 2024-04-24T23:18:43.047 INFO:tasks.workunit.client.0.smithi006.stdout:mv -f .deps/rbt.Tpo .deps/rbt.Po 2024-04-24T23:18:43.053 INFO:tasks.workunit.client.0.smithi006.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_tg.o -MD -MP -MF .deps/ffsb_tg.Tpo -c -o ffsb_tg.o ffsb_tg.c 2024-04-24T23:18:43.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:43.222+0000 7efe97e00640 1 -- 172.21.15.6:0/1150771274 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efea8068550 con 0x7efea80c9230 2024-04-24T23:18:43.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:43.223+0000 7efe9ea00640 1 -- 172.21.15.6:0/1150771274 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6078 (secure 0 0 0) 0x7efe9007e060 con 0x7efea80c9230 2024-04-24T23:18:43.225 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:18:43.225 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:18:42.788711+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4282,"mds_2":4235,"mds_3":4232,"mds_4":4197},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4197":{"gid":4197,"name":"a","rank":4,"incarnation":19,"state":"up:rejoin","state_seq":51,"addr":"172.21.15.6:6834/105267896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":105267896},{"type":"v1","addr":"172.21.15.6:6834","nonce":105267896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4232":{"gid":4232,"name":"b","rank":3,"incarnation":15,"state":"up:active","state_seq":15,"addr":"172.21.15.190:6838/2032280158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2032280158},{"type":"v1","addr":"172.21.15.190:6838","nonce":2032280158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4235":{"gid":4235,"name":"d","rank":2,"incarnation":13,"state":"up:active","state_seq":14,"addr":"172.21.15.190:6839/1572872451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":1572872451},{"type":"v1","addr":"172.21.15.190:6839","nonce":1572872451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4282":{"gid":4282,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":12,"addr":"172.21.15.190:6837/580315269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":580315269},{"type":"v1","addr":"172.21.15.190:6837","nonce":580315269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/489092393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":489092393},{"type":"v1","addr":"172.21.15.6:6835","nonce":489092393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4306,"qdb_cluster":[4306,4282,4235,4232]},"id":1}]} 2024-04-24T23:18:43.226 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 22 2024-04-24T23:18:43.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:43.225+0000 7efe97e00640 1 -- 172.21.15.6:0/1150771274 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7efe8406be60 msgr2=0x7efe8406e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:18:43.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:43.225+0000 7efe97e00640 1 --2- 172.21.15.6:0/1150771274 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7efe8406be60 0x7efe8406e320 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7efe980011e0 tx=0x7efe98007d30 comp rx=0 tx=0).stop 2024-04-24T23:18:43.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:43.226+0000 7efe97e00640 1 -- 172.21.15.6:0/1150771274 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efea80c9230 msgr2=0x7efea8148890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:18:43.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:43.226+0000 7efe97e00640 1 --2- 172.21.15.6:0/1150771274 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efea80c9230 0x7efea8148890 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7efe900098f0 tx=0x7efe900046c0 comp rx=0 tx=0).stop 2024-04-24T23:18:43.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:43.226+0000 7efe97e00640 1 -- 172.21.15.6:0/1150771274 shutdown_connections 2024-04-24T23:18:43.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:43.226+0000 7efe97e00640 1 --2- 172.21.15.6:0/1150771274 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7efe8406be60 0x7efe8406e320 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:43.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:43.226+0000 7efe97e00640 1 --2- 172.21.15.6:0/1150771274 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efea8151bb0 0x7efea81559c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:43.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:43.226+0000 7efe97e00640 1 --2- 172.21.15.6:0/1150771274 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efea80c9230 0x7efea8148890 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:43.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:43.226+0000 7efe97e00640 1 --2- 172.21.15.6:0/1150771274 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efea8071530 0x7efea814a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:43.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:43.226+0000 7efe97e00640 1 -- 172.21.15.6:0/1150771274 >> 172.21.15.6:0/1150771274 conn(0x7efea806b0e0 msgr2=0x7efea80c5c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:18:43.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:43.226+0000 7efe97e00640 1 -- 172.21.15.6:0/1150771274 shutdown_connections 2024-04-24T23:18:43.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:43.226+0000 7efe97e00640 1 -- 172.21.15.6:0/1150771274 wait complete. 2024-04-24T23:18:43.237 INFO:tasks.workunit.client.0.smithi006.stdout:mv -f .deps/ffsb_tg.Tpo .deps/ffsb_tg.Po 2024-04-24T23:18:43.243 INFO:tasks.workunit.client.0.smithi006.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_fs.o -MD -MP -MF .deps/ffsb_fs.Tpo -c -o ffsb_fs.o ffsb_fs.c 2024-04-24T23:18:43.248 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-24T23:18:43.470 INFO:tasks.workunit.client.0.smithi006.stdout:mv -f .deps/ffsb_fs.Tpo .deps/ffsb_fs.Po 2024-04-24T23:18:43.477 INFO:tasks.workunit.client.0.smithi006.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_thread.o -MD -MP -MF .deps/ffsb_thread.Tpo -c -o ffsb_thread.o ffsb_thread.c 2024-04-24T23:18:43.604 INFO:tasks.workunit.client.0.smithi006.stdout:mv -f .deps/ffsb_thread.Tpo .deps/ffsb_thread.Po 2024-04-24T23:18:43.609 INFO:tasks.workunit.client.0.smithi006.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_op.o -MD -MP -MF .deps/ffsb_op.Tpo -c -o ffsb_op.o ffsb_op.c 2024-04-24T23:18:43.716 INFO:tasks.workunit.client.0.smithi006.stdout:mv -f .deps/ffsb_op.Tpo .deps/ffsb_op.Po 2024-04-24T23:18:43.721 INFO:tasks.workunit.client.0.smithi006.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT util.o -MD -MP -MF .deps/util.Tpo -c -o util.o util.c 2024-04-24T23:18:43.875 INFO:tasks.workunit.client.0.smithi006.stdout:mv -f .deps/util.Tpo .deps/util.Po 2024-04-24T23:18:43.881 INFO:tasks.workunit.client.0.smithi006.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT parser.o -MD -MP -MF .deps/parser.Tpo -c -o parser.o parser.c 2024-04-24T23:18:43.937 INFO:tasks.workunit.client.0.smithi006.stderr:In file included from parser.h:22, 2024-04-24T23:18:43.937 INFO:tasks.workunit.client.0.smithi006.stderr: from parser.c:25: 2024-04-24T23:18:43.937 INFO:tasks.workunit.client.0.smithi006.stderr:parser.c: In function ‘init_filesys’: 2024-04-24T23:18:43.938 INFO:tasks.workunit.client.0.smithi006.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-04-24T23:18:43.938 INFO:tasks.workunit.client.0.smithi006.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-04-24T23:18:43.938 INFO:tasks.workunit.client.0.smithi006.stderr: | ^ 2024-04-24T23:18:43.938 INFO:tasks.workunit.client.0.smithi006.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-04-24T23:18:43.938 INFO:tasks.workunit.client.0.smithi006.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-04-24T23:18:43.938 INFO:tasks.workunit.client.0.smithi006.stderr: | ^~~~~~~~ 2024-04-24T23:18:43.938 INFO:tasks.workunit.client.0.smithi006.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-04-24T23:18:43.938 INFO:tasks.workunit.client.0.smithi006.stderr: 36 | container_of(ptr, type, member) 2024-04-24T23:18:43.938 INFO:tasks.workunit.client.0.smithi006.stderr: | ^~~~~~~~~~~~ 2024-04-24T23:18:43.938 INFO:tasks.workunit.client.0.smithi006.stderr:list.h:39:20: note: in expansion of macro ‘list_entry’ 2024-04-24T23:18:43.938 INFO:tasks.workunit.client.0.smithi006.stderr: 39 | for (pos = list_entry((head)->next, typeof(*pos), member); \ 2024-04-24T23:18:43.938 INFO:tasks.workunit.client.0.smithi006.stderr: | ^~~~~~~~~~ 2024-04-24T23:18:43.938 INFO:tasks.workunit.client.0.smithi006.stderr:parser.c:934:17: note: in expansion of macro ‘list_for_each_entry’ 2024-04-24T23:18:43.938 INFO:tasks.workunit.client.0.smithi006.stderr: 934 | list_for_each_entry(tmp_list, &list_head->list, list) 2024-04-24T23:18:43.939 INFO:tasks.workunit.client.0.smithi006.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-04-24T23:18:43.939 INFO:tasks.workunit.client.0.smithi006.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-04-24T23:18:43.939 INFO:tasks.workunit.client.0.smithi006.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-04-24T23:18:43.939 INFO:tasks.workunit.client.0.smithi006.stderr: | ^ 2024-04-24T23:18:43.939 INFO:tasks.workunit.client.0.smithi006.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-04-24T23:18:43.939 INFO:tasks.workunit.client.0.smithi006.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-04-24T23:18:43.939 INFO:tasks.workunit.client.0.smithi006.stderr: | ^~~~~~~~ 2024-04-24T23:18:43.939 INFO:tasks.workunit.client.0.smithi006.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-04-24T23:18:43.939 INFO:tasks.workunit.client.0.smithi006.stderr: 36 | container_of(ptr, type, member) 2024-04-24T23:18:43.939 INFO:tasks.workunit.client.0.smithi006.stderr: | ^~~~~~~~~~~~ 2024-04-24T23:18:43.939 INFO:tasks.workunit.client.0.smithi006.stderr:list.h:41:20: note: in expansion of macro ‘list_entry’ 2024-04-24T23:18:43.939 INFO:tasks.workunit.client.0.smithi006.stderr: 41 | pos = list_entry(pos->member.next, typeof(*pos), member)) 2024-04-24T23:18:43.939 INFO:tasks.workunit.client.0.smithi006.stderr: | ^~~~~~~~~~ 2024-04-24T23:18:43.940 INFO:tasks.workunit.client.0.smithi006.stderr:parser.c:934:17: note: in expansion of macro ‘list_for_each_entry’ 2024-04-24T23:18:43.940 INFO:tasks.workunit.client.0.smithi006.stderr: 934 | list_for_each_entry(tmp_list, &list_head->list, list) 2024-04-24T23:18:43.940 INFO:tasks.workunit.client.0.smithi006.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-04-24T23:18:43.941 INFO:tasks.workunit.client.0.smithi006.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-04-24T23:18:43.941 INFO:tasks.workunit.client.0.smithi006.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-04-24T23:18:43.941 INFO:tasks.workunit.client.0.smithi006.stderr: | ^ 2024-04-24T23:18:43.941 INFO:tasks.workunit.client.0.smithi006.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-04-24T23:18:43.941 INFO:tasks.workunit.client.0.smithi006.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-04-24T23:18:43.941 INFO:tasks.workunit.client.0.smithi006.stderr: | ^~~~~~~~ 2024-04-24T23:18:43.941 INFO:tasks.workunit.client.0.smithi006.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-04-24T23:18:43.941 INFO:tasks.workunit.client.0.smithi006.stderr: 36 | container_of(ptr, type, member) 2024-04-24T23:18:43.941 INFO:tasks.workunit.client.0.smithi006.stderr: | ^~~~~~~~~~~~ 2024-04-24T23:18:43.941 INFO:tasks.workunit.client.0.smithi006.stderr:list.h:39:20: note: in expansion of macro ‘list_entry’ 2024-04-24T23:18:43.941 INFO:tasks.workunit.client.0.smithi006.stderr: 39 | for (pos = list_entry((head)->next, typeof(*pos), member); \ 2024-04-24T23:18:43.941 INFO:tasks.workunit.client.0.smithi006.stderr: | ^~~~~~~~~~ 2024-04-24T23:18:43.942 INFO:tasks.workunit.client.0.smithi006.stderr:parser.c:941:17: note: in expansion of macro ‘list_for_each_entry’ 2024-04-24T23:18:43.942 INFO:tasks.workunit.client.0.smithi006.stderr: 941 | list_for_each_entry(tmp_list, &list_head->list, list) { 2024-04-24T23:18:43.942 INFO:tasks.workunit.client.0.smithi006.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-04-24T23:18:43.942 INFO:tasks.workunit.client.0.smithi006.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-04-24T23:18:43.942 INFO:tasks.workunit.client.0.smithi006.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-04-24T23:18:43.942 INFO:tasks.workunit.client.0.smithi006.stderr: | ^ 2024-04-24T23:18:43.942 INFO:tasks.workunit.client.0.smithi006.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-04-24T23:18:43.942 INFO:tasks.workunit.client.0.smithi006.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-04-24T23:18:43.942 INFO:tasks.workunit.client.0.smithi006.stderr: | ^~~~~~~~ 2024-04-24T23:18:43.942 INFO:tasks.workunit.client.0.smithi006.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-04-24T23:18:43.942 INFO:tasks.workunit.client.0.smithi006.stderr: 36 | container_of(ptr, type, member) 2024-04-24T23:18:43.942 INFO:tasks.workunit.client.0.smithi006.stderr: | ^~~~~~~~~~~~ 2024-04-24T23:18:43.942 INFO:tasks.workunit.client.0.smithi006.stderr:list.h:41:20: note: in expansion of macro ‘list_entry’ 2024-04-24T23:18:43.942 INFO:tasks.workunit.client.0.smithi006.stderr: 41 | pos = list_entry(pos->member.next, typeof(*pos), member)) 2024-04-24T23:18:43.943 INFO:tasks.workunit.client.0.smithi006.stderr: | ^~~~~~~~~~ 2024-04-24T23:18:43.943 INFO:tasks.workunit.client.0.smithi006.stderr:parser.c:941:17: note: in expansion of macro ‘list_for_each_entry’ 2024-04-24T23:18:43.943 INFO:tasks.workunit.client.0.smithi006.stderr: 941 | list_for_each_entry(tmp_list, &list_head->list, list) { 2024-04-24T23:18:43.943 INFO:tasks.workunit.client.0.smithi006.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-04-24T23:18:43.943 INFO:tasks.workunit.client.0.smithi006.stderr:parser.c: In function ‘init_tg_stats’: 2024-04-24T23:18:43.943 INFO:tasks.workunit.client.0.smithi006.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-04-24T23:18:43.943 INFO:tasks.workunit.client.0.smithi006.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-04-24T23:18:43.943 INFO:tasks.workunit.client.0.smithi006.stderr: | ^ 2024-04-24T23:18:43.943 INFO:tasks.workunit.client.0.smithi006.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-04-24T23:18:43.943 INFO:tasks.workunit.client.0.smithi006.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-04-24T23:18:43.943 INFO:tasks.workunit.client.0.smithi006.stderr: | ^~~~~~~~ 2024-04-24T23:18:43.943 INFO:tasks.workunit.client.0.smithi006.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-04-24T23:18:43.943 INFO:tasks.workunit.client.0.smithi006.stderr: 36 | container_of(ptr, type, member) 2024-04-24T23:18:43.943 INFO:tasks.workunit.client.0.smithi006.stderr: | ^~~~~~~~~~~~ 2024-04-24T23:18:43.944 INFO:tasks.workunit.client.0.smithi006.stderr:list.h:39:20: note: in expansion of macro ‘list_entry’ 2024-04-24T23:18:43.944 INFO:tasks.workunit.client.0.smithi006.stderr: 39 | for (pos = list_entry((head)->next, typeof(*pos), member); \ 2024-04-24T23:18:43.944 INFO:tasks.workunit.client.0.smithi006.stderr: | ^~~~~~~~~~ 2024-04-24T23:18:43.944 INFO:tasks.workunit.client.0.smithi006.stderr:parser.c:971:41: note: in expansion of macro ‘list_for_each_entry’ 2024-04-24T23:18:43.944 INFO:tasks.workunit.client.0.smithi006.stderr: 971 | list_for_each_entry(tmp_list, 2024-04-24T23:18:43.944 INFO:tasks.workunit.client.0.smithi006.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-04-24T23:18:43.944 INFO:tasks.workunit.client.0.smithi006.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-04-24T23:18:43.944 INFO:tasks.workunit.client.0.smithi006.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-04-24T23:18:43.944 INFO:tasks.workunit.client.0.smithi006.stderr: | ^ 2024-04-24T23:18:43.944 INFO:tasks.workunit.client.0.smithi006.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-04-24T23:18:43.944 INFO:tasks.workunit.client.0.smithi006.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-04-24T23:18:43.944 INFO:tasks.workunit.client.0.smithi006.stderr: | ^~~~~~~~ 2024-04-24T23:18:43.944 INFO:tasks.workunit.client.0.smithi006.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-04-24T23:18:43.944 INFO:tasks.workunit.client.0.smithi006.stderr: 36 | container_of(ptr, type, member) 2024-04-24T23:18:43.945 INFO:tasks.workunit.client.0.smithi006.stderr: | ^~~~~~~~~~~~ 2024-04-24T23:18:43.945 INFO:tasks.workunit.client.0.smithi006.stderr:list.h:41:20: note: in expansion of macro ‘list_entry’ 2024-04-24T23:18:43.945 INFO:tasks.workunit.client.0.smithi006.stderr: 41 | pos = list_entry(pos->member.next, typeof(*pos), member)) 2024-04-24T23:18:43.945 INFO:tasks.workunit.client.0.smithi006.stderr: | ^~~~~~~~~~ 2024-04-24T23:18:43.945 INFO:tasks.workunit.client.0.smithi006.stderr:parser.c:971:41: note: in expansion of macro ‘list_for_each_entry’ 2024-04-24T23:18:43.945 INFO:tasks.workunit.client.0.smithi006.stderr: 971 | list_for_each_entry(tmp_list, 2024-04-24T23:18:43.945 INFO:tasks.workunit.client.0.smithi006.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-04-24T23:18:43.945 INFO:tasks.workunit.client.0.smithi006.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-04-24T23:18:43.945 INFO:tasks.workunit.client.0.smithi006.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-04-24T23:18:43.945 INFO:tasks.workunit.client.0.smithi006.stderr: | ^ 2024-04-24T23:18:43.945 INFO:tasks.workunit.client.0.smithi006.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-04-24T23:18:43.945 INFO:tasks.workunit.client.0.smithi006.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-04-24T23:18:43.945 INFO:tasks.workunit.client.0.smithi006.stderr: | ^~~~~~~~ 2024-04-24T23:18:43.945 INFO:tasks.workunit.client.0.smithi006.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-04-24T23:18:43.946 INFO:tasks.workunit.client.0.smithi006.stderr: 36 | container_of(ptr, type, member) 2024-04-24T23:18:43.946 INFO:tasks.workunit.client.0.smithi006.stderr: | ^~~~~~~~~~~~ 2024-04-24T23:18:43.946 INFO:tasks.workunit.client.0.smithi006.stderr:list.h:39:20: note: in expansion of macro ‘list_entry’ 2024-04-24T23:18:43.946 INFO:tasks.workunit.client.0.smithi006.stderr: 39 | for (pos = list_entry((head)->next, typeof(*pos), member); \ 2024-04-24T23:18:43.946 INFO:tasks.workunit.client.0.smithi006.stderr: | ^~~~~~~~~~ 2024-04-24T23:18:43.946 INFO:tasks.workunit.client.0.smithi006.stderr:parser.c:980:41: note: in expansion of macro ‘list_for_each_entry’ 2024-04-24T23:18:43.946 INFO:tasks.workunit.client.0.smithi006.stderr: 980 | list_for_each_entry(tmp_list, 2024-04-24T23:18:43.946 INFO:tasks.workunit.client.0.smithi006.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-04-24T23:18:43.946 INFO:tasks.workunit.client.0.smithi006.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-04-24T23:18:43.946 INFO:tasks.workunit.client.0.smithi006.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-04-24T23:18:43.946 INFO:tasks.workunit.client.0.smithi006.stderr: | ^ 2024-04-24T23:18:43.946 INFO:tasks.workunit.client.0.smithi006.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-04-24T23:18:43.946 INFO:tasks.workunit.client.0.smithi006.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-04-24T23:18:43.946 INFO:tasks.workunit.client.0.smithi006.stderr: | ^~~~~~~~ 2024-04-24T23:18:43.946 INFO:tasks.workunit.client.0.smithi006.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-04-24T23:18:43.947 INFO:tasks.workunit.client.0.smithi006.stderr: 36 | container_of(ptr, type, member) 2024-04-24T23:18:43.947 INFO:tasks.workunit.client.0.smithi006.stderr: | ^~~~~~~~~~~~ 2024-04-24T23:18:43.947 INFO:tasks.workunit.client.0.smithi006.stderr:list.h:41:20: note: in expansion of macro ‘list_entry’ 2024-04-24T23:18:43.947 INFO:tasks.workunit.client.0.smithi006.stderr: 41 | pos = list_entry(pos->member.next, typeof(*pos), member)) 2024-04-24T23:18:43.947 INFO:tasks.workunit.client.0.smithi006.stderr: | ^~~~~~~~~~ 2024-04-24T23:18:43.947 INFO:tasks.workunit.client.0.smithi006.stderr:parser.c:980:41: note: in expansion of macro ‘list_for_each_entry’ 2024-04-24T23:18:43.947 INFO:tasks.workunit.client.0.smithi006.stderr: 980 | list_for_each_entry(tmp_list, 2024-04-24T23:18:43.947 INFO:tasks.workunit.client.0.smithi006.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-04-24T23:18:43.996 INFO:tasks.workunit.client.0.smithi006.stderr:parser.c: In function ‘get_optstr’: 2024-04-24T23:18:43.996 INFO:tasks.workunit.client.0.smithi006.stderr:parser.c:206:23: warning: ‘strnlen’ specified bound 4095 exceeds source size 1024 [-Wstringop-overread] 2024-04-24T23:18:43.996 INFO:tasks.workunit.client.0.smithi006.stderr: 206 | len = strnlen(temp, 4095) + 1; 2024-04-24T23:18:43.997 INFO:tasks.workunit.client.0.smithi006.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-04-24T23:18:43.997 INFO:tasks.workunit.client.0.smithi006.stderr:parser.c:200:14: note: source object declared here 2024-04-24T23:18:43.997 INFO:tasks.workunit.client.0.smithi006.stderr: 200 | char temp[BUFSIZE]; 2024-04-24T23:18:43.997 INFO:tasks.workunit.client.0.smithi006.stderr: | ^~~~ 2024-04-24T23:18:44.004 INFO:tasks.workunit.client.0.smithi006.stderr:parser.c: In function ‘size64_convert’: 2024-04-24T23:18:44.004 INFO:tasks.workunit.client.0.smithi006.stderr:parser.c:119:9: warning: ‘memcpy’ reading 3 bytes from a region of size 2 [-Wstringop-overread] 2024-04-24T23:18:44.004 INFO:tasks.workunit.client.0.smithi006.stderr: 119 | memcpy(unit, "\0", 3); 2024-04-24T23:18:44.004 INFO:tasks.workunit.client.0.smithi006.stderr: | ^~~~~~~~~~~~~~~~~~~~~ 2024-04-24T23:18:44.365 INFO:tasks.workunit.client.0.smithi006.stdout:mv -f .deps/parser.Tpo .deps/parser.Po 2024-04-24T23:18:44.371 INFO:tasks.workunit.client.0.smithi006.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_fc.o -MD -MP -MF .deps/ffsb_fc.Tpo -c -o ffsb_fc.o ffsb_fc.c 2024-04-24T23:18:44.474 INFO:tasks.workunit.client.0.smithi006.stdout:mv -f .deps/ffsb_fc.Tpo .deps/ffsb_fc.Po 2024-04-24T23:18:44.481 INFO:tasks.workunit.client.0.smithi006.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_stats.o -MD -MP -MF .deps/ffsb_stats.Tpo -c -o ffsb_stats.o ffsb_stats.c 2024-04-24T23:18:44.627 INFO:tasks.workunit.client.0.smithi006.stdout:mv -f .deps/ffsb_stats.Tpo .deps/ffsb_stats.Po 2024-04-24T23:18:44.630 INFO:tasks.workunit.client.0.smithi006.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT list.o -MD -MP -MF .deps/list.Tpo -c -o list.o list.c 2024-04-24T23:18:44.689 INFO:tasks.workunit.client.0.smithi006.stdout:mv -f .deps/list.Tpo .deps/list.Po 2024-04-24T23:18:44.697 INFO:tasks.workunit.client.0.smithi006.stdout:gcc -D_GNU_SOURCE -g -O2 -D_REENTRANT -o ffsb fileops.o rand.o main.o fh.o filelist.o metaops.o rwlock.o cirlist.o rbt.o ffsb_tg.o ffsb_fs.o ffsb_thread.o ffsb_op.o util.o parser.o ffsb_fc.o ffsb_stats.o list.o -lpthread -lm 2024-04-24T23:18:44.768 INFO:tasks.workunit.client.0.smithi006.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb' 2024-04-24T23:18:44.769 INFO:tasks.workunit.client.0.smithi006.stderr:+ cd .. 2024-04-24T23:18:44.769 INFO:tasks.workunit.client.0.smithi006.stderr:+ mkdir tmp 2024-04-24T23:18:44.774 INFO:tasks.workunit.client.0.smithi006.stderr:+ cd tmp 2024-04-24T23:18:44.775 INFO:tasks.workunit.client.0.smithi006.stderr:+ for f in $mydir/*.ffsb 2024-04-24T23:18:44.775 INFO:tasks.workunit.client.0.smithi006.stderr:+ ../ffsb/ffsb /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/random_write.32.ffsb 2024-04-24T23:18:44.777 INFO:tasks.workunit.client.0.smithi006.stdout:FFSB version 6.0-RC2 started 2024-04-24T23:18:44.777 INFO:tasks.workunit.client.0.smithi006.stdout: 2024-04-24T23:18:44.786 INFO:tasks.workunit.client.0.smithi006.stdout:benchmark time = 300 2024-04-24T23:18:44.786 INFO:tasks.workunit.client.0.smithi006.stdout:ThreadGroup 0 2024-04-24T23:18:44.786 INFO:tasks.workunit.client.0.smithi006.stdout:================ 2024-04-24T23:18:44.786 INFO:tasks.workunit.client.0.smithi006.stdout: num_threads = 32 2024-04-24T23:18:44.786 INFO:tasks.workunit.client.0.smithi006.stdout: 2024-04-24T23:18:44.786 INFO:tasks.workunit.client.0.smithi006.stdout: read_random = off 2024-04-24T23:18:44.786 INFO:tasks.workunit.client.0.smithi006.stdout: read_size = 0 (0B) 2024-04-24T23:18:44.786 INFO:tasks.workunit.client.0.smithi006.stdout: read_blocksize = 0 (0B) 2024-04-24T23:18:44.786 INFO:tasks.workunit.client.0.smithi006.stdout: read_skip = off 2024-04-24T23:18:44.787 INFO:tasks.workunit.client.0.smithi006.stdout: read_skipsize = 0 (0B) 2024-04-24T23:18:44.787 INFO:tasks.workunit.client.0.smithi006.stdout: 2024-04-24T23:18:44.787 INFO:tasks.workunit.client.0.smithi006.stdout: write_random = on 2024-04-24T23:18:44.787 INFO:tasks.workunit.client.0.smithi006.stdout: write_size = 5242880 (5MB) 2024-04-24T23:18:44.787 INFO:tasks.workunit.client.0.smithi006.stdout: fsync_file = 0 2024-04-24T23:18:44.787 INFO:tasks.workunit.client.0.smithi006.stdout: write_blocksize = 4096 (4KB) 2024-04-24T23:18:44.787 INFO:tasks.workunit.client.0.smithi006.stdout: wait time = 0 2024-04-24T23:18:44.787 INFO:tasks.workunit.client.0.smithi006.stdout: 2024-04-24T23:18:44.787 INFO:tasks.workunit.client.0.smithi006.stdout: op weights 2024-04-24T23:18:44.787 INFO:tasks.workunit.client.0.smithi006.stdout: read = 0 (0.00%) 2024-04-24T23:18:44.787 INFO:tasks.workunit.client.0.smithi006.stdout: readall = 0 (0.00%) 2024-04-24T23:18:44.787 INFO:tasks.workunit.client.0.smithi006.stdout: write = 1 (100.00%) 2024-04-24T23:18:44.787 INFO:tasks.workunit.client.0.smithi006.stdout: create = 0 (0.00%) 2024-04-24T23:18:44.787 INFO:tasks.workunit.client.0.smithi006.stdout: append = 0 (0.00%) 2024-04-24T23:18:44.788 INFO:tasks.workunit.client.0.smithi006.stdout: delete = 0 (0.00%) 2024-04-24T23:18:44.788 INFO:tasks.workunit.client.0.smithi006.stdout: metaop = 0 (0.00%) 2024-04-24T23:18:44.788 INFO:tasks.workunit.client.0.smithi006.stdout: createdir = 0 (0.00%) 2024-04-24T23:18:44.788 INFO:tasks.workunit.client.0.smithi006.stdout: stat = 0 (0.00%) 2024-04-24T23:18:44.788 INFO:tasks.workunit.client.0.smithi006.stdout: writeall = 0 (0.00%) 2024-04-24T23:18:44.788 INFO:tasks.workunit.client.0.smithi006.stdout: writeall_fsync = 0 (0.00%) 2024-04-24T23:18:44.788 INFO:tasks.workunit.client.0.smithi006.stdout: open_close = 0 (0.00%) 2024-04-24T23:18:44.788 INFO:tasks.workunit.client.0.smithi006.stdout: write_fsync = 0 (0.00%) 2024-04-24T23:18:44.788 INFO:tasks.workunit.client.0.smithi006.stdout: create_fsync = 0 (0.00%) 2024-04-24T23:18:44.788 INFO:tasks.workunit.client.0.smithi006.stdout: append_fsync = 0 (0.00%) 2024-04-24T23:18:44.788 INFO:tasks.workunit.client.0.smithi006.stdout: 2024-04-24T23:18:44.788 INFO:tasks.workunit.client.0.smithi006.stdout:FileSystem . 2024-04-24T23:18:44.788 INFO:tasks.workunit.client.0.smithi006.stdout:========== 2024-04-24T23:18:44.788 INFO:tasks.workunit.client.0.smithi006.stdout: num_dirs = 0 2024-04-24T23:18:44.788 INFO:tasks.workunit.client.0.smithi006.stdout: starting files = 128 2024-04-24T23:18:44.789 INFO:tasks.workunit.client.0.smithi006.stdout: 2024-04-24T23:18:44.789 INFO:tasks.workunit.client.0.smithi006.stdout: min file size = 104857600 (100MB) 2024-04-24T23:18:44.789 INFO:tasks.workunit.client.0.smithi006.stdout: max file size = 104857600 (100MB) 2024-04-24T23:18:44.789 INFO:tasks.workunit.client.0.smithi006.stdout: directio = off 2024-04-24T23:18:44.789 INFO:tasks.workunit.client.0.smithi006.stdout: alignedio = on 2024-04-24T23:18:44.789 INFO:tasks.workunit.client.0.smithi006.stdout: bufferedio = off 2024-04-24T23:18:44.789 INFO:tasks.workunit.client.0.smithi006.stdout: 2024-04-24T23:18:44.789 INFO:tasks.workunit.client.0.smithi006.stdout: aging is off 2024-04-24T23:18:44.790 INFO:tasks.workunit.client.0.smithi006.stdout: current utilization = 0.04% 2024-04-24T23:18:44.790 INFO:tasks.workunit.client.0.smithi006.stdout: 2024-04-24T23:18:44.790 INFO:tasks.workunit.client.0.smithi006.stdout:checking existing fs: . 2024-04-24T23:18:44.790 INFO:tasks.workunit.client.0.smithi006.stdout:recreating new fileset 2024-04-24T23:18:44.790 INFO:tasks.workunit.client.0.smithi006.stderr:opendir: No such file or directory 2024-04-24T23:18:45.248 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:18:45.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:45.450+0000 7f435f200640 1 Processor -- start 2024-04-24T23:18:45.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:45.450+0000 7f435f200640 1 -- start start 2024-04-24T23:18:45.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:45.452+0000 7f435f200640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f43580ccc50 0x7f435806db80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:18:45.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:45.452+0000 7f435f200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f435806e0c0 0x7f4358154840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:18:45.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:45.452+0000 7f435f200640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f435815e550 con 0x7f43580cc870 2024-04-24T23:18:45.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:45.452+0000 7f435f200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f435815e810 con 0x7f435806e0c0 2024-04-24T23:18:45.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:45.452+0000 7f435f200640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f435815eaf0 con 0x7f43580ccc50 2024-04-24T23:18:45.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:45.453+0000 7f435de00640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f43580cc870 0x7f435815b300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.6:42562/0 (socket says 172.21.15.6:42562) 2024-04-24T23:18:45.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:45.453+0000 7f435de00640 1 -- 172.21.15.6:0/3665224990 learned_addr learned my addr 172.21.15.6:0/3665224990 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:18:45.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:45.453+0000 7f435d400640 1 --2- 172.21.15.6:0/3665224990 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f43580ccc50 0x7f435806db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:18:45.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:45.453+0000 7f435e800640 1 --2- 172.21.15.6:0/3665224990 >> v2:172.21.15.6:3300/0 conn(0x7f435806e0c0 0x7f4358154840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:18:45.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:45.453+0000 7f435ca00640 1 -- 172.21.15.6:0/3665224990 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1625823716 0 0) 0x7f435815e550 con 0x7f43580cc870 2024-04-24T23:18:45.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:45.453+0000 7f435ca00640 1 -- 172.21.15.6:0/3665224990 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4344003520 con 0x7f43580cc870 2024-04-24T23:18:45.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:45.453+0000 7f435d400640 1 -- 172.21.15.6:0/3665224990 >> v2:172.21.15.6:3300/0 conn(0x7f435806e0c0 msgr2=0x7f4358154840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:18:45.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:45.453+0000 7f435d400640 1 --2- 172.21.15.6:0/3665224990 >> v2:172.21.15.6:3300/0 conn(0x7f435806e0c0 0x7f4358154840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:45.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:45.453+0000 7f435d400640 1 -- 172.21.15.6:0/3665224990 >> v1:172.21.15.190:6789/0 conn(0x7f43580cc870 legacy=0x7f435815b300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:18:45.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:45.453+0000 7f435d400640 1 -- 172.21.15.6:0/3665224990 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f435815ab00 con 0x7f43580ccc50 2024-04-24T23:18:45.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:45.454+0000 7f435d400640 1 --2- 172.21.15.6:0/3665224990 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f43580ccc50 0x7f435806db80 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f43480097a0 tx=0x7f434802a210 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dab2d5aa3b0b739b server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:18:45.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:45.455+0000 7f435ca00640 1 -- 172.21.15.6:0/3665224990 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4348037070 con 0x7f43580ccc50 2024-04-24T23:18:45.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:45.455+0000 7f435ca00640 1 -- 172.21.15.6:0/3665224990 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4348032440 con 0x7f43580ccc50 2024-04-24T23:18:45.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:45.455+0000 7f435ca00640 1 -- 172.21.15.6:0/3665224990 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4348032810 con 0x7f43580ccc50 2024-04-24T23:18:45.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:45.455+0000 7f435f200640 1 -- 172.21.15.6:0/3665224990 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f43580ccc50 msgr2=0x7f435806db80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:18:45.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:45.455+0000 7f435f200640 1 --2- 172.21.15.6:0/3665224990 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f43580ccc50 0x7f435806db80 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f43480097a0 tx=0x7f434802a210 comp rx=0 tx=0).stop 2024-04-24T23:18:45.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:45.455+0000 7f435f200640 1 -- 172.21.15.6:0/3665224990 shutdown_connections 2024-04-24T23:18:45.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:45.455+0000 7f435f200640 1 --2- 172.21.15.6:0/3665224990 >> v2:172.21.15.6:3300/0 conn(0x7f435806e0c0 0x7f4358154840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:45.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:45.455+0000 7f435f200640 1 --2- 172.21.15.6:0/3665224990 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f43580ccc50 0x7f435806db80 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:45.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:45.455+0000 7f435f200640 1 -- 172.21.15.6:0/3665224990 >> 172.21.15.6:0/3665224990 conn(0x7f435806ed80 msgr2=0x7f435806f190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:18:45.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:45.455+0000 7f435f200640 1 -- 172.21.15.6:0/3665224990 shutdown_connections 2024-04-24T23:18:45.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:45.455+0000 7f435f200640 1 -- 172.21.15.6:0/3665224990 wait complete. 2024-04-24T23:18:45.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:45.457+0000 7f435f200640 1 Processor -- start 2024-04-24T23:18:45.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:45.457+0000 7f435f200640 1 -- start start 2024-04-24T23:18:45.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:45.458+0000 7f435f200640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f435806e0c0 0x7f43580c5e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:18:45.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:45.458+0000 7f435f200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f43580cc870 0x7f4358150c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:18:45.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:45.458+0000 7f435f200640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f43580ccc50 0x7f435806b350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:18:45.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:45.458+0000 7f435f200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f435815edd0 con 0x7f43580cc870 2024-04-24T23:18:45.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:45.458+0000 7f435f200640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f43580c6be0 con 0x7f43580ccc50 2024-04-24T23:18:45.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:45.458+0000 7f435f200640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f435815dfe0 con 0x7f435806e0c0 2024-04-24T23:18:45.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:45.458+0000 7f435d400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f43580cc870 0x7f4358150c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:18:45.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:45.458+0000 7f435d400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f43580cc870 0x7f4358150c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57964/0 (socket says 172.21.15.6:57964) 2024-04-24T23:18:45.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:45.458+0000 7f435d400640 1 -- 172.21.15.6:0/471204036 learned_addr learned my addr 172.21.15.6:0/471204036 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:18:45.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:45.458+0000 7f435de00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f435806e0c0 0x7f43580c5e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:18:45.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:45.458+0000 7f435e800640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f43580ccc50 0x7f435806b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:18:45.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:45.459+0000 7f435d400640 1 -- 172.21.15.6:0/471204036 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f435806e0c0 msgr2=0x7f43580c5e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:18:45.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:45.459+0000 7f435d400640 1 --2- 172.21.15.6:0/471204036 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f435806e0c0 0x7f43580c5e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:45.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:45.459+0000 7f435d400640 1 -- 172.21.15.6:0/471204036 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f43580ccc50 msgr2=0x7f435806b350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:18:45.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:45.459+0000 7f435d400640 1 --2- 172.21.15.6:0/471204036 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f43580ccc50 0x7f435806b350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:45.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:45.459+0000 7f435d400640 1 -- 172.21.15.6:0/471204036 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4358148e50 con 0x7f43580cc870 2024-04-24T23:18:45.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:45.459+0000 7f435d400640 1 --2- 172.21.15.6:0/471204036 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f43580cc870 0x7f4358150c00 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f4348035040 tx=0x7f43480046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:18:45.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:45.459+0000 7f434e000640 1 -- 172.21.15.6:0/471204036 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4348041030 con 0x7f43580cc870 2024-04-24T23:18:45.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:45.459+0000 7f434e000640 1 -- 172.21.15.6:0/471204036 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f434803d4e0 con 0x7f43580cc870 2024-04-24T23:18:45.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:45.459+0000 7f435f200640 1 -- 172.21.15.6:0/471204036 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43581f0530 con 0x7f43580cc870 2024-04-24T23:18:45.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:45.459+0000 7f434e000640 1 -- 172.21.15.6:0/471204036 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4348037040 con 0x7f43580cc870 2024-04-24T23:18:45.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:45.460+0000 7f435f200640 1 -- 172.21.15.6:0/471204036 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43581f0af0 con 0x7f43580cc870 2024-04-24T23:18:45.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:45.461+0000 7f435f200640 1 -- 172.21.15.6:0/471204036 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4358075b30 con 0x7f43580cc870 2024-04-24T23:18:45.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:45.462+0000 7f434e000640 1 -- 172.21.15.6:0/471204036 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f4348044070 con 0x7f43580cc870 2024-04-24T23:18:45.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:45.462+0000 7f434e000640 1 --2- 172.21.15.6:0/471204036 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f433c06be30 0x7f433c06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:18:45.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:45.463+0000 7f434e000640 1 -- 172.21.15.6:0/471204036 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6676+0+0 (secure 0 0 0) 0x7f43480ab8c0 con 0x7f43580cc870 2024-04-24T23:18:45.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:45.463+0000 7f435de00640 1 --2- 172.21.15.6:0/471204036 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f433c06be30 0x7f433c06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:18:45.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:45.463+0000 7f435de00640 1 --2- 172.21.15.6:0/471204036 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f433c06be30 0x7f433c06e2f0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f4350001560 tx=0x7f4350001a90 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:18:45.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:45.467+0000 7f434e000640 1 -- 172.21.15.6:0/471204036 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f43480797c0 con 0x7f43580cc870 2024-04-24T23:18:45.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:45.713+0000 7f435f200640 1 -- 172.21.15.6:0/471204036 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f435806db80 con 0x7f43580cc870 2024-04-24T23:18:45.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:45.714+0000 7f434e000640 1 -- 172.21.15.6:0/471204036 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6078 (secure 0 0 0) 0x7f434807dac0 con 0x7f43580cc870 2024-04-24T23:18:45.717 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:18:45.717 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:18:42.788711+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4282,"mds_2":4235,"mds_3":4232,"mds_4":4197},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4197":{"gid":4197,"name":"a","rank":4,"incarnation":19,"state":"up:rejoin","state_seq":51,"addr":"172.21.15.6:6834/105267896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":105267896},{"type":"v1","addr":"172.21.15.6:6834","nonce":105267896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4232":{"gid":4232,"name":"b","rank":3,"incarnation":15,"state":"up:active","state_seq":15,"addr":"172.21.15.190:6838/2032280158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2032280158},{"type":"v1","addr":"172.21.15.190:6838","nonce":2032280158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4235":{"gid":4235,"name":"d","rank":2,"incarnation":13,"state":"up:active","state_seq":14,"addr":"172.21.15.190:6839/1572872451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":1572872451},{"type":"v1","addr":"172.21.15.190:6839","nonce":1572872451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4282":{"gid":4282,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":12,"addr":"172.21.15.190:6837/580315269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":580315269},{"type":"v1","addr":"172.21.15.190:6837","nonce":580315269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/489092393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":489092393},{"type":"v1","addr":"172.21.15.6:6835","nonce":489092393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4306,"qdb_cluster":[4306,4282,4235,4232]},"id":1}]} 2024-04-24T23:18:45.717 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 22 2024-04-24T23:18:45.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:45.717+0000 7f435f200640 1 -- 172.21.15.6:0/471204036 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f433c06be30 msgr2=0x7f433c06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:18:45.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:45.717+0000 7f435f200640 1 --2- 172.21.15.6:0/471204036 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f433c06be30 0x7f433c06e2f0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f4350001560 tx=0x7f4350001a90 comp rx=0 tx=0).stop 2024-04-24T23:18:45.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:45.717+0000 7f435f200640 1 -- 172.21.15.6:0/471204036 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f43580cc870 msgr2=0x7f4358150c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:18:45.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:45.717+0000 7f435f200640 1 --2- 172.21.15.6:0/471204036 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f43580cc870 0x7f4358150c00 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f4348035040 tx=0x7f43480046c0 comp rx=0 tx=0).stop 2024-04-24T23:18:45.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:45.717+0000 7f435f200640 1 -- 172.21.15.6:0/471204036 shutdown_connections 2024-04-24T23:18:45.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:45.717+0000 7f435f200640 1 --2- 172.21.15.6:0/471204036 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f433c06be30 0x7f433c06e2f0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:45.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:45.717+0000 7f435f200640 1 --2- 172.21.15.6:0/471204036 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f43580ccc50 0x7f435806b350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:45.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:45.717+0000 7f435f200640 1 --2- 172.21.15.6:0/471204036 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f43580cc870 0x7f4358150c00 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:45.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:45.717+0000 7f435f200640 1 --2- 172.21.15.6:0/471204036 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f435806e0c0 0x7f43580c5e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:45.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:45.717+0000 7f435f200640 1 -- 172.21.15.6:0/471204036 >> 172.21.15.6:0/471204036 conn(0x7f435806ed80 msgr2=0x7f4358151460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:18:45.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:45.717+0000 7f435f200640 1 -- 172.21.15.6:0/471204036 shutdown_connections 2024-04-24T23:18:45.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:45.717+0000 7f435f200640 1 -- 172.21.15.6:0/471204036 wait complete. 2024-04-24T23:18:45.737 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-24T23:18:47.738 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:18:47.830 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-04-24T23:18:47.828+0000 7fef20000640 -1 quiesce.mds.0 failed to decode message of type 1286 v1: End of buffer [buffer:2] 2024-04-24T23:18:47.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:47.938+0000 7f0df2400640 1 Processor -- start 2024-04-24T23:18:47.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:47.939+0000 7f0df2400640 1 -- start start 2024-04-24T23:18:47.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:47.941+0000 7f0df2400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f0dec15e150 0x7f0dec1664f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:18:47.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:47.941+0000 7f0df2400640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0dec15ad00 con 0x7f0dec154ca0 2024-04-24T23:18:47.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:47.941+0000 7f0df2400640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0dec15e580 con 0x7f0dec15b7a0 2024-04-24T23:18:47.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:47.941+0000 7f0df2400640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f0dec155080 con 0x7f0dec15e150 2024-04-24T23:18:47.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:47.941+0000 7f0deb400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f0dec154ca0 0x7f0dec0ca500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:43242/0 (socket says 172.21.15.6:43242) 2024-04-24T23:18:47.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:47.941+0000 7f0deb400640 1 -- 172.21.15.6:0/1589479546 learned_addr learned my addr 172.21.15.6:0/1589479546 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:18:47.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:47.941+0000 7f0debe00640 1 --2- 172.21.15.6:0/1589479546 >> v2:172.21.15.6:3300/0 conn(0x7f0dec15e150 0x7f0dec1664f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:18:47.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:47.942+0000 7f0dea000640 1 -- 172.21.15.6:0/1589479546 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4130026684 0 0) 0x7f0dec15ad00 con 0x7f0dec154ca0 2024-04-24T23:18:47.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:47.942+0000 7f0dea000640 1 -- 172.21.15.6:0/1589479546 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0dd0003520 con 0x7f0dec154ca0 2024-04-24T23:18:47.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:47.942+0000 7f0debe00640 1 -- 172.21.15.6:0/1589479546 >> v1:172.21.15.190:6789/0 conn(0x7f0dec15b7a0 legacy=0x7f0dec0caed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:18:47.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:47.942+0000 7f0debe00640 1 -- 172.21.15.6:0/1589479546 >> v1:172.21.15.6:6789/0 conn(0x7f0dec154ca0 legacy=0x7f0dec0ca500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:18:47.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:47.942+0000 7f0debe00640 1 -- 172.21.15.6:0/1589479546 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0dec06c440 con 0x7f0dec15e150 2024-04-24T23:18:47.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:47.943+0000 7f0debe00640 1 --2- 172.21.15.6:0/1589479546 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0dec15e150 0x7f0dec1664f0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f0de4009d90 tx=0x7f0de402a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=100584cdf48e1fa2 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:18:47.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:47.944+0000 7f0dea000640 1 -- 172.21.15.6:0/1589479546 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0de4007740 con 0x7f0dec15e150 2024-04-24T23:18:47.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:47.944+0000 7f0dea000640 1 -- 172.21.15.6:0/1589479546 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0de40078e0 con 0x7f0dec15e150 2024-04-24T23:18:47.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:47.944+0000 7f0dea000640 1 -- 172.21.15.6:0/1589479546 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0de4007c90 con 0x7f0dec15e150 2024-04-24T23:18:47.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:47.944+0000 7f0df2400640 1 -- 172.21.15.6:0/1589479546 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0dec15e150 msgr2=0x7f0dec1664f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:18:47.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:47.944+0000 7f0df2400640 1 --2- 172.21.15.6:0/1589479546 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0dec15e150 0x7f0dec1664f0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f0de4009d90 tx=0x7f0de402a660 comp rx=0 tx=0).stop 2024-04-24T23:18:47.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:47.944+0000 7f0df2400640 1 -- 172.21.15.6:0/1589479546 shutdown_connections 2024-04-24T23:18:47.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:47.944+0000 7f0df2400640 1 --2- 172.21.15.6:0/1589479546 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0dec15e150 0x7f0dec1664f0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:47.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:47.944+0000 7f0df2400640 1 -- 172.21.15.6:0/1589479546 >> 172.21.15.6:0/1589479546 conn(0x7f0dec070280 msgr2=0x7f0dec06f320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:18:47.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:47.945+0000 7f0df2400640 1 -- 172.21.15.6:0/1589479546 shutdown_connections 2024-04-24T23:18:47.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:47.945+0000 7f0df2400640 1 -- 172.21.15.6:0/1589479546 wait complete. 2024-04-24T23:18:47.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:47.947+0000 7f0df2400640 1 Processor -- start 2024-04-24T23:18:47.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:47.947+0000 7f0df2400640 1 -- start start 2024-04-24T23:18:47.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:47.949+0000 7f0df2400640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0dec154ca0 0x7f0dec1e7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:18:47.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:47.949+0000 7f0df2400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0dec15b7a0 0x7f0dec15c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:18:47.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:47.949+0000 7f0df2400640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0dec15e150 0x7f0dec14cc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:18:47.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:47.949+0000 7f0df2400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f0dec15afa0 con 0x7f0dec15b7a0 2024-04-24T23:18:47.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:47.949+0000 7f0df2400640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f0dec06c1f0 con 0x7f0dec154ca0 2024-04-24T23:18:47.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:47.949+0000 7f0df2400640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f0dec15b280 con 0x7f0dec15e150 2024-04-24T23:18:47.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:47.949+0000 7f0deaa00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0dec15b7a0 0x7f0dec15c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:18:47.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:47.949+0000 7f0deaa00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0dec15b7a0 0x7f0dec15c650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57980/0 (socket says 172.21.15.6:57980) 2024-04-24T23:18:47.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:47.949+0000 7f0deaa00640 1 -- 172.21.15.6:0/308518897 learned_addr learned my addr 172.21.15.6:0/308518897 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:18:47.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:47.949+0000 7f0debe00640 1 --2- 172.21.15.6:0/308518897 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0dec15e150 0x7f0dec14cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:18:47.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:47.949+0000 7f0deb400640 1 --2- 172.21.15.6:0/308518897 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0dec154ca0 0x7f0dec1e7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:18:47.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:47.950+0000 7f0deaa00640 1 -- 172.21.15.6:0/308518897 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0dec15e150 msgr2=0x7f0dec14cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:18:47.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:47.950+0000 7f0deaa00640 1 --2- 172.21.15.6:0/308518897 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0dec15e150 0x7f0dec14cc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:47.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:47.950+0000 7f0deaa00640 1 -- 172.21.15.6:0/308518897 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0dec154ca0 msgr2=0x7f0dec1e7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:18:47.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:47.950+0000 7f0deaa00640 1 --2- 172.21.15.6:0/308518897 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0dec154ca0 0x7f0dec1e7be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:47.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:47.950+0000 7f0deaa00640 1 -- 172.21.15.6:0/308518897 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0dec1514e0 con 0x7f0dec15b7a0 2024-04-24T23:18:47.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:47.950+0000 7f0deaa00640 1 --2- 172.21.15.6:0/308518897 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0dec15b7a0 0x7f0dec15c650 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f0dd800c400 tx=0x7f0dd800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:18:47.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:47.950+0000 7f0de3400640 1 -- 172.21.15.6:0/308518897 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dd80052d0 con 0x7f0dec15b7a0 2024-04-24T23:18:47.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:47.950+0000 7f0de3400640 1 -- 172.21.15.6:0/308518897 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0dd8005490 con 0x7f0dec15b7a0 2024-04-24T23:18:47.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:47.950+0000 7f0de3400640 1 -- 172.21.15.6:0/308518897 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dd8007260 con 0x7f0dec15b7a0 2024-04-24T23:18:47.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:47.950+0000 7f0df2400640 1 -- 172.21.15.6:0/308518897 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0dec1517a0 con 0x7f0dec15b7a0 2024-04-24T23:18:47.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:47.951+0000 7f0df2400640 1 -- 172.21.15.6:0/308518897 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0dec15be50 con 0x7f0dec15b7a0 2024-04-24T23:18:47.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:47.952+0000 7f0df2400640 1 -- 172.21.15.6:0/308518897 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0dec06e1e0 con 0x7f0dec15b7a0 2024-04-24T23:18:47.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:47.953+0000 7f0de3400640 1 -- 172.21.15.6:0/308518897 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f0dd8007400 con 0x7f0dec15b7a0 2024-04-24T23:18:47.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:47.954+0000 7f0de3400640 1 --2- 172.21.15.6:0/308518897 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f0dcc06be30 0x7f0dcc06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:18:47.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:47.954+0000 7f0de3400640 1 -- 172.21.15.6:0/308518897 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6676+0+0 (secure 0 0 0) 0x7f0dd808a7b0 con 0x7f0dec15b7a0 2024-04-24T23:18:47.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:47.954+0000 7f0deb400640 1 --2- 172.21.15.6:0/308518897 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f0dcc06be30 0x7f0dcc06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:18:47.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:47.955+0000 7f0deb400640 1 --2- 172.21.15.6:0/308518897 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f0dcc06be30 0x7f0dcc06e2f0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f0ddc005880 tx=0x7f0ddc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:18:47.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:47.958+0000 7f0de3400640 1 -- 172.21.15.6:0/308518897 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f0dec06e1e0 con 0x7f0dec15b7a0 2024-04-24T23:18:48.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:48.207+0000 7f0df2400640 1 -- 172.21.15.6:0/308518897 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0dec0767b0 con 0x7f0dec15b7a0 2024-04-24T23:18:48.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:48.208+0000 7f0de3400640 1 -- 172.21.15.6:0/308518897 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+6083 (secure 0 0 0) 0x7f0dd80586a0 con 0x7f0dec15b7a0 2024-04-24T23:18:48.211 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:18:48.211 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:18:47.823843+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4282,"mds_2":4235,"mds_3":4232,"mds_4":4197},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4197":{"gid":4197,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":53,"addr":"172.21.15.6:6834/105267896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":105267896},{"type":"v1","addr":"172.21.15.6:6834","nonce":105267896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4232":{"gid":4232,"name":"b","rank":3,"incarnation":15,"state":"up:active","state_seq":15,"addr":"172.21.15.190:6838/2032280158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2032280158},{"type":"v1","addr":"172.21.15.190:6838","nonce":2032280158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4235":{"gid":4235,"name":"d","rank":2,"incarnation":13,"state":"up:active","state_seq":14,"addr":"172.21.15.190:6839/1572872451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":1572872451},{"type":"v1","addr":"172.21.15.190:6839","nonce":1572872451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4282":{"gid":4282,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":12,"addr":"172.21.15.190:6837/580315269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":580315269},{"type":"v1","addr":"172.21.15.190:6837","nonce":580315269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/489092393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":489092393},{"type":"v1","addr":"172.21.15.6:6835","nonce":489092393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4306,"qdb_cluster":[4197,4232,4235,4282,4306]},"id":1}]} 2024-04-24T23:18:48.211 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 23 2024-04-24T23:18:48.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:48.212+0000 7f0df2400640 1 -- 172.21.15.6:0/308518897 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f0dcc06be30 msgr2=0x7f0dcc06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:18:48.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:48.212+0000 7f0df2400640 1 --2- 172.21.15.6:0/308518897 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f0dcc06be30 0x7f0dcc06e2f0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f0ddc005880 tx=0x7f0ddc009040 comp rx=0 tx=0).stop 2024-04-24T23:18:48.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:48.212+0000 7f0df2400640 1 -- 172.21.15.6:0/308518897 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0dec15b7a0 msgr2=0x7f0dec15c650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:18:48.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:48.212+0000 7f0df2400640 1 --2- 172.21.15.6:0/308518897 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0dec15b7a0 0x7f0dec15c650 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f0dd800c400 tx=0x7f0dd800c8c0 comp rx=0 tx=0).stop 2024-04-24T23:18:48.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:48.212+0000 7f0df2400640 1 -- 172.21.15.6:0/308518897 shutdown_connections 2024-04-24T23:18:48.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:48.212+0000 7f0df2400640 1 --2- 172.21.15.6:0/308518897 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f0dcc06be30 0x7f0dcc06e2f0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:48.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:48.212+0000 7f0df2400640 1 --2- 172.21.15.6:0/308518897 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0dec15e150 0x7f0dec14cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:48.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:48.212+0000 7f0df2400640 1 --2- 172.21.15.6:0/308518897 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0dec15b7a0 0x7f0dec15c650 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:48.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:48.212+0000 7f0df2400640 1 --2- 172.21.15.6:0/308518897 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0dec154ca0 0x7f0dec1e7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:48.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:48.212+0000 7f0df2400640 1 -- 172.21.15.6:0/308518897 >> 172.21.15.6:0/308518897 conn(0x7f0dec070280 msgr2=0x7f0dec06f320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:18:48.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:48.212+0000 7f0df2400640 1 -- 172.21.15.6:0/308518897 shutdown_connections 2024-04-24T23:18:48.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:48.212+0000 7f0df2400640 1 -- 172.21.15.6:0/308518897 wait complete. 2024-04-24T23:18:48.232 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a has gained rank=4, replacing gid=4203 2024-04-24T23:18:48.232 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 3 secs before reviving mds.e 2024-04-24T23:18:51.233 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.e 2024-04-24T23:18:51.233 INFO:tasks.ceph.mds.e:Restarting daemon 2024-04-24T23:18:51.234 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i e 2024-04-24T23:18:51.241 INFO:tasks.ceph.mds.e:Started 2024-04-24T23:18:51.241 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:18:51.388 INFO:tasks.ceph.mds.e.smithi006.stdout:starting mds.e at 2024-04-24T23:18:51.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:51.419+0000 7fb362400640 1 Processor -- start 2024-04-24T23:18:51.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:51.419+0000 7fb362400640 1 -- start start 2024-04-24T23:18:51.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:51.421+0000 7fb362400640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb35c157950 0x7fb35c157d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:18:51.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:51.421+0000 7fb362400640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb35c0c8390 con 0x7fb35c151bb0 2024-04-24T23:18:51.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:51.421+0000 7fb362400640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb35c071d10 con 0x7fb35c071530 2024-04-24T23:18:51.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:51.421+0000 7fb362400640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fb35c071ff0 con 0x7fb35c157950 2024-04-24T23:18:51.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:51.421+0000 7fb35b400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fb35c151bb0 0x7fb35c0c37f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:43256/0 (socket says 172.21.15.6:43256) 2024-04-24T23:18:51.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:51.421+0000 7fb35b400640 1 -- 172.21.15.6:0/3634236849 learned_addr learned my addr 172.21.15.6:0/3634236849 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:18:51.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:51.421+0000 7fb35be00640 1 --2- 172.21.15.6:0/3634236849 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb35c157950 0x7fb35c157d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:18:51.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:51.422+0000 7fb35a000640 1 -- 172.21.15.6:0/3634236849 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2422174597 0 0) 0x7fb35c071d10 con 0x7fb35c071530 2024-04-24T23:18:51.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:51.422+0000 7fb35a000640 1 -- 172.21.15.6:0/3634236849 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb340005730 con 0x7fb35c071530 2024-04-24T23:18:51.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:51.422+0000 7fb35be00640 1 -- 172.21.15.6:0/3634236849 >> v1:172.21.15.190:6789/0 conn(0x7fb35c071530 legacy=0x7fb35c0c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:18:51.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:51.422+0000 7fb35be00640 1 -- 172.21.15.6:0/3634236849 >> v1:172.21.15.6:6789/0 conn(0x7fb35c151bb0 legacy=0x7fb35c0c37f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:18:51.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:51.422+0000 7fb35be00640 1 -- 172.21.15.6:0/3634236849 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb35c0722d0 con 0x7fb35c157950 2024-04-24T23:18:51.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:51.423+0000 7fb35be00640 1 --2- 172.21.15.6:0/3634236849 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb35c157950 0x7fb35c157d60 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fb354009dc0 tx=0x7fb35402a760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=df999af6bcb6a63f server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:18:51.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:51.423+0000 7fb35a000640 1 -- 172.21.15.6:0/3634236849 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb354007740 con 0x7fb35c157950 2024-04-24T23:18:51.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:51.424+0000 7fb35a000640 1 -- 172.21.15.6:0/3634236849 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb3540078e0 con 0x7fb35c157950 2024-04-24T23:18:51.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:51.424+0000 7fb35a000640 1 -- 172.21.15.6:0/3634236849 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb354007c90 con 0x7fb35c157950 2024-04-24T23:18:51.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:51.424+0000 7fb362400640 1 -- 172.21.15.6:0/3634236849 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb35c157950 msgr2=0x7fb35c157d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:18:51.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:51.424+0000 7fb362400640 1 --2- 172.21.15.6:0/3634236849 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb35c157950 0x7fb35c157d60 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fb354009dc0 tx=0x7fb35402a760 comp rx=0 tx=0).stop 2024-04-24T23:18:51.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:51.424+0000 7fb362400640 1 -- 172.21.15.6:0/3634236849 shutdown_connections 2024-04-24T23:18:51.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:51.424+0000 7fb362400640 1 --2- 172.21.15.6:0/3634236849 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb35c157950 0x7fb35c157d60 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:51.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:51.424+0000 7fb362400640 1 -- 172.21.15.6:0/3634236849 >> 172.21.15.6:0/3634236849 conn(0x7fb35c06b0e0 msgr2=0x7fb35c06b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:18:51.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:51.424+0000 7fb362400640 1 -- 172.21.15.6:0/3634236849 shutdown_connections 2024-04-24T23:18:51.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:51.424+0000 7fb362400640 1 -- 172.21.15.6:0/3634236849 wait complete. 2024-04-24T23:18:51.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:51.426+0000 7fb362400640 1 Processor -- start 2024-04-24T23:18:51.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:51.426+0000 7fb362400640 1 -- start start 2024-04-24T23:18:51.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:51.427+0000 7fb362400640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb35c071530 0x7fb35c1f2400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:18:51.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:51.427+0000 7fb362400640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb35c151bb0 0x7fb35c1f1a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:18:51.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:51.427+0000 7fb362400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb35c157950 0x7fb35c14e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:18:51.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:51.427+0000 7fb362400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fb35c0c7f70 con 0x7fb35c157950 2024-04-24T23:18:51.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:51.427+0000 7fb362400640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fb35c1582a0 con 0x7fb35c151bb0 2024-04-24T23:18:51.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:51.427+0000 7fb362400640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fb35c0c80f0 con 0x7fb35c071530 2024-04-24T23:18:51.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:51.428+0000 7fb35b400640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb35c071530 0x7fb35c1f2400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:18:51.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:51.428+0000 7fb35b400640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb35c071530 0x7fb35c1f2400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.6:51092/0 (socket says 172.21.15.6:51092) 2024-04-24T23:18:51.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:51.428+0000 7fb35b400640 1 -- 172.21.15.6:0/3625114296 learned_addr learned my addr 172.21.15.6:0/3625114296 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:18:51.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:51.428+0000 7fb35be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb35c157950 0x7fb35c14e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:18:51.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:51.428+0000 7fb35aa00640 1 --2- 172.21.15.6:0/3625114296 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb35c151bb0 0x7fb35c1f1a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:18:51.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:51.428+0000 7fb35b400640 1 -- 172.21.15.6:0/3625114296 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb35c151bb0 msgr2=0x7fb35c1f1a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:18:51.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:51.428+0000 7fb35b400640 1 --2- 172.21.15.6:0/3625114296 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb35c151bb0 0x7fb35c1f1a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:51.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:51.428+0000 7fb35b400640 1 -- 172.21.15.6:0/3625114296 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb35c157950 msgr2=0x7fb35c14e770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:18:51.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:51.428+0000 7fb35b400640 1 --2- 172.21.15.6:0/3625114296 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb35c157950 0x7fb35c14e770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:51.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:51.428+0000 7fb35b400640 1 -- 172.21.15.6:0/3625114296 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb35c1431c0 con 0x7fb35c071530 2024-04-24T23:18:51.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:51.428+0000 7fb35b400640 1 --2- 172.21.15.6:0/3625114296 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb35c071530 0x7fb35c1f2400 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fb34c00c400 tx=0x7fb34c00c8c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:18:51.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:51.429+0000 7fb353400640 1 -- 172.21.15.6:0/3625114296 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb34c017070 con 0x7fb35c071530 2024-04-24T23:18:51.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:51.429+0000 7fb353400640 1 -- 172.21.15.6:0/3625114296 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb34c005ce0 con 0x7fb35c071530 2024-04-24T23:18:51.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:51.429+0000 7fb353400640 1 -- 172.21.15.6:0/3625114296 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb34c0072f0 con 0x7fb35c071530 2024-04-24T23:18:51.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:51.429+0000 7fb362400640 1 -- 172.21.15.6:0/3625114296 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb35c143650 con 0x7fb35c071530 2024-04-24T23:18:51.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:51.429+0000 7fb362400640 1 -- 172.21.15.6:0/3625114296 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb35c141520 con 0x7fb35c071530 2024-04-24T23:18:51.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:51.430+0000 7fb362400640 1 -- 172.21.15.6:0/3625114296 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb35c06dd40 con 0x7fb35c071530 2024-04-24T23:18:51.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:51.431+0000 7fb353400640 1 -- 172.21.15.6:0/3625114296 <== mon.2 v2:172.21.15.190:3301/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fb34c01a070 con 0x7fb35c071530 2024-04-24T23:18:51.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:51.432+0000 7fb353400640 1 --2- 172.21.15.6:0/3625114296 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fb33c06be30 0x7fb33c06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:18:51.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:51.432+0000 7fb353400640 1 -- 172.21.15.6:0/3625114296 <== mon.2 v2:172.21.15.190:3301/0 5 ==== osd_map(27..27 src has 1..27) ==== 6676+0+0 (secure 0 0 0) 0x7fb34c08a6e0 con 0x7fb35c071530 2024-04-24T23:18:51.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:51.433+0000 7fb35aa00640 1 --2- 172.21.15.6:0/3625114296 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fb33c06be30 0x7fb33c06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:18:51.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:51.433+0000 7fb35aa00640 1 --2- 172.21.15.6:0/3625114296 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fb33c06be30 0x7fb33c06e2f0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fb35c1430d0 tx=0x7fb348009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:18:51.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:51.436+0000 7fb353400640 1 -- 172.21.15.6:0/3625114296 <== mon.2 v2:172.21.15.190:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fb35c06dd40 con 0x7fb35c071530 2024-04-24T23:18:51.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:51.678+0000 7fb362400640 1 -- 172.21.15.6:0/3625114296 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb35c0767b0 con 0x7fb35c071530 2024-04-24T23:18:51.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:51.679+0000 7fb353400640 1 -- 172.21.15.6:0/3625114296 <== mon.2 v2:172.21.15.190:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+6083 (secure 0 0 0) 0x7fb35c1431c0 con 0x7fb35c071530 2024-04-24T23:18:51.681 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:18:51.682 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:18:47.823843+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4282,"mds_2":4235,"mds_3":4232,"mds_4":4197},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4197":{"gid":4197,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":53,"addr":"172.21.15.6:6834/105267896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":105267896},{"type":"v1","addr":"172.21.15.6:6834","nonce":105267896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4232":{"gid":4232,"name":"b","rank":3,"incarnation":15,"state":"up:active","state_seq":15,"addr":"172.21.15.190:6838/2032280158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2032280158},{"type":"v1","addr":"172.21.15.190:6838","nonce":2032280158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4235":{"gid":4235,"name":"d","rank":2,"incarnation":13,"state":"up:active","state_seq":14,"addr":"172.21.15.190:6839/1572872451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":1572872451},{"type":"v1","addr":"172.21.15.190:6839","nonce":1572872451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4282":{"gid":4282,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":12,"addr":"172.21.15.190:6837/580315269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":580315269},{"type":"v1","addr":"172.21.15.190:6837","nonce":580315269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/489092393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":489092393},{"type":"v1","addr":"172.21.15.6:6835","nonce":489092393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4306,"qdb_cluster":[4197,4232,4235,4282,4306]},"id":1}]} 2024-04-24T23:18:51.682 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 23 2024-04-24T23:18:51.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:51.682+0000 7fb362400640 1 -- 172.21.15.6:0/3625114296 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fb33c06be30 msgr2=0x7fb33c06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:18:51.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:51.682+0000 7fb362400640 1 --2- 172.21.15.6:0/3625114296 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fb33c06be30 0x7fb33c06e2f0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fb35c1430d0 tx=0x7fb348009040 comp rx=0 tx=0).stop 2024-04-24T23:18:51.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:51.682+0000 7fb362400640 1 -- 172.21.15.6:0/3625114296 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb35c071530 msgr2=0x7fb35c1f2400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:18:51.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:51.682+0000 7fb362400640 1 --2- 172.21.15.6:0/3625114296 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb35c071530 0x7fb35c1f2400 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fb34c00c400 tx=0x7fb34c00c8c0 comp rx=0 tx=0).stop 2024-04-24T23:18:51.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:51.682+0000 7fb362400640 1 -- 172.21.15.6:0/3625114296 shutdown_connections 2024-04-24T23:18:51.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:51.682+0000 7fb362400640 1 --2- 172.21.15.6:0/3625114296 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fb33c06be30 0x7fb33c06e2f0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:51.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:51.682+0000 7fb362400640 1 --2- 172.21.15.6:0/3625114296 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb35c157950 0x7fb35c14e770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:51.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:51.682+0000 7fb362400640 1 --2- 172.21.15.6:0/3625114296 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb35c151bb0 0x7fb35c1f1a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:51.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:51.682+0000 7fb362400640 1 --2- 172.21.15.6:0/3625114296 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb35c071530 0x7fb35c1f2400 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:51.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:51.682+0000 7fb362400640 1 -- 172.21.15.6:0/3625114296 >> 172.21.15.6:0/3625114296 conn(0x7fb35c06b0e0 msgr2=0x7fb35c06e2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:18:51.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:51.682+0000 7fb362400640 1 -- 172.21.15.6:0/3625114296 shutdown_connections 2024-04-24T23:18:51.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:51.682+0000 7fb362400640 1 -- 172.21.15.6:0/3625114296 wait complete. 2024-04-24T23:18:51.700 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is in active, standby or standby-replay 2024-04-24T23:18:51.907 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-04-24T23:18:51.905+0000 7fef1ce00640 -1 mds.pinger is_rank_lagging: rank=4 was never sent ping request. 2024-04-24T23:18:53.701 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:18:53.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:53.994+0000 7f0230600640 1 Processor -- start 2024-04-24T23:18:53.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:53.994+0000 7f0230600640 1 -- start start 2024-04-24T23:18:53.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:53.996+0000 7f0230600640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f022815c1b0 0x7f02280cb810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:18:53.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:53.996+0000 7f0230600640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f022815e2b0 0x7f022815e760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:18:53.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:53.996+0000 7f0230600640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f02280ca900 con 0x7f02280ca520 2024-04-24T23:18:53.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:53.996+0000 7f0230600640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f02280ccf70 con 0x7f022815e2b0 2024-04-24T23:18:53.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:53.996+0000 7f0230600640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f02280cc360 con 0x7f022815c1b0 2024-04-24T23:18:53.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:53.996+0000 7f022e600640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f022815e2b0 0x7f022815e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:18:53.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:53.996+0000 7f022d200640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f022815c1b0 0x7f02280cb810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:18:53.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:53.996+0000 7f022dc00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f02280ca520 0x7f02280cd400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:43272/0 (socket says 172.21.15.6:43272) 2024-04-24T23:18:53.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:53.996+0000 7f022e600640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f022815e2b0 0x7f022815e760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:33004/0 (socket says 172.21.15.6:33004) 2024-04-24T23:18:53.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:53.996+0000 7f022d200640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f022815c1b0 0x7f02280cb810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.6:51106/0 (socket says 172.21.15.6:51106) 2024-04-24T23:18:53.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:53.996+0000 7f022dc00640 1 -- 172.21.15.6:0/3047984778 learned_addr learned my addr 172.21.15.6:0/3047984778 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:18:54.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:53.997+0000 7f022e600640 1 -- 172.21.15.6:0/3047984778 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f022815c1b0 msgr2=0x7f02280cb810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:18:54.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:53.997+0000 7f022e600640 1 --2- 172.21.15.6:0/3047984778 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f022815c1b0 0x7f02280cb810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:54.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:53.997+0000 7f022e600640 1 -- 172.21.15.6:0/3047984778 >> v1:172.21.15.6:6789/0 conn(0x7f02280ca520 legacy=0x7f02280cd400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:18:54.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:53.997+0000 7f022e600640 1 -- 172.21.15.6:0/3047984778 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02280cc640 con 0x7f022815e2b0 2024-04-24T23:18:54.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:53.997+0000 7f022d200640 1 --2- 172.21.15.6:0/3047984778 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f022815c1b0 0x7f02280cb810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:18:54.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:53.997+0000 7f0227e00640 1 -- 172.21.15.6:0/3047984778 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1466428685 0 0) 0x7f02280ca900 con 0x7f02280ca520 2024-04-24T23:18:54.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:53.998+0000 7f022e600640 1 --2- 172.21.15.6:0/3047984778 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f022815e2b0 0x7f022815e760 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f0220009d90 tx=0x7f022002a6f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ef24749648367e71 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:18:54.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:53.998+0000 7f0227e00640 1 -- 172.21.15.6:0/3047984778 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0220007740 con 0x7f022815e2b0 2024-04-24T23:18:54.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:53.998+0000 7f0227e00640 1 -- 172.21.15.6:0/3047984778 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f02200078e0 con 0x7f022815e2b0 2024-04-24T23:18:54.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:53.998+0000 7f0227e00640 1 -- 172.21.15.6:0/3047984778 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0220007cb0 con 0x7f022815e2b0 2024-04-24T23:18:54.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:53.999+0000 7f0230600640 1 -- 172.21.15.6:0/3047984778 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f022815e2b0 msgr2=0x7f022815e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:18:54.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:53.999+0000 7f0230600640 1 --2- 172.21.15.6:0/3047984778 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f022815e2b0 0x7f022815e760 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f0220009d90 tx=0x7f022002a6f0 comp rx=0 tx=0).stop 2024-04-24T23:18:54.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:53.999+0000 7f0230600640 1 -- 172.21.15.6:0/3047984778 shutdown_connections 2024-04-24T23:18:54.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:53.999+0000 7f0230600640 1 --2- 172.21.15.6:0/3047984778 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f022815e2b0 0x7f022815e760 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:54.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:53.999+0000 7f0230600640 1 --2- 172.21.15.6:0/3047984778 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f022815c1b0 0x7f02280cb810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:54.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:53.999+0000 7f0230600640 1 -- 172.21.15.6:0/3047984778 >> 172.21.15.6:0/3047984778 conn(0x7f022806c0f0 msgr2=0x7f0228066ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:18:54.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:53.999+0000 7f0230600640 1 -- 172.21.15.6:0/3047984778 shutdown_connections 2024-04-24T23:18:54.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:54.000+0000 7f0230600640 1 -- 172.21.15.6:0/3047984778 wait complete. 2024-04-24T23:18:54.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:54.001+0000 7f0230600640 1 Processor -- start 2024-04-24T23:18:54.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:54.002+0000 7f0230600640 1 -- start start 2024-04-24T23:18:54.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:54.003+0000 7f0230600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f02280ca520 0x7f02280c8fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:18:54.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:54.003+0000 7f0230600640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f022815c1b0 0x7f022806f450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:18:54.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:54.003+0000 7f0230600640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f022815e2b0 0x7f022806f990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:18:54.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:54.003+0000 7f0230600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f0228067890 con 0x7f02280ca520 2024-04-24T23:18:54.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:54.003+0000 7f0230600640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f02280ccc00 con 0x7f022815c1b0 2024-04-24T23:18:54.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:54.003+0000 7f0230600640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f0228067bc0 con 0x7f022815e2b0 2024-04-24T23:18:54.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:54.003+0000 7f022d200640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f022815c1b0 0x7f022806f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:18:54.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:54.003+0000 7f022d200640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f022815c1b0 0x7f022806f450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:33010/0 (socket says 172.21.15.6:33010) 2024-04-24T23:18:54.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:54.003+0000 7f022d200640 1 -- 172.21.15.6:0/1487379161 learned_addr learned my addr 172.21.15.6:0/1487379161 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:18:54.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:54.003+0000 7f022dc00640 1 --2- 172.21.15.6:0/1487379161 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f02280ca520 0x7f02280c8fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:18:54.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:54.003+0000 7f022e600640 1 --2- 172.21.15.6:0/1487379161 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f022815e2b0 0x7f022806f990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:18:54.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:54.004+0000 7f022d200640 1 -- 172.21.15.6:0/1487379161 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f022815e2b0 msgr2=0x7f022806f990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:18:54.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:54.004+0000 7f022d200640 1 --2- 172.21.15.6:0/1487379161 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f022815e2b0 0x7f022806f990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:54.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:54.004+0000 7f022d200640 1 -- 172.21.15.6:0/1487379161 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f02280ca520 msgr2=0x7f02280c8fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:18:54.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:54.004+0000 7f022d200640 1 --2- 172.21.15.6:0/1487379161 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f02280ca520 0x7f02280c8fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:54.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:54.004+0000 7f022d200640 1 -- 172.21.15.6:0/1487379161 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02281f4a70 con 0x7f022815c1b0 2024-04-24T23:18:54.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:54.004+0000 7f022dc00640 1 --2- 172.21.15.6:0/1487379161 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f02280ca520 0x7f02280c8fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:18:54.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:54.004+0000 7f022d200640 1 --2- 172.21.15.6:0/1487379161 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f022815c1b0 0x7f022806f450 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f021c00ebb0 tx=0x7f021c00c520 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:18:54.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:54.004+0000 7f0225600640 1 -- 172.21.15.6:0/1487379161 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f021c017070 con 0x7f022815c1b0 2024-04-24T23:18:54.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:54.004+0000 7f0225600640 1 -- 172.21.15.6:0/1487379161 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f021c00cc30 con 0x7f022815c1b0 2024-04-24T23:18:54.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:54.004+0000 7f0230600640 1 -- 172.21.15.6:0/1487379161 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0228155d90 con 0x7f022815c1b0 2024-04-24T23:18:54.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:54.004+0000 7f0225600640 1 -- 172.21.15.6:0/1487379161 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f021c013470 con 0x7f022815c1b0 2024-04-24T23:18:54.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:54.005+0000 7f0230600640 1 -- 172.21.15.6:0/1487379161 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02281f4fd0 con 0x7f022815c1b0 2024-04-24T23:18:54.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:54.006+0000 7f021aa00640 1 -- 172.21.15.6:0/1487379161 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f02280767b0 con 0x7f022815c1b0 2024-04-24T23:18:54.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:54.007+0000 7f0225600640 1 -- 172.21.15.6:0/1487379161 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f021c01a070 con 0x7f022815c1b0 2024-04-24T23:18:54.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:54.008+0000 7f0225600640 1 --2- 172.21.15.6:0/1487379161 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f020806be60 0x7f020806e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:18:54.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:54.008+0000 7f0225600640 1 -- 172.21.15.6:0/1487379161 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6676+0+0 (secure 0 0 0) 0x7f021c08b190 con 0x7f022815c1b0 2024-04-24T23:18:54.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:54.008+0000 7f022dc00640 1 --2- 172.21.15.6:0/1487379161 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f020806be60 0x7f020806e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:18:54.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:54.009+0000 7f022dc00640 1 --2- 172.21.15.6:0/1487379161 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f020806be60 0x7f020806e320 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f0214000f60 tx=0x7f0214007c10 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:18:54.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:54.014+0000 7f0225600640 1 -- 172.21.15.6:0/1487379161 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f02281f4a70 con 0x7f022815c1b0 2024-04-24T23:18:54.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:54.512+0000 7f021aa00640 1 -- 172.21.15.6:0/1487379161 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f022815e020 con 0x7f022815c1b0 2024-04-24T23:18:54.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:54.513+0000 7f0225600640 1 -- 172.21.15.6:0/1487379161 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+6904 (secure 0 0 0) 0x7f021c058ff0 con 0x7f022815c1b0 2024-04-24T23:18:54.515 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:18:54.516 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5257,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/3240971655","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":3240971655},{"type":"v1","addr":"172.21.15.6:6837","nonce":3240971655}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":24}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:18:47.823843+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4282,"mds_2":4235,"mds_3":4232,"mds_4":4197},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4197":{"gid":4197,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":53,"addr":"172.21.15.6:6834/105267896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":105267896},{"type":"v1","addr":"172.21.15.6:6834","nonce":105267896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4232":{"gid":4232,"name":"b","rank":3,"incarnation":15,"state":"up:active","state_seq":15,"addr":"172.21.15.190:6838/2032280158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2032280158},{"type":"v1","addr":"172.21.15.190:6838","nonce":2032280158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4235":{"gid":4235,"name":"d","rank":2,"incarnation":13,"state":"up:active","state_seq":14,"addr":"172.21.15.190:6839/1572872451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":1572872451},{"type":"v1","addr":"172.21.15.190:6839","nonce":1572872451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4282":{"gid":4282,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":12,"addr":"172.21.15.190:6837/580315269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":580315269},{"type":"v1","addr":"172.21.15.190:6837","nonce":580315269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/489092393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":489092393},{"type":"v1","addr":"172.21.15.6:6835","nonce":489092393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4306,"qdb_cluster":[4306,4282,4235,4232,4197]},"id":1}]} 2024-04-24T23:18:54.516 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 24 2024-04-24T23:18:54.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:54.516+0000 7f021aa00640 1 -- 172.21.15.6:0/1487379161 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f020806be60 msgr2=0x7f020806e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:18:54.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:54.516+0000 7f021aa00640 1 --2- 172.21.15.6:0/1487379161 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f020806be60 0x7f020806e320 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f0214000f60 tx=0x7f0214007c10 comp rx=0 tx=0).stop 2024-04-24T23:18:54.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:54.516+0000 7f021aa00640 1 -- 172.21.15.6:0/1487379161 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f022815c1b0 msgr2=0x7f022806f450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:18:54.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:54.516+0000 7f021aa00640 1 --2- 172.21.15.6:0/1487379161 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f022815c1b0 0x7f022806f450 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f021c00ebb0 tx=0x7f021c00c520 comp rx=0 tx=0).stop 2024-04-24T23:18:54.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:54.516+0000 7f021aa00640 1 -- 172.21.15.6:0/1487379161 shutdown_connections 2024-04-24T23:18:54.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:54.516+0000 7f021aa00640 1 --2- 172.21.15.6:0/1487379161 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f020806be60 0x7f020806e320 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:54.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:54.516+0000 7f021aa00640 1 --2- 172.21.15.6:0/1487379161 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f022815e2b0 0x7f022806f990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:54.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:54.516+0000 7f021aa00640 1 --2- 172.21.15.6:0/1487379161 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f022815c1b0 0x7f022806f450 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:54.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:54.516+0000 7f021aa00640 1 --2- 172.21.15.6:0/1487379161 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f02280ca520 0x7f02280c8fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:18:54.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:54.517+0000 7f021aa00640 1 -- 172.21.15.6:0/1487379161 >> 172.21.15.6:0/1487379161 conn(0x7f022806c0f0 msgr2=0x7f02280cec00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:18:54.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:54.517+0000 7f021aa00640 1 -- 172.21.15.6:0/1487379161 shutdown_connections 2024-04-24T23:18:54.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:18:54.517+0000 7f021aa00640 1 -- 172.21.15.6:0/1487379161 wait complete. 2024-04-24T23:18:54.542 INFO:tasks.mds_thrash.fs.[cephfs]:mds.e reported in up:standby state 2024-04-24T23:18:54.543 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 67 secs before thrashing 2024-04-24T23:19:03.066 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-24T23:19:03.074 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-24T23:19:33.182 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-24T23:19:33.188 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-24T23:19:52.860 INFO:tasks.workunit.client.0.smithi006.stdout:fs setup took 68 secs 2024-04-24T23:20:01.544 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:20:01.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:01.727+0000 7fbfbbe00640 1 Processor -- start 2024-04-24T23:20:01.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:01.727+0000 7fbfbbe00640 1 -- start start 2024-04-24T23:20:01.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:01.729+0000 7fbfbbe00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbfbc0cc1e0 0x7fbfbc156a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:20:01.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:01.729+0000 7fbfbbe00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbfbc06d370 con 0x7fbfbc0c7e80 2024-04-24T23:20:01.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:01.729+0000 7fbfbbe00640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbfbc06d650 con 0x7fbfbc06ce30 2024-04-24T23:20:01.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:01.729+0000 7fbfbbe00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fbfbc06def0 con 0x7fbfbc0cc1e0 2024-04-24T23:20:01.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:01.729+0000 7fbfbaa00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbfbc0cc1e0 0x7fbfbc156a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:20:01.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:01.729+0000 7fbfbb400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fbfbc0c7e80 0x7fbfbc1506b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:33176/0 (socket says 172.21.15.6:33176) 2024-04-24T23:20:01.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:01.729+0000 7fbfbb400640 1 -- 172.21.15.6:0/2079654931 learned_addr learned my addr 172.21.15.6:0/2079654931 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:20:01.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:01.730+0000 7fbfb9600640 1 -- 172.21.15.6:0/2079654931 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4263611255 0 0) 0x7fbfbc06d370 con 0x7fbfbc0c7e80 2024-04-24T23:20:01.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:01.730+0000 7fbfb9600640 1 -- 172.21.15.6:0/2079654931 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbfa0003520 con 0x7fbfbc0c7e80 2024-04-24T23:20:01.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:01.730+0000 7fbfb9600640 1 -- 172.21.15.6:0/2079654931 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2772310529 0 0) 0x7fbfbc06d650 con 0x7fbfbc06ce30 2024-04-24T23:20:01.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:01.730+0000 7fbfb9600640 1 -- 172.21.15.6:0/2079654931 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbfbc06d370 con 0x7fbfbc06ce30 2024-04-24T23:20:01.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:01.730+0000 7fbfb9600640 1 -- 172.21.15.6:0/2079654931 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3162131891 0 0) 0x7fbfbc06d370 con 0x7fbfbc06ce30 2024-04-24T23:20:01.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:01.730+0000 7fbfb9600640 1 -- 172.21.15.6:0/2079654931 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbfbc0cc1e0 msgr2=0x7fbfbc156a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:20:01.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:01.730+0000 7fbfb9600640 1 --2- 172.21.15.6:0/2079654931 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbfbc0cc1e0 0x7fbfbc156a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:01.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:01.730+0000 7fbfb9600640 1 -- 172.21.15.6:0/2079654931 >> v1:172.21.15.6:6789/0 conn(0x7fbfbc0c7e80 legacy=0x7fbfbc1506b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:20:01.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:01.730+0000 7fbfb9600640 1 -- 172.21.15.6:0/2079654931 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfbc06d930 con 0x7fbfbc06ce30 2024-04-24T23:20:01.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:01.730+0000 7fbfb9600640 1 -- 172.21.15.6:0/2079654931 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3463268701 0 0) 0x7fbfac001670 con 0x7fbfbc06ce30 2024-04-24T23:20:01.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:01.730+0000 7fbfb9600640 1 -- 172.21.15.6:0/2079654931 >> v1:172.21.15.190:6789/0 conn(0x7fbfbc06ce30 legacy=0x7fbfbc0c7770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:20:01.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:01.730+0000 7fbfb9600640 1 --2- 172.21.15.6:0/2079654931 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbfa0004ba0 0x7fbfa0004fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:20:01.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:01.730+0000 7fbfb9600640 1 --2- 172.21.15.6:0/2079654931 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbfa00037a0 0x7fbfa0003c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:20:01.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:01.730+0000 7fbfb9600640 1 --2- 172.21.15.6:0/2079654931 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbfa0004160 0x7fbfa000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:20:01.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:01.730+0000 7fbfb9600640 1 -- 172.21.15.6:0/2079654931 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fbfbc06def0 con 0x7fbfa0004ba0 2024-04-24T23:20:01.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:01.730+0000 7fbfb9600640 1 -- 172.21.15.6:0/2079654931 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fbfa000bfe0 con 0x7fbfa0004160 2024-04-24T23:20:01.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:01.730+0000 7fbfb9600640 1 -- 172.21.15.6:0/2079654931 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fbfa000c1e0 con 0x7fbfa00037a0 2024-04-24T23:20:01.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:01.731+0000 7fbfbb400640 1 --2- 172.21.15.6:0/2079654931 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbfa0004160 0x7fbfa000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:20:01.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:01.731+0000 7fbfba000640 1 --2- 172.21.15.6:0/2079654931 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbfa00037a0 0x7fbfa0003c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:20:01.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:01.731+0000 7fbfbaa00640 1 --2- 172.21.15.6:0/2079654931 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbfa0004ba0 0x7fbfa0004fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:20:01.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:01.732+0000 7fbfba000640 1 -- 172.21.15.6:0/2079654931 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbfa0004160 msgr2=0x7fbfa000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:20:01.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:01.732+0000 7fbfba000640 1 --2- 172.21.15.6:0/2079654931 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbfa0004160 0x7fbfa000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:01.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:01.732+0000 7fbfba000640 1 -- 172.21.15.6:0/2079654931 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbfa0004ba0 msgr2=0x7fbfa0004fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:20:01.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:01.732+0000 7fbfba000640 1 --2- 172.21.15.6:0/2079654931 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbfa0004ba0 0x7fbfa0004fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:01.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:01.732+0000 7fbfba000640 1 -- 172.21.15.6:0/2079654931 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfbc06d370 con 0x7fbfa00037a0 2024-04-24T23:20:01.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:01.732+0000 7fbfbaa00640 1 --2- 172.21.15.6:0/2079654931 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbfa0004ba0 0x7fbfa0004fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:20:01.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:01.732+0000 7fbfbb400640 1 --2- 172.21.15.6:0/2079654931 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbfa0004160 0x7fbfa000b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:20:01.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:01.735+0000 7fbfba000640 1 --2- 172.21.15.6:0/2079654931 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbfa00037a0 0x7fbfa0003c20 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fbfac007eb0 tx=0x7fbfa0005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=64ad75585d31a2f3 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:20:01.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:01.735+0000 7fbfb9600640 1 -- 172.21.15.6:0/2079654931 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfac036070 con 0x7fbfa00037a0 2024-04-24T23:20:01.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:01.735+0000 7fbfb9600640 1 -- 172.21.15.6:0/2079654931 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbfac031400 con 0x7fbfa00037a0 2024-04-24T23:20:01.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:01.735+0000 7fbfb9600640 1 -- 172.21.15.6:0/2079654931 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfac007ad0 con 0x7fbfa00037a0 2024-04-24T23:20:01.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:01.736+0000 7fbfbbe00640 1 -- 172.21.15.6:0/2079654931 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbfa00037a0 msgr2=0x7fbfa0003c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:20:01.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:01.736+0000 7fbfbbe00640 1 --2- 172.21.15.6:0/2079654931 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbfa00037a0 0x7fbfa0003c20 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fbfac007eb0 tx=0x7fbfa0005830 comp rx=0 tx=0).stop 2024-04-24T23:20:01.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:01.736+0000 7fbfbb400640 1 -- 172.21.15.6:0/2079654931 reap_dead start 2024-04-24T23:20:01.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:01.736+0000 7fbfbbe00640 1 -- 172.21.15.6:0/2079654931 shutdown_connections 2024-04-24T23:20:01.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:01.736+0000 7fbfbbe00640 1 -- 172.21.15.6:0/2079654931 >> 172.21.15.6:0/2079654931 conn(0x7fbfbc06ed80 msgr2=0x7fbfbc06f190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:20:01.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:01.737+0000 7fbfbbe00640 1 -- 172.21.15.6:0/2079654931 shutdown_connections 2024-04-24T23:20:01.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:01.737+0000 7fbfbbe00640 1 -- 172.21.15.6:0/2079654931 wait complete. 2024-04-24T23:20:01.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:01.742+0000 7fbfbbe00640 1 Processor -- start 2024-04-24T23:20:01.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:01.742+0000 7fbfbbe00640 1 -- start start 2024-04-24T23:20:01.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:01.744+0000 7fbfbbe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbfbc148c30 0x7fbfbc1f0180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:20:01.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:01.744+0000 7fbfbbe00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbfbc149010 0x7fbfbc1ecc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:20:01.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:01.744+0000 7fbfbbe00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbfbc0c5390 0x7fbfbc06b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:20:01.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:01.744+0000 7fbfbbe00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fbfbc06e0b0 con 0x7fbfbc148c30 2024-04-24T23:20:01.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:01.744+0000 7fbfbbe00640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fbfbc071020 con 0x7fbfbc149010 2024-04-24T23:20:01.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:01.744+0000 7fbfbbe00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fbfbc06e230 con 0x7fbfbc0c5390 2024-04-24T23:20:01.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:01.744+0000 7fbfbaa00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbfbc148c30 0x7fbfbc1f0180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:20:01.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:01.744+0000 7fbfbb400640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbfbc0c5390 0x7fbfbc06b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:20:01.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:01.744+0000 7fbfbb400640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbfbc0c5390 0x7fbfbc06b0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.6:42920/0 (socket says 172.21.15.6:42920) 2024-04-24T23:20:01.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:01.744+0000 7fbfbb400640 1 -- 172.21.15.6:0/3466618621 learned_addr learned my addr 172.21.15.6:0/3466618621 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:20:01.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:01.745+0000 7fbfba000640 1 --2- 172.21.15.6:0/3466618621 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbfbc149010 0x7fbfbc1ecc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:20:01.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:01.745+0000 7fbfbaa00640 1 -- 172.21.15.6:0/3466618621 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbfbc0c5390 msgr2=0x7fbfbc06b0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:20:01.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:01.745+0000 7fbfbaa00640 1 --2- 172.21.15.6:0/3466618621 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbfbc0c5390 0x7fbfbc06b0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:01.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:01.745+0000 7fbfbaa00640 1 -- 172.21.15.6:0/3466618621 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbfbc149010 msgr2=0x7fbfbc1ecc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:20:01.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:01.745+0000 7fbfbaa00640 1 --2- 172.21.15.6:0/3466618621 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbfbc149010 0x7fbfbc1ecc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:01.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:01.745+0000 7fbfbaa00640 1 -- 172.21.15.6:0/3466618621 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfbc1ec200 con 0x7fbfbc148c30 2024-04-24T23:20:01.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:01.746+0000 7fbfbb400640 1 --2- 172.21.15.6:0/3466618621 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbfbc0c5390 0x7fbfbc06b0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:20:01.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:01.746+0000 7fbfba000640 1 --2- 172.21.15.6:0/3466618621 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbfbc149010 0x7fbfbc1ecc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:20:01.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:01.746+0000 7fbfbaa00640 1 --2- 172.21.15.6:0/3466618621 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbfbc148c30 0x7fbfbc1f0180 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fbfbc0ccbb0 tx=0x7fbfa8015660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:20:01.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:01.747+0000 7fbfb2a00640 1 -- 172.21.15.6:0/3466618621 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfa801e020 con 0x7fbfbc148c30 2024-04-24T23:20:01.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:01.748+0000 7fbfbbe00640 1 -- 172.21.15.6:0/3466618621 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbfbc158440 con 0x7fbfbc148c30 2024-04-24T23:20:01.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:01.748+0000 7fbfb2a00640 1 -- 172.21.15.6:0/3466618621 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbfa801a680 con 0x7fbfbc148c30 2024-04-24T23:20:01.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:01.748+0000 7fbfb2a00640 1 -- 172.21.15.6:0/3466618621 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfa801e020 con 0x7fbfbc148c30 2024-04-24T23:20:01.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:01.749+0000 7fbfbbe00640 1 -- 172.21.15.6:0/3466618621 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbfbc158a00 con 0x7fbfbc148c30 2024-04-24T23:20:01.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:01.753+0000 7fbfbbe00640 1 -- 172.21.15.6:0/3466618621 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbfbc075b30 con 0x7fbfbc148c30 2024-04-24T23:20:01.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:01.754+0000 7fbfb2a00640 1 -- 172.21.15.6:0/3466618621 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fbfa8021020 con 0x7fbfbc148c30 2024-04-24T23:20:01.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:01.756+0000 7fbfb2a00640 1 --2- 172.21.15.6:0/3466618621 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fbf9c06c030 0x7fbf9c06e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:20:01.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:01.757+0000 7fbfba000640 1 --2- 172.21.15.6:0/3466618621 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fbf9c06c030 0x7fbf9c06e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:20:01.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:01.757+0000 7fbfb2a00640 1 -- 172.21.15.6:0/3466618621 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6676+0+0 (secure 0 0 0) 0x7fbfa808fc70 con 0x7fbfbc148c30 2024-04-24T23:20:01.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:01.766+0000 7fbfba000640 1 --2- 172.21.15.6:0/3466618621 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fbf9c06c030 0x7fbf9c06e4f0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fbfac033000 tx=0x7fbfac009030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:20:01.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:01.772+0000 7fbfb2a00640 1 -- 172.21.15.6:0/3466618621 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fbfa8028020 con 0x7fbfbc148c30 2024-04-24T23:20:02.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:02.191+0000 7fbfbbe00640 1 -- 172.21.15.6:0/3466618621 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbfbc154dc0 con 0x7fbfbc148c30 2024-04-24T23:20:02.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:02.195+0000 7fbfb2a00640 1 -- 172.21.15.6:0/3466618621 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+6904 (secure 0 0 0) 0x7fbfbc1ec200 con 0x7fbfbc148c30 2024-04-24T23:20:02.198 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:20:02.198 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5257,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/3240971655","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":3240971655},{"type":"v1","addr":"172.21.15.6:6837","nonce":3240971655}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":24}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:18:47.823843+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4282,"mds_2":4235,"mds_3":4232,"mds_4":4197},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4197":{"gid":4197,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":53,"addr":"172.21.15.6:6834/105267896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":105267896},{"type":"v1","addr":"172.21.15.6:6834","nonce":105267896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4232":{"gid":4232,"name":"b","rank":3,"incarnation":15,"state":"up:active","state_seq":15,"addr":"172.21.15.190:6838/2032280158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2032280158},{"type":"v1","addr":"172.21.15.190:6838","nonce":2032280158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4235":{"gid":4235,"name":"d","rank":2,"incarnation":13,"state":"up:active","state_seq":14,"addr":"172.21.15.190:6839/1572872451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":1572872451},{"type":"v1","addr":"172.21.15.190:6839","nonce":1572872451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4282":{"gid":4282,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":12,"addr":"172.21.15.190:6837/580315269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":580315269},{"type":"v1","addr":"172.21.15.190:6837","nonce":580315269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/489092393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":489092393},{"type":"v1","addr":"172.21.15.6:6835","nonce":489092393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4306,"qdb_cluster":[4306,4282,4235,4232,4197]},"id":1}]} 2024-04-24T23:20:02.198 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 24 2024-04-24T23:20:02.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:02.198+0000 7fbfbbe00640 1 -- 172.21.15.6:0/3466618621 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fbf9c06c030 msgr2=0x7fbf9c06e4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:20:02.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:02.198+0000 7fbfbbe00640 1 --2- 172.21.15.6:0/3466618621 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fbf9c06c030 0x7fbf9c06e4f0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fbfac033000 tx=0x7fbfac009030 comp rx=0 tx=0).stop 2024-04-24T23:20:02.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:02.198+0000 7fbfbbe00640 1 -- 172.21.15.6:0/3466618621 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbfbc148c30 msgr2=0x7fbfbc1f0180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:20:02.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:02.198+0000 7fbfbbe00640 1 --2- 172.21.15.6:0/3466618621 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbfbc148c30 0x7fbfbc1f0180 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fbfbc0ccbb0 tx=0x7fbfa8015660 comp rx=0 tx=0).stop 2024-04-24T23:20:02.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:02.199+0000 7fbfbbe00640 1 -- 172.21.15.6:0/3466618621 shutdown_connections 2024-04-24T23:20:02.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:02.199+0000 7fbfbbe00640 1 --2- 172.21.15.6:0/3466618621 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fbf9c06c030 0x7fbf9c06e4f0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:02.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:02.199+0000 7fbfbbe00640 1 --2- 172.21.15.6:0/3466618621 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbfbc0c5390 0x7fbfbc06b0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:02.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:02.199+0000 7fbfbbe00640 1 --2- 172.21.15.6:0/3466618621 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbfbc149010 0x7fbfbc1ecc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:02.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:02.199+0000 7fbfbbe00640 1 --2- 172.21.15.6:0/3466618621 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbfbc148c30 0x7fbfbc1f0180 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:02.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:02.199+0000 7fbfbbe00640 1 -- 172.21.15.6:0/3466618621 >> 172.21.15.6:0/3466618621 conn(0x7fbfbc06ed80 msgr2=0x7fbfbc071ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:20:02.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:02.199+0000 7fbfbbe00640 1 -- 172.21.15.6:0/3466618621 shutdown_connections 2024-04-24T23:20:02.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:02.200+0000 7fbfbbe00640 1 -- 172.21.15.6:0/3466618621 wait complete. 2024-04-24T23:20:02.226 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.a (rank=4) 2024-04-24T23:20:02.227 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-24T23:20:02.227 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T23:20:02.253 INFO:tasks.ceph.mds.a:Stopped 2024-04-24T23:20:02.253 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:20:02.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:02.522+0000 7f8b46400640 1 Processor -- start 2024-04-24T23:20:02.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:02.522+0000 7f8b46400640 1 -- start start 2024-04-24T23:20:02.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:02.523+0000 7f8b46400640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f8b400c8f50 0x7f8b4006e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:20:02.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:02.523+0000 7f8b46400640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8b400c9330 0x7f8b4006e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:20:02.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:02.523+0000 7f8b46400640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8b400c6e70 con 0x7f8b40152bc0 2024-04-24T23:20:02.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:02.523+0000 7f8b46400640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f8b400c73d0 con 0x7f8b400c8f50 2024-04-24T23:20:02.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:02.523+0000 7f8b46400640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f8b400c8ac0 con 0x7f8b400c9330 2024-04-24T23:20:02.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:02.523+0000 7f8b3ea00640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f8b400c8f50 0x7f8b4006e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:20:02.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:02.523+0000 7f8b3f400640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f8b40152bc0 0x7f8b40159390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.6:41002/0 (socket says 172.21.15.6:41002) 2024-04-24T23:20:02.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:02.523+0000 7f8b3fe00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8b400c9330 0x7f8b4006e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:20:02.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:02.523+0000 7f8b3f400640 1 -- 172.21.15.6:0/455828757 learned_addr learned my addr 172.21.15.6:0/455828757 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:20:02.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:02.524+0000 7f8b3ea00640 1 -- 172.21.15.6:0/455828757 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8b400c9330 msgr2=0x7f8b4006e8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:20:02.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:02.524+0000 7f8b3ea00640 1 --2- 172.21.15.6:0/455828757 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8b400c9330 0x7f8b4006e8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:02.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:02.524+0000 7f8b3ea00640 1 -- 172.21.15.6:0/455828757 >> v1:172.21.15.190:6789/0 conn(0x7f8b40152bc0 legacy=0x7f8b40159390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:20:02.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:02.524+0000 7f8b3ea00640 1 -- 172.21.15.6:0/455828757 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b400c7110 con 0x7f8b400c8f50 2024-04-24T23:20:02.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:02.524+0000 7f8b3e000640 1 -- 172.21.15.6:0/455828757 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1747340283 0 0) 0x7f8b400c6e70 con 0x7f8b40152bc0 2024-04-24T23:20:02.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:02.525+0000 7f8b3ea00640 1 --2- 172.21.15.6:0/455828757 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8b400c8f50 0x7f8b4006e3a0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f8b300097a0 tx=0x7f8b3002a100 comp rx=0 tx=0).ready entity=mon.1 client_cookie=532188733085a61c server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:20:02.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:02.525+0000 7f8b3e000640 1 -- 172.21.15.6:0/455828757 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b30037070 con 0x7f8b400c8f50 2024-04-24T23:20:02.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:02.525+0000 7f8b3e000640 1 -- 172.21.15.6:0/455828757 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8b3002ae50 con 0x7f8b400c8f50 2024-04-24T23:20:02.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:02.525+0000 7f8b3e000640 1 -- 172.21.15.6:0/455828757 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b30032660 con 0x7f8b400c8f50 2024-04-24T23:20:02.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:02.526+0000 7f8b46400640 1 -- 172.21.15.6:0/455828757 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8b400c8f50 msgr2=0x7f8b4006e3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:20:02.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:02.526+0000 7f8b46400640 1 --2- 172.21.15.6:0/455828757 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8b400c8f50 0x7f8b4006e3a0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f8b300097a0 tx=0x7f8b3002a100 comp rx=0 tx=0).stop 2024-04-24T23:20:02.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:02.526+0000 7f8b46400640 1 -- 172.21.15.6:0/455828757 shutdown_connections 2024-04-24T23:20:02.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:02.526+0000 7f8b46400640 1 --2- 172.21.15.6:0/455828757 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8b400c9330 0x7f8b4006e8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:02.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:02.526+0000 7f8b46400640 1 --2- 172.21.15.6:0/455828757 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8b400c8f50 0x7f8b4006e3a0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:02.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:02.526+0000 7f8b46400640 1 -- 172.21.15.6:0/455828757 >> 172.21.15.6:0/455828757 conn(0x7f8b4006b0e0 msgr2=0x7f8b4006b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:20:02.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:02.526+0000 7f8b46400640 1 -- 172.21.15.6:0/455828757 shutdown_connections 2024-04-24T23:20:02.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:02.526+0000 7f8b46400640 1 -- 172.21.15.6:0/455828757 wait complete. 2024-04-24T23:20:02.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:02.528+0000 7f8b46400640 1 Processor -- start 2024-04-24T23:20:02.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:02.528+0000 7f8b46400640 1 -- start start 2024-04-24T23:20:02.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:02.529+0000 7f8b46400640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8b400c8f50 0x7f8b40072050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:20:02.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:02.529+0000 7f8b46400640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8b400c9330 0x7f8b401e3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:20:02.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:02.529+0000 7f8b46400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8b40152bc0 0x7f8b401f0720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:20:02.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:02.529+0000 7f8b46400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f8b400c8750 con 0x7f8b40152bc0 2024-04-24T23:20:02.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:02.529+0000 7f8b46400640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f8b400c8470 con 0x7f8b400c9330 2024-04-24T23:20:02.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:02.529+0000 7f8b46400640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f8b40152fa0 con 0x7f8b400c8f50 2024-04-24T23:20:02.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:02.530+0000 7f8b3f400640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8b400c8f50 0x7f8b40072050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:20:02.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:02.531+0000 7f8b3f400640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8b400c8f50 0x7f8b40072050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.6:42936/0 (socket says 172.21.15.6:42936) 2024-04-24T23:20:02.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:02.531+0000 7f8b3f400640 1 -- 172.21.15.6:0/2668663524 learned_addr learned my addr 172.21.15.6:0/2668663524 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:20:02.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:02.531+0000 7f8b3fe00640 1 --2- 172.21.15.6:0/2668663524 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8b40152bc0 0x7f8b401f0720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:20:02.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:02.532+0000 7f8b3ea00640 1 --2- 172.21.15.6:0/2668663524 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8b400c9330 0x7f8b401e3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:20:02.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:02.532+0000 7f8b3fe00640 1 -- 172.21.15.6:0/2668663524 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8b400c8f50 msgr2=0x7f8b40072050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:20:02.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:02.532+0000 7f8b3fe00640 1 --2- 172.21.15.6:0/2668663524 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8b400c8f50 0x7f8b40072050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:02.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:02.532+0000 7f8b3fe00640 1 -- 172.21.15.6:0/2668663524 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8b400c9330 msgr2=0x7f8b401e3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:20:02.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:02.532+0000 7f8b3fe00640 1 --2- 172.21.15.6:0/2668663524 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8b400c9330 0x7f8b401e3bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:02.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:02.532+0000 7f8b3fe00640 1 -- 172.21.15.6:0/2668663524 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b40072900 con 0x7f8b40152bc0 2024-04-24T23:20:02.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:02.532+0000 7f8b3ea00640 1 --2- 172.21.15.6:0/2668663524 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8b400c9330 0x7f8b401e3bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:20:02.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:02.533+0000 7f8b3fe00640 1 --2- 172.21.15.6:0/2668663524 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8b40152bc0 0x7f8b401f0720 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f8b2c00e820 tx=0x7f8b2c00ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:20:02.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:02.534+0000 7f8b37400640 1 -- 172.21.15.6:0/2668663524 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b2c017070 con 0x7f8b40152bc0 2024-04-24T23:20:02.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:02.534+0000 7f8b46400640 1 -- 172.21.15.6:0/2668663524 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b401f3c30 con 0x7f8b40152bc0 2024-04-24T23:20:02.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:02.534+0000 7f8b37400640 1 -- 172.21.15.6:0/2668663524 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8b2c00c940 con 0x7f8b40152bc0 2024-04-24T23:20:02.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:02.534+0000 7f8b37400640 1 -- 172.21.15.6:0/2668663524 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b2c00cd10 con 0x7f8b40152bc0 2024-04-24T23:20:02.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:02.534+0000 7f8b46400640 1 -- 172.21.15.6:0/2668663524 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b401f41f0 con 0x7f8b40152bc0 2024-04-24T23:20:02.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:02.536+0000 7f8b46400640 1 -- 172.21.15.6:0/2668663524 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b18005290 con 0x7f8b40152bc0 2024-04-24T23:20:02.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:02.540+0000 7f8b37400640 1 -- 172.21.15.6:0/2668663524 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f8b2c01a070 con 0x7f8b40152bc0 2024-04-24T23:20:02.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:02.544+0000 7f8b37400640 1 --2- 172.21.15.6:0/2668663524 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f8b2006bf00 0x7f8b2006e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:20:02.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:02.544+0000 7f8b3f400640 1 --2- 172.21.15.6:0/2668663524 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f8b2006bf00 0x7f8b2006e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:20:02.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:02.545+0000 7f8b37400640 1 -- 172.21.15.6:0/2668663524 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6676+0+0 (secure 0 0 0) 0x7f8b2c08b140 con 0x7f8b40152bc0 2024-04-24T23:20:02.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:02.545+0000 7f8b3f400640 1 --2- 172.21.15.6:0/2668663524 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f8b2006bf00 0x7f8b2006e3c0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f8b380012e0 tx=0x7f8b38009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:20:02.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:02.547+0000 7f8b37400640 1 -- 172.21.15.6:0/2668663524 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f8b2c059020 con 0x7f8b40152bc0 2024-04-24T23:20:02.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:02.927+0000 7f8b46400640 1 -- 172.21.15.6:0/2668663524 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8b18005580 con 0x7f8b40152bc0 2024-04-24T23:20:02.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:02.928+0000 7f8b37400640 1 -- 172.21.15.6:0/2668663524 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+6904 (secure 0 0 0) 0x7f8b2c05d320 con 0x7f8b40152bc0 2024-04-24T23:20:02.930 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:20:02.930 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5257,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/3240971655","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":3240971655},{"type":"v1","addr":"172.21.15.6:6837","nonce":3240971655}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":24}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:18:47.823843+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4282,"mds_2":4235,"mds_3":4232,"mds_4":4197},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4197":{"gid":4197,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":53,"addr":"172.21.15.6:6834/105267896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":105267896},{"type":"v1","addr":"172.21.15.6:6834","nonce":105267896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4232":{"gid":4232,"name":"b","rank":3,"incarnation":15,"state":"up:active","state_seq":15,"addr":"172.21.15.190:6838/2032280158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2032280158},{"type":"v1","addr":"172.21.15.190:6838","nonce":2032280158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4235":{"gid":4235,"name":"d","rank":2,"incarnation":13,"state":"up:active","state_seq":14,"addr":"172.21.15.190:6839/1572872451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":1572872451},{"type":"v1","addr":"172.21.15.190:6839","nonce":1572872451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4282":{"gid":4282,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":12,"addr":"172.21.15.190:6837/580315269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":580315269},{"type":"v1","addr":"172.21.15.190:6837","nonce":580315269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/489092393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":489092393},{"type":"v1","addr":"172.21.15.6:6835","nonce":489092393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4306,"qdb_cluster":[4306,4282,4235,4232,4197]},"id":1}]} 2024-04-24T23:20:02.931 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 24 2024-04-24T23:20:02.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:02.930+0000 7f8b46400640 1 -- 172.21.15.6:0/2668663524 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f8b2006bf00 msgr2=0x7f8b2006e3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:20:02.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:02.930+0000 7f8b46400640 1 --2- 172.21.15.6:0/2668663524 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f8b2006bf00 0x7f8b2006e3c0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f8b380012e0 tx=0x7f8b38009040 comp rx=0 tx=0).stop 2024-04-24T23:20:02.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:02.930+0000 7f8b46400640 1 -- 172.21.15.6:0/2668663524 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8b40152bc0 msgr2=0x7f8b401f0720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:20:02.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:02.930+0000 7f8b46400640 1 --2- 172.21.15.6:0/2668663524 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8b40152bc0 0x7f8b401f0720 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f8b2c00e820 tx=0x7f8b2c00ece0 comp rx=0 tx=0).stop 2024-04-24T23:20:02.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:02.930+0000 7f8b46400640 1 -- 172.21.15.6:0/2668663524 shutdown_connections 2024-04-24T23:20:02.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:02.930+0000 7f8b46400640 1 --2- 172.21.15.6:0/2668663524 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f8b2006bf00 0x7f8b2006e3c0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:02.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:02.930+0000 7f8b46400640 1 --2- 172.21.15.6:0/2668663524 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8b40152bc0 0x7f8b401f0720 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:02.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:02.930+0000 7f8b46400640 1 --2- 172.21.15.6:0/2668663524 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8b400c9330 0x7f8b401e3bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:02.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:02.930+0000 7f8b46400640 1 --2- 172.21.15.6:0/2668663524 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8b400c8f50 0x7f8b40072050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:02.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:02.930+0000 7f8b46400640 1 -- 172.21.15.6:0/2668663524 >> 172.21.15.6:0/2668663524 conn(0x7f8b4006b0e0 msgr2=0x7f8b40150b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:20:02.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:02.931+0000 7f8b46400640 1 -- 172.21.15.6:0/2668663524 shutdown_connections 2024-04-24T23:20:02.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:02.931+0000 7f8b46400640 1 -- 172.21.15.6:0/2668663524 wait complete. 2024-04-24T23:20:02.949 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-04-24T23:20:03.531 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-24T23:20:03.537 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-24T23:20:04.950 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:20:05.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:05.185+0000 7feb24a00640 1 Processor -- start 2024-04-24T23:20:05.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:05.185+0000 7feb24a00640 1 -- start start 2024-04-24T23:20:05.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:05.187+0000 7feb24a00640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7feb20159110 0x7feb20072420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:20:05.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:05.187+0000 7feb24a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7feb200c7dc0 0x7feb201623a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:20:05.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:05.188+0000 7feb24a00640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feb200c7230 con 0x7feb200c81c0 2024-04-24T23:20:05.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:05.188+0000 7feb24a00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7feb201594f0 con 0x7feb200c7dc0 2024-04-24T23:20:05.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:05.188+0000 7feb24a00640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7feb200c74d0 con 0x7feb20159110 2024-04-24T23:20:05.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:05.188+0000 7feb1de00640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7feb20159110 0x7feb20072420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:20:05.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:05.188+0000 7feb1de00640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7feb20159110 0x7feb20072420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:51284/0 (socket says 172.21.15.6:51284) 2024-04-24T23:20:05.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:05.188+0000 7feb1de00640 1 -- 172.21.15.6:0/1845415199 learned_addr learned my addr 172.21.15.6:0/1845415199 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:20:05.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:05.189+0000 7feb1ca00640 1 -- 172.21.15.6:0/1845415199 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4110086375 0 0) 0x7feb200c7230 con 0x7feb200c81c0 2024-04-24T23:20:05.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:05.189+0000 7feb1ca00640 1 -- 172.21.15.6:0/1845415199 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feb04003520 con 0x7feb200c81c0 2024-04-24T23:20:05.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:05.189+0000 7feb1de00640 1 -- 172.21.15.6:0/1845415199 >> v2:172.21.15.6:3300/0 conn(0x7feb200c7dc0 msgr2=0x7feb201623a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:20:05.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:05.189+0000 7feb1de00640 1 --2- 172.21.15.6:0/1845415199 >> v2:172.21.15.6:3300/0 conn(0x7feb200c7dc0 0x7feb201623a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:05.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:05.189+0000 7feb1de00640 1 -- 172.21.15.6:0/1845415199 >> v1:172.21.15.190:6789/0 conn(0x7feb200c81c0 legacy=0x7feb201628e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:20:05.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:05.189+0000 7feb1de00640 1 -- 172.21.15.6:0/1845415199 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb2006e440 con 0x7feb20159110 2024-04-24T23:20:05.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:05.191+0000 7feb1de00640 1 --2- 172.21.15.6:0/1845415199 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feb20159110 0x7feb20072420 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7feb0c009870 tx=0x7feb0c02a320 comp rx=0 tx=0).ready entity=mon.1 client_cookie=242541eb838138f3 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:20:05.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:05.191+0000 7feb1ca00640 1 -- 172.21.15.6:0/1845415199 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb0c037070 con 0x7feb20159110 2024-04-24T23:20:05.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:05.191+0000 7feb1ca00640 1 -- 172.21.15.6:0/1845415199 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feb0c032640 con 0x7feb20159110 2024-04-24T23:20:05.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:05.191+0000 7feb1ca00640 1 -- 172.21.15.6:0/1845415199 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb0c0329f0 con 0x7feb20159110 2024-04-24T23:20:05.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:05.193+0000 7feb24a00640 1 -- 172.21.15.6:0/1845415199 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feb20159110 msgr2=0x7feb20072420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:20:05.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:05.193+0000 7feb24a00640 1 --2- 172.21.15.6:0/1845415199 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feb20159110 0x7feb20072420 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7feb0c009870 tx=0x7feb0c02a320 comp rx=0 tx=0).stop 2024-04-24T23:20:05.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:05.195+0000 7feb24a00640 1 -- 172.21.15.6:0/1845415199 shutdown_connections 2024-04-24T23:20:05.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:05.195+0000 7feb24a00640 1 --2- 172.21.15.6:0/1845415199 >> v2:172.21.15.6:3300/0 conn(0x7feb200c7dc0 0x7feb201623a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:05.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:05.195+0000 7feb24a00640 1 --2- 172.21.15.6:0/1845415199 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feb20159110 0x7feb20072420 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:05.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:05.195+0000 7feb24a00640 1 -- 172.21.15.6:0/1845415199 >> 172.21.15.6:0/1845415199 conn(0x7feb2006b0e0 msgr2=0x7feb2006b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:20:05.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:05.196+0000 7feb24a00640 1 -- 172.21.15.6:0/1845415199 shutdown_connections 2024-04-24T23:20:05.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:05.196+0000 7feb24a00640 1 -- 172.21.15.6:0/1845415199 wait complete. 2024-04-24T23:20:05.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:05.197+0000 7feb24a00640 1 Processor -- start 2024-04-24T23:20:05.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:05.197+0000 7feb24a00640 1 -- start start 2024-04-24T23:20:05.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:05.202+0000 7feb24a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feb200c7dc0 0x7feb201601e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:20:05.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:05.202+0000 7feb24a00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feb200c81c0 0x7feb20160720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:20:05.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:05.202+0000 7feb24a00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feb20159110 0x7feb2015ba90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:20:05.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:05.202+0000 7feb24a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7feb2006ece0 con 0x7feb200c7dc0 2024-04-24T23:20:05.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:05.202+0000 7feb24a00640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7feb2006ea00 con 0x7feb20159110 2024-04-24T23:20:05.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:05.202+0000 7feb24a00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7feb20072960 con 0x7feb200c81c0 2024-04-24T23:20:05.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:05.203+0000 7feb1de00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feb200c7dc0 0x7feb201601e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:20:05.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:05.204+0000 7feb1de00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feb200c7dc0 0x7feb201601e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:48084/0 (socket says 172.21.15.6:48084) 2024-04-24T23:20:05.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:05.204+0000 7feb1de00640 1 -- 172.21.15.6:0/1053071065 learned_addr learned my addr 172.21.15.6:0/1053071065 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:20:05.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:05.204+0000 7feb1e800640 1 --2- 172.21.15.6:0/1053071065 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feb20159110 0x7feb2015ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:20:05.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:05.205+0000 7feb1de00640 1 -- 172.21.15.6:0/1053071065 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feb200c81c0 msgr2=0x7feb20160720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-24T23:20:05.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:05.205+0000 7feb1de00640 1 --2- 172.21.15.6:0/1053071065 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feb200c81c0 0x7feb20160720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:05.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:05.205+0000 7feb1de00640 1 -- 172.21.15.6:0/1053071065 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feb20159110 msgr2=0x7feb2015ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:20:05.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:05.205+0000 7feb1de00640 1 --2- 172.21.15.6:0/1053071065 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feb20159110 0x7feb2015ba90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:05.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:05.205+0000 7feb1de00640 1 -- 172.21.15.6:0/1053071065 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb20159ac0 con 0x7feb200c7dc0 2024-04-24T23:20:05.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:05.209+0000 7feb1de00640 1 --2- 172.21.15.6:0/1053071065 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feb200c7dc0 0x7feb201601e0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7feb0c0090b0 tx=0x7feb0c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:20:05.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:05.210+0000 7feb16000640 1 -- 172.21.15.6:0/1053071065 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb0c037040 con 0x7feb200c7dc0 2024-04-24T23:20:05.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:05.210+0000 7feb16000640 1 -- 172.21.15.6:0/1053071065 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feb0c004220 con 0x7feb200c7dc0 2024-04-24T23:20:05.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:05.210+0000 7feb16000640 1 -- 172.21.15.6:0/1053071065 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb0c03c670 con 0x7feb200c7dc0 2024-04-24T23:20:05.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:05.210+0000 7feb24a00640 1 -- 172.21.15.6:0/1053071065 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb20158260 con 0x7feb200c7dc0 2024-04-24T23:20:05.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:05.210+0000 7feb24a00640 1 -- 172.21.15.6:0/1053071065 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb20161b10 con 0x7feb200c7dc0 2024-04-24T23:20:05.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:05.216+0000 7feb24a00640 1 -- 172.21.15.6:0/1053071065 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feb200c6fb0 con 0x7feb200c7dc0 2024-04-24T23:20:05.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:05.219+0000 7feb16000640 1 -- 172.21.15.6:0/1053071065 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7feb0c03a070 con 0x7feb200c7dc0 2024-04-24T23:20:05.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:05.220+0000 7feb16000640 1 --2- 172.21.15.6:0/1053071065 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7feb0006be30 0x7feb0006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:20:05.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:05.220+0000 7feb16000640 1 -- 172.21.15.6:0/1053071065 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6676+0+0 (secure 0 0 0) 0x7feb0c0ab9a0 con 0x7feb200c7dc0 2024-04-24T23:20:05.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:05.220+0000 7feb1d400640 1 --2- 172.21.15.6:0/1053071065 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7feb0006be30 0x7feb0006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:20:05.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:05.221+0000 7feb1d400640 1 --2- 172.21.15.6:0/1053071065 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7feb0006be30 0x7feb0006e2f0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7feb2015fce0 tx=0x7feb10006c20 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:20:05.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:05.225+0000 7feb16000640 1 -- 172.21.15.6:0/1053071065 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7feb200c6fb0 con 0x7feb200c7dc0 2024-04-24T23:20:05.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:05.642+0000 7feb24a00640 1 -- 172.21.15.6:0/1053071065 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7feb200c7970 con 0x7feb200c7dc0 2024-04-24T23:20:05.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:05.643+0000 7feb16000640 1 -- 172.21.15.6:0/1053071065 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+6904 (secure 0 0 0) 0x7feb0c04a030 con 0x7feb200c7dc0 2024-04-24T23:20:05.645 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:20:05.645 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5257,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/3240971655","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":3240971655},{"type":"v1","addr":"172.21.15.6:6837","nonce":3240971655}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":24}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:18:47.823843+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4282,"mds_2":4235,"mds_3":4232,"mds_4":4197},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4197":{"gid":4197,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":53,"addr":"172.21.15.6:6834/105267896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":105267896},{"type":"v1","addr":"172.21.15.6:6834","nonce":105267896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4232":{"gid":4232,"name":"b","rank":3,"incarnation":15,"state":"up:active","state_seq":15,"addr":"172.21.15.190:6838/2032280158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2032280158},{"type":"v1","addr":"172.21.15.190:6838","nonce":2032280158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4235":{"gid":4235,"name":"d","rank":2,"incarnation":13,"state":"up:active","state_seq":14,"addr":"172.21.15.190:6839/1572872451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":1572872451},{"type":"v1","addr":"172.21.15.190:6839","nonce":1572872451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4282":{"gid":4282,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":12,"addr":"172.21.15.190:6837/580315269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":580315269},{"type":"v1","addr":"172.21.15.190:6837","nonce":580315269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/489092393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":489092393},{"type":"v1","addr":"172.21.15.6:6835","nonce":489092393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4306,"qdb_cluster":[4306,4282,4235,4232,4197]},"id":1}]} 2024-04-24T23:20:05.645 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 24 2024-04-24T23:20:05.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:05.648+0000 7feb24a00640 1 -- 172.21.15.6:0/1053071065 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7feb0006be30 msgr2=0x7feb0006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:20:05.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:05.648+0000 7feb24a00640 1 --2- 172.21.15.6:0/1053071065 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7feb0006be30 0x7feb0006e2f0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7feb2015fce0 tx=0x7feb10006c20 comp rx=0 tx=0).stop 2024-04-24T23:20:05.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:05.648+0000 7feb24a00640 1 -- 172.21.15.6:0/1053071065 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feb200c7dc0 msgr2=0x7feb201601e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:20:05.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:05.648+0000 7feb24a00640 1 --2- 172.21.15.6:0/1053071065 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feb200c7dc0 0x7feb201601e0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7feb0c0090b0 tx=0x7feb0c0046c0 comp rx=0 tx=0).stop 2024-04-24T23:20:05.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:05.650+0000 7feb24a00640 1 -- 172.21.15.6:0/1053071065 shutdown_connections 2024-04-24T23:20:05.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:05.650+0000 7feb24a00640 1 --2- 172.21.15.6:0/1053071065 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7feb0006be30 0x7feb0006e2f0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:05.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:05.650+0000 7feb24a00640 1 --2- 172.21.15.6:0/1053071065 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feb200c81c0 0x7feb20160720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:05.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:05.650+0000 7feb24a00640 1 --2- 172.21.15.6:0/1053071065 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feb20159110 0x7feb2015ba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:05.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:05.650+0000 7feb24a00640 1 --2- 172.21.15.6:0/1053071065 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feb200c7dc0 0x7feb201601e0 secure :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7feb0c0090b0 tx=0x7feb0c0046c0 comp rx=0 tx=0).stop 2024-04-24T23:20:05.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:05.650+0000 7feb24a00640 1 -- 172.21.15.6:0/1053071065 >> 172.21.15.6:0/1053071065 conn(0x7feb2006b0e0 msgr2=0x7feb201540c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:20:05.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:05.654+0000 7feb24a00640 1 -- 172.21.15.6:0/1053071065 shutdown_connections 2024-04-24T23:20:05.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:05.654+0000 7feb24a00640 1 -- 172.21.15.6:0/1053071065 wait complete. 2024-04-24T23:20:05.680 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-04-24T23:20:07.680 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:20:07.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:07.915+0000 7fdd7c600640 1 Processor -- start 2024-04-24T23:20:07.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:07.916+0000 7fdd7c600640 1 -- start start 2024-04-24T23:20:07.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:07.928+0000 7fdd7c600640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fdd741583c0 0x7fdd7415af70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:20:07.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:07.928+0000 7fdd7c600640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdd7406f440 0x7fdd7415b4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:20:07.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:07.928+0000 7fdd7c600640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdd740c7ed0 con 0x7fdd7406baa0 2024-04-24T23:20:07.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:07.928+0000 7fdd7c600640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7fdd74070aa0 con 0x7fdd741583c0 2024-04-24T23:20:07.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:07.928+0000 7fdd7c600640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fdd7406fe90 con 0x7fdd7406f440 2024-04-24T23:20:07.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:07.928+0000 7fdd79c00640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fdd741583c0 0x7fdd7415af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:20:07.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:07.928+0000 7fdd7a600640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdd7406f440 0x7fdd7415b4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:20:07.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:07.928+0000 7fdd79c00640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fdd741583c0 0x7fdd7415af70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:51306/0 (socket says 172.21.15.6:51306) 2024-04-24T23:20:07.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:07.928+0000 7fdd79c00640 1 -- 172.21.15.6:0/892788502 learned_addr learned my addr 172.21.15.6:0/892788502 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:20:07.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:07.929+0000 7fdd79c00640 1 -- 172.21.15.6:0/892788502 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdd7406f440 msgr2=0x7fdd7415b4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:20:07.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:07.929+0000 7fdd79c00640 1 --2- 172.21.15.6:0/892788502 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdd7406f440 0x7fdd7415b4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:07.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:07.929+0000 7fdd79c00640 1 -- 172.21.15.6:0/892788502 >> v1:172.21.15.6:6789/0 conn(0x7fdd7406baa0 legacy=0x7fdd740c85a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:20:07.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:07.929+0000 7fdd79c00640 1 -- 172.21.15.6:0/892788502 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd74070170 con 0x7fdd741583c0 2024-04-24T23:20:07.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:07.930+0000 7fdd79c00640 1 --2- 172.21.15.6:0/892788502 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdd741583c0 0x7fdd7415af70 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fdd680098a0 tx=0x7fdd6802a330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cd10e400eb91ce20 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:20:07.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:07.930+0000 7fdd73e00640 1 -- 172.21.15.6:0/892788502 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd68037070 con 0x7fdd741583c0 2024-04-24T23:20:07.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:07.930+0000 7fdd73e00640 1 -- 172.21.15.6:0/892788502 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd68032640 con 0x7fdd741583c0 2024-04-24T23:20:07.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:07.930+0000 7fdd73e00640 1 -- 172.21.15.6:0/892788502 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd68032a10 con 0x7fdd741583c0 2024-04-24T23:20:07.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:07.931+0000 7fdd7c600640 1 -- 172.21.15.6:0/892788502 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdd741583c0 msgr2=0x7fdd7415af70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:20:07.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:07.931+0000 7fdd7c600640 1 --2- 172.21.15.6:0/892788502 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdd741583c0 0x7fdd7415af70 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fdd680098a0 tx=0x7fdd6802a330 comp rx=0 tx=0).stop 2024-04-24T23:20:07.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:07.931+0000 7fdd7c600640 1 -- 172.21.15.6:0/892788502 shutdown_connections 2024-04-24T23:20:07.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:07.931+0000 7fdd7c600640 1 --2- 172.21.15.6:0/892788502 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdd7406f440 0x7fdd7415b4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:07.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:07.931+0000 7fdd7c600640 1 --2- 172.21.15.6:0/892788502 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdd741583c0 0x7fdd7415af70 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:07.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:07.931+0000 7fdd7c600640 1 -- 172.21.15.6:0/892788502 >> 172.21.15.6:0/892788502 conn(0x7fdd740cd290 msgr2=0x7fdd740cd670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:20:07.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:07.931+0000 7fdd7c600640 1 -- 172.21.15.6:0/892788502 shutdown_connections 2024-04-24T23:20:07.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:07.931+0000 7fdd7c600640 1 -- 172.21.15.6:0/892788502 wait complete. 2024-04-24T23:20:07.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:07.933+0000 7fdd7c600640 1 Processor -- start 2024-04-24T23:20:07.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:07.933+0000 7fdd7c600640 1 -- start start 2024-04-24T23:20:07.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:07.934+0000 7fdd7c600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdd7406baa0 0x7fdd741f0940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:20:07.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:07.934+0000 7fdd7c600640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdd7406f440 0x7fdd740cbbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:20:07.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:07.934+0000 7fdd7c600640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdd741583c0 0x7fdd740c6a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:20:07.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:07.934+0000 7fdd7c600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fdd740c7b90 con 0x7fdd7406baa0 2024-04-24T23:20:07.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:07.934+0000 7fdd7c600640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fdd74070730 con 0x7fdd741583c0 2024-04-24T23:20:07.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:07.934+0000 7fdd7c600640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fdd740c8cb0 con 0x7fdd7406f440 2024-04-24T23:20:07.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:07.934+0000 7fdd79c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdd7406baa0 0x7fdd741f0940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:20:07.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:07.935+0000 7fdd79c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdd7406baa0 0x7fdd741f0940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:48100/0 (socket says 172.21.15.6:48100) 2024-04-24T23:20:07.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:07.935+0000 7fdd79c00640 1 -- 172.21.15.6:0/2468914377 learned_addr learned my addr 172.21.15.6:0/2468914377 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:20:07.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:07.935+0000 7fdd7a600640 1 --2- 172.21.15.6:0/2468914377 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdd741583c0 0x7fdd740c6a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:20:07.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:07.935+0000 7fdd79200640 1 --2- 172.21.15.6:0/2468914377 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdd7406f440 0x7fdd740cbbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:20:07.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:07.935+0000 7fdd79c00640 1 -- 172.21.15.6:0/2468914377 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdd7406f440 msgr2=0x7fdd740cbbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:20:07.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:07.935+0000 7fdd79c00640 1 --2- 172.21.15.6:0/2468914377 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdd7406f440 0x7fdd740cbbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:07.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:07.935+0000 7fdd79c00640 1 -- 172.21.15.6:0/2468914377 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdd741583c0 msgr2=0x7fdd740c6a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:20:07.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:07.935+0000 7fdd79c00640 1 --2- 172.21.15.6:0/2468914377 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdd741583c0 0x7fdd740c6a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:07.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:07.936+0000 7fdd79c00640 1 -- 172.21.15.6:0/2468914377 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd740cc200 con 0x7fdd7406baa0 2024-04-24T23:20:07.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:07.936+0000 7fdd79200640 1 --2- 172.21.15.6:0/2468914377 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdd7406f440 0x7fdd740cbbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:20:07.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:07.936+0000 7fdd7a600640 1 --2- 172.21.15.6:0/2468914377 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdd741583c0 0x7fdd740c6a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:20:07.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:07.936+0000 7fdd79c00640 1 --2- 172.21.15.6:0/2468914377 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdd7406baa0 0x7fdd741f0940 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fdd68000c00 tx=0x7fdd680046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:20:07.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:07.937+0000 7fdd71600640 1 -- 172.21.15.6:0/2468914377 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd68037040 con 0x7fdd7406baa0 2024-04-24T23:20:07.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:07.937+0000 7fdd71600640 1 -- 172.21.15.6:0/2468914377 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd68042070 con 0x7fdd7406baa0 2024-04-24T23:20:07.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:07.937+0000 7fdd71600640 1 -- 172.21.15.6:0/2468914377 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd680309b0 con 0x7fdd7406baa0 2024-04-24T23:20:07.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:07.939+0000 7fdd7c600640 1 -- 172.21.15.6:0/2468914377 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd740cc4e0 con 0x7fdd7406baa0 2024-04-24T23:20:07.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:07.940+0000 7fdd7c600640 1 -- 172.21.15.6:0/2468914377 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd740ccaa0 con 0x7fdd7406baa0 2024-04-24T23:20:07.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:07.942+0000 7fdd7c600640 1 -- 172.21.15.6:0/2468914377 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd74068430 con 0x7fdd7406baa0 2024-04-24T23:20:07.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:07.942+0000 7fdd71600640 1 -- 172.21.15.6:0/2468914377 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fdd68030b50 con 0x7fdd7406baa0 2024-04-24T23:20:07.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:07.943+0000 7fdd71600640 1 --2- 172.21.15.6:0/2468914377 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fdd5406be80 0x7fdd5406e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:20:07.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:07.943+0000 7fdd71600640 1 -- 172.21.15.6:0/2468914377 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6676+0+0 (secure 0 0 0) 0x7fdd680aba80 con 0x7fdd7406baa0 2024-04-24T23:20:07.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:07.944+0000 7fdd79200640 1 --2- 172.21.15.6:0/2468914377 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fdd5406be80 0x7fdd5406e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:20:07.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:07.945+0000 7fdd79200640 1 --2- 172.21.15.6:0/2468914377 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fdd5406be80 0x7fdd5406e340 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fdd741f4f70 tx=0x7fdd60009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:20:07.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:07.950+0000 7fdd71600640 1 -- 172.21.15.6:0/2468914377 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fdd680ade20 con 0x7fdd7406baa0 2024-04-24T23:20:08.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:08.223+0000 7fdd7c600640 1 -- 172.21.15.6:0/2468914377 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdd7406bf60 con 0x7fdd7406baa0 2024-04-24T23:20:08.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:08.224+0000 7fdd71600640 1 -- 172.21.15.6:0/2468914377 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+6904 (secure 0 0 0) 0x7fdd6807a040 con 0x7fdd7406baa0 2024-04-24T23:20:08.226 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:20:08.226 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5257,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/3240971655","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":3240971655},{"type":"v1","addr":"172.21.15.6:6837","nonce":3240971655}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":24}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:18:47.823843+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4282,"mds_2":4235,"mds_3":4232,"mds_4":4197},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4197":{"gid":4197,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":53,"addr":"172.21.15.6:6834/105267896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":105267896},{"type":"v1","addr":"172.21.15.6:6834","nonce":105267896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4232":{"gid":4232,"name":"b","rank":3,"incarnation":15,"state":"up:active","state_seq":15,"addr":"172.21.15.190:6838/2032280158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2032280158},{"type":"v1","addr":"172.21.15.190:6838","nonce":2032280158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4235":{"gid":4235,"name":"d","rank":2,"incarnation":13,"state":"up:active","state_seq":14,"addr":"172.21.15.190:6839/1572872451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":1572872451},{"type":"v1","addr":"172.21.15.190:6839","nonce":1572872451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4282":{"gid":4282,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":12,"addr":"172.21.15.190:6837/580315269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":580315269},{"type":"v1","addr":"172.21.15.190:6837","nonce":580315269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/489092393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":489092393},{"type":"v1","addr":"172.21.15.6:6835","nonce":489092393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4306,"qdb_cluster":[4306,4282,4235,4232,4197]},"id":1}]} 2024-04-24T23:20:08.227 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 24 2024-04-24T23:20:08.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:08.226+0000 7fdd7c600640 1 -- 172.21.15.6:0/2468914377 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fdd5406be80 msgr2=0x7fdd5406e340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:20:08.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:08.226+0000 7fdd7c600640 1 --2- 172.21.15.6:0/2468914377 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fdd5406be80 0x7fdd5406e340 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fdd741f4f70 tx=0x7fdd60009040 comp rx=0 tx=0).stop 2024-04-24T23:20:08.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:08.226+0000 7fdd7c600640 1 -- 172.21.15.6:0/2468914377 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdd7406baa0 msgr2=0x7fdd741f0940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:20:08.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:08.226+0000 7fdd7c600640 1 --2- 172.21.15.6:0/2468914377 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdd7406baa0 0x7fdd741f0940 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fdd68000c00 tx=0x7fdd680046c0 comp rx=0 tx=0).stop 2024-04-24T23:20:08.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:08.226+0000 7fdd7c600640 1 -- 172.21.15.6:0/2468914377 shutdown_connections 2024-04-24T23:20:08.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:08.226+0000 7fdd7c600640 1 --2- 172.21.15.6:0/2468914377 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fdd5406be80 0x7fdd5406e340 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:08.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:08.226+0000 7fdd7c600640 1 --2- 172.21.15.6:0/2468914377 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdd7406f440 0x7fdd740cbbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:08.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:08.226+0000 7fdd7c600640 1 --2- 172.21.15.6:0/2468914377 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdd741583c0 0x7fdd740c6a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:08.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:08.226+0000 7fdd7c600640 1 --2- 172.21.15.6:0/2468914377 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdd7406baa0 0x7fdd741f0940 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:08.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:08.226+0000 7fdd7c600640 1 -- 172.21.15.6:0/2468914377 >> 172.21.15.6:0/2468914377 conn(0x7fdd740cd290 msgr2=0x7fdd74152cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:20:08.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:08.227+0000 7fdd7c600640 1 -- 172.21.15.6:0/2468914377 shutdown_connections 2024-04-24T23:20:08.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:08.227+0000 7fdd7c600640 1 -- 172.21.15.6:0/2468914377 wait complete. 2024-04-24T23:20:08.247 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-04-24T23:20:09.023 INFO:tasks.workunit.client.0.smithi006.stdout:Syncing()...16 sec 2024-04-24T23:20:09.027 INFO:tasks.workunit.client.0.smithi006.stdout:Starting Actual Benchmark At: Wed Apr 24 23:20:09 2024 2024-04-24T23:20:09.027 INFO:tasks.workunit.client.0.smithi006.stdout: 2024-04-24T23:20:10.248 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:20:10.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:10.585+0000 7f8ef5a00640 1 Processor -- start 2024-04-24T23:20:10.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:10.585+0000 7f8ef5a00640 1 -- start start 2024-04-24T23:20:10.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:10.591+0000 7f8ef5a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f8ef0072460 0x7f8ef01626a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:20:10.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:10.591+0000 7f8ef5a00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8ef015eec0 con 0x7f8ef006b350 2024-04-24T23:20:10.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:10.591+0000 7f8ef5a00640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8ef00c8a30 con 0x7f8ef006b730 2024-04-24T23:20:10.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:10.591+0000 7f8ef5a00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f8ef015ec60 con 0x7f8ef0072460 2024-04-24T23:20:10.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:10.592+0000 7f8eee000640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f8ef006b730 0x7f8ef0158a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.6:46750/0 (socket says 172.21.15.6:46750) 2024-04-24T23:20:10.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:10.592+0000 7f8eee000640 1 -- 172.21.15.6:0/1610863755 learned_addr learned my addr 172.21.15.6:0/1610863755 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:20:10.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:10.593+0000 7f8eef400640 1 --2- 172.21.15.6:0/1610863755 >> v2:172.21.15.6:3300/0 conn(0x7f8ef0072460 0x7f8ef01626a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:20:10.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:10.593+0000 7f8eed600640 1 -- 172.21.15.6:0/1610863755 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3068399797 0 0) 0x7f8ef00c8a30 con 0x7f8ef006b730 2024-04-24T23:20:10.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:10.593+0000 7f8eed600640 1 -- 172.21.15.6:0/1610863755 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8ed4003520 con 0x7f8ef006b730 2024-04-24T23:20:10.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:10.593+0000 7f8eed600640 1 -- 172.21.15.6:0/1610863755 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1864540529 0 0) 0x7f8ed4003520 con 0x7f8ef006b730 2024-04-24T23:20:10.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:10.593+0000 7f8eed600640 1 -- 172.21.15.6:0/1610863755 >> v2:172.21.15.6:3300/0 conn(0x7f8ef0072460 msgr2=0x7f8ef01626a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:20:10.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:10.594+0000 7f8eed600640 1 --2- 172.21.15.6:0/1610863755 >> v2:172.21.15.6:3300/0 conn(0x7f8ef0072460 0x7f8ef01626a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:10.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:10.594+0000 7f8eed600640 1 -- 172.21.15.6:0/1610863755 >> v1:172.21.15.6:6789/0 conn(0x7f8ef006b350 legacy=0x7f8ef00c93e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:20:10.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:10.594+0000 7f8eed600640 1 -- 172.21.15.6:0/1610863755 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ef015f160 con 0x7f8ef006b730 2024-04-24T23:20:10.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:10.594+0000 7f8eed600640 1 -- 172.21.15.6:0/1610863755 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3463268701 0 0) 0x7f8edc001650 con 0x7f8ef006b730 2024-04-24T23:20:10.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:10.594+0000 7f8eed600640 1 -- 172.21.15.6:0/1610863755 >> v1:172.21.15.190:6789/0 conn(0x7f8ef006b730 legacy=0x7f8ef0158a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:20:10.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:10.594+0000 7f8eed600640 1 --2- 172.21.15.6:0/1610863755 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8ed40039b0 0x7f8ed4003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:20:10.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:10.594+0000 7f8eed600640 1 --2- 172.21.15.6:0/1610863755 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8ed4004800 0x7f8ed4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:20:10.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:10.594+0000 7f8eed600640 1 --2- 172.21.15.6:0/1610863755 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8ed40052a0 0x7f8ed400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:20:10.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:10.594+0000 7f8eed600640 1 -- 172.21.15.6:0/1610863755 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f8ef015ec60 con 0x7f8ed4004800 2024-04-24T23:20:10.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:10.594+0000 7f8eed600640 1 -- 172.21.15.6:0/1610863755 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f8ed400c150 con 0x7f8ed40039b0 2024-04-24T23:20:10.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:10.594+0000 7f8eed600640 1 -- 172.21.15.6:0/1610863755 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f8ed400c350 con 0x7f8ed40052a0 2024-04-24T23:20:10.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:10.595+0000 7f8eef400640 1 --2- 172.21.15.6:0/1610863755 >> v2:172.21.15.6:3300/0 conn(0x7f8ef0072460 0x7f8ef01626a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:20:10.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:10.596+0000 7f8eef400640 1 --2- 172.21.15.6:0/1610863755 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8ed40052a0 0x7f8ed400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:20:10.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:10.597+0000 7f8eee000640 1 --2- 172.21.15.6:0/1610863755 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8ed4004800 0x7f8ed4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:20:10.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:10.599+0000 7f8eef400640 1 -- 172.21.15.6:0/1610863755 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8ed40039b0 msgr2=0x7f8ed4003df0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-24T23:20:10.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:10.599+0000 7f8eef400640 1 --2- 172.21.15.6:0/1610863755 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8ed40039b0 0x7f8ed4003df0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:10.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:10.599+0000 7f8eef400640 1 -- 172.21.15.6:0/1610863755 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8ed4004800 msgr2=0x7f8ed4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:20:10.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:10.599+0000 7f8eef400640 1 --2- 172.21.15.6:0/1610863755 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8ed4004800 0x7f8ed4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:10.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:10.599+0000 7f8eef400640 1 -- 172.21.15.6:0/1610863755 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ef015f160 con 0x7f8ed40052a0 2024-04-24T23:20:10.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:10.600+0000 7f8eef400640 1 -- 172.21.15.6:0/1610863755 reap_dead start 2024-04-24T23:20:10.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:10.600+0000 7f8eef400640 1 --2- 172.21.15.6:0/1610863755 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8ed40052a0 0x7f8ed400b9c0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f8ee001da40 tx=0x7f8ed4005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=13bcf22164857815 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:20:10.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:10.602+0000 7f8eed600640 1 -- 172.21.15.6:0/1610863755 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ee001d640 con 0x7f8ed40052a0 2024-04-24T23:20:10.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:10.602+0000 7f8eed600640 1 -- 172.21.15.6:0/1610863755 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ee0020040 con 0x7f8ed40052a0 2024-04-24T23:20:10.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:10.603+0000 7f8ef5a00640 1 -- 172.21.15.6:0/1610863755 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8ed40052a0 msgr2=0x7f8ed400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:20:10.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:10.604+0000 7f8eed600640 1 -- 172.21.15.6:0/1610863755 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ee001d640 con 0x7f8ed40052a0 2024-04-24T23:20:10.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:10.604+0000 7f8ef5a00640 1 --2- 172.21.15.6:0/1610863755 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8ed40052a0 0x7f8ed400b9c0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f8ee001da40 tx=0x7f8ed4005830 comp rx=0 tx=0).stop 2024-04-24T23:20:10.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:10.605+0000 7f8ef5a00640 1 -- 172.21.15.6:0/1610863755 shutdown_connections 2024-04-24T23:20:10.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:10.605+0000 7f8ef5a00640 1 --2- 172.21.15.6:0/1610863755 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8ed40052a0 0x7f8ed400b9c0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:10.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:10.605+0000 7f8ef5a00640 1 -- 172.21.15.6:0/1610863755 >> 172.21.15.6:0/1610863755 conn(0x7f8ef006cc90 msgr2=0x7f8ef006d070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:20:10.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:10.605+0000 7f8ef5a00640 1 -- 172.21.15.6:0/1610863755 shutdown_connections 2024-04-24T23:20:10.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:10.606+0000 7f8ef5a00640 1 -- 172.21.15.6:0/1610863755 wait complete. 2024-04-24T23:20:10.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:10.609+0000 7f8ef5a00640 1 Processor -- start 2024-04-24T23:20:10.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:10.610+0000 7f8ef5a00640 1 -- start start 2024-04-24T23:20:10.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:10.613+0000 7f8ef5a00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8ed40052a0 0x7f8ef0147eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:20:10.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:10.613+0000 7f8ef5a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8ef0156520 0x7f8ef0145760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:20:10.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:10.613+0000 7f8ef5a00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8ef0157ec0 0x7f8ef0143c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:20:10.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:10.614+0000 7f8ef5a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f8ef015f9e0 con 0x7f8ef0156520 2024-04-24T23:20:10.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:10.614+0000 7f8ef5a00640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f8ef015f700 con 0x7f8ed40052a0 2024-04-24T23:20:10.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:10.614+0000 7f8ef5a00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f8ef00c93e0 con 0x7f8ef0157ec0 2024-04-24T23:20:10.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:10.614+0000 7f8eef400640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8ef0157ec0 0x7f8ef0143c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:20:10.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:10.614+0000 7f8eef400640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8ef0157ec0 0x7f8ef0143c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.6:40440/0 (socket says 172.21.15.6:40440) 2024-04-24T23:20:10.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:10.614+0000 7f8eef400640 1 -- 172.21.15.6:0/734997969 learned_addr learned my addr 172.21.15.6:0/734997969 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:20:10.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:10.614+0000 7f8eeea00640 1 --2- 172.21.15.6:0/734997969 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8ed40052a0 0x7f8ef0147eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:20:10.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:10.615+0000 7f8eee000640 1 --2- 172.21.15.6:0/734997969 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8ef0156520 0x7f8ef0145760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:20:10.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:10.615+0000 7f8eef400640 1 -- 172.21.15.6:0/734997969 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8ed40052a0 msgr2=0x7f8ef0147eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:20:10.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:10.615+0000 7f8eef400640 1 --2- 172.21.15.6:0/734997969 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8ed40052a0 0x7f8ef0147eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:10.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:10.615+0000 7f8eef400640 1 -- 172.21.15.6:0/734997969 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8ef0156520 msgr2=0x7f8ef0145760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:20:10.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:10.616+0000 7f8eef400640 1 --2- 172.21.15.6:0/734997969 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8ef0156520 0x7f8ef0145760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:10.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:10.616+0000 7f8eef400640 1 -- 172.21.15.6:0/734997969 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ef0145ca0 con 0x7f8ef0157ec0 2024-04-24T23:20:10.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:10.617+0000 7f8eef400640 1 --2- 172.21.15.6:0/734997969 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8ef0157ec0 0x7f8ef0143c00 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f8ee001d170 tx=0x7f8ee0004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:20:10.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:10.617+0000 7f8eee000640 1 --2- 172.21.15.6:0/734997969 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8ef0156520 0x7f8ef0145760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:20:10.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:10.617+0000 7f8ee6a00640 1 -- 172.21.15.6:0/734997969 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ee0050070 con 0x7f8ef0157ec0 2024-04-24T23:20:10.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:10.617+0000 7f8ee6a00640 1 -- 172.21.15.6:0/734997969 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ee0003e70 con 0x7f8ef0157ec0 2024-04-24T23:20:10.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:10.617+0000 7f8ee6a00640 1 -- 172.21.15.6:0/734997969 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ee0042680 con 0x7f8ef0157ec0 2024-04-24T23:20:10.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:10.619+0000 7f8ef5a00640 1 -- 172.21.15.6:0/734997969 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ef015dd10 con 0x7f8ef0157ec0 2024-04-24T23:20:10.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:10.619+0000 7f8ef5a00640 1 -- 172.21.15.6:0/734997969 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ef015e2d0 con 0x7f8ef0157ec0 2024-04-24T23:20:10.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:10.621+0000 7f8ef5a00640 1 -- 172.21.15.6:0/734997969 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ef0156050 con 0x7f8ef0157ec0 2024-04-24T23:20:10.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:10.623+0000 7f8ee6a00640 1 -- 172.21.15.6:0/734997969 <== mon.2 v2:172.21.15.190:3301/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f8ee0053070 con 0x7f8ef0157ec0 2024-04-24T23:20:10.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:10.625+0000 7f8ee6a00640 1 --2- 172.21.15.6:0/734997969 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f8ed006bf00 0x7f8ed006e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:20:10.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:10.625+0000 7f8ee6a00640 1 -- 172.21.15.6:0/734997969 <== mon.2 v2:172.21.15.190:3301/0 5 ==== osd_map(27..27 src has 1..27) ==== 6676+0+0 (secure 0 0 0) 0x7f8ee00bcb20 con 0x7f8ef0157ec0 2024-04-24T23:20:10.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:10.625+0000 7f8eeea00640 1 --2- 172.21.15.6:0/734997969 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f8ed006bf00 0x7f8ed006e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:20:10.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:10.626+0000 7f8eeea00640 1 --2- 172.21.15.6:0/734997969 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f8ed006bf00 0x7f8ed006e3c0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f8ef014cb60 tx=0x7f8ee8009450 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:20:10.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:10.633+0000 7f8ee6a00640 1 -- 172.21.15.6:0/734997969 <== mon.2 v2:172.21.15.190:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f8ef0156050 con 0x7f8ef0157ec0 2024-04-24T23:20:11.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:11.286+0000 7f8ef5a00640 1 -- 172.21.15.6:0/734997969 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8ef00c7160 con 0x7f8ef0157ec0 2024-04-24T23:20:11.296 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:20:11.296 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5257,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/3240971655","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":3240971655},{"type":"v1","addr":"172.21.15.6:6837","nonce":3240971655}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":24}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:18:47.823843+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4282,"mds_2":4235,"mds_3":4232,"mds_4":4197},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4197":{"gid":4197,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":53,"addr":"172.21.15.6:6834/105267896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":105267896},{"type":"v1","addr":"172.21.15.6:6834","nonce":105267896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4232":{"gid":4232,"name":"b","rank":3,"incarnation":15,"state":"up:active","state_seq":15,"addr":"172.21.15.190:6838/2032280158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2032280158},{"type":"v1","addr":"172.21.15.190:6838","nonce":2032280158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4235":{"gid":4235,"name":"d","rank":2,"incarnation":13,"state":"up:active","state_seq":14,"addr":"172.21.15.190:6839/1572872451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":1572872451},{"type":"v1","addr":"172.21.15.190:6839","nonce":1572872451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4282":{"gid":4282,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":12,"addr":"172.21.15.190:6837/580315269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":580315269},{"type":"v1","addr":"172.21.15.190:6837","nonce":580315269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/489092393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":489092393},{"type":"v1","addr":"172.21.15.6:6835","nonce":489092393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4306,"qdb_cluster":[4306,4282,4235,4232,4197]},"id":1}]} 2024-04-24T23:20:11.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:11.291+0000 7f8ee6a00640 1 -- 172.21.15.6:0/734997969 <== mon.2 v2:172.21.15.190:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+6904 (secure 0 0 0) 0x7f8ef00c7160 con 0x7f8ef0157ec0 2024-04-24T23:20:11.296 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 24 2024-04-24T23:20:11.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:11.299+0000 7f8ef5a00640 1 -- 172.21.15.6:0/734997969 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f8ed006bf00 msgr2=0x7f8ed006e3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:20:11.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:11.299+0000 7f8ef5a00640 1 --2- 172.21.15.6:0/734997969 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f8ed006bf00 0x7f8ed006e3c0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f8ef014cb60 tx=0x7f8ee8009450 comp rx=0 tx=0).stop 2024-04-24T23:20:11.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:11.299+0000 7f8ef5a00640 1 -- 172.21.15.6:0/734997969 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8ef0157ec0 msgr2=0x7f8ef0143c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:20:11.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:11.299+0000 7f8ef5a00640 1 --2- 172.21.15.6:0/734997969 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8ef0157ec0 0x7f8ef0143c00 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f8ee001d170 tx=0x7f8ee0004730 comp rx=0 tx=0).stop 2024-04-24T23:20:11.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:11.299+0000 7f8ef5a00640 1 -- 172.21.15.6:0/734997969 shutdown_connections 2024-04-24T23:20:11.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:11.299+0000 7f8ef5a00640 1 --2- 172.21.15.6:0/734997969 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f8ed006bf00 0x7f8ed006e3c0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:11.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:11.299+0000 7f8ef5a00640 1 --2- 172.21.15.6:0/734997969 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8ef0157ec0 0x7f8ef0143c00 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:11.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:11.299+0000 7f8ef5a00640 1 --2- 172.21.15.6:0/734997969 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8ef0156520 0x7f8ef0145760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:11.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:11.299+0000 7f8ef5a00640 1 --2- 172.21.15.6:0/734997969 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8ed40052a0 0x7f8ef0147eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:11.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:11.299+0000 7f8ef5a00640 1 -- 172.21.15.6:0/734997969 >> 172.21.15.6:0/734997969 conn(0x7f8ef006cc90 msgr2=0x7f8ef0148f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:20:11.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:11.300+0000 7f8ef5a00640 1 -- 172.21.15.6:0/734997969 shutdown_connections 2024-04-24T23:20:11.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:11.300+0000 7f8ef5a00640 1 -- 172.21.15.6:0/734997969 wait complete. 2024-04-24T23:20:11.333 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-04-24T23:20:13.334 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:20:13.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:13.786+0000 7f4e8bc00640 1 Processor -- start 2024-04-24T23:20:13.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:13.786+0000 7f4e8bc00640 1 -- start start 2024-04-24T23:20:13.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:13.789+0000 7f4e8bc00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f4e8406eba0 0x7f4e840c8fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:20:13.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:13.789+0000 7f4e8bc00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4e8406f010 0x7f4e840c9510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:20:13.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:13.791+0000 7f4e8bc00640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4e84071880 con 0x7f4e841593b0 2024-04-24T23:20:13.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:13.791+0000 7f4e8bc00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f4e840c7520 con 0x7f4e8406eba0 2024-04-24T23:20:13.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:13.791+0000 7f4e8bc00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f4e8406e710 con 0x7f4e8406f010 2024-04-24T23:20:13.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:13.791+0000 7f4e89200640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f4e841593b0 0x7f4e840c6e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.6:46762/0 (socket says 172.21.15.6:46762) 2024-04-24T23:20:13.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:13.791+0000 7f4e89200640 1 -- 172.21.15.6:0/2144685029 learned_addr learned my addr 172.21.15.6:0/2144685029 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:20:13.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:13.791+0000 7f4e83e00640 1 --2- 172.21.15.6:0/2144685029 >> v2:172.21.15.6:3300/0 conn(0x7f4e8406eba0 0x7f4e840c8fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:20:13.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:13.792+0000 7f4e89c00640 1 --2- 172.21.15.6:0/2144685029 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4e8406f010 0x7f4e840c9510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:20:13.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:13.793+0000 7f4e83400640 1 -- 172.21.15.6:0/2144685029 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1178590218 0 0) 0x7f4e84071880 con 0x7f4e841593b0 2024-04-24T23:20:13.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:13.793+0000 7f4e83400640 1 -- 172.21.15.6:0/2144685029 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4e6c003520 con 0x7f4e841593b0 2024-04-24T23:20:13.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:13.793+0000 7f4e83e00640 1 -- 172.21.15.6:0/2144685029 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4e8406f010 msgr2=0x7f4e840c9510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:20:13.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:13.793+0000 7f4e83e00640 1 --2- 172.21.15.6:0/2144685029 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4e8406f010 0x7f4e840c9510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:13.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:13.793+0000 7f4e83e00640 1 -- 172.21.15.6:0/2144685029 >> v1:172.21.15.190:6789/0 conn(0x7f4e841593b0 legacy=0x7f4e840c6e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:20:13.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:13.793+0000 7f4e83e00640 1 -- 172.21.15.6:0/2144685029 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e8415f200 con 0x7f4e8406eba0 2024-04-24T23:20:13.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:13.795+0000 7f4e83e00640 1 --2- 172.21.15.6:0/2144685029 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4e8406eba0 0x7f4e840c8fd0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f4e700097a0 tx=0x7f4e7002a250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a70cbc93ce4160e server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:20:13.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:13.796+0000 7f4e83400640 1 -- 172.21.15.6:0/2144685029 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e70037070 con 0x7f4e8406eba0 2024-04-24T23:20:13.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:13.797+0000 7f4e83400640 1 -- 172.21.15.6:0/2144685029 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e70032440 con 0x7f4e8406eba0 2024-04-24T23:20:13.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:13.797+0000 7f4e83400640 1 -- 172.21.15.6:0/2144685029 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e70037070 con 0x7f4e8406eba0 2024-04-24T23:20:13.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:13.801+0000 7f4e8bc00640 1 -- 172.21.15.6:0/2144685029 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4e8406eba0 msgr2=0x7f4e840c8fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:20:13.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:13.801+0000 7f4e8bc00640 1 --2- 172.21.15.6:0/2144685029 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4e8406eba0 0x7f4e840c8fd0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f4e700097a0 tx=0x7f4e7002a250 comp rx=0 tx=0).stop 2024-04-24T23:20:13.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:13.803+0000 7f4e8bc00640 1 -- 172.21.15.6:0/2144685029 shutdown_connections 2024-04-24T23:20:13.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:13.804+0000 7f4e8bc00640 1 --2- 172.21.15.6:0/2144685029 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4e8406f010 0x7f4e840c9510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:13.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:13.804+0000 7f4e8bc00640 1 --2- 172.21.15.6:0/2144685029 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4e8406eba0 0x7f4e840c8fd0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:13.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:13.804+0000 7f4e8bc00640 1 -- 172.21.15.6:0/2144685029 >> 172.21.15.6:0/2144685029 conn(0x7f4e8406b0e0 msgr2=0x7f4e8406b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:20:13.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:13.805+0000 7f4e8bc00640 1 -- 172.21.15.6:0/2144685029 shutdown_connections 2024-04-24T23:20:13.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:13.805+0000 7f4e8bc00640 1 -- 172.21.15.6:0/2144685029 wait complete. 2024-04-24T23:20:13.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:13.808+0000 7f4e8bc00640 1 Processor -- start 2024-04-24T23:20:13.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:13.811+0000 7f4e8bc00640 1 -- start start 2024-04-24T23:20:13.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:13.815+0000 7f4e8bc00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4e8406eba0 0x7f4e8415e510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:20:13.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:13.815+0000 7f4e8bc00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4e8406f010 0x7f4e84152680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:20:13.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:13.815+0000 7f4e8bc00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4e841593b0 0x7f4e84152bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:20:13.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:13.816+0000 7f4e83e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4e8406f010 0x7f4e84152680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:20:13.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:13.816+0000 7f4e89c00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4e841593b0 0x7f4e84152bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:20:13.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:13.816+0000 7f4e89c00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4e841593b0 0x7f4e84152bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:51344/0 (socket says 172.21.15.6:51344) 2024-04-24T23:20:13.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:13.816+0000 7f4e89c00640 1 -- 172.21.15.6:0/3633318267 learned_addr learned my addr 172.21.15.6:0/3633318267 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:20:13.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:13.819+0000 7f4e8bc00640 1 -- 172.21.15.6:0/3633318267 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f4e8415fa80 con 0x7f4e8406f010 2024-04-24T23:20:13.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:13.819+0000 7f4e8bc00640 1 -- 172.21.15.6:0/3633318267 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f4e8415f7a0 con 0x7f4e841593b0 2024-04-24T23:20:13.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:13.819+0000 7f4e8bc00640 1 -- 172.21.15.6:0/3633318267 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f4e8415fd60 con 0x7f4e8406eba0 2024-04-24T23:20:13.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:13.819+0000 7f4e89200640 1 --2- 172.21.15.6:0/3633318267 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4e8406eba0 0x7f4e8415e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:20:13.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:13.819+0000 7f4e89200640 1 -- 172.21.15.6:0/3633318267 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4e841593b0 msgr2=0x7f4e84152bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:20:13.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:13.819+0000 7f4e89200640 1 --2- 172.21.15.6:0/3633318267 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4e841593b0 0x7f4e84152bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:13.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:13.819+0000 7f4e89200640 1 -- 172.21.15.6:0/3633318267 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4e8406f010 msgr2=0x7f4e84152680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:20:13.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:13.820+0000 7f4e89200640 1 --2- 172.21.15.6:0/3633318267 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4e8406f010 0x7f4e84152680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:13.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:13.820+0000 7f4e89200640 1 -- 172.21.15.6:0/3633318267 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e841fe2f0 con 0x7f4e8406eba0 2024-04-24T23:20:13.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:13.821+0000 7f4e83e00640 1 --2- 172.21.15.6:0/3633318267 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4e8406f010 0x7f4e84152680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:20:13.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:13.822+0000 7f4e89200640 1 --2- 172.21.15.6:0/3633318267 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4e8406eba0 0x7f4e8415e510 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f4e7800c430 tx=0x7f4e7800c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:20:13.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:13.822+0000 7f4e80c00640 1 -- 172.21.15.6:0/3633318267 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e78017070 con 0x7f4e8406eba0 2024-04-24T23:20:13.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:13.823+0000 7f4e80c00640 1 -- 172.21.15.6:0/3633318267 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e78005ce0 con 0x7f4e8406eba0 2024-04-24T23:20:13.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:13.823+0000 7f4e80c00640 1 -- 172.21.15.6:0/3633318267 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e780072f0 con 0x7f4e8406eba0 2024-04-24T23:20:13.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:13.823+0000 7f4e8bc00640 1 -- 172.21.15.6:0/3633318267 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e8415d750 con 0x7f4e8406eba0 2024-04-24T23:20:13.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:13.823+0000 7f4e8bc00640 1 -- 172.21.15.6:0/3633318267 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e8415dd10 con 0x7f4e8406eba0 2024-04-24T23:20:13.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:13.824+0000 7f4e8bc00640 1 -- 172.21.15.6:0/3633318267 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e60005290 con 0x7f4e8406eba0 2024-04-24T23:20:13.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:13.826+0000 7f4e80c00640 1 -- 172.21.15.6:0/3633318267 <== mon.2 v2:172.21.15.190:3301/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f4e7801a070 con 0x7f4e8406eba0 2024-04-24T23:20:13.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:13.827+0000 7f4e80c00640 1 --2- 172.21.15.6:0/3633318267 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f4e6406be30 0x7f4e6406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:20:13.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:13.827+0000 7f4e80c00640 1 -- 172.21.15.6:0/3633318267 <== mon.2 v2:172.21.15.190:3301/0 5 ==== osd_map(27..27 src has 1..27) ==== 6676+0+0 (secure 0 0 0) 0x7f4e7808a5d0 con 0x7f4e8406eba0 2024-04-24T23:20:13.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:13.829+0000 7f4e83e00640 1 --2- 172.21.15.6:0/3633318267 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f4e6406be30 0x7f4e6406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:20:13.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:13.830+0000 7f4e83e00640 1 --2- 172.21.15.6:0/3633318267 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f4e6406be30 0x7f4e6406e2f0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f4e70034000 tx=0x7f4e7002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:20:13.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:13.831+0000 7f4e80c00640 1 -- 172.21.15.6:0/3633318267 <== mon.2 v2:172.21.15.190:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f4e78007e20 con 0x7f4e8406eba0 2024-04-24T23:20:14.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:14.403+0000 7f4e8bc00640 1 -- 172.21.15.6:0/3633318267 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4e60005580 con 0x7f4e8406eba0 2024-04-24T23:20:14.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:14.404+0000 7f4e80c00640 1 -- 172.21.15.6:0/3633318267 <== mon.2 v2:172.21.15.190:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+6904 (secure 0 0 0) 0x7f4e78058430 con 0x7f4e8406eba0 2024-04-24T23:20:14.407 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:20:14.407 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5257,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/3240971655","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":3240971655},{"type":"v1","addr":"172.21.15.6:6837","nonce":3240971655}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":24}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:18:47.823843+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4282,"mds_2":4235,"mds_3":4232,"mds_4":4197},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4197":{"gid":4197,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":53,"addr":"172.21.15.6:6834/105267896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":105267896},{"type":"v1","addr":"172.21.15.6:6834","nonce":105267896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4232":{"gid":4232,"name":"b","rank":3,"incarnation":15,"state":"up:active","state_seq":15,"addr":"172.21.15.190:6838/2032280158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2032280158},{"type":"v1","addr":"172.21.15.190:6838","nonce":2032280158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4235":{"gid":4235,"name":"d","rank":2,"incarnation":13,"state":"up:active","state_seq":14,"addr":"172.21.15.190:6839/1572872451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":1572872451},{"type":"v1","addr":"172.21.15.190:6839","nonce":1572872451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4282":{"gid":4282,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":12,"addr":"172.21.15.190:6837/580315269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":580315269},{"type":"v1","addr":"172.21.15.190:6837","nonce":580315269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/489092393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":489092393},{"type":"v1","addr":"172.21.15.6:6835","nonce":489092393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4306,"qdb_cluster":[4306,4282,4235,4232,4197]},"id":1}]} 2024-04-24T23:20:14.407 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 24 2024-04-24T23:20:14.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:14.416+0000 7f4e8bc00640 1 -- 172.21.15.6:0/3633318267 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f4e6406be30 msgr2=0x7f4e6406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:20:14.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:14.416+0000 7f4e8bc00640 1 --2- 172.21.15.6:0/3633318267 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f4e6406be30 0x7f4e6406e2f0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f4e70034000 tx=0x7f4e7002e000 comp rx=0 tx=0).stop 2024-04-24T23:20:14.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:14.417+0000 7f4e8bc00640 1 -- 172.21.15.6:0/3633318267 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4e8406eba0 msgr2=0x7f4e8415e510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:20:14.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:14.417+0000 7f4e8bc00640 1 --2- 172.21.15.6:0/3633318267 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4e8406eba0 0x7f4e8415e510 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f4e7800c430 tx=0x7f4e7800c8f0 comp rx=0 tx=0).stop 2024-04-24T23:20:14.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:14.417+0000 7f4e8bc00640 1 -- 172.21.15.6:0/3633318267 shutdown_connections 2024-04-24T23:20:14.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:14.417+0000 7f4e8bc00640 1 --2- 172.21.15.6:0/3633318267 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f4e6406be30 0x7f4e6406e2f0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:14.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:14.417+0000 7f4e8bc00640 1 --2- 172.21.15.6:0/3633318267 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4e841593b0 0x7f4e84152bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:14.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:14.417+0000 7f4e8bc00640 1 --2- 172.21.15.6:0/3633318267 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4e8406f010 0x7f4e84152680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:14.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:14.417+0000 7f4e8bc00640 1 --2- 172.21.15.6:0/3633318267 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4e8406eba0 0x7f4e8415e510 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:14.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:14.417+0000 7f4e8bc00640 1 -- 172.21.15.6:0/3633318267 >> 172.21.15.6:0/3633318267 conn(0x7f4e8406b0e0 msgr2=0x7f4e841584a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:20:14.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:14.418+0000 7f4e8bc00640 1 -- 172.21.15.6:0/3633318267 shutdown_connections 2024-04-24T23:20:14.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:14.418+0000 7f4e8bc00640 1 -- 172.21.15.6:0/3633318267 wait complete. 2024-04-24T23:20:14.483 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-04-24T23:20:16.483 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:20:16.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:16.754+0000 7f6032600640 1 Processor -- start 2024-04-24T23:20:16.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:16.754+0000 7f6032600640 1 -- start start 2024-04-24T23:20:16.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:16.756+0000 7f6032600640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f602c0c8f50 0x7f602c06e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:20:16.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:16.756+0000 7f6032600640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f602c0c9330 0x7f602c06e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:20:16.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:16.756+0000 7f6032600640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f602c0c6e70 con 0x7f602c152bc0 2024-04-24T23:20:16.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:16.756+0000 7f6032600640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f602c0c73d0 con 0x7f602c0c8f50 2024-04-24T23:20:16.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:16.756+0000 7f6032600640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f602c0c8ac0 con 0x7f602c0c9330 2024-04-24T23:20:16.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:16.756+0000 7f602b400640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f602c152bc0 0x7f602c159390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.6:56860/0 (socket says 172.21.15.6:56860) 2024-04-24T23:20:16.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:16.756+0000 7f602b400640 1 -- 172.21.15.6:0/1413621575 learned_addr learned my addr 172.21.15.6:0/1413621575 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:20:16.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:16.757+0000 7f602a000640 1 -- 172.21.15.6:0/1413621575 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1484982868 0 0) 0x7f602c0c6e70 con 0x7f602c152bc0 2024-04-24T23:20:16.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:16.757+0000 7f602a000640 1 -- 172.21.15.6:0/1413621575 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6010003520 con 0x7f602c152bc0 2024-04-24T23:20:16.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:16.757+0000 7f602a000640 1 -- 172.21.15.6:0/1413621575 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1531144046 0 0) 0x7f6010003520 con 0x7f602c152bc0 2024-04-24T23:20:16.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:16.757+0000 7f602a000640 1 -- 172.21.15.6:0/1413621575 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f602c0c9330 msgr2=0x7f602c06e8e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-24T23:20:16.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:16.757+0000 7f602a000640 1 --2- 172.21.15.6:0/1413621575 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f602c0c9330 0x7f602c06e8e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:16.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:16.757+0000 7f602a000640 1 -- 172.21.15.6:0/1413621575 >> v2:172.21.15.190:3300/0 conn(0x7f602c0c8f50 msgr2=0x7f602c06e3a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-24T23:20:16.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:16.757+0000 7f602a000640 1 --2- 172.21.15.6:0/1413621575 >> v2:172.21.15.190:3300/0 conn(0x7f602c0c8f50 0x7f602c06e3a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:16.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:16.757+0000 7f602a000640 1 -- 172.21.15.6:0/1413621575 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f602c0c7110 con 0x7f602c152bc0 2024-04-24T23:20:16.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:16.758+0000 7f602a000640 1 -- 172.21.15.6:0/1413621575 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3463268701 0 0) 0x7f6018001690 con 0x7f602c152bc0 2024-04-24T23:20:16.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:16.758+0000 7f602a000640 1 -- 172.21.15.6:0/1413621575 >> v1:172.21.15.190:6789/0 conn(0x7f602c152bc0 legacy=0x7f602c159390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:20:16.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:16.758+0000 7f602a000640 1 --2- 172.21.15.6:0/1413621575 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f60100039b0 0x7f6010003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:20:16.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:16.759+0000 7f602a000640 1 --2- 172.21.15.6:0/1413621575 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6010004800 0x7f6010004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:20:16.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:16.759+0000 7f602a000640 1 --2- 172.21.15.6:0/1413621575 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f60100053c0 0x7f601000fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:20:16.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:16.759+0000 7f602a000640 1 -- 172.21.15.6:0/1413621575 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f602c0c73d0 con 0x7f60100039b0 2024-04-24T23:20:16.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:16.759+0000 7f602a000640 1 -- 172.21.15.6:0/1413621575 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f602c0c8ac0 con 0x7f60100053c0 2024-04-24T23:20:16.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:16.759+0000 7f602a000640 1 -- 172.21.15.6:0/1413621575 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f60100106c0 con 0x7f6010004800 2024-04-24T23:20:16.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:16.759+0000 7f602b400640 1 --2- 172.21.15.6:0/1413621575 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f60100039b0 0x7f6010003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:20:16.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:16.760+0000 7f602aa00640 1 --2- 172.21.15.6:0/1413621575 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6010004800 0x7f6010004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:20:16.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:16.760+0000 7f602aa00640 1 -- 172.21.15.6:0/1413621575 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f60100053c0 msgr2=0x7f601000fdd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-24T23:20:16.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:16.760+0000 7f602aa00640 1 --2- 172.21.15.6:0/1413621575 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f60100053c0 0x7f601000fdd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:16.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:16.760+0000 7f602aa00640 1 -- 172.21.15.6:0/1413621575 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f60100039b0 msgr2=0x7f6010003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:20:16.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:16.760+0000 7f602aa00640 1 --2- 172.21.15.6:0/1413621575 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f60100039b0 0x7f6010003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:16.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:16.760+0000 7f602aa00640 1 -- 172.21.15.6:0/1413621575 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6010003520 con 0x7f6010004800 2024-04-24T23:20:16.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:16.762+0000 7f602aa00640 1 --2- 172.21.15.6:0/1413621575 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6010004800 0x7f6010004c80 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f6024009e60 tx=0x7f6010005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c5f56e5cd06e97c8 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:20:16.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:16.762+0000 7f602a000640 1 -- 172.21.15.6:0/1413621575 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6024033070 con 0x7f6010004800 2024-04-24T23:20:16.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:16.762+0000 7f602a000640 1 -- 172.21.15.6:0/1413621575 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f602402f640 con 0x7f6010004800 2024-04-24T23:20:16.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:16.762+0000 7f602a000640 1 -- 172.21.15.6:0/1413621575 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6024009a30 con 0x7f6010004800 2024-04-24T23:20:16.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:16.763+0000 7f6032600640 1 -- 172.21.15.6:0/1413621575 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6010004800 msgr2=0x7f6010004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:20:16.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:16.763+0000 7f6032600640 1 --2- 172.21.15.6:0/1413621575 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6010004800 0x7f6010004c80 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f6024009e60 tx=0x7f6010005830 comp rx=0 tx=0).stop 2024-04-24T23:20:16.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:16.764+0000 7f602be00640 1 -- 172.21.15.6:0/1413621575 reap_dead start 2024-04-24T23:20:16.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:16.764+0000 7f6032600640 1 -- 172.21.15.6:0/1413621575 shutdown_connections 2024-04-24T23:20:16.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:16.764+0000 7f6032600640 1 -- 172.21.15.6:0/1413621575 >> 172.21.15.6:0/1413621575 conn(0x7f602c06b0e0 msgr2=0x7f602c06b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:20:16.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:16.764+0000 7f6032600640 1 -- 172.21.15.6:0/1413621575 shutdown_connections 2024-04-24T23:20:16.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:16.764+0000 7f6032600640 1 -- 172.21.15.6:0/1413621575 wait complete. 2024-04-24T23:20:16.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:16.767+0000 7f6032600640 1 Processor -- start 2024-04-24T23:20:16.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:16.767+0000 7f6032600640 1 -- start start 2024-04-24T23:20:16.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:16.769+0000 7f6032600640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f602c0c1980 0x7f602c155cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:20:16.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:16.769+0000 7f6032600640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f602c0c1d60 0x7f602c14e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:20:16.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:16.769+0000 7f6032600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f602c070f00 0x7f602c14eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:20:16.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:16.769+0000 7f6032600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f602c0c8750 con 0x7f602c070f00 2024-04-24T23:20:16.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:16.769+0000 7f6032600640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f602c0c8470 con 0x7f602c0c1d60 2024-04-24T23:20:16.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:16.769+0000 7f6032600640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f602c152fa0 con 0x7f602c0c1980 2024-04-24T23:20:16.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:16.769+0000 7f602aa00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f602c0c1d60 0x7f602c14e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:20:16.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:16.769+0000 7f602aa00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f602c0c1d60 0x7f602c14e5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:37098/0 (socket says 172.21.15.6:37098) 2024-04-24T23:20:16.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:16.769+0000 7f602aa00640 1 -- 172.21.15.6:0/275983191 learned_addr learned my addr 172.21.15.6:0/275983191 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:20:16.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:16.769+0000 7f602b400640 1 --2- 172.21.15.6:0/275983191 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f602c0c1980 0x7f602c155cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:20:16.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:16.770+0000 7f602aa00640 1 -- 172.21.15.6:0/275983191 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f602c0c1980 msgr2=0x7f602c155cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:20:16.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:16.770+0000 7f602aa00640 1 --2- 172.21.15.6:0/275983191 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f602c0c1980 0x7f602c155cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:16.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:16.770+0000 7f602aa00640 1 -- 172.21.15.6:0/275983191 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f602c070f00 msgr2=0x7f602c14eb30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-24T23:20:16.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:16.770+0000 7f602aa00640 1 --2- 172.21.15.6:0/275983191 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f602c070f00 0x7f602c14eb30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:16.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:16.770+0000 7f602aa00640 1 -- 172.21.15.6:0/275983191 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f602c1f9410 con 0x7f602c0c1d60 2024-04-24T23:20:16.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:16.770+0000 7f602aa00640 1 --2- 172.21.15.6:0/275983191 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f602c0c1d60 0x7f602c14e5f0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f6024009f80 tx=0x7f6024004a80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:20:16.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:16.770+0000 7f6023400640 1 -- 172.21.15.6:0/275983191 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6024033040 con 0x7f602c0c1d60 2024-04-24T23:20:16.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:16.770+0000 7f6023400640 1 -- 172.21.15.6:0/275983191 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6024039690 con 0x7f602c0c1d60 2024-04-24T23:20:16.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:16.770+0000 7f6023400640 1 -- 172.21.15.6:0/275983191 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6024039a60 con 0x7f602c0c1d60 2024-04-24T23:20:16.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:16.770+0000 7f6032600640 1 -- 172.21.15.6:0/275983191 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f602c072990 con 0x7f602c0c1d60 2024-04-24T23:20:16.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:16.771+0000 7f6032600640 1 -- 172.21.15.6:0/275983191 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f602c1e6150 con 0x7f602c0c1d60 2024-04-24T23:20:16.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:16.774+0000 7f6023400640 1 -- 172.21.15.6:0/275983191 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f6024036070 con 0x7f602c0c1d60 2024-04-24T23:20:16.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:16.775+0000 7f6032600640 1 -- 172.21.15.6:0/275983191 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f602c1e6350 con 0x7f602c0c1d60 2024-04-24T23:20:16.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:16.776+0000 7f6023400640 1 --2- 172.21.15.6:0/275983191 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f600c06beb0 0x7f600c06e370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:20:16.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:16.776+0000 7f6023400640 1 -- 172.21.15.6:0/275983191 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6676+0+0 (secure 0 0 0) 0x7f60240bd5d0 con 0x7f602c0c1d60 2024-04-24T23:20:16.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:16.777+0000 7f602b400640 1 --2- 172.21.15.6:0/275983191 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f600c06beb0 0x7f600c06e370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:20:16.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:16.778+0000 7f602b400640 1 --2- 172.21.15.6:0/275983191 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f600c06beb0 0x7f600c06e370 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f6018004840 tx=0x7f6018009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:20:16.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:16.785+0000 7f6023400640 1 -- 172.21.15.6:0/275983191 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f602c1e6350 con 0x7f602c0c1d60 2024-04-24T23:20:17.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:17.528+0000 7f6032600640 1 -- 172.21.15.6:0/275983191 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f602c0767b0 con 0x7f602c0c1d60 2024-04-24T23:20:17.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:17.529+0000 7f6023400640 1 -- 172.21.15.6:0/275983191 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+6904 (secure 0 0 0) 0x7f6024008030 con 0x7f602c0c1d60 2024-04-24T23:20:17.532 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:20:17.532 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5257,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/3240971655","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":3240971655},{"type":"v1","addr":"172.21.15.6:6837","nonce":3240971655}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":24}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:18:47.823843+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4282,"mds_2":4235,"mds_3":4232,"mds_4":4197},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4197":{"gid":4197,"name":"a","rank":4,"incarnation":19,"state":"up:active","state_seq":53,"addr":"172.21.15.6:6834/105267896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":105267896},{"type":"v1","addr":"172.21.15.6:6834","nonce":105267896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4232":{"gid":4232,"name":"b","rank":3,"incarnation":15,"state":"up:active","state_seq":15,"addr":"172.21.15.190:6838/2032280158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2032280158},{"type":"v1","addr":"172.21.15.190:6838","nonce":2032280158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4235":{"gid":4235,"name":"d","rank":2,"incarnation":13,"state":"up:active","state_seq":14,"addr":"172.21.15.190:6839/1572872451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":1572872451},{"type":"v1","addr":"172.21.15.190:6839","nonce":1572872451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4282":{"gid":4282,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":12,"addr":"172.21.15.190:6837/580315269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":580315269},{"type":"v1","addr":"172.21.15.190:6837","nonce":580315269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/489092393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":489092393},{"type":"v1","addr":"172.21.15.6:6835","nonce":489092393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4306,"qdb_cluster":[4306,4282,4235,4232,4197]},"id":1}]} 2024-04-24T23:20:17.532 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 24 2024-04-24T23:20:17.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:17.539+0000 7f6032600640 1 -- 172.21.15.6:0/275983191 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f600c06beb0 msgr2=0x7f600c06e370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:20:17.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:17.539+0000 7f6032600640 1 --2- 172.21.15.6:0/275983191 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f600c06beb0 0x7f600c06e370 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f6018004840 tx=0x7f6018009040 comp rx=0 tx=0).stop 2024-04-24T23:20:17.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:17.539+0000 7f6032600640 1 -- 172.21.15.6:0/275983191 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f602c0c1d60 msgr2=0x7f602c14e5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:20:17.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:17.539+0000 7f6032600640 1 --2- 172.21.15.6:0/275983191 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f602c0c1d60 0x7f602c14e5f0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f6024009f80 tx=0x7f6024004a80 comp rx=0 tx=0).stop 2024-04-24T23:20:17.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:17.543+0000 7f6032600640 1 -- 172.21.15.6:0/275983191 shutdown_connections 2024-04-24T23:20:17.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:17.543+0000 7f6032600640 1 --2- 172.21.15.6:0/275983191 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f600c06beb0 0x7f600c06e370 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:17.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:17.543+0000 7f6032600640 1 --2- 172.21.15.6:0/275983191 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f602c070f00 0x7f602c14eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:17.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:17.543+0000 7f6032600640 1 --2- 172.21.15.6:0/275983191 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f602c0c1d60 0x7f602c14e5f0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:17.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:17.543+0000 7f6032600640 1 --2- 172.21.15.6:0/275983191 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f602c0c1980 0x7f602c155cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:17.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:17.543+0000 7f6032600640 1 -- 172.21.15.6:0/275983191 >> 172.21.15.6:0/275983191 conn(0x7f602c06b0e0 msgr2=0x7f602c1500e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:20:17.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:17.543+0000 7f6032600640 1 -- 172.21.15.6:0/275983191 shutdown_connections 2024-04-24T23:20:17.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:17.544+0000 7f6032600640 1 -- 172.21.15.6:0/275983191 wait complete. 2024-04-24T23:20:17.600 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-04-24T23:20:19.601 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:20:19.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:19.933+0000 7fb700a00640 1 Processor -- start 2024-04-24T23:20:19.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:19.934+0000 7fb700a00640 1 -- start start 2024-04-24T23:20:19.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:19.936+0000 7fb700a00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb6fc06dd40 0x7fb6fc150a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:20:19.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:19.936+0000 7fb700a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fb6fc0cc930 0x7fb6fc0c8ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:20:19.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:19.936+0000 7fb700a00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb6fc0c7720 con 0x7fb6fc0ccd50 2024-04-24T23:20:19.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:19.936+0000 7fb700a00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7fb6fc15d230 con 0x7fb6fc0cc930 2024-04-24T23:20:19.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:19.936+0000 7fb700a00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fb6fc150fa0 con 0x7fb6fc06dd40 2024-04-24T23:20:19.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:19.937+0000 7fb6fb400640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb6fc06dd40 0x7fb6fc150a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:20:19.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:19.937+0000 7fb6fb400640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb6fc06dd40 0x7fb6fc150a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.6:56920/0 (socket says 172.21.15.6:56920) 2024-04-24T23:20:19.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:19.937+0000 7fb6fb400640 1 -- 172.21.15.6:0/2399723460 learned_addr learned my addr 172.21.15.6:0/2399723460 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:20:19.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:19.937+0000 7fb6fb400640 1 -- 172.21.15.6:0/2399723460 >> v2:172.21.15.6:3300/0 conn(0x7fb6fc0cc930 msgr2=0x7fb6fc0c8ae0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-24T23:20:19.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:19.937+0000 7fb6fb400640 1 --2- 172.21.15.6:0/2399723460 >> v2:172.21.15.6:3300/0 conn(0x7fb6fc0cc930 0x7fb6fc0c8ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:19.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:19.937+0000 7fb6fb400640 1 -- 172.21.15.6:0/2399723460 >> v1:172.21.15.6:6789/0 conn(0x7fb6fc0ccd50 legacy=0x7fb6fc159e70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-24T23:20:19.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:19.938+0000 7fb6fb400640 1 -- 172.21.15.6:0/2399723460 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6fc06e2a0 con 0x7fb6fc06dd40 2024-04-24T23:20:19.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:19.948+0000 7fb6fb400640 1 --2- 172.21.15.6:0/2399723460 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb6fc06dd40 0x7fb6fc150a60 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fb6fc0c78f0 tx=0x7fb6ec03b2b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e53cdde3e9274b89 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:20:19.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:19.949+0000 7fb6fa000640 1 -- 172.21.15.6:0/2399723460 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6ec048070 con 0x7fb6fc06dd40 2024-04-24T23:20:19.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:19.949+0000 7fb6fa000640 1 -- 172.21.15.6:0/2399723460 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb6ec043440 con 0x7fb6fc06dd40 2024-04-24T23:20:19.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:19.949+0000 7fb6fa000640 1 -- 172.21.15.6:0/2399723460 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6ec043810 con 0x7fb6fc06dd40 2024-04-24T23:20:19.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:19.950+0000 7fb700a00640 1 -- 172.21.15.6:0/2399723460 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb6fc06dd40 msgr2=0x7fb6fc150a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:20:19.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:19.950+0000 7fb700a00640 1 --2- 172.21.15.6:0/2399723460 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb6fc06dd40 0x7fb6fc150a60 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fb6fc0c78f0 tx=0x7fb6ec03b2b0 comp rx=0 tx=0).stop 2024-04-24T23:20:19.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:19.952+0000 7fb700a00640 1 -- 172.21.15.6:0/2399723460 shutdown_connections 2024-04-24T23:20:19.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:19.952+0000 7fb700a00640 1 --2- 172.21.15.6:0/2399723460 >> v2:172.21.15.6:3300/0 conn(0x7fb6fc0cc930 0x7fb6fc0c8ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:19.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:19.952+0000 7fb700a00640 1 --2- 172.21.15.6:0/2399723460 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb6fc06dd40 0x7fb6fc150a60 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:19.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:19.952+0000 7fb700a00640 1 -- 172.21.15.6:0/2399723460 >> 172.21.15.6:0/2399723460 conn(0x7fb6fc06ed80 msgr2=0x7fb6fc06f190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:20:19.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:19.953+0000 7fb700a00640 1 -- 172.21.15.6:0/2399723460 shutdown_connections 2024-04-24T23:20:19.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:19.953+0000 7fb700a00640 1 -- 172.21.15.6:0/2399723460 wait complete. 2024-04-24T23:20:19.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:19.955+0000 7fb700a00640 1 Processor -- start 2024-04-24T23:20:19.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:19.957+0000 7fb700a00640 1 -- start start 2024-04-24T23:20:19.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:19.959+0000 7fb700a00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb6fc06dd40 0x7fb6fc1f01b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:20:19.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:19.959+0000 7fb700a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb6fc0cc930 0x7fb6fc14b950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:20:19.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:19.959+0000 7fb700a00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb6fc0ccd50 0x7fb6fc142a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:20:19.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:19.959+0000 7fb700a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fb6fc0c7f80 con 0x7fb6fc0cc930 2024-04-24T23:20:19.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:19.959+0000 7fb700a00640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fb6fc0c7ca0 con 0x7fb6fc06dd40 2024-04-24T23:20:19.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:19.959+0000 7fb700a00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fb6fc0c7240 con 0x7fb6fc0ccd50 2024-04-24T23:20:19.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:19.959+0000 7fb6fbe00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb6fc0ccd50 0x7fb6fc142a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:20:19.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:19.959+0000 7fb6fbe00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb6fc0ccd50 0x7fb6fc142a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.6:56924/0 (socket says 172.21.15.6:56924) 2024-04-24T23:20:19.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:19.959+0000 7fb6fbe00640 1 -- 172.21.15.6:0/3422671087 learned_addr learned my addr 172.21.15.6:0/3422671087 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:20:19.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:19.962+0000 7fb6fbe00640 1 -- 172.21.15.6:0/3422671087 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb6fc06dd40 msgr2=0x7fb6fc1f01b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-24T23:20:19.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:19.962+0000 7fb6fbe00640 1 --2- 172.21.15.6:0/3422671087 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb6fc06dd40 0x7fb6fc1f01b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:19.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:19.962+0000 7fb6fbe00640 1 -- 172.21.15.6:0/3422671087 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb6fc0cc930 msgr2=0x7fb6fc14b950 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-24T23:20:19.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:19.962+0000 7fb6fbe00640 1 --2- 172.21.15.6:0/3422671087 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb6fc0cc930 0x7fb6fc14b950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:19.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:19.962+0000 7fb6fbe00640 1 -- 172.21.15.6:0/3422671087 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6fc1ef9f0 con 0x7fb6fc0ccd50 2024-04-24T23:20:19.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:19.962+0000 7fb6fbe00640 1 --2- 172.21.15.6:0/3422671087 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb6fc0ccd50 0x7fb6fc142a10 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fb6f400b9a0 tx=0x7fb6f400be60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:20:19.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:19.962+0000 7fb6f3400640 1 -- 172.21.15.6:0/3422671087 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6f4019070 con 0x7fb6fc0ccd50 2024-04-24T23:20:19.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:19.962+0000 7fb6f3400640 1 -- 172.21.15.6:0/3422671087 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb6f400cba0 con 0x7fb6fc0ccd50 2024-04-24T23:20:19.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:19.962+0000 7fb6f3400640 1 -- 172.21.15.6:0/3422671087 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6f4014440 con 0x7fb6fc0ccd50 2024-04-24T23:20:19.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:19.964+0000 7fb700a00640 1 -- 172.21.15.6:0/3422671087 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6fc1f06f0 con 0x7fb6fc0ccd50 2024-04-24T23:20:19.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:19.964+0000 7fb700a00640 1 -- 172.21.15.6:0/3422671087 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6fc14f960 con 0x7fb6fc0ccd50 2024-04-24T23:20:19.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:19.969+0000 7fb700a00640 1 -- 172.21.15.6:0/3422671087 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb6fc0729d0 con 0x7fb6fc0ccd50 2024-04-24T23:20:19.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:19.969+0000 7fb6f3400640 1 -- 172.21.15.6:0/3422671087 <== mon.2 v2:172.21.15.190:3301/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fb6f40145e0 con 0x7fb6fc0ccd50 2024-04-24T23:20:19.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:19.971+0000 7fb6f3400640 1 --2- 172.21.15.6:0/3422671087 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fb6dc06be60 0x7fb6dc06e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:20:19.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:19.972+0000 7fb6f3400640 1 -- 172.21.15.6:0/3422671087 <== mon.2 v2:172.21.15.190:3301/0 5 ==== osd_map(28..28 src has 1..28) ==== 6762+0+0 (secure 0 0 0) 0x7fb6f408ae60 con 0x7fb6fc0ccd50 2024-04-24T23:20:19.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:19.972+0000 7fb6fb400640 1 --2- 172.21.15.6:0/3422671087 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fb6dc06be60 0x7fb6dc06e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:20:19.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:19.973+0000 7fb6fb400640 1 --2- 172.21.15.6:0/3422671087 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fb6dc06be60 0x7fb6dc06e320 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fb6ec045000 tx=0x7fb6ec03f000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:20:19.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:19.979+0000 7fb6f3400640 1 -- 172.21.15.6:0/3422671087 <== mon.2 v2:172.21.15.190:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fb6f4059480 con 0x7fb6fc0ccd50 2024-04-24T23:20:20.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:20.496+0000 7fb700a00640 1 -- 172.21.15.6:0/3422671087 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb6fc075b30 con 0x7fb6fc0ccd50 2024-04-24T23:20:20.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:20.497+0000 7fb6f3400640 1 -- 172.21.15.6:0/3422671087 <== mon.2 v2:172.21.15.190:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6080 (secure 0 0 0) 0x7fb6f4058d80 con 0x7fb6fc0ccd50 2024-04-24T23:20:20.499 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:20:20.500 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:20:17.585709+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4282,"mds_2":4235,"mds_3":4232,"mds_4":5257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4232":{"gid":4232,"name":"b","rank":3,"incarnation":15,"state":"up:active","state_seq":15,"addr":"172.21.15.190:6838/2032280158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2032280158},{"type":"v1","addr":"172.21.15.190:6838","nonce":2032280158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4235":{"gid":4235,"name":"d","rank":2,"incarnation":13,"state":"up:active","state_seq":14,"addr":"172.21.15.190:6839/1572872451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":1572872451},{"type":"v1","addr":"172.21.15.190:6839","nonce":1572872451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4282":{"gid":4282,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":12,"addr":"172.21.15.190:6837/580315269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":580315269},{"type":"v1","addr":"172.21.15.190:6837","nonce":580315269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/489092393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":489092393},{"type":"v1","addr":"172.21.15.6:6835","nonce":489092393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5257":{"gid":5257,"name":"e","rank":4,"incarnation":25,"state":"up:replay","state_seq":1,"addr":"172.21.15.6:6837/3240971655","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":3240971655},{"type":"v1","addr":"172.21.15.6:6837","nonce":3240971655}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4306,"qdb_cluster":[4232,4235,4282,4306]},"id":1}]} 2024-04-24T23:20:20.500 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 25 2024-04-24T23:20:20.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:20.500+0000 7fb700a00640 1 -- 172.21.15.6:0/3422671087 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fb6dc06be60 msgr2=0x7fb6dc06e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:20:20.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:20.500+0000 7fb700a00640 1 --2- 172.21.15.6:0/3422671087 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fb6dc06be60 0x7fb6dc06e320 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fb6ec045000 tx=0x7fb6ec03f000 comp rx=0 tx=0).stop 2024-04-24T23:20:20.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:20.500+0000 7fb700a00640 1 -- 172.21.15.6:0/3422671087 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb6fc0ccd50 msgr2=0x7fb6fc142a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:20:20.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:20.500+0000 7fb700a00640 1 --2- 172.21.15.6:0/3422671087 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb6fc0ccd50 0x7fb6fc142a10 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fb6f400b9a0 tx=0x7fb6f400be60 comp rx=0 tx=0).stop 2024-04-24T23:20:20.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:20.501+0000 7fb700a00640 1 -- 172.21.15.6:0/3422671087 shutdown_connections 2024-04-24T23:20:20.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:20.501+0000 7fb700a00640 1 --2- 172.21.15.6:0/3422671087 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fb6dc06be60 0x7fb6dc06e320 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:20.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:20.501+0000 7fb700a00640 1 --2- 172.21.15.6:0/3422671087 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb6fc0ccd50 0x7fb6fc142a10 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:20.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:20.501+0000 7fb700a00640 1 --2- 172.21.15.6:0/3422671087 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb6fc0cc930 0x7fb6fc14b950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:20.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:20.501+0000 7fb700a00640 1 --2- 172.21.15.6:0/3422671087 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb6fc06dd40 0x7fb6fc1f01b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:20.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:20.501+0000 7fb700a00640 1 -- 172.21.15.6:0/3422671087 >> 172.21.15.6:0/3422671087 conn(0x7fb6fc06ed80 msgr2=0x7fb6fc14c7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:20:20.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:20.501+0000 7fb700a00640 1 -- 172.21.15.6:0/3422671087 shutdown_connections 2024-04-24T23:20:20.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:20.501+0000 7fb700a00640 1 -- 172.21.15.6:0/3422671087 wait complete. 2024-04-24T23:20:20.535 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a down, removed from mdsmap 2024-04-24T23:20:20.536 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2024-04-24T23:20:20.536 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:20:20.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:20.884+0000 7f189e200640 1 Processor -- start 2024-04-24T23:20:20.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:20.884+0000 7f189e200640 1 -- start start 2024-04-24T23:20:20.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:20.887+0000 7f189e200640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f18980c8c40 0x7f18980c9070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:20:20.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:20.887+0000 7f1897e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f18980cc9f0 0x7f189806da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:50212/0 (socket says 172.21.15.6:50212) 2024-04-24T23:20:20.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:20.888+0000 7f189e200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1898071910 con 0x7f18980cc9f0 2024-04-24T23:20:20.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:20.888+0000 7f189e200640 1 -- 172.21.15.6:0/3307686542 --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f189806cf60 con 0x7f1898154a60 2024-04-24T23:20:20.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:20.888+0000 7f189e200640 1 -- 172.21.15.6:0/3307686542 --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f189806d570 con 0x7f18980c8c40 2024-04-24T23:20:20.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:20.888+0000 7f1897e00640 1 -- 172.21.15.6:0/3307686542 learned_addr learned my addr 172.21.15.6:0/3307686542 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:20:20.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:20.890+0000 7f1897400640 1 -- 172.21.15.6:0/3307686542 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3973023260 0 0) 0x7f189806cf60 con 0x7f1898154a60 2024-04-24T23:20:20.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:20.891+0000 7f1897400640 1 -- 172.21.15.6:0/3307686542 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f187c003520 con 0x7f1898154a60 2024-04-24T23:20:20.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:20.892+0000 7f189d800640 1 --2- 172.21.15.6:0/3307686542 >> v2:172.21.15.190:3300/0 conn(0x7f18980c8c40 0x7f18980c9070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:20:20.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:20.893+0000 7f1897400640 1 -- 172.21.15.6:0/3307686542 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4149940524 0 0) 0x7f1898071910 con 0x7f18980cc9f0 2024-04-24T23:20:20.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:20.893+0000 7f1897400640 1 -- 172.21.15.6:0/3307686542 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f189806cf60 con 0x7f18980cc9f0 2024-04-24T23:20:20.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:20.893+0000 7f1897400640 1 -- 172.21.15.6:0/3307686542 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 701141575 0 0) 0x7f187c003520 con 0x7f1898154a60 2024-04-24T23:20:20.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:20.893+0000 7f189d800640 1 -- 172.21.15.6:0/3307686542 >> v1:172.21.15.190:6789/0 conn(0x7f1898154a60 legacy=0x7f189815b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:20:20.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:20.893+0000 7f189d800640 1 -- 172.21.15.6:0/3307686542 >> v1:172.21.15.6:6789/0 conn(0x7f18980cc9f0 legacy=0x7f189806da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:20:20.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:20.893+0000 7f189d800640 1 -- 172.21.15.6:0/3307686542 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1898071350 con 0x7f18980c8c40 2024-04-24T23:20:20.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:20.898+0000 7f189d800640 1 --2- 172.21.15.6:0/3307686542 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f18980c8c40 0x7f18980c9070 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f189806d6f0 tx=0x7f189002a590 comp rx=0 tx=0).ready entity=mon.1 client_cookie=445fc9d147f79625 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:20:20.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:20.900+0000 7f1897400640 1 -- 172.21.15.6:0/3307686542 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1888001670 con 0x7f18980c8c40 2024-04-24T23:20:20.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:20.900+0000 7f1897400640 1 -- 172.21.15.6:0/3307686542 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1890007540 con 0x7f18980c8c40 2024-04-24T23:20:20.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:20.900+0000 7f1897400640 1 -- 172.21.15.6:0/3307686542 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1890007910 con 0x7f18980c8c40 2024-04-24T23:20:20.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:20.901+0000 7f189e200640 1 -- 172.21.15.6:0/3307686542 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f18980c8c40 msgr2=0x7f18980c9070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:20:20.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:20.901+0000 7f189e200640 1 --2- 172.21.15.6:0/3307686542 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f18980c8c40 0x7f18980c9070 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f189806d6f0 tx=0x7f189002a590 comp rx=0 tx=0).stop 2024-04-24T23:20:20.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:20.901+0000 7f189e200640 1 -- 172.21.15.6:0/3307686542 shutdown_connections 2024-04-24T23:20:20.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:20.901+0000 7f189e200640 1 --2- 172.21.15.6:0/3307686542 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f18980c8c40 0x7f18980c9070 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:20.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:20.901+0000 7f189e200640 1 -- 172.21.15.6:0/3307686542 >> 172.21.15.6:0/3307686542 conn(0x7f189806ed80 msgr2=0x7f189806f190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:20:20.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:20.903+0000 7f189e200640 1 -- 172.21.15.6:0/3307686542 shutdown_connections 2024-04-24T23:20:20.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:20.904+0000 7f189e200640 1 -- 172.21.15.6:0/3307686542 wait complete. 2024-04-24T23:20:20.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:20.908+0000 7f189e200640 1 Processor -- start 2024-04-24T23:20:20.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:20.908+0000 7f189e200640 1 -- start start 2024-04-24T23:20:20.919 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:20.911+0000 7f189e200640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f18980cc9f0 0x7f189814fee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:20:20.919 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:20.911+0000 7f189e200640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1898154a60 0x7f1898150420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:20:20.919 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:20.911+0000 7f189e200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f18980c3f80 0x7f18980c5a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:20:20.919 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:20.911+0000 7f189e200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f18980726a0 con 0x7f18980c3f80 2024-04-24T23:20:20.919 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:20.911+0000 7f189e200640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f189806d200 con 0x7f1898154a60 2024-04-24T23:20:20.919 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:20.911+0000 7f189e200640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f1898072820 con 0x7f18980cc9f0 2024-04-24T23:20:20.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:20.922+0000 7f189d800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f18980c3f80 0x7f18980c5a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:20:20.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:20.922+0000 7f1897e00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1898154a60 0x7f1898150420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:20:20.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:20.922+0000 7f189d800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f18980c3f80 0x7f18980c5a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36904/0 (socket says 172.21.15.6:36904) 2024-04-24T23:20:20.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:20.922+0000 7f189d800640 1 -- 172.21.15.6:0/45675558 learned_addr learned my addr 172.21.15.6:0/45675558 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:20:20.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:20.922+0000 7f189ce00640 1 --2- 172.21.15.6:0/45675558 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f18980cc9f0 0x7f189814fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:20:20.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:20.924+0000 7f189d800640 1 -- 172.21.15.6:0/45675558 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f18980cc9f0 msgr2=0x7f189814fee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:20:20.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:20.929+0000 7f189d800640 1 --2- 172.21.15.6:0/45675558 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f18980cc9f0 0x7f189814fee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:20.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:20.929+0000 7f189d800640 1 -- 172.21.15.6:0/45675558 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1898154a60 msgr2=0x7f1898150420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:20:20.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:20.929+0000 7f189d800640 1 --2- 172.21.15.6:0/45675558 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1898154a60 0x7f1898150420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:20.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:20.929+0000 7f189d800640 1 -- 172.21.15.6:0/45675558 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f189814f860 con 0x7f18980c3f80 2024-04-24T23:20:20.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:20.929+0000 7f1897e00640 1 --2- 172.21.15.6:0/45675558 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1898154a60 0x7f1898150420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:20:20.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:20.929+0000 7f189ce00640 1 --2- 172.21.15.6:0/45675558 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f18980cc9f0 0x7f189814fee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:20:20.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:20.930+0000 7f189d800640 1 --2- 172.21.15.6:0/45675558 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f18980c3f80 0x7f18980c5a60 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f1890009e80 tx=0x7f1890004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:20:20.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:20.933+0000 7f1894c00640 1 -- 172.21.15.6:0/45675558 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1890040070 con 0x7f18980c3f80 2024-04-24T23:20:20.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:20.933+0000 7f1894c00640 1 -- 172.21.15.6:0/45675558 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1890031b40 con 0x7f18980c3f80 2024-04-24T23:20:20.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:20.933+0000 7f1894c00640 1 -- 172.21.15.6:0/45675558 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1890040070 con 0x7f18980c3f80 2024-04-24T23:20:20.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:20.936+0000 7f189e200640 1 -- 172.21.15.6:0/45675558 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18980c53b0 con 0x7f18980c3f80 2024-04-24T23:20:20.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:20.939+0000 7f189e200640 1 -- 172.21.15.6:0/45675558 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18981f4c60 con 0x7f18980c3f80 2024-04-24T23:20:20.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:20.951+0000 7f1894c00640 1 -- 172.21.15.6:0/45675558 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f1890039040 con 0x7f18980c3f80 2024-04-24T23:20:20.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:20.952+0000 7f189e200640 1 -- 172.21.15.6:0/45675558 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1898075b30 con 0x7f18980c3f80 2024-04-24T23:20:20.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:20.953+0000 7f1894c00640 1 --2- 172.21.15.6:0/45675558 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f187806c030 0x7f187806e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:20:20.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:20.955+0000 7f189ce00640 1 --2- 172.21.15.6:0/45675558 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f187806c030 0x7f187806e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:20:20.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:20.960+0000 7f1894c00640 1 -- 172.21.15.6:0/45675558 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6762+0+0 (secure 0 0 0) 0x7f1890078530 con 0x7f18980c3f80 2024-04-24T23:20:20.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:20.960+0000 7f189ce00640 1 --2- 172.21.15.6:0/45675558 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f187806c030 0x7f187806e4f0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f18880010e0 tx=0x7f1888009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:20:20.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:20.971+0000 7f1894c00640 1 -- 172.21.15.6:0/45675558 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f189007ab90 con 0x7f18980c3f80 2024-04-24T23:20:21.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:21.602+0000 7f189e200640 1 -- 172.21.15.6:0/45675558 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f189814fc40 con 0x7f18980c3f80 2024-04-24T23:20:21.605 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:20:21.606 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:20:17.585709+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4282,"mds_2":4235,"mds_3":4232,"mds_4":5257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4232":{"gid":4232,"name":"b","rank":3,"incarnation":15,"state":"up:active","state_seq":15,"addr":"172.21.15.190:6838/2032280158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2032280158},{"type":"v1","addr":"172.21.15.190:6838","nonce":2032280158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4235":{"gid":4235,"name":"d","rank":2,"incarnation":13,"state":"up:active","state_seq":14,"addr":"172.21.15.190:6839/1572872451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":1572872451},{"type":"v1","addr":"172.21.15.190:6839","nonce":1572872451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4282":{"gid":4282,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":12,"addr":"172.21.15.190:6837/580315269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":580315269},{"type":"v1","addr":"172.21.15.190:6837","nonce":580315269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/489092393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":489092393},{"type":"v1","addr":"172.21.15.6:6835","nonce":489092393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5257":{"gid":5257,"name":"e","rank":4,"incarnation":25,"state":"up:replay","state_seq":1,"addr":"172.21.15.6:6837/3240971655","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":3240971655},{"type":"v1","addr":"172.21.15.6:6837","nonce":3240971655}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4306,"qdb_cluster":[4232,4235,4282,4306]},"id":1}]} 2024-04-24T23:20:21.606 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 25 2024-04-24T23:20:21.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:21.603+0000 7f1894c00640 1 -- 172.21.15.6:0/45675558 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6080 (secure 0 0 0) 0x7f189007a490 con 0x7f18980c3f80 2024-04-24T23:20:21.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:21.605+0000 7f189e200640 1 -- 172.21.15.6:0/45675558 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f187806c030 msgr2=0x7f187806e4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:20:21.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:21.605+0000 7f189e200640 1 --2- 172.21.15.6:0/45675558 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f187806c030 0x7f187806e4f0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f18880010e0 tx=0x7f1888009040 comp rx=0 tx=0).stop 2024-04-24T23:20:21.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:21.605+0000 7f189e200640 1 -- 172.21.15.6:0/45675558 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f18980c3f80 msgr2=0x7f18980c5a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:20:21.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:21.605+0000 7f189e200640 1 --2- 172.21.15.6:0/45675558 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f18980c3f80 0x7f18980c5a60 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f1890009e80 tx=0x7f1890004a40 comp rx=0 tx=0).stop 2024-04-24T23:20:21.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:21.605+0000 7f189e200640 1 -- 172.21.15.6:0/45675558 shutdown_connections 2024-04-24T23:20:21.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:21.605+0000 7f189e200640 1 --2- 172.21.15.6:0/45675558 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f187806c030 0x7f187806e4f0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:21.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:21.605+0000 7f189e200640 1 --2- 172.21.15.6:0/45675558 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f18980c3f80 0x7f18980c5a60 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:21.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:21.605+0000 7f189e200640 1 --2- 172.21.15.6:0/45675558 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1898154a60 0x7f1898150420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:21.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:21.605+0000 7f189e200640 1 --2- 172.21.15.6:0/45675558 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f18980cc9f0 0x7f189814fee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:21.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:21.605+0000 7f189e200640 1 -- 172.21.15.6:0/45675558 >> 172.21.15.6:0/45675558 conn(0x7f189806ed80 msgr2=0x7f1898149710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:20:21.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:21.605+0000 7f189e200640 1 -- 172.21.15.6:0/45675558 shutdown_connections 2024-04-24T23:20:21.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:21.606+0000 7f189e200640 1 -- 172.21.15.6:0/45675558 wait complete. 2024-04-24T23:20:21.873 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 25, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-24T23:15:36.400716+0000", "damaged": [], "data_pools": [ 3, 4 ], "enabled": true, "epoch": 25, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4232": { "addr": "172.21.15.190:6838/2032280158", "addrs": { "addrvec": [ { "addr": "172.21.15.190:6835", "nonce": 2032280158, "type": "v2" }, { "addr": "172.21.15.190:6838", "nonce": 2032280158, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4232, "incarnation": 15, "join_fscid": -1, "name": "b", "rank": 3, "state": "up:active", "state_seq": 15 }, "gid_4235": { "addr": "172.21.15.190:6839/1572872451", "addrs": { "addrvec": [ { "addr": "172.21.15.190:6836", "nonce": 1572872451, "type": "v2" }, { "addr": "172.21.15.190:6839", "nonce": 1572872451, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4235, "incarnation": 13, "join_fscid": -1, "name": "d", "rank": 2, "state": "up:active", "state_seq": 14 }, "gid_4282": { "addr": "172.21.15.190:6837/580315269", "addrs": { "addrvec": [ { "addr": "172.21.15.190:6834", "nonce": 580315269, "type": "v2" }, { "addr": "172.21.15.190:6837", "nonce": 580315269, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4282, "incarnation": 9, "join_fscid": -1, "name": "f", "rank": 1, "state": "up:active", "state_seq": 12 }, "gid_4306": { "addr": "172.21.15.6:6835/489092393", "addrs": { "addrvec": [ { "addr": "172.21.15.6:6833", "nonce": 489092393, "type": "v2" }, { "addr": "172.21.15.6:6835", "nonce": 489092393, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4306, "incarnation": 4, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 4 }, "gid_5257": { "addr": "172.21.15.6:6837/3240971655", "addrs": { "addrvec": [ { "addr": "172.21.15.6:6836", "nonce": 3240971655, "type": "v2" }, { "addr": "172.21.15.6:6837", "nonce": 3240971655, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5257, "incarnation": 25, "join_fscid": -1, "name": "e", "rank": 4, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 28, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-04-24T23:20:17.585709+0000", "qdb_cluster": [ 4232, 4235, 4282, 4306 ], "qdb_leader": 4306, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4306, "mds_1": 4282, "mds_2": 4235, "mds_3": 4232, "mds_4": 5257 } } } ], "standbys": [] } 2024-04-24T23:20:23.874 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:20:24.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:24.270+0000 7f1cf6200640 1 Processor -- start 2024-04-24T23:20:24.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:24.270+0000 7f1cf6200640 1 -- start start 2024-04-24T23:20:24.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:24.271+0000 7f1cf6200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f1cf0165150 0x7f1cf016b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:20:24.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:24.272+0000 7f1cf6200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1cf0072370 con 0x7f1cf00cc600 2024-04-24T23:20:24.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:24.272+0000 7f1cf6200640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1cf0072650 con 0x7f1cf00c3f80 2024-04-24T23:20:24.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:24.272+0000 7f1cf6200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f1cf016bb30 con 0x7f1cf0165150 2024-04-24T23:20:24.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:24.275+0000 7f1cf5800640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f1cf00cc600 0x7f1cf0167ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:50228/0 (socket says 172.21.15.6:50228) 2024-04-24T23:20:24.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:24.275+0000 7f1cf5800640 1 -- 172.21.15.6:0/3124355552 learned_addr learned my addr 172.21.15.6:0/3124355552 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:20:24.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:24.275+0000 7f1cf4e00640 1 --2- 172.21.15.6:0/3124355552 >> v2:172.21.15.6:3300/0 conn(0x7f1cf0165150 0x7f1cf016b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:20:24.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:24.278+0000 7f1cef400640 1 -- 172.21.15.6:0/3124355552 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2417245217 0 0) 0x7f1cf0072370 con 0x7f1cf00cc600 2024-04-24T23:20:24.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:24.278+0000 7f1cf4e00640 1 -- 172.21.15.6:0/3124355552 >> v1:172.21.15.190:6789/0 conn(0x7f1cf00c3f80 legacy=0x7f1cf006ddd0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-24T23:20:24.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:24.278+0000 7f1cf4e00640 1 -- 172.21.15.6:0/3124355552 >> v1:172.21.15.6:6789/0 conn(0x7f1cf00cc600 legacy=0x7f1cf0167ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:20:24.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:24.279+0000 7f1cf4e00640 1 -- 172.21.15.6:0/3124355552 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1cf00710c0 con 0x7f1cf0165150 2024-04-24T23:20:24.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:24.280+0000 7f1cf4e00640 1 --2- 172.21.15.6:0/3124355552 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1cf0165150 0x7f1cf016b5f0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f1ce40097a0 tx=0x7f1ce402a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d1e4446a7b0d384 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:20:24.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:24.286+0000 7f1cef400640 1 -- 172.21.15.6:0/3124355552 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ce4037070 con 0x7f1cf0165150 2024-04-24T23:20:24.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:24.286+0000 7f1cef400640 1 -- 172.21.15.6:0/3124355552 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ce4032440 con 0x7f1cf0165150 2024-04-24T23:20:24.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:24.286+0000 7f1cef400640 1 -- 172.21.15.6:0/3124355552 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ce4032810 con 0x7f1cf0165150 2024-04-24T23:20:24.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:24.287+0000 7f1cf6200640 1 -- 172.21.15.6:0/3124355552 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1cf0165150 msgr2=0x7f1cf016b5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:20:24.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:24.288+0000 7f1cf6200640 1 --2- 172.21.15.6:0/3124355552 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1cf0165150 0x7f1cf016b5f0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f1ce40097a0 tx=0x7f1ce402a1e0 comp rx=0 tx=0).stop 2024-04-24T23:20:24.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:24.295+0000 7f1cf6200640 1 -- 172.21.15.6:0/3124355552 shutdown_connections 2024-04-24T23:20:24.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:24.295+0000 7f1cf6200640 1 --2- 172.21.15.6:0/3124355552 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1cf0165150 0x7f1cf016b5f0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:24.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:24.295+0000 7f1cf6200640 1 -- 172.21.15.6:0/3124355552 >> 172.21.15.6:0/3124355552 conn(0x7f1cf006ed80 msgr2=0x7f1cf006f190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:20:24.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:24.296+0000 7f1cf6200640 1 -- 172.21.15.6:0/3124355552 shutdown_connections 2024-04-24T23:20:24.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:24.296+0000 7f1cf6200640 1 -- 172.21.15.6:0/3124355552 wait complete. 2024-04-24T23:20:24.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:24.306+0000 7f1cf6200640 1 Processor -- start 2024-04-24T23:20:24.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:24.306+0000 7f1cf6200640 1 -- start start 2024-04-24T23:20:24.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:24.315+0000 7f1cf6200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1cf00c3f80 0x7f1cf00cb370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:20:24.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:24.316+0000 7f1cf6200640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1cf00cc600 0x7f1cf016b130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:20:24.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:24.316+0000 7f1cf6200640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1cf0165150 0x7f1cf0168ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:20:24.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:24.321+0000 7f1cf6200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f1cf00c3ba0 con 0x7f1cf00c3f80 2024-04-24T23:20:24.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:24.321+0000 7f1cf6200640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f1cf0070e70 con 0x7f1cf00cc600 2024-04-24T23:20:24.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:24.321+0000 7f1cf6200640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f1cf01681d0 con 0x7f1cf0165150 2024-04-24T23:20:24.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:24.317+0000 7f1cf5800640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1cf0165150 0x7f1cf0168ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:20:24.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:24.321+0000 7f1cf5800640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1cf0165150 0x7f1cf0168ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.6:56934/0 (socket says 172.21.15.6:56934) 2024-04-24T23:20:24.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:24.321+0000 7f1cf4e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1cf00c3f80 0x7f1cf00cb370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:20:24.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:24.321+0000 7f1cf5800640 1 -- 172.21.15.6:0/2548464070 learned_addr learned my addr 172.21.15.6:0/2548464070 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:20:24.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:24.322+0000 7f1cefe00640 1 --2- 172.21.15.6:0/2548464070 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1cf00cc600 0x7f1cf016b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:20:24.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:24.323+0000 7f1cefe00640 1 -- 172.21.15.6:0/2548464070 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1cf0165150 msgr2=0x7f1cf0168ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:20:24.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:24.323+0000 7f1cefe00640 1 --2- 172.21.15.6:0/2548464070 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1cf0165150 0x7f1cf0168ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:24.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:24.323+0000 7f1cefe00640 1 -- 172.21.15.6:0/2548464070 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1cf00c3f80 msgr2=0x7f1cf00cb370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:20:24.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:24.323+0000 7f1cefe00640 1 --2- 172.21.15.6:0/2548464070 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1cf00c3f80 0x7f1cf00cb370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:24.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:24.324+0000 7f1cefe00640 1 -- 172.21.15.6:0/2548464070 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1cf008bf80 con 0x7f1cf00cc600 2024-04-24T23:20:24.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:24.324+0000 7f1cf4e00640 1 --2- 172.21.15.6:0/2548464070 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1cf00c3f80 0x7f1cf00cb370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:20:24.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:24.328+0000 7f1cefe00640 1 --2- 172.21.15.6:0/2548464070 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1cf00cc600 0x7f1cf016b130 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f1cdc009fd0 tx=0x7f1cdc00b7b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:20:24.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:24.328+0000 7f1cecc00640 1 -- 172.21.15.6:0/2548464070 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cdc015020 con 0x7f1cf00cc600 2024-04-24T23:20:24.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:24.328+0000 7f1cecc00640 1 -- 172.21.15.6:0/2548464070 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1cdc011830 con 0x7f1cf00cc600 2024-04-24T23:20:24.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:24.328+0000 7f1cecc00640 1 -- 172.21.15.6:0/2548464070 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cdc011c20 con 0x7f1cf00cc600 2024-04-24T23:20:24.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:24.329+0000 7f1cf6200640 1 -- 172.21.15.6:0/2548464070 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1cf008b120 con 0x7f1cf00cc600 2024-04-24T23:20:24.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:24.330+0000 7f1cf6200640 1 -- 172.21.15.6:0/2548464070 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1cf008b6e0 con 0x7f1cf00cc600 2024-04-24T23:20:24.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:24.332+0000 7f1ce2000640 1 -- 172.21.15.6:0/2548464070 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1cf0075b30 con 0x7f1cf00cc600 2024-04-24T23:20:24.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:24.334+0000 7f1cecc00640 1 -- 172.21.15.6:0/2548464070 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f1cdc018020 con 0x7f1cf00cc600 2024-04-24T23:20:24.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:24.335+0000 7f1cecc00640 1 --2- 172.21.15.6:0/2548464070 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f1cd006be30 0x7f1cd006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:20:24.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:24.335+0000 7f1cecc00640 1 -- 172.21.15.6:0/2548464070 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6762+0+0 (secure 0 0 0) 0x7f1cdc08b0c0 con 0x7f1cf00cc600 2024-04-24T23:20:24.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:24.336+0000 7f1cf4e00640 1 --2- 172.21.15.6:0/2548464070 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f1cd006be30 0x7f1cd006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:20:24.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:24.338+0000 7f1cf4e00640 1 --2- 172.21.15.6:0/2548464070 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f1cd006be30 0x7f1cd006e2f0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f1ce4034000 tx=0x7f1ce402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:20:24.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:24.359+0000 7f1cecc00640 1 -- 172.21.15.6:0/2548464070 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f1cdc01f020 con 0x7f1cf00cc600 2024-04-24T23:20:24.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:24.913+0000 7f1ce2000640 1 -- 172.21.15.6:0/2548464070 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1cf006d890 con 0x7f1cf00cc600 2024-04-24T23:20:24.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:24.914+0000 7f1cecc00640 1 -- 172.21.15.6:0/2548464070 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6080 (secure 0 0 0) 0x7f1cdc0595b0 con 0x7f1cf00cc600 2024-04-24T23:20:24.918 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:20:24.918 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:20:17.585709+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4282,"mds_2":4235,"mds_3":4232,"mds_4":5257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4232":{"gid":4232,"name":"b","rank":3,"incarnation":15,"state":"up:active","state_seq":15,"addr":"172.21.15.190:6838/2032280158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2032280158},{"type":"v1","addr":"172.21.15.190:6838","nonce":2032280158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4235":{"gid":4235,"name":"d","rank":2,"incarnation":13,"state":"up:active","state_seq":14,"addr":"172.21.15.190:6839/1572872451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":1572872451},{"type":"v1","addr":"172.21.15.190:6839","nonce":1572872451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4282":{"gid":4282,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":12,"addr":"172.21.15.190:6837/580315269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":580315269},{"type":"v1","addr":"172.21.15.190:6837","nonce":580315269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/489092393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":489092393},{"type":"v1","addr":"172.21.15.6:6835","nonce":489092393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5257":{"gid":5257,"name":"e","rank":4,"incarnation":25,"state":"up:replay","state_seq":1,"addr":"172.21.15.6:6837/3240971655","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":3240971655},{"type":"v1","addr":"172.21.15.6:6837","nonce":3240971655}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4306,"qdb_cluster":[4232,4235,4282,4306]},"id":1}]} 2024-04-24T23:20:24.918 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 25 2024-04-24T23:20:24.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:24.921+0000 7f1ce2000640 1 -- 172.21.15.6:0/2548464070 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f1cd006be30 msgr2=0x7f1cd006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:20:24.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:24.921+0000 7f1ce2000640 1 --2- 172.21.15.6:0/2548464070 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f1cd006be30 0x7f1cd006e2f0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f1ce4034000 tx=0x7f1ce402e000 comp rx=0 tx=0).stop 2024-04-24T23:20:24.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:24.921+0000 7f1ce2000640 1 -- 172.21.15.6:0/2548464070 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1cf00cc600 msgr2=0x7f1cf016b130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:20:24.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:24.921+0000 7f1ce2000640 1 --2- 172.21.15.6:0/2548464070 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1cf00cc600 0x7f1cf016b130 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f1cdc009fd0 tx=0x7f1cdc00b7b0 comp rx=0 tx=0).stop 2024-04-24T23:20:24.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:24.921+0000 7f1ce2000640 1 -- 172.21.15.6:0/2548464070 shutdown_connections 2024-04-24T23:20:24.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:24.921+0000 7f1ce2000640 1 --2- 172.21.15.6:0/2548464070 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f1cd006be30 0x7f1cd006e2f0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:24.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:24.921+0000 7f1ce2000640 1 --2- 172.21.15.6:0/2548464070 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1cf0165150 0x7f1cf0168ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:24.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:24.921+0000 7f1ce2000640 1 --2- 172.21.15.6:0/2548464070 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1cf00cc600 0x7f1cf016b130 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:24.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:24.921+0000 7f1ce2000640 1 --2- 172.21.15.6:0/2548464070 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1cf00c3f80 0x7f1cf00cb370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:24.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:24.921+0000 7f1ce2000640 1 -- 172.21.15.6:0/2548464070 >> 172.21.15.6:0/2548464070 conn(0x7f1cf006ed80 msgr2=0x7f1cf0074630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:20:24.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:24.922+0000 7f1ce2000640 1 -- 172.21.15.6:0/2548464070 shutdown_connections 2024-04-24T23:20:24.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:24.922+0000 7f1ce2000640 1 -- 172.21.15.6:0/2548464070 wait complete. 2024-04-24T23:20:24.958 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-24T23:20:26.959 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:20:27.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:27.228+0000 7f38fcc00640 1 Processor -- start 2024-04-24T23:20:27.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:27.229+0000 7f38fcc00640 1 -- start start 2024-04-24T23:20:27.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:27.231+0000 7f38fcc00640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f38f8150cd0 0x7f38f80cc930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:20:27.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:27.231+0000 7f38fcc00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f38f80cce70 0x7f38f8159e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:20:27.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:27.231+0000 7f38fcc00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f38f806e580 con 0x7f38f806dd40 2024-04-24T23:20:27.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:27.231+0000 7f38fcc00640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f38f80c7720 con 0x7f38f8150cd0 2024-04-24T23:20:27.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:27.231+0000 7f38fcc00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f38f806e2a0 con 0x7f38f80cce70 2024-04-24T23:20:27.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:27.235+0000 7f38f7400640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f38f8150cd0 0x7f38f80cc930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:20:27.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:27.235+0000 7f38f6a00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f38f80cce70 0x7f38f8159e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:20:27.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:27.235+0000 7f38f6a00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f38f80cce70 0x7f38f8159e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.6:51258/0 (socket says 172.21.15.6:51258) 2024-04-24T23:20:27.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:27.235+0000 7f38f6a00640 1 -- 172.21.15.6:0/3328053914 learned_addr learned my addr 172.21.15.6:0/3328053914 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:20:27.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:27.235+0000 7f38f6a00640 1 -- 172.21.15.6:0/3328053914 >> v2:172.21.15.190:3300/0 conn(0x7f38f8150cd0 msgr2=0x7f38f80cc930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:20:27.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:27.235+0000 7f38f6a00640 1 --2- 172.21.15.6:0/3328053914 >> v2:172.21.15.190:3300/0 conn(0x7f38f8150cd0 0x7f38f80cc930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:27.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:27.235+0000 7f38f6a00640 1 -- 172.21.15.6:0/3328053914 >> v1:172.21.15.6:6789/0 conn(0x7f38f806dd40 legacy=0x7f38f80c8ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:20:27.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:27.236+0000 7f38f6a00640 1 -- 172.21.15.6:0/3328053914 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38f80c79c0 con 0x7f38f80cce70 2024-04-24T23:20:27.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:27.237+0000 7f38f6a00640 1 --2- 172.21.15.6:0/3328053914 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f38f80cce70 0x7f38f8159e70 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f38e8006dd0 tx=0x7f38e802e5b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e4165683dcbe089b server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:20:27.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:27.239+0000 7f38f6000640 1 -- 172.21.15.6:0/3328053914 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38e803a070 con 0x7f38f80cce70 2024-04-24T23:20:27.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:27.239+0000 7f38f6000640 1 -- 172.21.15.6:0/3328053914 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f38e8033710 con 0x7f38f80cce70 2024-04-24T23:20:27.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:27.239+0000 7f38f6000640 1 -- 172.21.15.6:0/3328053914 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38e8033ae0 con 0x7f38f80cce70 2024-04-24T23:20:27.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:27.240+0000 7f38fcc00640 1 -- 172.21.15.6:0/3328053914 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f38f80cce70 msgr2=0x7f38f8159e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:20:27.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:27.240+0000 7f38fcc00640 1 --2- 172.21.15.6:0/3328053914 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f38f80cce70 0x7f38f8159e70 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f38e8006dd0 tx=0x7f38e802e5b0 comp rx=0 tx=0).stop 2024-04-24T23:20:27.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:27.241+0000 7f38fcc00640 1 -- 172.21.15.6:0/3328053914 shutdown_connections 2024-04-24T23:20:27.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:27.241+0000 7f38fcc00640 1 --2- 172.21.15.6:0/3328053914 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f38f80cce70 0x7f38f8159e70 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:27.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:27.241+0000 7f38fcc00640 1 --2- 172.21.15.6:0/3328053914 >> v2:172.21.15.190:3300/0 conn(0x7f38f8150cd0 0x7f38f80cc930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:27.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:27.241+0000 7f38fcc00640 1 -- 172.21.15.6:0/3328053914 >> 172.21.15.6:0/3328053914 conn(0x7f38f806ed80 msgr2=0x7f38f806f190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:20:27.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:27.242+0000 7f38fcc00640 1 -- 172.21.15.6:0/3328053914 shutdown_connections 2024-04-24T23:20:27.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:27.242+0000 7f38fcc00640 1 -- 172.21.15.6:0/3328053914 wait complete. 2024-04-24T23:20:27.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:27.245+0000 7f38fcc00640 1 Processor -- start 2024-04-24T23:20:27.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:27.245+0000 7f38fcc00640 1 -- start start 2024-04-24T23:20:27.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:27.246+0000 7f38fcc00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f38f806dd40 0x7f38f81ee5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:20:27.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:27.246+0000 7f38fcc00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f38f80cce70 0x7f38f81ef2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:20:27.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:27.246+0000 7f38fcc00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f38f8150cd0 0x7f38f81ef830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:20:27.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:27.246+0000 7f38fcc00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f38f80c7240 con 0x7f38f8150cd0 2024-04-24T23:20:27.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:27.247+0000 7f38fcc00640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f38f80c7f80 con 0x7f38f80cce70 2024-04-24T23:20:27.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:27.247+0000 7f38fcc00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f38f80c91f0 con 0x7f38f806dd40 2024-04-24T23:20:27.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:27.247+0000 7f38f6a00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f38f80cce70 0x7f38f81ef2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:20:27.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:27.247+0000 7f38f6a00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f38f80cce70 0x7f38f81ef2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:53986/0 (socket says 172.21.15.6:53986) 2024-04-24T23:20:27.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:27.247+0000 7f38f6a00640 1 -- 172.21.15.6:0/1366511651 learned_addr learned my addr 172.21.15.6:0/1366511651 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:20:27.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:27.247+0000 7f38f7400640 1 --2- 172.21.15.6:0/1366511651 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f38f806dd40 0x7f38f81ee5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:20:27.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:27.247+0000 7f38f7e00640 1 --2- 172.21.15.6:0/1366511651 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f38f8150cd0 0x7f38f81ef830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:20:27.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:27.248+0000 7f38f6a00640 1 -- 172.21.15.6:0/1366511651 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f38f806dd40 msgr2=0x7f38f81ee5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:20:27.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:27.248+0000 7f38f6a00640 1 --2- 172.21.15.6:0/1366511651 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f38f806dd40 0x7f38f81ee5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:27.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:27.248+0000 7f38f6a00640 1 -- 172.21.15.6:0/1366511651 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f38f8150cd0 msgr2=0x7f38f81ef830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:20:27.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:27.248+0000 7f38f6a00640 1 --2- 172.21.15.6:0/1366511651 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f38f8150cd0 0x7f38f81ef830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:27.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:27.248+0000 7f38f6a00640 1 -- 172.21.15.6:0/1366511651 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38f8155af0 con 0x7f38f80cce70 2024-04-24T23:20:27.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:27.249+0000 7f38f6a00640 1 --2- 172.21.15.6:0/1366511651 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f38f80cce70 0x7f38f81ef2f0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f38e8009990 tx=0x7f38e80046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:20:27.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:27.249+0000 7f38f7e00640 1 --2- 172.21.15.6:0/1366511651 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f38f8150cd0 0x7f38f81ef830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:20:27.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:27.249+0000 7f38ef400640 1 -- 172.21.15.6:0/1366511651 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38e803a030 con 0x7f38f80cce70 2024-04-24T23:20:27.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:27.249+0000 7f38ef400640 1 -- 172.21.15.6:0/1366511651 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f38e803c050 con 0x7f38f80cce70 2024-04-24T23:20:27.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:27.249+0000 7f38ef400640 1 -- 172.21.15.6:0/1366511651 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38e80122f0 con 0x7f38f80cce70 2024-04-24T23:20:27.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:27.250+0000 7f38fcc00640 1 -- 172.21.15.6:0/1366511651 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38f8155850 con 0x7f38f80cce70 2024-04-24T23:20:27.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:27.252+0000 7f38fcc00640 1 -- 172.21.15.6:0/1366511651 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38f81fde40 con 0x7f38f80cce70 2024-04-24T23:20:27.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:27.254+0000 7f38fcc00640 1 -- 172.21.15.6:0/1366511651 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f38f8075b30 con 0x7f38f80cce70 2024-04-24T23:20:27.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:27.256+0000 7f38ef400640 1 -- 172.21.15.6:0/1366511651 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f38e8004030 con 0x7f38f80cce70 2024-04-24T23:20:27.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:27.258+0000 7f38ef400640 1 --2- 172.21.15.6:0/1366511651 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f38d806be60 0x7f38d806e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:20:27.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:27.258+0000 7f38ef400640 1 -- 172.21.15.6:0/1366511651 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6762+0+0 (secure 0 0 0) 0x7f38e80abbd0 con 0x7f38f80cce70 2024-04-24T23:20:27.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:27.258+0000 7f38f7400640 1 --2- 172.21.15.6:0/1366511651 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f38d806be60 0x7f38d806e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:20:27.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:27.260+0000 7f38f7400640 1 --2- 172.21.15.6:0/1366511651 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f38d806be60 0x7f38d806e320 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f38e4006fd0 tx=0x7f38e4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:20:27.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:27.264+0000 7f38ef400640 1 -- 172.21.15.6:0/1366511651 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f38e80abe20 con 0x7f38f80cce70 2024-04-24T23:20:28.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:27.999+0000 7f38fcc00640 1 -- 172.21.15.6:0/1366511651 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f38f81542f0 con 0x7f38f80cce70 2024-04-24T23:20:28.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:28.008+0000 7f38ef400640 1 -- 172.21.15.6:0/1366511651 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6080 (secure 0 0 0) 0x7f38e807a2f0 con 0x7f38f80cce70 2024-04-24T23:20:28.014 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:20:28.014 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:20:17.585709+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4282,"mds_2":4235,"mds_3":4232,"mds_4":5257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4232":{"gid":4232,"name":"b","rank":3,"incarnation":15,"state":"up:active","state_seq":15,"addr":"172.21.15.190:6838/2032280158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2032280158},{"type":"v1","addr":"172.21.15.190:6838","nonce":2032280158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4235":{"gid":4235,"name":"d","rank":2,"incarnation":13,"state":"up:active","state_seq":14,"addr":"172.21.15.190:6839/1572872451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":1572872451},{"type":"v1","addr":"172.21.15.190:6839","nonce":1572872451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4282":{"gid":4282,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":12,"addr":"172.21.15.190:6837/580315269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":580315269},{"type":"v1","addr":"172.21.15.190:6837","nonce":580315269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/489092393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":489092393},{"type":"v1","addr":"172.21.15.6:6835","nonce":489092393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5257":{"gid":5257,"name":"e","rank":4,"incarnation":25,"state":"up:replay","state_seq":1,"addr":"172.21.15.6:6837/3240971655","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":3240971655},{"type":"v1","addr":"172.21.15.6:6837","nonce":3240971655}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4306,"qdb_cluster":[4232,4235,4282,4306]},"id":1}]} 2024-04-24T23:20:28.014 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 25 2024-04-24T23:20:28.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:28.016+0000 7f38fcc00640 1 -- 172.21.15.6:0/1366511651 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f38d806be60 msgr2=0x7f38d806e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:20:28.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:28.016+0000 7f38fcc00640 1 --2- 172.21.15.6:0/1366511651 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f38d806be60 0x7f38d806e320 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f38e4006fd0 tx=0x7f38e4008040 comp rx=0 tx=0).stop 2024-04-24T23:20:28.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:28.016+0000 7f38fcc00640 1 -- 172.21.15.6:0/1366511651 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f38f80cce70 msgr2=0x7f38f81ef2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:20:28.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:28.016+0000 7f38fcc00640 1 --2- 172.21.15.6:0/1366511651 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f38f80cce70 0x7f38f81ef2f0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f38e8009990 tx=0x7f38e80046c0 comp rx=0 tx=0).stop 2024-04-24T23:20:28.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:28.017+0000 7f38fcc00640 1 -- 172.21.15.6:0/1366511651 shutdown_connections 2024-04-24T23:20:28.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:28.017+0000 7f38fcc00640 1 --2- 172.21.15.6:0/1366511651 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f38d806be60 0x7f38d806e320 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:28.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:28.017+0000 7f38fcc00640 1 --2- 172.21.15.6:0/1366511651 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f38f8150cd0 0x7f38f81ef830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:28.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:28.017+0000 7f38fcc00640 1 --2- 172.21.15.6:0/1366511651 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f38f80cce70 0x7f38f81ef2f0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:28.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:28.017+0000 7f38fcc00640 1 --2- 172.21.15.6:0/1366511651 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f38f806dd40 0x7f38f81ee5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:28.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:28.017+0000 7f38fcc00640 1 -- 172.21.15.6:0/1366511651 >> 172.21.15.6:0/1366511651 conn(0x7f38f806ed80 msgr2=0x7f38f81550a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:20:28.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:28.017+0000 7f38fcc00640 1 -- 172.21.15.6:0/1366511651 shutdown_connections 2024-04-24T23:20:28.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:28.017+0000 7f38fcc00640 1 -- 172.21.15.6:0/1366511651 wait complete. 2024-04-24T23:20:28.073 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-24T23:20:30.074 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:20:30.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:30.413+0000 7f7f30200640 1 Processor -- start 2024-04-24T23:20:30.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:30.414+0000 7f7f30200640 1 -- start start 2024-04-24T23:20:30.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:30.416+0000 7f7f30200640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7f28152d80 0x7f7f280c8d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:20:30.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:30.416+0000 7f7f30200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f7f281592a0 0x7f7f2815c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:20:30.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:30.417+0000 7f7f30200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7f28159770 con 0x7f7f2815cb10 2024-04-24T23:20:30.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:30.417+0000 7f7f30200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f7f280c5500 con 0x7f7f281592a0 2024-04-24T23:20:30.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:30.417+0000 7f7f30200640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f7f2815c140 con 0x7f7f28152d80 2024-04-24T23:20:30.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:30.417+0000 7f7f2d800640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7f28152d80 0x7f7f280c8d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:20:30.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:30.417+0000 7f7f2d800640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7f28152d80 0x7f7f280c8d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.6:51272/0 (socket says 172.21.15.6:51272) 2024-04-24T23:20:30.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:30.417+0000 7f7f2d800640 1 -- 172.21.15.6:0/1267777343 learned_addr learned my addr 172.21.15.6:0/1267777343 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:20:30.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:30.418+0000 7f7f2d800640 1 -- 172.21.15.6:0/1267777343 >> v2:172.21.15.6:3300/0 conn(0x7f7f281592a0 msgr2=0x7f7f2815c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:20:30.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:30.418+0000 7f7f2d800640 1 --2- 172.21.15.6:0/1267777343 >> v2:172.21.15.6:3300/0 conn(0x7f7f281592a0 0x7f7f2815c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:30.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:30.418+0000 7f7f2d800640 1 -- 172.21.15.6:0/1267777343 >> v1:172.21.15.6:6789/0 conn(0x7f7f2815cb10 legacy=0x7f7f28070c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:20:30.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:30.418+0000 7f7f2d800640 1 -- 172.21.15.6:0/1267777343 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f280c3470 con 0x7f7f28152d80 2024-04-24T23:20:30.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:30.419+0000 7f7f2d800640 1 --2- 172.21.15.6:0/1267777343 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7f28152d80 0x7f7f280c8d20 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f7f20009e50 tx=0x7f7f2002a6d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9e5eca70c900c5a8 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:20:30.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:30.420+0000 7f7f27e00640 1 -- 172.21.15.6:0/1267777343 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f20036070 con 0x7f7f28152d80 2024-04-24T23:20:30.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:30.420+0000 7f7f27e00640 1 -- 172.21.15.6:0/1267777343 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f20031600 con 0x7f7f28152d80 2024-04-24T23:20:30.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:30.420+0000 7f7f27e00640 1 -- 172.21.15.6:0/1267777343 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f200319d0 con 0x7f7f28152d80 2024-04-24T23:20:30.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:30.421+0000 7f7f30200640 1 -- 172.21.15.6:0/1267777343 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7f28152d80 msgr2=0x7f7f280c8d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:20:30.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:30.421+0000 7f7f30200640 1 --2- 172.21.15.6:0/1267777343 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7f28152d80 0x7f7f280c8d20 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f7f20009e50 tx=0x7f7f2002a6d0 comp rx=0 tx=0).stop 2024-04-24T23:20:30.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:30.421+0000 7f7f30200640 1 -- 172.21.15.6:0/1267777343 shutdown_connections 2024-04-24T23:20:30.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:30.421+0000 7f7f30200640 1 --2- 172.21.15.6:0/1267777343 >> v2:172.21.15.6:3300/0 conn(0x7f7f281592a0 0x7f7f2815c5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:30.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:30.421+0000 7f7f30200640 1 --2- 172.21.15.6:0/1267777343 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7f28152d80 0x7f7f280c8d20 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:30.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:30.421+0000 7f7f30200640 1 -- 172.21.15.6:0/1267777343 >> 172.21.15.6:0/1267777343 conn(0x7f7f28070280 msgr2=0x7f7f2806f320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:20:30.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:30.422+0000 7f7f30200640 1 -- 172.21.15.6:0/1267777343 shutdown_connections 2024-04-24T23:20:30.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:30.422+0000 7f7f30200640 1 -- 172.21.15.6:0/1267777343 wait complete. 2024-04-24T23:20:30.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:30.423+0000 7f7f30200640 1 Processor -- start 2024-04-24T23:20:30.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:30.424+0000 7f7f30200640 1 -- start start 2024-04-24T23:20:30.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:30.425+0000 7f7f30200640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7f28152d80 0x7f7f281efbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:20:30.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:30.425+0000 7f7f30200640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7f281592a0 0x7f7f2814a140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:20:30.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:30.425+0000 7f7f30200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7f2815cb10 0x7f7f280cd750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:20:30.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:30.425+0000 7f7f30200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f7f2815bdd0 con 0x7f7f2815cb10 2024-04-24T23:20:30.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:30.425+0000 7f7f30200640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f7f280c3a30 con 0x7f7f281592a0 2024-04-24T23:20:30.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:30.425+0000 7f7f30200640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f7f28159cd0 con 0x7f7f28152d80 2024-04-24T23:20:30.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:30.425+0000 7f7f2ce00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7f281592a0 0x7f7f2814a140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:20:30.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:30.426+0000 7f7f2ce00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7f281592a0 0x7f7f2814a140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:54002/0 (socket says 172.21.15.6:54002) 2024-04-24T23:20:30.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:30.426+0000 7f7f2ce00640 1 -- 172.21.15.6:0/419717570 learned_addr learned my addr 172.21.15.6:0/419717570 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:20:30.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:30.426+0000 7f7f2d800640 1 --2- 172.21.15.6:0/419717570 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7f28152d80 0x7f7f281efbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:20:30.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:30.427+0000 7f7f2e200640 1 --2- 172.21.15.6:0/419717570 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7f2815cb10 0x7f7f280cd750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:20:30.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:30.427+0000 7f7f2ce00640 1 -- 172.21.15.6:0/419717570 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7f28152d80 msgr2=0x7f7f281efbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:20:30.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:30.427+0000 7f7f2ce00640 1 --2- 172.21.15.6:0/419717570 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7f28152d80 0x7f7f281efbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:30.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:30.427+0000 7f7f2ce00640 1 -- 172.21.15.6:0/419717570 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7f2815cb10 msgr2=0x7f7f280cd750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:20:30.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:30.427+0000 7f7f2ce00640 1 --2- 172.21.15.6:0/419717570 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7f2815cb10 0x7f7f280cd750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:30.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:30.427+0000 7f7f2ce00640 1 -- 172.21.15.6:0/419717570 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f280ca280 con 0x7f7f281592a0 2024-04-24T23:20:30.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:30.427+0000 7f7f2ce00640 1 --2- 172.21.15.6:0/419717570 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7f281592a0 0x7f7f2814a140 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f7f1c00b4b0 tx=0x7f7f1c00b970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:20:30.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:30.428+0000 7f7f25600640 1 -- 172.21.15.6:0/419717570 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f1c014070 con 0x7f7f281592a0 2024-04-24T23:20:30.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:30.428+0000 7f7f25600640 1 -- 172.21.15.6:0/419717570 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f1c010610 con 0x7f7f281592a0 2024-04-24T23:20:30.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:30.428+0000 7f7f25600640 1 -- 172.21.15.6:0/419717570 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f1c0109e0 con 0x7f7f281592a0 2024-04-24T23:20:30.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:30.428+0000 7f7f30200640 1 -- 172.21.15.6:0/419717570 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f280ca560 con 0x7f7f281592a0 2024-04-24T23:20:30.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:30.429+0000 7f7f30200640 1 -- 172.21.15.6:0/419717570 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f2814eb10 con 0x7f7f281592a0 2024-04-24T23:20:30.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:30.432+0000 7f7f25600640 1 -- 172.21.15.6:0/419717570 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f7f1c017070 con 0x7f7f281592a0 2024-04-24T23:20:30.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:30.432+0000 7f7f30200640 1 -- 172.21.15.6:0/419717570 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7f28068b40 con 0x7f7f281592a0 2024-04-24T23:20:30.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:30.433+0000 7f7f25600640 1 --2- 172.21.15.6:0/419717570 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f7f0806bf00 0x7f7f0806e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:20:30.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:30.433+0000 7f7f25600640 1 -- 172.21.15.6:0/419717570 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6762+0+0 (secure 0 0 0) 0x7f7f1c08ad90 con 0x7f7f281592a0 2024-04-24T23:20:30.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:30.434+0000 7f7f2d800640 1 --2- 172.21.15.6:0/419717570 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f7f0806bf00 0x7f7f0806e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:20:30.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:30.435+0000 7f7f2d800640 1 --2- 172.21.15.6:0/419717570 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f7f0806bf00 0x7f7f0806e3c0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f7f20033000 tx=0x7f7f20009000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:20:30.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:30.439+0000 7f7f25600640 1 -- 172.21.15.6:0/419717570 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f7f28068b40 con 0x7f7f281592a0 2024-04-24T23:20:30.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:30.777+0000 7f7f30200640 1 -- 172.21.15.6:0/419717570 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7f280c7e30 con 0x7f7f281592a0 2024-04-24T23:20:30.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:30.779+0000 7f7f25600640 1 -- 172.21.15.6:0/419717570 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6080 (secure 0 0 0) 0x7f7f280c7e30 con 0x7f7f281592a0 2024-04-24T23:20:30.788 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:20:30.788 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:20:17.585709+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4282,"mds_2":4235,"mds_3":4232,"mds_4":5257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4232":{"gid":4232,"name":"b","rank":3,"incarnation":15,"state":"up:active","state_seq":15,"addr":"172.21.15.190:6838/2032280158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2032280158},{"type":"v1","addr":"172.21.15.190:6838","nonce":2032280158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4235":{"gid":4235,"name":"d","rank":2,"incarnation":13,"state":"up:active","state_seq":14,"addr":"172.21.15.190:6839/1572872451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":1572872451},{"type":"v1","addr":"172.21.15.190:6839","nonce":1572872451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4282":{"gid":4282,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":12,"addr":"172.21.15.190:6837/580315269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":580315269},{"type":"v1","addr":"172.21.15.190:6837","nonce":580315269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/489092393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":489092393},{"type":"v1","addr":"172.21.15.6:6835","nonce":489092393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5257":{"gid":5257,"name":"e","rank":4,"incarnation":25,"state":"up:replay","state_seq":1,"addr":"172.21.15.6:6837/3240971655","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":3240971655},{"type":"v1","addr":"172.21.15.6:6837","nonce":3240971655}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4306,"qdb_cluster":[4232,4235,4282,4306]},"id":1}]} 2024-04-24T23:20:30.789 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 25 2024-04-24T23:20:30.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:30.786+0000 7f7f30200640 1 -- 172.21.15.6:0/419717570 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f7f0806bf00 msgr2=0x7f7f0806e3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:20:30.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:30.786+0000 7f7f30200640 1 --2- 172.21.15.6:0/419717570 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f7f0806bf00 0x7f7f0806e3c0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f7f20033000 tx=0x7f7f20009000 comp rx=0 tx=0).stop 2024-04-24T23:20:30.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:30.786+0000 7f7f30200640 1 -- 172.21.15.6:0/419717570 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7f281592a0 msgr2=0x7f7f2814a140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:20:30.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:30.786+0000 7f7f30200640 1 --2- 172.21.15.6:0/419717570 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7f281592a0 0x7f7f2814a140 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f7f1c00b4b0 tx=0x7f7f1c00b970 comp rx=0 tx=0).stop 2024-04-24T23:20:30.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:30.788+0000 7f7f30200640 1 -- 172.21.15.6:0/419717570 shutdown_connections 2024-04-24T23:20:30.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:30.788+0000 7f7f30200640 1 --2- 172.21.15.6:0/419717570 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f7f0806bf00 0x7f7f0806e3c0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:30.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:30.788+0000 7f7f30200640 1 --2- 172.21.15.6:0/419717570 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7f2815cb10 0x7f7f280cd750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:30.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:30.788+0000 7f7f30200640 1 --2- 172.21.15.6:0/419717570 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7f281592a0 0x7f7f2814a140 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:30.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:30.788+0000 7f7f30200640 1 --2- 172.21.15.6:0/419717570 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7f28152d80 0x7f7f281efbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:30.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:30.788+0000 7f7f30200640 1 -- 172.21.15.6:0/419717570 >> 172.21.15.6:0/419717570 conn(0x7f7f28070280 msgr2=0x7f7f280ce290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:20:30.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:30.789+0000 7f7f30200640 1 -- 172.21.15.6:0/419717570 shutdown_connections 2024-04-24T23:20:30.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:30.789+0000 7f7f30200640 1 -- 172.21.15.6:0/419717570 wait complete. 2024-04-24T23:20:30.825 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-24T23:20:32.826 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:20:33.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:33.277+0000 7fc771e00640 1 Processor -- start 2024-04-24T23:20:33.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:33.278+0000 7fc771e00640 1 -- start start 2024-04-24T23:20:33.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:33.283+0000 7fc771e00640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fc76404c310 0x7fc764045a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:20:33.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:33.283+0000 7fc771e00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc76404b7f0 0x7fc7640dfa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:20:33.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:33.283+0000 7fc771e00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc7640dbb90 con 0x7fc7640074b0 2024-04-24T23:20:33.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:33.283+0000 7fc771e00640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7fc764051030 con 0x7fc76404c310 2024-04-24T23:20:33.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:33.283+0000 7fc771e00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fc76404bc80 con 0x7fc76404b7f0 2024-04-24T23:20:33.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:33.284+0000 7fc76aa00640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fc76404c310 0x7fc764045a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:20:33.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:33.284+0000 7fc76aa00640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fc76404c310 0x7fc764045a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:54014/0 (socket says 172.21.15.6:54014) 2024-04-24T23:20:33.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:33.284+0000 7fc76aa00640 1 -- 172.21.15.6:0/3373820636 learned_addr learned my addr 172.21.15.6:0/3373820636 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:20:33.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:33.286+0000 7fc76b400640 1 --2- 172.21.15.6:0/3373820636 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc76404b7f0 0x7fc7640dfa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:20:33.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:33.286+0000 7fc76aa00640 1 -- 172.21.15.6:0/3373820636 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc76404b7f0 msgr2=0x7fc7640dfa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:20:33.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:33.286+0000 7fc76aa00640 1 --2- 172.21.15.6:0/3373820636 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc76404b7f0 0x7fc7640dfa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:33.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:33.286+0000 7fc76aa00640 1 -- 172.21.15.6:0/3373820636 >> v1:172.21.15.6:6789/0 conn(0x7fc7640074b0 legacy=0x7fc7640d5070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:20:33.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:33.286+0000 7fc76aa00640 1 -- 172.21.15.6:0/3373820636 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7640dbe70 con 0x7fc76404c310 2024-04-24T23:20:33.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:33.290+0000 7fc76aa00640 1 --2- 172.21.15.6:0/3373820636 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc76404c310 0x7fc764045a50 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fc75c009870 tx=0x7fc75c02a2f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=481d5c83ded55914 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:20:33.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:33.294+0000 7fc769600640 1 -- 172.21.15.6:0/3373820636 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc75c037070 con 0x7fc76404c310 2024-04-24T23:20:33.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:33.294+0000 7fc769600640 1 -- 172.21.15.6:0/3373820636 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc75c032640 con 0x7fc76404c310 2024-04-24T23:20:33.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:33.294+0000 7fc769600640 1 -- 172.21.15.6:0/3373820636 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc75c032a10 con 0x7fc76404c310 2024-04-24T23:20:33.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:33.295+0000 7fc771e00640 1 -- 172.21.15.6:0/3373820636 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc76404c310 msgr2=0x7fc764045a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:20:33.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:33.295+0000 7fc771e00640 1 --2- 172.21.15.6:0/3373820636 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc76404c310 0x7fc764045a50 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fc75c009870 tx=0x7fc75c02a2f0 comp rx=0 tx=0).stop 2024-04-24T23:20:33.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:33.297+0000 7fc771e00640 1 -- 172.21.15.6:0/3373820636 shutdown_connections 2024-04-24T23:20:33.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:33.297+0000 7fc771e00640 1 --2- 172.21.15.6:0/3373820636 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc76404b7f0 0x7fc7640dfa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:33.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:33.297+0000 7fc771e00640 1 --2- 172.21.15.6:0/3373820636 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc76404c310 0x7fc764045a50 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:33.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:33.297+0000 7fc771e00640 1 -- 172.21.15.6:0/3373820636 >> 172.21.15.6:0/3373820636 conn(0x7fc764050180 msgr2=0x7fc764050590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:20:33.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:33.297+0000 7fc771e00640 1 -- 172.21.15.6:0/3373820636 shutdown_connections 2024-04-24T23:20:33.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:33.297+0000 7fc771e00640 1 -- 172.21.15.6:0/3373820636 wait complete. 2024-04-24T23:20:33.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:33.301+0000 7fc771e00640 1 Processor -- start 2024-04-24T23:20:33.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:33.302+0000 7fc771e00640 1 -- start start 2024-04-24T23:20:33.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:33.304+0000 7fc771e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc7640074b0 0x7fc7640d1bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:20:33.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:33.306+0000 7fc771e00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc76404b7f0 0x7fc7641818f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:20:33.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:33.306+0000 7fc771e00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc76404c310 0x7fc764181e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:20:33.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:33.307+0000 7fc771e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fc76404b2d0 con 0x7fc7640074b0 2024-04-24T23:20:33.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:33.307+0000 7fc771e00640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fc76404aff0 con 0x7fc76404b7f0 2024-04-24T23:20:33.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:33.307+0000 7fc771e00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fc7640d5780 con 0x7fc76404c310 2024-04-24T23:20:33.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:33.307+0000 7fc76b400640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc76404c310 0x7fc764181e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:20:33.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:33.307+0000 7fc76b400640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc76404c310 0x7fc764181e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.6:51296/0 (socket says 172.21.15.6:51296) 2024-04-24T23:20:33.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:33.307+0000 7fc76b400640 1 -- 172.21.15.6:0/1363263007 learned_addr learned my addr 172.21.15.6:0/1363263007 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:20:33.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:33.307+0000 7fc76a000640 1 --2- 172.21.15.6:0/1363263007 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc76404b7f0 0x7fc7641818f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:20:33.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:33.307+0000 7fc76b400640 1 -- 172.21.15.6:0/1363263007 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc76404b7f0 msgr2=0x7fc7641818f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:20:33.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:33.307+0000 7fc76b400640 1 --2- 172.21.15.6:0/1363263007 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc76404b7f0 0x7fc7641818f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:33.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:33.307+0000 7fc76b400640 1 -- 172.21.15.6:0/1363263007 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc7640074b0 msgr2=0x7fc7640d1bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:20:33.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:33.307+0000 7fc76b400640 1 --2- 172.21.15.6:0/1363263007 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc7640074b0 0x7fc7640d1bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:33.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:33.307+0000 7fc76b400640 1 -- 172.21.15.6:0/1363263007 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc764046a60 con 0x7fc76404c310 2024-04-24T23:20:33.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:33.309+0000 7fc76a000640 1 --2- 172.21.15.6:0/1363263007 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc76404b7f0 0x7fc7641818f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:20:33.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:33.311+0000 7fc76b400640 1 --2- 172.21.15.6:0/1363263007 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc76404c310 0x7fc764181e30 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fc76c050e00 tx=0x7fc76c076c40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:20:33.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:33.313+0000 7fc763400640 1 -- 172.21.15.6:0/1363263007 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc76c068b40 con 0x7fc76404c310 2024-04-24T23:20:33.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:33.313+0000 7fc763400640 1 -- 172.21.15.6:0/1363263007 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc76c068ce0 con 0x7fc76404c310 2024-04-24T23:20:33.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:33.313+0000 7fc763400640 1 -- 172.21.15.6:0/1363263007 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc76c096420 con 0x7fc76404c310 2024-04-24T23:20:33.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:33.319+0000 7fc771e00640 1 -- 172.21.15.6:0/1363263007 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7641856d0 con 0x7fc76404c310 2024-04-24T23:20:33.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:33.319+0000 7fc771e00640 1 -- 172.21.15.6:0/1363263007 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc764185970 con 0x7fc76404c310 2024-04-24T23:20:33.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:33.321+0000 7fc771e00640 1 -- 172.21.15.6:0/1363263007 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc7640052c0 con 0x7fc76404c310 2024-04-24T23:20:33.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:33.336+0000 7fc763400640 1 -- 172.21.15.6:0/1363263007 <== mon.2 v2:172.21.15.190:3301/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fc76c0965c0 con 0x7fc76404c310 2024-04-24T23:20:33.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:33.338+0000 7fc763400640 1 --2- 172.21.15.6:0/1363263007 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fc74c06bf00 0x7fc74c06e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:20:33.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:33.338+0000 7fc763400640 1 -- 172.21.15.6:0/1363263007 <== mon.2 v2:172.21.15.190:3301/0 5 ==== osd_map(28..28 src has 1..28) ==== 6762+0+0 (secure 0 0 0) 0x7fc76c107aa0 con 0x7fc76404c310 2024-04-24T23:20:33.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:33.339+0000 7fc76aa00640 1 --2- 172.21.15.6:0/1363263007 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fc74c06bf00 0x7fc74c06e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:20:33.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:33.341+0000 7fc76aa00640 1 --2- 172.21.15.6:0/1363263007 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fc74c06bf00 0x7fc74c06e3c0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fc75c034000 tx=0x7fc75c02c000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:20:33.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:33.345+0000 7fc763400640 1 -- 172.21.15.6:0/1363263007 <== mon.2 v2:172.21.15.190:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fc764046a60 con 0x7fc76404c310 2024-04-24T23:20:33.963 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:20:33.964 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:20:17.585709+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4282,"mds_2":4235,"mds_3":4232,"mds_4":5257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4232":{"gid":4232,"name":"b","rank":3,"incarnation":15,"state":"up:active","state_seq":15,"addr":"172.21.15.190:6838/2032280158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2032280158},{"type":"v1","addr":"172.21.15.190:6838","nonce":2032280158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4235":{"gid":4235,"name":"d","rank":2,"incarnation":13,"state":"up:active","state_seq":14,"addr":"172.21.15.190:6839/1572872451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":1572872451},{"type":"v1","addr":"172.21.15.190:6839","nonce":1572872451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4282":{"gid":4282,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":12,"addr":"172.21.15.190:6837/580315269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":580315269},{"type":"v1","addr":"172.21.15.190:6837","nonce":580315269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/489092393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":489092393},{"type":"v1","addr":"172.21.15.6:6835","nonce":489092393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5257":{"gid":5257,"name":"e","rank":4,"incarnation":25,"state":"up:replay","state_seq":1,"addr":"172.21.15.6:6837/3240971655","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":3240971655},{"type":"v1","addr":"172.21.15.6:6837","nonce":3240971655}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4306,"qdb_cluster":[4232,4235,4282,4306]},"id":1}]} 2024-04-24T23:20:33.964 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 25 2024-04-24T23:20:33.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:33.953+0000 7fc771e00640 1 -- 172.21.15.6:0/1363263007 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc7640d45f0 con 0x7fc76404c310 2024-04-24T23:20:33.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:33.956+0000 7fc763400640 1 -- 172.21.15.6:0/1363263007 <== mon.2 v2:172.21.15.190:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6080 (secure 0 0 0) 0x7fc76c0d5f80 con 0x7fc76404c310 2024-04-24T23:20:33.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:33.972+0000 7fc771e00640 1 -- 172.21.15.6:0/1363263007 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fc74c06bf00 msgr2=0x7fc74c06e3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:20:33.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:33.972+0000 7fc771e00640 1 --2- 172.21.15.6:0/1363263007 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fc74c06bf00 0x7fc74c06e3c0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fc75c034000 tx=0x7fc75c02c000 comp rx=0 tx=0).stop 2024-04-24T23:20:33.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:33.972+0000 7fc771e00640 1 -- 172.21.15.6:0/1363263007 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc76404c310 msgr2=0x7fc764181e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:20:33.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:33.972+0000 7fc771e00640 1 --2- 172.21.15.6:0/1363263007 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc76404c310 0x7fc764181e30 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fc76c050e00 tx=0x7fc76c076c40 comp rx=0 tx=0).stop 2024-04-24T23:20:33.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:33.975+0000 7fc771e00640 1 -- 172.21.15.6:0/1363263007 shutdown_connections 2024-04-24T23:20:33.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:33.975+0000 7fc771e00640 1 --2- 172.21.15.6:0/1363263007 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fc74c06bf00 0x7fc74c06e3c0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:33.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:33.975+0000 7fc771e00640 1 --2- 172.21.15.6:0/1363263007 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc76404c310 0x7fc764181e30 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:33.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:33.975+0000 7fc771e00640 1 --2- 172.21.15.6:0/1363263007 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc76404b7f0 0x7fc7641818f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:33.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:33.975+0000 7fc771e00640 1 --2- 172.21.15.6:0/1363263007 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc7640074b0 0x7fc7640d1bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:33.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:33.975+0000 7fc771e00640 1 -- 172.21.15.6:0/1363263007 >> 172.21.15.6:0/1363263007 conn(0x7fc764050180 msgr2=0x7fc764004900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:20:33.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:33.975+0000 7fc771e00640 1 -- 172.21.15.6:0/1363263007 shutdown_connections 2024-04-24T23:20:33.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:33.975+0000 7fc771e00640 1 -- 172.21.15.6:0/1363263007 wait complete. 2024-04-24T23:20:33.992 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-24T23:20:34.005 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-24T23:20:34.007 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-24T23:20:36.007 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:20:36.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:36.355+0000 7f4eddc00640 1 Processor -- start 2024-04-24T23:20:36.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:36.355+0000 7f4eddc00640 1 -- start start 2024-04-24T23:20:36.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:36.357+0000 7f4eddc00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f4ed80c7dc0 0x7f4ed8072530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:20:36.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:36.357+0000 7f4eddc00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4ed80c74c0 con 0x7f4ed816abf0 2024-04-24T23:20:36.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:36.357+0000 7f4eddc00640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4ed80c71e0 con 0x7f4ed8167180 2024-04-24T23:20:36.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:36.357+0000 7f4eddc00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f4ed8168570 con 0x7f4ed80c7dc0 2024-04-24T23:20:36.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:36.358+0000 7f4ed6a00640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f4ed8167180 0x7f4ed806edb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.6:53148/0 (socket says 172.21.15.6:53148) 2024-04-24T23:20:36.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:36.358+0000 7f4ed6a00640 1 -- 172.21.15.6:0/1711739114 learned_addr learned my addr 172.21.15.6:0/1711739114 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:20:36.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:36.361+0000 7f4ed5600640 1 -- 172.21.15.6:0/1711739114 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 255350510 0 0) 0x7f4ed80c74c0 con 0x7f4ed816abf0 2024-04-24T23:20:36.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:36.366+0000 7f4ed6000640 1 --2- 172.21.15.6:0/1711739114 >> v2:172.21.15.6:3300/0 conn(0x7f4ed80c7dc0 0x7f4ed8072530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:20:36.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:36.378+0000 7f4ed5600640 1 -- 172.21.15.6:0/1711739114 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4ebc003db0 con 0x7f4ed816abf0 2024-04-24T23:20:36.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:36.378+0000 7f4ed5600640 1 -- 172.21.15.6:0/1711739114 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2809988898 0 0) 0x7f4ed80c71e0 con 0x7f4ed8167180 2024-04-24T23:20:36.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:36.378+0000 7f4ed5600640 1 -- 172.21.15.6:0/1711739114 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4ed80c74c0 con 0x7f4ed8167180 2024-04-24T23:20:36.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:36.378+0000 7f4ed5600640 1 -- 172.21.15.6:0/1711739114 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3582321203 0 0) 0x7f4ed80c74c0 con 0x7f4ed8167180 2024-04-24T23:20:36.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:36.378+0000 7f4ed5600640 1 -- 172.21.15.6:0/1711739114 >> v2:172.21.15.6:3300/0 conn(0x7f4ed80c7dc0 msgr2=0x7f4ed8072530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:20:36.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:36.378+0000 7f4ed5600640 1 --2- 172.21.15.6:0/1711739114 >> v2:172.21.15.6:3300/0 conn(0x7f4ed80c7dc0 0x7f4ed8072530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:36.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:36.378+0000 7f4ed5600640 1 -- 172.21.15.6:0/1711739114 >> v1:172.21.15.6:6789/0 conn(0x7f4ed816abf0 legacy=0x7f4ed816b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:20:36.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:36.379+0000 7f4ed5600640 1 -- 172.21.15.6:0/1711739114 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ed8071770 con 0x7f4ed8167180 2024-04-24T23:20:36.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:36.379+0000 7f4ed5600640 1 -- 172.21.15.6:0/1711739114 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3463268701 0 0) 0x7f4ed00019f0 con 0x7f4ed8167180 2024-04-24T23:20:36.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:36.379+0000 7f4ed5600640 1 -- 172.21.15.6:0/1711739114 >> v1:172.21.15.190:6789/0 conn(0x7f4ed8167180 legacy=0x7f4ed806edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:20:36.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:36.379+0000 7f4ed5600640 1 --2- 172.21.15.6:0/1711739114 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4ebc0038c0 0x7f4ebc004030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:20:36.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:36.379+0000 7f4ed5600640 1 --2- 172.21.15.6:0/1711739114 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4ebc0045c0 0x7f4ebc004a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:20:36.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:36.379+0000 7f4ed5600640 1 --2- 172.21.15.6:0/1711739114 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4ebc005190 0x7f4ebc005640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:20:36.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:36.379+0000 7f4ed5600640 1 -- 172.21.15.6:0/1711739114 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f4ebc005e50 con 0x7f4ebc0038c0 2024-04-24T23:20:36.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:36.379+0000 7f4ed5600640 1 -- 172.21.15.6:0/1711739114 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f4ebc00c250 con 0x7f4ebc0045c0 2024-04-24T23:20:36.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:36.379+0000 7f4ed5600640 1 -- 172.21.15.6:0/1711739114 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f4ebc00c450 con 0x7f4ebc005190 2024-04-24T23:20:36.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:36.379+0000 7f4ed6000640 1 --2- 172.21.15.6:0/1711739114 >> v2:172.21.15.6:3300/0 conn(0x7f4ed80c7dc0 0x7f4ed8072530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:20:36.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:36.379+0000 7f4ed7400640 1 --2- 172.21.15.6:0/1711739114 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4ebc005190 0x7f4ebc005640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:20:36.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:36.379+0000 7f4ed6a00640 1 --2- 172.21.15.6:0/1711739114 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4ebc0038c0 0x7f4ebc004030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:20:36.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:36.380+0000 7f4ed7400640 1 -- 172.21.15.6:0/1711739114 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4ebc0045c0 msgr2=0x7f4ebc004a40 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-24T23:20:36.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:36.380+0000 7f4ed7400640 1 --2- 172.21.15.6:0/1711739114 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4ebc0045c0 0x7f4ebc004a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:36.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:36.380+0000 7f4ed7400640 1 -- 172.21.15.6:0/1711739114 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4ebc0038c0 msgr2=0x7f4ebc004030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:20:36.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:36.380+0000 7f4ed7400640 1 --2- 172.21.15.6:0/1711739114 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4ebc0038c0 0x7f4ebc004030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:36.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:36.380+0000 7f4ed7400640 1 -- 172.21.15.6:0/1711739114 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ed80c74c0 con 0x7f4ebc005190 2024-04-24T23:20:36.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:36.381+0000 7f4ed7400640 1 -- 172.21.15.6:0/1711739114 reap_dead start 2024-04-24T23:20:36.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:36.382+0000 7f4ed7400640 1 --2- 172.21.15.6:0/1711739114 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4ebc005190 0x7f4ebc005640 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f4ec4007f60 tx=0x7f4ebc0060c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=41aa94274520123a server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:20:36.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:36.385+0000 7f4ed5600640 1 -- 172.21.15.6:0/1711739114 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ec4007b30 con 0x7f4ebc005190 2024-04-24T23:20:36.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:36.385+0000 7f4ed5600640 1 -- 172.21.15.6:0/1711739114 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ec4005270 con 0x7f4ebc005190 2024-04-24T23:20:36.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:36.385+0000 7f4ed5600640 1 -- 172.21.15.6:0/1711739114 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ec40037f0 con 0x7f4ebc005190 2024-04-24T23:20:36.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:36.386+0000 7f4eddc00640 1 -- 172.21.15.6:0/1711739114 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4ebc005190 msgr2=0x7f4ebc005640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:20:36.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:36.386+0000 7f4eddc00640 1 --2- 172.21.15.6:0/1711739114 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4ebc005190 0x7f4ebc005640 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f4ec4007f60 tx=0x7f4ebc0060c0 comp rx=0 tx=0).stop 2024-04-24T23:20:36.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:36.386+0000 7f4eddc00640 1 -- 172.21.15.6:0/1711739114 shutdown_connections 2024-04-24T23:20:36.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:36.386+0000 7f4eddc00640 1 --2- 172.21.15.6:0/1711739114 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4ebc005190 0x7f4ebc005640 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:36.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:36.386+0000 7f4eddc00640 1 -- 172.21.15.6:0/1711739114 >> 172.21.15.6:0/1711739114 conn(0x7f4ed806b0e0 msgr2=0x7f4ed806b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:20:36.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:36.386+0000 7f4eddc00640 1 -- 172.21.15.6:0/1711739114 shutdown_connections 2024-04-24T23:20:36.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:36.386+0000 7f4eddc00640 1 -- 172.21.15.6:0/1711739114 wait complete. 2024-04-24T23:20:36.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:36.390+0000 7f4eddc00640 1 Processor -- start 2024-04-24T23:20:36.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:36.391+0000 7f4eddc00640 1 -- start start 2024-04-24T23:20:36.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:36.396+0000 7f4eddc00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4ebc005190 0x7f4ed8164b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:20:36.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:36.397+0000 7f4eddc00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4ed81687e0 0x7f4ed81650a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:20:36.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:36.397+0000 7f4eddc00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4ed815d530 0x7f4ed815d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:20:36.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:36.397+0000 7f4eddc00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f4ed8071d30 con 0x7f4ebc005190 2024-04-24T23:20:36.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:36.397+0000 7f4eddc00640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f4ed80720a0 con 0x7f4ed81687e0 2024-04-24T23:20:36.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:36.397+0000 7f4eddc00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f4ed80c6e10 con 0x7f4ed815d530 2024-04-24T23:20:36.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:36.398+0000 7f4ed7400640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4ed815d530 0x7f4ed815d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:20:36.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:36.398+0000 7f4ed7400640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4ed815d530 0x7f4ed815d9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.6:54212/0 (socket says 172.21.15.6:54212) 2024-04-24T23:20:36.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:36.398+0000 7f4ed7400640 1 -- 172.21.15.6:0/4234376638 learned_addr learned my addr 172.21.15.6:0/4234376638 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:20:36.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:36.398+0000 7f4ed6000640 1 --2- 172.21.15.6:0/4234376638 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4ed81687e0 0x7f4ed81650a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:20:36.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:36.400+0000 7f4ed6a00640 1 --2- 172.21.15.6:0/4234376638 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4ebc005190 0x7f4ed8164b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:20:36.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:36.401+0000 7f4ed7400640 1 -- 172.21.15.6:0/4234376638 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4ed81687e0 msgr2=0x7f4ed81650a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:20:36.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:36.401+0000 7f4ed7400640 1 --2- 172.21.15.6:0/4234376638 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4ed81687e0 0x7f4ed81650a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:36.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:36.402+0000 7f4ed7400640 1 -- 172.21.15.6:0/4234376638 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4ebc005190 msgr2=0x7f4ed8164b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:20:36.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:36.404+0000 7f4ed7400640 1 --2- 172.21.15.6:0/4234376638 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4ebc005190 0x7f4ed8164b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:36.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:36.404+0000 7f4ed7400640 1 -- 172.21.15.6:0/4234376638 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ed8165950 con 0x7f4ed815d530 2024-04-24T23:20:36.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:36.404+0000 7f4ed6000640 1 --2- 172.21.15.6:0/4234376638 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4ed81687e0 0x7f4ed81650a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:20:36.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:36.404+0000 7f4ed6a00640 1 --2- 172.21.15.6:0/4234376638 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4ebc005190 0x7f4ed8164b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:20:36.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:36.406+0000 7f4ed7400640 1 --2- 172.21.15.6:0/4234376638 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4ed815d530 0x7f4ed815d9e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f4ec400c1d0 tx=0x7f4ec4001790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:20:36.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:36.407+0000 7f4ecea00640 1 -- 172.21.15.6:0/4234376638 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ec40011d0 con 0x7f4ed815d530 2024-04-24T23:20:36.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:36.408+0000 7f4ecea00640 1 -- 172.21.15.6:0/4234376638 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ec4041070 con 0x7f4ed815d530 2024-04-24T23:20:36.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:36.408+0000 7f4eddc00640 1 -- 172.21.15.6:0/4234376638 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ed8162500 con 0x7f4ed815d530 2024-04-24T23:20:36.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:36.408+0000 7f4eddc00640 1 -- 172.21.15.6:0/4234376638 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ed8162ac0 con 0x7f4ed815d530 2024-04-24T23:20:36.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:36.408+0000 7f4ecea00640 1 -- 172.21.15.6:0/4234376638 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ec4005420 con 0x7f4ed815d530 2024-04-24T23:20:36.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:36.410+0000 7f4eddc00640 1 -- 172.21.15.6:0/4234376638 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4eb0005290 con 0x7f4ed815d530 2024-04-24T23:20:36.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:36.418+0000 7f4ecea00640 1 -- 172.21.15.6:0/4234376638 <== mon.2 v2:172.21.15.190:3301/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f4ec403a070 con 0x7f4ed815d530 2024-04-24T23:20:36.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:36.420+0000 7f4ecea00640 1 --2- 172.21.15.6:0/4234376638 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f4eb806bf30 0x7f4eb806e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:20:36.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:36.420+0000 7f4ecea00640 1 -- 172.21.15.6:0/4234376638 <== mon.2 v2:172.21.15.190:3301/0 5 ==== osd_map(28..28 src has 1..28) ==== 6762+0+0 (secure 0 0 0) 0x7f4ec40aab70 con 0x7f4ed815d530 2024-04-24T23:20:36.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:36.421+0000 7f4ed6a00640 1 --2- 172.21.15.6:0/4234376638 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f4eb806bf30 0x7f4eb806e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:20:36.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:36.421+0000 7f4ecea00640 1 -- 172.21.15.6:0/4234376638 <== mon.2 v2:172.21.15.190:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f4ec40d82d0 con 0x7f4ed815d530 2024-04-24T23:20:36.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:36.424+0000 7f4ed6a00640 1 --2- 172.21.15.6:0/4234376638 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f4eb806bf30 0x7f4eb806e3f0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f4ed0001390 tx=0x7f4ed0006040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:20:36.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:36.925+0000 7f4eddc00640 1 -- 172.21.15.6:0/4234376638 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4eb0005580 con 0x7f4ed815d530 2024-04-24T23:20:36.931 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:20:36.931 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:20:17.585709+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4282,"mds_2":4235,"mds_3":4232,"mds_4":5257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4232":{"gid":4232,"name":"b","rank":3,"incarnation":15,"state":"up:active","state_seq":15,"addr":"172.21.15.190:6838/2032280158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2032280158},{"type":"v1","addr":"172.21.15.190:6838","nonce":2032280158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4235":{"gid":4235,"name":"d","rank":2,"incarnation":13,"state":"up:active","state_seq":14,"addr":"172.21.15.190:6839/1572872451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":1572872451},{"type":"v1","addr":"172.21.15.190:6839","nonce":1572872451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4282":{"gid":4282,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":12,"addr":"172.21.15.190:6837/580315269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":580315269},{"type":"v1","addr":"172.21.15.190:6837","nonce":580315269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/489092393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":489092393},{"type":"v1","addr":"172.21.15.6:6835","nonce":489092393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5257":{"gid":5257,"name":"e","rank":4,"incarnation":25,"state":"up:replay","state_seq":1,"addr":"172.21.15.6:6837/3240971655","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":3240971655},{"type":"v1","addr":"172.21.15.6:6837","nonce":3240971655}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4306,"qdb_cluster":[4232,4235,4282,4306]},"id":1}]} 2024-04-24T23:20:36.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:36.926+0000 7f4ecea00640 1 -- 172.21.15.6:0/4234376638 <== mon.2 v2:172.21.15.190:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6080 (secure 0 0 0) 0x7f4ec4079200 con 0x7f4ed815d530 2024-04-24T23:20:36.932 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 25 2024-04-24T23:20:36.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:36.930+0000 7f4eddc00640 1 -- 172.21.15.6:0/4234376638 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f4eb806bf30 msgr2=0x7f4eb806e3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:20:36.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:36.930+0000 7f4eddc00640 1 --2- 172.21.15.6:0/4234376638 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f4eb806bf30 0x7f4eb806e3f0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f4ed0001390 tx=0x7f4ed0006040 comp rx=0 tx=0).stop 2024-04-24T23:20:36.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:36.930+0000 7f4eddc00640 1 -- 172.21.15.6:0/4234376638 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4ed815d530 msgr2=0x7f4ed815d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:20:36.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:36.930+0000 7f4eddc00640 1 --2- 172.21.15.6:0/4234376638 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4ed815d530 0x7f4ed815d9e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f4ec400c1d0 tx=0x7f4ec4001790 comp rx=0 tx=0).stop 2024-04-24T23:20:36.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:36.931+0000 7f4eddc00640 1 -- 172.21.15.6:0/4234376638 shutdown_connections 2024-04-24T23:20:36.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:36.931+0000 7f4eddc00640 1 --2- 172.21.15.6:0/4234376638 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f4eb806bf30 0x7f4eb806e3f0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:36.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:36.931+0000 7f4eddc00640 1 --2- 172.21.15.6:0/4234376638 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4ed815d530 0x7f4ed815d9e0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:36.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:36.931+0000 7f4eddc00640 1 --2- 172.21.15.6:0/4234376638 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4ed81687e0 0x7f4ed81650a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:36.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:36.931+0000 7f4eddc00640 1 --2- 172.21.15.6:0/4234376638 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4ebc005190 0x7f4ed8164b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:36.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:36.931+0000 7f4eddc00640 1 -- 172.21.15.6:0/4234376638 >> 172.21.15.6:0/4234376638 conn(0x7f4ed806b0e0 msgr2=0x7f4ed80cb9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:20:36.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:36.932+0000 7f4eddc00640 1 -- 172.21.15.6:0/4234376638 shutdown_connections 2024-04-24T23:20:36.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:36.932+0000 7f4eddc00640 1 -- 172.21.15.6:0/4234376638 wait complete. 2024-04-24T23:20:36.979 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 25, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-24T23:15:36.400716+0000", "damaged": [], "data_pools": [ 3, 4 ], "enabled": true, "epoch": 25, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4232": { "addr": "172.21.15.190:6838/2032280158", "addrs": { "addrvec": [ { "addr": "172.21.15.190:6835", "nonce": 2032280158, "type": "v2" }, { "addr": "172.21.15.190:6838", "nonce": 2032280158, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4232, "incarnation": 15, "join_fscid": -1, "name": "b", "rank": 3, "state": "up:active", "state_seq": 15 }, "gid_4235": { "addr": "172.21.15.190:6839/1572872451", "addrs": { "addrvec": [ { "addr": "172.21.15.190:6836", "nonce": 1572872451, "type": "v2" }, { "addr": "172.21.15.190:6839", "nonce": 1572872451, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4235, "incarnation": 13, "join_fscid": -1, "name": "d", "rank": 2, "state": "up:active", "state_seq": 14 }, "gid_4282": { "addr": "172.21.15.190:6837/580315269", "addrs": { "addrvec": [ { "addr": "172.21.15.190:6834", "nonce": 580315269, "type": "v2" }, { "addr": "172.21.15.190:6837", "nonce": 580315269, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4282, "incarnation": 9, "join_fscid": -1, "name": "f", "rank": 1, "state": "up:active", "state_seq": 12 }, "gid_4306": { "addr": "172.21.15.6:6835/489092393", "addrs": { "addrvec": [ { "addr": "172.21.15.6:6833", "nonce": 489092393, "type": "v2" }, { "addr": "172.21.15.6:6835", "nonce": 489092393, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4306, "incarnation": 4, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 4 }, "gid_5257": { "addr": "172.21.15.6:6837/3240971655", "addrs": { "addrvec": [ { "addr": "172.21.15.6:6836", "nonce": 3240971655, "type": "v2" }, { "addr": "172.21.15.6:6837", "nonce": 3240971655, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5257, "incarnation": 25, "join_fscid": -1, "name": "e", "rank": 4, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 28, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-04-24T23:20:17.585709+0000", "qdb_cluster": [ 4232, 4235, 4282, 4306 ], "qdb_leader": 4306, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4306, "mds_1": 4282, "mds_2": 4235, "mds_3": 4232, "mds_4": 5257 } } } ], "standbys": [] } 2024-04-24T23:20:38.979 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:20:39.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:39.336+0000 7f6882400640 1 Processor -- start 2024-04-24T23:20:39.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:39.336+0000 7f6882400640 1 -- start start 2024-04-24T23:20:39.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:39.338+0000 7f6882400640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f687c150990 0x7f687c15a980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:20:39.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:39.338+0000 7f6882400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f687c157470 0x7f687c161370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:20:39.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:39.338+0000 7f687b400640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f687c150990 0x7f687c15a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:20:39.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:39.339+0000 7f687b400640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f687c150990 0x7f687c15a980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:46224/0 (socket says 172.21.15.6:46224) 2024-04-24T23:20:39.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:39.339+0000 7f687aa00640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f687c073270 0x7f687c0c9110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.6:53156/0 (socket says 172.21.15.6:53156) 2024-04-24T23:20:39.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:39.339+0000 7f687aa00640 1 -- 172.21.15.6:0/4003958975 learned_addr learned my addr 172.21.15.6:0/4003958975 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:20:39.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:39.339+0000 7f6882400640 1 -- 172.21.15.6:0/4003958975 --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f687c15aec0 con 0x7f687c073270 2024-04-24T23:20:39.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:39.339+0000 7f6882400640 1 -- 172.21.15.6:0/4003958975 --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f687c0c8c50 con 0x7f687c157470 2024-04-24T23:20:39.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:39.341+0000 7f6882400640 1 -- 172.21.15.6:0/4003958975 --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f687c0c8690 con 0x7f687c150990 2024-04-24T23:20:39.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:39.341+0000 7f687be00640 1 --2- 172.21.15.6:0/4003958975 >> v2:172.21.15.6:3300/0 conn(0x7f687c157470 0x7f687c161370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:20:39.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:39.342+0000 7f687b400640 1 -- 172.21.15.6:0/4003958975 >> v2:172.21.15.6:3300/0 conn(0x7f687c157470 msgr2=0x7f687c161370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:20:39.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:39.342+0000 7f687b400640 1 --2- 172.21.15.6:0/4003958975 >> v2:172.21.15.6:3300/0 conn(0x7f687c157470 0x7f687c161370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:39.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:39.342+0000 7f687b400640 1 -- 172.21.15.6:0/4003958975 >> v1:172.21.15.190:6789/0 conn(0x7f687c073270 legacy=0x7f687c0c9110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:20:39.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:39.342+0000 7f687b400640 1 -- 172.21.15.6:0/4003958975 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f687c0c8970 con 0x7f687c150990 2024-04-24T23:20:39.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:39.343+0000 7f687b400640 1 --2- 172.21.15.6:0/4003958975 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f687c150990 0x7f687c15a980 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f687c0c71a0 tx=0x7f687402c5d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d1f6ec610c595ee6 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:20:39.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:39.344+0000 7f687a000640 1 -- 172.21.15.6:0/4003958975 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f687400b040 con 0x7f687c150990 2024-04-24T23:20:39.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:39.344+0000 7f687a000640 1 -- 172.21.15.6:0/4003958975 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6874033600 con 0x7f687c150990 2024-04-24T23:20:39.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:39.346+0000 7f6882400640 1 -- 172.21.15.6:0/4003958975 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f687c150990 msgr2=0x7f687c15a980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:20:39.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:39.346+0000 7f687a000640 1 -- 172.21.15.6:0/4003958975 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f687400b040 con 0x7f687c150990 2024-04-24T23:20:39.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:39.347+0000 7f6882400640 1 --2- 172.21.15.6:0/4003958975 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f687c150990 0x7f687c15a980 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f687c0c71a0 tx=0x7f687402c5d0 comp rx=0 tx=0).stop 2024-04-24T23:20:39.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:39.347+0000 7f6882400640 1 -- 172.21.15.6:0/4003958975 shutdown_connections 2024-04-24T23:20:39.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:39.347+0000 7f6882400640 1 --2- 172.21.15.6:0/4003958975 >> v2:172.21.15.6:3300/0 conn(0x7f687c157470 0x7f687c161370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:39.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:39.347+0000 7f6882400640 1 --2- 172.21.15.6:0/4003958975 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f687c150990 0x7f687c15a980 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:39.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:39.347+0000 7f6882400640 1 -- 172.21.15.6:0/4003958975 >> 172.21.15.6:0/4003958975 conn(0x7f687c06b0e0 msgr2=0x7f687c06b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:20:39.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:39.348+0000 7f6882400640 1 -- 172.21.15.6:0/4003958975 shutdown_connections 2024-04-24T23:20:39.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:39.348+0000 7f6882400640 1 -- 172.21.15.6:0/4003958975 wait complete. 2024-04-24T23:20:39.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:39.351+0000 7f6882400640 1 Processor -- start 2024-04-24T23:20:39.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:39.354+0000 7f6882400640 1 -- start start 2024-04-24T23:20:39.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:39.355+0000 7f6882400640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f687c073270 0x7f687c152b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:20:39.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:39.355+0000 7f6882400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f687c150990 0x7f687c1ebe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:20:39.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:39.355+0000 7f6882400640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f687c157470 0x7f687c154860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:20:39.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:39.355+0000 7f6882400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f687c0c7a70 con 0x7f687c150990 2024-04-24T23:20:39.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:39.355+0000 7f6882400640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f687c15a460 con 0x7f687c073270 2024-04-24T23:20:39.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:39.355+0000 7f6882400640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f687c150d70 con 0x7f687c157470 2024-04-24T23:20:39.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:39.356+0000 7f687be00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f687c157470 0x7f687c154860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:20:39.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:39.356+0000 7f687b400640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f687c073270 0x7f687c152b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:20:39.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:39.356+0000 7f687be00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f687c157470 0x7f687c154860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.6:54220/0 (socket says 172.21.15.6:54220) 2024-04-24T23:20:39.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:39.356+0000 7f687be00640 1 -- 172.21.15.6:0/2252520184 learned_addr learned my addr 172.21.15.6:0/2252520184 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:20:39.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:39.356+0000 7f687be00640 1 -- 172.21.15.6:0/2252520184 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f687c073270 msgr2=0x7f687c152b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:20:39.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:39.356+0000 7f687be00640 1 --2- 172.21.15.6:0/2252520184 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f687c073270 0x7f687c152b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:39.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:39.357+0000 7f687be00640 1 -- 172.21.15.6:0/2252520184 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f687c150990 msgr2=0x7f687c1ebe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:20:39.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:39.357+0000 7f687be00640 1 --2- 172.21.15.6:0/2252520184 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f687c150990 0x7f687c1ebe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:39.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:39.357+0000 7f687be00640 1 -- 172.21.15.6:0/2252520184 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f687c151840 con 0x7f687c157470 2024-04-24T23:20:39.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:39.357+0000 7f687be00640 1 --2- 172.21.15.6:0/2252520184 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f687c157470 0x7f687c154860 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f686c007ea0 tx=0x7f686c007f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:20:39.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:39.358+0000 7f6873400640 1 -- 172.21.15.6:0/2252520184 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f686c013560 con 0x7f687c157470 2024-04-24T23:20:39.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:39.358+0000 7f6873400640 1 -- 172.21.15.6:0/2252520184 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f686c013700 con 0x7f687c157470 2024-04-24T23:20:39.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:39.358+0000 7f6873400640 1 -- 172.21.15.6:0/2252520184 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f686c013ad0 con 0x7f687c157470 2024-04-24T23:20:39.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:39.358+0000 7f6882400640 1 -- 172.21.15.6:0/2252520184 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f687c1540d0 con 0x7f687c157470 2024-04-24T23:20:39.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:39.358+0000 7f6882400640 1 -- 172.21.15.6:0/2252520184 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f687c151da0 con 0x7f687c157470 2024-04-24T23:20:39.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:39.361+0000 7f6882400640 1 -- 172.21.15.6:0/2252520184 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f687c068e00 con 0x7f687c157470 2024-04-24T23:20:39.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:39.364+0000 7f6873400640 1 -- 172.21.15.6:0/2252520184 <== mon.2 v2:172.21.15.190:3301/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f686c013c70 con 0x7f687c157470 2024-04-24T23:20:39.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:39.366+0000 7f6873400640 1 --2- 172.21.15.6:0/2252520184 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f685806bf30 0x7f685806e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:20:39.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:39.366+0000 7f6873400640 1 -- 172.21.15.6:0/2252520184 <== mon.2 v2:172.21.15.190:3301/0 5 ==== osd_map(28..28 src has 1..28) ==== 6762+0+0 (secure 0 0 0) 0x7f686c08ffa0 con 0x7f687c157470 2024-04-24T23:20:39.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:39.368+0000 7f6873400640 1 -- 172.21.15.6:0/2252520184 <== mon.2 v2:172.21.15.190:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f687c068e00 con 0x7f687c157470 2024-04-24T23:20:39.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:39.368+0000 7f687b400640 1 --2- 172.21.15.6:0/2252520184 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f685806bf30 0x7f685806e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:20:39.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:39.372+0000 7f687b400640 1 --2- 172.21.15.6:0/2252520184 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f685806bf30 0x7f685806e3f0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f6874036000 tx=0x7f687400c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:20:39.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:39.837+0000 7f6882400640 1 -- 172.21.15.6:0/2252520184 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f687c0767b0 con 0x7f687c157470 2024-04-24T23:20:39.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:39.838+0000 7f6873400640 1 -- 172.21.15.6:0/2252520184 <== mon.2 v2:172.21.15.190:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6080 (secure 0 0 0) 0x7f687c151840 con 0x7f687c157470 2024-04-24T23:20:39.839 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:20:39.839 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:20:17.585709+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4282,"mds_2":4235,"mds_3":4232,"mds_4":5257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4232":{"gid":4232,"name":"b","rank":3,"incarnation":15,"state":"up:active","state_seq":15,"addr":"172.21.15.190:6838/2032280158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2032280158},{"type":"v1","addr":"172.21.15.190:6838","nonce":2032280158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4235":{"gid":4235,"name":"d","rank":2,"incarnation":13,"state":"up:active","state_seq":14,"addr":"172.21.15.190:6839/1572872451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":1572872451},{"type":"v1","addr":"172.21.15.190:6839","nonce":1572872451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4282":{"gid":4282,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":12,"addr":"172.21.15.190:6837/580315269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":580315269},{"type":"v1","addr":"172.21.15.190:6837","nonce":580315269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/489092393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":489092393},{"type":"v1","addr":"172.21.15.6:6835","nonce":489092393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5257":{"gid":5257,"name":"e","rank":4,"incarnation":25,"state":"up:replay","state_seq":1,"addr":"172.21.15.6:6837/3240971655","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":3240971655},{"type":"v1","addr":"172.21.15.6:6837","nonce":3240971655}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4306,"qdb_cluster":[4232,4235,4282,4306]},"id":1}]} 2024-04-24T23:20:39.840 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 25 2024-04-24T23:20:39.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:39.843+0000 7f6882400640 1 -- 172.21.15.6:0/2252520184 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f685806bf30 msgr2=0x7f685806e3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:20:39.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:39.843+0000 7f6882400640 1 --2- 172.21.15.6:0/2252520184 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f685806bf30 0x7f685806e3f0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f6874036000 tx=0x7f687400c040 comp rx=0 tx=0).stop 2024-04-24T23:20:39.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:39.843+0000 7f6882400640 1 -- 172.21.15.6:0/2252520184 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f687c157470 msgr2=0x7f687c154860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:20:39.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:39.843+0000 7f6882400640 1 --2- 172.21.15.6:0/2252520184 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f687c157470 0x7f687c154860 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f686c007ea0 tx=0x7f686c007f90 comp rx=0 tx=0).stop 2024-04-24T23:20:39.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:39.844+0000 7f6882400640 1 -- 172.21.15.6:0/2252520184 shutdown_connections 2024-04-24T23:20:39.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:39.844+0000 7f6882400640 1 --2- 172.21.15.6:0/2252520184 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f685806bf30 0x7f685806e3f0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:39.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:39.844+0000 7f6882400640 1 --2- 172.21.15.6:0/2252520184 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f687c157470 0x7f687c154860 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:39.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:39.844+0000 7f6882400640 1 --2- 172.21.15.6:0/2252520184 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f687c150990 0x7f687c1ebe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:39.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:39.844+0000 7f6882400640 1 --2- 172.21.15.6:0/2252520184 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f687c073270 0x7f687c152b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:39.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:39.844+0000 7f6882400640 1 -- 172.21.15.6:0/2252520184 >> 172.21.15.6:0/2252520184 conn(0x7f687c06b0e0 msgr2=0x7f687c0c5550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:20:39.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:39.845+0000 7f6882400640 1 -- 172.21.15.6:0/2252520184 shutdown_connections 2024-04-24T23:20:39.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:39.845+0000 7f6882400640 1 -- 172.21.15.6:0/2252520184 wait complete. 2024-04-24T23:20:39.879 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-24T23:20:41.880 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:20:42.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:42.176+0000 7faba7c00640 1 Processor -- start 2024-04-24T23:20:42.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:42.176+0000 7faba7c00640 1 -- start start 2024-04-24T23:20:42.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:42.178+0000 7faba7c00640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7faba0073270 0x7faba0150990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:20:42.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:42.178+0000 7faba7c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7faba0157470 0x7faba00c9110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:20:42.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:42.178+0000 7faba7c00640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faba00c8690 con 0x7faba006bbd0 2024-04-24T23:20:42.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:42.178+0000 7faba7c00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7faba00c8c50 con 0x7faba0157470 2024-04-24T23:20:42.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:42.178+0000 7faba7c00640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7faba00c8970 con 0x7faba0073270 2024-04-24T23:20:42.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:42.179+0000 7faba5200640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7faba006bbd0 0x7faba015a980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.6:53170/0 (socket says 172.21.15.6:53170) 2024-04-24T23:20:42.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:42.179+0000 7faba5200640 1 -- 172.21.15.6:0/4278770388 learned_addr learned my addr 172.21.15.6:0/4278770388 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:20:42.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:42.179+0000 7fab9f400640 1 -- 172.21.15.6:0/4278770388 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1096824994 0 0) 0x7faba00c8690 con 0x7faba006bbd0 2024-04-24T23:20:42.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:42.179+0000 7fab9f400640 1 -- 172.21.15.6:0/4278770388 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fab84003520 con 0x7faba006bbd0 2024-04-24T23:20:42.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:42.179+0000 7fab9f400640 1 -- 172.21.15.6:0/4278770388 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 526626857 0 0) 0x7fab84003520 con 0x7faba006bbd0 2024-04-24T23:20:42.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:42.180+0000 7fab9f400640 1 -- 172.21.15.6:0/4278770388 >> v2:172.21.15.190:3300/0 conn(0x7faba0073270 msgr2=0x7faba0150990 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-24T23:20:42.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:42.180+0000 7fab9f400640 1 --2- 172.21.15.6:0/4278770388 >> v2:172.21.15.190:3300/0 conn(0x7faba0073270 0x7faba0150990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:42.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:42.180+0000 7fab9f400640 1 -- 172.21.15.6:0/4278770388 >> v2:172.21.15.6:3300/0 conn(0x7faba0157470 msgr2=0x7faba00c9110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:20:42.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:42.180+0000 7fab9f400640 1 --2- 172.21.15.6:0/4278770388 >> v2:172.21.15.6:3300/0 conn(0x7faba0157470 0x7faba00c9110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:42.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:42.180+0000 7fab9f400640 1 -- 172.21.15.6:0/4278770388 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faba015a180 con 0x7faba006bbd0 2024-04-24T23:20:42.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:42.180+0000 7fab9f400640 1 -- 172.21.15.6:0/4278770388 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3463268701 0 0) 0x7fab8c001670 con 0x7faba006bbd0 2024-04-24T23:20:42.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:42.180+0000 7fab9f400640 1 -- 172.21.15.6:0/4278770388 >> v1:172.21.15.190:6789/0 conn(0x7faba006bbd0 legacy=0x7faba015a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:20:42.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:42.180+0000 7fab9f400640 1 --2- 172.21.15.6:0/4278770388 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fab840039b0 0x7fab84003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:20:42.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:42.180+0000 7fab9f400640 1 --2- 172.21.15.6:0/4278770388 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fab84004800 0x7fab84004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:20:42.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:42.180+0000 7fab9f400640 1 --2- 172.21.15.6:0/4278770388 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fab840053c0 0x7fab8400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:20:42.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:42.180+0000 7fab9f400640 1 -- 172.21.15.6:0/4278770388 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7faba00c8c50 con 0x7fab840039b0 2024-04-24T23:20:42.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:42.180+0000 7fab9f400640 1 -- 172.21.15.6:0/4278770388 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7faba00c8970 con 0x7fab84004800 2024-04-24T23:20:42.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:42.180+0000 7fab9f400640 1 -- 172.21.15.6:0/4278770388 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fab8400c300 con 0x7fab840053c0 2024-04-24T23:20:42.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:42.181+0000 7faba5200640 1 --2- 172.21.15.6:0/4278770388 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fab840039b0 0x7fab84003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:20:42.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:42.181+0000 7faba5200640 1 -- 172.21.15.6:0/4278770388 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fab840053c0 msgr2=0x7fab8400b9c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-24T23:20:42.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:42.181+0000 7faba5200640 1 --2- 172.21.15.6:0/4278770388 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fab840053c0 0x7fab8400b9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:42.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:42.181+0000 7faba5200640 1 -- 172.21.15.6:0/4278770388 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fab84004800 msgr2=0x7fab84004c80 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-24T23:20:42.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:42.181+0000 7faba5200640 1 --2- 172.21.15.6:0/4278770388 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fab84004800 0x7fab84004c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:42.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:42.181+0000 7faba5200640 1 -- 172.21.15.6:0/4278770388 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faba015a180 con 0x7fab840039b0 2024-04-24T23:20:42.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:42.183+0000 7faba5200640 1 --2- 172.21.15.6:0/4278770388 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fab840039b0 0x7fab84003dd0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fab8c007d40 tx=0x7fab84005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24a4e90afaeca711 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:20:42.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:42.183+0000 7fab9f400640 1 -- 172.21.15.6:0/4278770388 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab8c007d80 con 0x7fab840039b0 2024-04-24T23:20:42.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:42.183+0000 7fab9f400640 1 -- 172.21.15.6:0/4278770388 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fab8c033070 con 0x7fab840039b0 2024-04-24T23:20:42.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:42.183+0000 7fab9f400640 1 -- 172.21.15.6:0/4278770388 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab8c005cf0 con 0x7fab840039b0 2024-04-24T23:20:42.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:42.184+0000 7faba5c00640 1 -- 172.21.15.6:0/4278770388 reap_dead start 2024-04-24T23:20:42.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:42.185+0000 7faba7c00640 1 -- 172.21.15.6:0/4278770388 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fab840039b0 msgr2=0x7fab84003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:20:42.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:42.185+0000 7faba7c00640 1 --2- 172.21.15.6:0/4278770388 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fab840039b0 0x7fab84003dd0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fab8c007d40 tx=0x7fab84005830 comp rx=0 tx=0).stop 2024-04-24T23:20:42.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:42.185+0000 7faba7c00640 1 -- 172.21.15.6:0/4278770388 shutdown_connections 2024-04-24T23:20:42.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:42.185+0000 7faba7c00640 1 --2- 172.21.15.6:0/4278770388 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fab840039b0 0x7fab84003dd0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:42.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:42.185+0000 7faba7c00640 1 -- 172.21.15.6:0/4278770388 >> 172.21.15.6:0/4278770388 conn(0x7faba006b0e0 msgr2=0x7faba006b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:20:42.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:42.186+0000 7faba7c00640 1 -- 172.21.15.6:0/4278770388 shutdown_connections 2024-04-24T23:20:42.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:42.186+0000 7faba7c00640 1 -- 172.21.15.6:0/4278770388 wait complete. 2024-04-24T23:20:42.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:42.195+0000 7faba7c00640 1 Processor -- start 2024-04-24T23:20:42.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:42.195+0000 7faba7c00640 1 -- start start 2024-04-24T23:20:42.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:42.197+0000 7faba7c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fab840039b0 0x7faba01efd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:20:42.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:42.197+0000 7faba7c00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faba01563f0 0x7faba0151cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:20:42.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:42.197+0000 7faba7c00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faba00cc8d0 0x7faba014bb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:20:42.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:42.198+0000 7faba5c00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faba00cc8d0 0x7faba014bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:20:42.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:42.198+0000 7faba5c00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faba00cc8d0 0x7faba014bb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:46262/0 (socket says 172.21.15.6:46262) 2024-04-24T23:20:42.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:42.198+0000 7fab9fe00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faba01563f0 0x7faba0151cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:20:42.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:42.198+0000 7fab9fe00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faba01563f0 0x7faba0151cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.6:54226/0 (socket says 172.21.15.6:54226) 2024-04-24T23:20:42.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:42.198+0000 7faba5200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fab840039b0 0x7faba01efd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:20:42.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:42.198+0000 7faba5c00640 1 -- 172.21.15.6:0/3437876616 learned_addr learned my addr 172.21.15.6:0/3437876616 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:20:42.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:42.198+0000 7faba7c00640 1 -- 172.21.15.6:0/3437876616 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7faba00c45f0 con 0x7fab840039b0 2024-04-24T23:20:42.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:42.198+0000 7faba7c00640 1 -- 172.21.15.6:0/3437876616 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7faba00c7a70 con 0x7faba00cc8d0 2024-04-24T23:20:42.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:42.198+0000 7faba7c00640 1 -- 172.21.15.6:0/3437876616 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7faba00c4770 con 0x7faba01563f0 2024-04-24T23:20:42.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:42.199+0000 7fab9fe00640 1 -- 172.21.15.6:0/3437876616 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faba00cc8d0 msgr2=0x7faba014bb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:20:42.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:42.199+0000 7fab9fe00640 1 --2- 172.21.15.6:0/3437876616 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faba00cc8d0 0x7faba014bb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:42.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:42.199+0000 7fab9fe00640 1 -- 172.21.15.6:0/3437876616 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fab840039b0 msgr2=0x7faba01efd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:20:42.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:42.202+0000 7fab9fe00640 1 --2- 172.21.15.6:0/3437876616 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fab840039b0 0x7faba01efd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:42.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:42.203+0000 7fab9fe00640 1 -- 172.21.15.6:0/3437876616 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faba0157d90 con 0x7faba01563f0 2024-04-24T23:20:42.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:42.203+0000 7faba5c00640 1 --2- 172.21.15.6:0/3437876616 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faba00cc8d0 0x7faba014bb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:20:42.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:42.203+0000 7faba5200640 1 --2- 172.21.15.6:0/3437876616 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fab840039b0 0x7faba01efd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:20:42.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:42.204+0000 7fab9fe00640 1 --2- 172.21.15.6:0/3437876616 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faba01563f0 0x7faba0151cf0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7faba0064c60 tx=0x7fab98009ef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:20:42.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:42.209+0000 7fab9cc00640 1 -- 172.21.15.6:0/3437876616 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab98019070 con 0x7faba01563f0 2024-04-24T23:20:42.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:42.211+0000 7fab9cc00640 1 -- 172.21.15.6:0/3437876616 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fab9800cd90 con 0x7faba01563f0 2024-04-24T23:20:42.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:42.211+0000 7faba7c00640 1 -- 172.21.15.6:0/3437876616 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faba0158070 con 0x7faba01563f0 2024-04-24T23:20:42.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:42.211+0000 7faba7c00640 1 -- 172.21.15.6:0/3437876616 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7faba01ef500 con 0x7faba01563f0 2024-04-24T23:20:42.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:42.212+0000 7fab9cc00640 1 -- 172.21.15.6:0/3437876616 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab98014680 con 0x7faba01563f0 2024-04-24T23:20:42.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:42.215+0000 7faba7c00640 1 -- 172.21.15.6:0/3437876616 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fab78005290 con 0x7faba01563f0 2024-04-24T23:20:42.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:42.215+0000 7fab9cc00640 1 -- 172.21.15.6:0/3437876616 <== mon.2 v2:172.21.15.190:3301/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fab9801c070 con 0x7faba01563f0 2024-04-24T23:20:42.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:42.217+0000 7fab9cc00640 1 --2- 172.21.15.6:0/3437876616 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fab8006be60 0x7fab8006e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:20:42.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:42.217+0000 7fab9cc00640 1 -- 172.21.15.6:0/3437876616 <== mon.2 v2:172.21.15.190:3301/0 5 ==== osd_map(28..28 src has 1..28) ==== 6762+0+0 (secure 0 0 0) 0x7fab9808b090 con 0x7faba01563f0 2024-04-24T23:20:42.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:42.221+0000 7faba5200640 1 --2- 172.21.15.6:0/3437876616 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fab8006be60 0x7fab8006e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:20:42.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:42.222+0000 7faba5200640 1 --2- 172.21.15.6:0/3437876616 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fab8006be60 0x7fab8006e320 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fab8c037000 tx=0x7fab8c03b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:20:42.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:42.223+0000 7fab9cc00640 1 -- 172.21.15.6:0/3437876616 <== mon.2 v2:172.21.15.190:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fab98058f80 con 0x7faba01563f0 2024-04-24T23:20:42.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:42.792+0000 7faba7c00640 1 -- 172.21.15.6:0/3437876616 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fab78005850 con 0x7faba01563f0 2024-04-24T23:20:42.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:42.793+0000 7fab9cc00640 1 -- 172.21.15.6:0/3437876616 <== mon.2 v2:172.21.15.190:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6080 (secure 0 0 0) 0x7fab9805d280 con 0x7faba01563f0 2024-04-24T23:20:42.795 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:20:42.795 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:20:17.585709+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4282,"mds_2":4235,"mds_3":4232,"mds_4":5257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4232":{"gid":4232,"name":"b","rank":3,"incarnation":15,"state":"up:active","state_seq":15,"addr":"172.21.15.190:6838/2032280158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2032280158},{"type":"v1","addr":"172.21.15.190:6838","nonce":2032280158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4235":{"gid":4235,"name":"d","rank":2,"incarnation":13,"state":"up:active","state_seq":14,"addr":"172.21.15.190:6839/1572872451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":1572872451},{"type":"v1","addr":"172.21.15.190:6839","nonce":1572872451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4282":{"gid":4282,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":12,"addr":"172.21.15.190:6837/580315269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":580315269},{"type":"v1","addr":"172.21.15.190:6837","nonce":580315269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/489092393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":489092393},{"type":"v1","addr":"172.21.15.6:6835","nonce":489092393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5257":{"gid":5257,"name":"e","rank":4,"incarnation":25,"state":"up:replay","state_seq":1,"addr":"172.21.15.6:6837/3240971655","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":3240971655},{"type":"v1","addr":"172.21.15.6:6837","nonce":3240971655}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4306,"qdb_cluster":[4232,4235,4282,4306]},"id":1}]} 2024-04-24T23:20:42.795 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 25 2024-04-24T23:20:42.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:42.802+0000 7faba7c00640 1 -- 172.21.15.6:0/3437876616 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fab8006be60 msgr2=0x7fab8006e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:20:42.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:42.802+0000 7faba7c00640 1 --2- 172.21.15.6:0/3437876616 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fab8006be60 0x7fab8006e320 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fab8c037000 tx=0x7fab8c03b040 comp rx=0 tx=0).stop 2024-04-24T23:20:42.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:42.802+0000 7faba7c00640 1 -- 172.21.15.6:0/3437876616 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faba01563f0 msgr2=0x7faba0151cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:20:42.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:42.802+0000 7faba7c00640 1 --2- 172.21.15.6:0/3437876616 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faba01563f0 0x7faba0151cf0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7faba0064c60 tx=0x7fab98009ef0 comp rx=0 tx=0).stop 2024-04-24T23:20:42.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:42.803+0000 7faba7c00640 1 -- 172.21.15.6:0/3437876616 shutdown_connections 2024-04-24T23:20:42.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:42.803+0000 7faba7c00640 1 --2- 172.21.15.6:0/3437876616 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fab8006be60 0x7fab8006e320 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:42.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:42.803+0000 7faba7c00640 1 --2- 172.21.15.6:0/3437876616 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faba01563f0 0x7faba0151cf0 secure :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7faba0064c60 tx=0x7fab98009ef0 comp rx=0 tx=0).stop 2024-04-24T23:20:42.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:42.803+0000 7faba7c00640 1 --2- 172.21.15.6:0/3437876616 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faba00cc8d0 0x7faba014bb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:42.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:42.803+0000 7faba7c00640 1 --2- 172.21.15.6:0/3437876616 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fab840039b0 0x7faba01efd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:42.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:42.803+0000 7faba7c00640 1 -- 172.21.15.6:0/3437876616 >> 172.21.15.6:0/3437876616 conn(0x7faba006b0e0 msgr2=0x7faba014fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:20:42.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:42.805+0000 7faba7c00640 1 -- 172.21.15.6:0/3437876616 shutdown_connections 2024-04-24T23:20:42.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:42.805+0000 7faba7c00640 1 -- 172.21.15.6:0/3437876616 wait complete. 2024-04-24T23:20:42.869 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-24T23:20:44.869 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:20:45.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:45.205+0000 7f5432400640 1 Processor -- start 2024-04-24T23:20:45.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:45.205+0000 7f5432400640 1 -- start start 2024-04-24T23:20:45.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:45.207+0000 7f5432400640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f542c1550c0 0x7f542c06fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:20:45.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:45.208+0000 7f5432400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f542c15e740 0x7f542c15ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:20:45.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:45.208+0000 7f5432400640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f542c15bab0 con 0x7f542c070550 2024-04-24T23:20:45.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:45.208+0000 7f5432400640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f542c0723b0 con 0x7f542c15e740 2024-04-24T23:20:45.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:45.208+0000 7f5432400640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f542c15e2b0 con 0x7f542c1550c0 2024-04-24T23:20:45.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:45.208+0000 7f542be00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f542c15e740 0x7f542c15ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:20:45.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:45.209+0000 7f542be00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f542c15e740 0x7f542c15ebb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:50474/0 (socket says 172.21.15.6:50474) 2024-04-24T23:20:45.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:45.209+0000 7f542be00640 1 -- 172.21.15.6:0/2900339577 learned_addr learned my addr 172.21.15.6:0/2900339577 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:20:45.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:45.210+0000 7f542be00640 1 -- 172.21.15.6:0/2900339577 >> v2:172.21.15.190:3300/0 conn(0x7f542c1550c0 msgr2=0x7f542c06fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:20:45.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:45.210+0000 7f542be00640 1 --2- 172.21.15.6:0/2900339577 >> v2:172.21.15.190:3300/0 conn(0x7f542c1550c0 0x7f542c06fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:45.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:45.210+0000 7f542be00640 1 -- 172.21.15.6:0/2900339577 >> v1:172.21.15.6:6789/0 conn(0x7f542c070550 legacy=0x7f542c072a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:20:45.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:45.210+0000 7f542be00640 1 -- 172.21.15.6:0/2900339577 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f542c15af30 con 0x7f542c15e740 2024-04-24T23:20:45.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:45.212+0000 7f542be00640 1 --2- 172.21.15.6:0/2900339577 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f542c15e740 0x7f542c15ebb0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f542c0ce3c0 tx=0x7f54240330c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f7bccbc938fe1f server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:20:45.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:45.212+0000 7f542a000640 1 -- 172.21.15.6:0/2900339577 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5424033940 con 0x7f542c15e740 2024-04-24T23:20:45.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:45.212+0000 7f542a000640 1 -- 172.21.15.6:0/2900339577 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5424033ae0 con 0x7f542c15e740 2024-04-24T23:20:45.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:45.212+0000 7f542a000640 1 -- 172.21.15.6:0/2900339577 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5424033940 con 0x7f542c15e740 2024-04-24T23:20:45.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:45.213+0000 7f5432400640 1 -- 172.21.15.6:0/2900339577 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f542c15e740 msgr2=0x7f542c15ebb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:20:45.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:45.213+0000 7f5432400640 1 --2- 172.21.15.6:0/2900339577 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f542c15e740 0x7f542c15ebb0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f542c0ce3c0 tx=0x7f54240330c0 comp rx=0 tx=0).stop 2024-04-24T23:20:45.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:45.214+0000 7f5432400640 1 -- 172.21.15.6:0/2900339577 shutdown_connections 2024-04-24T23:20:45.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:45.214+0000 7f5432400640 1 --2- 172.21.15.6:0/2900339577 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f542c15e740 0x7f542c15ebb0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:45.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:45.214+0000 7f5432400640 1 --2- 172.21.15.6:0/2900339577 >> v2:172.21.15.190:3300/0 conn(0x7f542c1550c0 0x7f542c06fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:45.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:45.214+0000 7f5432400640 1 -- 172.21.15.6:0/2900339577 >> 172.21.15.6:0/2900339577 conn(0x7f542c06c1f0 msgr2=0x7f542c06c600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:20:45.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:45.214+0000 7f5432400640 1 -- 172.21.15.6:0/2900339577 shutdown_connections 2024-04-24T23:20:45.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:45.214+0000 7f5432400640 1 -- 172.21.15.6:0/2900339577 wait complete. 2024-04-24T23:20:45.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:45.218+0000 7f5432400640 1 Processor -- start 2024-04-24T23:20:45.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:45.220+0000 7f5432400640 1 -- start start 2024-04-24T23:20:45.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:45.222+0000 7f5432400640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f542c070550 0x7f542c149090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:20:45.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:45.222+0000 7f5432400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f542c1550c0 0x7f542c15d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:20:45.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:45.222+0000 7f5432400640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f542c15e740 0x7f542c1edca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:20:45.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:45.222+0000 7f5432400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f542c15b7d0 con 0x7f542c1550c0 2024-04-24T23:20:45.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:45.222+0000 7f5432400640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f542c15b4f0 con 0x7f542c070550 2024-04-24T23:20:45.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:45.222+0000 7f5432400640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f542c070260 con 0x7f542c15e740 2024-04-24T23:20:45.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:45.222+0000 7f542aa00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f542c1550c0 0x7f542c15d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:20:45.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:45.223+0000 7f542aa00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f542c1550c0 0x7f542c15d190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:50478/0 (socket says 172.21.15.6:50478) 2024-04-24T23:20:45.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:45.223+0000 7f542aa00640 1 -- 172.21.15.6:0/616919258 learned_addr learned my addr 172.21.15.6:0/616919258 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:20:45.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:45.223+0000 7f542be00640 1 --2- 172.21.15.6:0/616919258 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f542c15e740 0x7f542c1edca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:20:45.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:45.224+0000 7f542aa00640 1 -- 172.21.15.6:0/616919258 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f542c15e740 msgr2=0x7f542c1edca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:20:45.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:45.224+0000 7f542aa00640 1 --2- 172.21.15.6:0/616919258 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f542c15e740 0x7f542c1edca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:45.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:45.224+0000 7f542aa00640 1 -- 172.21.15.6:0/616919258 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f542c070550 msgr2=0x7f542c149090 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-24T23:20:45.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:45.224+0000 7f542aa00640 1 --2- 172.21.15.6:0/616919258 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f542c070550 0x7f542c149090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:45.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:45.224+0000 7f542aa00640 1 -- 172.21.15.6:0/616919258 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f542c148360 con 0x7f542c1550c0 2024-04-24T23:20:45.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:45.224+0000 7f542aa00640 1 --2- 172.21.15.6:0/616919258 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f542c1550c0 0x7f542c15d190 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f541800c750 tx=0x7f541800cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:20:45.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:45.225+0000 7f5423400640 1 -- 172.21.15.6:0/616919258 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5418017070 con 0x7f542c1550c0 2024-04-24T23:20:45.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:45.225+0000 7f5423400640 1 -- 172.21.15.6:0/616919258 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5418005270 con 0x7f542c1550c0 2024-04-24T23:20:45.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:45.225+0000 7f5423400640 1 -- 172.21.15.6:0/616919258 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f541800a640 con 0x7f542c1550c0 2024-04-24T23:20:45.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:45.226+0000 7f5432400640 1 -- 172.21.15.6:0/616919258 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f542c149820 con 0x7f542c1550c0 2024-04-24T23:20:45.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:45.226+0000 7f5432400640 1 -- 172.21.15.6:0/616919258 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f542c149de0 con 0x7f542c1550c0 2024-04-24T23:20:45.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:45.227+0000 7f5432400640 1 -- 172.21.15.6:0/616919258 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f542c0767b0 con 0x7f542c1550c0 2024-04-24T23:20:45.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:45.230+0000 7f5423400640 1 -- 172.21.15.6:0/616919258 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f541801a070 con 0x7f542c1550c0 2024-04-24T23:20:45.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:45.231+0000 7f5423400640 1 --2- 172.21.15.6:0/616919258 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f540c06be80 0x7f540c06e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:20:45.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:45.231+0000 7f5423400640 1 -- 172.21.15.6:0/616919258 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6762+0+0 (secure 0 0 0) 0x7f541808ac20 con 0x7f542c1550c0 2024-04-24T23:20:45.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:45.231+0000 7f542b400640 1 --2- 172.21.15.6:0/616919258 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f540c06be80 0x7f540c06e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:20:45.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:45.233+0000 7f542b400640 1 --2- 172.21.15.6:0/616919258 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f540c06be80 0x7f540c06e340 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f541c0096b0 tx=0x7f541c006c20 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:20:45.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:45.242+0000 7f5423400640 1 -- 172.21.15.6:0/616919258 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f541808ae20 con 0x7f542c1550c0 2024-04-24T23:20:45.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:45.632+0000 7f5432400640 1 -- 172.21.15.6:0/616919258 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f542c070b10 con 0x7f542c1550c0 2024-04-24T23:20:45.634 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:20:45.634 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:20:17.585709+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4282,"mds_2":4235,"mds_3":4232,"mds_4":5257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4232":{"gid":4232,"name":"b","rank":3,"incarnation":15,"state":"up:active","state_seq":15,"addr":"172.21.15.190:6838/2032280158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2032280158},{"type":"v1","addr":"172.21.15.190:6838","nonce":2032280158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4235":{"gid":4235,"name":"d","rank":2,"incarnation":13,"state":"up:active","state_seq":14,"addr":"172.21.15.190:6839/1572872451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":1572872451},{"type":"v1","addr":"172.21.15.190:6839","nonce":1572872451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4282":{"gid":4282,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":12,"addr":"172.21.15.190:6837/580315269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":580315269},{"type":"v1","addr":"172.21.15.190:6837","nonce":580315269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/489092393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":489092393},{"type":"v1","addr":"172.21.15.6:6835","nonce":489092393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5257":{"gid":5257,"name":"e","rank":4,"incarnation":25,"state":"up:replay","state_seq":1,"addr":"172.21.15.6:6837/3240971655","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":3240971655},{"type":"v1","addr":"172.21.15.6:6837","nonce":3240971655}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4306,"qdb_cluster":[4232,4235,4282,4306]},"id":1}]} 2024-04-24T23:20:45.634 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 25 2024-04-24T23:20:45.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:45.632+0000 7f5423400640 1 -- 172.21.15.6:0/616919258 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6080 (secure 0 0 0) 0x7f5418088e20 con 0x7f542c1550c0 2024-04-24T23:20:45.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:45.635+0000 7f5432400640 1 -- 172.21.15.6:0/616919258 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f540c06be80 msgr2=0x7f540c06e340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:20:45.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:45.635+0000 7f5432400640 1 --2- 172.21.15.6:0/616919258 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f540c06be80 0x7f540c06e340 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f541c0096b0 tx=0x7f541c006c20 comp rx=0 tx=0).stop 2024-04-24T23:20:45.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:45.635+0000 7f5432400640 1 -- 172.21.15.6:0/616919258 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f542c1550c0 msgr2=0x7f542c15d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:20:45.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:45.635+0000 7f5432400640 1 --2- 172.21.15.6:0/616919258 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f542c1550c0 0x7f542c15d190 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f541800c750 tx=0x7f541800cc10 comp rx=0 tx=0).stop 2024-04-24T23:20:45.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:45.635+0000 7f5432400640 1 -- 172.21.15.6:0/616919258 shutdown_connections 2024-04-24T23:20:45.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:45.635+0000 7f5432400640 1 --2- 172.21.15.6:0/616919258 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f540c06be80 0x7f540c06e340 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:45.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:45.635+0000 7f5432400640 1 --2- 172.21.15.6:0/616919258 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f542c15e740 0x7f542c1edca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:45.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:45.635+0000 7f5432400640 1 --2- 172.21.15.6:0/616919258 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f542c1550c0 0x7f542c15d190 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:45.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:45.635+0000 7f5432400640 1 --2- 172.21.15.6:0/616919258 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f542c070550 0x7f542c149090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:45.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:45.635+0000 7f5432400640 1 -- 172.21.15.6:0/616919258 >> 172.21.15.6:0/616919258 conn(0x7f542c06c1f0 msgr2=0x7f542c159050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:20:45.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:45.635+0000 7f5432400640 1 -- 172.21.15.6:0/616919258 shutdown_connections 2024-04-24T23:20:45.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:45.635+0000 7f5432400640 1 -- 172.21.15.6:0/616919258 wait complete. 2024-04-24T23:20:45.775 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-24T23:20:47.776 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:20:48.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:48.094+0000 7f1b16200640 1 Processor -- start 2024-04-24T23:20:48.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:48.094+0000 7f1b16200640 1 -- start start 2024-04-24T23:20:48.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:48.098+0000 7f1b16200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f1b10151390 0x7f1b10157a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:20:48.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:48.098+0000 7f1b16200640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f1b100c56f0 0x7f1b100c5b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:20:48.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:48.098+0000 7f1b16200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1b10157f90 con 0x7f1b100caea0 2024-04-24T23:20:48.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:48.098+0000 7f1b16200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f1b10151770 con 0x7f1b10151390 2024-04-24T23:20:48.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:48.098+0000 7f1b16200640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f1b100c6080 con 0x7f1b100c56f0 2024-04-24T23:20:48.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:48.102+0000 7f1b0fe00640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f1b100c56f0 0x7f1b100c5b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:20:48.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:48.103+0000 7f1b0fe00640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f1b100c56f0 0x7f1b100c5b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:60676/0 (socket says 172.21.15.6:60676) 2024-04-24T23:20:48.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:48.103+0000 7f1b0fe00640 1 -- 172.21.15.6:0/2550786701 learned_addr learned my addr 172.21.15.6:0/2550786701 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:20:48.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:48.104+0000 7f1b0f400640 1 --2- 172.21.15.6:0/2550786701 >> v2:172.21.15.6:3300/0 conn(0x7f1b10151390 0x7f1b10157a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:20:48.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:48.107+0000 7f1b0f400640 1 -- 172.21.15.6:0/2550786701 >> v2:172.21.15.190:3300/0 conn(0x7f1b100c56f0 msgr2=0x7f1b100c5b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:20:48.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:48.107+0000 7f1b0f400640 1 --2- 172.21.15.6:0/2550786701 >> v2:172.21.15.190:3300/0 conn(0x7f1b100c56f0 0x7f1b100c5b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:48.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:48.107+0000 7f1b0f400640 1 -- 172.21.15.6:0/2550786701 >> v1:172.21.15.6:6789/0 conn(0x7f1b100caea0 legacy=0x7f1b100cbc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:20:48.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:48.107+0000 7f1b0f400640 1 -- 172.21.15.6:0/2550786701 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b1015a760 con 0x7f1b10151390 2024-04-24T23:20:48.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:48.108+0000 7f1b0f400640 1 --2- 172.21.15.6:0/2550786701 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1b10151390 0x7f1b10157a50 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f1b10158160 tx=0x7f1b0002e4c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9b1c74cc66320fd server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:20:48.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:48.109+0000 7f1b0e000640 1 -- 172.21.15.6:0/2550786701 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b0003a070 con 0x7f1b10151390 2024-04-24T23:20:48.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:48.109+0000 7f1b0e000640 1 -- 172.21.15.6:0/2550786701 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b00033710 con 0x7f1b10151390 2024-04-24T23:20:48.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:48.109+0000 7f1b0e000640 1 -- 172.21.15.6:0/2550786701 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b00033ae0 con 0x7f1b10151390 2024-04-24T23:20:48.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:48.110+0000 7f1b16200640 1 -- 172.21.15.6:0/2550786701 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1b10151390 msgr2=0x7f1b10157a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:20:48.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:48.110+0000 7f1b16200640 1 --2- 172.21.15.6:0/2550786701 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1b10151390 0x7f1b10157a50 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f1b10158160 tx=0x7f1b0002e4c0 comp rx=0 tx=0).stop 2024-04-24T23:20:48.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:48.111+0000 7f1b16200640 1 -- 172.21.15.6:0/2550786701 shutdown_connections 2024-04-24T23:20:48.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:48.111+0000 7f1b16200640 1 --2- 172.21.15.6:0/2550786701 >> v2:172.21.15.190:3300/0 conn(0x7f1b100c56f0 0x7f1b100c5b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:48.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:48.111+0000 7f1b16200640 1 --2- 172.21.15.6:0/2550786701 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1b10151390 0x7f1b10157a50 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:48.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:48.111+0000 7f1b16200640 1 -- 172.21.15.6:0/2550786701 >> 172.21.15.6:0/2550786701 conn(0x7f1b100cc9a0 msgr2=0x7f1b100ccd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:20:48.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:48.111+0000 7f1b16200640 1 -- 172.21.15.6:0/2550786701 shutdown_connections 2024-04-24T23:20:48.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:48.112+0000 7f1b16200640 1 -- 172.21.15.6:0/2550786701 wait complete. 2024-04-24T23:20:48.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:48.140+0000 7f1b16200640 1 Processor -- start 2024-04-24T23:20:48.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:48.140+0000 7f1b16200640 1 -- start start 2024-04-24T23:20:48.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:48.142+0000 7f1b16200640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1b100c56f0 0x7f1b10148610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:20:48.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:48.143+0000 7f1b0f400640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1b100c56f0 0x7f1b10148610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:20:48.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:48.143+0000 7f1b0f400640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1b100c56f0 0x7f1b10148610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.6:53974/0 (socket says 172.21.15.6:53974) 2024-04-24T23:20:48.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:48.144+0000 7f1b16200640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1b100caea0 0x7f1b10149f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:20:48.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:48.145+0000 7f1b16200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1b10151390 0x7f1b10159540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:20:48.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:48.146+0000 7f1b0f400640 1 -- 172.21.15.6:0/989409240 learned_addr learned my addr 172.21.15.6:0/989409240 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:20:48.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:48.146+0000 7f1b16200640 1 -- 172.21.15.6:0/989409240 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f1b1015b000 con 0x7f1b10151390 2024-04-24T23:20:48.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:48.146+0000 7f1b16200640 1 -- 172.21.15.6:0/989409240 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f1b1015ad20 con 0x7f1b100caea0 2024-04-24T23:20:48.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:48.146+0000 7f1b16200640 1 -- 172.21.15.6:0/989409240 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f1b1015b2e0 con 0x7f1b100c56f0 2024-04-24T23:20:48.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:48.147+0000 7f1b0ea00640 1 --2- 172.21.15.6:0/989409240 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1b100caea0 0x7f1b10149f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:20:48.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:48.148+0000 7f1b0fe00640 1 --2- 172.21.15.6:0/989409240 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1b10151390 0x7f1b10159540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:20:48.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:48.148+0000 7f1b0f400640 1 -- 172.21.15.6:0/989409240 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1b100caea0 msgr2=0x7f1b10149f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:20:48.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:48.148+0000 7f1b0f400640 1 --2- 172.21.15.6:0/989409240 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1b100caea0 0x7f1b10149f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:48.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:48.148+0000 7f1b0f400640 1 -- 172.21.15.6:0/989409240 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1b10151390 msgr2=0x7f1b10159540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:20:48.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:48.149+0000 7f1b0f400640 1 --2- 172.21.15.6:0/989409240 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1b10151390 0x7f1b10159540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:48.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:48.149+0000 7f1b0f400640 1 -- 172.21.15.6:0/989409240 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b1014b530 con 0x7f1b100c56f0 2024-04-24T23:20:48.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:48.149+0000 7f1b0fe00640 1 --2- 172.21.15.6:0/989409240 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1b10151390 0x7f1b10159540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:20:48.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:48.152+0000 7f1b0f400640 1 --2- 172.21.15.6:0/989409240 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1b100c56f0 0x7f1b10148610 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f1b0002eab0 tx=0x7f1b000046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:20:48.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:48.152+0000 7f1b07400640 1 -- 172.21.15.6:0/989409240 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b00044020 con 0x7f1b100c56f0 2024-04-24T23:20:48.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:48.153+0000 7f1b16200640 1 -- 172.21.15.6:0/989409240 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b1014cda0 con 0x7f1b100c56f0 2024-04-24T23:20:48.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:48.153+0000 7f1b16200640 1 -- 172.21.15.6:0/989409240 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b1014c060 con 0x7f1b100c56f0 2024-04-24T23:20:48.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:48.159+0000 7f1b07400640 1 -- 172.21.15.6:0/989409240 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b000335d0 con 0x7f1b100c56f0 2024-04-24T23:20:48.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:48.159+0000 7f1b16200640 1 -- 172.21.15.6:0/989409240 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b100767b0 con 0x7f1b100c56f0 2024-04-24T23:20:48.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:48.160+0000 7f1b07400640 1 -- 172.21.15.6:0/989409240 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b0003a040 con 0x7f1b100c56f0 2024-04-24T23:20:48.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:48.167+0000 7f1b07400640 1 -- 172.21.15.6:0/989409240 <== mon.2 v2:172.21.15.190:3301/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f1b00047020 con 0x7f1b100c56f0 2024-04-24T23:20:48.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:48.169+0000 7f1b07400640 1 --2- 172.21.15.6:0/989409240 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f1af006be30 0x7f1af006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:20:48.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:48.169+0000 7f1b07400640 1 -- 172.21.15.6:0/989409240 <== mon.2 v2:172.21.15.190:3301/0 5 ==== osd_map(28..28 src has 1..28) ==== 6762+0+0 (secure 0 0 0) 0x7f1b000ab9e0 con 0x7f1b100c56f0 2024-04-24T23:20:48.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:48.172+0000 7f1b0ea00640 1 --2- 172.21.15.6:0/989409240 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f1af006be30 0x7f1af006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:20:48.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:48.173+0000 7f1b07400640 1 -- 172.21.15.6:0/989409240 <== mon.2 v2:172.21.15.190:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f1b00079fe0 con 0x7f1b100c56f0 2024-04-24T23:20:48.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:48.173+0000 7f1b0ea00640 1 --2- 172.21.15.6:0/989409240 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f1af006be30 0x7f1af006e2f0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f1b10149940 tx=0x7f1afc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:20:48.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:48.798+0000 7f1b16200640 1 -- 172.21.15.6:0/989409240 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1b100cba50 con 0x7f1b100c56f0 2024-04-24T23:20:48.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:48.800+0000 7f1b07400640 1 -- 172.21.15.6:0/989409240 <== mon.2 v2:172.21.15.190:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6080 (secure 0 0 0) 0x7f1b0007ee20 con 0x7f1b100c56f0 2024-04-24T23:20:48.801 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:20:48.801 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:20:17.585709+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4282,"mds_2":4235,"mds_3":4232,"mds_4":5257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4232":{"gid":4232,"name":"b","rank":3,"incarnation":15,"state":"up:active","state_seq":15,"addr":"172.21.15.190:6838/2032280158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2032280158},{"type":"v1","addr":"172.21.15.190:6838","nonce":2032280158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4235":{"gid":4235,"name":"d","rank":2,"incarnation":13,"state":"up:active","state_seq":14,"addr":"172.21.15.190:6839/1572872451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":1572872451},{"type":"v1","addr":"172.21.15.190:6839","nonce":1572872451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4282":{"gid":4282,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":12,"addr":"172.21.15.190:6837/580315269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":580315269},{"type":"v1","addr":"172.21.15.190:6837","nonce":580315269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/489092393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":489092393},{"type":"v1","addr":"172.21.15.6:6835","nonce":489092393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5257":{"gid":5257,"name":"e","rank":4,"incarnation":25,"state":"up:replay","state_seq":1,"addr":"172.21.15.6:6837/3240971655","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":3240971655},{"type":"v1","addr":"172.21.15.6:6837","nonce":3240971655}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4306,"qdb_cluster":[4232,4235,4282,4306]},"id":1}]} 2024-04-24T23:20:48.802 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 25 2024-04-24T23:20:48.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:48.806+0000 7f1b04c00640 1 -- 172.21.15.6:0/989409240 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f1af006be30 msgr2=0x7f1af006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:20:48.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:48.806+0000 7f1b04c00640 1 --2- 172.21.15.6:0/989409240 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f1af006be30 0x7f1af006e2f0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f1b10149940 tx=0x7f1afc009040 comp rx=0 tx=0).stop 2024-04-24T23:20:48.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:48.806+0000 7f1b04c00640 1 -- 172.21.15.6:0/989409240 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1b100c56f0 msgr2=0x7f1b10148610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:20:48.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:48.806+0000 7f1b04c00640 1 --2- 172.21.15.6:0/989409240 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1b100c56f0 0x7f1b10148610 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f1b0002eab0 tx=0x7f1b000046c0 comp rx=0 tx=0).stop 2024-04-24T23:20:48.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:48.806+0000 7f1b04c00640 1 -- 172.21.15.6:0/989409240 shutdown_connections 2024-04-24T23:20:48.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:48.806+0000 7f1b04c00640 1 --2- 172.21.15.6:0/989409240 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f1af006be30 0x7f1af006e2f0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:48.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:48.806+0000 7f1b04c00640 1 --2- 172.21.15.6:0/989409240 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1b10151390 0x7f1b10159540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:48.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:48.806+0000 7f1b04c00640 1 --2- 172.21.15.6:0/989409240 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1b100caea0 0x7f1b10149f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:48.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:48.806+0000 7f1b04c00640 1 --2- 172.21.15.6:0/989409240 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1b100c56f0 0x7f1b10148610 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:48.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:48.806+0000 7f1b04c00640 1 -- 172.21.15.6:0/989409240 >> 172.21.15.6:0/989409240 conn(0x7f1b100cc9a0 msgr2=0x7f1b1014ea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:20:48.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:48.807+0000 7f1b04c00640 1 -- 172.21.15.6:0/989409240 shutdown_connections 2024-04-24T23:20:48.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:48.807+0000 7f1b04c00640 1 -- 172.21.15.6:0/989409240 wait complete. 2024-04-24T23:20:48.841 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-24T23:20:50.843 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:20:51.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:51.318+0000 7f2b6d400640 1 Processor -- start 2024-04-24T23:20:51.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:51.318+0000 7f2b6d400640 1 -- start start 2024-04-24T23:20:51.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:51.323+0000 7f2b6d400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f2b6806ce30 0x7f2b680cc600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:20:51.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:51.323+0000 7f2b6d400640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2b680c6be0 0x7f2b680ccb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:20:51.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:51.323+0000 7f2b6d400640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2b6806d390 con 0x7f2b68152bd0 2024-04-24T23:20:51.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:51.323+0000 7f2b6d400640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f2b6806d970 con 0x7f2b6806ce30 2024-04-24T23:20:51.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:51.323+0000 7f2b6d400640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f2b6806dbc0 con 0x7f2b680c6be0 2024-04-24T23:20:51.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:51.323+0000 7f2b6ca00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2b680c6be0 0x7f2b680ccb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:20:51.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:51.323+0000 7f2b6ca00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2b680c6be0 0x7f2b680ccb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.6:53980/0 (socket says 172.21.15.6:53980) 2024-04-24T23:20:51.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:51.323+0000 7f2b6ca00640 1 -- 172.21.15.6:0/2156477361 learned_addr learned my addr 172.21.15.6:0/2156477361 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:20:51.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:51.324+0000 7f2b6ca00640 1 -- 172.21.15.6:0/2156477361 >> v2:172.21.15.6:3300/0 conn(0x7f2b6806ce30 msgr2=0x7f2b680cc600 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-24T23:20:51.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:51.324+0000 7f2b6ca00640 1 --2- 172.21.15.6:0/2156477361 >> v2:172.21.15.6:3300/0 conn(0x7f2b6806ce30 0x7f2b680cc600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:51.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:51.324+0000 7f2b6ca00640 1 -- 172.21.15.6:0/2156477361 >> v1:172.21.15.6:6789/0 conn(0x7f2b68152bd0 legacy=0x7f2b68158e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:20:51.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:51.324+0000 7f2b6ca00640 1 -- 172.21.15.6:0/2156477361 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b6806dea0 con 0x7f2b680c6be0 2024-04-24T23:20:51.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:51.326+0000 7f2b6ca00640 1 --2- 172.21.15.6:0/2156477361 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2b680c6be0 0x7f2b680ccb40 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f2b60009e50 tx=0x7f2b6003b670 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9bc4ada1abb68227 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:20:51.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:51.326+0000 7f2b66a00640 1 -- 172.21.15.6:0/2156477361 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b60047070 con 0x7f2b680c6be0 2024-04-24T23:20:51.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:51.326+0000 7f2b66a00640 1 -- 172.21.15.6:0/2156477361 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2b6003be50 con 0x7f2b680c6be0 2024-04-24T23:20:51.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:51.326+0000 7f2b66a00640 1 -- 172.21.15.6:0/2156477361 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b60042820 con 0x7f2b680c6be0 2024-04-24T23:20:51.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:51.329+0000 7f2b6d400640 1 -- 172.21.15.6:0/2156477361 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2b680c6be0 msgr2=0x7f2b680ccb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:20:51.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:51.329+0000 7f2b6d400640 1 --2- 172.21.15.6:0/2156477361 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2b680c6be0 0x7f2b680ccb40 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f2b60009e50 tx=0x7f2b6003b670 comp rx=0 tx=0).stop 2024-04-24T23:20:51.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:51.331+0000 7f2b6d400640 1 -- 172.21.15.6:0/2156477361 shutdown_connections 2024-04-24T23:20:51.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:51.331+0000 7f2b6d400640 1 --2- 172.21.15.6:0/2156477361 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2b680c6be0 0x7f2b680ccb40 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:51.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:51.331+0000 7f2b6d400640 1 --2- 172.21.15.6:0/2156477361 >> v2:172.21.15.6:3300/0 conn(0x7f2b6806ce30 0x7f2b680cc600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:51.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:51.331+0000 7f2b6d400640 1 -- 172.21.15.6:0/2156477361 >> 172.21.15.6:0/2156477361 conn(0x7f2b6806ed80 msgr2=0x7f2b6806f190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:20:51.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:51.331+0000 7f2b6d400640 1 -- 172.21.15.6:0/2156477361 shutdown_connections 2024-04-24T23:20:51.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:51.331+0000 7f2b6d400640 1 -- 172.21.15.6:0/2156477361 wait complete. 2024-04-24T23:20:51.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:51.339+0000 7f2b6d400640 1 Processor -- start 2024-04-24T23:20:51.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:51.345+0000 7f2b6d400640 1 -- start start 2024-04-24T23:20:51.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:51.357+0000 7f2b6d400640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2b6806ce30 0x7f2b68148c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:20:51.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:51.357+0000 7f2b6d400640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2b680c6be0 0x7f2b68143000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:20:51.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:51.358+0000 7f2b67e00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2b6806ce30 0x7f2b68148c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:20:51.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:51.358+0000 7f2b67e00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2b6806ce30 0x7f2b68148c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:60702/0 (socket says 172.21.15.6:60702) 2024-04-24T23:20:51.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:51.358+0000 7f2b6d400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2b68152bd0 0x7f2b68157040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:20:51.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:51.359+0000 7f2b67400640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2b680c6be0 0x7f2b68143000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:20:51.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:51.359+0000 7f2b67e00640 1 -- 172.21.15.6:0/1831892550 learned_addr learned my addr 172.21.15.6:0/1831892550 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:20:51.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:51.359+0000 7f2b6ca00640 1 --2- 172.21.15.6:0/1831892550 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2b68152bd0 0x7f2b68157040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:20:51.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:51.360+0000 7f2b6d400640 1 -- 172.21.15.6:0/1831892550 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f2b68072650 con 0x7f2b68152bd0 2024-04-24T23:20:51.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:51.360+0000 7f2b6d400640 1 -- 172.21.15.6:0/1831892550 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f2b68071890 con 0x7f2b6806ce30 2024-04-24T23:20:51.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:51.360+0000 7f2b6d400640 1 -- 172.21.15.6:0/1831892550 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f2b680727d0 con 0x7f2b680c6be0 2024-04-24T23:20:51.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:51.361+0000 7f2b67e00640 1 -- 172.21.15.6:0/1831892550 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2b680c6be0 msgr2=0x7f2b68143000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:20:51.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:51.361+0000 7f2b67e00640 1 --2- 172.21.15.6:0/1831892550 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2b680c6be0 0x7f2b68143000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:51.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:51.361+0000 7f2b67e00640 1 -- 172.21.15.6:0/1831892550 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2b68152bd0 msgr2=0x7f2b68157040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:20:51.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:51.361+0000 7f2b67e00640 1 --2- 172.21.15.6:0/1831892550 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2b68152bd0 0x7f2b68157040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:51.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:51.361+0000 7f2b67e00640 1 -- 172.21.15.6:0/1831892550 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b681ec2c0 con 0x7f2b6806ce30 2024-04-24T23:20:51.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:51.363+0000 7f2b67e00640 1 --2- 172.21.15.6:0/1831892550 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2b6806ce30 0x7f2b68148c30 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f2b5800b3e0 tx=0x7f2b5800b8a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:20:51.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:51.363+0000 7f2b5fe00640 1 -- 172.21.15.6:0/1831892550 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b58015020 con 0x7f2b6806ce30 2024-04-24T23:20:51.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:51.363+0000 7f2b6d400640 1 -- 172.21.15.6:0/1831892550 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b681ec560 con 0x7f2b6806ce30 2024-04-24T23:20:51.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:51.363+0000 7f2b6d400640 1 -- 172.21.15.6:0/1831892550 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b681ece00 con 0x7f2b6806ce30 2024-04-24T23:20:51.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:51.364+0000 7f2b5fe00640 1 -- 172.21.15.6:0/1831892550 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2b58011830 con 0x7f2b6806ce30 2024-04-24T23:20:51.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:51.367+0000 7f2b5fe00640 1 -- 172.21.15.6:0/1831892550 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b58015020 con 0x7f2b6806ce30 2024-04-24T23:20:51.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:51.372+0000 7f2b5fe00640 1 -- 172.21.15.6:0/1831892550 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f2b58018020 con 0x7f2b6806ce30 2024-04-24T23:20:51.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:51.373+0000 7f2b6d400640 1 -- 172.21.15.6:0/1831892550 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b40005290 con 0x7f2b6806ce30 2024-04-24T23:20:51.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:51.375+0000 7f2b5fe00640 1 --2- 172.21.15.6:0/1831892550 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f2b4806bff0 0x7f2b4806e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:20:51.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:51.376+0000 7f2b67400640 1 --2- 172.21.15.6:0/1831892550 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f2b4806bff0 0x7f2b4806e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:20:51.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:51.377+0000 7f2b5fe00640 1 -- 172.21.15.6:0/1831892550 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6762+0+0 (secure 0 0 0) 0x7f2b5808ade0 con 0x7f2b6806ce30 2024-04-24T23:20:51.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:51.377+0000 7f2b67400640 1 --2- 172.21.15.6:0/1831892550 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f2b4806bff0 0x7f2b4806e4b0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f2b54006fd0 tx=0x7f2b54008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:20:51.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:51.386+0000 7f2b5fe00640 1 -- 172.21.15.6:0/1831892550 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f2b5801f020 con 0x7f2b6806ce30 2024-04-24T23:20:51.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:51.917+0000 7f2b6d400640 1 -- 172.21.15.6:0/1831892550 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2b40005580 con 0x7f2b6806ce30 2024-04-24T23:20:51.920 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:51.919+0000 7f2b5fe00640 1 -- 172.21.15.6:0/1831892550 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6080 (secure 0 0 0) 0x7f2b580592d0 con 0x7f2b6806ce30 2024-04-24T23:20:51.921 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:20:51.921 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:20:17.585709+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4282,"mds_2":4235,"mds_3":4232,"mds_4":5257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4232":{"gid":4232,"name":"b","rank":3,"incarnation":15,"state":"up:active","state_seq":15,"addr":"172.21.15.190:6838/2032280158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2032280158},{"type":"v1","addr":"172.21.15.190:6838","nonce":2032280158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4235":{"gid":4235,"name":"d","rank":2,"incarnation":13,"state":"up:active","state_seq":14,"addr":"172.21.15.190:6839/1572872451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":1572872451},{"type":"v1","addr":"172.21.15.190:6839","nonce":1572872451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4282":{"gid":4282,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":12,"addr":"172.21.15.190:6837/580315269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":580315269},{"type":"v1","addr":"172.21.15.190:6837","nonce":580315269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/489092393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":489092393},{"type":"v1","addr":"172.21.15.6:6835","nonce":489092393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5257":{"gid":5257,"name":"e","rank":4,"incarnation":25,"state":"up:replay","state_seq":1,"addr":"172.21.15.6:6837/3240971655","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":3240971655},{"type":"v1","addr":"172.21.15.6:6837","nonce":3240971655}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4306,"qdb_cluster":[4232,4235,4282,4306]},"id":1}]} 2024-04-24T23:20:51.921 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 25 2024-04-24T23:20:51.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:51.923+0000 7f2b6d400640 1 -- 172.21.15.6:0/1831892550 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f2b4806bff0 msgr2=0x7f2b4806e4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:20:51.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:51.923+0000 7f2b6d400640 1 --2- 172.21.15.6:0/1831892550 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f2b4806bff0 0x7f2b4806e4b0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f2b54006fd0 tx=0x7f2b54008040 comp rx=0 tx=0).stop 2024-04-24T23:20:51.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:51.923+0000 7f2b6d400640 1 -- 172.21.15.6:0/1831892550 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2b6806ce30 msgr2=0x7f2b68148c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:20:51.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:51.923+0000 7f2b6d400640 1 --2- 172.21.15.6:0/1831892550 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2b6806ce30 0x7f2b68148c30 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f2b5800b3e0 tx=0x7f2b5800b8a0 comp rx=0 tx=0).stop 2024-04-24T23:20:51.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:51.924+0000 7f2b6d400640 1 -- 172.21.15.6:0/1831892550 shutdown_connections 2024-04-24T23:20:51.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:51.924+0000 7f2b6d400640 1 --2- 172.21.15.6:0/1831892550 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f2b4806bff0 0x7f2b4806e4b0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:51.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:51.924+0000 7f2b6d400640 1 --2- 172.21.15.6:0/1831892550 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2b680c6be0 0x7f2b68143000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:51.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:51.924+0000 7f2b6d400640 1 --2- 172.21.15.6:0/1831892550 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2b68152bd0 0x7f2b68157040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:51.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:51.924+0000 7f2b6d400640 1 --2- 172.21.15.6:0/1831892550 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2b6806ce30 0x7f2b68148c30 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:51.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:51.924+0000 7f2b6d400640 1 -- 172.21.15.6:0/1831892550 >> 172.21.15.6:0/1831892550 conn(0x7f2b6806ed80 msgr2=0x7f2b68158e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:20:51.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:51.924+0000 7f2b6d400640 1 -- 172.21.15.6:0/1831892550 shutdown_connections 2024-04-24T23:20:51.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:51.925+0000 7f2b6d400640 1 -- 172.21.15.6:0/1831892550 wait complete. 2024-04-24T23:20:51.964 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 25, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-24T23:15:36.400716+0000", "damaged": [], "data_pools": [ 3, 4 ], "enabled": true, "epoch": 25, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4232": { "addr": "172.21.15.190:6838/2032280158", "addrs": { "addrvec": [ { "addr": "172.21.15.190:6835", "nonce": 2032280158, "type": "v2" }, { "addr": "172.21.15.190:6838", "nonce": 2032280158, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4232, "incarnation": 15, "join_fscid": -1, "name": "b", "rank": 3, "state": "up:active", "state_seq": 15 }, "gid_4235": { "addr": "172.21.15.190:6839/1572872451", "addrs": { "addrvec": [ { "addr": "172.21.15.190:6836", "nonce": 1572872451, "type": "v2" }, { "addr": "172.21.15.190:6839", "nonce": 1572872451, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4235, "incarnation": 13, "join_fscid": -1, "name": "d", "rank": 2, "state": "up:active", "state_seq": 14 }, "gid_4282": { "addr": "172.21.15.190:6837/580315269", "addrs": { "addrvec": [ { "addr": "172.21.15.190:6834", "nonce": 580315269, "type": "v2" }, { "addr": "172.21.15.190:6837", "nonce": 580315269, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4282, "incarnation": 9, "join_fscid": -1, "name": "f", "rank": 1, "state": "up:active", "state_seq": 12 }, "gid_4306": { "addr": "172.21.15.6:6835/489092393", "addrs": { "addrvec": [ { "addr": "172.21.15.6:6833", "nonce": 489092393, "type": "v2" }, { "addr": "172.21.15.6:6835", "nonce": 489092393, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4306, "incarnation": 4, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 4 }, "gid_5257": { "addr": "172.21.15.6:6837/3240971655", "addrs": { "addrvec": [ { "addr": "172.21.15.6:6836", "nonce": 3240971655, "type": "v2" }, { "addr": "172.21.15.6:6837", "nonce": 3240971655, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5257, "incarnation": 25, "join_fscid": -1, "name": "e", "rank": 4, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 28, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-04-24T23:20:17.585709+0000", "qdb_cluster": [ 4232, 4235, 4282, 4306 ], "qdb_leader": 4306, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4306, "mds_1": 4282, "mds_2": 4235, "mds_3": 4232, "mds_4": 5257 } } } ], "standbys": [] } 2024-04-24T23:20:53.964 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:20:54.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:54.342+0000 7fb2f7e00640 1 Processor -- start 2024-04-24T23:20:54.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:54.342+0000 7fb2f7e00640 1 -- start start 2024-04-24T23:20:54.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:54.344+0000 7fb2f7e00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb2f0050e80 0x7fb2f004b580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:20:54.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:54.344+0000 7fb2f7e00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb2f004beb0 con 0x7fb2f00dedd0 2024-04-24T23:20:54.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:54.344+0000 7fb2f7e00640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb2f0006c50 con 0x7fb2f00de2e0 2024-04-24T23:20:54.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:54.344+0000 7fb2f7e00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fb2f0007840 con 0x7fb2f0050e80 2024-04-24T23:20:54.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:54.345+0000 7fb2f6000640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb2f0050e80 0x7fb2f004b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:20:54.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:54.345+0000 7fb2f5600640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7fb2f00de2e0 0x7fb2f00de6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.6:37074/0 (socket says 172.21.15.6:37074) 2024-04-24T23:20:54.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:54.345+0000 7fb2f5600640 1 -- 172.21.15.6:0/2024693136 learned_addr learned my addr 172.21.15.6:0/2024693136 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:20:54.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:54.346+0000 7fb2fd800640 1 -- 172.21.15.6:0/2024693136 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 554413256 0 0) 0x7fb2f004beb0 con 0x7fb2f00dedd0 2024-04-24T23:20:54.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:54.346+0000 7fb2f6000640 1 -- 172.21.15.6:0/2024693136 >> v1:172.21.15.190:6789/0 conn(0x7fb2f00de2e0 legacy=0x7fb2f00de6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:20:54.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:54.346+0000 7fb2f6000640 1 -- 172.21.15.6:0/2024693136 >> v1:172.21.15.6:6789/0 conn(0x7fb2f00dedd0 legacy=0x7fb2f00e5670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:20:54.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:54.347+0000 7fb2f6000640 1 -- 172.21.15.6:0/2024693136 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2f0006f10 con 0x7fb2f0050e80 2024-04-24T23:20:54.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:54.348+0000 7fb2f6000640 1 --2- 172.21.15.6:0/2024693136 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb2f0050e80 0x7fb2f004b580 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fb2f00079c0 tx=0x7fb2f80a6930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9b283c237069f775 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:20:54.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:54.349+0000 7fb2fd800640 1 -- 172.21.15.6:0/2024693136 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2f80ac6b0 con 0x7fb2f0050e80 2024-04-24T23:20:54.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:54.349+0000 7fb2fd800640 1 -- 172.21.15.6:0/2024693136 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb2f80a9070 con 0x7fb2f0050e80 2024-04-24T23:20:54.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:54.349+0000 7fb2fd800640 1 -- 172.21.15.6:0/2024693136 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2f80ac6b0 con 0x7fb2f0050e80 2024-04-24T23:20:54.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:54.354+0000 7fb2f7e00640 1 -- 172.21.15.6:0/2024693136 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb2f0050e80 msgr2=0x7fb2f004b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:20:54.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:54.354+0000 7fb2f7e00640 1 --2- 172.21.15.6:0/2024693136 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb2f0050e80 0x7fb2f004b580 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fb2f00079c0 tx=0x7fb2f80a6930 comp rx=0 tx=0).stop 2024-04-24T23:20:54.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:54.355+0000 7fb2f7e00640 1 -- 172.21.15.6:0/2024693136 shutdown_connections 2024-04-24T23:20:54.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:54.355+0000 7fb2f7e00640 1 --2- 172.21.15.6:0/2024693136 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb2f0050e80 0x7fb2f004b580 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:54.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:54.355+0000 7fb2f7e00640 1 -- 172.21.15.6:0/2024693136 >> 172.21.15.6:0/2024693136 conn(0x7fb2f0050360 msgr2=0x7fb2f0050770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:20:54.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:54.357+0000 7fb2f7e00640 1 -- 172.21.15.6:0/2024693136 shutdown_connections 2024-04-24T23:20:54.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:54.357+0000 7fb2f7e00640 1 -- 172.21.15.6:0/2024693136 wait complete. 2024-04-24T23:20:54.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:54.365+0000 7fb2f7e00640 1 Processor -- start 2024-04-24T23:20:54.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:54.372+0000 7fb2f7e00640 1 -- start start 2024-04-24T23:20:54.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:54.376+0000 7fb2f7e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb2f0050e80 0x7fb2f00dc900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:20:54.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:54.378+0000 7fb2f6000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb2f0050e80 0x7fb2f00dc900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:20:54.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:54.378+0000 7fb2f6000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb2f0050e80 0x7fb2f00dc900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:50526/0 (socket says 172.21.15.6:50526) 2024-04-24T23:20:54.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:54.379+0000 7fb2f6000640 1 -- 172.21.15.6:0/1747325447 learned_addr learned my addr 172.21.15.6:0/1747325447 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:20:54.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:54.379+0000 7fb2f7e00640 1 --2- 172.21.15.6:0/1747325447 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb2f00de2e0 0x7fb2f00465c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:20:54.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:54.380+0000 7fb2f7e00640 1 --2- 172.21.15.6:0/1747325447 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb2f00dedd0 0x7fb2f00ce390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:20:54.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:54.380+0000 7fb2f5600640 1 --2- 172.21.15.6:0/1747325447 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb2f00de2e0 0x7fb2f00465c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:20:54.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:54.380+0000 7fb2f7e00640 1 -- 172.21.15.6:0/1747325447 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fb2f00d5070 con 0x7fb2f0050e80 2024-04-24T23:20:54.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:54.380+0000 7fb2f7e00640 1 -- 172.21.15.6:0/1747325447 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fb2f00074d0 con 0x7fb2f00de2e0 2024-04-24T23:20:54.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:54.380+0000 7fb2f7e00640 1 -- 172.21.15.6:0/1747325447 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fb2f004ace0 con 0x7fb2f00dedd0 2024-04-24T23:20:54.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:54.381+0000 7fb2f5600640 1 -- 172.21.15.6:0/1747325447 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb2f00dedd0 msgr2=0x7fb2f00ce390 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-24T23:20:54.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:54.381+0000 7fb2f5600640 1 --2- 172.21.15.6:0/1747325447 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb2f00dedd0 0x7fb2f00ce390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:54.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:54.381+0000 7fb2f5600640 1 -- 172.21.15.6:0/1747325447 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb2f0050e80 msgr2=0x7fb2f00dc900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:20:54.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:54.382+0000 7fb2f5600640 1 --2- 172.21.15.6:0/1747325447 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb2f0050e80 0x7fb2f00dc900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:54.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:54.382+0000 7fb2f5600640 1 -- 172.21.15.6:0/1747325447 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2f0170f40 con 0x7fb2f00de2e0 2024-04-24T23:20:54.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:54.382+0000 7fb2f6000640 1 --2- 172.21.15.6:0/1747325447 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb2f0050e80 0x7fb2f00dc900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:20:54.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:54.383+0000 7fb2f5600640 1 --2- 172.21.15.6:0/1747325447 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb2f00de2e0 0x7fb2f00465c0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fb2e800c750 tx=0x7fb2e800cc10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:20:54.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:54.383+0000 7fb2eea00640 1 -- 172.21.15.6:0/1747325447 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2e8018070 con 0x7fb2f00de2e0 2024-04-24T23:20:54.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:54.383+0000 7fb2eea00640 1 -- 172.21.15.6:0/1747325447 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb2e80072d0 con 0x7fb2f00de2e0 2024-04-24T23:20:54.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:54.383+0000 7fb2eea00640 1 -- 172.21.15.6:0/1747325447 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2e80047e0 con 0x7fb2f00de2e0 2024-04-24T23:20:54.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:54.384+0000 7fb2f7e00640 1 -- 172.21.15.6:0/1747325447 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2f01703c0 con 0x7fb2f00de2e0 2024-04-24T23:20:54.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:54.385+0000 7fb2f7e00640 1 -- 172.21.15.6:0/1747325447 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2f0170980 con 0x7fb2f00de2e0 2024-04-24T23:20:54.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:54.387+0000 7fb2f7e00640 1 -- 172.21.15.6:0/1747325447 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb2f0006620 con 0x7fb2f00de2e0 2024-04-24T23:20:54.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:54.394+0000 7fb2eea00640 1 -- 172.21.15.6:0/1747325447 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fb2e8004980 con 0x7fb2f00de2e0 2024-04-24T23:20:54.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:54.399+0000 7fb2eea00640 1 --2- 172.21.15.6:0/1747325447 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fb2d8070270 0x7fb2d8072730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:20:54.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:54.399+0000 7fb2eea00640 1 -- 172.21.15.6:0/1747325447 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6762+0+0 (secure 0 0 0) 0x7fb2e808ab00 con 0x7fb2f00de2e0 2024-04-24T23:20:54.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:54.399+0000 7fb2f6000640 1 --2- 172.21.15.6:0/1747325447 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fb2d8070270 0x7fb2d8072730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:20:54.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:54.399+0000 7fb2eea00640 1 -- 172.21.15.6:0/1747325447 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fb2e8021020 con 0x7fb2f00de2e0 2024-04-24T23:20:54.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:54.400+0000 7fb2f6000640 1 --2- 172.21.15.6:0/1747325447 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fb2d8070270 0x7fb2d8072730 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fb2f8075730 tx=0x7fb2f80ac000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:20:54.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:54.920+0000 7fb2f7e00640 1 -- 172.21.15.6:0/1747325447 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb2f004c0e0 con 0x7fb2f00de2e0 2024-04-24T23:20:54.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:54.928+0000 7fb2eea00640 1 -- 172.21.15.6:0/1747325447 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6080 (secure 0 0 0) 0x7fb2e8059070 con 0x7fb2f00de2e0 2024-04-24T23:20:54.930 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:20:54.930 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:20:17.585709+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4282,"mds_2":4235,"mds_3":4232,"mds_4":5257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4232":{"gid":4232,"name":"b","rank":3,"incarnation":15,"state":"up:active","state_seq":15,"addr":"172.21.15.190:6838/2032280158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2032280158},{"type":"v1","addr":"172.21.15.190:6838","nonce":2032280158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4235":{"gid":4235,"name":"d","rank":2,"incarnation":13,"state":"up:active","state_seq":14,"addr":"172.21.15.190:6839/1572872451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":1572872451},{"type":"v1","addr":"172.21.15.190:6839","nonce":1572872451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4282":{"gid":4282,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":12,"addr":"172.21.15.190:6837/580315269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":580315269},{"type":"v1","addr":"172.21.15.190:6837","nonce":580315269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/489092393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":489092393},{"type":"v1","addr":"172.21.15.6:6835","nonce":489092393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5257":{"gid":5257,"name":"e","rank":4,"incarnation":25,"state":"up:replay","state_seq":1,"addr":"172.21.15.6:6837/3240971655","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":3240971655},{"type":"v1","addr":"172.21.15.6:6837","nonce":3240971655}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4306,"qdb_cluster":[4232,4235,4282,4306]},"id":1}]} 2024-04-24T23:20:54.930 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 25 2024-04-24T23:20:54.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:54.934+0000 7fb2f7e00640 1 -- 172.21.15.6:0/1747325447 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fb2d8070270 msgr2=0x7fb2d8072730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:20:54.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:54.934+0000 7fb2f7e00640 1 --2- 172.21.15.6:0/1747325447 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fb2d8070270 0x7fb2d8072730 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fb2f8075730 tx=0x7fb2f80ac000 comp rx=0 tx=0).stop 2024-04-24T23:20:54.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:54.934+0000 7fb2f7e00640 1 -- 172.21.15.6:0/1747325447 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb2f00de2e0 msgr2=0x7fb2f00465c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:20:54.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:54.934+0000 7fb2f7e00640 1 --2- 172.21.15.6:0/1747325447 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb2f00de2e0 0x7fb2f00465c0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fb2e800c750 tx=0x7fb2e800cc10 comp rx=0 tx=0).stop 2024-04-24T23:20:54.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:54.936+0000 7fb2f7e00640 1 -- 172.21.15.6:0/1747325447 shutdown_connections 2024-04-24T23:20:54.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:54.936+0000 7fb2f7e00640 1 --2- 172.21.15.6:0/1747325447 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fb2d8070270 0x7fb2d8072730 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:54.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:54.936+0000 7fb2f7e00640 1 --2- 172.21.15.6:0/1747325447 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb2f00dedd0 0x7fb2f00ce390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:54.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:54.936+0000 7fb2f7e00640 1 --2- 172.21.15.6:0/1747325447 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb2f00de2e0 0x7fb2f00465c0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:54.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:54.936+0000 7fb2f7e00640 1 --2- 172.21.15.6:0/1747325447 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb2f0050e80 0x7fb2f00dc900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:54.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:54.936+0000 7fb2f7e00640 1 -- 172.21.15.6:0/1747325447 >> 172.21.15.6:0/1747325447 conn(0x7fb2f0050360 msgr2=0x7fb2f004c300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:20:54.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:54.941+0000 7fb2f7e00640 1 -- 172.21.15.6:0/1747325447 shutdown_connections 2024-04-24T23:20:54.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:54.941+0000 7fb2f7e00640 1 -- 172.21.15.6:0/1747325447 wait complete. 2024-04-24T23:20:55.012 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-24T23:20:57.013 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:20:57.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:57.386+0000 7effeea00640 1 Processor -- start 2024-04-24T23:20:57.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:57.387+0000 7effeea00640 1 -- start start 2024-04-24T23:20:57.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:57.391+0000 7effeea00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7effe8154b90 0x7effe815aa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:20:57.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:57.391+0000 7effeea00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7effe80c56d0 0x7effe80c65a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:20:57.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:57.391+0000 7effeea00640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7effe815b9d0 con 0x7effe80c52d0 2024-04-24T23:20:57.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:57.391+0000 7effeea00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7effe80c5b40 con 0x7effe8154b90 2024-04-24T23:20:57.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:57.391+0000 7effeea00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7effe815b130 con 0x7effe80c56d0 2024-04-24T23:20:57.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:57.394+0000 7effed600640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7effe8154b90 0x7effe815aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:20:57.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:57.394+0000 7effee000640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7effe80c56d0 0x7effe80c65a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:20:57.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:57.394+0000 7effee000640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7effe80c56d0 0x7effe80c65a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.6:51642/0 (socket says 172.21.15.6:51642) 2024-04-24T23:20:57.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:57.394+0000 7effee000640 1 -- 172.21.15.6:0/172068186 learned_addr learned my addr 172.21.15.6:0/172068186 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:20:57.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:57.395+0000 7effee000640 1 -- 172.21.15.6:0/172068186 >> v2:172.21.15.6:3300/0 conn(0x7effe8154b90 msgr2=0x7effe815aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:20:57.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:57.396+0000 7effee000640 1 --2- 172.21.15.6:0/172068186 >> v2:172.21.15.6:3300/0 conn(0x7effe8154b90 0x7effe815aa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:57.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:57.396+0000 7effee000640 1 -- 172.21.15.6:0/172068186 >> v1:172.21.15.190:6789/0 conn(0x7effe80c52d0 legacy=0x7effe80cb0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:20:57.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:57.396+0000 7effee000640 1 -- 172.21.15.6:0/172068186 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7effe815b410 con 0x7effe80c56d0 2024-04-24T23:20:57.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:57.396+0000 7effed600640 1 --2- 172.21.15.6:0/172068186 >> v2:172.21.15.6:3300/0 conn(0x7effe8154b90 0x7effe815aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:20:57.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:57.396+0000 7effe7e00640 1 -- 172.21.15.6:0/172068186 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3904239708 0 0) 0x7effe815b9d0 con 0x7effe80c52d0 2024-04-24T23:20:57.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:57.399+0000 7effee000640 1 --2- 172.21.15.6:0/172068186 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7effe80c56d0 0x7effe80c65a0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7effdc009870 tx=0x7effdc02a2f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=28a6b83f107a2c35 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:20:57.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:57.401+0000 7effe7e00640 1 -- 172.21.15.6:0/172068186 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effdc037070 con 0x7effe80c56d0 2024-04-24T23:20:57.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:57.401+0000 7effe7e00640 1 -- 172.21.15.6:0/172068186 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7effdc032640 con 0x7effe80c56d0 2024-04-24T23:20:57.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:57.401+0000 7effe7e00640 1 -- 172.21.15.6:0/172068186 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effdc032a10 con 0x7effe80c56d0 2024-04-24T23:20:57.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:57.402+0000 7effeea00640 1 -- 172.21.15.6:0/172068186 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7effe80c56d0 msgr2=0x7effe80c65a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:20:57.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:57.402+0000 7effeea00640 1 --2- 172.21.15.6:0/172068186 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7effe80c56d0 0x7effe80c65a0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7effdc009870 tx=0x7effdc02a2f0 comp rx=0 tx=0).stop 2024-04-24T23:20:57.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:57.403+0000 7effeea00640 1 -- 172.21.15.6:0/172068186 shutdown_connections 2024-04-24T23:20:57.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:57.403+0000 7effeea00640 1 --2- 172.21.15.6:0/172068186 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7effe80c56d0 0x7effe80c65a0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:57.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:57.403+0000 7effeea00640 1 --2- 172.21.15.6:0/172068186 >> v2:172.21.15.6:3300/0 conn(0x7effe8154b90 0x7effe815aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:57.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:57.403+0000 7effeea00640 1 -- 172.21.15.6:0/172068186 >> 172.21.15.6:0/172068186 conn(0x7effe806cd70 msgr2=0x7effe806d870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:20:57.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:57.403+0000 7effeea00640 1 -- 172.21.15.6:0/172068186 shutdown_connections 2024-04-24T23:20:57.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:57.403+0000 7effeea00640 1 -- 172.21.15.6:0/172068186 wait complete. 2024-04-24T23:20:57.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:57.407+0000 7effeea00640 1 Processor -- start 2024-04-24T23:20:57.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:57.407+0000 7effeea00640 1 -- start start 2024-04-24T23:20:57.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:57.410+0000 7effeea00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7effe80c52d0 0x7effe80c7630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:20:57.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:57.410+0000 7effeea00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7effe80c56d0 0x7effe80c7b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:20:57.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:57.410+0000 7effecc00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7effe80c56d0 0x7effe80c7b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:20:57.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:57.410+0000 7effecc00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7effe80c56d0 0x7effe80c7b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:42728/0 (socket says 172.21.15.6:42728) 2024-04-24T23:20:57.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:57.410+0000 7effecc00640 1 -- 172.21.15.6:0/1311756589 learned_addr learned my addr 172.21.15.6:0/1311756589 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:20:57.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:57.410+0000 7effeea00640 1 --2- 172.21.15.6:0/1311756589 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7effe8154b90 0x7effe81500d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:20:57.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:57.411+0000 7effeea00640 1 -- 172.21.15.6:0/1311756589 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7effe80c6080 con 0x7effe80c52d0 2024-04-24T23:20:57.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:57.411+0000 7effeea00640 1 -- 172.21.15.6:0/1311756589 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7effe80c5da0 con 0x7effe80c56d0 2024-04-24T23:20:57.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:57.411+0000 7effeea00640 1 -- 172.21.15.6:0/1311756589 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7effe80cb7c0 con 0x7effe8154b90 2024-04-24T23:20:57.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:57.411+0000 7effed600640 1 --2- 172.21.15.6:0/1311756589 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7effe80c52d0 0x7effe80c7630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:20:57.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:57.412+0000 7effee000640 1 --2- 172.21.15.6:0/1311756589 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7effe8154b90 0x7effe81500d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:20:57.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:57.412+0000 7effecc00640 1 -- 172.21.15.6:0/1311756589 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7effe8154b90 msgr2=0x7effe81500d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:20:57.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:57.412+0000 7effecc00640 1 --2- 172.21.15.6:0/1311756589 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7effe8154b90 0x7effe81500d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:57.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:57.412+0000 7effecc00640 1 -- 172.21.15.6:0/1311756589 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7effe80c52d0 msgr2=0x7effe80c7630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:20:57.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:57.412+0000 7effecc00640 1 --2- 172.21.15.6:0/1311756589 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7effe80c52d0 0x7effe80c7630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:57.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:57.412+0000 7effecc00640 1 -- 172.21.15.6:0/1311756589 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7effe80c8ee0 con 0x7effe80c56d0 2024-04-24T23:20:57.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:57.413+0000 7effee000640 1 --2- 172.21.15.6:0/1311756589 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7effe8154b90 0x7effe81500d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:20:57.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:57.413+0000 7effecc00640 1 --2- 172.21.15.6:0/1311756589 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7effe80c56d0 0x7effe80c7b70 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7effe000c7e0 tx=0x7effe000cca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:20:57.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:57.414+0000 7effe5600640 1 -- 172.21.15.6:0/1311756589 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effe0003da0 con 0x7effe80c56d0 2024-04-24T23:20:57.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:57.414+0000 7effe5600640 1 -- 172.21.15.6:0/1311756589 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7effe0005070 con 0x7effe80c56d0 2024-04-24T23:20:57.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:57.414+0000 7effe5600640 1 -- 172.21.15.6:0/1311756589 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effe0005460 con 0x7effe80c56d0 2024-04-24T23:20:57.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:57.414+0000 7effeea00640 1 -- 172.21.15.6:0/1311756589 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7effe815ac10 con 0x7effe80c56d0 2024-04-24T23:20:57.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:57.415+0000 7effeea00640 1 -- 172.21.15.6:0/1311756589 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7effe80c8c00 con 0x7effe80c56d0 2024-04-24T23:20:57.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:57.417+0000 7effe5600640 1 -- 172.21.15.6:0/1311756589 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7effe0005600 con 0x7effe80c56d0 2024-04-24T23:20:57.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:57.416+0000 7effeea00640 1 -- 172.21.15.6:0/1311756589 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7effe81526d0 con 0x7effe80c56d0 2024-04-24T23:20:57.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:57.420+0000 7effe5600640 1 --2- 172.21.15.6:0/1311756589 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7effc806be60 0x7effc806e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:20:57.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:57.421+0000 7effe5600640 1 -- 172.21.15.6:0/1311756589 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6762+0+0 (secure 0 0 0) 0x7effe008f060 con 0x7effe80c56d0 2024-04-24T23:20:57.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:57.421+0000 7effed600640 1 --2- 172.21.15.6:0/1311756589 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7effc806be60 0x7effc806e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:20:57.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:57.423+0000 7effe5600640 1 -- 172.21.15.6:0/1311756589 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7effe81526d0 con 0x7effe80c56d0 2024-04-24T23:20:57.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:57.423+0000 7effed600640 1 --2- 172.21.15.6:0/1311756589 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7effc806be60 0x7effc806e320 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7effd4009a30 tx=0x7effd4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:20:57.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:57.910+0000 7effeea00640 1 -- 172.21.15.6:0/1311756589 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7effe80cbc60 con 0x7effe80c56d0 2024-04-24T23:20:57.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:57.911+0000 7effe5600640 1 -- 172.21.15.6:0/1311756589 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6080 (secure 0 0 0) 0x7effe80c8ee0 con 0x7effe80c56d0 2024-04-24T23:20:57.915 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:20:57.915 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:20:17.585709+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4282,"mds_2":4235,"mds_3":4232,"mds_4":5257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4232":{"gid":4232,"name":"b","rank":3,"incarnation":15,"state":"up:active","state_seq":15,"addr":"172.21.15.190:6838/2032280158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2032280158},{"type":"v1","addr":"172.21.15.190:6838","nonce":2032280158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4235":{"gid":4235,"name":"d","rank":2,"incarnation":13,"state":"up:active","state_seq":14,"addr":"172.21.15.190:6839/1572872451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":1572872451},{"type":"v1","addr":"172.21.15.190:6839","nonce":1572872451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4282":{"gid":4282,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":12,"addr":"172.21.15.190:6837/580315269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":580315269},{"type":"v1","addr":"172.21.15.190:6837","nonce":580315269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/489092393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":489092393},{"type":"v1","addr":"172.21.15.6:6835","nonce":489092393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5257":{"gid":5257,"name":"e","rank":4,"incarnation":25,"state":"up:replay","state_seq":1,"addr":"172.21.15.6:6837/3240971655","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":3240971655},{"type":"v1","addr":"172.21.15.6:6837","nonce":3240971655}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4306,"qdb_cluster":[4232,4235,4282,4306]},"id":1}]} 2024-04-24T23:20:57.915 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 25 2024-04-24T23:20:57.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:57.915+0000 7effeea00640 1 -- 172.21.15.6:0/1311756589 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7effc806be60 msgr2=0x7effc806e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:20:57.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:57.915+0000 7effeea00640 1 --2- 172.21.15.6:0/1311756589 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7effc806be60 0x7effc806e320 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7effd4009a30 tx=0x7effd4008040 comp rx=0 tx=0).stop 2024-04-24T23:20:57.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:57.915+0000 7effeea00640 1 -- 172.21.15.6:0/1311756589 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7effe80c56d0 msgr2=0x7effe80c7b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:20:57.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:57.915+0000 7effeea00640 1 --2- 172.21.15.6:0/1311756589 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7effe80c56d0 0x7effe80c7b70 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7effe000c7e0 tx=0x7effe000cca0 comp rx=0 tx=0).stop 2024-04-24T23:20:57.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:57.922+0000 7effeea00640 1 -- 172.21.15.6:0/1311756589 shutdown_connections 2024-04-24T23:20:57.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:57.922+0000 7effeea00640 1 --2- 172.21.15.6:0/1311756589 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7effc806be60 0x7effc806e320 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:57.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:57.922+0000 7effeea00640 1 --2- 172.21.15.6:0/1311756589 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7effe8154b90 0x7effe81500d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:57.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:57.922+0000 7effeea00640 1 --2- 172.21.15.6:0/1311756589 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7effe80c56d0 0x7effe80c7b70 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:57.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:57.922+0000 7effeea00640 1 --2- 172.21.15.6:0/1311756589 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7effe80c52d0 0x7effe80c7630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:20:57.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:57.922+0000 7effeea00640 1 -- 172.21.15.6:0/1311756589 >> 172.21.15.6:0/1311756589 conn(0x7effe806cd70 msgr2=0x7effe8147650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:20:57.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:57.925+0000 7effeea00640 1 -- 172.21.15.6:0/1311756589 shutdown_connections 2024-04-24T23:20:57.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:20:57.926+0000 7effeea00640 1 -- 172.21.15.6:0/1311756589 wait complete. 2024-04-24T23:20:57.958 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-24T23:20:59.960 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:21:00.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:00.275+0000 7f6836c00640 1 Processor -- start 2024-04-24T23:21:00.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:00.276+0000 7f6836c00640 1 -- start start 2024-04-24T23:21:00.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:00.278+0000 7f6836c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f68300c7dc0 0x7f68301623a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:21:00.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:00.278+0000 7f6836c00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f68300c81c0 0x7f68301628e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:21:00.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:00.278+0000 7f6836c00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f68300c7230 con 0x7f6830159110 2024-04-24T23:21:00.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:00.278+0000 7f6836c00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f68301594f0 con 0x7f68300c7dc0 2024-04-24T23:21:00.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:00.278+0000 7f6836c00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f68300c74d0 con 0x7f68300c81c0 2024-04-24T23:21:00.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:00.278+0000 7f6834c00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f68300c81c0 0x7f68301628e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:21:00.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:00.279+0000 7f6834c00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f68300c81c0 0x7f68301628e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.6:51662/0 (socket says 172.21.15.6:51662) 2024-04-24T23:21:00.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:00.279+0000 7f6834c00640 1 -- 172.21.15.6:0/880885762 learned_addr learned my addr 172.21.15.6:0/880885762 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:21:00.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:00.279+0000 7f682f400640 1 --2- 172.21.15.6:0/880885762 >> v2:172.21.15.6:3300/0 conn(0x7f68300c7dc0 0x7f68301623a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:21:00.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:00.279+0000 7f6834c00640 1 -- 172.21.15.6:0/880885762 >> v2:172.21.15.6:3300/0 conn(0x7f68300c7dc0 msgr2=0x7f68301623a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:21:00.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:00.279+0000 7f6834c00640 1 --2- 172.21.15.6:0/880885762 >> v2:172.21.15.6:3300/0 conn(0x7f68300c7dc0 0x7f68301623a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:21:00.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:00.279+0000 7f6834c00640 1 -- 172.21.15.6:0/880885762 >> v1:172.21.15.6:6789/0 conn(0x7f6830159110 legacy=0x7f6830072420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:21:00.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:00.279+0000 7f6834c00640 1 -- 172.21.15.6:0/880885762 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f683006e440 con 0x7f68300c81c0 2024-04-24T23:21:00.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:00.279+0000 7f682f400640 1 --2- 172.21.15.6:0/880885762 >> v2:172.21.15.6:3300/0 conn(0x7f68300c7dc0 0x7f68301623a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:21:00.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:00.280+0000 7f6834c00640 1 --2- 172.21.15.6:0/880885762 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f68300c81c0 0x7f68301628e0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f68300cd3b0 tx=0x7f681c02a280 comp rx=0 tx=0).ready entity=mon.2 client_cookie=62bc2839d2fabaa4 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:21:00.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:00.281+0000 7f682ea00640 1 -- 172.21.15.6:0/880885762 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f681c037070 con 0x7f68300c81c0 2024-04-24T23:21:00.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:00.281+0000 7f682ea00640 1 -- 172.21.15.6:0/880885762 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f681c032440 con 0x7f68300c81c0 2024-04-24T23:21:00.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:00.281+0000 7f682ea00640 1 -- 172.21.15.6:0/880885762 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f681c032810 con 0x7f68300c81c0 2024-04-24T23:21:00.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:00.282+0000 7f6836c00640 1 -- 172.21.15.6:0/880885762 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f68300c81c0 msgr2=0x7f68301628e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:21:00.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:00.282+0000 7f6836c00640 1 --2- 172.21.15.6:0/880885762 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f68300c81c0 0x7f68301628e0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f68300cd3b0 tx=0x7f681c02a280 comp rx=0 tx=0).stop 2024-04-24T23:21:00.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:00.283+0000 7f6836c00640 1 -- 172.21.15.6:0/880885762 shutdown_connections 2024-04-24T23:21:00.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:00.283+0000 7f6836c00640 1 --2- 172.21.15.6:0/880885762 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f68300c81c0 0x7f68301628e0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:21:00.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:00.283+0000 7f6836c00640 1 --2- 172.21.15.6:0/880885762 >> v2:172.21.15.6:3300/0 conn(0x7f68300c7dc0 0x7f68301623a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:21:00.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:00.283+0000 7f6836c00640 1 -- 172.21.15.6:0/880885762 >> 172.21.15.6:0/880885762 conn(0x7f683006b0e0 msgr2=0x7f683006b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:21:00.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:00.284+0000 7f6836c00640 1 -- 172.21.15.6:0/880885762 shutdown_connections 2024-04-24T23:21:00.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:00.285+0000 7f6836c00640 1 -- 172.21.15.6:0/880885762 wait complete. 2024-04-24T23:21:00.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:00.293+0000 7f6836c00640 1 Processor -- start 2024-04-24T23:21:00.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:00.293+0000 7f6836c00640 1 -- start start 2024-04-24T23:21:00.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:00.295+0000 7f6836c00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f68300c7dc0 0x7f683015dab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:21:00.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:00.298+0000 7f6836c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f68300c81c0 0x7f6830143430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:21:00.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:00.298+0000 7f6836c00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6830159110 0x7f68301597c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:21:00.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:00.299+0000 7f6836c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f683006ece0 con 0x7f68300c81c0 2024-04-24T23:21:00.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:00.299+0000 7f6836c00640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f683006ea00 con 0x7f6830159110 2024-04-24T23:21:00.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:00.300+0000 7f6834c00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6830159110 0x7f68301597c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:21:00.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:00.300+0000 7f6834c00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6830159110 0x7f68301597c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:42740/0 (socket says 172.21.15.6:42740) 2024-04-24T23:21:00.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:00.300+0000 7f6834c00640 1 -- 172.21.15.6:0/1602476768 learned_addr learned my addr 172.21.15.6:0/1602476768 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:21:00.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:00.300+0000 7f6836c00640 1 -- 172.21.15.6:0/1602476768 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f683006f200 con 0x7f68300c7dc0 2024-04-24T23:21:00.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:00.300+0000 7f682f400640 1 --2- 172.21.15.6:0/1602476768 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f68300c81c0 0x7f6830143430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:21:00.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:00.301+0000 7f682fe00640 1 --2- 172.21.15.6:0/1602476768 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f68300c7dc0 0x7f683015dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:21:00.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:00.304+0000 7f682f400640 1 -- 172.21.15.6:0/1602476768 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f68300c7dc0 msgr2=0x7f683015dab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:21:00.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:00.304+0000 7f682f400640 1 --2- 172.21.15.6:0/1602476768 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f68300c7dc0 0x7f683015dab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:21:00.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:00.305+0000 7f682f400640 1 -- 172.21.15.6:0/1602476768 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6830159110 msgr2=0x7f68301597c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:21:00.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:00.305+0000 7f682f400640 1 --2- 172.21.15.6:0/1602476768 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6830159110 0x7f68301597c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:21:00.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:00.306+0000 7f682f400640 1 -- 172.21.15.6:0/1602476768 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6830142590 con 0x7f68300c81c0 2024-04-24T23:21:00.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:00.306+0000 7f6834c00640 1 --2- 172.21.15.6:0/1602476768 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6830159110 0x7f68301597c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:21:00.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:00.307+0000 7f682f400640 1 --2- 172.21.15.6:0/1602476768 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f68300c81c0 0x7f6830143430 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f682000e880 tx=0x7f682000ed40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:21:00.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:00.308+0000 7f6827e00640 1 -- 172.21.15.6:0/1602476768 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f682000f040 con 0x7f68300c81c0 2024-04-24T23:21:00.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:00.309+0000 7f6827e00640 1 -- 172.21.15.6:0/1602476768 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6820004230 con 0x7f68300c81c0 2024-04-24T23:21:00.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:00.309+0000 7f6836c00640 1 -- 172.21.15.6:0/1602476768 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6830144410 con 0x7f68300c81c0 2024-04-24T23:21:00.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:00.309+0000 7f6827e00640 1 -- 172.21.15.6:0/1602476768 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f682000c6d0 con 0x7f68300c81c0 2024-04-24T23:21:00.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:00.311+0000 7f6836c00640 1 -- 172.21.15.6:0/1602476768 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f683015ce20 con 0x7f68300c81c0 2024-04-24T23:21:00.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:00.313+0000 7f6836c00640 1 -- 172.21.15.6:0/1602476768 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f68300767b0 con 0x7f68300c81c0 2024-04-24T23:21:00.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:00.313+0000 7f6827e00640 1 -- 172.21.15.6:0/1602476768 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f682000c870 con 0x7f68300c81c0 2024-04-24T23:21:00.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:00.326+0000 7f6827e00640 1 --2- 172.21.15.6:0/1602476768 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f681006be80 0x7f681006e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:21:00.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:00.326+0000 7f6827e00640 1 -- 172.21.15.6:0/1602476768 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6762+0+0 (secure 0 0 0) 0x7f6820056500 con 0x7f68300c81c0 2024-04-24T23:21:00.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:00.327+0000 7f6827e00640 1 -- 172.21.15.6:0/1602476768 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f6820017020 con 0x7f68300c81c0 2024-04-24T23:21:00.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:00.329+0000 7f682fe00640 1 --2- 172.21.15.6:0/1602476768 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f681006be80 0x7f681006e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:21:00.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:00.330+0000 7f682fe00640 1 --2- 172.21.15.6:0/1602476768 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f681006be80 0x7f681006e340 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f6828009f60 tx=0x7f68280094f0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:21:00.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:00.787+0000 7f6836c00640 1 -- 172.21.15.6:0/1602476768 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f683015d020 con 0x7f68300c81c0 2024-04-24T23:21:00.788 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:21:00.789 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:20:17.585709+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4282,"mds_2":4235,"mds_3":4232,"mds_4":5257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4232":{"gid":4232,"name":"b","rank":3,"incarnation":15,"state":"up:active","state_seq":15,"addr":"172.21.15.190:6838/2032280158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2032280158},{"type":"v1","addr":"172.21.15.190:6838","nonce":2032280158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4235":{"gid":4235,"name":"d","rank":2,"incarnation":13,"state":"up:active","state_seq":14,"addr":"172.21.15.190:6839/1572872451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":1572872451},{"type":"v1","addr":"172.21.15.190:6839","nonce":1572872451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4282":{"gid":4282,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":12,"addr":"172.21.15.190:6837/580315269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":580315269},{"type":"v1","addr":"172.21.15.190:6837","nonce":580315269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/489092393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":489092393},{"type":"v1","addr":"172.21.15.6:6835","nonce":489092393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5257":{"gid":5257,"name":"e","rank":4,"incarnation":25,"state":"up:replay","state_seq":1,"addr":"172.21.15.6:6837/3240971655","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":3240971655},{"type":"v1","addr":"172.21.15.6:6837","nonce":3240971655}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4306,"qdb_cluster":[4232,4235,4282,4306]},"id":1}]} 2024-04-24T23:21:00.789 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 25 2024-04-24T23:21:00.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:00.787+0000 7f6827e00640 1 -- 172.21.15.6:0/1602476768 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6080 (secure 0 0 0) 0x7f6820058460 con 0x7f68300c81c0 2024-04-24T23:21:00.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:00.789+0000 7f6836c00640 1 -- 172.21.15.6:0/1602476768 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f681006be80 msgr2=0x7f681006e340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:21:00.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:00.790+0000 7f6836c00640 1 --2- 172.21.15.6:0/1602476768 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f681006be80 0x7f681006e340 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f6828009f60 tx=0x7f68280094f0 comp rx=0 tx=0).stop 2024-04-24T23:21:00.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:00.790+0000 7f6836c00640 1 -- 172.21.15.6:0/1602476768 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f68300c81c0 msgr2=0x7f6830143430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:21:00.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:00.790+0000 7f6836c00640 1 --2- 172.21.15.6:0/1602476768 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f68300c81c0 0x7f6830143430 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f682000e880 tx=0x7f682000ed40 comp rx=0 tx=0).stop 2024-04-24T23:21:00.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:00.790+0000 7f6836c00640 1 -- 172.21.15.6:0/1602476768 shutdown_connections 2024-04-24T23:21:00.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:00.790+0000 7f6836c00640 1 --2- 172.21.15.6:0/1602476768 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f681006be80 0x7f681006e340 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:21:00.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:00.790+0000 7f6836c00640 1 --2- 172.21.15.6:0/1602476768 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6830159110 0x7f68301597c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:21:00.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:00.790+0000 7f6836c00640 1 --2- 172.21.15.6:0/1602476768 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f68300c81c0 0x7f6830143430 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:21:00.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:00.790+0000 7f6836c00640 1 --2- 172.21.15.6:0/1602476768 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f68300c7dc0 0x7f683015dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:21:00.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:00.790+0000 7f6836c00640 1 -- 172.21.15.6:0/1602476768 >> 172.21.15.6:0/1602476768 conn(0x7f683006b0e0 msgr2=0x7f6830155fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:21:00.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:00.790+0000 7f6836c00640 1 -- 172.21.15.6:0/1602476768 shutdown_connections 2024-04-24T23:21:00.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:00.790+0000 7f6836c00640 1 -- 172.21.15.6:0/1602476768 wait complete. 2024-04-24T23:21:00.892 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-24T23:21:02.893 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:21:03.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:03.232+0000 7f1ca9800640 1 Processor -- start 2024-04-24T23:21:03.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:03.232+0000 7f1ca9800640 1 -- start start 2024-04-24T23:21:03.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:03.234+0000 7f1ca9800640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1ca40cc870 0x7f1ca4070e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:21:03.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:03.234+0000 7f1ca9800640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f1ca40ccc50 0x7f1ca415f790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:21:03.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:03.234+0000 7f1ca9800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f1ca4162ac0 0x7f1ca4158a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:21:03.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:03.234+0000 7f1ca9800640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f1ca40713b0 con 0x7f1ca4162ac0 2024-04-24T23:21:03.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:03.234+0000 7f1ca9800640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f1ca4162350 con 0x7f1ca40ccc50 2024-04-24T23:21:03.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:03.234+0000 7f1ca9800640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f1ca40715e0 con 0x7f1ca40cc870 2024-04-24T23:21:03.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:03.235+0000 7f1ca3e00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1ca40cc870 0x7f1ca4070e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:21:03.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:03.235+0000 7f1ca3400640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f1ca40ccc50 0x7f1ca415f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:21:03.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:03.235+0000 7f1ca8e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f1ca4162ac0 0x7f1ca4158a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:21:03.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:03.236+0000 7f1ca8e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f1ca4162ac0 0x7f1ca4158a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45062/0 (socket says 172.21.15.6:45062) 2024-04-24T23:21:03.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:03.236+0000 7f1ca8e00640 1 -- 172.21.15.6:0/399530759 learned_addr learned my addr 172.21.15.6:0/399530759 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:21:03.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:03.236+0000 7f1ca3e00640 1 -- 172.21.15.6:0/399530759 >> v2:172.21.15.190:3300/0 conn(0x7f1ca40ccc50 msgr2=0x7f1ca415f790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:21:03.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:03.236+0000 7f1ca3e00640 1 --2- 172.21.15.6:0/399530759 >> v2:172.21.15.190:3300/0 conn(0x7f1ca40ccc50 0x7f1ca415f790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:21:03.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:03.236+0000 7f1ca3e00640 1 -- 172.21.15.6:0/399530759 >> v2:172.21.15.6:3300/0 conn(0x7f1ca4162ac0 msgr2=0x7f1ca4158a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:21:03.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:03.236+0000 7f1ca3e00640 1 --2- 172.21.15.6:0/399530759 >> v2:172.21.15.6:3300/0 conn(0x7f1ca4162ac0 0x7f1ca4158a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:21:03.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:03.236+0000 7f1ca3e00640 1 -- 172.21.15.6:0/399530759 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ca406dcb0 con 0x7f1ca40cc870 2024-04-24T23:21:03.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:03.238+0000 7f1ca3e00640 1 --2- 172.21.15.6:0/399530759 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1ca40cc870 0x7f1ca4070e70 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f1ca406de80 tx=0x7f1c9402a130 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f46060d3f58006 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:21:03.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:03.239+0000 7f1ca2a00640 1 -- 172.21.15.6:0/399530759 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c94037070 con 0x7f1ca40cc870 2024-04-24T23:21:03.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:03.239+0000 7f1ca2a00640 1 -- 172.21.15.6:0/399530759 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1c94032440 con 0x7f1ca40cc870 2024-04-24T23:21:03.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:03.239+0000 7f1ca2a00640 1 -- 172.21.15.6:0/399530759 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c940327f0 con 0x7f1ca40cc870 2024-04-24T23:21:03.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:03.240+0000 7f1ca9800640 1 -- 172.21.15.6:0/399530759 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1ca40cc870 msgr2=0x7f1ca4070e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:21:03.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:03.240+0000 7f1ca9800640 1 --2- 172.21.15.6:0/399530759 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1ca40cc870 0x7f1ca4070e70 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f1ca406de80 tx=0x7f1c9402a130 comp rx=0 tx=0).stop 2024-04-24T23:21:03.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:03.240+0000 7f1ca9800640 1 -- 172.21.15.6:0/399530759 shutdown_connections 2024-04-24T23:21:03.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:03.240+0000 7f1ca9800640 1 --2- 172.21.15.6:0/399530759 >> v2:172.21.15.6:3300/0 conn(0x7f1ca4162ac0 0x7f1ca4158a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:21:03.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:03.240+0000 7f1ca9800640 1 --2- 172.21.15.6:0/399530759 >> v2:172.21.15.190:3300/0 conn(0x7f1ca40ccc50 0x7f1ca415f790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:21:03.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:03.240+0000 7f1ca9800640 1 --2- 172.21.15.6:0/399530759 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1ca40cc870 0x7f1ca4070e70 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:21:03.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:03.240+0000 7f1ca9800640 1 -- 172.21.15.6:0/399530759 >> 172.21.15.6:0/399530759 conn(0x7f1ca406ed80 msgr2=0x7f1ca406f190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:21:03.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:03.241+0000 7f1ca9800640 1 -- 172.21.15.6:0/399530759 shutdown_connections 2024-04-24T23:21:03.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:03.241+0000 7f1ca9800640 1 -- 172.21.15.6:0/399530759 wait complete. 2024-04-24T23:21:03.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:03.243+0000 7f1ca9800640 1 Processor -- start 2024-04-24T23:21:03.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:03.243+0000 7f1ca9800640 1 -- start start 2024-04-24T23:21:03.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:03.244+0000 7f1ca9800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1ca40cc870 0x7f1ca40cb970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:21:03.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:03.244+0000 7f1ca9800640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1ca40ccc50 0x7f1ca4158820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:21:03.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:03.244+0000 7f1ca9800640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1ca4162ac0 0x7f1ca41f84e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:21:03.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:03.244+0000 7f1ca9800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f1ca40724e0 con 0x7f1ca40cc870 2024-04-24T23:21:03.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:03.244+0000 7f1ca9800640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f1ca415fcd0 con 0x7f1ca40ccc50 2024-04-24T23:21:03.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:03.244+0000 7f1ca9800640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f1ca406df50 con 0x7f1ca4162ac0 2024-04-24T23:21:03.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:03.245+0000 7f1ca3400640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1ca40ccc50 0x7f1ca4158820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:21:03.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:03.245+0000 7f1ca3400640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1ca40ccc50 0x7f1ca4158820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:42754/0 (socket says 172.21.15.6:42754) 2024-04-24T23:21:03.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:03.245+0000 7f1ca3400640 1 -- 172.21.15.6:0/4163352379 learned_addr learned my addr 172.21.15.6:0/4163352379 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:21:03.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:03.245+0000 7f1ca3e00640 1 --2- 172.21.15.6:0/4163352379 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1ca40cc870 0x7f1ca40cb970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:21:03.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:03.245+0000 7f1ca8e00640 1 --2- 172.21.15.6:0/4163352379 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1ca4162ac0 0x7f1ca41f84e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:21:03.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:03.246+0000 7f1ca3400640 1 -- 172.21.15.6:0/4163352379 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1ca4162ac0 msgr2=0x7f1ca41f84e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:21:03.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:03.246+0000 7f1ca3400640 1 --2- 172.21.15.6:0/4163352379 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1ca4162ac0 0x7f1ca41f84e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:21:03.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:03.246+0000 7f1ca3400640 1 -- 172.21.15.6:0/4163352379 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1ca40cc870 msgr2=0x7f1ca40cb970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:21:03.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:03.246+0000 7f1ca3400640 1 --2- 172.21.15.6:0/4163352379 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1ca40cc870 0x7f1ca40cb970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:21:03.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:03.246+0000 7f1ca3400640 1 -- 172.21.15.6:0/4163352379 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ca41e9ee0 con 0x7f1ca40ccc50 2024-04-24T23:21:03.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:03.246+0000 7f1ca3e00640 1 --2- 172.21.15.6:0/4163352379 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1ca40cc870 0x7f1ca40cb970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:21:03.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:03.247+0000 7f1ca3400640 1 --2- 172.21.15.6:0/4163352379 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1ca40ccc50 0x7f1ca4158820 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f1c9000e8c0 tx=0x7f1c9000ed80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:21:03.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:03.247+0000 7f1c9be00640 1 -- 172.21.15.6:0/4163352379 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c90017070 con 0x7f1ca40ccc50 2024-04-24T23:21:03.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:03.247+0000 7f1ca9800640 1 -- 172.21.15.6:0/4163352379 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ca41ea1c0 con 0x7f1ca40ccc50 2024-04-24T23:21:03.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:03.247+0000 7f1c9be00640 1 -- 172.21.15.6:0/4163352379 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1c9000c940 con 0x7f1ca40ccc50 2024-04-24T23:21:03.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:03.247+0000 7f1c9be00640 1 -- 172.21.15.6:0/4163352379 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c9000cd10 con 0x7f1ca40ccc50 2024-04-24T23:21:03.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:03.248+0000 7f1ca9800640 1 -- 172.21.15.6:0/4163352379 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ca41ea780 con 0x7f1ca40ccc50 2024-04-24T23:21:03.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:03.249+0000 7f1ca9800640 1 -- 172.21.15.6:0/4163352379 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ca4075b30 con 0x7f1ca40ccc50 2024-04-24T23:21:03.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:03.251+0000 7f1c9be00640 1 -- 172.21.15.6:0/4163352379 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f1c9001a070 con 0x7f1ca40ccc50 2024-04-24T23:21:03.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:03.252+0000 7f1c9be00640 1 --2- 172.21.15.6:0/4163352379 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f1c8406be30 0x7f1c8406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:21:03.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:03.252+0000 7f1c9be00640 1 -- 172.21.15.6:0/4163352379 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6762+0+0 (secure 0 0 0) 0x7f1c9008b330 con 0x7f1ca40ccc50 2024-04-24T23:21:03.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:03.252+0000 7f1ca3e00640 1 --2- 172.21.15.6:0/4163352379 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f1c8406be30 0x7f1c8406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:21:03.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:03.254+0000 7f1ca3e00640 1 --2- 172.21.15.6:0/4163352379 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f1c8406be30 0x7f1c8406e2f0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f1c94034000 tx=0x7f1c9402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:21:03.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:03.261+0000 7f1c9be00640 1 -- 172.21.15.6:0/4163352379 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f1c90059150 con 0x7f1ca40ccc50 2024-04-24T23:21:03.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:03.754+0000 7f1ca9800640 1 -- 172.21.15.6:0/4163352379 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1ca415c9b0 con 0x7f1ca40ccc50 2024-04-24T23:21:03.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:03.755+0000 7f1c9be00640 1 -- 172.21.15.6:0/4163352379 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6080 (secure 0 0 0) 0x7f1c9005d450 con 0x7f1ca40ccc50 2024-04-24T23:21:03.758 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:21:03.758 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:20:17.585709+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4282,"mds_2":4235,"mds_3":4232,"mds_4":5257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4232":{"gid":4232,"name":"b","rank":3,"incarnation":15,"state":"up:active","state_seq":15,"addr":"172.21.15.190:6838/2032280158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2032280158},{"type":"v1","addr":"172.21.15.190:6838","nonce":2032280158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4235":{"gid":4235,"name":"d","rank":2,"incarnation":13,"state":"up:active","state_seq":14,"addr":"172.21.15.190:6839/1572872451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":1572872451},{"type":"v1","addr":"172.21.15.190:6839","nonce":1572872451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4282":{"gid":4282,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":12,"addr":"172.21.15.190:6837/580315269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":580315269},{"type":"v1","addr":"172.21.15.190:6837","nonce":580315269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/489092393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":489092393},{"type":"v1","addr":"172.21.15.6:6835","nonce":489092393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5257":{"gid":5257,"name":"e","rank":4,"incarnation":25,"state":"up:replay","state_seq":1,"addr":"172.21.15.6:6837/3240971655","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":3240971655},{"type":"v1","addr":"172.21.15.6:6837","nonce":3240971655}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4306,"qdb_cluster":[4232,4235,4282,4306]},"id":1}]} 2024-04-24T23:21:03.758 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 25 2024-04-24T23:21:03.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:03.761+0000 7f1ca9800640 1 -- 172.21.15.6:0/4163352379 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f1c8406be30 msgr2=0x7f1c8406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:21:03.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:03.761+0000 7f1ca9800640 1 --2- 172.21.15.6:0/4163352379 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f1c8406be30 0x7f1c8406e2f0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f1c94034000 tx=0x7f1c9402e000 comp rx=0 tx=0).stop 2024-04-24T23:21:03.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:03.761+0000 7f1ca9800640 1 -- 172.21.15.6:0/4163352379 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1ca40ccc50 msgr2=0x7f1ca4158820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:21:03.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:03.761+0000 7f1ca9800640 1 --2- 172.21.15.6:0/4163352379 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1ca40ccc50 0x7f1ca4158820 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f1c9000e8c0 tx=0x7f1c9000ed80 comp rx=0 tx=0).stop 2024-04-24T23:21:03.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:03.761+0000 7f1ca9800640 1 -- 172.21.15.6:0/4163352379 shutdown_connections 2024-04-24T23:21:03.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:03.761+0000 7f1ca9800640 1 --2- 172.21.15.6:0/4163352379 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f1c8406be30 0x7f1c8406e2f0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:21:03.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:03.761+0000 7f1ca9800640 1 --2- 172.21.15.6:0/4163352379 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1ca4162ac0 0x7f1ca41f84e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:21:03.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:03.761+0000 7f1ca9800640 1 --2- 172.21.15.6:0/4163352379 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1ca40ccc50 0x7f1ca4158820 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:21:03.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:03.761+0000 7f1ca9800640 1 --2- 172.21.15.6:0/4163352379 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1ca40cc870 0x7f1ca40cb970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:21:03.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:03.761+0000 7f1ca9800640 1 -- 172.21.15.6:0/4163352379 >> 172.21.15.6:0/4163352379 conn(0x7f1ca406ed80 msgr2=0x7f1ca4153700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:21:03.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:03.761+0000 7f1ca9800640 1 -- 172.21.15.6:0/4163352379 shutdown_connections 2024-04-24T23:21:03.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:03.761+0000 7f1ca9800640 1 -- 172.21.15.6:0/4163352379 wait complete. 2024-04-24T23:21:03.790 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-24T23:21:05.537 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-24T23:21:05.547 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-24T23:21:05.791 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:21:06.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:06.087+0000 7f0eabe00640 1 Processor -- start 2024-04-24T23:21:06.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:06.087+0000 7f0eabe00640 1 -- start start 2024-04-24T23:21:06.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:06.089+0000 7f0eabe00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f0eac158c50 0x7f0eac0c9a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:21:06.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:06.089+0000 7f0eabe00640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f0eac0c9f70 0x7f0eac162780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:21:06.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:06.089+0000 7f0eabe00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0eac163150 con 0x7f0eac162cc0 2024-04-24T23:21:06.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:06.089+0000 7f0eabe00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f0eac1622f0 con 0x7f0eac158c50 2024-04-24T23:21:06.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:06.089+0000 7f0eabe00640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f0eac15ed00 con 0x7f0eac0c9f70 2024-04-24T23:21:06.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:06.090+0000 7f0eaa000640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f0eac0c9f70 0x7f0eac162780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:21:06.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:06.090+0000 7f0eaa000640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f0eac0c9f70 0x7f0eac162780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:49446/0 (socket says 172.21.15.6:49446) 2024-04-24T23:21:06.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:06.090+0000 7f0eaa000640 1 -- 172.21.15.6:0/3466081812 learned_addr learned my addr 172.21.15.6:0/3466081812 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:21:06.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:06.090+0000 7f0eaaa00640 1 --2- 172.21.15.6:0/3466081812 >> v2:172.21.15.6:3300/0 conn(0x7f0eac158c50 0x7f0eac0c9a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:21:06.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:06.092+0000 7f0eaa000640 1 -- 172.21.15.6:0/3466081812 >> v2:172.21.15.6:3300/0 conn(0x7f0eac158c50 msgr2=0x7f0eac0c9a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:21:06.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:06.092+0000 7f0ea9600640 1 -- 172.21.15.6:0/3466081812 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2547772752 0 0) 0x7f0eac163150 con 0x7f0eac162cc0 2024-04-24T23:21:06.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:06.092+0000 7f0eaa000640 1 --2- 172.21.15.6:0/3466081812 >> v2:172.21.15.6:3300/0 conn(0x7f0eac158c50 0x7f0eac0c9a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:21:06.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:06.092+0000 7f0eaa000640 1 -- 172.21.15.6:0/3466081812 >> v1:172.21.15.6:6789/0 conn(0x7f0eac162cc0 legacy=0x7f0eac072840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:21:06.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:06.092+0000 7f0eaa000640 1 -- 172.21.15.6:0/3466081812 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0eac0ca3b0 con 0x7f0eac0c9f70 2024-04-24T23:21:06.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:06.095+0000 7f0eaaa00640 1 --2- 172.21.15.6:0/3466081812 >> v2:172.21.15.6:3300/0 conn(0x7f0eac158c50 0x7f0eac0c9a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:21:06.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:06.099+0000 7f0eaa000640 1 --2- 172.21.15.6:0/3466081812 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0eac0c9f70 0x7f0eac162780 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f0eac073560 tx=0x7f0e9c02a270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=321fa0f9ac84eb55 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:21:06.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:06.101+0000 7f0ea9600640 1 -- 172.21.15.6:0/3466081812 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e9c037070 con 0x7f0eac0c9f70 2024-04-24T23:21:06.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:06.101+0000 7f0ea9600640 1 -- 172.21.15.6:0/3466081812 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0e9c032440 con 0x7f0eac0c9f70 2024-04-24T23:21:06.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:06.101+0000 7f0ea9600640 1 -- 172.21.15.6:0/3466081812 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e9c032810 con 0x7f0eac0c9f70 2024-04-24T23:21:06.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:06.105+0000 7f0eabe00640 1 -- 172.21.15.6:0/3466081812 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0eac0c9f70 msgr2=0x7f0eac162780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:21:06.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:06.105+0000 7f0eabe00640 1 --2- 172.21.15.6:0/3466081812 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0eac0c9f70 0x7f0eac162780 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f0eac073560 tx=0x7f0e9c02a270 comp rx=0 tx=0).stop 2024-04-24T23:21:06.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:06.107+0000 7f0eabe00640 1 -- 172.21.15.6:0/3466081812 shutdown_connections 2024-04-24T23:21:06.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:06.107+0000 7f0eabe00640 1 --2- 172.21.15.6:0/3466081812 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0eac0c9f70 0x7f0eac162780 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:21:06.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:06.107+0000 7f0eabe00640 1 --2- 172.21.15.6:0/3466081812 >> v2:172.21.15.6:3300/0 conn(0x7f0eac158c50 0x7f0eac0c9a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:21:06.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:06.107+0000 7f0eabe00640 1 -- 172.21.15.6:0/3466081812 >> 172.21.15.6:0/3466081812 conn(0x7f0eac06c1f0 msgr2=0x7f0eac06c5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:21:06.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:06.107+0000 7f0eabe00640 1 -- 172.21.15.6:0/3466081812 shutdown_connections 2024-04-24T23:21:06.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:06.107+0000 7f0eabe00640 1 -- 172.21.15.6:0/3466081812 wait complete. 2024-04-24T23:21:06.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:06.114+0000 7f0eabe00640 1 Processor -- start 2024-04-24T23:21:06.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:06.115+0000 7f0eabe00640 1 -- start start 2024-04-24T23:21:06.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:06.117+0000 7f0eabe00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0eac0c9f70 0x7f0eac06f9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:21:06.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:06.117+0000 7f0eabe00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0eac158c50 0x7f0eac1f82a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:21:06.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:06.117+0000 7f0eabe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0eac162cc0 0x7f0eac1f87e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:21:06.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:06.117+0000 7f0eabe00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f0eac15f540 con 0x7f0eac162cc0 2024-04-24T23:21:06.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:06.117+0000 7f0eabe00640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f0eac15f260 con 0x7f0eac158c50 2024-04-24T23:21:06.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:06.117+0000 7f0eabe00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f0eac15f820 con 0x7f0eac0c9f70 2024-04-24T23:21:06.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:06.118+0000 7f0eaaa00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0eac0c9f70 0x7f0eac06f9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:21:06.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:06.118+0000 7f0eaaa00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0eac0c9f70 0x7f0eac06f9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.6:54654/0 (socket says 172.21.15.6:54654) 2024-04-24T23:21:06.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:06.118+0000 7f0eaaa00640 1 -- 172.21.15.6:0/4292703942 learned_addr learned my addr 172.21.15.6:0/4292703942 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:21:06.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:06.118+0000 7f0eaaa00640 1 -- 172.21.15.6:0/4292703942 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0eac158c50 msgr2=0x7f0eac1f82a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-24T23:21:06.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:06.118+0000 7f0eaaa00640 1 --2- 172.21.15.6:0/4292703942 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0eac158c50 0x7f0eac1f82a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:21:06.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:06.118+0000 7f0eaaa00640 1 -- 172.21.15.6:0/4292703942 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0eac162cc0 msgr2=0x7f0eac1f87e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-24T23:21:06.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:06.118+0000 7f0eaaa00640 1 --2- 172.21.15.6:0/4292703942 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0eac162cc0 0x7f0eac1f87e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:21:06.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:06.118+0000 7f0eaaa00640 1 -- 172.21.15.6:0/4292703942 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0eac161140 con 0x7f0eac0c9f70 2024-04-24T23:21:06.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:06.119+0000 7f0eaaa00640 1 --2- 172.21.15.6:0/4292703942 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0eac0c9f70 0x7f0eac06f9c0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f0e9800c820 tx=0x7f0e9800cc20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:21:06.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:06.119+0000 7f0ea2a00640 1 -- 172.21.15.6:0/4292703942 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e98015070 con 0x7f0eac0c9f70 2024-04-24T23:21:06.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:06.119+0000 7f0ea2a00640 1 -- 172.21.15.6:0/4292703942 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0e98007a30 con 0x7f0eac0c9f70 2024-04-24T23:21:06.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:06.119+0000 7f0ea2a00640 1 -- 172.21.15.6:0/4292703942 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e98007de0 con 0x7f0eac0c9f70 2024-04-24T23:21:06.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:06.122+0000 7f0eabe00640 1 -- 172.21.15.6:0/4292703942 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0eac15fec0 con 0x7f0eac0c9f70 2024-04-24T23:21:06.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:06.123+0000 7f0eabe00640 1 -- 172.21.15.6:0/4292703942 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f0eac156080 con 0x7f0eac0c9f70 2024-04-24T23:21:06.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:06.125+0000 7f0eabe00640 1 -- 172.21.15.6:0/4292703942 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0eac075b30 con 0x7f0eac0c9f70 2024-04-24T23:21:06.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:06.125+0000 7f0ea2a00640 1 -- 172.21.15.6:0/4292703942 <== mon.2 v2:172.21.15.190:3301/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f0e98018070 con 0x7f0eac0c9f70 2024-04-24T23:21:06.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:06.127+0000 7f0ea2a00640 1 --2- 172.21.15.6:0/4292703942 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f0e8c08de90 0x7f0e8c090350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:21:06.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:06.127+0000 7f0ea2a00640 1 -- 172.21.15.6:0/4292703942 <== mon.2 v2:172.21.15.190:3301/0 5 ==== osd_map(28..28 src has 1..28) ==== 6762+0+0 (secure 0 0 0) 0x7f0e9808b230 con 0x7f0eac0c9f70 2024-04-24T23:21:06.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:06.130+0000 7f0eaa000640 1 --2- 172.21.15.6:0/4292703942 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f0e8c08de90 0x7f0e8c090350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:21:06.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:06.132+0000 7f0eaa000640 1 --2- 172.21.15.6:0/4292703942 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f0e8c08de90 0x7f0e8c090350 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f0e9c034000 tx=0x7f0e9c02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:21:06.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:06.132+0000 7f0ea2a00640 1 -- 172.21.15.6:0/4292703942 <== mon.2 v2:172.21.15.190:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f0e98059810 con 0x7f0eac0c9f70 2024-04-24T23:21:06.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:06.601+0000 7f0eabe00640 1 -- 172.21.15.6:0/4292703942 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0eac153580 con 0x7f0eac0c9f70 2024-04-24T23:21:06.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:06.602+0000 7f0ea2a00640 1 -- 172.21.15.6:0/4292703942 <== mon.2 v2:172.21.15.190:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6080 (secure 0 0 0) 0x7f0e98059110 con 0x7f0eac0c9f70 2024-04-24T23:21:06.607 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:21:06.607 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:20:17.585709+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4282,"mds_2":4235,"mds_3":4232,"mds_4":5257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4232":{"gid":4232,"name":"b","rank":3,"incarnation":15,"state":"up:active","state_seq":15,"addr":"172.21.15.190:6838/2032280158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2032280158},{"type":"v1","addr":"172.21.15.190:6838","nonce":2032280158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4235":{"gid":4235,"name":"d","rank":2,"incarnation":13,"state":"up:active","state_seq":14,"addr":"172.21.15.190:6839/1572872451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":1572872451},{"type":"v1","addr":"172.21.15.190:6839","nonce":1572872451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4282":{"gid":4282,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":12,"addr":"172.21.15.190:6837/580315269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":580315269},{"type":"v1","addr":"172.21.15.190:6837","nonce":580315269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/489092393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":489092393},{"type":"v1","addr":"172.21.15.6:6835","nonce":489092393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5257":{"gid":5257,"name":"e","rank":4,"incarnation":25,"state":"up:replay","state_seq":1,"addr":"172.21.15.6:6837/3240971655","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":3240971655},{"type":"v1","addr":"172.21.15.6:6837","nonce":3240971655}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4306,"qdb_cluster":[4232,4235,4282,4306]},"id":1}]} 2024-04-24T23:21:06.607 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 25 2024-04-24T23:21:06.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:06.611+0000 7f0eabe00640 1 -- 172.21.15.6:0/4292703942 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f0e8c08de90 msgr2=0x7f0e8c090350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:21:06.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:06.611+0000 7f0eabe00640 1 --2- 172.21.15.6:0/4292703942 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f0e8c08de90 0x7f0e8c090350 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f0e9c034000 tx=0x7f0e9c02e000 comp rx=0 tx=0).stop 2024-04-24T23:21:06.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:06.611+0000 7f0eabe00640 1 -- 172.21.15.6:0/4292703942 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0eac0c9f70 msgr2=0x7f0eac06f9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:21:06.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:06.611+0000 7f0eabe00640 1 --2- 172.21.15.6:0/4292703942 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0eac0c9f70 0x7f0eac06f9c0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f0e9800c820 tx=0x7f0e9800cc20 comp rx=0 tx=0).stop 2024-04-24T23:21:06.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:06.611+0000 7f0eabe00640 1 -- 172.21.15.6:0/4292703942 shutdown_connections 2024-04-24T23:21:06.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:06.611+0000 7f0eabe00640 1 --2- 172.21.15.6:0/4292703942 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f0e8c08de90 0x7f0e8c090350 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:21:06.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:06.611+0000 7f0eabe00640 1 --2- 172.21.15.6:0/4292703942 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0eac162cc0 0x7f0eac1f87e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:21:06.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:06.611+0000 7f0eabe00640 1 --2- 172.21.15.6:0/4292703942 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0eac158c50 0x7f0eac1f82a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:21:06.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:06.611+0000 7f0eabe00640 1 --2- 172.21.15.6:0/4292703942 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0eac0c9f70 0x7f0eac06f9c0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:21:06.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:06.611+0000 7f0eabe00640 1 -- 172.21.15.6:0/4292703942 >> 172.21.15.6:0/4292703942 conn(0x7f0eac06c1f0 msgr2=0x7f0eac06c5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:21:06.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:06.611+0000 7f0eabe00640 1 -- 172.21.15.6:0/4292703942 shutdown_connections 2024-04-24T23:21:06.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:06.611+0000 7f0eabe00640 1 -- 172.21.15.6:0/4292703942 wait complete. 2024-04-24T23:21:06.639 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 25, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-24T23:15:36.400716+0000", "damaged": [], "data_pools": [ 3, 4 ], "enabled": true, "epoch": 25, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4232": { "addr": "172.21.15.190:6838/2032280158", "addrs": { "addrvec": [ { "addr": "172.21.15.190:6835", "nonce": 2032280158, "type": "v2" }, { "addr": "172.21.15.190:6838", "nonce": 2032280158, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4232, "incarnation": 15, "join_fscid": -1, "name": "b", "rank": 3, "state": "up:active", "state_seq": 15 }, "gid_4235": { "addr": "172.21.15.190:6839/1572872451", "addrs": { "addrvec": [ { "addr": "172.21.15.190:6836", "nonce": 1572872451, "type": "v2" }, { "addr": "172.21.15.190:6839", "nonce": 1572872451, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4235, "incarnation": 13, "join_fscid": -1, "name": "d", "rank": 2, "state": "up:active", "state_seq": 14 }, "gid_4282": { "addr": "172.21.15.190:6837/580315269", "addrs": { "addrvec": [ { "addr": "172.21.15.190:6834", "nonce": 580315269, "type": "v2" }, { "addr": "172.21.15.190:6837", "nonce": 580315269, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4282, "incarnation": 9, "join_fscid": -1, "name": "f", "rank": 1, "state": "up:active", "state_seq": 12 }, "gid_4306": { "addr": "172.21.15.6:6835/489092393", "addrs": { "addrvec": [ { "addr": "172.21.15.6:6833", "nonce": 489092393, "type": "v2" }, { "addr": "172.21.15.6:6835", "nonce": 489092393, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4306, "incarnation": 4, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 4 }, "gid_5257": { "addr": "172.21.15.6:6837/3240971655", "addrs": { "addrvec": [ { "addr": "172.21.15.6:6836", "nonce": 3240971655, "type": "v2" }, { "addr": "172.21.15.6:6837", "nonce": 3240971655, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5257, "incarnation": 25, "join_fscid": -1, "name": "e", "rank": 4, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 28, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-04-24T23:20:17.585709+0000", "qdb_cluster": [ 4232, 4235, 4282, 4306 ], "qdb_leader": 4306, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4306, "mds_1": 4282, "mds_2": 4235, "mds_3": 4232, "mds_4": 5257 } } } ], "standbys": [] } 2024-04-24T23:21:08.640 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:21:08.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:08.964+0000 7f8f57200640 1 Processor -- start 2024-04-24T23:21:08.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:08.964+0000 7f8f57200640 1 -- start start 2024-04-24T23:21:08.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:08.967+0000 7f8f57200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f8f50050e80 0x7f8f500de350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:21:08.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:08.968+0000 7f8f57200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8f5004b6e0 con 0x7f8f500d4c00 2024-04-24T23:21:08.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:08.968+0000 7f8f57200640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8f5004b180 con 0x7f8f500076c0 2024-04-24T23:21:08.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:08.968+0000 7f8f57200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f8f50007230 con 0x7f8f50050e80 2024-04-24T23:21:08.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:08.969+0000 7f8f55400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f8f50050e80 0x7f8f500de350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:21:08.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:08.969+0000 7f8f5d000640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f8f500076c0 0x7f8f500de890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.6:34908/0 (socket says 172.21.15.6:34908) 2024-04-24T23:21:08.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:08.969+0000 7f8f5d000640 1 -- 172.21.15.6:0/2127328849 learned_addr learned my addr 172.21.15.6:0/2127328849 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:21:08.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:08.971+0000 7f8f55400640 1 -- 172.21.15.6:0/2127328849 >> v1:172.21.15.190:6789/0 conn(0x7f8f500076c0 legacy=0x7f8f500de890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:21:08.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:08.971+0000 7f8f54a00640 1 -- 172.21.15.6:0/2127328849 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1935048983 0 0) 0x7f8f48001110 con 0x7f8f500076c0 2024-04-24T23:21:08.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:08.971+0000 7f8f55400640 1 -- 172.21.15.6:0/2127328849 >> v1:172.21.15.6:6789/0 conn(0x7f8f500d4c00 legacy=0x7f8f50045300 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-24T23:21:08.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:08.971+0000 7f8f55400640 1 -- 172.21.15.6:0/2127328849 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f5004b400 con 0x7f8f50050e80 2024-04-24T23:21:08.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:08.977+0000 7f8f55400640 1 --2- 172.21.15.6:0/2127328849 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8f50050e80 0x7f8f500de350 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f8f5802d700 tx=0x7f8f580a6ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=510e8a21dee4ad92 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:21:08.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:08.978+0000 7f8f54a00640 1 -- 172.21.15.6:0/2127328849 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f580aa8f0 con 0x7f8f50050e80 2024-04-24T23:21:08.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:08.978+0000 7f8f54a00640 1 -- 172.21.15.6:0/2127328849 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f580aaa90 con 0x7f8f50050e80 2024-04-24T23:21:08.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:08.979+0000 7f8f54a00640 1 -- 172.21.15.6:0/2127328849 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f580aa8f0 con 0x7f8f50050e80 2024-04-24T23:21:08.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:08.979+0000 7f8f57200640 1 -- 172.21.15.6:0/2127328849 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8f50050e80 msgr2=0x7f8f500de350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:21:08.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:08.979+0000 7f8f57200640 1 --2- 172.21.15.6:0/2127328849 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8f50050e80 0x7f8f500de350 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f8f5802d700 tx=0x7f8f580a6ab0 comp rx=0 tx=0).stop 2024-04-24T23:21:08.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:08.980+0000 7f8f57200640 1 -- 172.21.15.6:0/2127328849 shutdown_connections 2024-04-24T23:21:08.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:08.980+0000 7f8f57200640 1 --2- 172.21.15.6:0/2127328849 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8f50050e80 0x7f8f500de350 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:21:08.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:08.980+0000 7f8f57200640 1 -- 172.21.15.6:0/2127328849 >> 172.21.15.6:0/2127328849 conn(0x7f8f50050360 msgr2=0x7f8f50050770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:21:08.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:08.981+0000 7f8f57200640 1 -- 172.21.15.6:0/2127328849 shutdown_connections 2024-04-24T23:21:08.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:08.981+0000 7f8f57200640 1 -- 172.21.15.6:0/2127328849 wait complete. 2024-04-24T23:21:08.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:08.985+0000 7f8f57200640 1 Processor -- start 2024-04-24T23:21:08.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:08.986+0000 7f8f57200640 1 -- start start 2024-04-24T23:21:08.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:08.990+0000 7f8f57200640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8f500076c0 0x7f8f501794b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:21:08.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:08.991+0000 7f8f57200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8f50050e80 0x7f8f5017d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:21:08.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:08.991+0000 7f8f57200640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8f500d4c00 0x7f8f500493c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:21:08.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:08.992+0000 7f8f57200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f8f500d4fe0 con 0x7f8f50050e80 2024-04-24T23:21:08.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:08.992+0000 7f8f57200640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f8f50006ec0 con 0x7f8f500d4c00 2024-04-24T23:21:08.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:08.992+0000 7f8f57200640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f8f500d5160 con 0x7f8f500076c0 2024-04-24T23:21:08.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:08.992+0000 7f8f55e00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8f500076c0 0x7f8f501794b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:21:08.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:08.992+0000 7f8f55e00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8f500076c0 0x7f8f501794b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.6:54664/0 (socket says 172.21.15.6:54664) 2024-04-24T23:21:08.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:08.992+0000 7f8f5d000640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8f500d4c00 0x7f8f500493c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:21:08.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:08.993+0000 7f8f5d000640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8f500d4c00 0x7f8f500493c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:49458/0 (socket says 172.21.15.6:49458) 2024-04-24T23:21:08.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:08.993+0000 7f8f5d000640 1 -- 172.21.15.6:0/2258671003 learned_addr learned my addr 172.21.15.6:0/2258671003 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:21:08.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:08.992+0000 7f8f55400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8f50050e80 0x7f8f5017d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:21:09.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:08.994+0000 7f8f55e00640 1 -- 172.21.15.6:0/2258671003 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8f500d4c00 msgr2=0x7f8f500493c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:21:09.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:08.994+0000 7f8f55e00640 1 --2- 172.21.15.6:0/2258671003 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8f500d4c00 0x7f8f500493c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:21:09.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:08.994+0000 7f8f55e00640 1 -- 172.21.15.6:0/2258671003 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8f50050e80 msgr2=0x7f8f5017d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:21:09.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:08.994+0000 7f8f55e00640 1 --2- 172.21.15.6:0/2258671003 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8f50050e80 0x7f8f5017d470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:21:09.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:08.994+0000 7f8f55e00640 1 -- 172.21.15.6:0/2258671003 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f500eb810 con 0x7f8f500076c0 2024-04-24T23:21:09.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:08.996+0000 7f8f55e00640 1 --2- 172.21.15.6:0/2258671003 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8f500076c0 0x7f8f501794b0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f8f44009fd0 tx=0x7f8f4400b870 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:21:09.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:08.998+0000 7f8f4e000640 1 -- 172.21.15.6:0/2258671003 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f44015020 con 0x7f8f500076c0 2024-04-24T23:21:09.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:08.998+0000 7f8f57200640 1 -- 172.21.15.6:0/2258671003 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f500ebab0 con 0x7f8f500076c0 2024-04-24T23:21:09.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:08.998+0000 7f8f57200640 1 -- 172.21.15.6:0/2258671003 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f5004a6d0 con 0x7f8f500076c0 2024-04-24T23:21:09.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:08.999+0000 7f8f4e000640 1 -- 172.21.15.6:0/2258671003 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f44011830 con 0x7f8f500076c0 2024-04-24T23:21:09.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:08.999+0000 7f8f4e000640 1 -- 172.21.15.6:0/2258671003 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f44015020 con 0x7f8f500076c0 2024-04-24T23:21:09.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:09.004+0000 7f8f43400640 1 -- 172.21.15.6:0/2258671003 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8f2c005290 con 0x7f8f500076c0 2024-04-24T23:21:09.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:09.013+0000 7f8f4e000640 1 -- 172.21.15.6:0/2258671003 <== mon.2 v2:172.21.15.190:3301/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f8f44018020 con 0x7f8f500076c0 2024-04-24T23:21:09.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:09.017+0000 7f8f4e000640 1 --2- 172.21.15.6:0/2258671003 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f8f3806c030 0x7f8f3806e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:21:09.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:09.017+0000 7f8f4e000640 1 -- 172.21.15.6:0/2258671003 <== mon.2 v2:172.21.15.190:3301/0 5 ==== osd_map(28..28 src has 1..28) ==== 6762+0+0 (secure 0 0 0) 0x7f8f4408ae00 con 0x7f8f500076c0 2024-04-24T23:21:09.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:09.025+0000 7f8f4e000640 1 -- 172.21.15.6:0/2258671003 <== mon.2 v2:172.21.15.190:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f8f4401f020 con 0x7f8f500076c0 2024-04-24T23:21:09.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:09.026+0000 7f8f55400640 1 --2- 172.21.15.6:0/2258671003 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f8f3806c030 0x7f8f3806e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:21:09.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:09.027+0000 7f8f55400640 1 --2- 172.21.15.6:0/2258671003 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f8f3806c030 0x7f8f3806e4f0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f8f5000dd40 tx=0x7f8f580ac000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:21:09.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:09.538+0000 7f8f43400640 1 -- 172.21.15.6:0/2258671003 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8f2c005580 con 0x7f8f500076c0 2024-04-24T23:21:09.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:09.542+0000 7f8f4e000640 1 -- 172.21.15.6:0/2258671003 <== mon.2 v2:172.21.15.190:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6082 (secure 0 0 0) 0x7f8f44058eb0 con 0x7f8f500076c0 2024-04-24T23:21:09.545 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:21:09.545 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:21:08.763908+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4282,"mds_2":4235,"mds_3":4232,"mds_4":5257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4232":{"gid":4232,"name":"b","rank":3,"incarnation":15,"state":"up:active","state_seq":15,"addr":"172.21.15.190:6838/2032280158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2032280158},{"type":"v1","addr":"172.21.15.190:6838","nonce":2032280158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4235":{"gid":4235,"name":"d","rank":2,"incarnation":13,"state":"up:active","state_seq":14,"addr":"172.21.15.190:6839/1572872451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":1572872451},{"type":"v1","addr":"172.21.15.190:6839","nonce":1572872451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4282":{"gid":4282,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":12,"addr":"172.21.15.190:6837/580315269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":580315269},{"type":"v1","addr":"172.21.15.190:6837","nonce":580315269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/489092393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":489092393},{"type":"v1","addr":"172.21.15.6:6835","nonce":489092393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5257":{"gid":5257,"name":"e","rank":4,"incarnation":25,"state":"up:resolve","state_seq":36,"addr":"172.21.15.6:6837/3240971655","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":3240971655},{"type":"v1","addr":"172.21.15.6:6837","nonce":3240971655}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4306,"qdb_cluster":[4306,4282,4235,4232]},"id":1}]} 2024-04-24T23:21:09.545 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 26 2024-04-24T23:21:09.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:09.548+0000 7f8f43400640 1 -- 172.21.15.6:0/2258671003 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f8f3806c030 msgr2=0x7f8f3806e4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:21:09.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:09.548+0000 7f8f43400640 1 --2- 172.21.15.6:0/2258671003 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f8f3806c030 0x7f8f3806e4f0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f8f5000dd40 tx=0x7f8f580ac000 comp rx=0 tx=0).stop 2024-04-24T23:21:09.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:09.549+0000 7f8f43400640 1 -- 172.21.15.6:0/2258671003 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8f500076c0 msgr2=0x7f8f501794b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:21:09.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:09.549+0000 7f8f43400640 1 --2- 172.21.15.6:0/2258671003 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8f500076c0 0x7f8f501794b0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f8f44009fd0 tx=0x7f8f4400b870 comp rx=0 tx=0).stop 2024-04-24T23:21:09.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:09.549+0000 7f8f43400640 1 -- 172.21.15.6:0/2258671003 shutdown_connections 2024-04-24T23:21:09.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:09.549+0000 7f8f43400640 1 --2- 172.21.15.6:0/2258671003 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f8f3806c030 0x7f8f3806e4f0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:21:09.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:09.549+0000 7f8f43400640 1 --2- 172.21.15.6:0/2258671003 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8f500d4c00 0x7f8f500493c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:21:09.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:09.550+0000 7f8f43400640 1 --2- 172.21.15.6:0/2258671003 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8f50050e80 0x7f8f5017d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:21:09.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:09.550+0000 7f8f43400640 1 --2- 172.21.15.6:0/2258671003 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8f500076c0 0x7f8f501794b0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:21:09.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:09.550+0000 7f8f43400640 1 -- 172.21.15.6:0/2258671003 >> 172.21.15.6:0/2258671003 conn(0x7f8f50050360 msgr2=0x7f8f50045300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:21:09.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:09.550+0000 7f8f43400640 1 -- 172.21.15.6:0/2258671003 shutdown_connections 2024-04-24T23:21:09.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:09.551+0000 7f8f43400640 1 -- 172.21.15.6:0/2258671003 wait complete. 2024-04-24T23:21:09.598 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-24T23:21:11.599 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:21:11.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:11.924+0000 7f4302400640 1 Processor -- start 2024-04-24T23:21:11.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:11.924+0000 7f4302400640 1 -- start start 2024-04-24T23:21:11.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:11.927+0000 7f4302400640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f42fc0c7dc0 0x7f42fc1623a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:21:11.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:11.927+0000 7f4302400640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42fc0c7230 con 0x7f42fc0c81c0 2024-04-24T23:21:11.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:11.927+0000 7f4302400640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42fc0c74d0 con 0x7f42fc159110 2024-04-24T23:21:11.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:11.927+0000 7f4302400640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f42fc1594f0 con 0x7f42fc0c7dc0 2024-04-24T23:21:11.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:11.927+0000 7f42faa00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f42fc0c7dc0 0x7f42fc1623a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:21:11.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:11.928+0000 7f42faa00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f42fc0c7dc0 0x7f42fc1623a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.6:54668/0 (socket says 172.21.15.6:54668) 2024-04-24T23:21:11.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:11.928+0000 7f42faa00640 1 -- 172.21.15.6:0/3352788969 learned_addr learned my addr 172.21.15.6:0/3352788969 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:21:11.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:11.928+0000 7f42fa000640 1 -- 172.21.15.6:0/3352788969 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2646216263 0 0) 0x7f42fc0c74d0 con 0x7f42fc159110 2024-04-24T23:21:11.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:11.928+0000 7f42fa000640 1 -- 172.21.15.6:0/3352788969 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f42e0003520 con 0x7f42fc159110 2024-04-24T23:21:11.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:11.929+0000 7f42faa00640 1 -- 172.21.15.6:0/3352788969 >> v1:172.21.15.190:6789/0 conn(0x7f42fc159110 legacy=0x7f42fc072420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:21:11.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:11.929+0000 7f42faa00640 1 -- 172.21.15.6:0/3352788969 >> v1:172.21.15.6:6789/0 conn(0x7f42fc0c81c0 legacy=0x7f42fc1628e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:21:11.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:11.929+0000 7f42fa000640 1 -- 172.21.15.6:0/3352788969 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 860631576 0 0) 0x7f42fc0c7230 con 0x7f42fc0c81c0 2024-04-24T23:21:11.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:11.929+0000 7f42faa00640 1 -- 172.21.15.6:0/3352788969 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42fc06e440 con 0x7f42fc0c7dc0 2024-04-24T23:21:11.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:11.932+0000 7f42faa00640 1 --2- 172.21.15.6:0/3352788969 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f42fc0c7dc0 0x7f42fc1623a0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f42ec0098a0 tx=0x7f42ec02a370 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f27ac4baea2c59ea server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:21:11.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:11.932+0000 7f42fa000640 1 -- 172.21.15.6:0/3352788969 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42ec037070 con 0x7f42fc0c7dc0 2024-04-24T23:21:11.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:11.933+0000 7f42fa000640 1 -- 172.21.15.6:0/3352788969 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f42ec032640 con 0x7f42fc0c7dc0 2024-04-24T23:21:11.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:11.933+0000 7f42fa000640 1 -- 172.21.15.6:0/3352788969 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42ec032a10 con 0x7f42fc0c7dc0 2024-04-24T23:21:11.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:11.934+0000 7f4302400640 1 -- 172.21.15.6:0/3352788969 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f42fc0c7dc0 msgr2=0x7f42fc1623a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:21:11.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:11.935+0000 7f4302400640 1 --2- 172.21.15.6:0/3352788969 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f42fc0c7dc0 0x7f42fc1623a0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f42ec0098a0 tx=0x7f42ec02a370 comp rx=0 tx=0).stop 2024-04-24T23:21:11.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:11.935+0000 7f4302400640 1 -- 172.21.15.6:0/3352788969 shutdown_connections 2024-04-24T23:21:11.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:11.935+0000 7f4302400640 1 --2- 172.21.15.6:0/3352788969 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f42fc0c7dc0 0x7f42fc1623a0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:21:11.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:11.935+0000 7f4302400640 1 -- 172.21.15.6:0/3352788969 >> 172.21.15.6:0/3352788969 conn(0x7f42fc06b0e0 msgr2=0x7f42fc06b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:21:11.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:11.940+0000 7f4302400640 1 -- 172.21.15.6:0/3352788969 shutdown_connections 2024-04-24T23:21:11.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:11.940+0000 7f4302400640 1 -- 172.21.15.6:0/3352788969 wait complete. 2024-04-24T23:21:11.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:11.944+0000 7f4302400640 1 Processor -- start 2024-04-24T23:21:11.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:11.944+0000 7f4302400640 1 -- start start 2024-04-24T23:21:11.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:11.945+0000 7f4302400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f42fc0c7dc0 0x7f42fc12d700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:21:11.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:11.945+0000 7f4302400640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f42fc0c81c0 0x7f42fc1f0ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:21:11.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:11.945+0000 7f4302400640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f42fc159110 0x7f42fc15cc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:21:11.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:11.945+0000 7f4302400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f42fc06ece0 con 0x7f42fc0c7dc0 2024-04-24T23:21:11.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:11.945+0000 7f4302400640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f42fc06ea00 con 0x7f42fc159110 2024-04-24T23:21:11.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:11.945+0000 7f4302400640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f42fc06f200 con 0x7f42fc0c81c0 2024-04-24T23:21:11.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:11.946+0000 7f42fbe00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f42fc159110 0x7f42fc15cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:21:11.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:11.946+0000 7f42fbe00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f42fc159110 0x7f42fc15cc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:49462/0 (socket says 172.21.15.6:49462) 2024-04-24T23:21:11.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:11.946+0000 7f42fbe00640 1 -- 172.21.15.6:0/1421607615 learned_addr learned my addr 172.21.15.6:0/1421607615 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:21:11.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:11.947+0000 7f42fb400640 1 --2- 172.21.15.6:0/1421607615 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f42fc0c7dc0 0x7f42fc12d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:21:11.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:11.947+0000 7f42fb400640 1 -- 172.21.15.6:0/1421607615 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f42fc0c81c0 msgr2=0x7f42fc1f0ae0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-24T23:21:11.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:11.947+0000 7f42fb400640 1 --2- 172.21.15.6:0/1421607615 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f42fc0c81c0 0x7f42fc1f0ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:21:11.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:11.947+0000 7f42fb400640 1 -- 172.21.15.6:0/1421607615 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f42fc159110 msgr2=0x7f42fc15cc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:21:11.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:11.947+0000 7f42fb400640 1 --2- 172.21.15.6:0/1421607615 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f42fc159110 0x7f42fc15cc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:21:11.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:11.947+0000 7f42fb400640 1 -- 172.21.15.6:0/1421607615 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42fc0c4c60 con 0x7f42fc0c7dc0 2024-04-24T23:21:11.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:11.948+0000 7f42fb400640 1 --2- 172.21.15.6:0/1421607615 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f42fc0c7dc0 0x7f42fc12d700 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f42f400c8d0 tx=0x7f42f400cd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:21:11.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:11.948+0000 7f42f3400640 1 -- 172.21.15.6:0/1421607615 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42f4003b70 con 0x7f42fc0c7dc0 2024-04-24T23:21:11.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:11.948+0000 7f42f3400640 1 -- 172.21.15.6:0/1421607615 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f42f4003d10 con 0x7f42fc0c7dc0 2024-04-24T23:21:11.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:11.948+0000 7f42f3400640 1 -- 172.21.15.6:0/1421607615 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42f4005310 con 0x7f42fc0c7dc0 2024-04-24T23:21:11.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:11.948+0000 7f4302400640 1 -- 172.21.15.6:0/1421607615 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42fc2015f0 con 0x7f42fc0c7dc0 2024-04-24T23:21:11.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:11.951+0000 7f4302400640 1 -- 172.21.15.6:0/1421607615 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42fc200830 con 0x7f42fc0c7dc0 2024-04-24T23:21:11.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:11.952+0000 7f42f3400640 1 -- 172.21.15.6:0/1421607615 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f42f40054b0 con 0x7f42fc0c7dc0 2024-04-24T23:21:11.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:11.953+0000 7f4302400640 1 -- 172.21.15.6:0/1421607615 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f42fc0c74d0 con 0x7f42fc0c7dc0 2024-04-24T23:21:11.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:11.955+0000 7f42f3400640 1 --2- 172.21.15.6:0/1421607615 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f42d806be80 0x7f42d806e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:21:11.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:11.955+0000 7f42faa00640 1 --2- 172.21.15.6:0/1421607615 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f42d806be80 0x7f42d806e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:21:11.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:11.955+0000 7f42f3400640 1 -- 172.21.15.6:0/1421607615 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6762+0+0 (secure 0 0 0) 0x7f42f408a8c0 con 0x7f42fc0c7dc0 2024-04-24T23:21:11.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:11.959+0000 7f42faa00640 1 --2- 172.21.15.6:0/1421607615 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f42d806be80 0x7f42d806e340 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f42ec034000 tx=0x7f42ec02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:21:11.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:11.967+0000 7f42f3400640 1 -- 172.21.15.6:0/1421607615 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f42fc0c74d0 con 0x7f42fc0c7dc0 2024-04-24T23:21:12.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:12.494+0000 7f4302400640 1 -- 172.21.15.6:0/1421607615 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f42fc0767b0 con 0x7f42fc0c7dc0 2024-04-24T23:21:12.502 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:21:12.502 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:21:08.763908+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4282,"mds_2":4235,"mds_3":4232,"mds_4":5257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4232":{"gid":4232,"name":"b","rank":3,"incarnation":15,"state":"up:active","state_seq":15,"addr":"172.21.15.190:6838/2032280158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2032280158},{"type":"v1","addr":"172.21.15.190:6838","nonce":2032280158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4235":{"gid":4235,"name":"d","rank":2,"incarnation":13,"state":"up:active","state_seq":14,"addr":"172.21.15.190:6839/1572872451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":1572872451},{"type":"v1","addr":"172.21.15.190:6839","nonce":1572872451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4282":{"gid":4282,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":12,"addr":"172.21.15.190:6837/580315269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":580315269},{"type":"v1","addr":"172.21.15.190:6837","nonce":580315269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/489092393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":489092393},{"type":"v1","addr":"172.21.15.6:6835","nonce":489092393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5257":{"gid":5257,"name":"e","rank":4,"incarnation":25,"state":"up:resolve","state_seq":36,"addr":"172.21.15.6:6837/3240971655","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":3240971655},{"type":"v1","addr":"172.21.15.6:6837","nonce":3240971655}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4306,"qdb_cluster":[4306,4282,4235,4232]},"id":1}]} 2024-04-24T23:21:12.503 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 26 2024-04-24T23:21:12.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:12.499+0000 7f42f3400640 1 -- 172.21.15.6:0/1421607615 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6082 (secure 0 0 0) 0x7f42f4058e40 con 0x7f42fc0c7dc0 2024-04-24T23:21:12.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:12.505+0000 7f4302400640 1 -- 172.21.15.6:0/1421607615 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f42d806be80 msgr2=0x7f42d806e340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:21:12.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:12.505+0000 7f4302400640 1 --2- 172.21.15.6:0/1421607615 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f42d806be80 0x7f42d806e340 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f42ec034000 tx=0x7f42ec02e000 comp rx=0 tx=0).stop 2024-04-24T23:21:12.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:12.505+0000 7f4302400640 1 -- 172.21.15.6:0/1421607615 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f42fc0c7dc0 msgr2=0x7f42fc12d700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:21:12.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:12.505+0000 7f4302400640 1 --2- 172.21.15.6:0/1421607615 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f42fc0c7dc0 0x7f42fc12d700 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f42f400c8d0 tx=0x7f42f400cd90 comp rx=0 tx=0).stop 2024-04-24T23:21:12.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:12.506+0000 7f4302400640 1 -- 172.21.15.6:0/1421607615 shutdown_connections 2024-04-24T23:21:12.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:12.506+0000 7f4302400640 1 --2- 172.21.15.6:0/1421607615 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f42d806be80 0x7f42d806e340 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:21:12.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:12.506+0000 7f4302400640 1 --2- 172.21.15.6:0/1421607615 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f42fc0c81c0 0x7f42fc1f0ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:21:12.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:12.506+0000 7f4302400640 1 --2- 172.21.15.6:0/1421607615 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f42fc159110 0x7f42fc15cc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:21:12.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:12.506+0000 7f4302400640 1 --2- 172.21.15.6:0/1421607615 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f42fc0c7dc0 0x7f42fc12d700 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:21:12.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:12.506+0000 7f4302400640 1 -- 172.21.15.6:0/1421607615 >> 172.21.15.6:0/1421607615 conn(0x7f42fc06b0e0 msgr2=0x7f42fc157440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:21:12.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:12.507+0000 7f4302400640 1 -- 172.21.15.6:0/1421607615 shutdown_connections 2024-04-24T23:21:12.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:12.507+0000 7f4302400640 1 -- 172.21.15.6:0/1421607615 wait complete. 2024-04-24T23:21:12.536 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-24T23:21:14.536 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:21:14.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:14.927+0000 7f8613200640 1 Processor -- start 2024-04-24T23:21:14.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:14.927+0000 7f8613200640 1 -- start start 2024-04-24T23:21:14.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:14.929+0000 7f8613200640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f860c067940 0x7f860c06c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:21:14.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:14.929+0000 7f8613200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f860c15b930 con 0x7f860c154c80 2024-04-24T23:21:14.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:14.930+0000 7f8613200640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f860c0c5500 con 0x7f860c0c62c0 2024-04-24T23:21:14.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:14.930+0000 7f8613200640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f860c0cb1a0 con 0x7f860c067940 2024-04-24T23:21:14.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:14.930+0000 7f8611200640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f860c067940 0x7f860c06c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:21:14.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:14.930+0000 7f8611200640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f860c067940 0x7f860c06c6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:48888/0 (socket says 172.21.15.6:48888) 2024-04-24T23:21:14.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:14.930+0000 7f8611200640 1 -- 172.21.15.6:0/72188316 learned_addr learned my addr 172.21.15.6:0/72188316 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:21:14.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:14.930+0000 7f8611200640 1 -- 172.21.15.6:0/72188316 >> v1:172.21.15.190:6789/0 conn(0x7f860c0c62c0 legacy=0x7f860c06bf90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-24T23:21:14.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:14.930+0000 7f8611200640 1 -- 172.21.15.6:0/72188316 >> v1:172.21.15.6:6789/0 conn(0x7f860c154c80 legacy=0x7f860c0cbb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:21:14.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:14.931+0000 7f8611200640 1 -- 172.21.15.6:0/72188316 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f860c0c57e0 con 0x7f860c067940 2024-04-24T23:21:14.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:14.934+0000 7f8611200640 1 --2- 172.21.15.6:0/72188316 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f860c067940 0x7f860c06c6a0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f85fc009870 tx=0x7f85fc02a1c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=adf295d9ead09ea2 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:21:14.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:14.934+0000 7f860aa00640 1 -- 172.21.15.6:0/72188316 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85fc037070 con 0x7f860c067940 2024-04-24T23:21:14.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:14.934+0000 7f860aa00640 1 -- 172.21.15.6:0/72188316 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f85fc032440 con 0x7f860c067940 2024-04-24T23:21:14.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:14.934+0000 7f860aa00640 1 -- 172.21.15.6:0/72188316 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85fc032810 con 0x7f860c067940 2024-04-24T23:21:14.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:14.935+0000 7f8613200640 1 -- 172.21.15.6:0/72188316 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f860c067940 msgr2=0x7f860c06c6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:21:14.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:14.936+0000 7f8613200640 1 --2- 172.21.15.6:0/72188316 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f860c067940 0x7f860c06c6a0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f85fc009870 tx=0x7f85fc02a1c0 comp rx=0 tx=0).stop 2024-04-24T23:21:14.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:14.936+0000 7f8613200640 1 -- 172.21.15.6:0/72188316 shutdown_connections 2024-04-24T23:21:14.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:14.936+0000 7f8613200640 1 --2- 172.21.15.6:0/72188316 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f860c067940 0x7f860c06c6a0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:21:14.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:14.936+0000 7f8613200640 1 -- 172.21.15.6:0/72188316 >> 172.21.15.6:0/72188316 conn(0x7f860c072810 msgr2=0x7f860c066ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:21:14.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:14.937+0000 7f8613200640 1 -- 172.21.15.6:0/72188316 shutdown_connections 2024-04-24T23:21:14.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:14.937+0000 7f8613200640 1 -- 172.21.15.6:0/72188316 wait complete. 2024-04-24T23:21:14.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:14.940+0000 7f8613200640 1 Processor -- start 2024-04-24T23:21:14.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:14.940+0000 7f8613200640 1 -- start start 2024-04-24T23:21:14.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:14.942+0000 7f8613200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f860c067940 0x7f860c153350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:21:14.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:14.943+0000 7f860be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f860c067940 0x7f860c153350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:21:14.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:14.943+0000 7f8613200640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f860c0c62c0 0x7f860c0c7fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:21:14.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:14.943+0000 7f8613200640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f860c154c80 0x7f860c0c8510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:21:14.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:14.943+0000 7f8613200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f860c15b580 con 0x7f860c067940 2024-04-24T23:21:14.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:14.943+0000 7f8613200640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f860c0c5e30 con 0x7f860c0c62c0 2024-04-24T23:21:14.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:14.943+0000 7f8613200640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f860c0c9f20 con 0x7f860c154c80 2024-04-24T23:21:14.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:14.944+0000 7f8611200640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f860c154c80 0x7f860c0c8510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:21:14.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:14.944+0000 7f8611200640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f860c154c80 0x7f860c0c8510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.6:35060/0 (socket says 172.21.15.6:35060) 2024-04-24T23:21:14.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:14.944+0000 7f8611200640 1 -- 172.21.15.6:0/2558560044 learned_addr learned my addr 172.21.15.6:0/2558560044 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:21:14.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:14.944+0000 7f860b400640 1 --2- 172.21.15.6:0/2558560044 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f860c0c62c0 0x7f860c0c7fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:21:14.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:14.946+0000 7f860b400640 1 -- 172.21.15.6:0/2558560044 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f860c154c80 msgr2=0x7f860c0c8510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:21:14.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:14.946+0000 7f860b400640 1 --2- 172.21.15.6:0/2558560044 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f860c154c80 0x7f860c0c8510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:21:14.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:14.946+0000 7f860b400640 1 -- 172.21.15.6:0/2558560044 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f860c067940 msgr2=0x7f860c153350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:21:14.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:14.946+0000 7f860b400640 1 --2- 172.21.15.6:0/2558560044 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f860c067940 0x7f860c153350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:21:14.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:14.946+0000 7f860b400640 1 -- 172.21.15.6:0/2558560044 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f860c148ea0 con 0x7f860c0c62c0 2024-04-24T23:21:14.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:14.946+0000 7f860be00640 1 --2- 172.21.15.6:0/2558560044 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f860c067940 0x7f860c153350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:21:14.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:14.946+0000 7f8611200640 1 --2- 172.21.15.6:0/2558560044 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f860c154c80 0x7f860c0c8510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:21:14.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:14.947+0000 7f860b400640 1 --2- 172.21.15.6:0/2558560044 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f860c0c62c0 0x7f860c0c7fd0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f85f8009fd0 tx=0x7f85f800b870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:21:14.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:14.947+0000 7f8603e00640 1 -- 172.21.15.6:0/2558560044 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85f8015020 con 0x7f860c0c62c0 2024-04-24T23:21:14.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:14.948+0000 7f8603e00640 1 -- 172.21.15.6:0/2558560044 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f85f8011850 con 0x7f860c0c62c0 2024-04-24T23:21:14.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:14.948+0000 7f8603e00640 1 -- 172.21.15.6:0/2558560044 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85f8011c80 con 0x7f860c0c62c0 2024-04-24T23:21:14.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:14.948+0000 7f8613200640 1 -- 172.21.15.6:0/2558560044 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f860c0c7810 con 0x7f860c0c62c0 2024-04-24T23:21:14.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:14.948+0000 7f8613200640 1 -- 172.21.15.6:0/2558560044 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f860c0ca4c0 con 0x7f860c0c62c0 2024-04-24T23:21:14.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:14.950+0000 7f8613200640 1 -- 172.21.15.6:0/2558560044 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f860c0767b0 con 0x7f860c0c62c0 2024-04-24T23:21:14.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:14.951+0000 7f8603e00640 1 -- 172.21.15.6:0/2558560044 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f85f8018020 con 0x7f860c0c62c0 2024-04-24T23:21:14.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:14.952+0000 7f8603e00640 1 --2- 172.21.15.6:0/2558560044 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f85ec06be60 0x7f85ec06e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:21:14.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:14.952+0000 7f8603e00640 1 -- 172.21.15.6:0/2558560044 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6762+0+0 (secure 0 0 0) 0x7f85f808b170 con 0x7f860c0c62c0 2024-04-24T23:21:14.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:14.953+0000 7f860be00640 1 --2- 172.21.15.6:0/2558560044 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f85ec06be60 0x7f85ec06e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:21:14.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:14.954+0000 7f860be00640 1 --2- 172.21.15.6:0/2558560044 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f85ec06be60 0x7f85ec06e320 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f8604009fc0 tx=0x7f8604009550 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:21:14.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:14.959+0000 7f8603e00640 1 -- 172.21.15.6:0/2558560044 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f85f801f020 con 0x7f860c0c62c0 2024-04-24T23:21:15.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:15.506+0000 7f8613200640 1 -- 172.21.15.6:0/2558560044 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f860c0cb360 con 0x7f860c0c62c0 2024-04-24T23:21:15.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:15.509+0000 7f8603e00640 1 -- 172.21.15.6:0/2558560044 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+6084 (secure 0 0 0) 0x7f85f8011e20 con 0x7f860c0c62c0 2024-04-24T23:21:15.511 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:21:15.511 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:21:14.071015+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4282,"mds_2":4235,"mds_3":4232,"mds_4":5257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4232":{"gid":4232,"name":"b","rank":3,"incarnation":15,"state":"up:active","state_seq":15,"addr":"172.21.15.190:6838/2032280158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2032280158},{"type":"v1","addr":"172.21.15.190:6838","nonce":2032280158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4235":{"gid":4235,"name":"d","rank":2,"incarnation":13,"state":"up:active","state_seq":14,"addr":"172.21.15.190:6839/1572872451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":1572872451},{"type":"v1","addr":"172.21.15.190:6839","nonce":1572872451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4282":{"gid":4282,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":12,"addr":"172.21.15.190:6837/580315269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":580315269},{"type":"v1","addr":"172.21.15.190:6837","nonce":580315269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/489092393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":489092393},{"type":"v1","addr":"172.21.15.6:6835","nonce":489092393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5257":{"gid":5257,"name":"e","rank":4,"incarnation":25,"state":"up:reconnect","state_seq":38,"addr":"172.21.15.6:6837/3240971655","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":3240971655},{"type":"v1","addr":"172.21.15.6:6837","nonce":3240971655}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4306,"qdb_cluster":[4232,4235,4282,4306]},"id":1}]} 2024-04-24T23:21:15.511 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 27 2024-04-24T23:21:15.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:15.515+0000 7f8613200640 1 -- 172.21.15.6:0/2558560044 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f85ec06be60 msgr2=0x7f85ec06e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:21:15.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:15.515+0000 7f8613200640 1 --2- 172.21.15.6:0/2558560044 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f85ec06be60 0x7f85ec06e320 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f8604009fc0 tx=0x7f8604009550 comp rx=0 tx=0).stop 2024-04-24T23:21:15.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:15.515+0000 7f8613200640 1 -- 172.21.15.6:0/2558560044 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f860c0c62c0 msgr2=0x7f860c0c7fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:21:15.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:15.515+0000 7f8613200640 1 --2- 172.21.15.6:0/2558560044 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f860c0c62c0 0x7f860c0c7fd0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f85f8009fd0 tx=0x7f85f800b870 comp rx=0 tx=0).stop 2024-04-24T23:21:15.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:15.515+0000 7f8613200640 1 -- 172.21.15.6:0/2558560044 shutdown_connections 2024-04-24T23:21:15.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:15.515+0000 7f8613200640 1 --2- 172.21.15.6:0/2558560044 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f85ec06be60 0x7f85ec06e320 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:21:15.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:15.515+0000 7f8613200640 1 --2- 172.21.15.6:0/2558560044 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f860c154c80 0x7f860c0c8510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:21:15.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:15.515+0000 7f8613200640 1 --2- 172.21.15.6:0/2558560044 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f860c0c62c0 0x7f860c0c7fd0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:21:15.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:15.515+0000 7f8613200640 1 --2- 172.21.15.6:0/2558560044 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f860c067940 0x7f860c153350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:21:15.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:15.515+0000 7f8613200640 1 -- 172.21.15.6:0/2558560044 >> 172.21.15.6:0/2558560044 conn(0x7f860c072810 msgr2=0x7f860c159170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:21:15.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:15.516+0000 7f8613200640 1 -- 172.21.15.6:0/2558560044 shutdown_connections 2024-04-24T23:21:15.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:15.516+0000 7f8613200640 1 -- 172.21.15.6:0/2558560044 wait complete. 2024-04-24T23:21:15.545 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-24T23:21:17.547 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:21:17.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:17.829+0000 7f9a84e00640 1 Processor -- start 2024-04-24T23:21:17.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:17.829+0000 7f9a84e00640 1 -- start start 2024-04-24T23:21:17.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:17.831+0000 7f9a84e00640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f9a800c7dc0 0x7f9a801623a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:21:17.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:17.831+0000 7f9a84e00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9a800c7230 con 0x7f9a80159110 2024-04-24T23:21:17.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:17.831+0000 7f9a84e00640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9a800c74d0 con 0x7f9a800c81c0 2024-04-24T23:21:17.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:17.831+0000 7f9a84e00640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f9a801594f0 con 0x7f9a800c7dc0 2024-04-24T23:21:17.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:17.831+0000 7f9a7de00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f9a80159110 0x7f9a80072420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:52212/0 (socket says 172.21.15.6:52212) 2024-04-24T23:21:17.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:17.831+0000 7f9a7de00640 1 -- 172.21.15.6:0/2079011541 learned_addr learned my addr 172.21.15.6:0/2079011541 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:21:17.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:17.832+0000 7f9a7ca00640 1 -- 172.21.15.6:0/2079011541 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2325719364 0 0) 0x7f9a800c74d0 con 0x7f9a800c81c0 2024-04-24T23:21:17.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:17.832+0000 7f9a7ca00640 1 -- 172.21.15.6:0/2079011541 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9a64003520 con 0x7f9a800c81c0 2024-04-24T23:21:17.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:17.832+0000 7f9a7ca00640 1 -- 172.21.15.6:0/2079011541 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2510258901 0 0) 0x7f9a64003520 con 0x7f9a800c81c0 2024-04-24T23:21:17.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:17.832+0000 7f9a7ca00640 1 -- 172.21.15.6:0/2079011541 >> v2:172.21.15.190:3300/0 conn(0x7f9a800c7dc0 msgr2=0x7f9a801623a0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-24T23:21:17.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:17.832+0000 7f9a7ca00640 1 --2- 172.21.15.6:0/2079011541 >> v2:172.21.15.190:3300/0 conn(0x7f9a800c7dc0 0x7f9a801623a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:21:17.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:17.832+0000 7f9a7ca00640 1 -- 172.21.15.6:0/2079011541 >> v1:172.21.15.6:6789/0 conn(0x7f9a80159110 legacy=0x7f9a80072420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:21:17.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:17.832+0000 7f9a7ca00640 1 -- 172.21.15.6:0/2079011541 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a8006e440 con 0x7f9a800c81c0 2024-04-24T23:21:17.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:17.832+0000 7f9a7ca00640 1 -- 172.21.15.6:0/2079011541 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3463268701 0 0) 0x7f9a780019f0 con 0x7f9a800c81c0 2024-04-24T23:21:17.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:17.832+0000 7f9a7ca00640 1 -- 172.21.15.6:0/2079011541 >> v1:172.21.15.190:6789/0 conn(0x7f9a800c81c0 legacy=0x7f9a801628e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:21:17.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:17.832+0000 7f9a7ca00640 1 --2- 172.21.15.6:0/2079011541 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9a64003a00 0x7f9a64003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:21:17.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:17.832+0000 7f9a7ca00640 1 --2- 172.21.15.6:0/2079011541 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9a64004800 0x7f9a64004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:21:17.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:17.832+0000 7f9a7ca00640 1 --2- 172.21.15.6:0/2079011541 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9a640053c0 0x7f9a6400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:21:17.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:17.832+0000 7f9a7ca00640 1 -- 172.21.15.6:0/2079011541 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f9a801594f0 con 0x7f9a640053c0 2024-04-24T23:21:17.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:17.832+0000 7f9a7ca00640 1 -- 172.21.15.6:0/2079011541 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f9a6400c220 con 0x7f9a64004800 2024-04-24T23:21:17.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:17.832+0000 7f9a7ca00640 1 -- 172.21.15.6:0/2079011541 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f9a6400c420 con 0x7f9a64003a00 2024-04-24T23:21:17.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:17.833+0000 7f9a7e800640 1 --2- 172.21.15.6:0/2079011541 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9a640053c0 0x7f9a6400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:21:17.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:17.834+0000 7f9a7e800640 1 -- 172.21.15.6:0/2079011541 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9a64003a00 msgr2=0x7f9a64003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:21:17.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:17.834+0000 7f9a7e800640 1 --2- 172.21.15.6:0/2079011541 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9a64003a00 0x7f9a64003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:21:17.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:17.834+0000 7f9a7e800640 1 -- 172.21.15.6:0/2079011541 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9a64004800 msgr2=0x7f9a64004c80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-24T23:21:17.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:17.834+0000 7f9a7e800640 1 --2- 172.21.15.6:0/2079011541 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9a64004800 0x7f9a64004c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:21:17.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:17.834+0000 7f9a7e800640 1 -- 172.21.15.6:0/2079011541 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a64003520 con 0x7f9a640053c0 2024-04-24T23:21:17.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:17.835+0000 7f9a7e800640 1 --2- 172.21.15.6:0/2079011541 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9a640053c0 0x7f9a6400b9c0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f9a7800c4c0 tx=0x7f9a64005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b235509c4bac26ba server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:21:17.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:17.835+0000 7f9a7e800640 1 -- 172.21.15.6:0/2079011541 reap_dead start 2024-04-24T23:21:17.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:17.836+0000 7f9a7ca00640 1 -- 172.21.15.6:0/2079011541 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a78033070 con 0x7f9a640053c0 2024-04-24T23:21:17.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:17.836+0000 7f9a7ca00640 1 -- 172.21.15.6:0/2079011541 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a7802f640 con 0x7f9a640053c0 2024-04-24T23:21:17.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:17.836+0000 7f9a7ca00640 1 -- 172.21.15.6:0/2079011541 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a7800c0e0 con 0x7f9a640053c0 2024-04-24T23:21:17.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:17.836+0000 7f9a84e00640 1 -- 172.21.15.6:0/2079011541 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9a640053c0 msgr2=0x7f9a6400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:21:17.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:17.836+0000 7f9a84e00640 1 --2- 172.21.15.6:0/2079011541 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9a640053c0 0x7f9a6400b9c0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f9a7800c4c0 tx=0x7f9a64005830 comp rx=0 tx=0).stop 2024-04-24T23:21:17.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:17.837+0000 7f9a84e00640 1 -- 172.21.15.6:0/2079011541 shutdown_connections 2024-04-24T23:21:17.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:17.837+0000 7f9a84e00640 1 --2- 172.21.15.6:0/2079011541 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9a640053c0 0x7f9a6400b9c0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:21:17.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:17.837+0000 7f9a84e00640 1 -- 172.21.15.6:0/2079011541 >> 172.21.15.6:0/2079011541 conn(0x7f9a8006b0e0 msgr2=0x7f9a8006b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:21:17.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:17.837+0000 7f9a84e00640 1 -- 172.21.15.6:0/2079011541 shutdown_connections 2024-04-24T23:21:17.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:17.837+0000 7f9a84e00640 1 -- 172.21.15.6:0/2079011541 wait complete. 2024-04-24T23:21:17.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:17.840+0000 7f9a84e00640 1 Processor -- start 2024-04-24T23:21:17.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:17.840+0000 7f9a84e00640 1 -- start start 2024-04-24T23:21:17.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:17.841+0000 7f9a84e00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9a640053c0 0x7f9a800c9080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:21:17.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:17.842+0000 7f9a84e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9a80161cc0 0x7f9a8015be10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:21:17.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:17.842+0000 7f9a7de00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9a640053c0 0x7f9a800c9080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:21:17.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:17.842+0000 7f9a7de00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9a640053c0 0x7f9a800c9080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.6:35078/0 (socket says 172.21.15.6:35078) 2024-04-24T23:21:17.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:17.842+0000 7f9a84e00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9a800cb900 0x7f9a8015c350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:21:17.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:17.843+0000 7f9a84e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f9a8006ece0 con 0x7f9a80161cc0 2024-04-24T23:21:17.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:17.843+0000 7f9a7e800640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9a800cb900 0x7f9a8015c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:21:17.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:17.843+0000 7f9a7e800640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9a800cb900 0x7f9a8015c350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:48904/0 (socket says 172.21.15.6:48904) 2024-04-24T23:21:17.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:17.843+0000 7f9a7e800640 1 -- 172.21.15.6:0/37970887 learned_addr learned my addr 172.21.15.6:0/37970887 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:21:17.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:17.843+0000 7f9a84e00640 1 -- 172.21.15.6:0/37970887 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f9a8006ea00 con 0x7f9a800cb900 2024-04-24T23:21:17.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:17.843+0000 7f9a84e00640 1 -- 172.21.15.6:0/37970887 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f9a8006f200 con 0x7f9a640053c0 2024-04-24T23:21:17.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:17.843+0000 7f9a7d400640 1 --2- 172.21.15.6:0/37970887 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9a80161cc0 0x7f9a8015be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:21:17.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:17.844+0000 7f9a7e800640 1 -- 172.21.15.6:0/37970887 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9a640053c0 msgr2=0x7f9a800c9080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:21:17.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:17.844+0000 7f9a7e800640 1 --2- 172.21.15.6:0/37970887 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9a640053c0 0x7f9a800c9080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:21:17.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:17.844+0000 7f9a7e800640 1 -- 172.21.15.6:0/37970887 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9a80161cc0 msgr2=0x7f9a8015be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:21:17.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:17.844+0000 7f9a7e800640 1 --2- 172.21.15.6:0/37970887 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9a80161cc0 0x7f9a8015be10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:21:17.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:17.844+0000 7f9a7e800640 1 -- 172.21.15.6:0/37970887 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a801600f0 con 0x7f9a800cb900 2024-04-24T23:21:17.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:17.844+0000 7f9a7de00640 1 --2- 172.21.15.6:0/37970887 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9a640053c0 0x7f9a800c9080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:21:17.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:17.845+0000 7f9a7e800640 1 --2- 172.21.15.6:0/37970887 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9a800cb900 0x7f9a8015c350 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f9a78000f80 tx=0x7f9a8006e610 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:21:17.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:17.845+0000 7f9a76000640 1 -- 172.21.15.6:0/37970887 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a78033040 con 0x7f9a800cb900 2024-04-24T23:21:17.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:17.845+0000 7f9a76000640 1 -- 172.21.15.6:0/37970887 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a7802f440 con 0x7f9a800cb900 2024-04-24T23:21:17.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:17.845+0000 7f9a76000640 1 -- 172.21.15.6:0/37970887 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a78007d90 con 0x7f9a800cb900 2024-04-24T23:21:17.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:17.845+0000 7f9a84e00640 1 -- 172.21.15.6:0/37970887 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a80160390 con 0x7f9a800cb900 2024-04-24T23:21:17.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:17.846+0000 7f9a84e00640 1 -- 172.21.15.6:0/37970887 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a80160950 con 0x7f9a800cb900 2024-04-24T23:21:17.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:17.847+0000 7f9a84e00640 1 -- 172.21.15.6:0/37970887 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a800767b0 con 0x7f9a800cb900 2024-04-24T23:21:17.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:17.848+0000 7f9a76000640 1 -- 172.21.15.6:0/37970887 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f9a78036070 con 0x7f9a800cb900 2024-04-24T23:21:17.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:17.850+0000 7f9a76000640 1 --2- 172.21.15.6:0/37970887 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f9a6006beb0 0x7f9a6006e370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:21:17.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:17.850+0000 7f9a76000640 1 -- 172.21.15.6:0/37970887 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6762+0+0 (secure 0 0 0) 0x7f9a780ab6b0 con 0x7f9a800cb900 2024-04-24T23:21:17.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:17.850+0000 7f9a7de00640 1 --2- 172.21.15.6:0/37970887 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f9a6006beb0 0x7f9a6006e370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:21:17.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:17.851+0000 7f9a7de00640 1 --2- 172.21.15.6:0/37970887 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f9a6006beb0 0x7f9a6006e370 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f9a6c005880 tx=0x7f9a6c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:21:17.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:17.857+0000 7f9a76000640 1 -- 172.21.15.6:0/37970887 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f9a7802c340 con 0x7f9a800cb900 2024-04-24T23:21:18.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:18.324+0000 7f9a84e00640 1 -- 172.21.15.6:0/37970887 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9a8006f7d0 con 0x7f9a800cb900 2024-04-24T23:21:18.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:18.325+0000 7f9a76000640 1 -- 172.21.15.6:0/37970887 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+6084 (secure 0 0 0) 0x7f9a78079b90 con 0x7f9a800cb900 2024-04-24T23:21:18.327 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:21:18.328 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:21:14.071015+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4282,"mds_2":4235,"mds_3":4232,"mds_4":5257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4232":{"gid":4232,"name":"b","rank":3,"incarnation":15,"state":"up:active","state_seq":15,"addr":"172.21.15.190:6838/2032280158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2032280158},{"type":"v1","addr":"172.21.15.190:6838","nonce":2032280158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4235":{"gid":4235,"name":"d","rank":2,"incarnation":13,"state":"up:active","state_seq":14,"addr":"172.21.15.190:6839/1572872451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":1572872451},{"type":"v1","addr":"172.21.15.190:6839","nonce":1572872451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4282":{"gid":4282,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":12,"addr":"172.21.15.190:6837/580315269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":580315269},{"type":"v1","addr":"172.21.15.190:6837","nonce":580315269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/489092393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":489092393},{"type":"v1","addr":"172.21.15.6:6835","nonce":489092393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5257":{"gid":5257,"name":"e","rank":4,"incarnation":25,"state":"up:reconnect","state_seq":38,"addr":"172.21.15.6:6837/3240971655","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":3240971655},{"type":"v1","addr":"172.21.15.6:6837","nonce":3240971655}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4306,"qdb_cluster":[4232,4235,4282,4306]},"id":1}]} 2024-04-24T23:21:18.328 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 27 2024-04-24T23:21:18.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:18.328+0000 7f9a84e00640 1 -- 172.21.15.6:0/37970887 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f9a6006beb0 msgr2=0x7f9a6006e370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:21:18.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:18.328+0000 7f9a84e00640 1 --2- 172.21.15.6:0/37970887 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f9a6006beb0 0x7f9a6006e370 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f9a6c005880 tx=0x7f9a6c009040 comp rx=0 tx=0).stop 2024-04-24T23:21:18.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:18.328+0000 7f9a84e00640 1 -- 172.21.15.6:0/37970887 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9a800cb900 msgr2=0x7f9a8015c350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:21:18.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:18.328+0000 7f9a84e00640 1 --2- 172.21.15.6:0/37970887 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9a800cb900 0x7f9a8015c350 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f9a78000f80 tx=0x7f9a8006e610 comp rx=0 tx=0).stop 2024-04-24T23:21:18.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:18.328+0000 7f9a84e00640 1 -- 172.21.15.6:0/37970887 shutdown_connections 2024-04-24T23:21:18.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:18.328+0000 7f9a84e00640 1 --2- 172.21.15.6:0/37970887 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f9a6006beb0 0x7f9a6006e370 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:21:18.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:18.328+0000 7f9a84e00640 1 --2- 172.21.15.6:0/37970887 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9a800cb900 0x7f9a8015c350 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:21:18.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:18.328+0000 7f9a84e00640 1 --2- 172.21.15.6:0/37970887 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9a80161cc0 0x7f9a8015be10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:21:18.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:18.328+0000 7f9a84e00640 1 --2- 172.21.15.6:0/37970887 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9a640053c0 0x7f9a800c9080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:21:18.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:18.328+0000 7f9a84e00640 1 -- 172.21.15.6:0/37970887 >> 172.21.15.6:0/37970887 conn(0x7f9a8006b0e0 msgr2=0x7f9a801573d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:21:18.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:18.328+0000 7f9a84e00640 1 -- 172.21.15.6:0/37970887 shutdown_connections 2024-04-24T23:21:18.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:18.328+0000 7f9a84e00640 1 -- 172.21.15.6:0/37970887 wait complete. 2024-04-24T23:21:18.356 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-24T23:21:20.357 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:21:20.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:20.648+0000 7efd97800640 1 Processor -- start 2024-04-24T23:21:20.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:20.650+0000 7efd97800640 1 -- start start 2024-04-24T23:21:20.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:20.652+0000 7efd97800640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efd90159110 0x7efd90072420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:21:20.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:20.652+0000 7efd97800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7efd900c7dc0 0x7efd901623a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:21:20.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:20.652+0000 7efd97800640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7efd900c81c0 0x7efd901628e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:21:20.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:20.652+0000 7efd97800640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7efd901594f0 con 0x7efd900c7dc0 2024-04-24T23:21:20.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:20.652+0000 7efd97800640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7efd900c7230 con 0x7efd900c81c0 2024-04-24T23:21:20.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:20.652+0000 7efd97800640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7efd900c74d0 con 0x7efd90159110 2024-04-24T23:21:20.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:20.652+0000 7efd95800640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7efd900c81c0 0x7efd901628e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:21:20.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:20.652+0000 7efd95800640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7efd900c81c0 0x7efd901628e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:48920/0 (socket says 172.21.15.6:48920) 2024-04-24T23:21:20.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:20.652+0000 7efd95800640 1 -- 172.21.15.6:0/2747671709 learned_addr learned my addr 172.21.15.6:0/2747671709 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:21:20.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:20.652+0000 7efd94e00640 1 --2- 172.21.15.6:0/2747671709 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efd90159110 0x7efd90072420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:21:20.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:20.653+0000 7efd8fe00640 1 --2- 172.21.15.6:0/2747671709 >> v2:172.21.15.6:3300/0 conn(0x7efd900c7dc0 0x7efd901623a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:21:20.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:20.653+0000 7efd95800640 1 -- 172.21.15.6:0/2747671709 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efd90159110 msgr2=0x7efd90072420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:21:20.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:20.653+0000 7efd95800640 1 --2- 172.21.15.6:0/2747671709 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efd90159110 0x7efd90072420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:21:20.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:20.653+0000 7efd95800640 1 -- 172.21.15.6:0/2747671709 >> v2:172.21.15.6:3300/0 conn(0x7efd900c7dc0 msgr2=0x7efd901623a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:21:20.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:20.653+0000 7efd95800640 1 --2- 172.21.15.6:0/2747671709 >> v2:172.21.15.6:3300/0 conn(0x7efd900c7dc0 0x7efd901623a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:21:20.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:20.653+0000 7efd95800640 1 -- 172.21.15.6:0/2747671709 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd9006e440 con 0x7efd900c81c0 2024-04-24T23:21:20.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:20.653+0000 7efd8fe00640 1 --2- 172.21.15.6:0/2747671709 >> v2:172.21.15.6:3300/0 conn(0x7efd900c7dc0 0x7efd901623a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:21:20.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:20.655+0000 7efd95800640 1 --2- 172.21.15.6:0/2747671709 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efd900c81c0 0x7efd901628e0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7efd900c3610 tx=0x7efd8802a730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=125387f00ab56fd3 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:21:20.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:20.656+0000 7efd8f400640 1 -- 172.21.15.6:0/2747671709 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd88036070 con 0x7efd900c81c0 2024-04-24T23:21:20.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:20.656+0000 7efd8f400640 1 -- 172.21.15.6:0/2747671709 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efd88031600 con 0x7efd900c81c0 2024-04-24T23:21:20.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:20.656+0000 7efd8f400640 1 -- 172.21.15.6:0/2747671709 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd880319d0 con 0x7efd900c81c0 2024-04-24T23:21:20.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:20.656+0000 7efd97800640 1 -- 172.21.15.6:0/2747671709 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efd900c81c0 msgr2=0x7efd901628e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:21:20.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:20.657+0000 7efd97800640 1 --2- 172.21.15.6:0/2747671709 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efd900c81c0 0x7efd901628e0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7efd900c3610 tx=0x7efd8802a730 comp rx=0 tx=0).stop 2024-04-24T23:21:20.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:20.657+0000 7efd97800640 1 -- 172.21.15.6:0/2747671709 shutdown_connections 2024-04-24T23:21:20.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:20.657+0000 7efd97800640 1 --2- 172.21.15.6:0/2747671709 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efd900c81c0 0x7efd901628e0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:21:20.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:20.657+0000 7efd97800640 1 --2- 172.21.15.6:0/2747671709 >> v2:172.21.15.6:3300/0 conn(0x7efd900c7dc0 0x7efd901623a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:21:20.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:20.657+0000 7efd97800640 1 --2- 172.21.15.6:0/2747671709 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efd90159110 0x7efd90072420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:21:20.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:20.657+0000 7efd97800640 1 -- 172.21.15.6:0/2747671709 >> 172.21.15.6:0/2747671709 conn(0x7efd9006b0e0 msgr2=0x7efd9006b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:21:20.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:20.657+0000 7efd97800640 1 -- 172.21.15.6:0/2747671709 shutdown_connections 2024-04-24T23:21:20.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:20.657+0000 7efd97800640 1 -- 172.21.15.6:0/2747671709 wait complete. 2024-04-24T23:21:20.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:20.660+0000 7efd97800640 1 Processor -- start 2024-04-24T23:21:20.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:20.660+0000 7efd97800640 1 -- start start 2024-04-24T23:21:20.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:20.661+0000 7efd97800640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efd900c7dc0 0x7efd9015b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:21:20.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:20.661+0000 7efd97800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efd900c81c0 0x7efd9015ea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:21:20.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:20.661+0000 7efd97800640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efd90159110 0x7efd9015ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:21:20.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:20.661+0000 7efd97800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7efd9006ece0 con 0x7efd900c81c0 2024-04-24T23:21:20.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:20.661+0000 7efd97800640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7efd9006ea00 con 0x7efd900c7dc0 2024-04-24T23:21:20.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:20.661+0000 7efd97800640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7efd90072960 con 0x7efd90159110 2024-04-24T23:21:20.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:20.663+0000 7efd8fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efd900c81c0 0x7efd9015ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:21:20.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:20.664+0000 7efd8fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efd900c81c0 0x7efd9015ea00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:48420/0 (socket says 172.21.15.6:48420) 2024-04-24T23:21:20.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:20.664+0000 7efd8fe00640 1 -- 172.21.15.6:0/3598936334 learned_addr learned my addr 172.21.15.6:0/3598936334 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:21:20.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:20.664+0000 7efd8fe00640 1 -- 172.21.15.6:0/3598936334 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efd90159110 msgr2=0x7efd9015ef40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-24T23:21:20.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:20.664+0000 7efd8fe00640 1 --2- 172.21.15.6:0/3598936334 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efd90159110 0x7efd9015ef40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:21:20.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:20.664+0000 7efd8fe00640 1 -- 172.21.15.6:0/3598936334 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efd900c7dc0 msgr2=0x7efd9015b3b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-24T23:21:20.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:20.664+0000 7efd8fe00640 1 --2- 172.21.15.6:0/3598936334 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efd900c7dc0 0x7efd9015b3b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:21:20.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:20.664+0000 7efd8fe00640 1 -- 172.21.15.6:0/3598936334 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd900c35d0 con 0x7efd900c81c0 2024-04-24T23:21:20.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:20.665+0000 7efd8fe00640 1 --2- 172.21.15.6:0/3598936334 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efd900c81c0 0x7efd9015ea00 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7efd8400c7c0 tx=0x7efd8400cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:21:20.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:20.665+0000 7efd8cc00640 1 -- 172.21.15.6:0/3598936334 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd84015070 con 0x7efd900c81c0 2024-04-24T23:21:20.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:20.665+0000 7efd97800640 1 -- 172.21.15.6:0/3598936334 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd90144ff0 con 0x7efd900c81c0 2024-04-24T23:21:20.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:20.665+0000 7efd97800640 1 -- 172.21.15.6:0/3598936334 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd9015fcb0 con 0x7efd900c81c0 2024-04-24T23:21:20.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:20.665+0000 7efd8cc00640 1 -- 172.21.15.6:0/3598936334 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efd84007a30 con 0x7efd900c81c0 2024-04-24T23:21:20.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:20.665+0000 7efd8cc00640 1 -- 172.21.15.6:0/3598936334 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd84007de0 con 0x7efd900c81c0 2024-04-24T23:21:20.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:20.667+0000 7efd97800640 1 -- 172.21.15.6:0/3598936334 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd6c005290 con 0x7efd900c81c0 2024-04-24T23:21:20.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:20.670+0000 7efd8cc00640 1 -- 172.21.15.6:0/3598936334 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7efd84018070 con 0x7efd900c81c0 2024-04-24T23:21:20.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:20.671+0000 7efd8cc00640 1 --2- 172.21.15.6:0/3598936334 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7efd7006be30 0x7efd7006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:21:20.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:20.671+0000 7efd8cc00640 1 -- 172.21.15.6:0/3598936334 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6762+0+0 (secure 0 0 0) 0x7efd8408b4f0 con 0x7efd900c81c0 2024-04-24T23:21:20.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:20.671+0000 7efd94e00640 1 --2- 172.21.15.6:0/3598936334 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7efd7006be30 0x7efd7006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:21:20.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:20.672+0000 7efd94e00640 1 --2- 172.21.15.6:0/3598936334 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7efd7006be30 0x7efd7006e2f0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7efd7c009930 tx=0x7efd7c009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:21:20.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:20.676+0000 7efd8cc00640 1 -- 172.21.15.6:0/3598936334 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7efd84059af0 con 0x7efd900c81c0 2024-04-24T23:21:21.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:21.319+0000 7efd97800640 1 -- 172.21.15.6:0/3598936334 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efd6c005580 con 0x7efd900c81c0 2024-04-24T23:21:21.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:21.320+0000 7efd8cc00640 1 -- 172.21.15.6:0/3598936334 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+6084 (secure 0 0 0) 0x7efd840593f0 con 0x7efd900c81c0 2024-04-24T23:21:21.326 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:21:21.326 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:21:14.071015+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4282,"mds_2":4235,"mds_3":4232,"mds_4":5257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4232":{"gid":4232,"name":"b","rank":3,"incarnation":15,"state":"up:active","state_seq":15,"addr":"172.21.15.190:6838/2032280158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2032280158},{"type":"v1","addr":"172.21.15.190:6838","nonce":2032280158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4235":{"gid":4235,"name":"d","rank":2,"incarnation":13,"state":"up:active","state_seq":14,"addr":"172.21.15.190:6839/1572872451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":1572872451},{"type":"v1","addr":"172.21.15.190:6839","nonce":1572872451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4282":{"gid":4282,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":12,"addr":"172.21.15.190:6837/580315269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":580315269},{"type":"v1","addr":"172.21.15.190:6837","nonce":580315269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/489092393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":489092393},{"type":"v1","addr":"172.21.15.6:6835","nonce":489092393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5257":{"gid":5257,"name":"e","rank":4,"incarnation":25,"state":"up:reconnect","state_seq":38,"addr":"172.21.15.6:6837/3240971655","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":3240971655},{"type":"v1","addr":"172.21.15.6:6837","nonce":3240971655}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4306,"qdb_cluster":[4232,4235,4282,4306]},"id":1}]} 2024-04-24T23:21:21.326 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 27 2024-04-24T23:21:21.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:21.324+0000 7efd97800640 1 -- 172.21.15.6:0/3598936334 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7efd7006be30 msgr2=0x7efd7006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:21:21.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:21.324+0000 7efd97800640 1 --2- 172.21.15.6:0/3598936334 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7efd7006be30 0x7efd7006e2f0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7efd7c009930 tx=0x7efd7c009190 comp rx=0 tx=0).stop 2024-04-24T23:21:21.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:21.324+0000 7efd97800640 1 -- 172.21.15.6:0/3598936334 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efd900c81c0 msgr2=0x7efd9015ea00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:21:21.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:21.324+0000 7efd97800640 1 --2- 172.21.15.6:0/3598936334 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efd900c81c0 0x7efd9015ea00 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7efd8400c7c0 tx=0x7efd8400cc80 comp rx=0 tx=0).stop 2024-04-24T23:21:21.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:21.325+0000 7efd97800640 1 -- 172.21.15.6:0/3598936334 shutdown_connections 2024-04-24T23:21:21.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:21.325+0000 7efd97800640 1 --2- 172.21.15.6:0/3598936334 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7efd7006be30 0x7efd7006e2f0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:21:21.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:21.325+0000 7efd97800640 1 --2- 172.21.15.6:0/3598936334 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efd90159110 0x7efd9015ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:21:21.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:21.325+0000 7efd97800640 1 --2- 172.21.15.6:0/3598936334 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efd900c81c0 0x7efd9015ea00 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:21:21.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:21.325+0000 7efd97800640 1 --2- 172.21.15.6:0/3598936334 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efd900c7dc0 0x7efd9015b3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:21:21.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:21.325+0000 7efd97800640 1 -- 172.21.15.6:0/3598936334 >> 172.21.15.6:0/3598936334 conn(0x7efd9006b0e0 msgr2=0x7efd900c9020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:21:21.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:21.326+0000 7efd97800640 1 -- 172.21.15.6:0/3598936334 shutdown_connections 2024-04-24T23:21:21.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:21.327+0000 7efd97800640 1 -- 172.21.15.6:0/3598936334 wait complete. 2024-04-24T23:21:21.358 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 27, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-24T23:15:36.400716+0000", "damaged": [], "data_pools": [ 3, 4 ], "enabled": true, "epoch": 27, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4232": { "addr": "172.21.15.190:6838/2032280158", "addrs": { "addrvec": [ { "addr": "172.21.15.190:6835", "nonce": 2032280158, "type": "v2" }, { "addr": "172.21.15.190:6838", "nonce": 2032280158, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4232, "incarnation": 15, "join_fscid": -1, "name": "b", "rank": 3, "state": "up:active", "state_seq": 15 }, "gid_4235": { "addr": "172.21.15.190:6839/1572872451", "addrs": { "addrvec": [ { "addr": "172.21.15.190:6836", "nonce": 1572872451, "type": "v2" }, { "addr": "172.21.15.190:6839", "nonce": 1572872451, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4235, "incarnation": 13, "join_fscid": -1, "name": "d", "rank": 2, "state": "up:active", "state_seq": 14 }, "gid_4282": { "addr": "172.21.15.190:6837/580315269", "addrs": { "addrvec": [ { "addr": "172.21.15.190:6834", "nonce": 580315269, "type": "v2" }, { "addr": "172.21.15.190:6837", "nonce": 580315269, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4282, "incarnation": 9, "join_fscid": -1, "name": "f", "rank": 1, "state": "up:active", "state_seq": 12 }, "gid_4306": { "addr": "172.21.15.6:6835/489092393", "addrs": { "addrvec": [ { "addr": "172.21.15.6:6833", "nonce": 489092393, "type": "v2" }, { "addr": "172.21.15.6:6835", "nonce": 489092393, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4306, "incarnation": 4, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 4 }, "gid_5257": { "addr": "172.21.15.6:6837/3240971655", "addrs": { "addrvec": [ { "addr": "172.21.15.6:6836", "nonce": 3240971655, "type": "v2" }, { "addr": "172.21.15.6:6837", "nonce": 3240971655, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5257, "incarnation": 25, "join_fscid": -1, "name": "e", "rank": 4, "state": "up:reconnect", "state_seq": 38 } }, "last_failure": 0, "last_failure_osd_epoch": 28, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-04-24T23:21:14.071015+0000", "qdb_cluster": [ 4232, 4235, 4282, 4306 ], "qdb_leader": 4306, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4306, "mds_1": 4282, "mds_2": 4235, "mds_3": 4232, "mds_4": 5257 } } } ], "standbys": [] } 2024-04-24T23:21:23.359 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:21:23.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:23.606+0000 7f6190c00640 1 Processor -- start 2024-04-24T23:21:23.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:23.608+0000 7f6190c00640 1 -- start start 2024-04-24T23:21:23.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:23.610+0000 7f6190c00640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f618c0c7080 0x7f618c06f020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:21:23.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:23.610+0000 7f6190c00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f618c072530 con 0x7f618c154b70 2024-04-24T23:21:23.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:23.610+0000 7f6190c00640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f618c0727d0 con 0x7f618c071000 2024-04-24T23:21:23.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:23.610+0000 7f6190c00640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f618c072a70 con 0x7f618c0c7080 2024-04-24T23:21:23.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:23.612+0000 7f618a800640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f618c071000 0x7f618c15b0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.6:38370/0 (socket says 172.21.15.6:38370) 2024-04-24T23:21:23.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:23.612+0000 7f618a800640 1 -- 172.21.15.6:0/1863426850 learned_addr learned my addr 172.21.15.6:0/1863426850 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:21:23.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:23.612+0000 7f6189400640 1 --2- 172.21.15.6:0/1863426850 >> v2:172.21.15.190:3300/0 conn(0x7f618c0c7080 0x7f618c06f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:21:23.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:23.614+0000 7f6188a00640 1 -- 172.21.15.6:0/1863426850 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 792441238 0 0) 0x7f618c072530 con 0x7f618c154b70 2024-04-24T23:21:23.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:23.614+0000 7f6188a00640 1 -- 172.21.15.6:0/1863426850 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6170003520 con 0x7f618c154b70 2024-04-24T23:21:23.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:23.614+0000 7f6189400640 1 -- 172.21.15.6:0/1863426850 >> v1:172.21.15.190:6789/0 conn(0x7f618c071000 legacy=0x7f618c15b0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:21:23.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:23.615+0000 7f6189400640 1 -- 172.21.15.6:0/1863426850 >> v1:172.21.15.6:6789/0 conn(0x7f618c154b70 legacy=0x7f618c15a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:21:23.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:23.615+0000 7f6189400640 1 -- 172.21.15.6:0/1863426850 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f618c06e540 con 0x7f618c0c7080 2024-04-24T23:21:23.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:23.617+0000 7f6189400640 1 --2- 172.21.15.6:0/1863426850 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f618c0c7080 0x7f618c06f020 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f61780097d0 tx=0x7f617802a1c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d6e5c2efb12e964c server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:21:23.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:23.617+0000 7f6188a00640 1 -- 172.21.15.6:0/1863426850 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6178037070 con 0x7f618c0c7080 2024-04-24T23:21:23.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:23.617+0000 7f6188a00640 1 -- 172.21.15.6:0/1863426850 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6178032440 con 0x7f618c0c7080 2024-04-24T23:21:23.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:23.617+0000 7f6188a00640 1 -- 172.21.15.6:0/1863426850 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6178032810 con 0x7f618c0c7080 2024-04-24T23:21:23.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:23.619+0000 7f6190c00640 1 -- 172.21.15.6:0/1863426850 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f618c0c7080 msgr2=0x7f618c06f020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:21:23.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:23.619+0000 7f6190c00640 1 --2- 172.21.15.6:0/1863426850 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f618c0c7080 0x7f618c06f020 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f61780097d0 tx=0x7f617802a1c0 comp rx=0 tx=0).stop 2024-04-24T23:21:23.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:23.625+0000 7f6190c00640 1 -- 172.21.15.6:0/1863426850 shutdown_connections 2024-04-24T23:21:23.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:23.625+0000 7f6190c00640 1 --2- 172.21.15.6:0/1863426850 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f618c0c7080 0x7f618c06f020 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:21:23.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:23.625+0000 7f6190c00640 1 -- 172.21.15.6:0/1863426850 >> 172.21.15.6:0/1863426850 conn(0x7f618c06b0e0 msgr2=0x7f618c06b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:21:23.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:23.625+0000 7f6190c00640 1 -- 172.21.15.6:0/1863426850 shutdown_connections 2024-04-24T23:21:23.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:23.625+0000 7f6190c00640 1 -- 172.21.15.6:0/1863426850 wait complete. 2024-04-24T23:21:23.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:23.627+0000 7f6190c00640 1 Processor -- start 2024-04-24T23:21:23.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:23.627+0000 7f6190c00640 1 -- start start 2024-04-24T23:21:23.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:23.629+0000 7f6190c00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f618c071000 0x7f618c1f4280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:21:23.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:23.629+0000 7f6190c00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f618c0c7080 0x7f618c150ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:21:23.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:23.629+0000 7f6190c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f618c154b70 0x7f618c149dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:21:23.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:23.629+0000 7f6190c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f618c15b7f0 con 0x7f618c154b70 2024-04-24T23:21:23.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:23.629+0000 7f6190c00640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f618c06eb00 con 0x7f618c0c7080 2024-04-24T23:21:23.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:23.629+0000 7f6190c00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f618c154f50 con 0x7f618c071000 2024-04-24T23:21:23.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:23.630+0000 7f6189400640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f618c0c7080 0x7f618c150ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:21:23.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:23.630+0000 7f6189400640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f618c0c7080 0x7f618c150ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:48944/0 (socket says 172.21.15.6:48944) 2024-04-24T23:21:23.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:23.630+0000 7f6189400640 1 -- 172.21.15.6:0/1868128685 learned_addr learned my addr 172.21.15.6:0/1868128685 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:21:23.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:23.630+0000 7f618a800640 1 --2- 172.21.15.6:0/1868128685 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f618c154b70 0x7f618c149dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:21:23.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:23.630+0000 7f6189e00640 1 --2- 172.21.15.6:0/1868128685 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f618c071000 0x7f618c1f4280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:21:23.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:23.630+0000 7f6189400640 1 -- 172.21.15.6:0/1868128685 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f618c071000 msgr2=0x7f618c1f4280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:21:23.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:23.630+0000 7f6189400640 1 --2- 172.21.15.6:0/1868128685 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f618c071000 0x7f618c1f4280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:21:23.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:23.630+0000 7f6189400640 1 -- 172.21.15.6:0/1868128685 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f618c154b70 msgr2=0x7f618c149dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:21:23.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:23.630+0000 7f6189400640 1 --2- 172.21.15.6:0/1868128685 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f618c154b70 0x7f618c149dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:21:23.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:23.630+0000 7f6189400640 1 -- 172.21.15.6:0/1868128685 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f618c159420 con 0x7f618c0c7080 2024-04-24T23:21:23.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:23.631+0000 7f6189400640 1 --2- 172.21.15.6:0/1868128685 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f618c0c7080 0x7f618c150ef0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f6178009720 tx=0x7f61780046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:21:23.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:23.631+0000 7f6182000640 1 -- 172.21.15.6:0/1868128685 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6178037040 con 0x7f618c0c7080 2024-04-24T23:21:23.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:23.631+0000 7f6182000640 1 -- 172.21.15.6:0/1868128685 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6178004230 con 0x7f618c0c7080 2024-04-24T23:21:23.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:23.631+0000 7f6182000640 1 -- 172.21.15.6:0/1868128685 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f617803c670 con 0x7f618c0c7080 2024-04-24T23:21:23.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:23.631+0000 7f6190c00640 1 -- 172.21.15.6:0/1868128685 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f618c0c42d0 con 0x7f618c0c7080 2024-04-24T23:21:23.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:23.632+0000 7f6190c00640 1 -- 172.21.15.6:0/1868128685 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f618c1f3a80 con 0x7f618c0c7080 2024-04-24T23:21:23.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:23.633+0000 7f6190c00640 1 -- 172.21.15.6:0/1868128685 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f618c1f38a0 con 0x7f618c0c7080 2024-04-24T23:21:23.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:23.635+0000 7f6182000640 1 -- 172.21.15.6:0/1868128685 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f617803a070 con 0x7f618c0c7080 2024-04-24T23:21:23.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:23.636+0000 7f6182000640 1 --2- 172.21.15.6:0/1868128685 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f616c06be60 0x7f616c06e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:21:23.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:23.636+0000 7f6182000640 1 -- 172.21.15.6:0/1868128685 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6762+0+0 (secure 0 0 0) 0x7f61780775f0 con 0x7f618c0c7080 2024-04-24T23:21:23.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:23.636+0000 7f6189e00640 1 --2- 172.21.15.6:0/1868128685 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f616c06be60 0x7f616c06e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:21:23.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:23.637+0000 7f6189e00640 1 --2- 172.21.15.6:0/1868128685 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f616c06be60 0x7f616c06e320 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f617c0010e0 tx=0x7f617c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:21:23.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:23.639+0000 7f6182000640 1 -- 172.21.15.6:0/1868128685 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f618c1f38a0 con 0x7f618c0c7080 2024-04-24T23:21:24.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:24.138+0000 7f6190c00640 1 -- 172.21.15.6:0/1868128685 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f618c0767b0 con 0x7f618c0c7080 2024-04-24T23:21:24.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:24.139+0000 7f6182000640 1 -- 172.21.15.6:0/1868128685 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+6081 (secure 0 0 0) 0x7f6178041020 con 0x7f618c0c7080 2024-04-24T23:21:24.145 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:21:24.146 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:21:23.121993+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4282,"mds_2":4235,"mds_3":4232,"mds_4":5257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4232":{"gid":4232,"name":"b","rank":3,"incarnation":15,"state":"up:active","state_seq":15,"addr":"172.21.15.190:6838/2032280158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2032280158},{"type":"v1","addr":"172.21.15.190:6838","nonce":2032280158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4235":{"gid":4235,"name":"d","rank":2,"incarnation":13,"state":"up:active","state_seq":14,"addr":"172.21.15.190:6839/1572872451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":1572872451},{"type":"v1","addr":"172.21.15.190:6839","nonce":1572872451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4282":{"gid":4282,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":12,"addr":"172.21.15.190:6837/580315269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":580315269},{"type":"v1","addr":"172.21.15.190:6837","nonce":580315269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/489092393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":489092393},{"type":"v1","addr":"172.21.15.6:6835","nonce":489092393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5257":{"gid":5257,"name":"e","rank":4,"incarnation":25,"state":"up:rejoin","state_seq":41,"addr":"172.21.15.6:6837/3240971655","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":3240971655},{"type":"v1","addr":"172.21.15.6:6837","nonce":3240971655}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4306,"qdb_cluster":[4306,4282,4235,4232]},"id":1}]} 2024-04-24T23:21:24.146 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 28 2024-04-24T23:21:24.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:24.144+0000 7f6190c00640 1 -- 172.21.15.6:0/1868128685 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f616c06be60 msgr2=0x7f616c06e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:21:24.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:24.145+0000 7f6190c00640 1 --2- 172.21.15.6:0/1868128685 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f616c06be60 0x7f616c06e320 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f617c0010e0 tx=0x7f617c009040 comp rx=0 tx=0).stop 2024-04-24T23:21:24.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:24.145+0000 7f6190c00640 1 -- 172.21.15.6:0/1868128685 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f618c0c7080 msgr2=0x7f618c150ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:21:24.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:24.145+0000 7f6190c00640 1 --2- 172.21.15.6:0/1868128685 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f618c0c7080 0x7f618c150ef0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f6178009720 tx=0x7f61780046c0 comp rx=0 tx=0).stop 2024-04-24T23:21:24.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:24.147+0000 7f6190c00640 1 -- 172.21.15.6:0/1868128685 shutdown_connections 2024-04-24T23:21:24.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:24.147+0000 7f6190c00640 1 --2- 172.21.15.6:0/1868128685 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f616c06be60 0x7f616c06e320 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:21:24.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:24.147+0000 7f6190c00640 1 --2- 172.21.15.6:0/1868128685 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f618c154b70 0x7f618c149dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:21:24.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:24.147+0000 7f6190c00640 1 --2- 172.21.15.6:0/1868128685 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f618c0c7080 0x7f618c150ef0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:21:24.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:24.147+0000 7f6190c00640 1 --2- 172.21.15.6:0/1868128685 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f618c071000 0x7f618c1f4280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:21:24.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:24.147+0000 7f6190c00640 1 -- 172.21.15.6:0/1868128685 >> 172.21.15.6:0/1868128685 conn(0x7f618c06b0e0 msgr2=0x7f618c0c7ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:21:24.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:24.147+0000 7f6190c00640 1 -- 172.21.15.6:0/1868128685 shutdown_connections 2024-04-24T23:21:24.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:24.150+0000 7f6190c00640 1 -- 172.21.15.6:0/1868128685 wait complete. 2024-04-24T23:21:24.180 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-24T23:21:26.181 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:21:26.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:26.530+0000 7f0499600640 1 Processor -- start 2024-04-24T23:21:26.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:26.530+0000 7f0499600640 1 -- start start 2024-04-24T23:21:26.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:26.532+0000 7f0499600640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f04940c7080 0x7f049406f020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:21:26.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:26.532+0000 7f0499600640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f0494071000 0x7f04940c80e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:21:26.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:26.532+0000 7f0499600640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0494072530 con 0x7f0494154b70 2024-04-24T23:21:26.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:26.532+0000 7f0499600640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f0494072a70 con 0x7f0494071000 2024-04-24T23:21:26.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:26.532+0000 7f0499600640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f04940727d0 con 0x7f04940c7080 2024-04-24T23:21:26.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:26.533+0000 7f0492000640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f04940c7080 0x7f049406f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:21:26.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:26.533+0000 7f0492000640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f04940c7080 0x7f049406f020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.6:41960/0 (socket says 172.21.15.6:41960) 2024-04-24T23:21:26.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:26.533+0000 7f0492000640 1 -- 172.21.15.6:0/108482416 learned_addr learned my addr 172.21.15.6:0/108482416 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:21:26.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:26.533+0000 7f0492000640 1 -- 172.21.15.6:0/108482416 >> v2:172.21.15.6:3300/0 conn(0x7f0494071000 msgr2=0x7f04940c80e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:21:26.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:26.533+0000 7f0492000640 1 --2- 172.21.15.6:0/108482416 >> v2:172.21.15.6:3300/0 conn(0x7f0494071000 0x7f04940c80e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:21:26.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:26.533+0000 7f0492000640 1 -- 172.21.15.6:0/108482416 >> v1:172.21.15.6:6789/0 conn(0x7f0494154b70 legacy=0x7f049415a9d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-24T23:21:26.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:26.533+0000 7f0492000640 1 -- 172.21.15.6:0/108482416 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f049406e540 con 0x7f04940c7080 2024-04-24T23:21:26.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:26.535+0000 7f0492000640 1 --2- 172.21.15.6:0/108482416 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f04940c7080 0x7f049406f020 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f0484009870 tx=0x7f048402a210 comp rx=0 tx=0).ready entity=mon.2 client_cookie=98a25fb735ecf9a7 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:21:26.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:26.535+0000 7f0491600640 1 -- 172.21.15.6:0/108482416 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0484037070 con 0x7f04940c7080 2024-04-24T23:21:26.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:26.535+0000 7f0491600640 1 -- 172.21.15.6:0/108482416 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0484032440 con 0x7f04940c7080 2024-04-24T23:21:26.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:26.535+0000 7f0491600640 1 -- 172.21.15.6:0/108482416 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0484032810 con 0x7f04940c7080 2024-04-24T23:21:26.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:26.536+0000 7f0499600640 1 -- 172.21.15.6:0/108482416 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f04940c7080 msgr2=0x7f049406f020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:21:26.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:26.536+0000 7f0499600640 1 --2- 172.21.15.6:0/108482416 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f04940c7080 0x7f049406f020 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f0484009870 tx=0x7f048402a210 comp rx=0 tx=0).stop 2024-04-24T23:21:26.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:26.536+0000 7f0499600640 1 -- 172.21.15.6:0/108482416 shutdown_connections 2024-04-24T23:21:26.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:26.536+0000 7f0499600640 1 --2- 172.21.15.6:0/108482416 >> v2:172.21.15.6:3300/0 conn(0x7f0494071000 0x7f04940c80e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:21:26.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:26.536+0000 7f0499600640 1 --2- 172.21.15.6:0/108482416 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f04940c7080 0x7f049406f020 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:21:26.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:26.536+0000 7f0499600640 1 -- 172.21.15.6:0/108482416 >> 172.21.15.6:0/108482416 conn(0x7f049406b0e0 msgr2=0x7f049406b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:21:26.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:26.537+0000 7f0499600640 1 -- 172.21.15.6:0/108482416 shutdown_connections 2024-04-24T23:21:26.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:26.537+0000 7f0499600640 1 -- 172.21.15.6:0/108482416 wait complete. 2024-04-24T23:21:26.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:26.539+0000 7f0499600640 1 Processor -- start 2024-04-24T23:21:26.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:26.540+0000 7f0499600640 1 -- start start 2024-04-24T23:21:26.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:26.542+0000 7f0499600640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0494071000 0x7f04941f6900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:21:26.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:26.542+0000 7f0499600640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f04940c7080 0x7f04941ea510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:21:26.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:26.542+0000 7f0499600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0494154b70 0x7f0494149520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:21:26.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:26.542+0000 7f0499600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f0494154f50 con 0x7f0494154b70 2024-04-24T23:21:26.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:26.543+0000 7f0499600640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f049406eb00 con 0x7f0494071000 2024-04-24T23:21:26.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:26.543+0000 7f0499600640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f04941550d0 con 0x7f04940c7080 2024-04-24T23:21:26.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:26.543+0000 7f0493400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0494154b70 0x7f0494149520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:21:26.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:26.543+0000 7f0493400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0494154b70 0x7f0494149520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36138/0 (socket says 172.21.15.6:36138) 2024-04-24T23:21:26.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:26.543+0000 7f0493400640 1 -- 172.21.15.6:0/774261690 learned_addr learned my addr 172.21.15.6:0/774261690 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:21:26.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:26.543+0000 7f0493400640 1 -- 172.21.15.6:0/774261690 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f04940c7080 msgr2=0x7f04941ea510 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-24T23:21:26.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:26.543+0000 7f0493400640 1 --2- 172.21.15.6:0/774261690 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f04940c7080 0x7f04941ea510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:21:26.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:26.543+0000 7f0493400640 1 -- 172.21.15.6:0/774261690 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0494071000 msgr2=0x7f04941f6900 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-24T23:21:26.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:26.543+0000 7f0493400640 1 --2- 172.21.15.6:0/774261690 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0494071000 0x7f04941f6900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:21:26.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:26.543+0000 7f0493400640 1 -- 172.21.15.6:0/774261690 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04941e5910 con 0x7f0494154b70 2024-04-24T23:21:26.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:26.544+0000 7f0493400640 1 --2- 172.21.15.6:0/774261690 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0494154b70 0x7f0494149520 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f048c00b970 tx=0x7f048c00be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:21:26.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:26.545+0000 7f048aa00640 1 -- 172.21.15.6:0/774261690 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f048c019070 con 0x7f0494154b70 2024-04-24T23:21:26.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:26.545+0000 7f048aa00640 1 -- 172.21.15.6:0/774261690 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f048c00cb60 con 0x7f0494154b70 2024-04-24T23:21:26.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:26.545+0000 7f048aa00640 1 -- 172.21.15.6:0/774261690 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f048c014440 con 0x7f0494154b70 2024-04-24T23:21:26.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:26.546+0000 7f0499600640 1 -- 172.21.15.6:0/774261690 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04941e5bd0 con 0x7f0494154b70 2024-04-24T23:21:26.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:26.546+0000 7f0499600640 1 -- 172.21.15.6:0/774261690 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04940cab00 con 0x7f0494154b70 2024-04-24T23:21:26.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:26.551+0000 7f048aa00640 1 -- 172.21.15.6:0/774261690 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f048c01c070 con 0x7f0494154b70 2024-04-24T23:21:26.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:26.553+0000 7f048aa00640 1 --2- 172.21.15.6:0/774261690 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f047406bf00 0x7f047406e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:21:26.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:26.553+0000 7f048aa00640 1 -- 172.21.15.6:0/774261690 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6762+0+0 (secure 0 0 0) 0x7f048c08b060 con 0x7f0494154b70 2024-04-24T23:21:26.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:26.556+0000 7f0492a00640 1 --2- 172.21.15.6:0/774261690 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f047406bf00 0x7f047406e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:21:26.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:26.556+0000 7f0499600640 1 -- 172.21.15.6:0/774261690 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f04940767b0 con 0x7f0494154b70 2024-04-24T23:21:26.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:26.558+0000 7f0492a00640 1 --2- 172.21.15.6:0/774261690 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f047406bf00 0x7f047406e3c0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f04800095d0 tx=0x7f0480006b30 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:21:26.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:26.567+0000 7f048aa00640 1 -- 172.21.15.6:0/774261690 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f048c059540 con 0x7f0494154b70 2024-04-24T23:21:27.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:27.031+0000 7f0499600640 1 -- 172.21.15.6:0/774261690 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0494068570 con 0x7f0494154b70 2024-04-24T23:21:27.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:27.031+0000 7f048aa00640 1 -- 172.21.15.6:0/774261690 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+6081 (secure 0 0 0) 0x7f048c058e40 con 0x7f0494154b70 2024-04-24T23:21:27.037 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:21:27.038 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:21:23.121993+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4282,"mds_2":4235,"mds_3":4232,"mds_4":5257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4232":{"gid":4232,"name":"b","rank":3,"incarnation":15,"state":"up:active","state_seq":15,"addr":"172.21.15.190:6838/2032280158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2032280158},{"type":"v1","addr":"172.21.15.190:6838","nonce":2032280158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4235":{"gid":4235,"name":"d","rank":2,"incarnation":13,"state":"up:active","state_seq":14,"addr":"172.21.15.190:6839/1572872451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":1572872451},{"type":"v1","addr":"172.21.15.190:6839","nonce":1572872451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4282":{"gid":4282,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":12,"addr":"172.21.15.190:6837/580315269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":580315269},{"type":"v1","addr":"172.21.15.190:6837","nonce":580315269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/489092393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":489092393},{"type":"v1","addr":"172.21.15.6:6835","nonce":489092393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5257":{"gid":5257,"name":"e","rank":4,"incarnation":25,"state":"up:rejoin","state_seq":41,"addr":"172.21.15.6:6837/3240971655","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":3240971655},{"type":"v1","addr":"172.21.15.6:6837","nonce":3240971655}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4306,"qdb_cluster":[4306,4282,4235,4232]},"id":1}]} 2024-04-24T23:21:27.038 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 28 2024-04-24T23:21:27.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:27.036+0000 7f0499600640 1 -- 172.21.15.6:0/774261690 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f047406bf00 msgr2=0x7f047406e3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:21:27.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:27.037+0000 7f0499600640 1 --2- 172.21.15.6:0/774261690 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f047406bf00 0x7f047406e3c0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f04800095d0 tx=0x7f0480006b30 comp rx=0 tx=0).stop 2024-04-24T23:21:27.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:27.037+0000 7f0499600640 1 -- 172.21.15.6:0/774261690 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0494154b70 msgr2=0x7f0494149520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:21:27.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:27.037+0000 7f0499600640 1 --2- 172.21.15.6:0/774261690 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0494154b70 0x7f0494149520 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f048c00b970 tx=0x7f048c00be30 comp rx=0 tx=0).stop 2024-04-24T23:21:27.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:27.037+0000 7f0499600640 1 -- 172.21.15.6:0/774261690 shutdown_connections 2024-04-24T23:21:27.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:27.037+0000 7f0499600640 1 --2- 172.21.15.6:0/774261690 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f047406bf00 0x7f047406e3c0 secure :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f04800095d0 tx=0x7f0480006b30 comp rx=0 tx=0).stop 2024-04-24T23:21:27.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:27.037+0000 7f0499600640 1 --2- 172.21.15.6:0/774261690 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f04940c7080 0x7f04941ea510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:21:27.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:27.037+0000 7f0499600640 1 --2- 172.21.15.6:0/774261690 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0494154b70 0x7f0494149520 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:21:27.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:27.037+0000 7f0499600640 1 --2- 172.21.15.6:0/774261690 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0494071000 0x7f04941f6900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:21:27.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:27.037+0000 7f0499600640 1 -- 172.21.15.6:0/774261690 >> 172.21.15.6:0/774261690 conn(0x7f049406b0e0 msgr2=0x7f0494151190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:21:27.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:27.038+0000 7f0499600640 1 -- 172.21.15.6:0/774261690 shutdown_connections 2024-04-24T23:21:27.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:27.038+0000 7f0499600640 1 -- 172.21.15.6:0/774261690 wait complete. 2024-04-24T23:21:27.068 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-24T23:21:29.069 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:21:29.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:29.395+0000 7fae5dc00640 1 Processor -- start 2024-04-24T23:21:29.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:29.395+0000 7fae5dc00640 1 -- start start 2024-04-24T23:21:29.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:29.397+0000 7fae5dc00640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fae580c7dc0 0x7fae58072530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:21:29.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:29.397+0000 7fae5dc00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fae5816abf0 0x7fae5816b040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:21:29.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:29.397+0000 7fae5dc00640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fae580c74c0 con 0x7fae58167180 2024-04-24T23:21:29.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:29.397+0000 7fae5dc00640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7fae58168570 con 0x7fae580c7dc0 2024-04-24T23:21:29.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:29.397+0000 7fae5dc00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fae580720a0 con 0x7fae5816abf0 2024-04-24T23:21:29.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:29.397+0000 7fae56000640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fae580c7dc0 0x7fae58072530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:21:29.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:29.397+0000 7fae56000640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fae580c7dc0 0x7fae58072530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:49584/0 (socket says 172.21.15.6:49584) 2024-04-24T23:21:29.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:29.397+0000 7fae56000640 1 -- 172.21.15.6:0/2329437116 learned_addr learned my addr 172.21.15.6:0/2329437116 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:21:29.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:29.398+0000 7fae55600640 1 -- 172.21.15.6:0/2329437116 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3166661360 0 0) 0x7fae580c74c0 con 0x7fae58167180 2024-04-24T23:21:29.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:29.422+0000 7fae55600640 1 -- 172.21.15.6:0/2329437116 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fae3c003500 con 0x7fae58167180 2024-04-24T23:21:29.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:29.398+0000 7fae57400640 1 --2- 172.21.15.6:0/2329437116 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fae5816abf0 0x7fae5816b040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:21:29.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:29.423+0000 7fae55600640 1 -- 172.21.15.6:0/2329437116 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1208028605 0 0) 0x7fae3c003500 con 0x7fae58167180 2024-04-24T23:21:29.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:29.423+0000 7fae55600640 1 -- 172.21.15.6:0/2329437116 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fae5816abf0 msgr2=0x7fae5816b040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:21:29.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:29.423+0000 7fae55600640 1 --2- 172.21.15.6:0/2329437116 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fae5816abf0 0x7fae5816b040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:21:29.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:29.423+0000 7fae55600640 1 -- 172.21.15.6:0/2329437116 >> v2:172.21.15.190:3300/0 conn(0x7fae580c7dc0 msgr2=0x7fae58072530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:21:29.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:29.423+0000 7fae55600640 1 --2- 172.21.15.6:0/2329437116 >> v2:172.21.15.190:3300/0 conn(0x7fae580c7dc0 0x7fae58072530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:21:29.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:29.423+0000 7fae55600640 1 -- 172.21.15.6:0/2329437116 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae580c71e0 con 0x7fae58167180 2024-04-24T23:21:29.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:29.423+0000 7fae55600640 1 -- 172.21.15.6:0/2329437116 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3463268701 0 0) 0x7fae500019f0 con 0x7fae58167180 2024-04-24T23:21:29.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:29.424+0000 7fae55600640 1 -- 172.21.15.6:0/2329437116 >> v1:172.21.15.190:6789/0 conn(0x7fae58167180 legacy=0x7fae5806edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:21:29.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:29.424+0000 7fae55600640 1 --2- 172.21.15.6:0/2329437116 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fae3c003b10 0x7fae3c003f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:21:29.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:29.424+0000 7fae55600640 1 --2- 172.21.15.6:0/2329437116 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fae5816abf0 0x7fae3c0045b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:21:29.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:29.424+0000 7fae55600640 1 --2- 172.21.15.6:0/2329437116 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fae3c004c30 0x7fae3c0050b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:21:29.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:29.424+0000 7fae55600640 1 -- 172.21.15.6:0/2329437116 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fae580720a0 con 0x7fae3c004c30 2024-04-24T23:21:29.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:29.424+0000 7fae55600640 1 -- 172.21.15.6:0/2329437116 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fae3c00dc70 con 0x7fae3c003b10 2024-04-24T23:21:29.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:29.424+0000 7fae55600640 1 -- 172.21.15.6:0/2329437116 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fae3c00de70 con 0x7fae5816abf0 2024-04-24T23:21:29.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:29.425+0000 7fae56000640 1 --2- 172.21.15.6:0/2329437116 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fae5816abf0 0x7fae3c0045b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:21:29.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:29.425+0000 7fae56a00640 1 --2- 172.21.15.6:0/2329437116 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fae3c003b10 0x7fae3c003f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:21:29.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:29.426+0000 7fae57400640 1 --2- 172.21.15.6:0/2329437116 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fae3c004c30 0x7fae3c0050b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:21:29.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:29.427+0000 7fae56000640 1 -- 172.21.15.6:0/2329437116 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fae3c003b10 msgr2=0x7fae3c003f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:21:29.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:29.427+0000 7fae56000640 1 --2- 172.21.15.6:0/2329437116 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fae3c003b10 0x7fae3c003f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:21:29.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:29.427+0000 7fae56000640 1 -- 172.21.15.6:0/2329437116 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fae3c004c30 msgr2=0x7fae3c0050b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:21:29.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:29.427+0000 7fae56000640 1 --2- 172.21.15.6:0/2329437116 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fae3c004c30 0x7fae3c0050b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:21:29.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:29.427+0000 7fae56000640 1 -- 172.21.15.6:0/2329437116 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae580c71e0 con 0x7fae5816abf0 2024-04-24T23:21:29.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:29.432+0000 7fae56000640 1 --2- 172.21.15.6:0/2329437116 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fae5816abf0 0x7fae3c0045b0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fae40009470 tx=0x7fae3c003890 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f412a34196f181ed server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:21:29.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:29.432+0000 7fae55600640 1 -- 172.21.15.6:0/2329437116 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae4003c070 con 0x7fae5816abf0 2024-04-24T23:21:29.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:29.432+0000 7fae55600640 1 -- 172.21.15.6:0/2329437116 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae40037410 con 0x7fae5816abf0 2024-04-24T23:21:29.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:29.432+0000 7fae55600640 1 -- 172.21.15.6:0/2329437116 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae400090c0 con 0x7fae5816abf0 2024-04-24T23:21:29.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:29.433+0000 7fae5dc00640 1 -- 172.21.15.6:0/2329437116 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fae5816abf0 msgr2=0x7fae3c0045b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:21:29.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:29.433+0000 7fae5dc00640 1 --2- 172.21.15.6:0/2329437116 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fae5816abf0 0x7fae3c0045b0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fae40009470 tx=0x7fae3c003890 comp rx=0 tx=0).stop 2024-04-24T23:21:29.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:29.433+0000 7fae57400640 1 -- 172.21.15.6:0/2329437116 reap_dead start 2024-04-24T23:21:29.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:29.434+0000 7fae5dc00640 1 -- 172.21.15.6:0/2329437116 shutdown_connections 2024-04-24T23:21:29.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:29.434+0000 7fae5dc00640 1 -- 172.21.15.6:0/2329437116 >> 172.21.15.6:0/2329437116 conn(0x7fae5806b0e0 msgr2=0x7fae5806b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:21:29.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:29.434+0000 7fae5dc00640 1 -- 172.21.15.6:0/2329437116 shutdown_connections 2024-04-24T23:21:29.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:29.434+0000 7fae5dc00640 1 -- 172.21.15.6:0/2329437116 wait complete. 2024-04-24T23:21:29.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:29.438+0000 7fae5dc00640 1 Processor -- start 2024-04-24T23:21:29.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:29.438+0000 7fae5dc00640 1 -- start start 2024-04-24T23:21:29.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:29.440+0000 7fae5dc00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fae5816abf0 0x7fae580c5d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:21:29.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:29.440+0000 7fae5dc00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fae5816a450 0x7fae5815ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:21:29.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:29.440+0000 7fae5dc00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fae58160060 0x7fae581604e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:21:29.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:29.440+0000 7fae5dc00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fae58071d30 con 0x7fae5816a450 2024-04-24T23:21:29.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:29.440+0000 7fae5dc00640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fae58071a50 con 0x7fae58160060 2024-04-24T23:21:29.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:29.440+0000 7fae5dc00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fae580c6e10 con 0x7fae5816abf0 2024-04-24T23:21:29.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:29.441+0000 7fae56a00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fae5816abf0 0x7fae580c5d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:21:29.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:29.441+0000 7fae56a00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fae5816abf0 0x7fae580c5d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.6:41990/0 (socket says 172.21.15.6:41990) 2024-04-24T23:21:29.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:29.441+0000 7fae56a00640 1 -- 172.21.15.6:0/2042592597 learned_addr learned my addr 172.21.15.6:0/2042592597 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:21:29.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:29.441+0000 7fae57400640 1 --2- 172.21.15.6:0/2042592597 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fae58160060 0x7fae581604e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:21:29.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:29.442+0000 7fae57400640 1 -- 172.21.15.6:0/2042592597 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fae5816abf0 msgr2=0x7fae580c5d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:21:29.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:29.442+0000 7fae57400640 1 --2- 172.21.15.6:0/2042592597 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fae5816abf0 0x7fae580c5d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:21:29.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:29.442+0000 7fae57400640 1 -- 172.21.15.6:0/2042592597 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fae5816a450 msgr2=0x7fae5815ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:21:29.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:29.442+0000 7fae57400640 1 --2- 172.21.15.6:0/2042592597 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fae5816a450 0x7fae5815ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:21:29.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:29.442+0000 7fae57400640 1 -- 172.21.15.6:0/2042592597 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae58160a20 con 0x7fae58160060 2024-04-24T23:21:29.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:29.442+0000 7fae57400640 1 --2- 172.21.15.6:0/2042592597 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fae58160060 0x7fae581604e0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fae5806bf20 tx=0x7fae48011920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:21:29.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:29.442+0000 7fae4ea00640 1 -- 172.21.15.6:0/2042592597 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae48015930 con 0x7fae58160060 2024-04-24T23:21:29.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:29.443+0000 7fae4ea00640 1 -- 172.21.15.6:0/2042592597 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae4801c070 con 0x7fae58160060 2024-04-24T23:21:29.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:29.443+0000 7fae4ea00640 1 -- 172.21.15.6:0/2042592597 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae48015d30 con 0x7fae58160060 2024-04-24T23:21:29.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:29.443+0000 7fae5dc00640 1 -- 172.21.15.6:0/2042592597 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae581fd150 con 0x7fae58160060 2024-04-24T23:21:29.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:29.443+0000 7fae5dc00640 1 -- 172.21.15.6:0/2042592597 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae581fd710 con 0x7fae58160060 2024-04-24T23:21:29.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:29.445+0000 7fae5dc00640 1 -- 172.21.15.6:0/2042592597 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fae580767b0 con 0x7fae58160060 2024-04-24T23:21:29.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:29.448+0000 7fae4ea00640 1 -- 172.21.15.6:0/2042592597 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fae48020410 con 0x7fae58160060 2024-04-24T23:21:29.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:29.451+0000 7fae4ea00640 1 --2- 172.21.15.6:0/2042592597 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fae3406bf00 0x7fae3406e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:21:29.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:29.451+0000 7fae4ea00640 1 -- 172.21.15.6:0/2042592597 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6762+0+0 (secure 0 0 0) 0x7fae4808ae50 con 0x7fae58160060 2024-04-24T23:21:29.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:29.452+0000 7fae56a00640 1 --2- 172.21.15.6:0/2042592597 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fae3406bf00 0x7fae3406e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:21:29.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:29.453+0000 7fae56a00640 1 --2- 172.21.15.6:0/2042592597 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fae3406bf00 0x7fae3406e3c0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fae50001330 tx=0x7fae50009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:21:29.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:29.454+0000 7fae4ea00640 1 -- 172.21.15.6:0/2042592597 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fae48025020 con 0x7fae58160060 2024-04-24T23:21:29.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:29.965+0000 7fae5dc00640 1 -- 172.21.15.6:0/2042592597 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fae580c7970 con 0x7fae58160060 2024-04-24T23:21:29.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:29.968+0000 7fae4ea00640 1 -- 172.21.15.6:0/2042592597 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+6086 (secure 0 0 0) 0x7fae4808c090 con 0x7fae58160060 2024-04-24T23:21:29.971 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:21:29.972 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:21:28.204416+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4282,"mds_2":4235,"mds_3":4232,"mds_4":5257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4232":{"gid":4232,"name":"b","rank":3,"incarnation":15,"state":"up:active","state_seq":15,"addr":"172.21.15.190:6838/2032280158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2032280158},{"type":"v1","addr":"172.21.15.190:6838","nonce":2032280158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4235":{"gid":4235,"name":"d","rank":2,"incarnation":13,"state":"up:active","state_seq":14,"addr":"172.21.15.190:6839/1572872451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":1572872451},{"type":"v1","addr":"172.21.15.190:6839","nonce":1572872451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4282":{"gid":4282,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":12,"addr":"172.21.15.190:6837/580315269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":580315269},{"type":"v1","addr":"172.21.15.190:6837","nonce":580315269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/489092393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":489092393},{"type":"v1","addr":"172.21.15.6:6835","nonce":489092393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5257":{"gid":5257,"name":"e","rank":4,"incarnation":25,"state":"up:active","state_seq":43,"addr":"172.21.15.6:6837/3240971655","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":3240971655},{"type":"v1","addr":"172.21.15.6:6837","nonce":3240971655}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4306,"qdb_cluster":[4232,4235,4282,5257,4306]},"id":1}]} 2024-04-24T23:21:29.972 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 29 2024-04-24T23:21:29.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:29.974+0000 7fae5dc00640 1 -- 172.21.15.6:0/2042592597 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fae3406bf00 msgr2=0x7fae3406e3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:21:29.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:29.974+0000 7fae5dc00640 1 --2- 172.21.15.6:0/2042592597 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fae3406bf00 0x7fae3406e3c0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fae50001330 tx=0x7fae50009040 comp rx=0 tx=0).stop 2024-04-24T23:21:29.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:29.974+0000 7fae5dc00640 1 -- 172.21.15.6:0/2042592597 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fae58160060 msgr2=0x7fae581604e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:21:29.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:29.974+0000 7fae5dc00640 1 --2- 172.21.15.6:0/2042592597 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fae58160060 0x7fae581604e0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fae5806bf20 tx=0x7fae48011920 comp rx=0 tx=0).stop 2024-04-24T23:21:29.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:29.974+0000 7fae5dc00640 1 -- 172.21.15.6:0/2042592597 shutdown_connections 2024-04-24T23:21:29.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:29.974+0000 7fae5dc00640 1 --2- 172.21.15.6:0/2042592597 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fae3406bf00 0x7fae3406e3c0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:21:29.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:29.974+0000 7fae5dc00640 1 --2- 172.21.15.6:0/2042592597 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fae58160060 0x7fae581604e0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:21:29.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:29.974+0000 7fae5dc00640 1 --2- 172.21.15.6:0/2042592597 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fae5816a450 0x7fae5815ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:21:29.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:29.974+0000 7fae5dc00640 1 --2- 172.21.15.6:0/2042592597 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fae5816abf0 0x7fae580c5d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:21:29.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:29.974+0000 7fae5dc00640 1 -- 172.21.15.6:0/2042592597 >> 172.21.15.6:0/2042592597 conn(0x7fae5806b0e0 msgr2=0x7fae5806b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:21:29.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:29.975+0000 7fae5dc00640 1 -- 172.21.15.6:0/2042592597 shutdown_connections 2024-04-24T23:21:29.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:29.976+0000 7fae5dc00640 1 -- 172.21.15.6:0/2042592597 wait complete. 2024-04-24T23:21:30.023 INFO:tasks.mds_thrash.fs.[cephfs]:mds.e has gained rank=4, replacing gid=4197 2024-04-24T23:21:30.023 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 1 secs before reviving mds.a 2024-04-24T23:21:31.024 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.a 2024-04-24T23:21:31.024 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-24T23:21:31.024 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-24T23:21:31.030 INFO:tasks.ceph.mds.a:Started 2024-04-24T23:21:31.031 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:21:31.947 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-04-24T23:21:31.945+0000 7fef1ce00640 -1 mds.pinger is_rank_lagging: rank=4 was never sent ping request. 2024-04-24T23:21:32.145 INFO:tasks.ceph.mds.a.smithi006.stdout:starting mds.a at 2024-04-24T23:21:32.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:32.198+0000 7f79c3e00640 1 Processor -- start 2024-04-24T23:21:32.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:32.198+0000 7f79c3e00640 1 -- start start 2024-04-24T23:21:32.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:32.202+0000 7f79c3e00640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f79c4152940 0x7f79c40ca830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:21:32.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:32.203+0000 7f79c3e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f79c40cb840 0x7f79c415c200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:21:32.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:32.203+0000 7f79c3400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f79c40cb840 0x7f79c415c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:21:32.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:32.204+0000 7f79c3400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f79c40cb840 0x7f79c415c200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36186/0 (socket says 172.21.15.6:36186) 2024-04-24T23:21:32.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:32.204+0000 7f79c3400640 1 -- 172.21.15.6:0/2187007078 learned_addr learned my addr 172.21.15.6:0/2187007078 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:21:32.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:32.204+0000 7f79c3e00640 1 -- 172.21.15.6:0/2187007078 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f79c41587a0 con 0x7f79c4152d20 2024-04-24T23:21:32.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:32.204+0000 7f79c3e00640 1 -- 172.21.15.6:0/2187007078 --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f79c415bd70 con 0x7f79c40cb840 2024-04-24T23:21:32.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:32.204+0000 7f79c3e00640 1 -- 172.21.15.6:0/2187007078 --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f79c4158a40 con 0x7f79c4152940 2024-04-24T23:21:32.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:32.205+0000 7f79c2a00640 1 --2- 172.21.15.6:0/2187007078 >> v2:172.21.15.190:3300/0 conn(0x7f79c4152940 0x7f79c40ca830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:21:32.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:32.205+0000 7f79c3400640 1 -- 172.21.15.6:0/2187007078 >> v2:172.21.15.190:3300/0 conn(0x7f79c4152940 msgr2=0x7f79c40ca830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:21:32.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:32.205+0000 7f79c3400640 1 --2- 172.21.15.6:0/2187007078 >> v2:172.21.15.190:3300/0 conn(0x7f79c4152940 0x7f79c40ca830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:21:32.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:32.205+0000 7f79c3400640 1 -- 172.21.15.6:0/2187007078 >> v1:172.21.15.6:6789/0 conn(0x7f79c4152d20 legacy=0x7f79c4070b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:21:32.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:32.205+0000 7f79c3400640 1 -- 172.21.15.6:0/2187007078 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79c4158ce0 con 0x7f79c40cb840 2024-04-24T23:21:32.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:32.208+0000 7f79c3400640 1 --2- 172.21.15.6:0/2187007078 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f79c40cb840 0x7f79c415c200 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f79c4071730 tx=0x7f79bc02a910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f4e43949936306e server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:21:32.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:32.209+0000 7f79c1600640 1 -- 172.21.15.6:0/2187007078 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79bc031420 con 0x7f79c40cb840 2024-04-24T23:21:32.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:32.210+0000 7f79c1600640 1 -- 172.21.15.6:0/2187007078 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f79bc0315c0 con 0x7f79c40cb840 2024-04-24T23:21:32.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:32.211+0000 7f79c1600640 1 -- 172.21.15.6:0/2187007078 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79bc031420 con 0x7f79c40cb840 2024-04-24T23:21:32.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:32.211+0000 7f79c3e00640 1 -- 172.21.15.6:0/2187007078 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f79c40cb840 msgr2=0x7f79c415c200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:21:32.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:32.211+0000 7f79c3e00640 1 --2- 172.21.15.6:0/2187007078 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f79c40cb840 0x7f79c415c200 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f79c4071730 tx=0x7f79bc02a910 comp rx=0 tx=0).stop 2024-04-24T23:21:32.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:32.214+0000 7f79c3e00640 1 -- 172.21.15.6:0/2187007078 shutdown_connections 2024-04-24T23:21:32.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:32.214+0000 7f79c3e00640 1 --2- 172.21.15.6:0/2187007078 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f79c40cb840 0x7f79c415c200 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:21:32.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:32.214+0000 7f79c3e00640 1 --2- 172.21.15.6:0/2187007078 >> v2:172.21.15.190:3300/0 conn(0x7f79c4152940 0x7f79c40ca830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:21:32.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:32.214+0000 7f79c3e00640 1 -- 172.21.15.6:0/2187007078 >> 172.21.15.6:0/2187007078 conn(0x7f79c406f500 msgr2=0x7f79c406b0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:21:32.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:32.214+0000 7f79c3e00640 1 -- 172.21.15.6:0/2187007078 shutdown_connections 2024-04-24T23:21:32.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:32.214+0000 7f79c3e00640 1 -- 172.21.15.6:0/2187007078 wait complete. 2024-04-24T23:21:32.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:32.217+0000 7f79c3e00640 1 Processor -- start 2024-04-24T23:21:32.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:32.218+0000 7f79c3e00640 1 -- start start 2024-04-24T23:21:32.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:32.221+0000 7f79c3e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f79c40cb840 0x7f79c40ce5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:21:32.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:32.221+0000 7f79c3e00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f79c4152940 0x7f79c40cc550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:21:32.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:32.221+0000 7f79c3e00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f79c40cc0a0 0x7f79c406d240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:21:32.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:32.222+0000 7f79c3e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f79c4159580 con 0x7f79c40cb840 2024-04-24T23:21:32.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:32.222+0000 7f79c3e00640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f79c41592a0 con 0x7f79c4152940 2024-04-24T23:21:32.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:32.222+0000 7f79c3e00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f79c40cad70 con 0x7f79c40cc0a0 2024-04-24T23:21:32.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:32.222+0000 7f79c2000640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f79c4152940 0x7f79c40cc550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:21:32.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:32.223+0000 7f79c2000640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f79c4152940 0x7f79c40cc550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:49638/0 (socket says 172.21.15.6:49638) 2024-04-24T23:21:32.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:32.223+0000 7f79c2000640 1 -- 172.21.15.6:0/1080935085 learned_addr learned my addr 172.21.15.6:0/1080935085 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:21:32.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:32.224+0000 7f79c2a00640 1 --2- 172.21.15.6:0/1080935085 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f79c40cb840 0x7f79c40ce5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:21:32.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:32.224+0000 7f79c3400640 1 --2- 172.21.15.6:0/1080935085 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f79c40cc0a0 0x7f79c406d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:21:32.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:32.224+0000 7f79c2000640 1 -- 172.21.15.6:0/1080935085 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f79c40cc0a0 msgr2=0x7f79c406d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:21:32.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:32.224+0000 7f79c2000640 1 --2- 172.21.15.6:0/1080935085 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f79c40cc0a0 0x7f79c406d240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:21:32.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:32.224+0000 7f79c2000640 1 -- 172.21.15.6:0/1080935085 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f79c40cb840 msgr2=0x7f79c40ce5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:21:32.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:32.224+0000 7f79c2000640 1 --2- 172.21.15.6:0/1080935085 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f79c40cb840 0x7f79c40ce5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:21:32.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:32.225+0000 7f79c2000640 1 -- 172.21.15.6:0/1080935085 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79c41f5820 con 0x7f79c4152940 2024-04-24T23:21:32.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:32.225+0000 7f79c2a00640 1 --2- 172.21.15.6:0/1080935085 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f79c40cb840 0x7f79c40ce5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:21:32.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:32.225+0000 7f79c2000640 1 --2- 172.21.15.6:0/1080935085 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f79c4152940 0x7f79c40cc550 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f79b000c430 tx=0x7f79b000c8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:21:32.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:32.225+0000 7f79baa00640 1 -- 172.21.15.6:0/1080935085 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79b0011790 con 0x7f79c4152940 2024-04-24T23:21:32.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:32.226+0000 7f79c3e00640 1 -- 172.21.15.6:0/1080935085 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79c41f8200 con 0x7f79c4152940 2024-04-24T23:21:32.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:32.226+0000 7f79c3e00640 1 -- 172.21.15.6:0/1080935085 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79c41f4d40 con 0x7f79c4152940 2024-04-24T23:21:32.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:32.226+0000 7f79baa00640 1 -- 172.21.15.6:0/1080935085 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f79b0017070 con 0x7f79c4152940 2024-04-24T23:21:32.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:32.226+0000 7f79baa00640 1 -- 172.21.15.6:0/1080935085 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79b0011bd0 con 0x7f79c4152940 2024-04-24T23:21:32.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:32.229+0000 7f79baa00640 1 -- 172.21.15.6:0/1080935085 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f79b001c460 con 0x7f79c4152940 2024-04-24T23:21:32.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:32.232+0000 7f79baa00640 1 --2- 172.21.15.6:0/1080935085 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f79a406be30 0x7f79a406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:21:32.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:32.232+0000 7f79baa00640 1 -- 172.21.15.6:0/1080935085 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6762+0+0 (secure 0 0 0) 0x7f79b008b020 con 0x7f79c4152940 2024-04-24T23:21:32.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:32.232+0000 7f79c2a00640 1 --2- 172.21.15.6:0/1080935085 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f79a406be30 0x7f79a406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:21:32.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:32.252+0000 7f79c3e00640 1 -- 172.21.15.6:0/1080935085 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f799c005290 con 0x7f79c4152940 2024-04-24T23:21:32.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:32.253+0000 7f79c2a00640 1 --2- 172.21.15.6:0/1080935085 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f79a406be30 0x7f79a406e2f0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f79b400aeb0 tx=0x7f79b4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:21:32.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:32.259+0000 7f79baa00640 1 -- 172.21.15.6:0/1080935085 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f79b00595a0 con 0x7f79c4152940 2024-04-24T23:21:32.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:32.950+0000 7f79c3e00640 1 -- 172.21.15.6:0/1080935085 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f799c005580 con 0x7f79c4152940 2024-04-24T23:21:32.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:32.951+0000 7f79baa00640 1 -- 172.21.15.6:0/1080935085 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+6907 (secure 0 0 0) 0x7f79b0058ea0 con 0x7f79c4152940 2024-04-24T23:21:32.953 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:21:32.953 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5081,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/1190759626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1190759626},{"type":"v1","addr":"172.21.15.6:6834","nonce":1190759626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30}],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:21:28.204416+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4282,"mds_2":4235,"mds_3":4232,"mds_4":5257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4232":{"gid":4232,"name":"b","rank":3,"incarnation":15,"state":"up:active","state_seq":15,"addr":"172.21.15.190:6838/2032280158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2032280158},{"type":"v1","addr":"172.21.15.190:6838","nonce":2032280158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4235":{"gid":4235,"name":"d","rank":2,"incarnation":13,"state":"up:active","state_seq":14,"addr":"172.21.15.190:6839/1572872451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":1572872451},{"type":"v1","addr":"172.21.15.190:6839","nonce":1572872451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4282":{"gid":4282,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":12,"addr":"172.21.15.190:6837/580315269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":580315269},{"type":"v1","addr":"172.21.15.190:6837","nonce":580315269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/489092393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":489092393},{"type":"v1","addr":"172.21.15.6:6835","nonce":489092393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5257":{"gid":5257,"name":"e","rank":4,"incarnation":25,"state":"up:active","state_seq":43,"addr":"172.21.15.6:6837/3240971655","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":3240971655},{"type":"v1","addr":"172.21.15.6:6837","nonce":3240971655}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4306,"qdb_cluster":[4306,5257,4282,4235,4232]},"id":1}]} 2024-04-24T23:21:32.956 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 30 2024-04-24T23:21:32.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:32.954+0000 7f79c3e00640 1 -- 172.21.15.6:0/1080935085 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f79a406be30 msgr2=0x7f79a406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:21:32.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:32.954+0000 7f79c3e00640 1 --2- 172.21.15.6:0/1080935085 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f79a406be30 0x7f79a406e2f0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f79b400aeb0 tx=0x7f79b4008040 comp rx=0 tx=0).stop 2024-04-24T23:21:32.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:32.955+0000 7f79c3e00640 1 -- 172.21.15.6:0/1080935085 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f79c4152940 msgr2=0x7f79c40cc550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:21:32.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:32.955+0000 7f79c3e00640 1 --2- 172.21.15.6:0/1080935085 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f79c4152940 0x7f79c40cc550 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f79b000c430 tx=0x7f79b000c8f0 comp rx=0 tx=0).stop 2024-04-24T23:21:32.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:32.955+0000 7f79c3e00640 1 -- 172.21.15.6:0/1080935085 shutdown_connections 2024-04-24T23:21:32.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:32.955+0000 7f79c3e00640 1 --2- 172.21.15.6:0/1080935085 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f79a406be30 0x7f79a406e2f0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:21:32.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:32.955+0000 7f79c3e00640 1 --2- 172.21.15.6:0/1080935085 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f79c40cc0a0 0x7f79c406d240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:21:32.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:32.955+0000 7f79c3e00640 1 --2- 172.21.15.6:0/1080935085 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f79c4152940 0x7f79c40cc550 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:21:32.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:32.955+0000 7f79c3e00640 1 --2- 172.21.15.6:0/1080935085 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f79c40cb840 0x7f79c40ce5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:21:32.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:32.955+0000 7f79c3e00640 1 -- 172.21.15.6:0/1080935085 >> 172.21.15.6:0/1080935085 conn(0x7f79c406f500 msgr2=0x7f79c415aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:21:32.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:32.955+0000 7f79c3e00640 1 -- 172.21.15.6:0/1080935085 shutdown_connections 2024-04-24T23:21:32.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:21:32.955+0000 7f79c3e00640 1 -- 172.21.15.6:0/1080935085 wait complete. 2024-04-24T23:21:34.566 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a reported in up:standby state 2024-04-24T23:21:34.566 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 12 secs before thrashing 2024-04-24T23:21:36.436 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-24T23:21:36.449 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-24T23:21:46.567 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:22:01.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:01.761+0000 7fea2de00640 1 Processor -- start 2024-04-24T23:22:01.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:01.761+0000 7fea2de00640 1 -- start start 2024-04-24T23:22:01.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:01.763+0000 7fea2de00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fea28073270 0x7fea28150900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:22:01.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:01.763+0000 7fea2de00640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fea28157470 0x7fea280c9110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:22:01.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:01.763+0000 7fea2de00640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fea28150e40 con 0x7fea2806bbd0 2024-04-24T23:22:01.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:01.763+0000 7fea2de00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7fea280c8c50 con 0x7fea28073270 2024-04-24T23:22:01.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:01.763+0000 7fea2de00640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7fea280c8690 con 0x7fea28157470 2024-04-24T23:22:01.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:01.763+0000 7fea26a00640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7fea2806bbd0 0x7fea2815a980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.6:46908/0 (socket says 172.21.15.6:46908) 2024-04-24T23:22:01.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:01.763+0000 7fea26a00640 1 -- 172.21.15.6:0/1220712294 learned_addr learned my addr 172.21.15.6:0/1220712294 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:22:01.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:01.763+0000 7fea27400640 1 --2- 172.21.15.6:0/1220712294 >> v2:172.21.15.190:3300/0 conn(0x7fea28157470 0x7fea280c9110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:22:01.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:01.764+0000 7fea25600640 1 -- 172.21.15.6:0/1220712294 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1802739407 0 0) 0x7fea10001110 con 0x7fea2806bbd0 2024-04-24T23:22:01.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:01.764+0000 7fea25600640 1 -- 172.21.15.6:0/1220712294 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fea0c003520 con 0x7fea2806bbd0 2024-04-24T23:22:01.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:01.764+0000 7fea27400640 1 -- 172.21.15.6:0/1220712294 >> v2:172.21.15.6:3300/0 conn(0x7fea28073270 msgr2=0x7fea28150900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:22:01.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:01.764+0000 7fea27400640 1 --2- 172.21.15.6:0/1220712294 >> v2:172.21.15.6:3300/0 conn(0x7fea28073270 0x7fea28150900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:22:01.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:01.764+0000 7fea27400640 1 -- 172.21.15.6:0/1220712294 >> v1:172.21.15.190:6789/0 conn(0x7fea2806bbd0 legacy=0x7fea2815a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:22:01.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:01.764+0000 7fea27400640 1 -- 172.21.15.6:0/1220712294 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea280c8970 con 0x7fea28157470 2024-04-24T23:22:01.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:01.765+0000 7fea27400640 1 --2- 172.21.15.6:0/1220712294 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fea28157470 0x7fea280c9110 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fea280c7150 tx=0x7fea2002a680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8e9d82c953774894 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:22:01.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:01.785+0000 7fea25600640 1 -- 172.21.15.6:0/1220712294 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea20007740 con 0x7fea28157470 2024-04-24T23:22:01.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:01.785+0000 7fea25600640 1 -- 172.21.15.6:0/1220712294 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea200078e0 con 0x7fea28157470 2024-04-24T23:22:01.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:01.785+0000 7fea25600640 1 -- 172.21.15.6:0/1220712294 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea20007cb0 con 0x7fea28157470 2024-04-24T23:22:01.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:01.786+0000 7fea2de00640 1 -- 172.21.15.6:0/1220712294 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fea28157470 msgr2=0x7fea280c9110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:22:01.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:01.786+0000 7fea2de00640 1 --2- 172.21.15.6:0/1220712294 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fea28157470 0x7fea280c9110 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fea280c7150 tx=0x7fea2002a680 comp rx=0 tx=0).stop 2024-04-24T23:22:01.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:01.788+0000 7fea2de00640 1 -- 172.21.15.6:0/1220712294 shutdown_connections 2024-04-24T23:22:01.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:01.788+0000 7fea2de00640 1 --2- 172.21.15.6:0/1220712294 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fea28157470 0x7fea280c9110 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:22:01.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:01.788+0000 7fea2de00640 1 --2- 172.21.15.6:0/1220712294 >> v2:172.21.15.6:3300/0 conn(0x7fea28073270 0x7fea28150900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:22:01.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:01.788+0000 7fea2de00640 1 -- 172.21.15.6:0/1220712294 >> 172.21.15.6:0/1220712294 conn(0x7fea2806b0e0 msgr2=0x7fea2806b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:22:01.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:01.793+0000 7fea2de00640 1 -- 172.21.15.6:0/1220712294 shutdown_connections 2024-04-24T23:22:01.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:01.793+0000 7fea2de00640 1 -- 172.21.15.6:0/1220712294 wait complete. 2024-04-24T23:22:01.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:01.805+0000 7fea2de00640 1 Processor -- start 2024-04-24T23:22:01.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:01.806+0000 7fea2de00640 1 -- start start 2024-04-24T23:22:01.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:01.823+0000 7fea2de00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fea2806bbd0 0x7fea280cbcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:22:01.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:01.824+0000 7fea2de00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fea28073270 0x7fea2806e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:22:01.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:01.824+0000 7fea2de00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fea28157470 0x7fea281535b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:22:01.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:01.825+0000 7fea26a00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fea2806bbd0 0x7fea280cbcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:22:01.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:01.825+0000 7fea26a00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fea2806bbd0 0x7fea280cbcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.6:38634/0 (socket says 172.21.15.6:38634) 2024-04-24T23:22:01.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:01.825+0000 7fea26a00640 1 -- 172.21.15.6:0/3563222788 learned_addr learned my addr 172.21.15.6:0/3563222788 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:22:01.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:01.825+0000 7fea27400640 1 --2- 172.21.15.6:0/3563222788 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fea28157470 0x7fea281535b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:22:01.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:01.826+0000 7fea2de00640 1 -- 172.21.15.6:0/3563222788 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fea280c7a70 con 0x7fea28157470 2024-04-24T23:22:01.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:01.826+0000 7fea2de00640 1 -- 172.21.15.6:0/3563222788 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fea2815a460 con 0x7fea28073270 2024-04-24T23:22:01.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:01.826+0000 7fea2de00640 1 -- 172.21.15.6:0/3563222788 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fea280c45f0 con 0x7fea2806bbd0 2024-04-24T23:22:01.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:01.827+0000 7fea26000640 1 --2- 172.21.15.6:0/3563222788 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fea28073270 0x7fea2806e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:22:01.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:01.828+0000 7fea26a00640 1 -- 172.21.15.6:0/3563222788 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fea28073270 msgr2=0x7fea2806e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:22:01.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:01.828+0000 7fea26a00640 1 --2- 172.21.15.6:0/3563222788 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fea28073270 0x7fea2806e3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:22:01.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:01.828+0000 7fea26a00640 1 -- 172.21.15.6:0/3563222788 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fea28157470 msgr2=0x7fea281535b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:22:01.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:01.828+0000 7fea26a00640 1 --2- 172.21.15.6:0/3563222788 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fea28157470 0x7fea281535b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:22:01.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:01.828+0000 7fea26a00640 1 -- 172.21.15.6:0/3563222788 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea28149bf0 con 0x7fea2806bbd0 2024-04-24T23:22:01.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:01.829+0000 7fea26a00640 1 --2- 172.21.15.6:0/3563222788 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fea2806bbd0 0x7fea280cbcc0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fea1000c730 tx=0x7fea1000cb30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:22:01.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:01.830+0000 7fea1f400640 1 -- 172.21.15.6:0/3563222788 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea10017070 con 0x7fea2806bbd0 2024-04-24T23:22:01.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:01.830+0000 7fea1f400640 1 -- 172.21.15.6:0/3563222788 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea10005ab0 con 0x7fea2806bbd0 2024-04-24T23:22:01.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:01.830+0000 7fea1f400640 1 -- 172.21.15.6:0/3563222788 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea10007e20 con 0x7fea2806bbd0 2024-04-24T23:22:01.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:01.832+0000 7fea2de00640 1 -- 172.21.15.6:0/3563222788 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea28149e90 con 0x7fea2806bbd0 2024-04-24T23:22:01.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:01.832+0000 7fea2de00640 1 -- 172.21.15.6:0/3563222788 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fea2814a450 con 0x7fea2806bbd0 2024-04-24T23:22:01.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:01.838+0000 7fea1f400640 1 -- 172.21.15.6:0/3563222788 <== mon.2 v2:172.21.15.190:3301/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fea1001a070 con 0x7fea2806bbd0 2024-04-24T23:22:01.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:01.840+0000 7fea2de00640 1 -- 172.21.15.6:0/3563222788 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea280cd0f0 con 0x7fea2806bbd0 2024-04-24T23:22:01.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:01.842+0000 7fea1f400640 1 --2- 172.21.15.6:0/3563222788 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fea0406bf00 0x7fea0406e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:22:01.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:01.842+0000 7fea1f400640 1 -- 172.21.15.6:0/3563222788 <== mon.2 v2:172.21.15.190:3301/0 5 ==== osd_map(28..28 src has 1..28) ==== 6762+0+0 (secure 0 0 0) 0x7fea1008a580 con 0x7fea2806bbd0 2024-04-24T23:22:01.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:01.851+0000 7fea1f400640 1 -- 172.21.15.6:0/3563222788 <== mon.2 v2:172.21.15.190:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fea280cd0f0 con 0x7fea2806bbd0 2024-04-24T23:22:01.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:01.853+0000 7fea26000640 1 --2- 172.21.15.6:0/3563222788 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fea0406bf00 0x7fea0406e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:22:01.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:01.855+0000 7fea26000640 1 --2- 172.21.15.6:0/3563222788 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fea0406bf00 0x7fea0406e3c0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fea280c7760 tx=0x7fea18009110 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:22:02.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:02.488+0000 7fea2de00640 1 -- 172.21.15.6:0/3563222788 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fea280767b0 con 0x7fea2806bbd0 2024-04-24T23:22:02.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:02.494+0000 7fea1f400640 1 -- 172.21.15.6:0/3563222788 <== mon.2 v2:172.21.15.190:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+6907 (secure 0 0 0) 0x7fea28150e40 con 0x7fea2806bbd0 2024-04-24T23:22:02.499 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:22:02.499 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5081,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/1190759626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1190759626},{"type":"v1","addr":"172.21.15.6:6834","nonce":1190759626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30}],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:21:28.204416+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4282,"mds_2":4235,"mds_3":4232,"mds_4":5257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4232":{"gid":4232,"name":"b","rank":3,"incarnation":15,"state":"up:active","state_seq":15,"addr":"172.21.15.190:6838/2032280158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2032280158},{"type":"v1","addr":"172.21.15.190:6838","nonce":2032280158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4235":{"gid":4235,"name":"d","rank":2,"incarnation":13,"state":"up:active","state_seq":14,"addr":"172.21.15.190:6839/1572872451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":1572872451},{"type":"v1","addr":"172.21.15.190:6839","nonce":1572872451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4282":{"gid":4282,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":12,"addr":"172.21.15.190:6837/580315269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":580315269},{"type":"v1","addr":"172.21.15.190:6837","nonce":580315269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/489092393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":489092393},{"type":"v1","addr":"172.21.15.6:6835","nonce":489092393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5257":{"gid":5257,"name":"e","rank":4,"incarnation":25,"state":"up:active","state_seq":43,"addr":"172.21.15.6:6837/3240971655","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":3240971655},{"type":"v1","addr":"172.21.15.6:6837","nonce":3240971655}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4306,"qdb_cluster":[4306,5257,4282,4235,4232]},"id":1}]} 2024-04-24T23:22:02.499 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 30 2024-04-24T23:22:02.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:02.498+0000 7fea2de00640 1 -- 172.21.15.6:0/3563222788 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fea0406bf00 msgr2=0x7fea0406e3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:22:02.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:02.498+0000 7fea2de00640 1 --2- 172.21.15.6:0/3563222788 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fea0406bf00 0x7fea0406e3c0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fea280c7760 tx=0x7fea18009110 comp rx=0 tx=0).stop 2024-04-24T23:22:02.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:02.498+0000 7fea2de00640 1 -- 172.21.15.6:0/3563222788 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fea2806bbd0 msgr2=0x7fea280cbcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:22:02.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:02.498+0000 7fea2de00640 1 --2- 172.21.15.6:0/3563222788 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fea2806bbd0 0x7fea280cbcc0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fea1000c730 tx=0x7fea1000cb30 comp rx=0 tx=0).stop 2024-04-24T23:22:02.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:02.500+0000 7fea2de00640 1 -- 172.21.15.6:0/3563222788 shutdown_connections 2024-04-24T23:22:02.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:02.500+0000 7fea2de00640 1 --2- 172.21.15.6:0/3563222788 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fea0406bf00 0x7fea0406e3c0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:22:02.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:02.500+0000 7fea2de00640 1 --2- 172.21.15.6:0/3563222788 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fea28157470 0x7fea281535b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:22:02.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:02.500+0000 7fea2de00640 1 --2- 172.21.15.6:0/3563222788 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fea28073270 0x7fea2806e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:22:02.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:02.500+0000 7fea2de00640 1 --2- 172.21.15.6:0/3563222788 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fea2806bbd0 0x7fea280cbcc0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:22:02.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:02.500+0000 7fea2de00640 1 -- 172.21.15.6:0/3563222788 >> 172.21.15.6:0/3563222788 conn(0x7fea2806b0e0 msgr2=0x7fea28070f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:22:02.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:02.505+0000 7fea2de00640 1 -- 172.21.15.6:0/3563222788 shutdown_connections 2024-04-24T23:22:02.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:02.505+0000 7fea2de00640 1 -- 172.21.15.6:0/3563222788 wait complete. 2024-04-24T23:22:02.542 INFO:tasks.mds_thrash.fs.[cephfs]:thrashing max_mds: 5 -> 2 2024-04-24T23:22:02.542 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 2 2024-04-24T23:22:06.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:06.073+0000 7fc845c00640 1 Processor -- start 2024-04-24T23:22:06.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:06.074+0000 7fc845c00640 1 -- start start 2024-04-24T23:22:06.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:06.077+0000 7fc845c00640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fc840150d50 0x7fc84015aab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:22:06.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:06.078+0000 7fc845c00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc84006ec50 con 0x7fc84006e3a0 2024-04-24T23:22:06.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:06.078+0000 7fc83ea00640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fc840150d50 0x7fc84015aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:22:06.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:06.078+0000 7fc83ea00640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fc840150d50 0x7fc84015aab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:48552/0 (socket says 172.21.15.6:48552) 2024-04-24T23:22:06.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:06.078+0000 7fc83ea00640 1 -- 172.21.15.6:0/25518051 learned_addr learned my addr 172.21.15.6:0/25518051 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:22:06.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:06.079+0000 7fc845c00640 1 -- 172.21.15.6:0/25518051 --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc84006ef30 con 0x7fc8400c8d30 2024-04-24T23:22:06.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:06.079+0000 7fc845c00640 1 -- 172.21.15.6:0/25518051 --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7fc84015aff0 con 0x7fc840150d50 2024-04-24T23:22:06.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:06.081+0000 7fc83d600640 1 -- 172.21.15.6:0/25518051 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 82918764 0 0) 0x7fc84006ef30 con 0x7fc8400c8d30 2024-04-24T23:22:06.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:06.081+0000 7fc83d600640 1 -- 172.21.15.6:0/25518051 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc824003520 con 0x7fc8400c8d30 2024-04-24T23:22:06.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:06.081+0000 7fc83d600640 1 -- 172.21.15.6:0/25518051 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3039840278 0 0) 0x7fc84006ec50 con 0x7fc84006e3a0 2024-04-24T23:22:06.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:06.081+0000 7fc83d600640 1 -- 172.21.15.6:0/25518051 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc84006ef30 con 0x7fc84006e3a0 2024-04-24T23:22:06.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:06.082+0000 7fc83d600640 1 -- 172.21.15.6:0/25518051 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4206965037 0 0) 0x7fc824003520 con 0x7fc8400c8d30 2024-04-24T23:22:06.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:06.082+0000 7fc83d600640 1 -- 172.21.15.6:0/25518051 >> v2:172.21.15.190:3300/0 conn(0x7fc840150d50 msgr2=0x7fc84015aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:22:06.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:06.082+0000 7fc83d600640 1 --2- 172.21.15.6:0/25518051 >> v2:172.21.15.190:3300/0 conn(0x7fc840150d50 0x7fc84015aab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:22:06.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:06.082+0000 7fc83d600640 1 -- 172.21.15.6:0/25518051 >> v1:172.21.15.6:6789/0 conn(0x7fc84006e3a0 legacy=0x7fc8400c6e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:22:06.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:06.083+0000 7fc83d600640 1 -- 172.21.15.6:0/25518051 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc84006f210 con 0x7fc8400c8d30 2024-04-24T23:22:06.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:06.083+0000 7fc83d600640 1 -- 172.21.15.6:0/25518051 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3463268701 0 0) 0x7fc830001670 con 0x7fc8400c8d30 2024-04-24T23:22:06.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:06.083+0000 7fc83d600640 1 -- 172.21.15.6:0/25518051 >> v1:172.21.15.190:6789/0 conn(0x7fc8400c8d30 legacy=0x7fc8400c9140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:22:06.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:06.083+0000 7fc83d600640 1 --2- 172.21.15.6:0/25518051 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc824004ba0 0x7fc824004f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:22:06.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:06.083+0000 7fc83d600640 1 --2- 172.21.15.6:0/25518051 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc8240037a0 0x7fc824003c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:22:06.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:06.083+0000 7fc83d600640 1 --2- 172.21.15.6:0/25518051 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc824004160 0x7fc82400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:22:06.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:06.083+0000 7fc83d600640 1 -- 172.21.15.6:0/25518051 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fc8240147c0 con 0x7fc8240037a0 2024-04-24T23:22:06.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:06.083+0000 7fc83d600640 1 -- 172.21.15.6:0/25518051 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fc824014940 con 0x7fc824004160 2024-04-24T23:22:06.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:06.083+0000 7fc83d600640 1 -- 172.21.15.6:0/25518051 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fc824014af0 con 0x7fc824004ba0 2024-04-24T23:22:06.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:06.085+0000 7fc83ea00640 1 --2- 172.21.15.6:0/25518051 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc824004ba0 0x7fc824004f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:22:06.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:06.086+0000 7fc83f400640 1 --2- 172.21.15.6:0/25518051 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc824004160 0x7fc82400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:22:06.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:06.086+0000 7fc83e000640 1 --2- 172.21.15.6:0/25518051 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc8240037a0 0x7fc824003c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:22:06.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:06.087+0000 7fc83f400640 1 -- 172.21.15.6:0/25518051 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc824004ba0 msgr2=0x7fc824004f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:22:06.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:06.087+0000 7fc83f400640 1 --2- 172.21.15.6:0/25518051 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc824004ba0 0x7fc824004f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:22:06.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:06.087+0000 7fc83f400640 1 -- 172.21.15.6:0/25518051 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc8240037a0 msgr2=0x7fc824003c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:22:06.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:06.087+0000 7fc83f400640 1 --2- 172.21.15.6:0/25518051 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc8240037a0 0x7fc824003c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:22:06.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:06.087+0000 7fc83f400640 1 -- 172.21.15.6:0/25518051 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc84006f210 con 0x7fc824004160 2024-04-24T23:22:06.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:06.089+0000 7fc83f400640 1 -- 172.21.15.6:0/25518051 reap_dead start 2024-04-24T23:22:06.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:06.094+0000 7fc83f400640 1 --2- 172.21.15.6:0/25518051 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc824004160 0x7fc82400b9c0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fc83000c160 tx=0x7fc824005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cbc4dd16a40dcd82 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:22:06.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:06.094+0000 7fc83d600640 1 -- 172.21.15.6:0/25518051 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc830036070 con 0x7fc824004160 2024-04-24T23:22:06.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:06.095+0000 7fc83d600640 1 -- 172.21.15.6:0/25518051 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc830007270 con 0x7fc824004160 2024-04-24T23:22:06.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:06.096+0000 7fc845c00640 1 -- 172.21.15.6:0/25518051 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc824004160 msgr2=0x7fc82400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:22:06.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:06.096+0000 7fc845c00640 1 --2- 172.21.15.6:0/25518051 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc824004160 0x7fc82400b9c0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fc83000c160 tx=0x7fc824005830 comp rx=0 tx=0).stop 2024-04-24T23:22:06.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:06.096+0000 7fc845c00640 1 -- 172.21.15.6:0/25518051 shutdown_connections 2024-04-24T23:22:06.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:06.096+0000 7fc845c00640 1 --2- 172.21.15.6:0/25518051 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc824004160 0x7fc82400b9c0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:22:06.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:06.097+0000 7fc845c00640 1 -- 172.21.15.6:0/25518051 >> 172.21.15.6:0/25518051 conn(0x7fc84006b0e0 msgr2=0x7fc84006b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:22:06.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:06.097+0000 7fc845c00640 1 -- 172.21.15.6:0/25518051 shutdown_connections 2024-04-24T23:22:06.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:06.097+0000 7fc845c00640 1 -- 172.21.15.6:0/25518051 wait complete. 2024-04-24T23:22:06.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:06.100+0000 7fc845c00640 1 Processor -- start 2024-04-24T23:22:06.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:06.101+0000 7fc845c00640 1 -- start start 2024-04-24T23:22:06.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:06.102+0000 7fc845c00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc824004160 0x7fc8401e0300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:22:06.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:06.102+0000 7fc845c00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc8401e0900 0x7fc8401dd5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:22:06.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:06.102+0000 7fc845c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc8401df240 0x7fc84014a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:22:06.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:06.102+0000 7fc845c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fc8400c8810 con 0x7fc8401df240 2024-04-24T23:22:06.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:06.102+0000 7fc845c00640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fc8400c8530 con 0x7fc824004160 2024-04-24T23:22:06.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:06.102+0000 7fc845c00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fc840151130 con 0x7fc8401e0900 2024-04-24T23:22:06.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:06.103+0000 7fc83f400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc8401df240 0x7fc84014a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:22:06.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:06.103+0000 7fc83f400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc8401df240 0x7fc84014a4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36236/0 (socket says 172.21.15.6:36236) 2024-04-24T23:22:06.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:06.103+0000 7fc83f400640 1 -- 172.21.15.6:0/1764689224 learned_addr learned my addr 172.21.15.6:0/1764689224 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:22:06.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:06.103+0000 7fc83e000640 1 --2- 172.21.15.6:0/1764689224 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc8401e0900 0x7fc8401dd5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:22:06.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:06.104+0000 7fc83f400640 1 -- 172.21.15.6:0/1764689224 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc8401e0900 msgr2=0x7fc8401dd5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:22:06.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:06.104+0000 7fc83f400640 1 --2- 172.21.15.6:0/1764689224 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc8401e0900 0x7fc8401dd5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:22:06.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:06.104+0000 7fc83f400640 1 -- 172.21.15.6:0/1764689224 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc824004160 msgr2=0x7fc8401e0300 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-24T23:22:06.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:06.104+0000 7fc83f400640 1 --2- 172.21.15.6:0/1764689224 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc824004160 0x7fc8401e0300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:22:06.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:06.104+0000 7fc83f400640 1 -- 172.21.15.6:0/1764689224 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8401ddae0 con 0x7fc8401df240 2024-04-24T23:22:06.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:06.105+0000 7fc83f400640 1 --2- 172.21.15.6:0/1764689224 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc8401df240 0x7fc84014a4a0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fc83000c280 tx=0x7fc830001970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:22:06.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:06.105+0000 7fc836a00640 1 -- 172.21.15.6:0/1764689224 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc830036070 con 0x7fc8401df240 2024-04-24T23:22:06.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:06.106+0000 7fc836a00640 1 -- 172.21.15.6:0/1764689224 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc830041070 con 0x7fc8401df240 2024-04-24T23:22:06.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:06.106+0000 7fc836a00640 1 -- 172.21.15.6:0/1764689224 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8300067d0 con 0x7fc8401df240 2024-04-24T23:22:06.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:06.106+0000 7fc845c00640 1 -- 172.21.15.6:0/1764689224 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8401dfa60 con 0x7fc8401df240 2024-04-24T23:22:06.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:06.106+0000 7fc845c00640 1 -- 172.21.15.6:0/1764689224 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8401e0020 con 0x7fc8401df240 2024-04-24T23:22:06.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:06.111+0000 7fc845c00640 1 -- 172.21.15.6:0/1764689224 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc818005290 con 0x7fc8401df240 2024-04-24T23:22:06.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:06.112+0000 7fc836a00640 1 -- 172.21.15.6:0/1764689224 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fc830007270 con 0x7fc8401df240 2024-04-24T23:22:06.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:06.114+0000 7fc836a00640 1 --2- 172.21.15.6:0/1764689224 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fc82006be60 0x7fc82006e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:22:06.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:06.115+0000 7fc836a00640 1 -- 172.21.15.6:0/1764689224 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6762+0+0 (secure 0 0 0) 0x7fc8300aad80 con 0x7fc8401df240 2024-04-24T23:22:06.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:06.131+0000 7fc83ea00640 1 --2- 172.21.15.6:0/1764689224 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fc82006be60 0x7fc82006e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:22:06.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:06.135+0000 7fc83ea00640 1 --2- 172.21.15.6:0/1764689224 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fc82006be60 0x7fc82006e320 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fc838007b00 tx=0x7fc838007760 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:22:06.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:06.136+0000 7fc836a00640 1 -- 172.21.15.6:0/1764689224 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fc830078cc0 con 0x7fc8401df240 2024-04-24T23:22:06.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:06.711+0000 7fc845c00640 1 -- 172.21.15.6:0/1764689224 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"} v 0) -- 0x7fc818005850 con 0x7fc8401df240 2024-04-24T23:22:07.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:07.459+0000 7fc836a00640 1 -- 172.21.15.6:0/1764689224 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]=0 v31) ==== 105+0+0 (secure 0 0 0) 0x7fc83007cfc0 con 0x7fc8401df240 2024-04-24T23:22:07.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:07.469+0000 7fc845c00640 1 -- 172.21.15.6:0/1764689224 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fc82006be60 msgr2=0x7fc82006e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:22:07.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:07.469+0000 7fc845c00640 1 --2- 172.21.15.6:0/1764689224 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fc82006be60 0x7fc82006e320 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fc838007b00 tx=0x7fc838007760 comp rx=0 tx=0).stop 2024-04-24T23:22:07.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:07.469+0000 7fc845c00640 1 -- 172.21.15.6:0/1764689224 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc8401df240 msgr2=0x7fc84014a4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:22:07.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:07.469+0000 7fc845c00640 1 --2- 172.21.15.6:0/1764689224 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc8401df240 0x7fc84014a4a0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fc83000c280 tx=0x7fc830001970 comp rx=0 tx=0).stop 2024-04-24T23:22:07.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:07.469+0000 7fc845c00640 1 -- 172.21.15.6:0/1764689224 shutdown_connections 2024-04-24T23:22:07.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:07.469+0000 7fc845c00640 1 --2- 172.21.15.6:0/1764689224 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fc82006be60 0x7fc82006e320 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:22:07.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:07.469+0000 7fc845c00640 1 --2- 172.21.15.6:0/1764689224 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc8401e0900 0x7fc8401dd5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:22:07.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:07.469+0000 7fc845c00640 1 --2- 172.21.15.6:0/1764689224 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc8401df240 0x7fc84014a4a0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:22:07.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:07.469+0000 7fc845c00640 1 --2- 172.21.15.6:0/1764689224 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc824004160 0x7fc8401e0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:22:07.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:07.469+0000 7fc845c00640 1 -- 172.21.15.6:0/1764689224 >> 172.21.15.6:0/1764689224 conn(0x7fc84006b0e0 msgr2=0x7fc84014b0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:22:07.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:07.471+0000 7fc845c00640 1 -- 172.21.15.6:0/1764689224 shutdown_connections 2024-04-24T23:22:07.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:07.471+0000 7fc845c00640 1 -- 172.21.15.6:0/1764689224 wait complete. 2024-04-24T23:22:07.517 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2024-04-24T23:22:07.517 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:22:09.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:09.662+0000 7fb34fa00640 1 Processor -- start 2024-04-24T23:22:09.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:09.663+0000 7fb34fa00640 1 -- start start 2024-04-24T23:22:09.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:09.665+0000 7fb34fa00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fb348067940 0x7fb34806c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:22:09.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:09.665+0000 7fb34fa00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb34815b930 con 0x7fb3480c62c0 2024-04-24T23:22:09.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:09.665+0000 7fb34fa00640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb3480c5500 con 0x7fb348154c80 2024-04-24T23:22:09.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:09.665+0000 7fb34fa00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7fb3480cb1a0 con 0x7fb348067940 2024-04-24T23:22:09.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:09.665+0000 7fb34d000640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7fb348154c80 0x7fb3480cbb90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.6:50656/0 (socket says 172.21.15.6:50656) 2024-04-24T23:22:09.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:09.665+0000 7fb34d000640 1 -- 172.21.15.6:0/449248116 learned_addr learned my addr 172.21.15.6:0/449248116 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:22:09.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:09.666+0000 7fb34da00640 1 --2- 172.21.15.6:0/449248116 >> v2:172.21.15.6:3300/0 conn(0x7fb348067940 0x7fb34806c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:22:09.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:09.666+0000 7fb347400640 1 -- 172.21.15.6:0/449248116 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2705376920 0 0) 0x7fb3480c5500 con 0x7fb348154c80 2024-04-24T23:22:09.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:09.666+0000 7fb347400640 1 -- 172.21.15.6:0/449248116 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb32c003520 con 0x7fb348154c80 2024-04-24T23:22:09.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:09.666+0000 7fb347400640 1 -- 172.21.15.6:0/449248116 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1315609248 0 0) 0x7fb32c003520 con 0x7fb348154c80 2024-04-24T23:22:09.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:09.666+0000 7fb347400640 1 -- 172.21.15.6:0/449248116 >> v2:172.21.15.6:3300/0 conn(0x7fb348067940 msgr2=0x7fb34806c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:22:09.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:09.666+0000 7fb347400640 1 --2- 172.21.15.6:0/449248116 >> v2:172.21.15.6:3300/0 conn(0x7fb348067940 0x7fb34806c6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:22:09.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:09.666+0000 7fb347400640 1 -- 172.21.15.6:0/449248116 >> v1:172.21.15.6:6789/0 conn(0x7fb3480c62c0 legacy=0x7fb34806bf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:22:09.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:09.666+0000 7fb347400640 1 -- 172.21.15.6:0/449248116 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3480c57e0 con 0x7fb348154c80 2024-04-24T23:22:09.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:09.666+0000 7fb347400640 1 -- 172.21.15.6:0/449248116 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3463268701 0 0) 0x7fb338001670 con 0x7fb348154c80 2024-04-24T23:22:09.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:09.666+0000 7fb347400640 1 -- 172.21.15.6:0/449248116 >> v1:172.21.15.190:6789/0 conn(0x7fb348154c80 legacy=0x7fb3480cbb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:22:09.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:09.666+0000 7fb347400640 1 --2- 172.21.15.6:0/449248116 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb32c0039b0 0x7fb32c003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:22:09.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:09.666+0000 7fb347400640 1 --2- 172.21.15.6:0/449248116 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb32c004800 0x7fb32c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:22:09.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:09.666+0000 7fb347400640 1 --2- 172.21.15.6:0/449248116 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb32c0051c0 0x7fb32c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:22:09.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:09.666+0000 7fb347400640 1 -- 172.21.15.6:0/449248116 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fb32c00c0c0 con 0x7fb32c004800 2024-04-24T23:22:09.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:09.666+0000 7fb347400640 1 -- 172.21.15.6:0/449248116 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fb32c00c270 con 0x7fb32c0039b0 2024-04-24T23:22:09.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:09.666+0000 7fb347400640 1 -- 172.21.15.6:0/449248116 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fb32c00c470 con 0x7fb32c0051c0 2024-04-24T23:22:09.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:09.668+0000 7fb34da00640 1 --2- 172.21.15.6:0/449248116 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb32c0051c0 0x7fb32c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:22:09.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:09.668+0000 7fb34da00640 1 -- 172.21.15.6:0/449248116 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb32c0039b0 msgr2=0x7fb32c003db0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-24T23:22:09.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:09.668+0000 7fb34da00640 1 --2- 172.21.15.6:0/449248116 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb32c0039b0 0x7fb32c003db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:22:09.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:09.668+0000 7fb34da00640 1 -- 172.21.15.6:0/449248116 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb32c004800 msgr2=0x7fb32c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:22:09.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:09.668+0000 7fb34da00640 1 --2- 172.21.15.6:0/449248116 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb32c004800 0x7fb32c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:22:09.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:09.668+0000 7fb34da00640 1 -- 172.21.15.6:0/449248116 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3480c57e0 con 0x7fb32c0051c0 2024-04-24T23:22:09.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:09.670+0000 7fb34da00640 1 -- 172.21.15.6:0/449248116 reap_dead start 2024-04-24T23:22:09.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:09.670+0000 7fb34da00640 1 --2- 172.21.15.6:0/449248116 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb32c0051c0 0x7fb32c00b9c0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fb334009ac0 tx=0x7fb32c005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7bec7f2bd7720e14 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:22:09.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:09.670+0000 7fb347400640 1 -- 172.21.15.6:0/449248116 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb334009690 con 0x7fb32c0051c0 2024-04-24T23:22:09.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:09.671+0000 7fb347400640 1 -- 172.21.15.6:0/449248116 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb334004020 con 0x7fb32c0051c0 2024-04-24T23:22:09.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:09.672+0000 7fb34fa00640 1 -- 172.21.15.6:0/449248116 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb32c0051c0 msgr2=0x7fb32c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:22:09.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:09.672+0000 7fb34fa00640 1 --2- 172.21.15.6:0/449248116 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb32c0051c0 0x7fb32c00b9c0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fb334009ac0 tx=0x7fb32c005830 comp rx=0 tx=0).stop 2024-04-24T23:22:09.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:09.673+0000 7fb34fa00640 1 -- 172.21.15.6:0/449248116 shutdown_connections 2024-04-24T23:22:09.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:09.673+0000 7fb34fa00640 1 --2- 172.21.15.6:0/449248116 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb32c0051c0 0x7fb32c00b9c0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:22:09.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:09.673+0000 7fb34fa00640 1 -- 172.21.15.6:0/449248116 >> 172.21.15.6:0/449248116 conn(0x7fb348072810 msgr2=0x7fb348066ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:22:09.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:09.674+0000 7fb34fa00640 1 -- 172.21.15.6:0/449248116 shutdown_connections 2024-04-24T23:22:09.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:09.675+0000 7fb34fa00640 1 -- 172.21.15.6:0/449248116 wait complete. 2024-04-24T23:22:09.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:09.677+0000 7fb34fa00640 1 Processor -- start 2024-04-24T23:22:09.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:09.678+0000 7fb34fa00640 1 -- start start 2024-04-24T23:22:09.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:09.680+0000 7fb34fa00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb32c0051c0 0x7fb3480cbfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:22:09.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:09.681+0000 7fb34fa00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb3480ca3d0 0x7fb3480cc4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:22:09.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:09.681+0000 7fb34fa00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb3481f8130 0x7fb3481f85b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:22:09.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:09.682+0000 7fb34da00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb3481f8130 0x7fb3481f85b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:22:09.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:09.682+0000 7fb34da00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb3481f8130 0x7fb3481f85b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36262/0 (socket says 172.21.15.6:36262) 2024-04-24T23:22:09.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:09.682+0000 7fb347e00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb3480ca3d0 0x7fb3480cc4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:22:09.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:09.683+0000 7fb347e00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb3480ca3d0 0x7fb3480cc4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.6:36226/0 (socket says 172.21.15.6:36226) 2024-04-24T23:22:09.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:09.681+0000 7fb34d000640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb32c0051c0 0x7fb3480cbfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:22:09.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:09.683+0000 7fb34d000640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb32c0051c0 0x7fb3480cbfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:48578/0 (socket says 172.21.15.6:48578) 2024-04-24T23:22:09.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:09.683+0000 7fb34da00640 1 -- 172.21.15.6:0/3607957951 learned_addr learned my addr 172.21.15.6:0/3607957951 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:22:09.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:09.683+0000 7fb34fa00640 1 -- 172.21.15.6:0/3607957951 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fb34815b580 con 0x7fb3481f8130 2024-04-24T23:22:09.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:09.683+0000 7fb34fa00640 1 -- 172.21.15.6:0/3607957951 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fb3480c5e30 con 0x7fb32c0051c0 2024-04-24T23:22:09.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:09.683+0000 7fb34fa00640 1 -- 172.21.15.6:0/3607957951 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fb3480cbb20 con 0x7fb3480ca3d0 2024-04-24T23:22:09.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:09.683+0000 7fb347e00640 1 -- 172.21.15.6:0/3607957951 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb32c0051c0 msgr2=0x7fb3480cbfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:22:09.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:09.683+0000 7fb347e00640 1 --2- 172.21.15.6:0/3607957951 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb32c0051c0 0x7fb3480cbfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:22:09.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:09.684+0000 7fb347e00640 1 -- 172.21.15.6:0/3607957951 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb3481f8130 msgr2=0x7fb3481f85b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:22:09.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:09.684+0000 7fb347e00640 1 --2- 172.21.15.6:0/3607957951 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb3481f8130 0x7fb3481f85b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:22:09.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:09.684+0000 7fb347e00640 1 -- 172.21.15.6:0/3607957951 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb348159140 con 0x7fb3480ca3d0 2024-04-24T23:22:09.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:09.684+0000 7fb347e00640 1 --2- 172.21.15.6:0/3607957951 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb3480ca3d0 0x7fb3480cc4f0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fb34000c7e0 tx=0x7fb34000cbe0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:22:09.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:09.684+0000 7fb344c00640 1 -- 172.21.15.6:0/3607957951 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb340017070 con 0x7fb3480ca3d0 2024-04-24T23:22:09.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:09.685+0000 7fb344c00640 1 -- 172.21.15.6:0/3607957951 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb340003da0 con 0x7fb3480ca3d0 2024-04-24T23:22:09.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:09.685+0000 7fb344c00640 1 -- 172.21.15.6:0/3607957951 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3400052b0 con 0x7fb3480ca3d0 2024-04-24T23:22:09.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:09.685+0000 7fb34fa00640 1 -- 172.21.15.6:0/3607957951 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb348154100 con 0x7fb3480ca3d0 2024-04-24T23:22:09.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:09.685+0000 7fb34fa00640 1 -- 172.21.15.6:0/3607957951 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb348153670 con 0x7fb3480ca3d0 2024-04-24T23:22:09.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:09.688+0000 7fb34fa00640 1 -- 172.21.15.6:0/3607957951 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb3480767b0 con 0x7fb3480ca3d0 2024-04-24T23:22:09.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:09.690+0000 7fb344c00640 1 -- 172.21.15.6:0/3607957951 <== mon.2 v2:172.21.15.190:3301/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fb340005450 con 0x7fb3480ca3d0 2024-04-24T23:22:09.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:09.692+0000 7fb344c00640 1 --2- 172.21.15.6:0/3607957951 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fb32806be30 0x7fb32806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:22:09.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:09.693+0000 7fb344c00640 1 -- 172.21.15.6:0/3607957951 <== mon.2 v2:172.21.15.190:3301/0 5 ==== osd_map(28..28 src has 1..28) ==== 6762+0+0 (secure 0 0 0) 0x7fb34008a730 con 0x7fb3480ca3d0 2024-04-24T23:22:09.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:09.693+0000 7fb34d000640 1 --2- 172.21.15.6:0/3607957951 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fb32806be30 0x7fb32806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:22:09.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:09.694+0000 7fb34d000640 1 --2- 172.21.15.6:0/3607957951 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fb32806be30 0x7fb32806e2f0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fb338001eb0 tx=0x7fb338001820 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:22:09.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:09.697+0000 7fb344c00640 1 -- 172.21.15.6:0/3607957951 <== mon.2 v2:172.21.15.190:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fb3400585d0 con 0x7fb3480ca3d0 2024-04-24T23:22:10.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:10.257+0000 7fb34fa00640 1 -- 172.21.15.6:0/3607957951 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb348159d10 con 0x7fb3480ca3d0 2024-04-24T23:22:10.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:10.260+0000 7fb344c00640 1 -- 172.21.15.6:0/3607957951 <== mon.2 v2:172.21.15.190:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+6904 (secure 0 0 0) 0x7fb34005c8d0 con 0x7fb3480ca3d0 2024-04-24T23:22:10.261 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:22:10.263 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5081,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/1190759626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1190759626},{"type":"v1","addr":"172.21.15.6:6834","nonce":1190759626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:22:07.460817+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4282,"mds_2":4235,"mds_3":4232,"mds_4":5257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4232":{"gid":4232,"name":"b","rank":3,"incarnation":15,"state":"up:active","state_seq":15,"addr":"172.21.15.190:6838/2032280158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2032280158},{"type":"v1","addr":"172.21.15.190:6838","nonce":2032280158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4235":{"gid":4235,"name":"d","rank":2,"incarnation":13,"state":"up:active","state_seq":14,"addr":"172.21.15.190:6839/1572872451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":1572872451},{"type":"v1","addr":"172.21.15.190:6839","nonce":1572872451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4282":{"gid":4282,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":12,"addr":"172.21.15.190:6837/580315269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":580315269},{"type":"v1","addr":"172.21.15.190:6837","nonce":580315269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/489092393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":489092393},{"type":"v1","addr":"172.21.15.6:6835","nonce":489092393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5257":{"gid":5257,"name":"e","rank":4,"incarnation":25,"state":"up:stopping","state_seq":43,"addr":"172.21.15.6:6837/3240971655","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":3240971655},{"type":"v1","addr":"172.21.15.6:6837","nonce":3240971655}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4306,"qdb_cluster":[4232,4235,4282,4306]},"id":1}]} 2024-04-24T23:22:10.263 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 32 2024-04-24T23:22:10.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:10.268+0000 7fb34fa00640 1 -- 172.21.15.6:0/3607957951 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fb32806be30 msgr2=0x7fb32806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:22:10.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:10.268+0000 7fb34fa00640 1 --2- 172.21.15.6:0/3607957951 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fb32806be30 0x7fb32806e2f0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fb338001eb0 tx=0x7fb338001820 comp rx=0 tx=0).stop 2024-04-24T23:22:10.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:10.272+0000 7fb34fa00640 1 -- 172.21.15.6:0/3607957951 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb3480ca3d0 msgr2=0x7fb3480cc4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:22:10.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:10.272+0000 7fb34fa00640 1 --2- 172.21.15.6:0/3607957951 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb3480ca3d0 0x7fb3480cc4f0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fb34000c7e0 tx=0x7fb34000cbe0 comp rx=0 tx=0).stop 2024-04-24T23:22:10.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:10.273+0000 7fb34fa00640 1 -- 172.21.15.6:0/3607957951 shutdown_connections 2024-04-24T23:22:10.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:10.273+0000 7fb34fa00640 1 --2- 172.21.15.6:0/3607957951 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fb32806be30 0x7fb32806e2f0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:22:10.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:10.273+0000 7fb34fa00640 1 --2- 172.21.15.6:0/3607957951 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb3480ca3d0 0x7fb3480cc4f0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:22:10.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:10.273+0000 7fb34fa00640 1 --2- 172.21.15.6:0/3607957951 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb3481f8130 0x7fb3481f85b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:22:10.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:10.273+0000 7fb34fa00640 1 --2- 172.21.15.6:0/3607957951 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb32c0051c0 0x7fb3480cbfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:22:10.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:10.273+0000 7fb34fa00640 1 -- 172.21.15.6:0/3607957951 >> 172.21.15.6:0/3607957951 conn(0x7fb348072810 msgr2=0x7fb34806c570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:22:10.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:10.273+0000 7fb34fa00640 1 -- 172.21.15.6:0/3607957951 shutdown_connections 2024-04-24T23:22:10.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:10.273+0000 7fb34fa00640 1 -- 172.21.15.6:0/3607957951 wait complete. 2024-04-24T23:22:10.321 INFO:tasks.mds_thrash.fs.[cephfs]:cluster is considered unstable while MDS are in up:stopping (!thrash_while_stopping) 2024-04-24T23:22:10.322 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 32, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-24T23:15:36.400716+0000", "damaged": [], "data_pools": [ 3, 4 ], "enabled": true, "epoch": 32, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4232": { "addr": "172.21.15.190:6838/2032280158", "addrs": { "addrvec": [ { "addr": "172.21.15.190:6835", "nonce": 2032280158, "type": "v2" }, { "addr": "172.21.15.190:6838", "nonce": 2032280158, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4232, "incarnation": 15, "join_fscid": -1, "name": "b", "rank": 3, "state": "up:active", "state_seq": 15 }, "gid_4235": { "addr": "172.21.15.190:6839/1572872451", "addrs": { "addrvec": [ { "addr": "172.21.15.190:6836", "nonce": 1572872451, "type": "v2" }, { "addr": "172.21.15.190:6839", "nonce": 1572872451, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4235, "incarnation": 13, "join_fscid": -1, "name": "d", "rank": 2, "state": "up:active", "state_seq": 14 }, "gid_4282": { "addr": "172.21.15.190:6837/580315269", "addrs": { "addrvec": [ { "addr": "172.21.15.190:6834", "nonce": 580315269, "type": "v2" }, { "addr": "172.21.15.190:6837", "nonce": 580315269, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4282, "incarnation": 9, "join_fscid": -1, "name": "f", "rank": 1, "state": "up:active", "state_seq": 12 }, "gid_4306": { "addr": "172.21.15.6:6835/489092393", "addrs": { "addrvec": [ { "addr": "172.21.15.6:6833", "nonce": 489092393, "type": "v2" }, { "addr": "172.21.15.6:6835", "nonce": 489092393, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4306, "incarnation": 4, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 4 }, "gid_5257": { "addr": "172.21.15.6:6837/3240971655", "addrs": { "addrvec": [ { "addr": "172.21.15.6:6836", "nonce": 3240971655, "type": "v2" }, { "addr": "172.21.15.6:6837", "nonce": 3240971655, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5257, "incarnation": 25, "join_fscid": -1, "name": "e", "rank": 4, "state": "up:stopping", "state_seq": 43 } }, "last_failure": 0, "last_failure_osd_epoch": 28, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-04-24T23:22:07.460817+0000", "qdb_cluster": [ 4232, 4235, 4282, 4306 ], "qdb_leader": 4306, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4306, "mds_1": 4282, "mds_2": 4235, "mds_3": 4232, "mds_4": 5257 } } } ], "standbys": [ { "addr": "172.21.15.6:6834/1190759626", "addrs": { "addrvec": [ { "addr": "172.21.15.6:6832", "nonce": 1190759626, "type": "v2" }, { "addr": "172.21.15.6:6834", "nonce": 1190759626, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 30, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5081, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-04-24T23:22:12.323 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:22:12.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:12.728+0000 7fc12b200640 1 Processor -- start 2024-04-24T23:22:12.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:12.729+0000 7fc12b200640 1 -- start start 2024-04-24T23:22:12.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:12.734+0000 7fc12b200640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fc1240d55b0 0x7fc124006b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:22:12.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:12.734+0000 7fc12b200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc1240073c0 con 0x7fc1240db150 2024-04-24T23:22:12.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:12.734+0000 7fc12b200640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc1240076a0 con 0x7fc1240dc700 2024-04-24T23:22:12.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:12.734+0000 7fc12b200640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7fc124007980 con 0x7fc1240d55b0 2024-04-24T23:22:12.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:12.735+0000 7fc12a800640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fc1240db150 0x7fc1240db580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:32832/0 (socket says 172.21.15.6:32832) 2024-04-24T23:22:12.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:12.735+0000 7fc12a800640 1 -- 172.21.15.6:0/1432255884 learned_addr learned my addr 172.21.15.6:0/1432255884 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:22:12.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:12.735+0000 7fc129e00640 1 --2- 172.21.15.6:0/1432255884 >> v2:172.21.15.190:3300/0 conn(0x7fc1240d55b0 0x7fc124006b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:22:12.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:12.736+0000 7fc128a00640 1 -- 172.21.15.6:0/1432255884 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2217265372 0 0) 0x7fc1240076a0 con 0x7fc1240dc700 2024-04-24T23:22:12.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:12.736+0000 7fc128a00640 1 -- 172.21.15.6:0/1432255884 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc11000bd30 con 0x7fc1240dc700 2024-04-24T23:22:12.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:12.737+0000 7fc128a00640 1 -- 172.21.15.6:0/1432255884 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2784355268 0 0) 0x7fc1240073c0 con 0x7fc1240db150 2024-04-24T23:22:12.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:12.737+0000 7fc128a00640 1 -- 172.21.15.6:0/1432255884 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc1240076a0 con 0x7fc1240db150 2024-04-24T23:22:12.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:12.737+0000 7fc128a00640 1 -- 172.21.15.6:0/1432255884 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 818332196 0 0) 0x7fc11000bd30 con 0x7fc1240dc700 2024-04-24T23:22:12.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:12.737+0000 7fc128a00640 1 -- 172.21.15.6:0/1432255884 >> v2:172.21.15.190:3300/0 conn(0x7fc1240d55b0 msgr2=0x7fc124006b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:22:12.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:12.737+0000 7fc128a00640 1 --2- 172.21.15.6:0/1432255884 >> v2:172.21.15.190:3300/0 conn(0x7fc1240d55b0 0x7fc124006b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:22:12.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:12.737+0000 7fc128a00640 1 -- 172.21.15.6:0/1432255884 >> v1:172.21.15.6:6789/0 conn(0x7fc1240db150 legacy=0x7fc1240db580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:22:12.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:12.737+0000 7fc128a00640 1 -- 172.21.15.6:0/1432255884 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc124007c60 con 0x7fc1240dc700 2024-04-24T23:22:12.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:12.737+0000 7fc128a00640 1 -- 172.21.15.6:0/1432255884 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3463268701 0 0) 0x7fc118001670 con 0x7fc1240dc700 2024-04-24T23:22:12.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:12.739+0000 7fc128a00640 1 -- 172.21.15.6:0/1432255884 >> v1:172.21.15.190:6789/0 conn(0x7fc1240dc700 legacy=0x7fc1240dcb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:22:12.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:12.739+0000 7fc128a00640 1 --2- 172.21.15.6:0/1432255884 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc11000c350 0x7fc11000c750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:22:12.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:12.739+0000 7fc128a00640 1 --2- 172.21.15.6:0/1432255884 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc11000d010 0x7fc11000d490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:22:12.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:12.739+0000 7fc128a00640 1 --2- 172.21.15.6:0/1432255884 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc11000d9d0 0x7fc1100141d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:22:12.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:12.739+0000 7fc128a00640 1 -- 172.21.15.6:0/1432255884 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fc124007980 con 0x7fc11000d9d0 2024-04-24T23:22:12.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:12.739+0000 7fc128a00640 1 -- 172.21.15.6:0/1432255884 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fc110014770 con 0x7fc11000d010 2024-04-24T23:22:12.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:12.739+0000 7fc128a00640 1 -- 172.21.15.6:0/1432255884 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fc110014970 con 0x7fc11000c350 2024-04-24T23:22:12.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:12.740+0000 7fc129e00640 1 --2- 172.21.15.6:0/1432255884 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc11000c350 0x7fc11000c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:22:12.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:12.742+0000 7fc12a800640 1 --2- 172.21.15.6:0/1432255884 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc11000d9d0 0x7fc1100141d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:22:12.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:12.742+0000 7fc129400640 1 --2- 172.21.15.6:0/1432255884 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc11000d010 0x7fc11000d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:22:12.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:12.743+0000 7fc129e00640 1 -- 172.21.15.6:0/1432255884 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc11000d010 msgr2=0x7fc11000d490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:22:12.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:12.743+0000 7fc129e00640 1 --2- 172.21.15.6:0/1432255884 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc11000d010 0x7fc11000d490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:22:12.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:12.743+0000 7fc129e00640 1 -- 172.21.15.6:0/1432255884 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc11000d9d0 msgr2=0x7fc1100141d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:22:12.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:12.743+0000 7fc129e00640 1 --2- 172.21.15.6:0/1432255884 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc11000d9d0 0x7fc1100141d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:22:12.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:12.743+0000 7fc129e00640 1 -- 172.21.15.6:0/1432255884 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc11000bd30 con 0x7fc11000c350 2024-04-24T23:22:12.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:12.745+0000 7fc12a800640 1 -- 172.21.15.6:0/1432255884 reap_dead start 2024-04-24T23:22:12.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:12.748+0000 7fc129e00640 1 --2- 172.21.15.6:0/1432255884 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc11000c350 0x7fc11000c750 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fc11c009a90 tx=0x7fc11000e040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dec0096428549987 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:22:12.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:12.748+0000 7fc128a00640 1 -- 172.21.15.6:0/1432255884 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc11c036070 con 0x7fc11000c350 2024-04-24T23:22:12.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:12.749+0000 7fc128a00640 1 -- 172.21.15.6:0/1432255884 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc11c031400 con 0x7fc11000c350 2024-04-24T23:22:12.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:12.750+0000 7fc12b200640 1 -- 172.21.15.6:0/1432255884 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc11000c350 msgr2=0x7fc11000c750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:22:12.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:12.750+0000 7fc12b200640 1 --2- 172.21.15.6:0/1432255884 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc11000c350 0x7fc11000c750 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fc11c009a90 tx=0x7fc11000e040 comp rx=0 tx=0).stop 2024-04-24T23:22:12.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:12.753+0000 7fc12b200640 1 -- 172.21.15.6:0/1432255884 shutdown_connections 2024-04-24T23:22:12.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:12.753+0000 7fc12b200640 1 --2- 172.21.15.6:0/1432255884 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc11000c350 0x7fc11000c750 secure :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fc11c009a90 tx=0x7fc11000e040 comp rx=0 tx=0).stop 2024-04-24T23:22:12.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:12.753+0000 7fc12b200640 1 -- 172.21.15.6:0/1432255884 >> 172.21.15.6:0/1432255884 conn(0x7fc124050a10 msgr2=0x7fc124050e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:22:12.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:12.761+0000 7fc12b200640 1 -- 172.21.15.6:0/1432255884 shutdown_connections 2024-04-24T23:22:12.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:12.761+0000 7fc12b200640 1 -- 172.21.15.6:0/1432255884 wait complete. 2024-04-24T23:22:12.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:12.764+0000 7fc12b200640 1 Processor -- start 2024-04-24T23:22:12.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:12.768+0000 7fc12b200640 1 -- start start 2024-04-24T23:22:12.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:12.774+0000 7fc12b200640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc1240cb810 0x7fc124174f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:22:12.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:12.774+0000 7fc12b200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc1240d7670 0x7fc1240d6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:22:12.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:12.774+0000 7fc12b200640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc1240d6750 0x7fc1240d3f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:22:12.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:12.775+0000 7fc12b200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fc12404d000 con 0x7fc1240d7670 2024-04-24T23:22:12.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:12.775+0000 7fc12b200640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fc124008220 con 0x7fc1240d6750 2024-04-24T23:22:12.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:12.775+0000 7fc12b200640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fc12404d180 con 0x7fc1240cb810 2024-04-24T23:22:12.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:12.775+0000 7fc12a800640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc1240d6750 0x7fc1240d3f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:22:12.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:12.775+0000 7fc12a800640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc1240d6750 0x7fc1240d3f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:48594/0 (socket says 172.21.15.6:48594) 2024-04-24T23:22:12.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:12.775+0000 7fc12a800640 1 -- 172.21.15.6:0/347720992 learned_addr learned my addr 172.21.15.6:0/347720992 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:22:12.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:12.776+0000 7fc129e00640 1 --2- 172.21.15.6:0/347720992 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc1240cb810 0x7fc124174f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:22:12.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:12.776+0000 7fc12a800640 1 -- 172.21.15.6:0/347720992 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc1240cb810 msgr2=0x7fc124174f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:22:12.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:12.776+0000 7fc12a800640 1 --2- 172.21.15.6:0/347720992 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc1240cb810 0x7fc124174f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:22:12.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:12.776+0000 7fc12a800640 1 -- 172.21.15.6:0/347720992 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc1240d7670 msgr2=0x7fc1240d6210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:22:12.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:12.776+0000 7fc129e00640 1 --2- 172.21.15.6:0/347720992 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc1240cb810 0x7fc124174f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:22:12.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:12.776+0000 7fc129400640 1 --2- 172.21.15.6:0/347720992 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc1240d7670 0x7fc1240d6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:22:12.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:12.778+0000 7fc12a800640 1 --2- 172.21.15.6:0/347720992 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc1240d7670 0x7fc1240d6210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:22:12.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:12.778+0000 7fc12a800640 1 -- 172.21.15.6:0/347720992 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1240d7a70 con 0x7fc1240d6750 2024-04-24T23:22:12.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:12.778+0000 7fc129400640 1 --2- 172.21.15.6:0/347720992 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc1240d7670 0x7fc1240d6210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:22:12.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:12.778+0000 7fc12a800640 1 --2- 172.21.15.6:0/347720992 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc1240d6750 0x7fc1240d3f90 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fc12c0502b0 tx=0x7fc12c068800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:22:12.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:12.779+0000 7fc122000640 1 -- 172.21.15.6:0/347720992 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc12c06f040 con 0x7fc1240d6750 2024-04-24T23:22:12.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:12.780+0000 7fc12b200640 1 -- 172.21.15.6:0/347720992 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1241754a0 con 0x7fc1240d6750 2024-04-24T23:22:12.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:12.780+0000 7fc12b200640 1 -- 172.21.15.6:0/347720992 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1240d7d10 con 0x7fc1240d6750 2024-04-24T23:22:12.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:12.788+0000 7fc117400640 1 -- 172.21.15.6:0/347720992 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc124005710 con 0x7fc1240d6750 2024-04-24T23:22:12.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:12.788+0000 7fc122000640 1 -- 172.21.15.6:0/347720992 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc12c068930 con 0x7fc1240d6750 2024-04-24T23:22:12.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:12.788+0000 7fc122000640 1 -- 172.21.15.6:0/347720992 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc12c06f040 con 0x7fc1240d6750 2024-04-24T23:22:12.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:12.793+0000 7fc122000640 1 -- 172.21.15.6:0/347720992 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fc12c073500 con 0x7fc1240d6750 2024-04-24T23:22:12.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:12.799+0000 7fc122000640 1 --2- 172.21.15.6:0/347720992 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fc10c06c060 0x7fc10c06e520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:22:12.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:12.799+0000 7fc129e00640 1 --2- 172.21.15.6:0/347720992 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fc10c06c060 0x7fc10c06e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:22:12.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:12.801+0000 7fc122000640 1 -- 172.21.15.6:0/347720992 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6762+0+0 (secure 0 0 0) 0x7fc12c106e60 con 0x7fc1240d6750 2024-04-24T23:22:12.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:12.801+0000 7fc122000640 1 -- 172.21.15.6:0/347720992 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fc12c107280 con 0x7fc1240d6750 2024-04-24T23:22:12.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:12.801+0000 7fc129e00640 1 --2- 172.21.15.6:0/347720992 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fc10c06c060 0x7fc10c06e520 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fc11c033000 tx=0x7fc11c008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:22:13.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:13.264+0000 7fc117400640 1 -- 172.21.15.6:0/347720992 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc124005cc0 con 0x7fc1240d6750 2024-04-24T23:22:13.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:13.274+0000 7fc122000640 1 -- 172.21.15.6:0/347720992 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+6904 (secure 0 0 0) 0x7fc12c0d5460 con 0x7fc1240d6750 2024-04-24T23:22:13.275 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:22:13.275 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5081,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/1190759626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1190759626},{"type":"v1","addr":"172.21.15.6:6834","nonce":1190759626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:22:07.460817+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4282,"mds_2":4235,"mds_3":4232,"mds_4":5257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4232":{"gid":4232,"name":"b","rank":3,"incarnation":15,"state":"up:active","state_seq":15,"addr":"172.21.15.190:6838/2032280158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2032280158},{"type":"v1","addr":"172.21.15.190:6838","nonce":2032280158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4235":{"gid":4235,"name":"d","rank":2,"incarnation":13,"state":"up:active","state_seq":14,"addr":"172.21.15.190:6839/1572872451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":1572872451},{"type":"v1","addr":"172.21.15.190:6839","nonce":1572872451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4282":{"gid":4282,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":12,"addr":"172.21.15.190:6837/580315269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":580315269},{"type":"v1","addr":"172.21.15.190:6837","nonce":580315269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/489092393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":489092393},{"type":"v1","addr":"172.21.15.6:6835","nonce":489092393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5257":{"gid":5257,"name":"e","rank":4,"incarnation":25,"state":"up:stopping","state_seq":43,"addr":"172.21.15.6:6837/3240971655","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":3240971655},{"type":"v1","addr":"172.21.15.6:6837","nonce":3240971655}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4306,"qdb_cluster":[4232,4235,4282,4306]},"id":1}]} 2024-04-24T23:22:13.275 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 32 2024-04-24T23:22:13.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:13.284+0000 7fc117400640 1 -- 172.21.15.6:0/347720992 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fc10c06c060 msgr2=0x7fc10c06e520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:22:13.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:13.284+0000 7fc117400640 1 --2- 172.21.15.6:0/347720992 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fc10c06c060 0x7fc10c06e520 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fc11c033000 tx=0x7fc11c008030 comp rx=0 tx=0).stop 2024-04-24T23:22:13.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:13.284+0000 7fc117400640 1 -- 172.21.15.6:0/347720992 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc1240d6750 msgr2=0x7fc1240d3f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:22:13.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:13.284+0000 7fc117400640 1 --2- 172.21.15.6:0/347720992 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc1240d6750 0x7fc1240d3f90 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fc12c0502b0 tx=0x7fc12c068800 comp rx=0 tx=0).stop 2024-04-24T23:22:13.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:13.284+0000 7fc117400640 1 -- 172.21.15.6:0/347720992 shutdown_connections 2024-04-24T23:22:13.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:13.284+0000 7fc117400640 1 --2- 172.21.15.6:0/347720992 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fc10c06c060 0x7fc10c06e520 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:22:13.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:13.284+0000 7fc117400640 1 --2- 172.21.15.6:0/347720992 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc1240d6750 0x7fc1240d3f90 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:22:13.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:13.284+0000 7fc117400640 1 --2- 172.21.15.6:0/347720992 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc1240d7670 0x7fc1240d6210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:22:13.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:13.284+0000 7fc117400640 1 --2- 172.21.15.6:0/347720992 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc1240cb810 0x7fc124174f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:22:13.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:13.284+0000 7fc117400640 1 -- 172.21.15.6:0/347720992 >> 172.21.15.6:0/347720992 conn(0x7fc124050a10 msgr2=0x7fc1240d9b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:22:13.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:13.285+0000 7fc117400640 1 -- 172.21.15.6:0/347720992 shutdown_connections 2024-04-24T23:22:13.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:13.285+0000 7fc117400640 1 -- 172.21.15.6:0/347720992 wait complete. 2024-04-24T23:22:13.319 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-24T23:22:14.146 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-24T23:22:14.154 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-24T23:22:15.320 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:22:15.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:15.782+0000 7f010b600640 1 Processor -- start 2024-04-24T23:22:15.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:15.782+0000 7f010b600640 1 -- start start 2024-04-24T23:22:15.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:15.784+0000 7f010b600640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f01040c7dc0 0x7f0104072530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:22:15.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:15.784+0000 7f010b600640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f010416abf0 0x7f010416b040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:22:15.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:15.784+0000 7f010b600640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f01040c74c0 con 0x7f0104167180 2024-04-24T23:22:15.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:15.784+0000 7f010b600640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f0104168570 con 0x7f01040c7dc0 2024-04-24T23:22:15.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:15.784+0000 7f010b600640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f01040720a0 con 0x7f010416abf0 2024-04-24T23:22:15.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:15.784+0000 7f0109600640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f010416abf0 0x7f010416b040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:22:15.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:15.784+0000 7f0109600640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f010416abf0 0x7f010416b040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.6:39052/0 (socket says 172.21.15.6:39052) 2024-04-24T23:22:15.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:15.784+0000 7f0109600640 1 -- 172.21.15.6:0/4177965267 learned_addr learned my addr 172.21.15.6:0/4177965267 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:22:15.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:15.786+0000 7f0103e00640 1 --2- 172.21.15.6:0/4177965267 >> v2:172.21.15.6:3300/0 conn(0x7f01040c7dc0 0x7f0104072530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:22:15.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:15.797+0000 7f0103400640 1 -- 172.21.15.6:0/4177965267 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1443125666 0 0) 0x7f01040c74c0 con 0x7f0104167180 2024-04-24T23:22:15.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:15.944+0000 7f0103400640 1 -- 172.21.15.6:0/4177965267 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f00ec007930 con 0x7f0104167180 2024-04-24T23:22:15.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:15.944+0000 7f0109600640 1 -- 172.21.15.6:0/4177965267 >> v2:172.21.15.6:3300/0 conn(0x7f01040c7dc0 msgr2=0x7f0104072530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:22:15.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:15.944+0000 7f0109600640 1 --2- 172.21.15.6:0/4177965267 >> v2:172.21.15.6:3300/0 conn(0x7f01040c7dc0 0x7f0104072530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:22:15.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:15.944+0000 7f0109600640 1 -- 172.21.15.6:0/4177965267 >> v1:172.21.15.6:6789/0 conn(0x7f0104167180 legacy=0x7f010406edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:22:15.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:15.944+0000 7f0109600640 1 -- 172.21.15.6:0/4177965267 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01040c71e0 con 0x7f010416abf0 2024-04-24T23:22:15.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:15.947+0000 7f0109600640 1 --2- 172.21.15.6:0/4177965267 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f010416abf0 0x7f010416b040 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f00fc008ec0 tx=0x7f00fc02aaf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fc7e62f5a932a12 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:22:15.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:15.947+0000 7f0103400640 1 -- 172.21.15.6:0/4177965267 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00fc02c850 con 0x7f010416abf0 2024-04-24T23:22:15.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:15.948+0000 7f0103400640 1 -- 172.21.15.6:0/4177965267 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f00fc02c9f0 con 0x7f010416abf0 2024-04-24T23:22:15.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:15.948+0000 7f0103400640 1 -- 172.21.15.6:0/4177965267 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00fc02cde0 con 0x7f010416abf0 2024-04-24T23:22:15.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:15.948+0000 7f010b600640 1 -- 172.21.15.6:0/4177965267 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f010416abf0 msgr2=0x7f010416b040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:22:15.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:15.949+0000 7f010b600640 1 --2- 172.21.15.6:0/4177965267 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f010416abf0 0x7f010416b040 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f00fc008ec0 tx=0x7f00fc02aaf0 comp rx=0 tx=0).stop 2024-04-24T23:22:15.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:15.949+0000 7f010b600640 1 -- 172.21.15.6:0/4177965267 shutdown_connections 2024-04-24T23:22:15.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:15.949+0000 7f010b600640 1 --2- 172.21.15.6:0/4177965267 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f010416abf0 0x7f010416b040 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:22:15.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:15.949+0000 7f010b600640 1 --2- 172.21.15.6:0/4177965267 >> v2:172.21.15.6:3300/0 conn(0x7f01040c7dc0 0x7f0104072530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:22:15.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:15.949+0000 7f010b600640 1 -- 172.21.15.6:0/4177965267 >> 172.21.15.6:0/4177965267 conn(0x7f010406b0e0 msgr2=0x7f010406b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:22:15.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:15.949+0000 7f010b600640 1 -- 172.21.15.6:0/4177965267 shutdown_connections 2024-04-24T23:22:15.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:15.949+0000 7f010b600640 1 -- 172.21.15.6:0/4177965267 wait complete. 2024-04-24T23:22:15.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:15.951+0000 7f010b600640 1 Processor -- start 2024-04-24T23:22:15.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:15.953+0000 7f010b600640 1 -- start start 2024-04-24T23:22:15.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:15.956+0000 7f010b600640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f01040c7dc0 0x7f01040c8470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:22:15.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:15.956+0000 7f010b600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0104167180 0x7f010420e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:22:15.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:15.956+0000 7f010b600640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f010416abf0 0x7f010420ec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:22:15.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:15.956+0000 7f010b600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f0104071d30 con 0x7f0104167180 2024-04-24T23:22:15.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:15.956+0000 7f010b600640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f0104071a50 con 0x7f010416abf0 2024-04-24T23:22:15.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:15.956+0000 7f010b600640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f01040c6e10 con 0x7f01040c7dc0 2024-04-24T23:22:15.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:15.956+0000 7f0108c00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f01040c7dc0 0x7f01040c8470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:22:15.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:15.956+0000 7f0108c00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f01040c7dc0 0x7f01040c8470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.6:39064/0 (socket says 172.21.15.6:39064) 2024-04-24T23:22:15.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:15.956+0000 7f0108c00640 1 -- 172.21.15.6:0/863193452 learned_addr learned my addr 172.21.15.6:0/863193452 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:22:15.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:15.956+0000 7f0109600640 1 --2- 172.21.15.6:0/863193452 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f010416abf0 0x7f010420ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:22:15.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:15.956+0000 7f0108c00640 1 -- 172.21.15.6:0/863193452 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f010416abf0 msgr2=0x7f010420ec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:22:15.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:15.956+0000 7f0108c00640 1 --2- 172.21.15.6:0/863193452 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f010416abf0 0x7f010420ec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:22:15.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:15.957+0000 7f0108c00640 1 -- 172.21.15.6:0/863193452 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0104167180 msgr2=0x7f010420e6f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-24T23:22:15.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:15.957+0000 7f0108c00640 1 --2- 172.21.15.6:0/863193452 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0104167180 0x7f010420e6f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:22:15.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:15.957+0000 7f0108c00640 1 -- 172.21.15.6:0/863193452 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01040c4f90 con 0x7f01040c7dc0 2024-04-24T23:22:15.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:15.957+0000 7f0108c00640 1 --2- 172.21.15.6:0/863193452 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f01040c7dc0 0x7f01040c8470 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f00f000c4c0 tx=0x7f00f000c980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:22:15.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:15.957+0000 7f0100c00640 1 -- 172.21.15.6:0/863193452 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00f00052d0 con 0x7f01040c7dc0 2024-04-24T23:22:15.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:15.957+0000 7f0100c00640 1 -- 172.21.15.6:0/863193452 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f00f0005470 con 0x7f01040c7dc0 2024-04-24T23:22:15.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:15.957+0000 7f0100c00640 1 -- 172.21.15.6:0/863193452 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00f0012930 con 0x7f01040c7dc0 2024-04-24T23:22:15.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:15.957+0000 7f010b600640 1 -- 172.21.15.6:0/863193452 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01040c5270 con 0x7f01040c7dc0 2024-04-24T23:22:15.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:15.958+0000 7f010b600640 1 -- 172.21.15.6:0/863193452 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f01040c5830 con 0x7f01040c7dc0 2024-04-24T23:22:15.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:15.959+0000 7f010b600640 1 -- 172.21.15.6:0/863193452 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f010406f570 con 0x7f01040c7dc0 2024-04-24T23:22:15.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:15.960+0000 7f0100c00640 1 -- 172.21.15.6:0/863193452 <== mon.2 v2:172.21.15.190:3301/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f00f0012ad0 con 0x7f01040c7dc0 2024-04-24T23:22:15.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:15.962+0000 7f0100c00640 1 --2- 172.21.15.6:0/863193452 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f00e406beb0 0x7f00e406e370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:22:15.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:15.962+0000 7f0100c00640 1 -- 172.21.15.6:0/863193452 <== mon.2 v2:172.21.15.190:3301/0 5 ==== osd_map(28..28 src has 1..28) ==== 6762+0+0 (secure 0 0 0) 0x7f00f008af30 con 0x7f01040c7dc0 2024-04-24T23:22:15.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:15.969+0000 7f0103e00640 1 --2- 172.21.15.6:0/863193452 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f00e406beb0 0x7f00e406e370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:22:15.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:15.970+0000 7f0100c00640 1 -- 172.21.15.6:0/863193452 <== mon.2 v2:172.21.15.190:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f010406f570 con 0x7f01040c7dc0 2024-04-24T23:22:15.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:15.977+0000 7f0103e00640 1 --2- 172.21.15.6:0/863193452 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f00e406beb0 0x7f00e406e370 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f0104165980 tx=0x7f00f8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:22:16.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:16.533+0000 7f010b600640 1 -- 172.21.15.6:0/863193452 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f01040c7970 con 0x7f01040c7dc0 2024-04-24T23:22:16.536 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:22:16.537 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5081,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/1190759626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1190759626},{"type":"v1","addr":"172.21.15.6:6834","nonce":1190759626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:22:07.460817+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4282,"mds_2":4235,"mds_3":4232,"mds_4":5257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4232":{"gid":4232,"name":"b","rank":3,"incarnation":15,"state":"up:active","state_seq":15,"addr":"172.21.15.190:6838/2032280158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2032280158},{"type":"v1","addr":"172.21.15.190:6838","nonce":2032280158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4235":{"gid":4235,"name":"d","rank":2,"incarnation":13,"state":"up:active","state_seq":14,"addr":"172.21.15.190:6839/1572872451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":1572872451},{"type":"v1","addr":"172.21.15.190:6839","nonce":1572872451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4282":{"gid":4282,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":12,"addr":"172.21.15.190:6837/580315269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":580315269},{"type":"v1","addr":"172.21.15.190:6837","nonce":580315269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/489092393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":489092393},{"type":"v1","addr":"172.21.15.6:6835","nonce":489092393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5257":{"gid":5257,"name":"e","rank":4,"incarnation":25,"state":"up:stopping","state_seq":43,"addr":"172.21.15.6:6837/3240971655","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":3240971655},{"type":"v1","addr":"172.21.15.6:6837","nonce":3240971655}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4306,"qdb_cluster":[4232,4235,4282,4306]},"id":1}]} 2024-04-24T23:22:16.537 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 32 2024-04-24T23:22:16.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:16.535+0000 7f0100c00640 1 -- 172.21.15.6:0/863193452 <== mon.2 v2:172.21.15.190:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+6904 (secure 0 0 0) 0x7f00f0059410 con 0x7f01040c7dc0 2024-04-24T23:22:16.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:16.539+0000 7f010b600640 1 -- 172.21.15.6:0/863193452 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f00e406beb0 msgr2=0x7f00e406e370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:22:16.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:16.539+0000 7f010b600640 1 --2- 172.21.15.6:0/863193452 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f00e406beb0 0x7f00e406e370 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f0104165980 tx=0x7f00f8008040 comp rx=0 tx=0).stop 2024-04-24T23:22:16.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:16.539+0000 7f010b600640 1 -- 172.21.15.6:0/863193452 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f01040c7dc0 msgr2=0x7f01040c8470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:22:16.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:16.540+0000 7f010b600640 1 --2- 172.21.15.6:0/863193452 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f01040c7dc0 0x7f01040c8470 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f00f000c4c0 tx=0x7f00f000c980 comp rx=0 tx=0).stop 2024-04-24T23:22:16.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:16.540+0000 7f010b600640 1 -- 172.21.15.6:0/863193452 shutdown_connections 2024-04-24T23:22:16.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:16.540+0000 7f010b600640 1 --2- 172.21.15.6:0/863193452 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f00e406beb0 0x7f00e406e370 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:22:16.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:16.540+0000 7f010b600640 1 --2- 172.21.15.6:0/863193452 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f010416abf0 0x7f010420ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:22:16.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:16.540+0000 7f010b600640 1 --2- 172.21.15.6:0/863193452 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0104167180 0x7f010420e6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:22:16.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:16.540+0000 7f010b600640 1 --2- 172.21.15.6:0/863193452 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f01040c7dc0 0x7f01040c8470 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:22:16.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:16.540+0000 7f010b600640 1 -- 172.21.15.6:0/863193452 >> 172.21.15.6:0/863193452 conn(0x7f010406b0e0 msgr2=0x7f01040ca9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:22:16.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:16.541+0000 7f010b600640 1 -- 172.21.15.6:0/863193452 shutdown_connections 2024-04-24T23:22:16.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:16.541+0000 7f010b600640 1 -- 172.21.15.6:0/863193452 wait complete. 2024-04-24T23:22:16.577 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-24T23:22:18.579 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:22:18.920 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:18.919+0000 7f9ccba00640 1 Processor -- start 2024-04-24T23:22:18.920 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:18.919+0000 7f9ccba00640 1 -- start start 2024-04-24T23:22:18.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:18.921+0000 7f9ccba00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f9cc40c62c0 0x7f9cc4067940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:22:18.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:18.921+0000 7f9ccba00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9cc415b930 con 0x7f9cc4154c80 2024-04-24T23:22:18.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:18.921+0000 7f9ccba00640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9cc40c5500 con 0x7f9cc406bf90 2024-04-24T23:22:18.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:18.921+0000 7f9ccba00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f9cc40cb1a0 con 0x7f9cc40c62c0 2024-04-24T23:22:18.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:18.923+0000 7f9cc9a00640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f9cc406bf90 0x7f9cc406c400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.6:45134/0 (socket says 172.21.15.6:45134) 2024-04-24T23:22:18.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:18.923+0000 7f9cc9a00640 1 -- 172.21.15.6:0/3621265760 learned_addr learned my addr 172.21.15.6:0/3621265760 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:22:18.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:18.924+0000 7f9cc3e00640 1 --2- 172.21.15.6:0/3621265760 >> v2:172.21.15.6:3300/0 conn(0x7f9cc40c62c0 0x7f9cc4067940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:22:18.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:18.924+0000 7f9cc3400640 1 -- 172.21.15.6:0/3621265760 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2103909019 0 0) 0x7f9cc40c5500 con 0x7f9cc406bf90 2024-04-24T23:22:18.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:18.924+0000 7f9cc3400640 1 -- 172.21.15.6:0/3621265760 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9cac003520 con 0x7f9cc406bf90 2024-04-24T23:22:18.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:18.925+0000 7f9cc3400640 1 -- 172.21.15.6:0/3621265760 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 894286885 0 0) 0x7f9cac003520 con 0x7f9cc406bf90 2024-04-24T23:22:18.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:18.925+0000 7f9cc3400640 1 -- 172.21.15.6:0/3621265760 >> v2:172.21.15.6:3300/0 conn(0x7f9cc40c62c0 msgr2=0x7f9cc4067940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:22:18.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:18.925+0000 7f9cc3400640 1 --2- 172.21.15.6:0/3621265760 >> v2:172.21.15.6:3300/0 conn(0x7f9cc40c62c0 0x7f9cc4067940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:22:18.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:18.925+0000 7f9cc3400640 1 -- 172.21.15.6:0/3621265760 >> v1:172.21.15.6:6789/0 conn(0x7f9cc4154c80 legacy=0x7f9cc40cbb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:22:18.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:18.925+0000 7f9cc3400640 1 -- 172.21.15.6:0/3621265760 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9cc40c57e0 con 0x7f9cc406bf90 2024-04-24T23:22:18.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:18.925+0000 7f9cc3400640 1 -- 172.21.15.6:0/3621265760 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3463268701 0 0) 0x7f9cbc0019f0 con 0x7f9cc406bf90 2024-04-24T23:22:18.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:18.925+0000 7f9cc3400640 1 -- 172.21.15.6:0/3621265760 >> v1:172.21.15.190:6789/0 conn(0x7f9cc406bf90 legacy=0x7f9cc406c400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:22:18.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:18.925+0000 7f9cc3400640 1 --2- 172.21.15.6:0/3621265760 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9cac0039b0 0x7f9cac003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:22:18.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:18.925+0000 7f9cc3400640 1 --2- 172.21.15.6:0/3621265760 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9cac004800 0x7f9cac004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:22:18.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:18.925+0000 7f9cc3400640 1 --2- 172.21.15.6:0/3621265760 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9cac0051c0 0x7f9cac00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:22:18.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:18.925+0000 7f9cc3400640 1 -- 172.21.15.6:0/3621265760 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f9cac00c090 con 0x7f9cac0039b0 2024-04-24T23:22:18.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:18.925+0000 7f9cc3400640 1 -- 172.21.15.6:0/3621265760 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f9cac00c240 con 0x7f9cac004800 2024-04-24T23:22:18.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:18.925+0000 7f9cc3400640 1 -- 172.21.15.6:0/3621265760 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f9cac00c440 con 0x7f9cac0051c0 2024-04-24T23:22:18.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:18.929+0000 7f9cc9a00640 1 --2- 172.21.15.6:0/3621265760 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9cac0051c0 0x7f9cac00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:22:18.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:18.929+0000 7f9cc3e00640 1 --2- 172.21.15.6:0/3621265760 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9cac004800 0x7f9cac004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:22:18.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:18.929+0000 7f9cc9a00640 1 -- 172.21.15.6:0/3621265760 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9cac004800 msgr2=0x7f9cac004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:22:18.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:18.929+0000 7f9cc9a00640 1 --2- 172.21.15.6:0/3621265760 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9cac004800 0x7f9cac004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:22:18.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:18.929+0000 7f9cc9a00640 1 -- 172.21.15.6:0/3621265760 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9cac0039b0 msgr2=0x7f9cac003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:22:18.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:18.929+0000 7f9cc9a00640 1 --2- 172.21.15.6:0/3621265760 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9cac0039b0 0x7f9cac003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:22:18.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:18.929+0000 7f9cc9a00640 1 -- 172.21.15.6:0/3621265760 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9cc40c57e0 con 0x7f9cac0051c0 2024-04-24T23:22:18.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:18.932+0000 7f9cc9a00640 1 -- 172.21.15.6:0/3621265760 reap_dead start 2024-04-24T23:22:18.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:18.933+0000 7f9cc9a00640 1 --2- 172.21.15.6:0/3621265760 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9cac0051c0 0x7f9cac00b9c0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f9cbc00a970 tx=0x7f9cac005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cab3e6af270f9bff server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:22:18.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:18.934+0000 7f9cc3400640 1 -- 172.21.15.6:0/3621265760 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cbc00a540 con 0x7f9cac0051c0 2024-04-24T23:22:18.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:18.934+0000 7f9cc3400640 1 -- 172.21.15.6:0/3621265760 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9cbc02f070 con 0x7f9cac0051c0 2024-04-24T23:22:18.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:18.934+0000 7f9cc3400640 1 -- 172.21.15.6:0/3621265760 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cbc02bde0 con 0x7f9cac0051c0 2024-04-24T23:22:18.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:18.936+0000 7f9ccba00640 1 -- 172.21.15.6:0/3621265760 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9cac0051c0 msgr2=0x7f9cac00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:22:18.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:18.936+0000 7f9ccba00640 1 --2- 172.21.15.6:0/3621265760 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9cac0051c0 0x7f9cac00b9c0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f9cbc00a970 tx=0x7f9cac005830 comp rx=0 tx=0).stop 2024-04-24T23:22:18.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:18.937+0000 7f9ccba00640 1 -- 172.21.15.6:0/3621265760 shutdown_connections 2024-04-24T23:22:18.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:18.938+0000 7f9ccba00640 1 --2- 172.21.15.6:0/3621265760 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9cac0051c0 0x7f9cac00b9c0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:22:18.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:18.938+0000 7f9ccba00640 1 -- 172.21.15.6:0/3621265760 >> 172.21.15.6:0/3621265760 conn(0x7f9cc4072810 msgr2=0x7f9cc4066ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:22:18.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:18.939+0000 7f9ccba00640 1 -- 172.21.15.6:0/3621265760 shutdown_connections 2024-04-24T23:22:18.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:18.939+0000 7f9ccba00640 1 -- 172.21.15.6:0/3621265760 wait complete. 2024-04-24T23:22:18.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:18.947+0000 7f9ccba00640 1 Processor -- start 2024-04-24T23:22:18.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:18.948+0000 7f9ccba00640 1 -- start start 2024-04-24T23:22:18.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:18.953+0000 7f9ccba00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9cac0051c0 0x7f9cc40c9530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:22:18.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:18.955+0000 7f9ccba00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9cc40c7e30 0x7f9cc4159c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:22:18.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:18.955+0000 7f9cc9000640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9cac0051c0 0x7f9cc40c9530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:22:18.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:18.955+0000 7f9cc9000640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9cac0051c0 0x7f9cc40c9530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.6:39084/0 (socket says 172.21.15.6:39084) 2024-04-24T23:22:18.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:18.956+0000 7f9cc9000640 1 -- 172.21.15.6:0/64509980 learned_addr learned my addr 172.21.15.6:0/64509980 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:22:18.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:18.956+0000 7f9ccba00640 1 --2- 172.21.15.6:0/64509980 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9cc415a170 0x7f9cc4149da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:22:18.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:18.956+0000 7f9ccba00640 1 -- 172.21.15.6:0/64509980 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f9cc415b580 con 0x7f9cc415a170 2024-04-24T23:22:18.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:18.956+0000 7f9ccba00640 1 -- 172.21.15.6:0/64509980 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f9cc40c5e30 con 0x7f9cc40c7e30 2024-04-24T23:22:18.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:18.958+0000 7f9ccba00640 1 -- 172.21.15.6:0/64509980 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f9cc40c8ae0 con 0x7f9cac0051c0 2024-04-24T23:22:18.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:18.958+0000 7f9cc3e00640 1 --2- 172.21.15.6:0/64509980 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9cc40c7e30 0x7f9cc4159c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:22:18.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:18.959+0000 7f9cc9a00640 1 --2- 172.21.15.6:0/64509980 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9cc415a170 0x7f9cc4149da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:22:18.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:18.961+0000 7f9cc9000640 1 -- 172.21.15.6:0/64509980 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9cc40c7e30 msgr2=0x7f9cc4159c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:22:18.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:18.961+0000 7f9cc9000640 1 --2- 172.21.15.6:0/64509980 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9cc40c7e30 0x7f9cc4159c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:22:18.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:18.961+0000 7f9cc9000640 1 -- 172.21.15.6:0/64509980 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9cc415a170 msgr2=0x7f9cc4149da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:22:18.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:18.961+0000 7f9cc9000640 1 --2- 172.21.15.6:0/64509980 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9cc415a170 0x7f9cc4149da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:22:18.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:18.961+0000 7f9cc9000640 1 -- 172.21.15.6:0/64509980 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9cc41501d0 con 0x7f9cac0051c0 2024-04-24T23:22:18.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:18.961+0000 7f9cc9a00640 1 --2- 172.21.15.6:0/64509980 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9cc415a170 0x7f9cc4149da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:22:18.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:18.961+0000 7f9cc3e00640 1 --2- 172.21.15.6:0/64509980 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9cc40c7e30 0x7f9cc4159c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:22:18.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:18.962+0000 7f9cc9000640 1 --2- 172.21.15.6:0/64509980 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9cac0051c0 0x7f9cc40c9530 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f9cb400c4f0 tx=0x7f9cb400c9b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:22:18.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:18.965+0000 7f9cc0c00640 1 -- 172.21.15.6:0/64509980 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cb4016030 con 0x7f9cac0051c0 2024-04-24T23:22:18.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:18.965+0000 7f9cc0c00640 1 -- 172.21.15.6:0/64509980 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9cb4007020 con 0x7f9cac0051c0 2024-04-24T23:22:18.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:18.965+0000 7f9ccba00640 1 -- 172.21.15.6:0/64509980 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9cc41f0f40 con 0x7f9cac0051c0 2024-04-24T23:22:18.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:18.965+0000 7f9ccba00640 1 -- 172.21.15.6:0/64509980 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f9cc40c8d30 con 0x7f9cac0051c0 2024-04-24T23:22:18.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:18.976+0000 7f9cc0c00640 1 -- 172.21.15.6:0/64509980 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cb40073f0 con 0x7f9cac0051c0 2024-04-24T23:22:18.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:18.982+0000 7f9cc0c00640 1 -- 172.21.15.6:0/64509980 <== mon.2 v2:172.21.15.190:3301/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f9cb4019070 con 0x7f9cac0051c0 2024-04-24T23:22:18.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:18.984+0000 7f9cc0c00640 1 --2- 172.21.15.6:0/64509980 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f9ca806bf00 0x7f9ca806e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:22:18.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:18.984+0000 7f9cc0c00640 1 -- 172.21.15.6:0/64509980 <== mon.2 v2:172.21.15.190:3301/0 5 ==== osd_map(28..28 src has 1..28) ==== 6762+0+0 (secure 0 0 0) 0x7f9cb408ae70 con 0x7f9cac0051c0 2024-04-24T23:22:18.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:18.985+0000 7f9cc3e00640 1 --2- 172.21.15.6:0/64509980 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f9ca806bf00 0x7f9ca806e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:22:18.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:18.986+0000 7f9cc3e00640 1 --2- 172.21.15.6:0/64509980 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f9ca806bf00 0x7f9ca806e3c0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f9cc415ade0 tx=0x7f9cb8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:22:18.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:18.987+0000 7f9ccba00640 1 -- 172.21.15.6:0/64509980 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ca0005290 con 0x7f9cac0051c0 2024-04-24T23:22:18.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:18.996+0000 7f9cc0c00640 1 -- 172.21.15.6:0/64509980 <== mon.2 v2:172.21.15.190:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f9cb4059350 con 0x7f9cac0051c0 2024-04-24T23:22:19.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:19.561+0000 7f9ccba00640 1 -- 172.21.15.6:0/64509980 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9ca0005580 con 0x7f9cac0051c0 2024-04-24T23:22:19.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:19.562+0000 7f9cc0c00640 1 -- 172.21.15.6:0/64509980 <== mon.2 v2:172.21.15.190:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+6904 (secure 0 0 0) 0x7f9cb4058c50 con 0x7f9cac0051c0 2024-04-24T23:22:19.563 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:22:19.563 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5081,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/1190759626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1190759626},{"type":"v1","addr":"172.21.15.6:6834","nonce":1190759626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:22:07.460817+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4282,"mds_2":4235,"mds_3":4232,"mds_4":5257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4232":{"gid":4232,"name":"b","rank":3,"incarnation":15,"state":"up:active","state_seq":15,"addr":"172.21.15.190:6838/2032280158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2032280158},{"type":"v1","addr":"172.21.15.190:6838","nonce":2032280158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4235":{"gid":4235,"name":"d","rank":2,"incarnation":13,"state":"up:active","state_seq":14,"addr":"172.21.15.190:6839/1572872451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":1572872451},{"type":"v1","addr":"172.21.15.190:6839","nonce":1572872451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4282":{"gid":4282,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":12,"addr":"172.21.15.190:6837/580315269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":580315269},{"type":"v1","addr":"172.21.15.190:6837","nonce":580315269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/489092393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":489092393},{"type":"v1","addr":"172.21.15.6:6835","nonce":489092393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5257":{"gid":5257,"name":"e","rank":4,"incarnation":25,"state":"up:stopping","state_seq":43,"addr":"172.21.15.6:6837/3240971655","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":3240971655},{"type":"v1","addr":"172.21.15.6:6837","nonce":3240971655}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4306,"qdb_cluster":[4232,4235,4282,4306]},"id":1}]} 2024-04-24T23:22:19.564 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 32 2024-04-24T23:22:19.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:19.566+0000 7f9ccba00640 1 -- 172.21.15.6:0/64509980 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f9ca806bf00 msgr2=0x7f9ca806e3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:22:19.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:19.566+0000 7f9ccba00640 1 --2- 172.21.15.6:0/64509980 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f9ca806bf00 0x7f9ca806e3c0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f9cc415ade0 tx=0x7f9cb8008040 comp rx=0 tx=0).stop 2024-04-24T23:22:19.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:19.566+0000 7f9ccba00640 1 -- 172.21.15.6:0/64509980 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9cac0051c0 msgr2=0x7f9cc40c9530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:22:19.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:19.566+0000 7f9ccba00640 1 --2- 172.21.15.6:0/64509980 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9cac0051c0 0x7f9cc40c9530 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f9cb400c4f0 tx=0x7f9cb400c9b0 comp rx=0 tx=0).stop 2024-04-24T23:22:19.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:19.567+0000 7f9ccba00640 1 -- 172.21.15.6:0/64509980 shutdown_connections 2024-04-24T23:22:19.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:19.567+0000 7f9ccba00640 1 --2- 172.21.15.6:0/64509980 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f9ca806bf00 0x7f9ca806e3c0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:22:19.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:19.567+0000 7f9ccba00640 1 --2- 172.21.15.6:0/64509980 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9cc415a170 0x7f9cc4149da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:22:19.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:19.567+0000 7f9ccba00640 1 --2- 172.21.15.6:0/64509980 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9cc40c7e30 0x7f9cc4159c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:22:19.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:19.567+0000 7f9ccba00640 1 --2- 172.21.15.6:0/64509980 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9cac0051c0 0x7f9cc40c9530 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:22:19.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:19.567+0000 7f9ccba00640 1 -- 172.21.15.6:0/64509980 >> 172.21.15.6:0/64509980 conn(0x7f9cc4072810 msgr2=0x7f9cc40b3810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:22:19.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:19.567+0000 7f9ccba00640 1 -- 172.21.15.6:0/64509980 shutdown_connections 2024-04-24T23:22:19.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:19.567+0000 7f9ccba00640 1 -- 172.21.15.6:0/64509980 wait complete. 2024-04-24T23:22:19.609 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-24T23:22:21.610 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:22:21.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:21.812+0000 7fd8c5800640 1 Processor -- start 2024-04-24T23:22:21.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:21.813+0000 7fd8c5800640 1 -- start start 2024-04-24T23:22:21.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:21.827+0000 7fd8c5800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd8c0150990 0x7fd8c015a980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:22:21.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:21.828+0000 7fd8c5800640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd8c0073270 0x7fd8c00c9110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:22:21.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:21.830+0000 7fd8c5800640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd8c015aec0 con 0x7fd8c006bbd0 2024-04-24T23:22:21.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:21.830+0000 7fd8c5800640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7fd8c00c8c50 con 0x7fd8c0150990 2024-04-24T23:22:21.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:21.830+0000 7fd8c5800640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fd8c00c8690 con 0x7fd8c0073270 2024-04-24T23:22:21.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:21.831+0000 7fd8bea00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd8c0150990 0x7fd8c015a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:22:21.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:21.831+0000 7fd8bea00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd8c0150990 0x7fd8c015a980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:37402/0 (socket says 172.21.15.6:37402) 2024-04-24T23:22:21.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:21.832+0000 7fd8bea00640 1 -- 172.21.15.6:0/1418560036 learned_addr learned my addr 172.21.15.6:0/1418560036 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:22:21.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:21.832+0000 7fd8bea00640 1 -- 172.21.15.6:0/1418560036 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd8c0073270 msgr2=0x7fd8c00c9110 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-24T23:22:21.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:21.833+0000 7fd8bea00640 1 --2- 172.21.15.6:0/1418560036 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd8c0073270 0x7fd8c00c9110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:22:21.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:21.833+0000 7fd8bea00640 1 -- 172.21.15.6:0/1418560036 >> v1:172.21.15.190:6789/0 conn(0x7fd8c006bbd0 legacy=0x7fd8c006e3a0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-24T23:22:21.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:21.833+0000 7fd8bea00640 1 -- 172.21.15.6:0/1418560036 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8c00c8970 con 0x7fd8c0150990 2024-04-24T23:22:21.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:21.837+0000 7fd8bea00640 1 --2- 172.21.15.6:0/1418560036 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd8c0150990 0x7fd8c015a980 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fd8b0009870 tx=0x7fd8b002a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2668d9e2c1d0dc9 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:22:21.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:21.841+0000 7fd8bd600640 1 -- 172.21.15.6:0/1418560036 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8b0037070 con 0x7fd8c0150990 2024-04-24T23:22:21.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:21.841+0000 7fd8bd600640 1 -- 172.21.15.6:0/1418560036 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd8b0032440 con 0x7fd8c0150990 2024-04-24T23:22:21.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:21.841+0000 7fd8bd600640 1 -- 172.21.15.6:0/1418560036 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8b0032810 con 0x7fd8c0150990 2024-04-24T23:22:21.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:21.844+0000 7fd8c5800640 1 -- 172.21.15.6:0/1418560036 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd8c0150990 msgr2=0x7fd8c015a980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:22:21.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:21.844+0000 7fd8c5800640 1 --2- 172.21.15.6:0/1418560036 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd8c0150990 0x7fd8c015a980 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fd8b0009870 tx=0x7fd8b002a230 comp rx=0 tx=0).stop 2024-04-24T23:22:21.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:21.846+0000 7fd8c5800640 1 -- 172.21.15.6:0/1418560036 shutdown_connections 2024-04-24T23:22:21.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:21.847+0000 7fd8c5800640 1 --2- 172.21.15.6:0/1418560036 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd8c0073270 0x7fd8c00c9110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:22:21.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:21.847+0000 7fd8c5800640 1 --2- 172.21.15.6:0/1418560036 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd8c0150990 0x7fd8c015a980 secure :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fd8b0009870 tx=0x7fd8b002a230 comp rx=0 tx=0).stop 2024-04-24T23:22:21.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:21.847+0000 7fd8c5800640 1 -- 172.21.15.6:0/1418560036 >> 172.21.15.6:0/1418560036 conn(0x7fd8c006b0e0 msgr2=0x7fd8c006b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:22:21.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:21.851+0000 7fd8c5800640 1 -- 172.21.15.6:0/1418560036 shutdown_connections 2024-04-24T23:22:21.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:21.851+0000 7fd8c5800640 1 -- 172.21.15.6:0/1418560036 wait complete. 2024-04-24T23:22:21.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:21.885+0000 7fd8c5800640 1 Processor -- start 2024-04-24T23:22:21.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:21.885+0000 7fd8c5800640 1 -- start start 2024-04-24T23:22:21.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:21.892+0000 7fd8c5800640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd8c006bbd0 0x7fd8c014ff10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:22:21.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:21.892+0000 7fd8c5800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd8c0073270 0x7fd8c01eb630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:22:21.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:21.892+0000 7fd8c5800640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd8c014f440 0x7fd8c014f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:22:21.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:21.892+0000 7fd8c5800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fd8c00c7a70 con 0x7fd8c0073270 2024-04-24T23:22:21.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:21.892+0000 7fd8c5800640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fd8c015a460 con 0x7fd8c014f440 2024-04-24T23:22:21.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:21.892+0000 7fd8c5800640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fd8c0150d70 con 0x7fd8c006bbd0 2024-04-24T23:22:21.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:21.898+0000 7fd8be000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd8c0073270 0x7fd8c01eb630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:22:21.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:21.899+0000 7fd8be000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd8c0073270 0x7fd8c01eb630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:37418/0 (socket says 172.21.15.6:37418) 2024-04-24T23:22:21.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:21.899+0000 7fd8be000640 1 -- 172.21.15.6:0/2645503899 learned_addr learned my addr 172.21.15.6:0/2645503899 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:22:21.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:21.899+0000 7fd8bf400640 1 --2- 172.21.15.6:0/2645503899 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd8c014f440 0x7fd8c014f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:22:21.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:21.901+0000 7fd8bf400640 1 -- 172.21.15.6:0/2645503899 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd8c006bbd0 msgr2=0x7fd8c014ff10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-24T23:22:21.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:21.901+0000 7fd8bf400640 1 --2- 172.21.15.6:0/2645503899 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd8c006bbd0 0x7fd8c014ff10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:22:21.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:21.901+0000 7fd8bf400640 1 -- 172.21.15.6:0/2645503899 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd8c0073270 msgr2=0x7fd8c01eb630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:22:21.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:21.906+0000 7fd8bf400640 1 --2- 172.21.15.6:0/2645503899 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd8c0073270 0x7fd8c01eb630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:22:21.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:21.906+0000 7fd8bf400640 1 -- 172.21.15.6:0/2645503899 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8c01f4f40 con 0x7fd8c014f440 2024-04-24T23:22:21.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:21.906+0000 7fd8be000640 1 --2- 172.21.15.6:0/2645503899 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd8c0073270 0x7fd8c01eb630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:22:21.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:21.913+0000 7fd8bf400640 1 --2- 172.21.15.6:0/2645503899 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd8c014f440 0x7fd8c014f8d0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fd8b800bdf0 tx=0x7fd8b8009ef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:22:21.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:21.913+0000 7fd8b6a00640 1 -- 172.21.15.6:0/2645503899 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8b8019070 con 0x7fd8c014f440 2024-04-24T23:22:21.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:21.913+0000 7fd8b6a00640 1 -- 172.21.15.6:0/2645503899 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd8b8014440 con 0x7fd8c014f440 2024-04-24T23:22:21.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:21.913+0000 7fd8c5800640 1 -- 172.21.15.6:0/2645503899 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8c01f51e0 con 0x7fd8c014f440 2024-04-24T23:22:21.919 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:21.913+0000 7fd8b6a00640 1 -- 172.21.15.6:0/2645503899 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8b8014830 con 0x7fd8c014f440 2024-04-24T23:22:21.919 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:21.913+0000 7fd8c5800640 1 -- 172.21.15.6:0/2645503899 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8c01f5780 con 0x7fd8c014f440 2024-04-24T23:22:21.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:21.921+0000 7fd8c5800640 1 -- 172.21.15.6:0/2645503899 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd898005290 con 0x7fd8c014f440 2024-04-24T23:22:21.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:21.923+0000 7fd8b6a00640 1 -- 172.21.15.6:0/2645503899 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fd8b801c070 con 0x7fd8c014f440 2024-04-24T23:22:21.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:21.925+0000 7fd8b6a00640 1 --2- 172.21.15.6:0/2645503899 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fd8a006be80 0x7fd8a006e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:22:21.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:21.925+0000 7fd8b6a00640 1 -- 172.21.15.6:0/2645503899 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6762+0+0 (secure 0 0 0) 0x7fd8b808b140 con 0x7fd8c014f440 2024-04-24T23:22:21.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:21.926+0000 7fd8bea00640 1 --2- 172.21.15.6:0/2645503899 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fd8a006be80 0x7fd8a006e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:22:21.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:21.928+0000 7fd8bea00640 1 --2- 172.21.15.6:0/2645503899 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fd8a006be80 0x7fd8a006e340 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fd8b0034000 tx=0x7fd8b002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:22:21.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:21.944+0000 7fd8b6a00640 1 -- 172.21.15.6:0/2645503899 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fd8b8058f50 con 0x7fd8c014f440 2024-04-24T23:22:22.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:22.512+0000 7fd8c5800640 1 -- 172.21.15.6:0/2645503899 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd898005580 con 0x7fd8c014f440 2024-04-24T23:22:22.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:22.513+0000 7fd8b6a00640 1 -- 172.21.15.6:0/2645503899 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+6904 (secure 0 0 0) 0x7fd8b805d250 con 0x7fd8c014f440 2024-04-24T23:22:22.519 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:22:22.519 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5081,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/1190759626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1190759626},{"type":"v1","addr":"172.21.15.6:6834","nonce":1190759626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:22:07.460817+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4282,"mds_2":4235,"mds_3":4232,"mds_4":5257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4232":{"gid":4232,"name":"b","rank":3,"incarnation":15,"state":"up:active","state_seq":15,"addr":"172.21.15.190:6838/2032280158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2032280158},{"type":"v1","addr":"172.21.15.190:6838","nonce":2032280158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4235":{"gid":4235,"name":"d","rank":2,"incarnation":13,"state":"up:active","state_seq":14,"addr":"172.21.15.190:6839/1572872451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":1572872451},{"type":"v1","addr":"172.21.15.190:6839","nonce":1572872451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4282":{"gid":4282,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":12,"addr":"172.21.15.190:6837/580315269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":580315269},{"type":"v1","addr":"172.21.15.190:6837","nonce":580315269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/489092393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":489092393},{"type":"v1","addr":"172.21.15.6:6835","nonce":489092393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5257":{"gid":5257,"name":"e","rank":4,"incarnation":25,"state":"up:stopping","state_seq":43,"addr":"172.21.15.6:6837/3240971655","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":3240971655},{"type":"v1","addr":"172.21.15.6:6837","nonce":3240971655}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4306,"qdb_cluster":[4232,4235,4282,4306]},"id":1}]} 2024-04-24T23:22:22.520 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 32 2024-04-24T23:22:22.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:22.520+0000 7fd8c5800640 1 -- 172.21.15.6:0/2645503899 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fd8a006be80 msgr2=0x7fd8a006e340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:22:22.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:22.520+0000 7fd8c5800640 1 --2- 172.21.15.6:0/2645503899 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fd8a006be80 0x7fd8a006e340 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fd8b0034000 tx=0x7fd8b002e000 comp rx=0 tx=0).stop 2024-04-24T23:22:22.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:22.520+0000 7fd8c5800640 1 -- 172.21.15.6:0/2645503899 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd8c014f440 msgr2=0x7fd8c014f8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:22:22.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:22.520+0000 7fd8c5800640 1 --2- 172.21.15.6:0/2645503899 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd8c014f440 0x7fd8c014f8d0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fd8b800bdf0 tx=0x7fd8b8009ef0 comp rx=0 tx=0).stop 2024-04-24T23:22:22.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:22.520+0000 7fd8c5800640 1 -- 172.21.15.6:0/2645503899 shutdown_connections 2024-04-24T23:22:22.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:22.520+0000 7fd8c5800640 1 --2- 172.21.15.6:0/2645503899 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fd8a006be80 0x7fd8a006e340 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:22:22.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:22.520+0000 7fd8c5800640 1 --2- 172.21.15.6:0/2645503899 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd8c014f440 0x7fd8c014f8d0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:22:22.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:22.520+0000 7fd8c5800640 1 --2- 172.21.15.6:0/2645503899 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd8c0073270 0x7fd8c01eb630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:22:22.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:22.520+0000 7fd8c5800640 1 --2- 172.21.15.6:0/2645503899 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd8c006bbd0 0x7fd8c014ff10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:22:22.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:22.520+0000 7fd8c5800640 1 -- 172.21.15.6:0/2645503899 >> 172.21.15.6:0/2645503899 conn(0x7fd8c006b0e0 msgr2=0x7fd8c00c9110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:22:22.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:22.520+0000 7fd8c5800640 1 -- 172.21.15.6:0/2645503899 shutdown_connections 2024-04-24T23:22:22.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:22.521+0000 7fd8c5800640 1 -- 172.21.15.6:0/2645503899 wait complete. 2024-04-24T23:22:22.553 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-24T23:22:24.554 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:22:24.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:24.937+0000 7fb7c1000640 1 Processor -- start 2024-04-24T23:22:24.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:24.937+0000 7fb7c1000640 1 -- start start 2024-04-24T23:22:24.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:24.939+0000 7fb7c1000640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fb7bc073270 0x7fb7bc0c9110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:22:24.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:24.939+0000 7fb7c1000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fb7bc157470 0x7fb7bc0c9650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:22:24.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:24.939+0000 7fb7c1000640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb7bc0c8690 con 0x7fb7bc150990 2024-04-24T23:22:24.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:24.939+0000 7fb7c1000640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7fb7bc0c8c50 con 0x7fb7bc157470 2024-04-24T23:22:24.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:24.939+0000 7fb7c1000640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7fb7bc0c8970 con 0x7fb7bc073270 2024-04-24T23:22:24.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:24.941+0000 7fb7b9400640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fb7bc073270 0x7fb7bc0c9110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:22:24.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:24.941+0000 7fb7b9e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fb7bc150990 0x7fb7bc15a980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:55438/0 (socket says 172.21.15.6:55438) 2024-04-24T23:22:24.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:24.941+0000 7fb7b9e00640 1 -- 172.21.15.6:0/1075942352 learned_addr learned my addr 172.21.15.6:0/1075942352 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:22:24.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:24.941+0000 7fb7ba800640 1 --2- 172.21.15.6:0/1075942352 >> v2:172.21.15.6:3300/0 conn(0x7fb7bc157470 0x7fb7bc0c9650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:22:24.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:24.941+0000 7fb7b9400640 1 -- 172.21.15.6:0/1075942352 >> v2:172.21.15.6:3300/0 conn(0x7fb7bc157470 msgr2=0x7fb7bc0c9650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:22:24.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:24.941+0000 7fb7b9400640 1 --2- 172.21.15.6:0/1075942352 >> v2:172.21.15.6:3300/0 conn(0x7fb7bc157470 0x7fb7bc0c9650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:22:24.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:24.941+0000 7fb7b9400640 1 -- 172.21.15.6:0/1075942352 >> v1:172.21.15.6:6789/0 conn(0x7fb7bc150990 legacy=0x7fb7bc15a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:22:24.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:24.941+0000 7fb7b9400640 1 -- 172.21.15.6:0/1075942352 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7bc15a180 con 0x7fb7bc073270 2024-04-24T23:22:24.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:24.942+0000 7fb7b9400640 1 --2- 172.21.15.6:0/1075942352 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb7bc073270 0x7fb7bc0c9110 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fb7bc0c39b0 tx=0x7fb7ac02a0e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=285ae48cad3aa2ce server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:22:24.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:24.944+0000 7fb7b8a00640 1 -- 172.21.15.6:0/1075942352 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7ac037070 con 0x7fb7bc073270 2024-04-24T23:22:24.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:24.944+0000 7fb7b8a00640 1 -- 172.21.15.6:0/1075942352 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb7ac02adf0 con 0x7fb7bc073270 2024-04-24T23:22:24.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:24.944+0000 7fb7b8a00640 1 -- 172.21.15.6:0/1075942352 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7ac032660 con 0x7fb7bc073270 2024-04-24T23:22:24.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:24.945+0000 7fb7c1000640 1 -- 172.21.15.6:0/1075942352 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb7bc073270 msgr2=0x7fb7bc0c9110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:22:24.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:24.945+0000 7fb7c1000640 1 --2- 172.21.15.6:0/1075942352 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb7bc073270 0x7fb7bc0c9110 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fb7bc0c39b0 tx=0x7fb7ac02a0e0 comp rx=0 tx=0).stop 2024-04-24T23:22:24.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:24.945+0000 7fb7c1000640 1 -- 172.21.15.6:0/1075942352 shutdown_connections 2024-04-24T23:22:24.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:24.945+0000 7fb7c1000640 1 --2- 172.21.15.6:0/1075942352 >> v2:172.21.15.6:3300/0 conn(0x7fb7bc157470 0x7fb7bc0c9650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:22:24.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:24.945+0000 7fb7c1000640 1 --2- 172.21.15.6:0/1075942352 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb7bc073270 0x7fb7bc0c9110 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:22:24.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:24.945+0000 7fb7c1000640 1 -- 172.21.15.6:0/1075942352 >> 172.21.15.6:0/1075942352 conn(0x7fb7bc06b0e0 msgr2=0x7fb7bc06b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:22:24.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:24.945+0000 7fb7c1000640 1 -- 172.21.15.6:0/1075942352 shutdown_connections 2024-04-24T23:22:24.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:24.945+0000 7fb7c1000640 1 -- 172.21.15.6:0/1075942352 wait complete. 2024-04-24T23:22:24.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:24.948+0000 7fb7c1000640 1 Processor -- start 2024-04-24T23:22:24.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:24.948+0000 7fb7c1000640 1 -- start start 2024-04-24T23:22:24.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:24.949+0000 7fb7c1000640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb7bc073270 0x7fb7bc159980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:22:24.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:24.949+0000 7fb7c1000640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb7bc150990 0x7fb7bc200d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:22:24.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:24.949+0000 7fb7c1000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb7bc157470 0x7fb7bc1e2640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:22:24.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:24.949+0000 7fb7c1000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fb7bc150d70 con 0x7fb7bc157470 2024-04-24T23:22:24.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:24.949+0000 7fb7c1000640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fb7bc0c7a70 con 0x7fb7bc073270 2024-04-24T23:22:24.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:24.949+0000 7fb7c1000640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fb7bc06bbd0 con 0x7fb7bc150990 2024-04-24T23:22:24.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:24.950+0000 7fb7b9e00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb7bc073270 0x7fb7bc159980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:22:24.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:24.950+0000 7fb7b9e00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb7bc073270 0x7fb7bc159980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:57556/0 (socket says 172.21.15.6:57556) 2024-04-24T23:22:24.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:24.950+0000 7fb7b9e00640 1 -- 172.21.15.6:0/1601580329 learned_addr learned my addr 172.21.15.6:0/1601580329 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:22:24.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:24.950+0000 7fb7ba800640 1 --2- 172.21.15.6:0/1601580329 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb7bc157470 0x7fb7bc1e2640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:22:24.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:24.950+0000 7fb7b9400640 1 --2- 172.21.15.6:0/1601580329 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb7bc150990 0x7fb7bc200d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:22:24.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:24.950+0000 7fb7b9400640 1 -- 172.21.15.6:0/1601580329 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb7bc073270 msgr2=0x7fb7bc159980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:22:24.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:24.950+0000 7fb7b9400640 1 --2- 172.21.15.6:0/1601580329 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb7bc073270 0x7fb7bc159980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:22:24.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:24.950+0000 7fb7b9400640 1 -- 172.21.15.6:0/1601580329 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb7bc157470 msgr2=0x7fb7bc1e2640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:22:24.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:24.951+0000 7fb7b9400640 1 --2- 172.21.15.6:0/1601580329 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb7bc157470 0x7fb7bc1e2640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:22:24.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:24.951+0000 7fb7b9400640 1 -- 172.21.15.6:0/1601580329 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7bc1fcf40 con 0x7fb7bc150990 2024-04-24T23:22:24.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:24.951+0000 7fb7ba800640 1 --2- 172.21.15.6:0/1601580329 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb7bc157470 0x7fb7bc1e2640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:22:24.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:24.951+0000 7fb7b9400640 1 --2- 172.21.15.6:0/1601580329 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb7bc150990 0x7fb7bc200d10 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fb7bc06f400 tx=0x7fb7ac0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:22:24.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:24.951+0000 7fb7b2000640 1 -- 172.21.15.6:0/1601580329 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7ac037040 con 0x7fb7bc150990 2024-04-24T23:22:24.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:24.951+0000 7fb7b2000640 1 -- 172.21.15.6:0/1601580329 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb7ac042070 con 0x7fb7bc150990 2024-04-24T23:22:24.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:24.951+0000 7fb7b2000640 1 -- 172.21.15.6:0/1601580329 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7ac03ba50 con 0x7fb7bc150990 2024-04-24T23:22:24.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:24.952+0000 7fb7c1000640 1 -- 172.21.15.6:0/1601580329 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7bc1fd1e0 con 0x7fb7bc150990 2024-04-24T23:22:24.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:24.952+0000 7fb7c1000640 1 -- 172.21.15.6:0/1601580329 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7bc1fd780 con 0x7fb7bc150990 2024-04-24T23:22:24.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:24.957+0000 7fb7c1000640 1 -- 172.21.15.6:0/1601580329 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb7bc0767b0 con 0x7fb7bc150990 2024-04-24T23:22:24.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:24.958+0000 7fb7b2000640 1 -- 172.21.15.6:0/1601580329 <== mon.2 v2:172.21.15.190:3301/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fb7ac03a070 con 0x7fb7bc150990 2024-04-24T23:22:24.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:24.960+0000 7fb7b2000640 1 --2- 172.21.15.6:0/1601580329 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fb79c06bf30 0x7fb79c06e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:22:24.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:24.960+0000 7fb7b2000640 1 -- 172.21.15.6:0/1601580329 <== mon.2 v2:172.21.15.190:3301/0 5 ==== osd_map(28..28 src has 1..28) ==== 6762+0+0 (secure 0 0 0) 0x7fb7ac0ab1a0 con 0x7fb7bc150990 2024-04-24T23:22:24.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:24.964+0000 7fb7b9e00640 1 --2- 172.21.15.6:0/1601580329 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fb79c06bf30 0x7fb79c06e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:22:24.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:24.965+0000 7fb7b9e00640 1 --2- 172.21.15.6:0/1601580329 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fb79c06bf30 0x7fb79c06e3f0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fb7a8007d50 tx=0x7fb7a8007ca0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:22:24.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:24.965+0000 7fb7b2000640 1 -- 172.21.15.6:0/1601580329 <== mon.2 v2:172.21.15.190:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fb7ac079150 con 0x7fb7bc150990 2024-04-24T23:22:25.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:25.495+0000 7fb7c1000640 1 -- 172.21.15.6:0/1601580329 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb7bc0c75d0 con 0x7fb7bc150990 2024-04-24T23:22:25.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:25.496+0000 7fb7b2000640 1 -- 172.21.15.6:0/1601580329 <== mon.2 v2:172.21.15.190:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+6904 (secure 0 0 0) 0x7fb7ac07d450 con 0x7fb7bc150990 2024-04-24T23:22:25.498 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:22:25.498 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5081,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/1190759626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1190759626},{"type":"v1","addr":"172.21.15.6:6834","nonce":1190759626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:22:07.460817+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4282,"mds_2":4235,"mds_3":4232,"mds_4":5257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4232":{"gid":4232,"name":"b","rank":3,"incarnation":15,"state":"up:active","state_seq":15,"addr":"172.21.15.190:6838/2032280158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2032280158},{"type":"v1","addr":"172.21.15.190:6838","nonce":2032280158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4235":{"gid":4235,"name":"d","rank":2,"incarnation":13,"state":"up:active","state_seq":14,"addr":"172.21.15.190:6839/1572872451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":1572872451},{"type":"v1","addr":"172.21.15.190:6839","nonce":1572872451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4282":{"gid":4282,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":12,"addr":"172.21.15.190:6837/580315269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":580315269},{"type":"v1","addr":"172.21.15.190:6837","nonce":580315269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/489092393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":489092393},{"type":"v1","addr":"172.21.15.6:6835","nonce":489092393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5257":{"gid":5257,"name":"e","rank":4,"incarnation":25,"state":"up:stopping","state_seq":43,"addr":"172.21.15.6:6837/3240971655","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":3240971655},{"type":"v1","addr":"172.21.15.6:6837","nonce":3240971655}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4306,"qdb_cluster":[4232,4235,4282,4306]},"id":1}]} 2024-04-24T23:22:25.498 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 32 2024-04-24T23:22:25.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:25.500+0000 7fb7c1000640 1 -- 172.21.15.6:0/1601580329 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fb79c06bf30 msgr2=0x7fb79c06e3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:22:25.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:25.500+0000 7fb7c1000640 1 --2- 172.21.15.6:0/1601580329 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fb79c06bf30 0x7fb79c06e3f0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fb7a8007d50 tx=0x7fb7a8007ca0 comp rx=0 tx=0).stop 2024-04-24T23:22:25.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:25.500+0000 7fb7c1000640 1 -- 172.21.15.6:0/1601580329 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb7bc150990 msgr2=0x7fb7bc200d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:22:25.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:25.500+0000 7fb7c1000640 1 --2- 172.21.15.6:0/1601580329 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb7bc150990 0x7fb7bc200d10 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fb7bc06f400 tx=0x7fb7ac0046c0 comp rx=0 tx=0).stop 2024-04-24T23:22:25.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:25.500+0000 7fb7c1000640 1 -- 172.21.15.6:0/1601580329 shutdown_connections 2024-04-24T23:22:25.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:25.500+0000 7fb7c1000640 1 --2- 172.21.15.6:0/1601580329 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fb79c06bf30 0x7fb79c06e3f0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:22:25.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:25.500+0000 7fb7c1000640 1 --2- 172.21.15.6:0/1601580329 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb7bc150990 0x7fb7bc200d10 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:22:25.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:25.500+0000 7fb7c1000640 1 --2- 172.21.15.6:0/1601580329 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb7bc157470 0x7fb7bc1e2640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:22:25.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:25.500+0000 7fb7c1000640 1 --2- 172.21.15.6:0/1601580329 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb7bc073270 0x7fb7bc159980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:22:25.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:25.500+0000 7fb7c1000640 1 -- 172.21.15.6:0/1601580329 >> 172.21.15.6:0/1601580329 conn(0x7fb7bc06b0e0 msgr2=0x7fb7bc06e170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:22:25.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:25.501+0000 7fb7c1000640 1 -- 172.21.15.6:0/1601580329 shutdown_connections 2024-04-24T23:22:25.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:25.501+0000 7fb7c1000640 1 -- 172.21.15.6:0/1601580329 wait complete. 2024-04-24T23:22:25.533 INFO:tasks.mds_thrash.fs.[cephfs]:cluster is considered unstable while MDS are in up:stopping (!thrash_while_stopping) 2024-04-24T23:22:25.534 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 32, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-24T23:15:36.400716+0000", "damaged": [], "data_pools": [ 3, 4 ], "enabled": true, "epoch": 32, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4232": { "addr": "172.21.15.190:6838/2032280158", "addrs": { "addrvec": [ { "addr": "172.21.15.190:6835", "nonce": 2032280158, "type": "v2" }, { "addr": "172.21.15.190:6838", "nonce": 2032280158, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4232, "incarnation": 15, "join_fscid": -1, "name": "b", "rank": 3, "state": "up:active", "state_seq": 15 }, "gid_4235": { "addr": "172.21.15.190:6839/1572872451", "addrs": { "addrvec": [ { "addr": "172.21.15.190:6836", "nonce": 1572872451, "type": "v2" }, { "addr": "172.21.15.190:6839", "nonce": 1572872451, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4235, "incarnation": 13, "join_fscid": -1, "name": "d", "rank": 2, "state": "up:active", "state_seq": 14 }, "gid_4282": { "addr": "172.21.15.190:6837/580315269", "addrs": { "addrvec": [ { "addr": "172.21.15.190:6834", "nonce": 580315269, "type": "v2" }, { "addr": "172.21.15.190:6837", "nonce": 580315269, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4282, "incarnation": 9, "join_fscid": -1, "name": "f", "rank": 1, "state": "up:active", "state_seq": 12 }, "gid_4306": { "addr": "172.21.15.6:6835/489092393", "addrs": { "addrvec": [ { "addr": "172.21.15.6:6833", "nonce": 489092393, "type": "v2" }, { "addr": "172.21.15.6:6835", "nonce": 489092393, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4306, "incarnation": 4, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 4 }, "gid_5257": { "addr": "172.21.15.6:6837/3240971655", "addrs": { "addrvec": [ { "addr": "172.21.15.6:6836", "nonce": 3240971655, "type": "v2" }, { "addr": "172.21.15.6:6837", "nonce": 3240971655, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5257, "incarnation": 25, "join_fscid": -1, "name": "e", "rank": 4, "state": "up:stopping", "state_seq": 43 } }, "last_failure": 0, "last_failure_osd_epoch": 28, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-04-24T23:22:07.460817+0000", "qdb_cluster": [ 4232, 4235, 4282, 4306 ], "qdb_leader": 4306, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4306, "mds_1": 4282, "mds_2": 4235, "mds_3": 4232, "mds_4": 5257 } } } ], "standbys": [ { "addr": "172.21.15.6:6834/1190759626", "addrs": { "addrvec": [ { "addr": "172.21.15.6:6832", "nonce": 1190759626, "type": "v2" }, { "addr": "172.21.15.6:6834", "nonce": 1190759626, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 30, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5081, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-04-24T23:22:27.535 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:22:30.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:30.049+0000 7f1334800640 1 Processor -- start 2024-04-24T23:22:30.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:30.049+0000 7f1334800640 1 -- start start 2024-04-24T23:22:30.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:30.051+0000 7f1334800640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f132c152bc0 0x7f132c159390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:22:30.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:30.051+0000 7f1334800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f132c0c8f50 0x7f132c15bf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:22:30.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:30.051+0000 7f1334800640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f132c0c6e70 con 0x7f132c0c9330 2024-04-24T23:22:30.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:30.051+0000 7f1334800640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f132c0c73d0 con 0x7f132c0c8f50 2024-04-24T23:22:30.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:30.051+0000 7f1334800640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f132c0c8ac0 con 0x7f132c152bc0 2024-04-24T23:22:30.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:30.054+0000 7f1331e00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f132c152bc0 0x7f132c159390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:22:30.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:30.054+0000 7f1331e00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f132c152bc0 0x7f132c159390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.6:49200/0 (socket says 172.21.15.6:49200) 2024-04-24T23:22:30.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:30.054+0000 7f1331e00640 1 -- 172.21.15.6:0/1124125768 learned_addr learned my addr 172.21.15.6:0/1124125768 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:22:30.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:30.054+0000 7f1331e00640 1 -- 172.21.15.6:0/1124125768 >> v2:172.21.15.6:3300/0 conn(0x7f132c0c8f50 msgr2=0x7f132c15bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:22:30.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:30.054+0000 7f1331e00640 1 --2- 172.21.15.6:0/1124125768 >> v2:172.21.15.6:3300/0 conn(0x7f132c0c8f50 0x7f132c15bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:22:30.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:30.054+0000 7f1331e00640 1 -- 172.21.15.6:0/1124125768 >> v1:172.21.15.6:6789/0 conn(0x7f132c0c9330 legacy=0x7f132c15c440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:22:30.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:30.054+0000 7f1331e00640 1 -- 172.21.15.6:0/1124125768 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f132c1598d0 con 0x7f132c152bc0 2024-04-24T23:22:30.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:30.056+0000 7f1331e00640 1 --2- 172.21.15.6:0/1124125768 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f132c152bc0 0x7f132c159390 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f132c06fbb0 tx=0x7f132002a160 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e584417005baf7f0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:22:30.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:30.058+0000 7f1330a00640 1 -- 172.21.15.6:0/1124125768 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1320037070 con 0x7f132c152bc0 2024-04-24T23:22:30.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:30.058+0000 7f1330a00640 1 -- 172.21.15.6:0/1124125768 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1320032440 con 0x7f132c152bc0 2024-04-24T23:22:30.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:30.058+0000 7f1330a00640 1 -- 172.21.15.6:0/1124125768 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1320032810 con 0x7f132c152bc0 2024-04-24T23:22:30.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:30.059+0000 7f1334800640 1 -- 172.21.15.6:0/1124125768 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f132c152bc0 msgr2=0x7f132c159390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:22:30.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:30.059+0000 7f1334800640 1 --2- 172.21.15.6:0/1124125768 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f132c152bc0 0x7f132c159390 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f132c06fbb0 tx=0x7f132002a160 comp rx=0 tx=0).stop 2024-04-24T23:22:30.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:30.059+0000 7f1334800640 1 -- 172.21.15.6:0/1124125768 shutdown_connections 2024-04-24T23:22:30.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:30.059+0000 7f1334800640 1 --2- 172.21.15.6:0/1124125768 >> v2:172.21.15.6:3300/0 conn(0x7f132c0c8f50 0x7f132c15bf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:22:30.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:30.059+0000 7f1334800640 1 --2- 172.21.15.6:0/1124125768 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f132c152bc0 0x7f132c159390 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:22:30.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:30.059+0000 7f1334800640 1 -- 172.21.15.6:0/1124125768 >> 172.21.15.6:0/1124125768 conn(0x7f132c06b0e0 msgr2=0x7f132c06b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:22:30.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:30.059+0000 7f1334800640 1 -- 172.21.15.6:0/1124125768 shutdown_connections 2024-04-24T23:22:30.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:30.059+0000 7f1334800640 1 -- 172.21.15.6:0/1124125768 wait complete. 2024-04-24T23:22:30.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:30.062+0000 7f1334800640 1 Processor -- start 2024-04-24T23:22:30.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:30.062+0000 7f1334800640 1 -- start start 2024-04-24T23:22:30.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:30.063+0000 7f1334800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f132c0c8f50 0x7f132c15c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:22:30.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:30.063+0000 7f1334800640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f132c0c9330 0x7f132c0ca460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:22:30.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:30.063+0000 7f1334800640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f132c152bc0 0x7f132c1f8180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:22:30.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:30.064+0000 7f1334800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f132c0c8470 con 0x7f132c0c8f50 2024-04-24T23:22:30.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:30.064+0000 7f1334800640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f132c0c6700 con 0x7f132c0c9330 2024-04-24T23:22:30.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:30.064+0000 7f1334800640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f132c0c8750 con 0x7f132c152bc0 2024-04-24T23:22:30.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:30.065+0000 7f1332800640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f132c152bc0 0x7f132c1f8180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:22:30.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:30.065+0000 7f1332800640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f132c152bc0 0x7f132c1f8180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.6:49202/0 (socket says 172.21.15.6:49202) 2024-04-24T23:22:30.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:30.065+0000 7f1332800640 1 -- 172.21.15.6:0/3877755524 learned_addr learned my addr 172.21.15.6:0/3877755524 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:22:30.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:30.065+0000 7f1331400640 1 --2- 172.21.15.6:0/3877755524 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f132c0c9330 0x7f132c0ca460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:22:30.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:30.065+0000 7f1331e00640 1 --2- 172.21.15.6:0/3877755524 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f132c0c8f50 0x7f132c15c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:22:30.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:30.065+0000 7f1332800640 1 -- 172.21.15.6:0/3877755524 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f132c0c9330 msgr2=0x7f132c0ca460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:22:30.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:30.065+0000 7f1332800640 1 --2- 172.21.15.6:0/3877755524 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f132c0c9330 0x7f132c0ca460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:22:30.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:30.065+0000 7f1332800640 1 -- 172.21.15.6:0/3877755524 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f132c0c8f50 msgr2=0x7f132c15c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:22:30.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:30.065+0000 7f1332800640 1 --2- 172.21.15.6:0/3877755524 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f132c0c8f50 0x7f132c15c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:22:30.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:30.065+0000 7f1332800640 1 -- 172.21.15.6:0/3877755524 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f132c0c30b0 con 0x7f132c152bc0 2024-04-24T23:22:30.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:30.067+0000 7f1332800640 1 --2- 172.21.15.6:0/3877755524 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f132c152bc0 0x7f132c1f8180 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f132800c7b0 tx=0x7f132800cbb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:22:30.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:30.067+0000 7f1326000640 1 -- 172.21.15.6:0/3877755524 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1328017070 con 0x7f132c152bc0 2024-04-24T23:22:30.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:30.067+0000 7f1334800640 1 -- 172.21.15.6:0/3877755524 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f132c159b70 con 0x7f132c152bc0 2024-04-24T23:22:30.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:30.067+0000 7f1334800640 1 -- 172.21.15.6:0/3877755524 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f132c15a130 con 0x7f132c152bc0 2024-04-24T23:22:30.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:30.067+0000 7f1326000640 1 -- 172.21.15.6:0/3877755524 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1328003b80 con 0x7f132c152bc0 2024-04-24T23:22:30.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:30.067+0000 7f1326000640 1 -- 172.21.15.6:0/3877755524 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1328005070 con 0x7f132c152bc0 2024-04-24T23:22:30.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:30.071+0000 7f1326000640 1 -- 172.21.15.6:0/3877755524 <== mon.2 v2:172.21.15.190:3301/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f132801a070 con 0x7f132c152bc0 2024-04-24T23:22:30.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:30.073+0000 7f1326000640 1 --2- 172.21.15.6:0/3877755524 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f131006bf00 0x7f131006e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:22:30.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:30.074+0000 7f1326000640 1 -- 172.21.15.6:0/3877755524 <== mon.2 v2:172.21.15.190:3301/0 5 ==== osd_map(28..28 src has 1..28) ==== 6762+0+0 (secure 0 0 0) 0x7f132808a800 con 0x7f132c152bc0 2024-04-24T23:22:30.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:30.074+0000 7f1331e00640 1 --2- 172.21.15.6:0/3877755524 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f131006bf00 0x7f131006e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:22:30.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:30.075+0000 7f1331e00640 1 --2- 172.21.15.6:0/3877755524 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f131006bf00 0x7f131006e3c0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f1320034000 tx=0x7f132002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:22:30.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:30.076+0000 7f1334800640 1 -- 172.21.15.6:0/3877755524 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1308005290 con 0x7f132c152bc0 2024-04-24T23:22:30.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:30.082+0000 7f1326000640 1 -- 172.21.15.6:0/3877755524 <== mon.2 v2:172.21.15.190:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f1328058720 con 0x7f132c152bc0 2024-04-24T23:22:30.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:30.528+0000 7f1334800640 1 -- 172.21.15.6:0/3877755524 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1308005580 con 0x7f132c152bc0 2024-04-24T23:22:30.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:30.529+0000 7f1326000640 1 -- 172.21.15.6:0/3877755524 <== mon.2 v2:172.21.15.190:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+6904 (secure 0 0 0) 0x7f132805ca20 con 0x7f132c152bc0 2024-04-24T23:22:30.531 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:22:30.532 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5081,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/1190759626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1190759626},{"type":"v1","addr":"172.21.15.6:6834","nonce":1190759626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:22:07.460817+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4282,"mds_2":4235,"mds_3":4232,"mds_4":5257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4232":{"gid":4232,"name":"b","rank":3,"incarnation":15,"state":"up:active","state_seq":15,"addr":"172.21.15.190:6838/2032280158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2032280158},{"type":"v1","addr":"172.21.15.190:6838","nonce":2032280158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4235":{"gid":4235,"name":"d","rank":2,"incarnation":13,"state":"up:active","state_seq":14,"addr":"172.21.15.190:6839/1572872451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":1572872451},{"type":"v1","addr":"172.21.15.190:6839","nonce":1572872451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4282":{"gid":4282,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":12,"addr":"172.21.15.190:6837/580315269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":580315269},{"type":"v1","addr":"172.21.15.190:6837","nonce":580315269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/489092393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":489092393},{"type":"v1","addr":"172.21.15.6:6835","nonce":489092393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5257":{"gid":5257,"name":"e","rank":4,"incarnation":25,"state":"up:stopping","state_seq":43,"addr":"172.21.15.6:6837/3240971655","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":3240971655},{"type":"v1","addr":"172.21.15.6:6837","nonce":3240971655}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4306,"qdb_cluster":[4232,4235,4282,4306]},"id":1}]} 2024-04-24T23:22:30.532 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 32 2024-04-24T23:22:30.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:30.533+0000 7f1334800640 1 -- 172.21.15.6:0/3877755524 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f131006bf00 msgr2=0x7f131006e3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:22:30.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:30.533+0000 7f1334800640 1 --2- 172.21.15.6:0/3877755524 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f131006bf00 0x7f131006e3c0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f1320034000 tx=0x7f132002e000 comp rx=0 tx=0).stop 2024-04-24T23:22:30.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:30.533+0000 7f1334800640 1 -- 172.21.15.6:0/3877755524 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f132c152bc0 msgr2=0x7f132c1f8180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:22:30.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:30.533+0000 7f1334800640 1 --2- 172.21.15.6:0/3877755524 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f132c152bc0 0x7f132c1f8180 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f132800c7b0 tx=0x7f132800cbb0 comp rx=0 tx=0).stop 2024-04-24T23:22:30.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:30.535+0000 7f1334800640 1 -- 172.21.15.6:0/3877755524 shutdown_connections 2024-04-24T23:22:30.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:30.535+0000 7f1334800640 1 --2- 172.21.15.6:0/3877755524 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f131006bf00 0x7f131006e3c0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:22:30.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:30.535+0000 7f1334800640 1 --2- 172.21.15.6:0/3877755524 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f132c152bc0 0x7f132c1f8180 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:22:30.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:30.535+0000 7f1334800640 1 --2- 172.21.15.6:0/3877755524 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f132c0c9330 0x7f132c0ca460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:22:30.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:30.535+0000 7f1334800640 1 --2- 172.21.15.6:0/3877755524 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f132c0c8f50 0x7f132c15c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:22:30.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:30.535+0000 7f1334800640 1 -- 172.21.15.6:0/3877755524 >> 172.21.15.6:0/3877755524 conn(0x7f132c06b0e0 msgr2=0x7f132c0cb490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:22:30.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:30.538+0000 7f1334800640 1 -- 172.21.15.6:0/3877755524 shutdown_connections 2024-04-24T23:22:30.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:30.539+0000 7f1334800640 1 -- 172.21.15.6:0/3877755524 wait complete. 2024-04-24T23:22:30.573 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-24T23:22:32.575 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:22:34.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:34.189+0000 7f509de00640 1 Processor -- start 2024-04-24T23:22:34.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:34.190+0000 7f509de00640 1 -- start start 2024-04-24T23:22:34.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:34.192+0000 7f509de00640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f5098154e60 0x7f50980c9110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:22:34.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:34.192+0000 7f509de00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5098155240 con 0x7f50980ca8c0 2024-04-24T23:22:34.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:34.192+0000 7f509de00640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f50980c7cc0 con 0x7f50980cad30 2024-04-24T23:22:34.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:34.192+0000 7f509de00640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f50980c9650 con 0x7f5098154e60 2024-04-24T23:22:34.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:34.194+0000 7f5096a00640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f5098154e60 0x7f50980c9110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:22:34.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:34.194+0000 7f5096a00640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f5098154e60 0x7f50980c9110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:57578/0 (socket says 172.21.15.6:57578) 2024-04-24T23:22:34.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:34.194+0000 7f5096a00640 1 -- 172.21.15.6:0/2519035155 learned_addr learned my addr 172.21.15.6:0/2519035155 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:22:34.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:34.196+0000 7f5095600640 1 -- 172.21.15.6:0/2519035155 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1097628575 0 0) 0x7f50980c7cc0 con 0x7f50980cad30 2024-04-24T23:22:34.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:34.197+0000 7f5095600640 1 -- 172.21.15.6:0/2519035155 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f507c003520 con 0x7f50980cad30 2024-04-24T23:22:34.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:34.197+0000 7f5096a00640 1 -- 172.21.15.6:0/2519035155 >> v1:172.21.15.190:6789/0 conn(0x7f50980cad30 legacy=0x7f509815ad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:22:34.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:34.197+0000 7f5096a00640 1 -- 172.21.15.6:0/2519035155 >> v1:172.21.15.6:6789/0 conn(0x7f50980ca8c0 legacy=0x7f50980cb370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:22:34.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:34.197+0000 7f5096a00640 1 -- 172.21.15.6:0/2519035155 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50980c9890 con 0x7f5098154e60 2024-04-24T23:22:34.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:34.202+0000 7f5096a00640 1 --2- 172.21.15.6:0/2519035155 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5098154e60 0x7f50980c9110 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f5088009870 tx=0x7f508802a2f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8c5400d58cb7d9b3 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:22:34.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:34.202+0000 7f5095600640 1 -- 172.21.15.6:0/2519035155 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5088037070 con 0x7f5098154e60 2024-04-24T23:22:34.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:34.203+0000 7f5095600640 1 -- 172.21.15.6:0/2519035155 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5088032640 con 0x7f5098154e60 2024-04-24T23:22:34.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:34.203+0000 7f5095600640 1 -- 172.21.15.6:0/2519035155 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5088032a10 con 0x7f5098154e60 2024-04-24T23:22:34.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:34.204+0000 7f509de00640 1 -- 172.21.15.6:0/2519035155 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5098154e60 msgr2=0x7f50980c9110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:22:34.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:34.204+0000 7f509de00640 1 --2- 172.21.15.6:0/2519035155 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5098154e60 0x7f50980c9110 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f5088009870 tx=0x7f508802a2f0 comp rx=0 tx=0).stop 2024-04-24T23:22:34.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:34.205+0000 7f509de00640 1 -- 172.21.15.6:0/2519035155 shutdown_connections 2024-04-24T23:22:34.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:34.205+0000 7f509de00640 1 --2- 172.21.15.6:0/2519035155 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5098154e60 0x7f50980c9110 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:22:34.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:34.205+0000 7f509de00640 1 -- 172.21.15.6:0/2519035155 >> 172.21.15.6:0/2519035155 conn(0x7f50980cc9a0 msgr2=0x7f50980ccd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:22:34.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:34.209+0000 7f509de00640 1 -- 172.21.15.6:0/2519035155 shutdown_connections 2024-04-24T23:22:34.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:34.209+0000 7f509de00640 1 -- 172.21.15.6:0/2519035155 wait complete. 2024-04-24T23:22:34.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:34.216+0000 7f509de00640 1 Processor -- start 2024-04-24T23:22:34.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:34.217+0000 7f509de00640 1 -- start start 2024-04-24T23:22:34.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:34.218+0000 7f509de00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f50980ca8c0 0x7f50981536f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:22:34.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:34.218+0000 7f509de00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5098154e60 0x7f50981568f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:22:34.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:34.218+0000 7f5096a00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f50980ca8c0 0x7f50981536f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:22:34.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:34.219+0000 7f5096000640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5098154e60 0x7f50981568f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:22:34.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:34.219+0000 7f5096000640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5098154e60 0x7f50981568f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.6:49206/0 (socket says 172.21.15.6:49206) 2024-04-24T23:22:34.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:34.219+0000 7f5096a00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f50980ca8c0 0x7f50981536f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:57586/0 (socket says 172.21.15.6:57586) 2024-04-24T23:22:34.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:34.219+0000 7f509de00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f50981583b0 0x7f5098156e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:22:34.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:34.220+0000 7f5097400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f50981583b0 0x7f5098156e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:22:34.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:34.220+0000 7f5096000640 1 -- 172.21.15.6:0/1819190521 learned_addr learned my addr 172.21.15.6:0/1819190521 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:22:34.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:34.220+0000 7f509de00640 1 -- 172.21.15.6:0/1819190521 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f50980c76c0 con 0x7f50981583b0 2024-04-24T23:22:34.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:34.220+0000 7f509de00640 1 -- 172.21.15.6:0/1819190521 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f50980c85d0 con 0x7f50980ca8c0 2024-04-24T23:22:34.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:34.220+0000 7f509de00640 1 -- 172.21.15.6:0/1819190521 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f50980c8a60 con 0x7f5098154e60 2024-04-24T23:22:34.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:34.221+0000 7f5096a00640 1 -- 172.21.15.6:0/1819190521 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5098154e60 msgr2=0x7f50981568f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:22:34.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:34.221+0000 7f5096a00640 1 --2- 172.21.15.6:0/1819190521 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5098154e60 0x7f50981568f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:22:34.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:34.221+0000 7f5096a00640 1 -- 172.21.15.6:0/1819190521 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f50981583b0 msgr2=0x7f5098156e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:22:34.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:34.221+0000 7f5096a00640 1 --2- 172.21.15.6:0/1819190521 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f50981583b0 0x7f5098156e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:22:34.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:34.221+0000 7f5096a00640 1 -- 172.21.15.6:0/1819190521 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f509814a610 con 0x7f50980ca8c0 2024-04-24T23:22:34.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:34.221+0000 7f5097400640 1 --2- 172.21.15.6:0/1819190521 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f50981583b0 0x7f5098156e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:22:34.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:34.222+0000 7f5096a00640 1 --2- 172.21.15.6:0/1819190521 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f50980ca8c0 0x7f50981536f0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f5098070420 tx=0x7f50880046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:22:34.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:34.223+0000 7f508f400640 1 -- 172.21.15.6:0/1819190521 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5088037040 con 0x7f50980ca8c0 2024-04-24T23:22:34.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:34.224+0000 7f509de00640 1 -- 172.21.15.6:0/1819190521 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5098155850 con 0x7f50980ca8c0 2024-04-24T23:22:34.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:34.224+0000 7f509de00640 1 -- 172.21.15.6:0/1819190521 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5098155e10 con 0x7f50980ca8c0 2024-04-24T23:22:34.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:34.224+0000 7f508f400640 1 -- 172.21.15.6:0/1819190521 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5088042070 con 0x7f50980ca8c0 2024-04-24T23:22:34.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:34.224+0000 7f508f400640 1 -- 172.21.15.6:0/1819190521 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5088030cd0 con 0x7f50980ca8c0 2024-04-24T23:22:34.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:34.226+0000 7f509de00640 1 -- 172.21.15.6:0/1819190521 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f509806e8e0 con 0x7f50980ca8c0 2024-04-24T23:22:34.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:34.230+0000 7f508f400640 1 -- 172.21.15.6:0/1819190521 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f5088064480 con 0x7f50980ca8c0 2024-04-24T23:22:34.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:34.234+0000 7f508f400640 1 --2- 172.21.15.6:0/1819190521 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f507806bf30 0x7f507806e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:22:34.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:34.234+0000 7f508f400640 1 -- 172.21.15.6:0/1819190521 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6762+0+0 (secure 0 0 0) 0x7f50880abf30 con 0x7f50980ca8c0 2024-04-24T23:22:34.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:34.235+0000 7f5096000640 1 --2- 172.21.15.6:0/1819190521 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f507806bf30 0x7f507806e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:22:34.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:34.236+0000 7f5096000640 1 --2- 172.21.15.6:0/1819190521 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f507806bf30 0x7f507806e3f0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f509814d9c0 tx=0x7f5084009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:22:34.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:34.239+0000 7f508f400640 1 -- 172.21.15.6:0/1819190521 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f509806e8e0 con 0x7f50980ca8c0 2024-04-24T23:22:34.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:34.729+0000 7f509de00640 1 -- 172.21.15.6:0/1819190521 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f50980767b0 con 0x7f50980ca8c0 2024-04-24T23:22:34.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:34.734+0000 7f508f400640 1 -- 172.21.15.6:0/1819190521 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+6904 (secure 0 0 0) 0x7f508807a530 con 0x7f50980ca8c0 2024-04-24T23:22:34.738 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:22:34.738 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5081,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/1190759626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1190759626},{"type":"v1","addr":"172.21.15.6:6834","nonce":1190759626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:22:07.460817+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4282,"mds_2":4235,"mds_3":4232,"mds_4":5257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4232":{"gid":4232,"name":"b","rank":3,"incarnation":15,"state":"up:active","state_seq":15,"addr":"172.21.15.190:6838/2032280158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2032280158},{"type":"v1","addr":"172.21.15.190:6838","nonce":2032280158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4235":{"gid":4235,"name":"d","rank":2,"incarnation":13,"state":"up:active","state_seq":14,"addr":"172.21.15.190:6839/1572872451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":1572872451},{"type":"v1","addr":"172.21.15.190:6839","nonce":1572872451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4282":{"gid":4282,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":12,"addr":"172.21.15.190:6837/580315269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":580315269},{"type":"v1","addr":"172.21.15.190:6837","nonce":580315269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/489092393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":489092393},{"type":"v1","addr":"172.21.15.6:6835","nonce":489092393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5257":{"gid":5257,"name":"e","rank":4,"incarnation":25,"state":"up:stopping","state_seq":43,"addr":"172.21.15.6:6837/3240971655","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":3240971655},{"type":"v1","addr":"172.21.15.6:6837","nonce":3240971655}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4306,"qdb_cluster":[4232,4235,4282,4306]},"id":1}]} 2024-04-24T23:22:34.738 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 32 2024-04-24T23:22:34.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:34.738+0000 7f509de00640 1 -- 172.21.15.6:0/1819190521 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f507806bf30 msgr2=0x7f507806e3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:22:34.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:34.738+0000 7f509de00640 1 --2- 172.21.15.6:0/1819190521 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f507806bf30 0x7f507806e3f0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f509814d9c0 tx=0x7f5084009040 comp rx=0 tx=0).stop 2024-04-24T23:22:34.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:34.738+0000 7f509de00640 1 -- 172.21.15.6:0/1819190521 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f50980ca8c0 msgr2=0x7f50981536f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:22:34.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:34.738+0000 7f509de00640 1 --2- 172.21.15.6:0/1819190521 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f50980ca8c0 0x7f50981536f0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f5098070420 tx=0x7f50880046c0 comp rx=0 tx=0).stop 2024-04-24T23:22:34.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:34.738+0000 7f509de00640 1 -- 172.21.15.6:0/1819190521 shutdown_connections 2024-04-24T23:22:34.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:34.738+0000 7f509de00640 1 --2- 172.21.15.6:0/1819190521 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f507806bf30 0x7f507806e3f0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:22:34.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:34.738+0000 7f509de00640 1 --2- 172.21.15.6:0/1819190521 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5098154e60 0x7f50981568f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:22:34.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:34.738+0000 7f509de00640 1 --2- 172.21.15.6:0/1819190521 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f50981583b0 0x7f5098156e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:22:34.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:34.738+0000 7f509de00640 1 --2- 172.21.15.6:0/1819190521 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f50980ca8c0 0x7f50981536f0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:22:34.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:34.738+0000 7f509de00640 1 -- 172.21.15.6:0/1819190521 >> 172.21.15.6:0/1819190521 conn(0x7f50980cc9a0 msgr2=0x7f50980ccd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:22:34.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:34.738+0000 7f509de00640 1 -- 172.21.15.6:0/1819190521 shutdown_connections 2024-04-24T23:22:34.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:34.738+0000 7f509de00640 1 -- 172.21.15.6:0/1819190521 wait complete. 2024-04-24T23:22:34.776 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-24T23:22:36.777 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:22:40.896 INFO:tasks.ceph.mds.e.smithi006.stdout:starting mds.e at 2024-04-24T23:22:45.438 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-24T23:22:45.447 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-24T23:22:46.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:46.872+0000 7f526b800640 1 Processor -- start 2024-04-24T23:22:46.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:46.873+0000 7f526b800640 1 -- start start 2024-04-24T23:22:46.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:46.878+0000 7f526b800640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f5264159110 0x7f5264072420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:22:46.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:46.879+0000 7f5268e00640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f5264159110 0x7f5264072420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:22:46.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:46.879+0000 7f5268e00640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f5264159110 0x7f5264072420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:39818/0 (socket says 172.21.15.6:39818) 2024-04-24T23:22:46.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:46.880+0000 7f5263e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f52640c7dc0 0x7f52641623a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:33288/0 (socket says 172.21.15.6:33288) 2024-04-24T23:22:46.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:46.881+0000 7f5268e00640 1 -- 172.21.15.6:0/2985112526 learned_addr learned my addr 172.21.15.6:0/2985112526 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:22:46.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:46.882+0000 7f526b800640 1 --2- 172.21.15.6:0/2985112526 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f52640c81c0 0x7f5264162ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:22:46.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:46.883+0000 7f5269800640 1 --2- 172.21.15.6:0/2985112526 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f52640c81c0 0x7f5264162ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:22:46.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:46.885+0000 7f526b800640 1 -- 172.21.15.6:0/2985112526 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f52640c7230 con 0x7f52640c7dc0 2024-04-24T23:22:46.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:46.885+0000 7f526b800640 1 -- 172.21.15.6:0/2985112526 --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f52641594f0 con 0x7f5264159110 2024-04-24T23:22:46.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:46.886+0000 7f5263400640 1 -- 172.21.15.6:0/2985112526 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2515804630 0 0) 0x7f52640c7230 con 0x7f52640c7dc0 2024-04-24T23:22:46.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:46.886+0000 7f526b800640 1 -- 172.21.15.6:0/2985112526 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f52640c74d0 con 0x7f52640c81c0 2024-04-24T23:22:46.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:46.887+0000 7f5263400640 1 -- 172.21.15.6:0/2985112526 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5248003520 con 0x7f52640c7dc0 2024-04-24T23:22:46.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:46.888+0000 7f5268e00640 1 -- 172.21.15.6:0/2985112526 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f52640c81c0 msgr2=0x7f5264162ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:22:46.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:46.889+0000 7f5263400640 1 -- 172.21.15.6:0/2985112526 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2641564545 0 0) 0x7f5248003520 con 0x7f52640c7dc0 2024-04-24T23:22:46.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:46.889+0000 7f5268e00640 1 --2- 172.21.15.6:0/2985112526 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f52640c81c0 0x7f5264162ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:22:46.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:46.890+0000 7f5268e00640 1 -- 172.21.15.6:0/2985112526 >> v1:172.21.15.6:6789/0 conn(0x7f52640c7dc0 legacy=0x7f52641623a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:22:46.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:46.890+0000 7f5269800640 1 --2- 172.21.15.6:0/2985112526 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f52640c81c0 0x7f5264162ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:22:46.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:46.890+0000 7f5268e00640 1 -- 172.21.15.6:0/2985112526 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f526406e440 con 0x7f5264159110 2024-04-24T23:22:46.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:46.895+0000 7f5268e00640 1 --2- 172.21.15.6:0/2985112526 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5264159110 0x7f5264072420 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f5254009870 tx=0x7f525402a290 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f6f8daeabbf95866 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:22:46.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:46.896+0000 7f5263400640 1 -- 172.21.15.6:0/2985112526 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5254037070 con 0x7f5264159110 2024-04-24T23:22:46.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:46.896+0000 7f5263400640 1 -- 172.21.15.6:0/2985112526 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f525402ad20 con 0x7f5264159110 2024-04-24T23:22:46.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:46.896+0000 7f5263400640 1 -- 172.21.15.6:0/2985112526 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5250001690 con 0x7f5264159110 2024-04-24T23:22:46.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:46.897+0000 7f526b800640 1 -- 172.21.15.6:0/2985112526 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5264159110 msgr2=0x7f5264072420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:22:46.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:46.897+0000 7f526b800640 1 --2- 172.21.15.6:0/2985112526 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5264159110 0x7f5264072420 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f5254009870 tx=0x7f525402a290 comp rx=0 tx=0).stop 2024-04-24T23:22:46.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:46.901+0000 7f526b800640 1 -- 172.21.15.6:0/2985112526 shutdown_connections 2024-04-24T23:22:46.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:46.901+0000 7f526b800640 1 --2- 172.21.15.6:0/2985112526 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f52640c81c0 0x7f5264162ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:22:46.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:46.901+0000 7f526b800640 1 --2- 172.21.15.6:0/2985112526 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5264159110 0x7f5264072420 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:22:46.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:46.901+0000 7f526b800640 1 -- 172.21.15.6:0/2985112526 >> 172.21.15.6:0/2985112526 conn(0x7f526406b0e0 msgr2=0x7f526406b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:22:46.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:46.903+0000 7f526b800640 1 -- 172.21.15.6:0/2985112526 shutdown_connections 2024-04-24T23:22:46.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:46.904+0000 7f526b800640 1 -- 172.21.15.6:0/2985112526 wait complete. 2024-04-24T23:22:46.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:46.910+0000 7f526b800640 1 Processor -- start 2024-04-24T23:22:46.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:46.910+0000 7f526b800640 1 -- start start 2024-04-24T23:22:46.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:46.917+0000 7f526b800640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f52640c7dc0 0x7f52640ca360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:22:46.920 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:46.918+0000 7f526b800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f52640c81c0 0x7f5264162a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:22:46.920 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:46.918+0000 7f526b800640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5264159110 0x7f52640c45e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:22:46.920 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:46.918+0000 7f526b800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f526406ece0 con 0x7f52640c81c0 2024-04-24T23:22:46.920 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:46.918+0000 7f526b800640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f526406ea00 con 0x7f52640c7dc0 2024-04-24T23:22:46.920 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:46.918+0000 7f526b800640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f5264072960 con 0x7f5264159110 2024-04-24T23:22:46.920 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:46.918+0000 7f5263e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f52640c81c0 0x7f5264162a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:22:46.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:46.918+0000 7f5263e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f52640c81c0 0x7f5264162a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:48302/0 (socket says 172.21.15.6:48302) 2024-04-24T23:22:46.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:46.918+0000 7f5263e00640 1 -- 172.21.15.6:0/1384866557 learned_addr learned my addr 172.21.15.6:0/1384866557 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:22:46.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:46.919+0000 7f5263e00640 1 -- 172.21.15.6:0/1384866557 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5264159110 msgr2=0x7f52640c45e0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-24T23:22:46.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:46.919+0000 7f5263e00640 1 --2- 172.21.15.6:0/1384866557 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5264159110 0x7f52640c45e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:22:46.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:46.919+0000 7f5263e00640 1 -- 172.21.15.6:0/1384866557 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f52640c7dc0 msgr2=0x7f52640ca360 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-24T23:22:46.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:46.919+0000 7f5263e00640 1 --2- 172.21.15.6:0/1384866557 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f52640c7dc0 0x7f52640ca360 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:22:46.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:46.919+0000 7f5263e00640 1 -- 172.21.15.6:0/1384866557 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52641f4d70 con 0x7f52640c81c0 2024-04-24T23:22:46.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:46.919+0000 7f5263e00640 1 --2- 172.21.15.6:0/1384866557 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f52640c81c0 0x7f5264162a70 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f525000c780 tx=0x7f525000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:22:46.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:46.922+0000 7f5260c00640 1 -- 172.21.15.6:0/1384866557 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5250018070 con 0x7f52640c81c0 2024-04-24T23:22:46.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:46.922+0000 7f5260c00640 1 -- 172.21.15.6:0/1384866557 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f52500072d0 con 0x7f52640c81c0 2024-04-24T23:22:46.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:46.922+0000 7f5260c00640 1 -- 172.21.15.6:0/1384866557 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5250018070 con 0x7f52640c81c0 2024-04-24T23:22:46.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:46.923+0000 7f526b800640 1 -- 172.21.15.6:0/1384866557 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52641f5050 con 0x7f52640c81c0 2024-04-24T23:22:46.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:46.923+0000 7f526b800640 1 -- 172.21.15.6:0/1384866557 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52641f5610 con 0x7f52640c81c0 2024-04-24T23:22:46.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:46.930+0000 7f5260c00640 1 -- 172.21.15.6:0/1384866557 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f5250007480 con 0x7f52640c81c0 2024-04-24T23:22:46.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:46.933+0000 7f5260c00640 1 --2- 172.21.15.6:0/1384866557 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f524406c060 0x7f524406e520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:22:46.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:46.934+0000 7f5260c00640 1 -- 172.21.15.6:0/1384866557 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6762+0+0 (secure 0 0 0) 0x7f525008a350 con 0x7f52640c81c0 2024-04-24T23:22:46.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:46.935+0000 7f5268e00640 1 --2- 172.21.15.6:0/1384866557 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f524406c060 0x7f524406e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:22:46.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:46.944+0000 7f5268e00640 1 --2- 172.21.15.6:0/1384866557 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f524406c060 0x7f524406e520 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f5254034000 tx=0x7f525402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:22:47.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:47.106+0000 7f526b800640 1 -- 172.21.15.6:0/1384866557 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f52640767b0 con 0x7f52640c81c0 2024-04-24T23:22:47.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:47.124+0000 7f5260c00640 1 -- 172.21.15.6:0/1384866557 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f52500588d0 con 0x7f52640c81c0 2024-04-24T23:22:47.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:47.657+0000 7f526b800640 1 -- 172.21.15.6:0/1384866557 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f52640c7970 con 0x7f52640c81c0 2024-04-24T23:22:47.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:47.666+0000 7f5260c00640 1 -- 172.21.15.6:0/1384866557 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+6885 (secure 0 0 0) 0x7f52500581d0 con 0x7f52640c81c0 2024-04-24T23:22:47.672 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:22:47.672 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5081,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/1190759626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1190759626},{"type":"v1","addr":"172.21.15.6:6834","nonce":1190759626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":5503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/3928731059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":3928731059},{"type":"v1","addr":"172.21.15.6:6837","nonce":3928731059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:22:37.858824+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1,2,3],"up":{"mds_0":4306,"mds_1":4282,"mds_2":4235,"mds_3":4232},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_4232":{"gid":4232,"name":"b","rank":3,"incarnation":15,"state":"up:stopping","state_seq":15,"addr":"172.21.15.190:6838/2032280158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2032280158},{"type":"v1","addr":"172.21.15.190:6838","nonce":2032280158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4235":{"gid":4235,"name":"d","rank":2,"incarnation":13,"state":"up:active","state_seq":14,"addr":"172.21.15.190:6839/1572872451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":1572872451},{"type":"v1","addr":"172.21.15.190:6839","nonce":1572872451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4282":{"gid":4282,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":12,"addr":"172.21.15.190:6837/580315269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":580315269},{"type":"v1","addr":"172.21.15.190:6837","nonce":580315269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/489092393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":489092393},{"type":"v1","addr":"172.21.15.6:6835","nonce":489092393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4306,"qdb_cluster":[4306,4282,4235]},"id":1}]} 2024-04-24T23:22:47.673 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 35 2024-04-24T23:22:47.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:47.672+0000 7f526b800640 1 -- 172.21.15.6:0/1384866557 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f524406c060 msgr2=0x7f524406e520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:22:47.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:47.672+0000 7f526b800640 1 --2- 172.21.15.6:0/1384866557 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f524406c060 0x7f524406e520 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f5254034000 tx=0x7f525402e000 comp rx=0 tx=0).stop 2024-04-24T23:22:47.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:47.672+0000 7f526b800640 1 -- 172.21.15.6:0/1384866557 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f52640c81c0 msgr2=0x7f5264162a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:22:47.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:47.672+0000 7f526b800640 1 --2- 172.21.15.6:0/1384866557 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f52640c81c0 0x7f5264162a70 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f525000c780 tx=0x7f525000cc40 comp rx=0 tx=0).stop 2024-04-24T23:22:47.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:47.673+0000 7f526b800640 1 -- 172.21.15.6:0/1384866557 shutdown_connections 2024-04-24T23:22:47.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:47.673+0000 7f526b800640 1 --2- 172.21.15.6:0/1384866557 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f524406c060 0x7f524406e520 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:22:47.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:47.673+0000 7f526b800640 1 --2- 172.21.15.6:0/1384866557 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5264159110 0x7f52640c45e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:22:47.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:47.673+0000 7f526b800640 1 --2- 172.21.15.6:0/1384866557 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f52640c81c0 0x7f5264162a70 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:22:47.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:47.673+0000 7f526b800640 1 --2- 172.21.15.6:0/1384866557 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f52640c7dc0 0x7f52640ca360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:22:47.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:47.673+0000 7f526b800640 1 -- 172.21.15.6:0/1384866557 >> 172.21.15.6:0/1384866557 conn(0x7f526406b0e0 msgr2=0x7f5264157790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:22:47.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:47.673+0000 7f526b800640 1 -- 172.21.15.6:0/1384866557 shutdown_connections 2024-04-24T23:22:47.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:47.673+0000 7f526b800640 1 -- 172.21.15.6:0/1384866557 wait complete. 2024-04-24T23:22:47.710 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-24T23:22:49.711 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:22:50.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:50.147+0000 7fa1e2a00640 1 Processor -- start 2024-04-24T23:22:50.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:50.147+0000 7fa1e2a00640 1 -- start start 2024-04-24T23:22:50.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:50.149+0000 7fa1e2a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa1dc0cb090 0x7fa1dc0d0820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:22:50.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:50.150+0000 7fa1e1600640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7fa1dc154d90 0x7fa1dc0cbe80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.6:52990/0 (socket says 172.21.15.6:52990) 2024-04-24T23:22:50.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:50.150+0000 7fa1e1600640 1 -- 172.21.15.6:0/2126177000 learned_addr learned my addr 172.21.15.6:0/2126177000 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:22:50.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:50.150+0000 7fa1e2a00640 1 -- 172.21.15.6:0/2126177000 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa1dc155170 con 0x7fa1dc0d0d60 2024-04-24T23:22:50.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:50.150+0000 7fa1e2a00640 1 -- 172.21.15.6:0/2126177000 --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa1dc0cc9a0 con 0x7fa1dc154d90 2024-04-24T23:22:50.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:50.150+0000 7fa1e0c00640 1 --2- 172.21.15.6:0/2126177000 >> v2:172.21.15.6:3300/0 conn(0x7fa1dc0cb090 0x7fa1dc0d0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:22:50.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:50.150+0000 7fa1e2a00640 1 -- 172.21.15.6:0/2126177000 --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7fa1dc15ad60 con 0x7fa1dc0cb090 2024-04-24T23:22:50.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:50.151+0000 7fa1dbe00640 1 -- 172.21.15.6:0/2126177000 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2430697402 0 0) 0x7fa1dc0cc9a0 con 0x7fa1dc154d90 2024-04-24T23:22:50.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:50.151+0000 7fa1dbe00640 1 -- 172.21.15.6:0/2126177000 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa1c4003520 con 0x7fa1dc154d90 2024-04-24T23:22:50.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:50.152+0000 7fa1dbe00640 1 -- 172.21.15.6:0/2126177000 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4142032968 0 0) 0x7fa1c4003520 con 0x7fa1dc154d90 2024-04-24T23:22:50.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:50.152+0000 7fa1dbe00640 1 -- 172.21.15.6:0/2126177000 >> v2:172.21.15.6:3300/0 conn(0x7fa1dc0cb090 msgr2=0x7fa1dc0d0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:22:50.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:50.152+0000 7fa1dbe00640 1 --2- 172.21.15.6:0/2126177000 >> v2:172.21.15.6:3300/0 conn(0x7fa1dc0cb090 0x7fa1dc0d0820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:22:50.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:50.152+0000 7fa1dbe00640 1 -- 172.21.15.6:0/2126177000 >> v1:172.21.15.6:6789/0 conn(0x7fa1dc0d0d60 legacy=0x7fa1dc15e000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:22:50.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:50.152+0000 7fa1dbe00640 1 -- 172.21.15.6:0/2126177000 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1dc15b040 con 0x7fa1dc154d90 2024-04-24T23:22:50.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:50.152+0000 7fa1dbe00640 1 -- 172.21.15.6:0/2126177000 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3463268701 0 0) 0x7fa1d00019f0 con 0x7fa1dc154d90 2024-04-24T23:22:50.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:50.152+0000 7fa1dbe00640 1 -- 172.21.15.6:0/2126177000 >> v1:172.21.15.190:6789/0 conn(0x7fa1dc154d90 legacy=0x7fa1dc0cbe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:22:50.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:50.152+0000 7fa1dbe00640 1 --2- 172.21.15.6:0/2126177000 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa1c40039b0 0x7fa1c4003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:22:50.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:50.152+0000 7fa1dbe00640 1 --2- 172.21.15.6:0/2126177000 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa1c4004800 0x7fa1c4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:22:50.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:50.152+0000 7fa1dbe00640 1 --2- 172.21.15.6:0/2126177000 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa1c4005210 0x7fa1c40141d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:22:50.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:50.152+0000 7fa1dbe00640 1 -- 172.21.15.6:0/2126177000 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fa1dc15ad60 con 0x7fa1c4004800 2024-04-24T23:22:50.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:50.152+0000 7fa1dbe00640 1 -- 172.21.15.6:0/2126177000 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fa1c40148d0 con 0x7fa1c4005210 2024-04-24T23:22:50.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:50.152+0000 7fa1dbe00640 1 -- 172.21.15.6:0/2126177000 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fa1c4014ad0 con 0x7fa1c40039b0 2024-04-24T23:22:50.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:50.153+0000 7fa1e2000640 1 --2- 172.21.15.6:0/2126177000 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa1c4005210 0x7fa1c40141d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:22:50.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:50.153+0000 7fa1e1600640 1 --2- 172.21.15.6:0/2126177000 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa1c40039b0 0x7fa1c4003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:22:50.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:50.153+0000 7fa1e2000640 1 -- 172.21.15.6:0/2126177000 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa1c40039b0 msgr2=0x7fa1c4003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:22:50.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:50.153+0000 7fa1e2000640 1 --2- 172.21.15.6:0/2126177000 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa1c40039b0 0x7fa1c4003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:22:50.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:50.153+0000 7fa1e2000640 1 -- 172.21.15.6:0/2126177000 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa1c4004800 msgr2=0x7fa1c4004c80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-24T23:22:50.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:50.153+0000 7fa1e2000640 1 --2- 172.21.15.6:0/2126177000 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa1c4004800 0x7fa1c4004c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:22:50.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:50.153+0000 7fa1e2000640 1 -- 172.21.15.6:0/2126177000 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1dc15b040 con 0x7fa1c4005210 2024-04-24T23:22:50.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:50.155+0000 7fa1e2000640 1 -- 172.21.15.6:0/2126177000 reap_dead start 2024-04-24T23:22:50.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:50.155+0000 7fa1e2000640 1 --2- 172.21.15.6:0/2126177000 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa1c4005210 0x7fa1c40141d0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fa1d4001830 tx=0x7fa1c4005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7534c0485fa66eed server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:22:50.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:50.156+0000 7fa1dbe00640 1 -- 172.21.15.6:0/2126177000 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1d4001430 con 0x7fa1c4005210 2024-04-24T23:22:50.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:50.156+0000 7fa1dbe00640 1 -- 172.21.15.6:0/2126177000 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa1d40052f0 con 0x7fa1c4005210 2024-04-24T23:22:50.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:50.157+0000 7fa1dbe00640 1 -- 172.21.15.6:0/2126177000 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1d402ac10 con 0x7fa1c4005210 2024-04-24T23:22:50.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:50.157+0000 7fa1e2a00640 1 -- 172.21.15.6:0/2126177000 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa1c4005210 msgr2=0x7fa1c40141d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:22:50.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:50.157+0000 7fa1e2a00640 1 --2- 172.21.15.6:0/2126177000 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa1c4005210 0x7fa1c40141d0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fa1d4001830 tx=0x7fa1c4005830 comp rx=0 tx=0).stop 2024-04-24T23:22:50.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:50.160+0000 7fa1e2a00640 1 -- 172.21.15.6:0/2126177000 shutdown_connections 2024-04-24T23:22:50.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:50.160+0000 7fa1e2a00640 1 --2- 172.21.15.6:0/2126177000 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa1c4005210 0x7fa1c40141d0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:22:50.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:50.160+0000 7fa1e2a00640 1 -- 172.21.15.6:0/2126177000 >> 172.21.15.6:0/2126177000 conn(0x7fa1dc06be10 msgr2=0x7fa1dc06c1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:22:50.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:50.161+0000 7fa1e2a00640 1 -- 172.21.15.6:0/2126177000 shutdown_connections 2024-04-24T23:22:50.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:50.161+0000 7fa1e2a00640 1 -- 172.21.15.6:0/2126177000 wait complete. 2024-04-24T23:22:50.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:50.164+0000 7fa1e2a00640 1 Processor -- start 2024-04-24T23:22:50.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:50.167+0000 7fa1e2a00640 1 -- start start 2024-04-24T23:22:50.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:50.170+0000 7fa1e2a00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa1c4005210 0x7fa1dc14e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:22:50.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:50.170+0000 7fa1e2a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa1dc1538b0 0x7fa1dc1ef270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:22:50.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:50.170+0000 7fa1e2a00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa1dc143380 0x7fa1dc1ef7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:22:50.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:50.170+0000 7fa1e1600640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa1c4005210 0x7fa1dc14e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:22:50.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:50.171+0000 7fa1e1600640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa1c4005210 0x7fa1dc14e020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.6:49914/0 (socket says 172.21.15.6:49914) 2024-04-24T23:22:50.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:50.171+0000 7fa1e0c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa1dc1538b0 0x7fa1dc1ef270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:22:50.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:50.171+0000 7fa1e1600640 1 -- 172.21.15.6:0/3605311745 learned_addr learned my addr 172.21.15.6:0/3605311745 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:22:50.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:50.171+0000 7fa1e2a00640 1 -- 172.21.15.6:0/3605311745 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fa1dc15aa30 con 0x7fa1dc1538b0 2024-04-24T23:22:50.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:50.171+0000 7fa1e2a00640 1 -- 172.21.15.6:0/3605311745 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fa1dc15b600 con 0x7fa1dc143380 2024-04-24T23:22:50.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:50.171+0000 7fa1e2a00640 1 -- 172.21.15.6:0/3605311745 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fa1dc0d0820 con 0x7fa1c4005210 2024-04-24T23:22:50.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:50.173+0000 7fa1e2000640 1 --2- 172.21.15.6:0/3605311745 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa1dc143380 0x7fa1dc1ef7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:22:50.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:50.174+0000 7fa1e1600640 1 -- 172.21.15.6:0/3605311745 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa1dc143380 msgr2=0x7fa1dc1ef7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:22:50.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:50.174+0000 7fa1e1600640 1 --2- 172.21.15.6:0/3605311745 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa1dc143380 0x7fa1dc1ef7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:22:50.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:50.174+0000 7fa1e1600640 1 -- 172.21.15.6:0/3605311745 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa1dc1538b0 msgr2=0x7fa1dc1ef270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:22:50.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:50.174+0000 7fa1e1600640 1 --2- 172.21.15.6:0/3605311745 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa1dc1538b0 0x7fa1dc1ef270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:22:50.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:50.174+0000 7fa1e1600640 1 -- 172.21.15.6:0/3605311745 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1dc154040 con 0x7fa1c4005210 2024-04-24T23:22:50.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:50.174+0000 7fa1e2000640 1 --2- 172.21.15.6:0/3605311745 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa1dc143380 0x7fa1dc1ef7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:22:50.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:50.174+0000 7fa1e0c00640 1 --2- 172.21.15.6:0/3605311745 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa1dc1538b0 0x7fa1dc1ef270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:22:50.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:50.175+0000 7fa1e1600640 1 --2- 172.21.15.6:0/3605311745 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa1c4005210 0x7fa1dc14e020 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fa1d0007a20 tx=0x7fa1d000a750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:22:50.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:50.177+0000 7fa1d9600640 1 -- 172.21.15.6:0/3605311745 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1d000ae00 con 0x7fa1c4005210 2024-04-24T23:22:50.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:50.177+0000 7fa1d9600640 1 -- 172.21.15.6:0/3605311745 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa1d0018070 con 0x7fa1c4005210 2024-04-24T23:22:50.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:50.177+0000 7fa1d9600640 1 -- 172.21.15.6:0/3605311745 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1d00127a0 con 0x7fa1c4005210 2024-04-24T23:22:50.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:50.177+0000 7fa1e2a00640 1 -- 172.21.15.6:0/3605311745 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1dc154320 con 0x7fa1c4005210 2024-04-24T23:22:50.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:50.177+0000 7fa1e2a00640 1 -- 172.21.15.6:0/3605311745 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1dc15a650 con 0x7fa1c4005210 2024-04-24T23:22:50.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:50.188+0000 7fa1e2a00640 1 -- 172.21.15.6:0/3605311745 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa1b8005290 con 0x7fa1c4005210 2024-04-24T23:22:50.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:50.189+0000 7fa1d9600640 1 -- 172.21.15.6:0/3605311745 <== mon.2 v2:172.21.15.190:3301/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fa1d0012940 con 0x7fa1c4005210 2024-04-24T23:22:50.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:50.192+0000 7fa1d9600640 1 --2- 172.21.15.6:0/3605311745 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fa1bc06bf30 0x7fa1bc06e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:22:50.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:50.192+0000 7fa1d9600640 1 -- 172.21.15.6:0/3605311745 <== mon.2 v2:172.21.15.190:3301/0 5 ==== osd_map(28..28 src has 1..28) ==== 6762+0+0 (secure 0 0 0) 0x7fa1d008bc00 con 0x7fa1c4005210 2024-04-24T23:22:50.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:50.195+0000 7fa1e0c00640 1 --2- 172.21.15.6:0/3605311745 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fa1bc06bf30 0x7fa1bc06e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:22:50.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:50.196+0000 7fa1e0c00640 1 --2- 172.21.15.6:0/3605311745 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fa1bc06bf30 0x7fa1bc06e3f0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fa1dc06c540 tx=0x7fa1c8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:22:50.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:50.199+0000 7fa1d9600640 1 -- 172.21.15.6:0/3605311745 <== mon.2 v2:172.21.15.190:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fa1d005a0e0 con 0x7fa1c4005210 2024-04-24T23:22:50.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:50.783+0000 7fa1e2a00640 1 -- 172.21.15.6:0/3605311745 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa1b8005580 con 0x7fa1c4005210 2024-04-24T23:22:50.787 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:22:50.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:50.785+0000 7fa1d9600640 1 -- 172.21.15.6:0/3605311745 <== mon.2 v2:172.21.15.190:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+6885 (secure 0 0 0) 0x7fa1d00599e0 con 0x7fa1c4005210 2024-04-24T23:22:50.788 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5081,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/1190759626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1190759626},{"type":"v1","addr":"172.21.15.6:6834","nonce":1190759626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":5503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/3928731059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":3928731059},{"type":"v1","addr":"172.21.15.6:6837","nonce":3928731059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:22:37.858824+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1,2,3],"up":{"mds_0":4306,"mds_1":4282,"mds_2":4235,"mds_3":4232},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_4232":{"gid":4232,"name":"b","rank":3,"incarnation":15,"state":"up:stopping","state_seq":15,"addr":"172.21.15.190:6838/2032280158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2032280158},{"type":"v1","addr":"172.21.15.190:6838","nonce":2032280158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4235":{"gid":4235,"name":"d","rank":2,"incarnation":13,"state":"up:active","state_seq":14,"addr":"172.21.15.190:6839/1572872451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":1572872451},{"type":"v1","addr":"172.21.15.190:6839","nonce":1572872451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4282":{"gid":4282,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":12,"addr":"172.21.15.190:6837/580315269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":580315269},{"type":"v1","addr":"172.21.15.190:6837","nonce":580315269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/489092393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":489092393},{"type":"v1","addr":"172.21.15.6:6835","nonce":489092393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4306,"qdb_cluster":[4306,4282,4235]},"id":1}]} 2024-04-24T23:22:50.791 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 35 2024-04-24T23:22:50.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:50.791+0000 7fa1e2a00640 1 -- 172.21.15.6:0/3605311745 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fa1bc06bf30 msgr2=0x7fa1bc06e3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:22:50.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:50.791+0000 7fa1e2a00640 1 --2- 172.21.15.6:0/3605311745 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fa1bc06bf30 0x7fa1bc06e3f0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fa1dc06c540 tx=0x7fa1c8008040 comp rx=0 tx=0).stop 2024-04-24T23:22:50.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:50.792+0000 7fa1e2a00640 1 -- 172.21.15.6:0/3605311745 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa1c4005210 msgr2=0x7fa1dc14e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:22:50.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:50.792+0000 7fa1e2a00640 1 --2- 172.21.15.6:0/3605311745 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa1c4005210 0x7fa1dc14e020 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fa1d0007a20 tx=0x7fa1d000a750 comp rx=0 tx=0).stop 2024-04-24T23:22:50.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:50.792+0000 7fa1e2a00640 1 -- 172.21.15.6:0/3605311745 shutdown_connections 2024-04-24T23:22:50.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:50.792+0000 7fa1e2a00640 1 --2- 172.21.15.6:0/3605311745 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fa1bc06bf30 0x7fa1bc06e3f0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:22:50.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:50.792+0000 7fa1e2a00640 1 --2- 172.21.15.6:0/3605311745 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa1dc143380 0x7fa1dc1ef7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:22:50.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:50.792+0000 7fa1e2a00640 1 --2- 172.21.15.6:0/3605311745 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa1dc1538b0 0x7fa1dc1ef270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:22:50.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:50.792+0000 7fa1e2a00640 1 --2- 172.21.15.6:0/3605311745 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa1c4005210 0x7fa1dc14e020 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:22:50.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:50.792+0000 7fa1e2a00640 1 -- 172.21.15.6:0/3605311745 >> 172.21.15.6:0/3605311745 conn(0x7fa1dc06be10 msgr2=0x7fa1dc14d5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:22:50.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:50.792+0000 7fa1e2a00640 1 -- 172.21.15.6:0/3605311745 shutdown_connections 2024-04-24T23:22:50.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:50.792+0000 7fa1e2a00640 1 -- 172.21.15.6:0/3605311745 wait complete. 2024-04-24T23:22:50.834 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-24T23:22:52.835 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:22:53.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:53.396+0000 7fd76ec00640 1 Processor -- start 2024-04-24T23:22:53.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:53.396+0000 7fd76ec00640 1 -- start start 2024-04-24T23:22:53.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:53.398+0000 7fd76ec00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd768154b70 0x7fd76806f020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:22:53.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:53.398+0000 7fd76ec00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd768072530 con 0x7fd768071000 2024-04-24T23:22:53.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:53.398+0000 7fd76ec00640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7680727d0 con 0x7fd7680c7080 2024-04-24T23:22:53.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:53.398+0000 7fd76ec00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7fd768072a70 con 0x7fd768154b70 2024-04-24T23:22:53.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:53.398+0000 7fd76cc00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fd768071000 0x7fd76815b0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:33304/0 (socket says 172.21.15.6:33304) 2024-04-24T23:22:53.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:53.398+0000 7fd76cc00640 1 -- 172.21.15.6:0/1283728463 learned_addr learned my addr 172.21.15.6:0/1283728463 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:22:53.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:53.399+0000 7fd766a00640 1 -- 172.21.15.6:0/1283728463 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2698993124 0 0) 0x7fd768072530 con 0x7fd768071000 2024-04-24T23:22:53.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:53.399+0000 7fd766a00640 1 -- 172.21.15.6:0/1283728463 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd74c003520 con 0x7fd768071000 2024-04-24T23:22:53.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:53.399+0000 7fd766a00640 1 -- 172.21.15.6:0/1283728463 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1752460907 0 0) 0x7fd74c003520 con 0x7fd768071000 2024-04-24T23:22:53.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:53.399+0000 7fd766a00640 1 -- 172.21.15.6:0/1283728463 >> v2:172.21.15.6:3300/0 conn(0x7fd768154b70 msgr2=0x7fd76806f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:22:53.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:53.400+0000 7fd767e00640 1 --2- 172.21.15.6:0/1283728463 >> v2:172.21.15.6:3300/0 conn(0x7fd768154b70 0x7fd76806f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:22:53.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:53.400+0000 7fd766a00640 1 --2- 172.21.15.6:0/1283728463 >> v2:172.21.15.6:3300/0 conn(0x7fd768154b70 0x7fd76806f020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:22:53.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:53.400+0000 7fd766a00640 1 -- 172.21.15.6:0/1283728463 >> v1:172.21.15.190:6789/0 conn(0x7fd7680c7080 legacy=0x7fd76815a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:22:53.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:53.400+0000 7fd766a00640 1 -- 172.21.15.6:0/1283728463 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd76806e540 con 0x7fd768071000 2024-04-24T23:22:53.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:53.400+0000 7fd766a00640 1 -- 172.21.15.6:0/1283728463 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3463268701 0 0) 0x7fd760001a10 con 0x7fd768071000 2024-04-24T23:22:53.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:53.400+0000 7fd766a00640 1 -- 172.21.15.6:0/1283728463 >> v1:172.21.15.6:6789/0 conn(0x7fd768071000 legacy=0x7fd76815b0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:22:53.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:53.400+0000 7fd766a00640 1 --2- 172.21.15.6:0/1283728463 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd74c0039b0 0x7fd74c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:22:53.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:53.400+0000 7fd766a00640 1 --2- 172.21.15.6:0/1283728463 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd74c004800 0x7fd74c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:22:53.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:53.400+0000 7fd766a00640 1 --2- 172.21.15.6:0/1283728463 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd74c005210 0x7fd74c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:22:53.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:53.400+0000 7fd766a00640 1 -- 172.21.15.6:0/1283728463 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fd768072a70 con 0x7fd74c005210 2024-04-24T23:22:53.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:53.400+0000 7fd766a00640 1 -- 172.21.15.6:0/1283728463 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fd74c00c160 con 0x7fd74c0039b0 2024-04-24T23:22:53.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:53.400+0000 7fd766a00640 1 -- 172.21.15.6:0/1283728463 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fd74c00c360 con 0x7fd74c004800 2024-04-24T23:22:53.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:53.401+0000 7fd767400640 1 --2- 172.21.15.6:0/1283728463 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd74c004800 0x7fd74c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:22:53.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:53.401+0000 7fd767400640 1 -- 172.21.15.6:0/1283728463 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd74c0039b0 msgr2=0x7fd74c003d90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-24T23:22:53.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:53.401+0000 7fd767400640 1 --2- 172.21.15.6:0/1283728463 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd74c0039b0 0x7fd74c003d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:22:53.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:53.401+0000 7fd767400640 1 -- 172.21.15.6:0/1283728463 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd74c005210 msgr2=0x7fd74c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:22:53.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:53.401+0000 7fd767400640 1 --2- 172.21.15.6:0/1283728463 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd74c005210 0x7fd74c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:22:53.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:53.401+0000 7fd767400640 1 -- 172.21.15.6:0/1283728463 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd76806e540 con 0x7fd74c004800 2024-04-24T23:22:53.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:53.402+0000 7fd767e00640 1 --2- 172.21.15.6:0/1283728463 >> v2:172.21.15.6:3300/0 conn(0x7fd768154b70 0x7fd76806f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:22:53.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:53.403+0000 7fd76cc00640 1 -- 172.21.15.6:0/1283728463 reap_dead start 2024-04-24T23:22:53.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:53.403+0000 7fd767400640 1 --2- 172.21.15.6:0/1283728463 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd74c004800 0x7fd74c004c80 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fd758001860 tx=0x7fd74c005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=509bb5097aa78a87 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:22:53.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:53.403+0000 7fd766a00640 1 -- 172.21.15.6:0/1283728463 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd758001430 con 0x7fd74c004800 2024-04-24T23:22:53.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:53.403+0000 7fd766a00640 1 -- 172.21.15.6:0/1283728463 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd758005330 con 0x7fd74c004800 2024-04-24T23:22:53.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:53.405+0000 7fd76ec00640 1 -- 172.21.15.6:0/1283728463 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd74c004800 msgr2=0x7fd74c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:22:53.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:53.405+0000 7fd76ec00640 1 --2- 172.21.15.6:0/1283728463 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd74c004800 0x7fd74c004c80 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fd758001860 tx=0x7fd74c005830 comp rx=0 tx=0).stop 2024-04-24T23:22:53.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:53.405+0000 7fd76ec00640 1 -- 172.21.15.6:0/1283728463 shutdown_connections 2024-04-24T23:22:53.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:53.405+0000 7fd76ec00640 1 --2- 172.21.15.6:0/1283728463 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd74c004800 0x7fd74c004c80 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:22:53.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:53.405+0000 7fd76ec00640 1 -- 172.21.15.6:0/1283728463 >> 172.21.15.6:0/1283728463 conn(0x7fd76806b0e0 msgr2=0x7fd76806b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:22:53.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:53.406+0000 7fd76ec00640 1 -- 172.21.15.6:0/1283728463 shutdown_connections 2024-04-24T23:22:53.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:53.406+0000 7fd76ec00640 1 -- 172.21.15.6:0/1283728463 wait complete. 2024-04-24T23:22:53.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:53.410+0000 7fd76ec00640 1 Processor -- start 2024-04-24T23:22:53.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:53.412+0000 7fd76ec00640 1 -- start start 2024-04-24T23:22:53.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:53.413+0000 7fd76ec00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd768150fd0 0x7fd7681f57f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:22:53.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:53.413+0000 7fd76ec00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd7681513b0 0x7fd7681f6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:22:53.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:53.413+0000 7fd76ec00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd76815eab0 0x7fd7681f8b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:22:53.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:53.413+0000 7fd76ec00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fd768154f50 con 0x7fd7681513b0 2024-04-24T23:22:53.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:53.413+0000 7fd76ec00640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fd76806eb00 con 0x7fd76815eab0 2024-04-24T23:22:53.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:53.413+0000 7fd76ec00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fd7681550d0 con 0x7fd768150fd0 2024-04-24T23:22:53.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:53.414+0000 7fd76cc00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd76815eab0 0x7fd7681f8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:22:53.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:53.414+0000 7fd767e00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd768150fd0 0x7fd7681f57f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:22:53.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:53.415+0000 7fd767e00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd768150fd0 0x7fd7681f57f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.6:49928/0 (socket says 172.21.15.6:49928) 2024-04-24T23:22:53.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:53.415+0000 7fd767e00640 1 -- 172.21.15.6:0/2482585390 learned_addr learned my addr 172.21.15.6:0/2482585390 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:22:53.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:53.416+0000 7fd767400640 1 --2- 172.21.15.6:0/2482585390 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd7681513b0 0x7fd7681f6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:22:53.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:53.416+0000 7fd767e00640 1 -- 172.21.15.6:0/2482585390 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd76815eab0 msgr2=0x7fd7681f8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:22:53.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:53.416+0000 7fd767e00640 1 --2- 172.21.15.6:0/2482585390 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd76815eab0 0x7fd7681f8b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:22:53.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:53.416+0000 7fd767e00640 1 -- 172.21.15.6:0/2482585390 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd7681513b0 msgr2=0x7fd7681f6070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:22:53.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:53.416+0000 7fd767e00640 1 --2- 172.21.15.6:0/2482585390 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd7681513b0 0x7fd7681f6070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:22:53.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:53.417+0000 7fd767e00640 1 -- 172.21.15.6:0/2482585390 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd76814c040 con 0x7fd768150fd0 2024-04-24T23:22:53.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:53.417+0000 7fd76cc00640 1 --2- 172.21.15.6:0/2482585390 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd76815eab0 0x7fd7681f8b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:22:53.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:53.418+0000 7fd767400640 1 --2- 172.21.15.6:0/2482585390 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd7681513b0 0x7fd7681f6070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:22:53.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:53.420+0000 7fd767e00640 1 --2- 172.21.15.6:0/2482585390 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd768150fd0 0x7fd7681f57f0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fd75400c7f0 tx=0x7fd75400ccb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:22:53.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:53.420+0000 7fd75fe00640 1 -- 172.21.15.6:0/2482585390 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd754015070 con 0x7fd768150fd0 2024-04-24T23:22:53.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:53.421+0000 7fd75fe00640 1 -- 172.21.15.6:0/2482585390 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7540077c0 con 0x7fd768150fd0 2024-04-24T23:22:53.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:53.421+0000 7fd75fe00640 1 -- 172.21.15.6:0/2482585390 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd754007bb0 con 0x7fd768150fd0 2024-04-24T23:22:53.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:53.422+0000 7fd76ec00640 1 -- 172.21.15.6:0/2482585390 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd76814ba80 con 0x7fd768150fd0 2024-04-24T23:22:53.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:53.423+0000 7fd76ec00640 1 -- 172.21.15.6:0/2482585390 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7681f4d10 con 0x7fd768150fd0 2024-04-24T23:22:53.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:53.425+0000 7fd76ec00640 1 -- 172.21.15.6:0/2482585390 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd7680767b0 con 0x7fd768150fd0 2024-04-24T23:22:53.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:53.427+0000 7fd75fe00640 1 -- 172.21.15.6:0/2482585390 <== mon.2 v2:172.21.15.190:3301/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fd754018070 con 0x7fd768150fd0 2024-04-24T23:22:53.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:53.431+0000 7fd75fe00640 1 --2- 172.21.15.6:0/2482585390 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fd74806be80 0x7fd74806e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:22:53.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:53.431+0000 7fd75fe00640 1 -- 172.21.15.6:0/2482585390 <== mon.2 v2:172.21.15.190:3301/0 5 ==== osd_map(28..28 src has 1..28) ==== 6762+0+0 (secure 0 0 0) 0x7fd75408b2d0 con 0x7fd768150fd0 2024-04-24T23:22:53.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:53.432+0000 7fd767400640 1 --2- 172.21.15.6:0/2482585390 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fd74806be80 0x7fd74806e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:22:53.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:53.433+0000 7fd767400640 1 --2- 172.21.15.6:0/2482585390 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fd74806be80 0x7fd74806e340 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fd758032000 tx=0x7fd758009030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:22:53.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:53.440+0000 7fd75fe00640 1 -- 172.21.15.6:0/2482585390 <== mon.2 v2:172.21.15.190:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fd754059940 con 0x7fd768150fd0 2024-04-24T23:22:53.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:53.992+0000 7fd76ec00640 1 -- 172.21.15.6:0/2482585390 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd768151df0 con 0x7fd768150fd0 2024-04-24T23:22:53.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:53.997+0000 7fd75fe00640 1 -- 172.21.15.6:0/2482585390 <== mon.2 v2:172.21.15.190:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+6885 (secure 0 0 0) 0x7fd754059240 con 0x7fd768150fd0 2024-04-24T23:22:53.999 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 35 2024-04-24T23:22:54.000 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:22:54.000 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5081,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/1190759626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1190759626},{"type":"v1","addr":"172.21.15.6:6834","nonce":1190759626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":5503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/3928731059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":3928731059},{"type":"v1","addr":"172.21.15.6:6837","nonce":3928731059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:22:37.858824+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1,2,3],"up":{"mds_0":4306,"mds_1":4282,"mds_2":4235,"mds_3":4232},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_4232":{"gid":4232,"name":"b","rank":3,"incarnation":15,"state":"up:stopping","state_seq":15,"addr":"172.21.15.190:6838/2032280158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2032280158},{"type":"v1","addr":"172.21.15.190:6838","nonce":2032280158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4235":{"gid":4235,"name":"d","rank":2,"incarnation":13,"state":"up:active","state_seq":14,"addr":"172.21.15.190:6839/1572872451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":1572872451},{"type":"v1","addr":"172.21.15.190:6839","nonce":1572872451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4282":{"gid":4282,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":12,"addr":"172.21.15.190:6837/580315269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":580315269},{"type":"v1","addr":"172.21.15.190:6837","nonce":580315269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/489092393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":489092393},{"type":"v1","addr":"172.21.15.6:6835","nonce":489092393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4306,"qdb_cluster":[4306,4282,4235]},"id":1}]} 2024-04-24T23:22:54.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:54.003+0000 7fd76ec00640 1 -- 172.21.15.6:0/2482585390 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fd74806be80 msgr2=0x7fd74806e340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:22:54.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:54.003+0000 7fd76ec00640 1 --2- 172.21.15.6:0/2482585390 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fd74806be80 0x7fd74806e340 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fd758032000 tx=0x7fd758009030 comp rx=0 tx=0).stop 2024-04-24T23:22:54.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:54.003+0000 7fd76ec00640 1 -- 172.21.15.6:0/2482585390 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd768150fd0 msgr2=0x7fd7681f57f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:22:54.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:54.003+0000 7fd76ec00640 1 --2- 172.21.15.6:0/2482585390 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd768150fd0 0x7fd7681f57f0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fd75400c7f0 tx=0x7fd75400ccb0 comp rx=0 tx=0).stop 2024-04-24T23:22:54.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:54.004+0000 7fd76ec00640 1 -- 172.21.15.6:0/2482585390 shutdown_connections 2024-04-24T23:22:54.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:54.004+0000 7fd76ec00640 1 --2- 172.21.15.6:0/2482585390 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fd74806be80 0x7fd74806e340 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:22:54.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:54.004+0000 7fd76ec00640 1 --2- 172.21.15.6:0/2482585390 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd76815eab0 0x7fd7681f8b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:22:54.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:54.004+0000 7fd76ec00640 1 --2- 172.21.15.6:0/2482585390 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd7681513b0 0x7fd7681f6070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:22:54.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:54.004+0000 7fd76ec00640 1 --2- 172.21.15.6:0/2482585390 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd768150fd0 0x7fd7681f57f0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:22:54.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:54.004+0000 7fd76ec00640 1 -- 172.21.15.6:0/2482585390 >> 172.21.15.6:0/2482585390 conn(0x7fd76806b0e0 msgr2=0x7fd7680c9000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:22:54.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:54.005+0000 7fd76ec00640 1 -- 172.21.15.6:0/2482585390 shutdown_connections 2024-04-24T23:22:54.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:54.005+0000 7fd76ec00640 1 -- 172.21.15.6:0/2482585390 wait complete. 2024-04-24T23:22:54.052 INFO:tasks.mds_thrash.fs.[cephfs]:cluster is considered unstable while MDS are in up:stopping (!thrash_while_stopping) 2024-04-24T23:22:54.053 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 35, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-24T23:15:36.400716+0000", "damaged": [], "data_pools": [ 3, 4 ], "enabled": true, "epoch": 34, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3 ], "info": { "gid_4232": { "addr": "172.21.15.190:6838/2032280158", "addrs": { "addrvec": [ { "addr": "172.21.15.190:6835", "nonce": 2032280158, "type": "v2" }, { "addr": "172.21.15.190:6838", "nonce": 2032280158, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4232, "incarnation": 15, "join_fscid": -1, "name": "b", "rank": 3, "state": "up:stopping", "state_seq": 15 }, "gid_4235": { "addr": "172.21.15.190:6839/1572872451", "addrs": { "addrvec": [ { "addr": "172.21.15.190:6836", "nonce": 1572872451, "type": "v2" }, { "addr": "172.21.15.190:6839", "nonce": 1572872451, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4235, "incarnation": 13, "join_fscid": -1, "name": "d", "rank": 2, "state": "up:active", "state_seq": 14 }, "gid_4282": { "addr": "172.21.15.190:6837/580315269", "addrs": { "addrvec": [ { "addr": "172.21.15.190:6834", "nonce": 580315269, "type": "v2" }, { "addr": "172.21.15.190:6837", "nonce": 580315269, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4282, "incarnation": 9, "join_fscid": -1, "name": "f", "rank": 1, "state": "up:active", "state_seq": 12 }, "gid_4306": { "addr": "172.21.15.6:6835/489092393", "addrs": { "addrvec": [ { "addr": "172.21.15.6:6833", "nonce": 489092393, "type": "v2" }, { "addr": "172.21.15.6:6835", "nonce": 489092393, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4306, "incarnation": 4, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 4 } }, "last_failure": 0, "last_failure_osd_epoch": 28, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-04-24T23:22:37.858824+0000", "qdb_cluster": [ 4306, 4282, 4235 ], "qdb_leader": 4306, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 4 ], "tableserver": 0, "up": { "mds_0": 4306, "mds_1": 4282, "mds_2": 4235, "mds_3": 4232 } } } ], "standbys": [ { "addr": "172.21.15.6:6834/1190759626", "addrs": { "addrvec": [ { "addr": "172.21.15.6:6832", "nonce": 1190759626, "type": "v2" }, { "addr": "172.21.15.6:6834", "nonce": 1190759626, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 30, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5081, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.6:6837/3928731059", "addrs": { "addrvec": [ { "addr": "172.21.15.6:6836", "nonce": 3928731059, "type": "v2" }, { "addr": "172.21.15.6:6837", "nonce": 3928731059, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 35, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5503, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-04-24T23:22:56.054 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:22:56.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:56.253+0000 7f88ffc00640 1 Processor -- start 2024-04-24T23:22:56.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:56.253+0000 7f88ffc00640 1 -- start start 2024-04-24T23:22:56.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:56.255+0000 7f88ffc00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f88f8072530 0x7f88f816b300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:22:56.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:56.255+0000 7f88ffc00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f88f8072980 con 0x7f88f80c7dc0 2024-04-24T23:22:56.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:56.255+0000 7f88ffc00640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f88f80c74c0 con 0x7f88f8167180 2024-04-24T23:22:56.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:56.255+0000 7f88ffc00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f88f8168570 con 0x7f88f8072530 2024-04-24T23:22:56.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:56.256+0000 7f88fdc00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f88f8072530 0x7f88f816b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:22:56.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:56.256+0000 7f88fdc00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f88f8072530 0x7f88f816b300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.6:41924/0 (socket says 172.21.15.6:41924) 2024-04-24T23:22:56.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:56.256+0000 7f88fdc00640 1 -- 172.21.15.6:0/3372317431 learned_addr learned my addr 172.21.15.6:0/3372317431 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:22:56.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:56.257+0000 7f88f7400640 1 -- 172.21.15.6:0/3372317431 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 111966579 0 0) 0x7f88f8072980 con 0x7f88f80c7dc0 2024-04-24T23:22:56.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:56.325+0000 7f88f7400640 1 -- 172.21.15.6:0/3372317431 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f88dc003500 con 0x7f88f80c7dc0 2024-04-24T23:22:56.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:56.325+0000 7f88f7400640 1 -- 172.21.15.6:0/3372317431 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3890206555 0 0) 0x7f88f80c74c0 con 0x7f88f8167180 2024-04-24T23:22:56.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:56.325+0000 7f88f7400640 1 -- 172.21.15.6:0/3372317431 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f88f8072980 con 0x7f88f8167180 2024-04-24T23:22:56.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:56.326+0000 7f88f7400640 1 -- 172.21.15.6:0/3372317431 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1697127743 0 0) 0x7f88f8072980 con 0x7f88f8167180 2024-04-24T23:22:56.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:56.326+0000 7f88f7400640 1 -- 172.21.15.6:0/3372317431 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f88f8072530 msgr2=0x7f88f816b300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:22:56.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:56.326+0000 7f88f7400640 1 --2- 172.21.15.6:0/3372317431 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f88f8072530 0x7f88f816b300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:22:56.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:56.326+0000 7f88f7400640 1 -- 172.21.15.6:0/3372317431 >> v1:172.21.15.6:6789/0 conn(0x7f88f80c7dc0 legacy=0x7f88f816abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:22:56.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:56.326+0000 7f88f7400640 1 -- 172.21.15.6:0/3372317431 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88f80c71e0 con 0x7f88f8167180 2024-04-24T23:22:56.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:56.326+0000 7f88f7400640 1 -- 172.21.15.6:0/3372317431 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3463268701 0 0) 0x7f88f00019f0 con 0x7f88f8167180 2024-04-24T23:22:56.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:56.327+0000 7f88f7400640 1 -- 172.21.15.6:0/3372317431 >> v1:172.21.15.190:6789/0 conn(0x7f88f8167180 legacy=0x7f88f806edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:22:56.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:56.327+0000 7f88f7400640 1 --2- 172.21.15.6:0/3372317431 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f88dc003d20 0x7f88dc004120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:22:56.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:56.327+0000 7f88f7400640 1 --2- 172.21.15.6:0/3372317431 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f88dc0046f0 0x7f88dc006c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:22:56.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:56.327+0000 7f88f7400640 1 --2- 172.21.15.6:0/3372317431 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f88dc007220 0x7f88dc0096b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:22:56.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:56.327+0000 7f88f7400640 1 -- 172.21.15.6:0/3372317431 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f88dc009ec0 con 0x7f88dc003d20 2024-04-24T23:22:56.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:56.327+0000 7f88f7400640 1 -- 172.21.15.6:0/3372317431 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f88dc00a070 con 0x7f88dc007220 2024-04-24T23:22:56.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:56.327+0000 7f88f7400640 1 -- 172.21.15.6:0/3372317431 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f88dc00a270 con 0x7f88dc0046f0 2024-04-24T23:22:56.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:56.327+0000 7f88fdc00640 1 --2- 172.21.15.6:0/3372317431 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f88f8072530 0x7f88f816b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:22:56.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:56.329+0000 7f88f7e00640 1 --2- 172.21.15.6:0/3372317431 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f88dc0046f0 0x7f88dc006c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:22:56.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:56.329+0000 7f88fdc00640 1 --2- 172.21.15.6:0/3372317431 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f88dc007220 0x7f88dc0096b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:22:56.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:56.330+0000 7f88fd200640 1 --2- 172.21.15.6:0/3372317431 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f88dc003d20 0x7f88dc004120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:22:56.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:56.331+0000 7f88f7e00640 1 -- 172.21.15.6:0/3372317431 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f88dc007220 msgr2=0x7f88dc0096b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:22:56.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:56.331+0000 7f88f7e00640 1 --2- 172.21.15.6:0/3372317431 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f88dc007220 0x7f88dc0096b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:22:56.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:56.331+0000 7f88f7e00640 1 -- 172.21.15.6:0/3372317431 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f88dc003d20 msgr2=0x7f88dc004120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:22:56.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:56.331+0000 7f88f7e00640 1 --2- 172.21.15.6:0/3372317431 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f88dc003d20 0x7f88dc004120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:22:56.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:56.332+0000 7f88f7e00640 1 -- 172.21.15.6:0/3372317431 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88f80c71e0 con 0x7f88dc0046f0 2024-04-24T23:22:56.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:56.334+0000 7f88f7e00640 1 --2- 172.21.15.6:0/3372317431 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f88dc0046f0 0x7f88dc006c00 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f88e4001900 tx=0x7f88dc004b20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d88b020dba41c13 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:22:56.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:56.334+0000 7f88f7400640 1 -- 172.21.15.6:0/3372317431 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88e4001520 con 0x7f88dc0046f0 2024-04-24T23:22:56.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:56.335+0000 7f88f7400640 1 -- 172.21.15.6:0/3372317431 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f88e402aca0 con 0x7f88dc0046f0 2024-04-24T23:22:56.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:56.336+0000 7f88ffc00640 1 -- 172.21.15.6:0/3372317431 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f88dc0046f0 msgr2=0x7f88dc006c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:22:56.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:56.336+0000 7f88f7400640 1 -- 172.21.15.6:0/3372317431 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88e4001520 con 0x7f88dc0046f0 2024-04-24T23:22:56.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:56.336+0000 7f88ffc00640 1 --2- 172.21.15.6:0/3372317431 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f88dc0046f0 0x7f88dc006c00 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f88e4001900 tx=0x7f88dc004b20 comp rx=0 tx=0).stop 2024-04-24T23:22:56.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:56.336+0000 7f88fdc00640 1 -- 172.21.15.6:0/3372317431 reap_dead start 2024-04-24T23:22:56.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:56.336+0000 7f88ffc00640 1 -- 172.21.15.6:0/3372317431 shutdown_connections 2024-04-24T23:22:56.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:56.336+0000 7f88ffc00640 1 -- 172.21.15.6:0/3372317431 >> 172.21.15.6:0/3372317431 conn(0x7f88f806b0e0 msgr2=0x7f88f806b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:22:56.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:56.337+0000 7f88ffc00640 1 -- 172.21.15.6:0/3372317431 shutdown_connections 2024-04-24T23:22:56.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:56.337+0000 7f88ffc00640 1 -- 172.21.15.6:0/3372317431 wait complete. 2024-04-24T23:22:56.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:56.341+0000 7f88ffc00640 1 Processor -- start 2024-04-24T23:22:56.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:56.341+0000 7f88ffc00640 1 -- start start 2024-04-24T23:22:56.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:56.344+0000 7f88ffc00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f88dc0046f0 0x7f88f81fd9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:22:56.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:56.344+0000 7f88ffc00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f88f81629d0 0x7f88f8162e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:22:56.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:56.344+0000 7f88ffc00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f88f8168d80 0x7f88f8169200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:22:56.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:56.347+0000 7f88fd200640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f88dc0046f0 0x7f88f81fd9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:22:56.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:56.348+0000 7f88fd200640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f88dc0046f0 0x7f88f81fd9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:60262/0 (socket says 172.21.15.6:60262) 2024-04-24T23:22:56.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:56.348+0000 7f88fdc00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f88f8168d80 0x7f88f8169200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:22:56.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:56.348+0000 7f88fd200640 1 -- 172.21.15.6:0/3849304070 learned_addr learned my addr 172.21.15.6:0/3849304070 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:22:56.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:56.348+0000 7f88ffc00640 1 -- 172.21.15.6:0/3849304070 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f88f8071a50 con 0x7f88f8168d80 2024-04-24T23:22:56.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:56.348+0000 7f88ffc00640 1 -- 172.21.15.6:0/3849304070 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f88f80720a0 con 0x7f88dc0046f0 2024-04-24T23:22:56.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:56.348+0000 7f88ffc00640 1 -- 172.21.15.6:0/3849304070 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f88f8071d30 con 0x7f88f81629d0 2024-04-24T23:22:56.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:56.349+0000 7f88f7e00640 1 --2- 172.21.15.6:0/3849304070 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f88f81629d0 0x7f88f8162e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:22:56.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:56.349+0000 7f88f7e00640 1 -- 172.21.15.6:0/3849304070 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f88dc0046f0 msgr2=0x7f88f81fd9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:22:56.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:56.349+0000 7f88f7e00640 1 --2- 172.21.15.6:0/3849304070 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f88dc0046f0 0x7f88f81fd9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:22:56.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:56.349+0000 7f88f7e00640 1 -- 172.21.15.6:0/3849304070 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f88f8168d80 msgr2=0x7f88f8169200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:22:56.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:56.349+0000 7f88f7e00640 1 --2- 172.21.15.6:0/3849304070 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f88f8168d80 0x7f88f8169200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:22:56.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:56.349+0000 7f88f7e00640 1 -- 172.21.15.6:0/3849304070 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88f8163350 con 0x7f88f81629d0 2024-04-24T23:22:56.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:56.349+0000 7f88fd200640 1 --2- 172.21.15.6:0/3849304070 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f88dc0046f0 0x7f88f81fd9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:22:56.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:56.349+0000 7f88fdc00640 1 --2- 172.21.15.6:0/3849304070 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f88f8168d80 0x7f88f8169200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:22:56.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:56.350+0000 7f88f7e00640 1 --2- 172.21.15.6:0/3849304070 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f88f81629d0 0x7f88f8162e10 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f88e4004000 tx=0x7f88e40012d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:22:56.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:56.350+0000 7f88f4c00640 1 -- 172.21.15.6:0/3849304070 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88e4001520 con 0x7f88f81629d0 2024-04-24T23:22:56.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:56.351+0000 7f88ffc00640 1 -- 172.21.15.6:0/3849304070 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88f8169740 con 0x7f88f81629d0 2024-04-24T23:22:56.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:56.351+0000 7f88f4c00640 1 -- 172.21.15.6:0/3849304070 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f88e402ae50 con 0x7f88f81629d0 2024-04-24T23:22:56.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:56.351+0000 7f88f4c00640 1 -- 172.21.15.6:0/3849304070 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88e40069e0 con 0x7f88f81629d0 2024-04-24T23:22:56.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:56.351+0000 7f88ffc00640 1 -- 172.21.15.6:0/3849304070 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f88f81f0cd0 con 0x7f88f81629d0 2024-04-24T23:22:56.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:56.352+0000 7f88ffc00640 1 -- 172.21.15.6:0/3849304070 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f88f806e660 con 0x7f88f81629d0 2024-04-24T23:22:56.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:56.354+0000 7f88f4c00640 1 -- 172.21.15.6:0/3849304070 <== mon.2 v2:172.21.15.190:3301/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f88e4006b80 con 0x7f88f81629d0 2024-04-24T23:22:56.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:56.356+0000 7f88f4c00640 1 --2- 172.21.15.6:0/3849304070 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f88d806be30 0x7f88d806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:22:56.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:56.356+0000 7f88f4c00640 1 -- 172.21.15.6:0/3849304070 <== mon.2 v2:172.21.15.190:3301/0 5 ==== osd_map(28..28 src has 1..28) ==== 6762+0+0 (secure 0 0 0) 0x7f88e40ab0d0 con 0x7f88f81629d0 2024-04-24T23:22:56.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:56.357+0000 7f88fd200640 1 --2- 172.21.15.6:0/3849304070 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f88d806be30 0x7f88d806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:22:56.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:56.361+0000 7f88fd200640 1 --2- 172.21.15.6:0/3849304070 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f88d806be30 0x7f88d806e2f0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f88f000f560 tx=0x7f88f0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:22:56.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:56.362+0000 7f88f4c00640 1 -- 172.21.15.6:0/3849304070 <== mon.2 v2:172.21.15.190:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f88f806e660 con 0x7f88f81629d0 2024-04-24T23:22:56.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:56.839+0000 7f88ffc00640 1 -- 172.21.15.6:0/3849304070 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f88f80767b0 con 0x7f88f81629d0 2024-04-24T23:22:56.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:56.841+0000 7f88f4c00640 1 -- 172.21.15.6:0/3849304070 <== mon.2 v2:172.21.15.190:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+6885 (secure 0 0 0) 0x7f88e4079660 con 0x7f88f81629d0 2024-04-24T23:22:56.846 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:22:56.846 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5081,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/1190759626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1190759626},{"type":"v1","addr":"172.21.15.6:6834","nonce":1190759626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":5503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/3928731059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":3928731059},{"type":"v1","addr":"172.21.15.6:6837","nonce":3928731059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:22:37.858824+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1,2,3],"up":{"mds_0":4306,"mds_1":4282,"mds_2":4235,"mds_3":4232},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_4232":{"gid":4232,"name":"b","rank":3,"incarnation":15,"state":"up:stopping","state_seq":15,"addr":"172.21.15.190:6838/2032280158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2032280158},{"type":"v1","addr":"172.21.15.190:6838","nonce":2032280158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4235":{"gid":4235,"name":"d","rank":2,"incarnation":13,"state":"up:active","state_seq":14,"addr":"172.21.15.190:6839/1572872451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":1572872451},{"type":"v1","addr":"172.21.15.190:6839","nonce":1572872451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4282":{"gid":4282,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":12,"addr":"172.21.15.190:6837/580315269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":580315269},{"type":"v1","addr":"172.21.15.190:6837","nonce":580315269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/489092393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":489092393},{"type":"v1","addr":"172.21.15.6:6835","nonce":489092393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4306,"qdb_cluster":[4306,4282,4235]},"id":1}]} 2024-04-24T23:22:56.846 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 35 2024-04-24T23:22:56.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:56.845+0000 7f88ffc00640 1 -- 172.21.15.6:0/3849304070 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f88d806be30 msgr2=0x7f88d806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:22:56.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:56.845+0000 7f88ffc00640 1 --2- 172.21.15.6:0/3849304070 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f88d806be30 0x7f88d806e2f0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f88f000f560 tx=0x7f88f0009040 comp rx=0 tx=0).stop 2024-04-24T23:22:56.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:56.846+0000 7f88ffc00640 1 -- 172.21.15.6:0/3849304070 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f88f81629d0 msgr2=0x7f88f8162e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:22:56.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:56.846+0000 7f88ffc00640 1 --2- 172.21.15.6:0/3849304070 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f88f81629d0 0x7f88f8162e10 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f88e4004000 tx=0x7f88e40012d0 comp rx=0 tx=0).stop 2024-04-24T23:22:56.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:56.846+0000 7f88ffc00640 1 -- 172.21.15.6:0/3849304070 shutdown_connections 2024-04-24T23:22:56.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:56.846+0000 7f88ffc00640 1 --2- 172.21.15.6:0/3849304070 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f88d806be30 0x7f88d806e2f0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:22:56.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:56.846+0000 7f88ffc00640 1 --2- 172.21.15.6:0/3849304070 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f88f81629d0 0x7f88f8162e10 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:22:56.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:56.846+0000 7f88ffc00640 1 --2- 172.21.15.6:0/3849304070 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f88f8168d80 0x7f88f8169200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:22:56.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:56.846+0000 7f88ffc00640 1 --2- 172.21.15.6:0/3849304070 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f88dc0046f0 0x7f88f81fd9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:22:56.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:56.846+0000 7f88ffc00640 1 -- 172.21.15.6:0/3849304070 >> 172.21.15.6:0/3849304070 conn(0x7f88f806b0e0 msgr2=0x7f88f806ed40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:22:56.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:56.847+0000 7f88ffc00640 1 -- 172.21.15.6:0/3849304070 shutdown_connections 2024-04-24T23:22:56.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:56.847+0000 7f88ffc00640 1 -- 172.21.15.6:0/3849304070 wait complete. 2024-04-24T23:22:56.883 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-24T23:22:58.884 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:22:59.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:59.213+0000 7f6420800640 1 Processor -- start 2024-04-24T23:22:59.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:59.214+0000 7f6420800640 1 -- start start 2024-04-24T23:22:59.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:59.216+0000 7f6420800640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6418152d80 0x7f64180c8d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:22:59.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:59.216+0000 7f6420800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f64181592a0 0x7f641815c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:22:59.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:59.216+0000 7f6420800640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f641815cb10 0x7f6418070c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:22:59.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:59.217+0000 7f641de00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6418152d80 0x7f64180c8d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:22:59.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:59.217+0000 7f6420800640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f64180c5500 con 0x7f64181592a0 2024-04-24T23:22:59.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:59.217+0000 7f6420800640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f641815c140 con 0x7f641815cb10 2024-04-24T23:22:59.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:59.217+0000 7f6420800640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f6418159770 con 0x7f6418152d80 2024-04-24T23:22:59.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:59.217+0000 7f641e800640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f641815cb10 0x7f6418070c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:22:59.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:59.218+0000 7f641e800640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f641815cb10 0x7f6418070c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:60278/0 (socket says 172.21.15.6:60278) 2024-04-24T23:22:59.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:59.218+0000 7f641e800640 1 -- 172.21.15.6:0/235622961 learned_addr learned my addr 172.21.15.6:0/235622961 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:22:59.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:59.218+0000 7f641d400640 1 --2- 172.21.15.6:0/235622961 >> v2:172.21.15.6:3300/0 conn(0x7f64181592a0 0x7f641815c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:22:59.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:59.220+0000 7f641d400640 1 -- 172.21.15.6:0/235622961 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6418152d80 msgr2=0x7f64180c8d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:22:59.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:59.220+0000 7f641d400640 1 --2- 172.21.15.6:0/235622961 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6418152d80 0x7f64180c8d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:22:59.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:59.220+0000 7f641d400640 1 -- 172.21.15.6:0/235622961 >> v2:172.21.15.190:3300/0 conn(0x7f641815cb10 msgr2=0x7f6418070c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:22:59.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:59.220+0000 7f641d400640 1 --2- 172.21.15.6:0/235622961 >> v2:172.21.15.190:3300/0 conn(0x7f641815cb10 0x7f6418070c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:22:59.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:59.220+0000 7f641d400640 1 -- 172.21.15.6:0/235622961 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64180c3470 con 0x7f64181592a0 2024-04-24T23:22:59.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:59.220+0000 7f641de00640 1 --2- 172.21.15.6:0/235622961 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6418152d80 0x7f64180c8d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:22:59.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:59.220+0000 7f641e800640 1 --2- 172.21.15.6:0/235622961 >> v2:172.21.15.190:3300/0 conn(0x7f641815cb10 0x7f6418070c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:22:59.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:59.222+0000 7f641d400640 1 --2- 172.21.15.6:0/235622961 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f64181592a0 0x7f641815c5d0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f6418071750 tx=0x7f640802a100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e39d420b44549bb server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:22:59.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:59.225+0000 7f641ca00640 1 -- 172.21.15.6:0/235622961 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6408037070 con 0x7f64181592a0 2024-04-24T23:22:59.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:59.225+0000 7f641ca00640 1 -- 172.21.15.6:0/235622961 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f640802ae50 con 0x7f64181592a0 2024-04-24T23:22:59.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:59.226+0000 7f641ca00640 1 -- 172.21.15.6:0/235622961 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6408032660 con 0x7f64181592a0 2024-04-24T23:22:59.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:59.227+0000 7f6420800640 1 -- 172.21.15.6:0/235622961 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f64181592a0 msgr2=0x7f641815c5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:22:59.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:59.227+0000 7f6420800640 1 --2- 172.21.15.6:0/235622961 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f64181592a0 0x7f641815c5d0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f6418071750 tx=0x7f640802a100 comp rx=0 tx=0).stop 2024-04-24T23:22:59.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:59.227+0000 7f6420800640 1 -- 172.21.15.6:0/235622961 shutdown_connections 2024-04-24T23:22:59.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:59.227+0000 7f6420800640 1 --2- 172.21.15.6:0/235622961 >> v2:172.21.15.190:3300/0 conn(0x7f641815cb10 0x7f6418070c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:22:59.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:59.227+0000 7f6420800640 1 --2- 172.21.15.6:0/235622961 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f64181592a0 0x7f641815c5d0 secure :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f6418071750 tx=0x7f640802a100 comp rx=0 tx=0).stop 2024-04-24T23:22:59.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:59.227+0000 7f6420800640 1 --2- 172.21.15.6:0/235622961 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6418152d80 0x7f64180c8d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:22:59.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:59.227+0000 7f6420800640 1 -- 172.21.15.6:0/235622961 >> 172.21.15.6:0/235622961 conn(0x7f6418070280 msgr2=0x7f641806f320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:22:59.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:59.231+0000 7f6420800640 1 -- 172.21.15.6:0/235622961 shutdown_connections 2024-04-24T23:22:59.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:59.231+0000 7f6420800640 1 -- 172.21.15.6:0/235622961 wait complete. 2024-04-24T23:22:59.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:59.234+0000 7f6420800640 1 Processor -- start 2024-04-24T23:22:59.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:59.238+0000 7f6420800640 1 -- start start 2024-04-24T23:22:59.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:59.243+0000 7f6420800640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6418152d80 0x7f64180c9a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:22:59.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:59.243+0000 7f6420800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f641815cb10 0x7f64180c9f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:22:59.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:59.244+0000 7f6420800640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f64181e48b0 0x7f64180c61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:22:59.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:59.244+0000 7f641e800640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f64181e48b0 0x7f64180c61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:22:59.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:59.244+0000 7f6420800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f641815bdd0 con 0x7f641815cb10 2024-04-24T23:22:59.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:59.244+0000 7f6420800640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f64180c3a30 con 0x7f64181e48b0 2024-04-24T23:22:59.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:59.244+0000 7f6420800640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f6418159cd0 con 0x7f6418152d80 2024-04-24T23:22:59.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:59.247+0000 7f641d400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f641815cb10 0x7f64180c9f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:22:59.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:59.244+0000 7f641e800640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f64181e48b0 0x7f64180c61e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:60280/0 (socket says 172.21.15.6:60280) 2024-04-24T23:22:59.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:59.248+0000 7f641e800640 1 -- 172.21.15.6:0/3832137845 learned_addr learned my addr 172.21.15.6:0/3832137845 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:22:59.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:59.248+0000 7f641de00640 1 --2- 172.21.15.6:0/3832137845 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6418152d80 0x7f64180c9a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:22:59.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:59.249+0000 7f641de00640 1 -- 172.21.15.6:0/3832137845 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f64181e48b0 msgr2=0x7f64180c61e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:22:59.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:59.249+0000 7f641de00640 1 --2- 172.21.15.6:0/3832137845 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f64181e48b0 0x7f64180c61e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:22:59.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:59.249+0000 7f641de00640 1 -- 172.21.15.6:0/3832137845 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f641815cb10 msgr2=0x7f64180c9f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:22:59.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:59.249+0000 7f641e800640 1 --2- 172.21.15.6:0/3832137845 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f64181e48b0 0x7f64180c61e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:22:59.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:59.253+0000 7f641de00640 1 --2- 172.21.15.6:0/3832137845 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f641815cb10 0x7f64180c9f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:22:59.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:59.253+0000 7f641de00640 1 -- 172.21.15.6:0/3832137845 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64181e4490 con 0x7f6418152d80 2024-04-24T23:22:59.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:59.253+0000 7f641d400640 1 --2- 172.21.15.6:0/3832137845 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f641815cb10 0x7f64180c9f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:22:59.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:59.255+0000 7f641de00640 1 --2- 172.21.15.6:0/3832137845 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6418152d80 0x7f64180c9a20 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f640c00bda0 tx=0x7f640c009490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:22:59.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:59.257+0000 7f6412000640 1 -- 172.21.15.6:0/3832137845 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f640c004230 con 0x7f6418152d80 2024-04-24T23:22:59.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:59.258+0000 7f6420800640 1 -- 172.21.15.6:0/3832137845 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f641814fb10 con 0x7f6418152d80 2024-04-24T23:22:59.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:59.258+0000 7f6420800640 1 -- 172.21.15.6:0/3832137845 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f64181ef490 con 0x7f6418152d80 2024-04-24T23:22:59.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:59.261+0000 7f6412000640 1 -- 172.21.15.6:0/3832137845 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f640c0043d0 con 0x7f6418152d80 2024-04-24T23:22:59.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:59.261+0000 7f6412000640 1 -- 172.21.15.6:0/3832137845 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f640c004230 con 0x7f6418152d80 2024-04-24T23:22:59.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:59.264+0000 7f6420800640 1 -- 172.21.15.6:0/3832137845 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f63f4005290 con 0x7f6418152d80 2024-04-24T23:22:59.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:59.267+0000 7f6412000640 1 -- 172.21.15.6:0/3832137845 <== mon.2 v2:172.21.15.190:3301/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f640c014960 con 0x7f6418152d80 2024-04-24T23:22:59.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:59.269+0000 7f6412000640 1 --2- 172.21.15.6:0/3832137845 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f63fc06c060 0x7f63fc06e520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:22:59.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:59.269+0000 7f6412000640 1 -- 172.21.15.6:0/3832137845 <== mon.2 v2:172.21.15.190:3301/0 5 ==== osd_map(28..28 src has 1..28) ==== 6762+0+0 (secure 0 0 0) 0x7f640c092ff0 con 0x7f6418152d80 2024-04-24T23:22:59.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:59.277+0000 7f641d400640 1 --2- 172.21.15.6:0/3832137845 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f63fc06c060 0x7f63fc06e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:22:59.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:59.278+0000 7f641d400640 1 --2- 172.21.15.6:0/3832137845 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f63fc06c060 0x7f63fc06e520 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f6408034000 tx=0x7f640802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:22:59.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:59.284+0000 7f6412000640 1 -- 172.21.15.6:0/3832137845 <== mon.2 v2:172.21.15.190:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f640c01a070 con 0x7f6418152d80 2024-04-24T23:22:59.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:59.864+0000 7f6420800640 1 -- 172.21.15.6:0/3832137845 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f63f4005580 con 0x7f6418152d80 2024-04-24T23:22:59.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:59.865+0000 7f6412000640 1 -- 172.21.15.6:0/3832137845 <== mon.2 v2:172.21.15.190:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+6885 (secure 0 0 0) 0x7f640c01a070 con 0x7f6418152d80 2024-04-24T23:22:59.871 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:22:59.871 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5081,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/1190759626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1190759626},{"type":"v1","addr":"172.21.15.6:6834","nonce":1190759626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":5503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/3928731059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":3928731059},{"type":"v1","addr":"172.21.15.6:6837","nonce":3928731059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:22:37.858824+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1,2,3],"up":{"mds_0":4306,"mds_1":4282,"mds_2":4235,"mds_3":4232},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_4232":{"gid":4232,"name":"b","rank":3,"incarnation":15,"state":"up:stopping","state_seq":15,"addr":"172.21.15.190:6838/2032280158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2032280158},{"type":"v1","addr":"172.21.15.190:6838","nonce":2032280158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4235":{"gid":4235,"name":"d","rank":2,"incarnation":13,"state":"up:active","state_seq":14,"addr":"172.21.15.190:6839/1572872451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":1572872451},{"type":"v1","addr":"172.21.15.190:6839","nonce":1572872451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4282":{"gid":4282,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":12,"addr":"172.21.15.190:6837/580315269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":580315269},{"type":"v1","addr":"172.21.15.190:6837","nonce":580315269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/489092393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":489092393},{"type":"v1","addr":"172.21.15.6:6835","nonce":489092393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4306,"qdb_cluster":[4306,4282,4235]},"id":1}]} 2024-04-24T23:22:59.871 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 35 2024-04-24T23:22:59.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:59.872+0000 7f6420800640 1 -- 172.21.15.6:0/3832137845 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f63fc06c060 msgr2=0x7f63fc06e520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:22:59.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:59.872+0000 7f6420800640 1 --2- 172.21.15.6:0/3832137845 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f63fc06c060 0x7f63fc06e520 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f6408034000 tx=0x7f640802e000 comp rx=0 tx=0).stop 2024-04-24T23:22:59.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:59.872+0000 7f6420800640 1 -- 172.21.15.6:0/3832137845 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6418152d80 msgr2=0x7f64180c9a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:22:59.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:59.872+0000 7f6420800640 1 --2- 172.21.15.6:0/3832137845 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6418152d80 0x7f64180c9a20 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f640c00bda0 tx=0x7f640c009490 comp rx=0 tx=0).stop 2024-04-24T23:22:59.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:59.875+0000 7f6420800640 1 -- 172.21.15.6:0/3832137845 shutdown_connections 2024-04-24T23:22:59.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:59.875+0000 7f6420800640 1 --2- 172.21.15.6:0/3832137845 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f63fc06c060 0x7f63fc06e520 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:22:59.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:59.875+0000 7f6420800640 1 --2- 172.21.15.6:0/3832137845 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f64181e48b0 0x7f64180c61e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:22:59.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:59.875+0000 7f6420800640 1 --2- 172.21.15.6:0/3832137845 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f641815cb10 0x7f64180c9f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:22:59.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:59.875+0000 7f6420800640 1 --2- 172.21.15.6:0/3832137845 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6418152d80 0x7f64180c9a20 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:22:59.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:59.875+0000 7f6420800640 1 -- 172.21.15.6:0/3832137845 >> 172.21.15.6:0/3832137845 conn(0x7f6418070280 msgr2=0x7f6418158a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:22:59.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:59.876+0000 7f6420800640 1 -- 172.21.15.6:0/3832137845 shutdown_connections 2024-04-24T23:22:59.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:22:59.876+0000 7f6420800640 1 -- 172.21.15.6:0/3832137845 wait complete. 2024-04-24T23:22:59.913 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-24T23:23:01.914 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:23:02.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:02.246+0000 7f2457e00640 1 Processor -- start 2024-04-24T23:23:02.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:02.247+0000 7f2457e00640 1 -- start start 2024-04-24T23:23:02.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:02.248+0000 7f2457e00640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f24580cc870 0x7f245816b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:23:02.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:02.248+0000 7f2457e00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2458164a90 0x7f2458164f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:23:02.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:02.249+0000 7f2457e00640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2458072370 con 0x7f24580ccc50 2024-04-24T23:23:02.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:02.249+0000 7f2457e00640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f245816bb30 con 0x7f24580cc870 2024-04-24T23:23:02.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:02.249+0000 7f2457e00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f2458070e70 con 0x7f2458164a90 2024-04-24T23:23:02.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:02.252+0000 7f2456a00640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f24580cc870 0x7f245816b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:23:02.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:02.252+0000 7f2456a00640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f24580cc870 0x7f245816b5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:60286/0 (socket says 172.21.15.6:60286) 2024-04-24T23:23:02.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:02.252+0000 7f2456a00640 1 -- 172.21.15.6:0/3628648566 learned_addr learned my addr 172.21.15.6:0/3628648566 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:23:02.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:02.253+0000 7f2457400640 1 --2- 172.21.15.6:0/3628648566 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2458164a90 0x7f2458164f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:23:02.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:02.254+0000 7f2455600640 1 -- 172.21.15.6:0/3628648566 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2583528563 0 0) 0x7f2458072370 con 0x7f24580ccc50 2024-04-24T23:23:02.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:02.359+0000 7f2455600640 1 -- 172.21.15.6:0/3628648566 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f243c003520 con 0x7f24580ccc50 2024-04-24T23:23:02.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:02.360+0000 7f2457400640 1 -- 172.21.15.6:0/3628648566 >> v2:172.21.15.190:3300/0 conn(0x7f24580cc870 msgr2=0x7f245816b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:23:02.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:02.360+0000 7f2457400640 1 --2- 172.21.15.6:0/3628648566 >> v2:172.21.15.190:3300/0 conn(0x7f24580cc870 0x7f245816b5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:02.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:02.360+0000 7f2457400640 1 -- 172.21.15.6:0/3628648566 >> v1:172.21.15.190:6789/0 conn(0x7f24580ccc50 legacy=0x7f245806ddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:23:02.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:02.360+0000 7f2457400640 1 -- 172.21.15.6:0/3628648566 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2458072650 con 0x7f2458164a90 2024-04-24T23:23:02.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:02.366+0000 7f2457400640 1 --2- 172.21.15.6:0/3628648566 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2458164a90 0x7f2458164f00 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f2450007b50 tx=0x7f245002ac10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3f31d141427bc58e server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:23:02.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:02.366+0000 7f2455600640 1 -- 172.21.15.6:0/3628648566 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2450038070 con 0x7f2458164a90 2024-04-24T23:23:02.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:02.366+0000 7f2455600640 1 -- 172.21.15.6:0/3628648566 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f245002b960 con 0x7f2458164a90 2024-04-24T23:23:02.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:02.366+0000 7f2455600640 1 -- 172.21.15.6:0/3628648566 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f245002bd30 con 0x7f2458164a90 2024-04-24T23:23:02.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:02.370+0000 7f2457e00640 1 -- 172.21.15.6:0/3628648566 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2458164a90 msgr2=0x7f2458164f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:23:02.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:02.370+0000 7f2457e00640 1 --2- 172.21.15.6:0/3628648566 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2458164a90 0x7f2458164f00 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f2450007b50 tx=0x7f245002ac10 comp rx=0 tx=0).stop 2024-04-24T23:23:02.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:02.370+0000 7f2457e00640 1 -- 172.21.15.6:0/3628648566 shutdown_connections 2024-04-24T23:23:02.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:02.370+0000 7f2457e00640 1 --2- 172.21.15.6:0/3628648566 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2458164a90 0x7f2458164f00 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:02.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:02.370+0000 7f2457e00640 1 --2- 172.21.15.6:0/3628648566 >> v2:172.21.15.190:3300/0 conn(0x7f24580cc870 0x7f245816b5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:02.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:02.370+0000 7f2457e00640 1 -- 172.21.15.6:0/3628648566 >> 172.21.15.6:0/3628648566 conn(0x7f245806ed80 msgr2=0x7f245806f190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:23:02.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:02.373+0000 7f2457e00640 1 -- 172.21.15.6:0/3628648566 shutdown_connections 2024-04-24T23:23:02.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:02.373+0000 7f2457e00640 1 -- 172.21.15.6:0/3628648566 wait complete. 2024-04-24T23:23:02.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:02.388+0000 7f2457e00640 1 Processor -- start 2024-04-24T23:23:02.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:02.388+0000 7f2457e00640 1 -- start start 2024-04-24T23:23:02.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:02.398+0000 7f2457e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f24580cc870 0x7f2458160500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:23:02.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:02.398+0000 7f2457e00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f24580ccc50 0x7f24580c6020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:23:02.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:02.398+0000 7f2457e00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2458164a90 0x7f2458169a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:23:02.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:02.398+0000 7f2457e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f24580713a0 con 0x7f24580cc870 2024-04-24T23:23:02.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:02.398+0000 7f2457e00640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f24580c3ba0 con 0x7f24580ccc50 2024-04-24T23:23:02.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:02.398+0000 7f2457e00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f2458167ac0 con 0x7f2458164a90 2024-04-24T23:23:02.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:02.398+0000 7f2456000640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f24580ccc50 0x7f24580c6020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:23:02.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:02.399+0000 7f2456000640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f24580ccc50 0x7f24580c6020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:60294/0 (socket says 172.21.15.6:60294) 2024-04-24T23:23:02.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:02.399+0000 7f2456000640 1 -- 172.21.15.6:0/2719789728 learned_addr learned my addr 172.21.15.6:0/2719789728 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:23:02.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:02.399+0000 7f2457400640 1 --2- 172.21.15.6:0/2719789728 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2458164a90 0x7f2458169a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:23:02.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:02.399+0000 7f2456a00640 1 --2- 172.21.15.6:0/2719789728 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f24580cc870 0x7f2458160500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:23:02.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:02.403+0000 7f2456a00640 1 -- 172.21.15.6:0/2719789728 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2458164a90 msgr2=0x7f2458169a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:23:02.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:02.403+0000 7f2456a00640 1 --2- 172.21.15.6:0/2719789728 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2458164a90 0x7f2458169a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:02.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:02.403+0000 7f2456a00640 1 -- 172.21.15.6:0/2719789728 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f24580ccc50 msgr2=0x7f24580c6020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:23:02.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:02.403+0000 7f2456a00640 1 --2- 172.21.15.6:0/2719789728 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f24580ccc50 0x7f24580c6020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:02.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:02.403+0000 7f2456a00640 1 -- 172.21.15.6:0/2719789728 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24582050b0 con 0x7f24580cc870 2024-04-24T23:23:02.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:02.403+0000 7f2456000640 1 --2- 172.21.15.6:0/2719789728 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f24580ccc50 0x7f24580c6020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:23:02.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:02.405+0000 7f2456a00640 1 --2- 172.21.15.6:0/2719789728 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f24580cc870 0x7f2458160500 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f244800da80 tx=0x7f244800df40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:23:02.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:02.409+0000 7f244ea00640 1 -- 172.21.15.6:0/2719789728 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f244800a990 con 0x7f24580cc870 2024-04-24T23:23:02.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:02.410+0000 7f244ea00640 1 -- 172.21.15.6:0/2719789728 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f244800ab30 con 0x7f24580cc870 2024-04-24T23:23:02.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:02.410+0000 7f244ea00640 1 -- 172.21.15.6:0/2719789728 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2448017420 con 0x7f24580cc870 2024-04-24T23:23:02.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:02.410+0000 7f2457e00640 1 -- 172.21.15.6:0/2719789728 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2458205320 con 0x7f24580cc870 2024-04-24T23:23:02.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:02.410+0000 7f2457e00640 1 -- 172.21.15.6:0/2719789728 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24580c45c0 con 0x7f24580cc870 2024-04-24T23:23:02.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:02.418+0000 7f2457e00640 1 -- 172.21.15.6:0/2719789728 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2430005290 con 0x7f24580cc870 2024-04-24T23:23:02.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:02.421+0000 7f244ea00640 1 -- 172.21.15.6:0/2719789728 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f2448020440 con 0x7f24580cc870 2024-04-24T23:23:02.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:02.423+0000 7f244ea00640 1 --2- 172.21.15.6:0/2719789728 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f243806bf00 0x7f243806e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:23:02.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:02.423+0000 7f244ea00640 1 -- 172.21.15.6:0/2719789728 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6762+0+0 (secure 0 0 0) 0x7f244808af70 con 0x7f24580cc870 2024-04-24T23:23:02.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:02.424+0000 7f2456000640 1 --2- 172.21.15.6:0/2719789728 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f243806bf00 0x7f243806e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:23:02.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:02.427+0000 7f2456000640 1 --2- 172.21.15.6:0/2719789728 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f243806bf00 0x7f243806e3c0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f24581fbab0 tx=0x7f2444009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:23:02.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:02.433+0000 7f244ea00640 1 -- 172.21.15.6:0/2719789728 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f2448012070 con 0x7f24580cc870 2024-04-24T23:23:03.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:03.070+0000 7f2457e00640 1 -- 172.21.15.6:0/2719789728 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2430005580 con 0x7f24580cc870 2024-04-24T23:23:03.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:03.072+0000 7f244ea00640 1 -- 172.21.15.6:0/2719789728 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+6885 (secure 0 0 0) 0x7f2448059570 con 0x7f24580cc870 2024-04-24T23:23:03.076 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:23:03.076 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5081,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/1190759626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1190759626},{"type":"v1","addr":"172.21.15.6:6834","nonce":1190759626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":5503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/3928731059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":3928731059},{"type":"v1","addr":"172.21.15.6:6837","nonce":3928731059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:22:37.858824+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1,2,3],"up":{"mds_0":4306,"mds_1":4282,"mds_2":4235,"mds_3":4232},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_4232":{"gid":4232,"name":"b","rank":3,"incarnation":15,"state":"up:stopping","state_seq":15,"addr":"172.21.15.190:6838/2032280158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2032280158},{"type":"v1","addr":"172.21.15.190:6838","nonce":2032280158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4235":{"gid":4235,"name":"d","rank":2,"incarnation":13,"state":"up:active","state_seq":14,"addr":"172.21.15.190:6839/1572872451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":1572872451},{"type":"v1","addr":"172.21.15.190:6839","nonce":1572872451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4282":{"gid":4282,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":12,"addr":"172.21.15.190:6837/580315269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":580315269},{"type":"v1","addr":"172.21.15.190:6837","nonce":580315269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/489092393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":489092393},{"type":"v1","addr":"172.21.15.6:6835","nonce":489092393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4306,"qdb_cluster":[4306,4282,4235]},"id":1}]} 2024-04-24T23:23:03.076 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 35 2024-04-24T23:23:03.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:03.079+0000 7f2457e00640 1 -- 172.21.15.6:0/2719789728 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f243806bf00 msgr2=0x7f243806e3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:23:03.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:03.079+0000 7f2457e00640 1 --2- 172.21.15.6:0/2719789728 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f243806bf00 0x7f243806e3c0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f24581fbab0 tx=0x7f2444009040 comp rx=0 tx=0).stop 2024-04-24T23:23:03.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:03.079+0000 7f2457e00640 1 -- 172.21.15.6:0/2719789728 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f24580cc870 msgr2=0x7f2458160500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:23:03.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:03.079+0000 7f2457e00640 1 --2- 172.21.15.6:0/2719789728 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f24580cc870 0x7f2458160500 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f244800da80 tx=0x7f244800df40 comp rx=0 tx=0).stop 2024-04-24T23:23:03.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:03.080+0000 7f2457e00640 1 -- 172.21.15.6:0/2719789728 shutdown_connections 2024-04-24T23:23:03.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:03.080+0000 7f2457e00640 1 --2- 172.21.15.6:0/2719789728 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f243806bf00 0x7f243806e3c0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:03.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:03.080+0000 7f2457e00640 1 --2- 172.21.15.6:0/2719789728 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2458164a90 0x7f2458169a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:03.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:03.080+0000 7f2457e00640 1 --2- 172.21.15.6:0/2719789728 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f24580ccc50 0x7f24580c6020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:03.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:03.080+0000 7f2457e00640 1 --2- 172.21.15.6:0/2719789728 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f24580cc870 0x7f2458160500 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:03.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:03.080+0000 7f2457e00640 1 -- 172.21.15.6:0/2719789728 >> 172.21.15.6:0/2719789728 conn(0x7f245806ed80 msgr2=0x7f245816b3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:23:03.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:03.082+0000 7f2457e00640 1 -- 172.21.15.6:0/2719789728 shutdown_connections 2024-04-24T23:23:03.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:03.082+0000 7f2457e00640 1 -- 172.21.15.6:0/2719789728 wait complete. 2024-04-24T23:23:03.113 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-24T23:23:05.115 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:23:05.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:05.409+0000 7f8fc8a00640 1 Processor -- start 2024-04-24T23:23:05.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:05.409+0000 7f8fc8a00640 1 -- start start 2024-04-24T23:23:05.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:05.412+0000 7f8fc8a00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8fc40c7dc0 0x7f8fc41623a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:23:05.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:05.412+0000 7f8fc8a00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8fc40c7230 con 0x7f8fc4159110 2024-04-24T23:23:05.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:05.412+0000 7f8fc8a00640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8fc40c74d0 con 0x7f8fc40c81c0 2024-04-24T23:23:05.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:05.412+0000 7f8fc8a00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f8fc41594f0 con 0x7f8fc40c7dc0 2024-04-24T23:23:05.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:05.413+0000 7f8fc1e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f8fc4159110 0x7f8fc4072420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:49258/0 (socket says 172.21.15.6:49258) 2024-04-24T23:23:05.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:05.413+0000 7f8fc1e00640 1 -- 172.21.15.6:0/3769126814 learned_addr learned my addr 172.21.15.6:0/3769126814 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:23:05.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:05.413+0000 7f8fc1400640 1 --2- 172.21.15.6:0/3769126814 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8fc40c7dc0 0x7f8fc41623a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:23:05.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:05.414+0000 7f8fc0a00640 1 -- 172.21.15.6:0/3769126814 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2669212574 0 0) 0x7f8fc40c7230 con 0x7f8fc4159110 2024-04-24T23:23:05.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:05.414+0000 7f8fc0a00640 1 -- 172.21.15.6:0/3769126814 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8fa8003520 con 0x7f8fc4159110 2024-04-24T23:23:05.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:05.414+0000 7f8fc0a00640 1 -- 172.21.15.6:0/3769126814 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 846717427 0 0) 0x7f8fc40c74d0 con 0x7f8fc40c81c0 2024-04-24T23:23:05.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:05.414+0000 7f8fc0a00640 1 -- 172.21.15.6:0/3769126814 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8fc40c7230 con 0x7f8fc40c81c0 2024-04-24T23:23:05.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:05.414+0000 7f8fc1400640 1 -- 172.21.15.6:0/3769126814 >> v1:172.21.15.190:6789/0 conn(0x7f8fc40c81c0 legacy=0x7f8fc41628e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:23:05.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:05.414+0000 7f8fc1400640 1 -- 172.21.15.6:0/3769126814 >> v1:172.21.15.6:6789/0 conn(0x7f8fc4159110 legacy=0x7f8fc4072420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:23:05.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:05.414+0000 7f8fc0a00640 1 -- 172.21.15.6:0/3769126814 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4235734051 0 0) 0x7f8fa8003520 con 0x7f8fc4159110 2024-04-24T23:23:05.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:05.415+0000 7f8fc1400640 1 -- 172.21.15.6:0/3769126814 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fc406e440 con 0x7f8fc40c7dc0 2024-04-24T23:23:05.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:05.419+0000 7f8fc1400640 1 --2- 172.21.15.6:0/3769126814 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8fc40c7dc0 0x7f8fc41623a0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f8fb00097d0 tx=0x7f8fb002a230 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e5fbc031ce22a0d9 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:23:05.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:05.419+0000 7f8fc0a00640 1 -- 172.21.15.6:0/3769126814 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fb0037070 con 0x7f8fc40c7dc0 2024-04-24T23:23:05.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:05.420+0000 7f8fc0a00640 1 -- 172.21.15.6:0/3769126814 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8fb002acc0 con 0x7f8fc40c7dc0 2024-04-24T23:23:05.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:05.420+0000 7f8fc0a00640 1 -- 172.21.15.6:0/3769126814 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fb4001690 con 0x7f8fc40c7dc0 2024-04-24T23:23:05.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:05.422+0000 7f8fc8a00640 1 -- 172.21.15.6:0/3769126814 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8fc40c7dc0 msgr2=0x7f8fc41623a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:23:05.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:05.422+0000 7f8fc8a00640 1 --2- 172.21.15.6:0/3769126814 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8fc40c7dc0 0x7f8fc41623a0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f8fb00097d0 tx=0x7f8fb002a230 comp rx=0 tx=0).stop 2024-04-24T23:23:05.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:05.423+0000 7f8fc8a00640 1 -- 172.21.15.6:0/3769126814 shutdown_connections 2024-04-24T23:23:05.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:05.423+0000 7f8fc8a00640 1 --2- 172.21.15.6:0/3769126814 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8fc40c7dc0 0x7f8fc41623a0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:05.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:05.423+0000 7f8fc8a00640 1 -- 172.21.15.6:0/3769126814 >> 172.21.15.6:0/3769126814 conn(0x7f8fc406b0e0 msgr2=0x7f8fc406b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:23:05.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:05.433+0000 7f8fc8a00640 1 -- 172.21.15.6:0/3769126814 shutdown_connections 2024-04-24T23:23:05.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:05.433+0000 7f8fc8a00640 1 -- 172.21.15.6:0/3769126814 wait complete. 2024-04-24T23:23:05.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:05.437+0000 7f8fc8a00640 1 Processor -- start 2024-04-24T23:23:05.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:05.446+0000 7f8fc8a00640 1 -- start start 2024-04-24T23:23:05.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:05.448+0000 7f8fc8a00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8fc40c7dc0 0x7f8fc415bc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:23:05.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:05.448+0000 7f8fc8a00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8fc40c81c0 0x7f8fc41e8980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:23:05.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:05.449+0000 7f8fc8a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8fc4159110 0x7f8fc41e8ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:23:05.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:05.449+0000 7f8fc1e00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8fc40c7dc0 0x7f8fc415bc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:23:05.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:05.449+0000 7f8fc1e00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8fc40c7dc0 0x7f8fc415bc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.6:41266/0 (socket says 172.21.15.6:41266) 2024-04-24T23:23:05.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:05.449+0000 7f8fc8a00640 1 -- 172.21.15.6:0/1329604220 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f8fc406ece0 con 0x7f8fc4159110 2024-04-24T23:23:05.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:05.449+0000 7f8fc8a00640 1 -- 172.21.15.6:0/1329604220 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f8fc406ea00 con 0x7f8fc40c81c0 2024-04-24T23:23:05.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:05.449+0000 7f8fc8a00640 1 -- 172.21.15.6:0/1329604220 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f8fc406f200 con 0x7f8fc40c7dc0 2024-04-24T23:23:05.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:05.449+0000 7f8fc1400640 1 --2- 172.21.15.6:0/1329604220 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8fc40c81c0 0x7f8fc41e8980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:23:05.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:05.450+0000 7f8fc2800640 1 --2- 172.21.15.6:0/1329604220 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8fc4159110 0x7f8fc41e8ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:23:05.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:05.449+0000 7f8fc1e00640 1 -- 172.21.15.6:0/1329604220 learned_addr learned my addr 172.21.15.6:0/1329604220 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:23:05.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:05.451+0000 7f8fc2800640 1 -- 172.21.15.6:0/1329604220 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8fc40c7dc0 msgr2=0x7f8fc415bc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:23:05.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:05.451+0000 7f8fc2800640 1 --2- 172.21.15.6:0/1329604220 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8fc40c7dc0 0x7f8fc415bc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:05.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:05.451+0000 7f8fc2800640 1 -- 172.21.15.6:0/1329604220 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8fc40c81c0 msgr2=0x7f8fc41e8980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:23:05.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:05.451+0000 7f8fc2800640 1 --2- 172.21.15.6:0/1329604220 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8fc40c81c0 0x7f8fc41e8980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:05.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:05.451+0000 7f8fc2800640 1 -- 172.21.15.6:0/1329604220 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fc41563d0 con 0x7f8fc4159110 2024-04-24T23:23:05.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:05.457+0000 7f8fc2800640 1 --2- 172.21.15.6:0/1329604220 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8fc4159110 0x7f8fc41e8ec0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f8fbc00c730 tx=0x7f8fbc00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:23:05.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:05.458+0000 7f8fc1400640 1 --2- 172.21.15.6:0/1329604220 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8fc40c81c0 0x7f8fc41e8980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:23:05.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:05.458+0000 7f8fba000640 1 -- 172.21.15.6:0/1329604220 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fbc003b70 con 0x7f8fc4159110 2024-04-24T23:23:05.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:05.459+0000 7f8fc8a00640 1 -- 172.21.15.6:0/1329604220 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8fc40c62b0 con 0x7f8fc4159110 2024-04-24T23:23:05.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:05.459+0000 7f8fc8a00640 1 -- 172.21.15.6:0/1329604220 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8fc40c6550 con 0x7f8fc4159110 2024-04-24T23:23:05.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:05.461+0000 7f8fba000640 1 -- 172.21.15.6:0/1329604220 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8fbc003d10 con 0x7f8fc4159110 2024-04-24T23:23:05.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:05.461+0000 7f8fba000640 1 -- 172.21.15.6:0/1329604220 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fbc003b70 con 0x7f8fc4159110 2024-04-24T23:23:05.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:05.467+0000 7f8fc8a00640 1 -- 172.21.15.6:0/1329604220 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8f9c005290 con 0x7f8fc4159110 2024-04-24T23:23:05.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:05.481+0000 7f8fba000640 1 -- 172.21.15.6:0/1329604220 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f8fbc0050d0 con 0x7f8fc4159110 2024-04-24T23:23:05.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:05.484+0000 7f8fba000640 1 --2- 172.21.15.6:0/1329604220 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f8fa406c030 0x7f8fa406e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:23:05.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:05.484+0000 7f8fba000640 1 -- 172.21.15.6:0/1329604220 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6762+0+0 (secure 0 0 0) 0x7f8fbc08ad70 con 0x7f8fc4159110 2024-04-24T23:23:05.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:05.488+0000 7f8fc1e00640 1 --2- 172.21.15.6:0/1329604220 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f8fa406c030 0x7f8fa406e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:23:05.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:05.491+0000 7f8fc1e00640 1 --2- 172.21.15.6:0/1329604220 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f8fa406c030 0x7f8fa406e4f0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f8fb4001800 tx=0x7f8fb40016f0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:23:05.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:05.502+0000 7f8fba000640 1 -- 172.21.15.6:0/1329604220 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f8fbc059260 con 0x7f8fc4159110 2024-04-24T23:23:06.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:06.132+0000 7f8fc8a00640 1 -- 172.21.15.6:0/1329604220 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8f9c005580 con 0x7f8fc4159110 2024-04-24T23:23:06.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:06.133+0000 7f8fba000640 1 -- 172.21.15.6:0/1329604220 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+6885 (secure 0 0 0) 0x7f8fbc058b60 con 0x7f8fc4159110 2024-04-24T23:23:06.138 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:23:06.139 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5081,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/1190759626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1190759626},{"type":"v1","addr":"172.21.15.6:6834","nonce":1190759626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":5503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/3928731059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":3928731059},{"type":"v1","addr":"172.21.15.6:6837","nonce":3928731059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:22:37.858824+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1,2,3],"up":{"mds_0":4306,"mds_1":4282,"mds_2":4235,"mds_3":4232},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_4232":{"gid":4232,"name":"b","rank":3,"incarnation":15,"state":"up:stopping","state_seq":15,"addr":"172.21.15.190:6838/2032280158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2032280158},{"type":"v1","addr":"172.21.15.190:6838","nonce":2032280158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4235":{"gid":4235,"name":"d","rank":2,"incarnation":13,"state":"up:active","state_seq":14,"addr":"172.21.15.190:6839/1572872451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":1572872451},{"type":"v1","addr":"172.21.15.190:6839","nonce":1572872451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4282":{"gid":4282,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":12,"addr":"172.21.15.190:6837/580315269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":580315269},{"type":"v1","addr":"172.21.15.190:6837","nonce":580315269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/489092393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":489092393},{"type":"v1","addr":"172.21.15.6:6835","nonce":489092393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4306,"qdb_cluster":[4306,4282,4235]},"id":1}]} 2024-04-24T23:23:06.139 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 35 2024-04-24T23:23:06.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:06.140+0000 7f8fc8a00640 1 -- 172.21.15.6:0/1329604220 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f8fa406c030 msgr2=0x7f8fa406e4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:23:06.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:06.141+0000 7f8fc8a00640 1 --2- 172.21.15.6:0/1329604220 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f8fa406c030 0x7f8fa406e4f0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f8fb4001800 tx=0x7f8fb40016f0 comp rx=0 tx=0).stop 2024-04-24T23:23:06.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:06.141+0000 7f8fc8a00640 1 -- 172.21.15.6:0/1329604220 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8fc4159110 msgr2=0x7f8fc41e8ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:23:06.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:06.141+0000 7f8fc8a00640 1 --2- 172.21.15.6:0/1329604220 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8fc4159110 0x7f8fc41e8ec0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f8fbc00c730 tx=0x7f8fbc00cbf0 comp rx=0 tx=0).stop 2024-04-24T23:23:06.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:06.143+0000 7f8fc8a00640 1 -- 172.21.15.6:0/1329604220 shutdown_connections 2024-04-24T23:23:06.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:06.143+0000 7f8fc8a00640 1 --2- 172.21.15.6:0/1329604220 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f8fa406c030 0x7f8fa406e4f0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:06.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:06.143+0000 7f8fc8a00640 1 --2- 172.21.15.6:0/1329604220 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8fc4159110 0x7f8fc41e8ec0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:06.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:06.143+0000 7f8fc8a00640 1 --2- 172.21.15.6:0/1329604220 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8fc40c81c0 0x7f8fc41e8980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:06.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:06.143+0000 7f8fc8a00640 1 --2- 172.21.15.6:0/1329604220 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8fc40c7dc0 0x7f8fc415bc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:06.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:06.143+0000 7f8fc8a00640 1 -- 172.21.15.6:0/1329604220 >> 172.21.15.6:0/1329604220 conn(0x7f8fc406b0e0 msgr2=0x7f8fc4153cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:23:06.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:06.147+0000 7f8fc8a00640 1 -- 172.21.15.6:0/1329604220 shutdown_connections 2024-04-24T23:23:06.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:06.147+0000 7f8fc8a00640 1 -- 172.21.15.6:0/1329604220 wait complete. 2024-04-24T23:23:06.189 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-24T23:23:08.191 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:23:08.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:08.602+0000 7fa4fb000640 1 Processor -- start 2024-04-24T23:23:08.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:08.602+0000 7fa4fb000640 1 -- start start 2024-04-24T23:23:08.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:08.605+0000 7fa4fb000640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fa4f41518e0 0x7fa4f40c74c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:23:08.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:08.605+0000 7fa4fb000640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa4f415ff90 0x7fa4f4162380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:23:08.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:08.605+0000 7fa4fb000640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa4f4071d20 con 0x7fa4f4157680 2024-04-24T23:23:08.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:08.605+0000 7fa4fb000640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7fa4f40c5920 con 0x7fa4f41518e0 2024-04-24T23:23:08.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:08.605+0000 7fa4fb000640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fa4f40c6fe0 con 0x7fa4f415ff90 2024-04-24T23:23:08.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:08.605+0000 7fa4f3e00640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fa4f41518e0 0x7fa4f40c74c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:23:08.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:08.605+0000 7fa4f3e00640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fa4f41518e0 0x7fa4f40c74c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:41906/0 (socket says 172.21.15.6:41906) 2024-04-24T23:23:08.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:08.605+0000 7fa4f3400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fa4f4157680 0x7fa4f40cc240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:49268/0 (socket says 172.21.15.6:49268) 2024-04-24T23:23:08.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:08.605+0000 7fa4f3400640 1 -- 172.21.15.6:0/3509855210 learned_addr learned my addr 172.21.15.6:0/3509855210 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:23:08.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:08.606+0000 7fa4f9000640 1 --2- 172.21.15.6:0/3509855210 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa4f415ff90 0x7fa4f4162380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:23:08.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:08.606+0000 7fa4f3e00640 1 -- 172.21.15.6:0/3509855210 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa4f415ff90 msgr2=0x7fa4f4162380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:23:08.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:08.606+0000 7fa4f3e00640 1 --2- 172.21.15.6:0/3509855210 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa4f415ff90 0x7fa4f4162380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:08.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:08.606+0000 7fa4f3e00640 1 -- 172.21.15.6:0/3509855210 >> v1:172.21.15.6:6789/0 conn(0x7fa4f4157680 legacy=0x7fa4f40cc240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:23:08.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:08.606+0000 7fa4f3e00640 1 -- 172.21.15.6:0/3509855210 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4f40c5080 con 0x7fa4f41518e0 2024-04-24T23:23:08.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:08.608+0000 7fa4f3e00640 1 --2- 172.21.15.6:0/3509855210 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa4f41518e0 0x7fa4f40c74c0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fa4e80097a0 tx=0x7fa4e802c390 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6a307bd98165ce8a server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:23:08.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:08.609+0000 7fa4f2a00640 1 -- 172.21.15.6:0/3509855210 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4e8039070 con 0x7fa4f41518e0 2024-04-24T23:23:08.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:08.609+0000 7fa4f2a00640 1 -- 172.21.15.6:0/3509855210 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa4e8034640 con 0x7fa4f41518e0 2024-04-24T23:23:08.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:08.609+0000 7fa4f2a00640 1 -- 172.21.15.6:0/3509855210 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4e8034a10 con 0x7fa4f41518e0 2024-04-24T23:23:08.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:08.612+0000 7fa4fb000640 1 -- 172.21.15.6:0/3509855210 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa4f41518e0 msgr2=0x7fa4f40c74c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:23:08.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:08.612+0000 7fa4fb000640 1 --2- 172.21.15.6:0/3509855210 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa4f41518e0 0x7fa4f40c74c0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fa4e80097a0 tx=0x7fa4e802c390 comp rx=0 tx=0).stop 2024-04-24T23:23:08.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:08.613+0000 7fa4fb000640 1 -- 172.21.15.6:0/3509855210 shutdown_connections 2024-04-24T23:23:08.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:08.614+0000 7fa4fb000640 1 --2- 172.21.15.6:0/3509855210 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa4f415ff90 0x7fa4f4162380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:08.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:08.614+0000 7fa4fb000640 1 --2- 172.21.15.6:0/3509855210 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa4f41518e0 0x7fa4f40c74c0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:08.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:08.614+0000 7fa4fb000640 1 -- 172.21.15.6:0/3509855210 >> 172.21.15.6:0/3509855210 conn(0x7fa4f406ec20 msgr2=0x7fa4f406f030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:23:08.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:08.617+0000 7fa4fb000640 1 -- 172.21.15.6:0/3509855210 shutdown_connections 2024-04-24T23:23:08.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:08.618+0000 7fa4fb000640 1 -- 172.21.15.6:0/3509855210 wait complete. 2024-04-24T23:23:08.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:08.620+0000 7fa4fb000640 1 Processor -- start 2024-04-24T23:23:08.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:08.621+0000 7fa4fb000640 1 -- start start 2024-04-24T23:23:08.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:08.622+0000 7fa4fb000640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa4f41518e0 0x7fa4f415b200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:23:08.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:08.622+0000 7fa4fb000640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa4f4157680 0x7fa4f414e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:23:08.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:08.622+0000 7fa4fb000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa4f415ff90 0x7fa4f41f7fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:23:08.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:08.623+0000 7fa4f9000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa4f415ff90 0x7fa4f41f7fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:23:08.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:08.623+0000 7fa4f9000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa4f415ff90 0x7fa4f41f7fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:46922/0 (socket says 172.21.15.6:46922) 2024-04-24T23:23:08.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:08.623+0000 7fa4f3400640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa4f4157680 0x7fa4f414e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:23:08.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:08.624+0000 7fa4f3400640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa4f4157680 0x7fa4f414e1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.6:41278/0 (socket says 172.21.15.6:41278) 2024-04-24T23:23:08.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:08.624+0000 7fa4f9000640 1 -- 172.21.15.6:0/3169441439 learned_addr learned my addr 172.21.15.6:0/3169441439 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:23:08.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:08.624+0000 7fa4fb000640 1 -- 172.21.15.6:0/3169441439 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fa4f4151cc0 con 0x7fa4f415ff90 2024-04-24T23:23:08.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:08.624+0000 7fa4fb000640 1 -- 172.21.15.6:0/3169441439 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fa4f40c5640 con 0x7fa4f41518e0 2024-04-24T23:23:08.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:08.624+0000 7fa4fb000640 1 -- 172.21.15.6:0/3169441439 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fa4f4151e40 con 0x7fa4f4157680 2024-04-24T23:23:08.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:08.624+0000 7fa4f3e00640 1 --2- 172.21.15.6:0/3169441439 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa4f41518e0 0x7fa4f415b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:23:08.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:08.624+0000 7fa4f3e00640 1 -- 172.21.15.6:0/3169441439 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa4f4157680 msgr2=0x7fa4f414e1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:23:08.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:08.624+0000 7fa4f3e00640 1 --2- 172.21.15.6:0/3169441439 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa4f4157680 0x7fa4f414e1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:08.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:08.625+0000 7fa4f3e00640 1 -- 172.21.15.6:0/3169441439 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa4f415ff90 msgr2=0x7fa4f41f7fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:23:08.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:08.625+0000 7fa4f3e00640 1 --2- 172.21.15.6:0/3169441439 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa4f415ff90 0x7fa4f41f7fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:08.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:08.625+0000 7fa4f3e00640 1 -- 172.21.15.6:0/3169441439 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4f415a760 con 0x7fa4f41518e0 2024-04-24T23:23:08.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:08.625+0000 7fa4f9000640 1 --2- 172.21.15.6:0/3169441439 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa4f415ff90 0x7fa4f41f7fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:23:08.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:08.625+0000 7fa4f3e00640 1 --2- 172.21.15.6:0/3169441439 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa4f41518e0 0x7fa4f415b200 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fa4e8033000 tx=0x7fa4e80046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:23:08.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:08.626+0000 7fa4e7e00640 1 -- 172.21.15.6:0/3169441439 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4e8039040 con 0x7fa4f41518e0 2024-04-24T23:23:08.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:08.626+0000 7fa4e7e00640 1 -- 172.21.15.6:0/3169441439 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa4e8042070 con 0x7fa4f41518e0 2024-04-24T23:23:08.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:08.626+0000 7fa4e7e00640 1 -- 172.21.15.6:0/3169441439 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4e8032910 con 0x7fa4f41518e0 2024-04-24T23:23:08.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:08.627+0000 7fa4fb000640 1 -- 172.21.15.6:0/3169441439 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4f415b740 con 0x7fa4f41518e0 2024-04-24T23:23:08.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:08.628+0000 7fa4fb000640 1 -- 172.21.15.6:0/3169441439 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4f41f8b60 con 0x7fa4f41518e0 2024-04-24T23:23:08.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:08.630+0000 7fa4fb000640 1 -- 172.21.15.6:0/3169441439 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa4f40c3500 con 0x7fa4f41518e0 2024-04-24T23:23:08.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:08.634+0000 7fa4e7e00640 1 -- 172.21.15.6:0/3169441439 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fa4e8032ab0 con 0x7fa4f41518e0 2024-04-24T23:23:08.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:08.635+0000 7fa4e7e00640 1 --2- 172.21.15.6:0/3169441439 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fa4d406bf00 0x7fa4d406e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:23:08.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:08.636+0000 7fa4e7e00640 1 -- 172.21.15.6:0/3169441439 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6762+0+0 (secure 0 0 0) 0x7fa4e80ab390 con 0x7fa4f41518e0 2024-04-24T23:23:08.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:08.636+0000 7fa4f3400640 1 --2- 172.21.15.6:0/3169441439 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fa4d406bf00 0x7fa4d406e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:23:08.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:08.638+0000 7fa4e7e00640 1 -- 172.21.15.6:0/3169441439 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fa4f40c3500 con 0x7fa4f41518e0 2024-04-24T23:23:08.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:08.639+0000 7fa4f3400640 1 --2- 172.21.15.6:0/3169441439 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fa4d406bf00 0x7fa4d406e3c0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fa4f40cbf10 tx=0x7fa4e0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:23:09.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:09.160+0000 7fa4fb000640 1 -- 172.21.15.6:0/3169441439 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa4f4075b30 con 0x7fa4f41518e0 2024-04-24T23:23:09.174 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:23:09.174 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5081,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/1190759626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1190759626},{"type":"v1","addr":"172.21.15.6:6834","nonce":1190759626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":5503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/3928731059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":3928731059},{"type":"v1","addr":"172.21.15.6:6837","nonce":3928731059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:23:08.146813+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1,2],"up":{"mds_0":4306,"mds_1":4282,"mds_2":4235},"failed":[],"damaged":[],"stopped":[3,4],"info":{"gid_4235":{"gid":4235,"name":"d","rank":2,"incarnation":13,"state":"up:stopping","state_seq":14,"addr":"172.21.15.190:6839/1572872451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":1572872451},{"type":"v1","addr":"172.21.15.190:6839","nonce":1572872451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4282":{"gid":4282,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":12,"addr":"172.21.15.190:6837/580315269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":580315269},{"type":"v1","addr":"172.21.15.190:6837","nonce":580315269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/489092393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":489092393},{"type":"v1","addr":"172.21.15.6:6835","nonce":489092393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4306,"qdb_cluster":[4282,4306]},"id":1}]} 2024-04-24T23:23:09.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:09.169+0000 7fa4e7e00640 1 -- 172.21.15.6:0/3169441439 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+6039 (secure 0 0 0) 0x7fa4e80792b0 con 0x7fa4f41518e0 2024-04-24T23:23:09.175 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 37 2024-04-24T23:23:09.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:09.175+0000 7fa4fb000640 1 -- 172.21.15.6:0/3169441439 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fa4d406bf00 msgr2=0x7fa4d406e3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:23:09.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:09.175+0000 7fa4fb000640 1 --2- 172.21.15.6:0/3169441439 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fa4d406bf00 0x7fa4d406e3c0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fa4f40cbf10 tx=0x7fa4e0009040 comp rx=0 tx=0).stop 2024-04-24T23:23:09.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:09.175+0000 7fa4fb000640 1 -- 172.21.15.6:0/3169441439 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa4f41518e0 msgr2=0x7fa4f415b200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:23:09.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:09.175+0000 7fa4fb000640 1 --2- 172.21.15.6:0/3169441439 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa4f41518e0 0x7fa4f415b200 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fa4e8033000 tx=0x7fa4e80046c0 comp rx=0 tx=0).stop 2024-04-24T23:23:09.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:09.175+0000 7fa4fb000640 1 -- 172.21.15.6:0/3169441439 shutdown_connections 2024-04-24T23:23:09.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:09.175+0000 7fa4fb000640 1 --2- 172.21.15.6:0/3169441439 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fa4d406bf00 0x7fa4d406e3c0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:09.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:09.175+0000 7fa4fb000640 1 --2- 172.21.15.6:0/3169441439 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa4f4157680 0x7fa4f414e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:09.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:09.175+0000 7fa4fb000640 1 --2- 172.21.15.6:0/3169441439 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa4f415ff90 0x7fa4f41f7fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:09.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:09.175+0000 7fa4fb000640 1 --2- 172.21.15.6:0/3169441439 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa4f41518e0 0x7fa4f415b200 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:09.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:09.175+0000 7fa4fb000640 1 -- 172.21.15.6:0/3169441439 >> 172.21.15.6:0/3169441439 conn(0x7fa4f406ec20 msgr2=0x7fa4f406f000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:23:09.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:09.175+0000 7fa4fb000640 1 -- 172.21.15.6:0/3169441439 shutdown_connections 2024-04-24T23:23:09.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:09.175+0000 7fa4fb000640 1 -- 172.21.15.6:0/3169441439 wait complete. 2024-04-24T23:23:09.216 INFO:tasks.mds_thrash.fs.[cephfs]:cluster is considered unstable while MDS are in up:stopping (!thrash_while_stopping) 2024-04-24T23:23:09.217 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 37, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-24T23:15:36.400716+0000", "damaged": [], "data_pools": [ 3, 4 ], "enabled": true, "epoch": 37, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_4235": { "addr": "172.21.15.190:6839/1572872451", "addrs": { "addrvec": [ { "addr": "172.21.15.190:6836", "nonce": 1572872451, "type": "v2" }, { "addr": "172.21.15.190:6839", "nonce": 1572872451, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4235, "incarnation": 13, "join_fscid": -1, "name": "d", "rank": 2, "state": "up:stopping", "state_seq": 14 }, "gid_4282": { "addr": "172.21.15.190:6837/580315269", "addrs": { "addrvec": [ { "addr": "172.21.15.190:6834", "nonce": 580315269, "type": "v2" }, { "addr": "172.21.15.190:6837", "nonce": 580315269, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4282, "incarnation": 9, "join_fscid": -1, "name": "f", "rank": 1, "state": "up:active", "state_seq": 12 }, "gid_4306": { "addr": "172.21.15.6:6835/489092393", "addrs": { "addrvec": [ { "addr": "172.21.15.6:6833", "nonce": 489092393, "type": "v2" }, { "addr": "172.21.15.6:6835", "nonce": 489092393, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4306, "incarnation": 4, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 4 } }, "last_failure": 0, "last_failure_osd_epoch": 28, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-04-24T23:23:08.146813+0000", "qdb_cluster": [ 4282, 4306 ], "qdb_leader": 4306, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 3, 4 ], "tableserver": 0, "up": { "mds_0": 4306, "mds_1": 4282, "mds_2": 4235 } } } ], "standbys": [ { "addr": "172.21.15.6:6834/1190759626", "addrs": { "addrvec": [ { "addr": "172.21.15.6:6832", "nonce": 1190759626, "type": "v2" }, { "addr": "172.21.15.6:6834", "nonce": 1190759626, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 30, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5081, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.6:6837/3928731059", "addrs": { "addrvec": [ { "addr": "172.21.15.6:6836", "nonce": 3928731059, "type": "v2" }, { "addr": "172.21.15.6:6837", "nonce": 3928731059, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 35, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5503, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-04-24T23:23:09.436 INFO:tasks.ceph.mds.b.smithi190.stdout:starting mds.b at 2024-04-24T23:23:11.218 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:23:11.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:11.847+0000 7fe3eb200640 1 Processor -- start 2024-04-24T23:23:11.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:11.848+0000 7fe3eb200640 1 -- start start 2024-04-24T23:23:11.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:11.852+0000 7fe3eb200640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fe3ec0cca10 0x7fe3ec0c36e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:23:11.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:11.852+0000 7fe3e9e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fe3ec151ac0 0x7fe3ec070e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:49276/0 (socket says 172.21.15.6:49276) 2024-04-24T23:23:11.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:11.852+0000 7fe3e9e00640 1 -- 172.21.15.6:0/716553281 learned_addr learned my addr 172.21.15.6:0/716553281 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:23:11.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:11.853+0000 7fe3e9400640 1 --2- 172.21.15.6:0/716553281 >> v2:172.21.15.190:3300/0 conn(0x7fe3ec0cca10 0x7fe3ec0c36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:23:11.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:11.853+0000 7fe3eb200640 1 --2- 172.21.15.6:0/716553281 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe3ec0c50e0 0x7fe3ec0c54e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:23:11.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:11.853+0000 7fe3eb200640 1 -- 172.21.15.6:0/716553281 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe3ec0c7b00 con 0x7fe3ec151ac0 2024-04-24T23:23:11.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:11.853+0000 7fe3eb200640 1 -- 172.21.15.6:0/716553281 --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7fe3ec15bb70 con 0x7fe3ec0cca10 2024-04-24T23:23:11.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:11.853+0000 7fe3eb200640 1 -- 172.21.15.6:0/716553281 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fe3ec072530 con 0x7fe3ec0c50e0 2024-04-24T23:23:11.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:11.854+0000 7fe3e8a00640 1 -- 172.21.15.6:0/716553281 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1444228509 0 0) 0x7fe3ec0c7b00 con 0x7fe3ec151ac0 2024-04-24T23:23:11.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:11.854+0000 7fe3e8a00640 1 -- 172.21.15.6:0/716553281 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe3d0003520 con 0x7fe3ec151ac0 2024-04-24T23:23:11.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:11.856+0000 7fe3e8a00640 1 -- 172.21.15.6:0/716553281 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4208746703 0 0) 0x7fe3d0003520 con 0x7fe3ec151ac0 2024-04-24T23:23:11.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:11.856+0000 7fe3e8a00640 1 -- 172.21.15.6:0/716553281 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe3ec0c50e0 msgr2=0x7fe3ec0c54e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:23:11.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:11.856+0000 7fe3e8a00640 1 --2- 172.21.15.6:0/716553281 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe3ec0c50e0 0x7fe3ec0c54e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:11.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:11.856+0000 7fe3e8a00640 1 -- 172.21.15.6:0/716553281 >> v2:172.21.15.190:3300/0 conn(0x7fe3ec0cca10 msgr2=0x7fe3ec0c36e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:23:11.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:11.856+0000 7fe3e8a00640 1 --2- 172.21.15.6:0/716553281 >> v2:172.21.15.190:3300/0 conn(0x7fe3ec0cca10 0x7fe3ec0c36e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:11.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:11.856+0000 7fe3e8a00640 1 -- 172.21.15.6:0/716553281 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3ec0c6d80 con 0x7fe3ec151ac0 2024-04-24T23:23:11.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:11.856+0000 7fe3e8a00640 1 -- 172.21.15.6:0/716553281 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3463268701 0 0) 0x7fe3dc001670 con 0x7fe3ec151ac0 2024-04-24T23:23:11.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:11.856+0000 7fe3e8a00640 1 -- 172.21.15.6:0/716553281 >> v1:172.21.15.6:6789/0 conn(0x7fe3ec151ac0 legacy=0x7fe3ec070e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:23:11.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:11.856+0000 7fe3e8a00640 1 --2- 172.21.15.6:0/716553281 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe3d00039b0 0x7fe3d0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:23:11.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:11.856+0000 7fe3e8a00640 1 --2- 172.21.15.6:0/716553281 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe3ec0c50e0 0x7fe3d0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:23:11.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:11.856+0000 7fe3e8a00640 1 --2- 172.21.15.6:0/716553281 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe3d0004d40 0x7fe3d00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:23:11.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:11.856+0000 7fe3e8a00640 1 -- 172.21.15.6:0/716553281 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fe3ec15bb70 con 0x7fe3d00039b0 2024-04-24T23:23:11.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:11.856+0000 7fe3e8a00640 1 -- 172.21.15.6:0/716553281 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fe3ec072530 con 0x7fe3d0004d40 2024-04-24T23:23:11.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:11.856+0000 7fe3e8a00640 1 -- 172.21.15.6:0/716553281 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fe3d000ddf0 con 0x7fe3ec0c50e0 2024-04-24T23:23:11.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:11.857+0000 7fe3ea800640 1 --2- 172.21.15.6:0/716553281 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe3d0004d40 0x7fe3d00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:23:11.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:11.857+0000 7fe3e9400640 1 --2- 172.21.15.6:0/716553281 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe3ec0c50e0 0x7fe3d0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:23:11.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:11.857+0000 7fe3e9e00640 1 --2- 172.21.15.6:0/716553281 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe3d00039b0 0x7fe3d0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:23:11.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:11.858+0000 7fe3e9400640 1 -- 172.21.15.6:0/716553281 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe3d0004d40 msgr2=0x7fe3d00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:23:11.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:11.858+0000 7fe3e9400640 1 --2- 172.21.15.6:0/716553281 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe3d0004d40 0x7fe3d00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:11.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:11.858+0000 7fe3e9400640 1 -- 172.21.15.6:0/716553281 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe3d00039b0 msgr2=0x7fe3d0003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:23:11.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:11.858+0000 7fe3e9400640 1 --2- 172.21.15.6:0/716553281 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe3d00039b0 0x7fe3d0003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:11.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:11.858+0000 7fe3e9400640 1 -- 172.21.15.6:0/716553281 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3ec0c6d80 con 0x7fe3ec0c50e0 2024-04-24T23:23:11.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:11.861+0000 7fe3e9400640 1 --2- 172.21.15.6:0/716553281 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe3ec0c50e0 0x7fe3d0004800 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fe3d400ba30 tx=0x7fe3d0005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8d99998687f8912b server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:23:11.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:11.862+0000 7fe3e8a00640 1 -- 172.21.15.6:0/716553281 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3d4030070 con 0x7fe3ec0c50e0 2024-04-24T23:23:11.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:11.862+0000 7fe3e8a00640 1 -- 172.21.15.6:0/716553281 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe3d402ad50 con 0x7fe3ec0c50e0 2024-04-24T23:23:11.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:11.863+0000 7fe3eb200640 1 -- 172.21.15.6:0/716553281 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe3ec0c50e0 msgr2=0x7fe3d0004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:23:11.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:11.863+0000 7fe3eb200640 1 --2- 172.21.15.6:0/716553281 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe3ec0c50e0 0x7fe3d0004800 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fe3d400ba30 tx=0x7fe3d0005830 comp rx=0 tx=0).stop 2024-04-24T23:23:11.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:11.866+0000 7fe3ea800640 1 -- 172.21.15.6:0/716553281 reap_dead start 2024-04-24T23:23:11.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:11.866+0000 7fe3eb200640 1 -- 172.21.15.6:0/716553281 shutdown_connections 2024-04-24T23:23:11.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:11.866+0000 7fe3eb200640 1 -- 172.21.15.6:0/716553281 >> 172.21.15.6:0/716553281 conn(0x7fe3ec06ed80 msgr2=0x7fe3ec06f190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:23:11.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:11.866+0000 7fe3eb200640 1 -- 172.21.15.6:0/716553281 shutdown_connections 2024-04-24T23:23:11.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:11.867+0000 7fe3eb200640 1 -- 172.21.15.6:0/716553281 wait complete. 2024-04-24T23:23:11.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:11.869+0000 7fe3eb200640 1 Processor -- start 2024-04-24T23:23:11.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:11.873+0000 7fe3eb200640 1 -- start start 2024-04-24T23:23:11.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:11.876+0000 7fe3eb200640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe3ec157c50 0x7fe3ec1ec710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:23:11.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:11.879+0000 7fe3eb200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe3ec158030 0x7fe3ec1ecc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:23:11.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:11.879+0000 7fe3eb200640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe3ec13a900 0x7fe3ec13ad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:23:11.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:11.880+0000 7fe3ea800640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe3ec13a900 0x7fe3ec13ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:23:11.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:11.880+0000 7fe3ea800640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe3ec13a900 0x7fe3ec13ad90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.6:41306/0 (socket says 172.21.15.6:41306) 2024-04-24T23:23:11.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:11.881+0000 7fe3ea800640 1 -- 172.21.15.6:0/3158252511 learned_addr learned my addr 172.21.15.6:0/3158252511 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:23:11.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:11.881+0000 7fe3eb200640 1 -- 172.21.15.6:0/3158252511 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fe3ec15b890 con 0x7fe3ec158030 2024-04-24T23:23:11.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:11.881+0000 7fe3eb200640 1 -- 172.21.15.6:0/3158252511 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fe3ec0c7670 con 0x7fe3ec157c50 2024-04-24T23:23:11.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:11.881+0000 7fe3e9e00640 1 --2- 172.21.15.6:0/3158252511 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe3ec157c50 0x7fe3ec1ec710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:23:11.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:11.882+0000 7fe3eb200640 1 -- 172.21.15.6:0/3158252511 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fe3ec0c7020 con 0x7fe3ec13a900 2024-04-24T23:23:11.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:11.882+0000 7fe3e9400640 1 --2- 172.21.15.6:0/3158252511 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe3ec158030 0x7fe3ec1ecc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:23:11.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:11.883+0000 7fe3ea800640 1 -- 172.21.15.6:0/3158252511 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe3ec157c50 msgr2=0x7fe3ec1ec710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:23:11.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:11.883+0000 7fe3ea800640 1 --2- 172.21.15.6:0/3158252511 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe3ec157c50 0x7fe3ec1ec710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:11.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:11.883+0000 7fe3ea800640 1 -- 172.21.15.6:0/3158252511 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe3ec158030 msgr2=0x7fe3ec1ecc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:23:11.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:11.883+0000 7fe3ea800640 1 --2- 172.21.15.6:0/3158252511 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe3ec158030 0x7fe3ec1ecc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:11.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:11.883+0000 7fe3ea800640 1 -- 172.21.15.6:0/3158252511 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3ec13b2d0 con 0x7fe3ec13a900 2024-04-24T23:23:11.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:11.884+0000 7fe3ea800640 1 --2- 172.21.15.6:0/3158252511 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe3ec13a900 0x7fe3ec13ad90 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fe3e4000f80 tx=0x7fe3e400ed80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:23:11.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:11.884+0000 7fe3e2000640 1 -- 172.21.15.6:0/3158252511 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3e400ca80 con 0x7fe3ec13a900 2024-04-24T23:23:11.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:11.884+0000 7fe3e2000640 1 -- 172.21.15.6:0/3158252511 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe3e4006e50 con 0x7fe3ec13a900 2024-04-24T23:23:11.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:11.884+0000 7fe3e2000640 1 -- 172.21.15.6:0/3158252511 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3e400ce80 con 0x7fe3ec13a900 2024-04-24T23:23:11.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:11.885+0000 7fe3eb200640 1 -- 172.21.15.6:0/3158252511 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3ec1efc70 con 0x7fe3ec13a900 2024-04-24T23:23:11.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:11.885+0000 7fe3eb200640 1 -- 172.21.15.6:0/3158252511 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3ec1f0230 con 0x7fe3ec13a900 2024-04-24T23:23:11.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:11.888+0000 7fe3e2000640 1 -- 172.21.15.6:0/3158252511 <== mon.2 v2:172.21.15.190:3301/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fe3e4016450 con 0x7fe3ec13a900 2024-04-24T23:23:11.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:11.888+0000 7fe3eb200640 1 -- 172.21.15.6:0/3158252511 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe3ec075b30 con 0x7fe3ec13a900 2024-04-24T23:23:11.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:11.889+0000 7fe3e2000640 1 --2- 172.21.15.6:0/3158252511 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fe3c806beb0 0x7fe3c806e370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:23:11.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:11.889+0000 7fe3e2000640 1 -- 172.21.15.6:0/3158252511 <== mon.2 v2:172.21.15.190:3301/0 5 ==== osd_map(28..28 src has 1..28) ==== 6762+0+0 (secure 0 0 0) 0x7fe3e408b430 con 0x7fe3ec13a900 2024-04-24T23:23:11.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:11.890+0000 7fe3e9e00640 1 --2- 172.21.15.6:0/3158252511 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fe3c806beb0 0x7fe3c806e370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:23:11.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:11.890+0000 7fe3e9e00640 1 --2- 172.21.15.6:0/3158252511 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fe3c806beb0 0x7fe3c806e370 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fe3dc0017e0 tx=0x7fe3dc007040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:23:11.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:11.898+0000 7fe3e2000640 1 -- 172.21.15.6:0/3158252511 <== mon.2 v2:172.21.15.190:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fe3e4059910 con 0x7fe3ec13a900 2024-04-24T23:23:12.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:12.508+0000 7fe3eb200640 1 -- 172.21.15.6:0/3158252511 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe3ec0c3da0 con 0x7fe3ec13a900 2024-04-24T23:23:12.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:12.508+0000 7fe3e2000640 1 -- 172.21.15.6:0/3158252511 <== mon.2 v2:172.21.15.190:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) ==== 94+0+6867 (secure 0 0 0) 0x7fe3e4059210 con 0x7fe3ec13a900 2024-04-24T23:23:12.511 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:23:12.511 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":38,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5081,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/1190759626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1190759626},{"type":"v1","addr":"172.21.15.6:6834","nonce":1190759626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":5503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/3928731059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":3928731059},{"type":"v1","addr":"172.21.15.6:6837","nonce":3928731059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":35},{"gid":5551,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6838/3253046315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":3253046315},{"type":"v1","addr":"172.21.15.190:6838","nonce":3253046315}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":38}],"filesystems":[{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:23:08.146813+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1,2],"up":{"mds_0":4306,"mds_1":4282,"mds_2":4235},"failed":[],"damaged":[],"stopped":[3,4],"info":{"gid_4235":{"gid":4235,"name":"d","rank":2,"incarnation":13,"state":"up:stopping","state_seq":14,"addr":"172.21.15.190:6839/1572872451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":1572872451},{"type":"v1","addr":"172.21.15.190:6839","nonce":1572872451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4282":{"gid":4282,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":12,"addr":"172.21.15.190:6837/580315269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":580315269},{"type":"v1","addr":"172.21.15.190:6837","nonce":580315269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/489092393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":489092393},{"type":"v1","addr":"172.21.15.6:6835","nonce":489092393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4306,"qdb_cluster":[4306,4282]},"id":1}]} 2024-04-24T23:23:12.511 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 38 2024-04-24T23:23:12.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:12.513+0000 7fe3eb200640 1 -- 172.21.15.6:0/3158252511 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fe3c806beb0 msgr2=0x7fe3c806e370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:23:12.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:12.513+0000 7fe3eb200640 1 --2- 172.21.15.6:0/3158252511 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fe3c806beb0 0x7fe3c806e370 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fe3dc0017e0 tx=0x7fe3dc007040 comp rx=0 tx=0).stop 2024-04-24T23:23:12.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:12.513+0000 7fe3eb200640 1 -- 172.21.15.6:0/3158252511 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe3ec13a900 msgr2=0x7fe3ec13ad90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:23:12.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:12.513+0000 7fe3eb200640 1 --2- 172.21.15.6:0/3158252511 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe3ec13a900 0x7fe3ec13ad90 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fe3e4000f80 tx=0x7fe3e400ed80 comp rx=0 tx=0).stop 2024-04-24T23:23:12.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:12.513+0000 7fe3eb200640 1 -- 172.21.15.6:0/3158252511 shutdown_connections 2024-04-24T23:23:12.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:12.513+0000 7fe3eb200640 1 --2- 172.21.15.6:0/3158252511 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fe3c806beb0 0x7fe3c806e370 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:12.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:12.513+0000 7fe3eb200640 1 --2- 172.21.15.6:0/3158252511 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe3ec13a900 0x7fe3ec13ad90 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:12.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:12.513+0000 7fe3eb200640 1 --2- 172.21.15.6:0/3158252511 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe3ec158030 0x7fe3ec1ecc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:12.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:12.513+0000 7fe3eb200640 1 --2- 172.21.15.6:0/3158252511 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe3ec157c50 0x7fe3ec1ec710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:12.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:12.513+0000 7fe3eb200640 1 -- 172.21.15.6:0/3158252511 >> 172.21.15.6:0/3158252511 conn(0x7fe3ec06ed80 msgr2=0x7fe3ec14f6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:23:12.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:12.513+0000 7fe3eb200640 1 -- 172.21.15.6:0/3158252511 shutdown_connections 2024-04-24T23:23:12.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:12.514+0000 7fe3eb200640 1 -- 172.21.15.6:0/3158252511 wait complete. 2024-04-24T23:23:12.545 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-24T23:23:14.546 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:23:14.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:14.960+0000 7fbb43200640 1 Processor -- start 2024-04-24T23:23:14.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:14.960+0000 7fbb43200640 1 -- start start 2024-04-24T23:23:14.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:14.963+0000 7fbb43200640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fbb440c7e80 0x7fbb44156a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:23:14.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:14.963+0000 7fbb43200640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbb4406ce30 0x7fbb441506b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:23:14.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:14.963+0000 7fbb41e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fbb440c8d50 0x7fbb440c7770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:38920/0 (socket says 172.21.15.6:38920) 2024-04-24T23:23:14.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:14.963+0000 7fbb41e00640 1 -- 172.21.15.6:0/191232069 learned_addr learned my addr 172.21.15.6:0/191232069 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:23:14.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:14.964+0000 7fbb42800640 1 --2- 172.21.15.6:0/191232069 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbb4406ce30 0x7fbb441506b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:23:14.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:14.964+0000 7fbb43200640 1 -- 172.21.15.6:0/191232069 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb4406d370 con 0x7fbb440c8d50 2024-04-24T23:23:14.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:14.964+0000 7fbb43200640 1 -- 172.21.15.6:0/191232069 --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7fbb4406def0 con 0x7fbb440c7e80 2024-04-24T23:23:14.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:14.964+0000 7fbb43200640 1 -- 172.21.15.6:0/191232069 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fbb4406d650 con 0x7fbb4406ce30 2024-04-24T23:23:14.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:14.964+0000 7fbb42800640 1 -- 172.21.15.6:0/191232069 >> v2:172.21.15.190:3300/0 conn(0x7fbb440c7e80 msgr2=0x7fbb44156a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:23:14.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:14.967+0000 7fbb41400640 1 --2- 172.21.15.6:0/191232069 >> v2:172.21.15.190:3300/0 conn(0x7fbb440c7e80 0x7fbb44156a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:23:14.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:14.967+0000 7fbb42800640 1 --2- 172.21.15.6:0/191232069 >> v2:172.21.15.190:3300/0 conn(0x7fbb440c7e80 0x7fbb44156a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:14.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:14.967+0000 7fbb42800640 1 -- 172.21.15.6:0/191232069 >> v1:172.21.15.6:6789/0 conn(0x7fbb440c8d50 legacy=0x7fbb440c7770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:23:14.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:14.967+0000 7fbb42800640 1 -- 172.21.15.6:0/191232069 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb4406d930 con 0x7fbb4406ce30 2024-04-24T23:23:14.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:14.968+0000 7fbb40a00640 1 -- 172.21.15.6:0/191232069 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1221604798 0 0) 0x7fbb4406d370 con 0x7fbb440c8d50 2024-04-24T23:23:14.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:14.968+0000 7fbb41400640 1 --2- 172.21.15.6:0/191232069 >> v2:172.21.15.190:3300/0 conn(0x7fbb440c7e80 0x7fbb44156a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:23:14.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:14.969+0000 7fbb42800640 1 --2- 172.21.15.6:0/191232069 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbb4406ce30 0x7fbb441506b0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fbb3400a8c0 tx=0x7fbb3402b350 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e06bdc0690c8a5b7 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:23:14.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:14.969+0000 7fbb40a00640 1 -- 172.21.15.6:0/191232069 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb34038070 con 0x7fbb4406ce30 2024-04-24T23:23:14.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:14.969+0000 7fbb40a00640 1 -- 172.21.15.6:0/191232069 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb34033640 con 0x7fbb4406ce30 2024-04-24T23:23:14.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:14.969+0000 7fbb40a00640 1 -- 172.21.15.6:0/191232069 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb34033a10 con 0x7fbb4406ce30 2024-04-24T23:23:14.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:14.970+0000 7fbb43200640 1 -- 172.21.15.6:0/191232069 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbb4406ce30 msgr2=0x7fbb441506b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:23:14.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:14.970+0000 7fbb43200640 1 --2- 172.21.15.6:0/191232069 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbb4406ce30 0x7fbb441506b0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fbb3400a8c0 tx=0x7fbb3402b350 comp rx=0 tx=0).stop 2024-04-24T23:23:14.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:14.971+0000 7fbb43200640 1 -- 172.21.15.6:0/191232069 shutdown_connections 2024-04-24T23:23:14.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:14.971+0000 7fbb43200640 1 --2- 172.21.15.6:0/191232069 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbb4406ce30 0x7fbb441506b0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:14.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:14.971+0000 7fbb43200640 1 --2- 172.21.15.6:0/191232069 >> v2:172.21.15.190:3300/0 conn(0x7fbb440c7e80 0x7fbb44156a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:14.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:14.971+0000 7fbb43200640 1 -- 172.21.15.6:0/191232069 >> 172.21.15.6:0/191232069 conn(0x7fbb4406ed80 msgr2=0x7fbb4406f190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:23:14.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:14.971+0000 7fbb43200640 1 -- 172.21.15.6:0/191232069 shutdown_connections 2024-04-24T23:23:14.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:14.971+0000 7fbb43200640 1 -- 172.21.15.6:0/191232069 wait complete. 2024-04-24T23:23:14.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:14.973+0000 7fbb43200640 1 Processor -- start 2024-04-24T23:23:14.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:14.974+0000 7fbb43200640 1 -- start start 2024-04-24T23:23:14.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:14.975+0000 7fbb43200640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbb4406ce30 0x7fbb441f4e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:23:14.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:14.975+0000 7fbb43200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbb440c7e80 0x7fbb440c5530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:23:14.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:14.975+0000 7fbb43200640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbb440c8d50 0x7fbb44148260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:23:14.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:14.975+0000 7fbb43200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fbb440cc1e0 con 0x7fbb440c7e80 2024-04-24T23:23:14.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:14.975+0000 7fbb43200640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fbb44071020 con 0x7fbb440c8d50 2024-04-24T23:23:14.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:14.975+0000 7fbb43200640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fbb440cc360 con 0x7fbb4406ce30 2024-04-24T23:23:14.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:14.976+0000 7fbb41e00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbb4406ce30 0x7fbb441f4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:23:14.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:14.976+0000 7fbb41e00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbb4406ce30 0x7fbb441f4e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.6:33862/0 (socket says 172.21.15.6:33862) 2024-04-24T23:23:14.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:14.976+0000 7fbb41e00640 1 -- 172.21.15.6:0/3863557776 learned_addr learned my addr 172.21.15.6:0/3863557776 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:23:14.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:14.976+0000 7fbb42800640 1 --2- 172.21.15.6:0/3863557776 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbb440c8d50 0x7fbb44148260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:23:14.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:14.976+0000 7fbb41400640 1 --2- 172.21.15.6:0/3863557776 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbb440c7e80 0x7fbb440c5530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:23:14.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:14.977+0000 7fbb41e00640 1 -- 172.21.15.6:0/3863557776 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbb440c8d50 msgr2=0x7fbb44148260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:23:14.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:14.977+0000 7fbb41e00640 1 --2- 172.21.15.6:0/3863557776 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbb440c8d50 0x7fbb44148260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:14.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:14.977+0000 7fbb41e00640 1 -- 172.21.15.6:0/3863557776 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbb440c7e80 msgr2=0x7fbb440c5530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:23:14.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:14.977+0000 7fbb41e00640 1 --2- 172.21.15.6:0/3863557776 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbb440c7e80 0x7fbb440c5530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:14.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:14.977+0000 7fbb41e00640 1 -- 172.21.15.6:0/3863557776 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb44154380 con 0x7fbb4406ce30 2024-04-24T23:23:14.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:14.977+0000 7fbb41400640 1 --2- 172.21.15.6:0/3863557776 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbb440c7e80 0x7fbb440c5530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:23:14.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:14.978+0000 7fbb41e00640 1 --2- 172.21.15.6:0/3863557776 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbb4406ce30 0x7fbb441f4e90 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fbb3000b4d0 tx=0x7fbb3000b990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:23:14.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:14.978+0000 7fbb3a000640 1 -- 172.21.15.6:0/3863557776 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb30004e30 con 0x7fbb4406ce30 2024-04-24T23:23:14.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:14.978+0000 7fbb3a000640 1 -- 172.21.15.6:0/3863557776 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb30015070 con 0x7fbb4406ce30 2024-04-24T23:23:14.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:14.978+0000 7fbb3a000640 1 -- 172.21.15.6:0/3863557776 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb300062b0 con 0x7fbb4406ce30 2024-04-24T23:23:14.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:14.978+0000 7fbb43200640 1 -- 172.21.15.6:0/3863557776 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb44152780 con 0x7fbb4406ce30 2024-04-24T23:23:14.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:14.979+0000 7fbb43200640 1 -- 172.21.15.6:0/3863557776 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb44152d40 con 0x7fbb4406ce30 2024-04-24T23:23:14.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:14.980+0000 7fbb3a000640 1 -- 172.21.15.6:0/3863557776 <== mon.2 v2:172.21.15.190:3301/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fbb30006450 con 0x7fbb4406ce30 2024-04-24T23:23:14.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:14.981+0000 7fbb43200640 1 -- 172.21.15.6:0/3863557776 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb44075b30 con 0x7fbb4406ce30 2024-04-24T23:23:14.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:14.982+0000 7fbb3a000640 1 --2- 172.21.15.6:0/3863557776 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fbb2406be60 0x7fbb2406e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:23:14.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:14.982+0000 7fbb3a000640 1 -- 172.21.15.6:0/3863557776 <== mon.2 v2:172.21.15.190:3301/0 5 ==== osd_map(28..28 src has 1..28) ==== 6762+0+0 (secure 0 0 0) 0x7fbb3008aff0 con 0x7fbb4406ce30 2024-04-24T23:23:14.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:14.982+0000 7fbb41400640 1 --2- 172.21.15.6:0/3863557776 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fbb2406be60 0x7fbb2406e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:23:14.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:14.983+0000 7fbb41400640 1 --2- 172.21.15.6:0/3863557776 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fbb2406be60 0x7fbb2406e320 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fbb440ca2f0 tx=0x7fbb3c007ab0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:23:14.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:14.986+0000 7fbb3a000640 1 -- 172.21.15.6:0/3863557776 <== mon.2 v2:172.21.15.190:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fbb30059540 con 0x7fbb4406ce30 2024-04-24T23:23:15.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:15.440+0000 7fbb43200640 1 -- 172.21.15.6:0/3863557776 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbb4414f340 con 0x7fbb4406ce30 2024-04-24T23:23:15.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:15.441+0000 7fbb3a000640 1 -- 172.21.15.6:0/3863557776 <== mon.2 v2:172.21.15.190:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) ==== 94+0+6867 (secure 0 0 0) 0x7fbb30058e40 con 0x7fbb4406ce30 2024-04-24T23:23:15.443 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:23:15.444 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":38,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5081,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/1190759626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1190759626},{"type":"v1","addr":"172.21.15.6:6834","nonce":1190759626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":5503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/3928731059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":3928731059},{"type":"v1","addr":"172.21.15.6:6837","nonce":3928731059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":35},{"gid":5551,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6838/3253046315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":3253046315},{"type":"v1","addr":"172.21.15.190:6838","nonce":3253046315}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":38}],"filesystems":[{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:23:08.146813+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1,2],"up":{"mds_0":4306,"mds_1":4282,"mds_2":4235},"failed":[],"damaged":[],"stopped":[3,4],"info":{"gid_4235":{"gid":4235,"name":"d","rank":2,"incarnation":13,"state":"up:stopping","state_seq":14,"addr":"172.21.15.190:6839/1572872451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":1572872451},{"type":"v1","addr":"172.21.15.190:6839","nonce":1572872451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4282":{"gid":4282,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":12,"addr":"172.21.15.190:6837/580315269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":580315269},{"type":"v1","addr":"172.21.15.190:6837","nonce":580315269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/489092393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":489092393},{"type":"v1","addr":"172.21.15.6:6835","nonce":489092393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4306,"qdb_cluster":[4306,4282]},"id":1}]} 2024-04-24T23:23:15.444 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 38 2024-04-24T23:23:15.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:15.446+0000 7fbb43200640 1 -- 172.21.15.6:0/3863557776 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fbb2406be60 msgr2=0x7fbb2406e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:23:15.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:15.446+0000 7fbb43200640 1 --2- 172.21.15.6:0/3863557776 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fbb2406be60 0x7fbb2406e320 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fbb440ca2f0 tx=0x7fbb3c007ab0 comp rx=0 tx=0).stop 2024-04-24T23:23:15.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:15.446+0000 7fbb43200640 1 -- 172.21.15.6:0/3863557776 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbb4406ce30 msgr2=0x7fbb441f4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:23:15.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:15.446+0000 7fbb43200640 1 --2- 172.21.15.6:0/3863557776 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbb4406ce30 0x7fbb441f4e90 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fbb3000b4d0 tx=0x7fbb3000b990 comp rx=0 tx=0).stop 2024-04-24T23:23:15.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:15.447+0000 7fbb43200640 1 -- 172.21.15.6:0/3863557776 shutdown_connections 2024-04-24T23:23:15.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:15.447+0000 7fbb43200640 1 --2- 172.21.15.6:0/3863557776 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fbb2406be60 0x7fbb2406e320 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:15.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:15.447+0000 7fbb43200640 1 --2- 172.21.15.6:0/3863557776 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbb440c8d50 0x7fbb44148260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:15.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:15.447+0000 7fbb43200640 1 --2- 172.21.15.6:0/3863557776 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbb440c7e80 0x7fbb440c5530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:15.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:15.447+0000 7fbb43200640 1 --2- 172.21.15.6:0/3863557776 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbb4406ce30 0x7fbb441f4e90 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:15.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:15.447+0000 7fbb43200640 1 -- 172.21.15.6:0/3863557776 >> 172.21.15.6:0/3863557776 conn(0x7fbb4406ed80 msgr2=0x7fbb44149eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:23:15.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:15.448+0000 7fbb43200640 1 -- 172.21.15.6:0/3863557776 shutdown_connections 2024-04-24T23:23:15.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:15.448+0000 7fbb43200640 1 -- 172.21.15.6:0/3863557776 wait complete. 2024-04-24T23:23:15.490 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-24T23:23:17.060 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-24T23:23:17.067 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-24T23:23:17.491 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:23:17.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:17.769+0000 7f39d9c00640 1 Processor -- start 2024-04-24T23:23:17.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:17.770+0000 7f39d9c00640 1 -- start start 2024-04-24T23:23:17.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:17.771+0000 7f39d9c00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f39d40c6e10 0x7f39d40c7e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:23:17.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:17.771+0000 7f39d9c00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f39d4071880 con 0x7f39d41593b0 2024-04-24T23:23:17.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:17.771+0000 7f39d9c00640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f39d4072900 con 0x7f39d40c8390 2024-04-24T23:23:17.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:17.771+0000 7f39d9c00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f39d4072470 con 0x7f39d40c6e10 2024-04-24T23:23:17.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:17.771+0000 7f39d2000640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f39d40c6e10 0x7f39d40c7e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:23:17.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:17.771+0000 7f39d3400640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f39d40c8390 0x7f39d4162460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.6:42566/0 (socket says 172.21.15.6:42566) 2024-04-24T23:23:17.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:17.771+0000 7f39d3400640 1 -- 172.21.15.6:0/505083775 learned_addr learned my addr 172.21.15.6:0/505083775 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:23:17.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:17.772+0000 7f39d2000640 1 -- 172.21.15.6:0/505083775 >> v1:172.21.15.190:6789/0 conn(0x7f39d40c8390 legacy=0x7f39d4162460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:23:17.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:17.772+0000 7f39d2000640 1 -- 172.21.15.6:0/505083775 >> v1:172.21.15.6:6789/0 conn(0x7f39d41593b0 legacy=0x7f39d406edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:23:17.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:17.772+0000 7f39d1600640 1 -- 172.21.15.6:0/505083775 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2883254033 0 0) 0x7f39d4072900 con 0x7f39d40c8390 2024-04-24T23:23:17.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:17.772+0000 7f39d2000640 1 -- 172.21.15.6:0/505083775 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39d40c7410 con 0x7f39d40c6e10 2024-04-24T23:23:17.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:17.773+0000 7f39d2000640 1 --2- 172.21.15.6:0/505083775 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f39d40c6e10 0x7f39d40c7e50 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f39c00097d0 tx=0x7f39c002a250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=23cb0621bcd39232 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:23:17.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:17.773+0000 7f39d1600640 1 -- 172.21.15.6:0/505083775 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39c0037070 con 0x7f39d40c6e10 2024-04-24T23:23:17.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:17.773+0000 7f39d1600640 1 -- 172.21.15.6:0/505083775 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f39c0032440 con 0x7f39d40c6e10 2024-04-24T23:23:17.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:17.773+0000 7f39d1600640 1 -- 172.21.15.6:0/505083775 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39c0032810 con 0x7f39d40c6e10 2024-04-24T23:23:17.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:17.774+0000 7f39d9c00640 1 -- 172.21.15.6:0/505083775 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f39d40c6e10 msgr2=0x7f39d40c7e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:23:17.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:17.774+0000 7f39d9c00640 1 --2- 172.21.15.6:0/505083775 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f39d40c6e10 0x7f39d40c7e50 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f39c00097d0 tx=0x7f39c002a250 comp rx=0 tx=0).stop 2024-04-24T23:23:17.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:17.775+0000 7f39d9c00640 1 -- 172.21.15.6:0/505083775 shutdown_connections 2024-04-24T23:23:17.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:17.775+0000 7f39d9c00640 1 --2- 172.21.15.6:0/505083775 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f39d40c6e10 0x7f39d40c7e50 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:17.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:17.775+0000 7f39d9c00640 1 -- 172.21.15.6:0/505083775 >> 172.21.15.6:0/505083775 conn(0x7f39d406b0e0 msgr2=0x7f39d406b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:23:17.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:17.775+0000 7f39d9c00640 1 -- 172.21.15.6:0/505083775 shutdown_connections 2024-04-24T23:23:17.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:17.775+0000 7f39d9c00640 1 -- 172.21.15.6:0/505083775 wait complete. 2024-04-24T23:23:17.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:17.778+0000 7f39d9c00640 1 Processor -- start 2024-04-24T23:23:17.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:17.779+0000 7f39d9c00640 1 -- start start 2024-04-24T23:23:17.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:17.780+0000 7f39d9c00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f39d40c6e10 0x7f39d408c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:23:17.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:17.780+0000 7f39d9c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f39d40c8390 0x7f39d41538c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:23:17.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:17.780+0000 7f39d9c00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f39d41593b0 0x7f39d4155e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:23:17.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:17.780+0000 7f39d9c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f39d4072100 con 0x7f39d40c8390 2024-04-24T23:23:17.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:17.780+0000 7f39d9c00640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f39d4071e20 con 0x7f39d41593b0 2024-04-24T23:23:17.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:17.780+0000 7f39d9c00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f39d4163080 con 0x7f39d40c6e10 2024-04-24T23:23:17.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:17.781+0000 7f39d3400640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f39d41593b0 0x7f39d4155e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:23:17.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:17.781+0000 7f39d3400640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f39d41593b0 0x7f39d4155e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:44042/0 (socket says 172.21.15.6:44042) 2024-04-24T23:23:17.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:17.781+0000 7f39d3400640 1 -- 172.21.15.6:0/2170519150 learned_addr learned my addr 172.21.15.6:0/2170519150 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:23:17.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:17.782+0000 7f39d3400640 1 -- 172.21.15.6:0/2170519150 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f39d40c6e10 msgr2=0x7f39d408c820 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-24T23:23:17.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:17.782+0000 7f39d3400640 1 --2- 172.21.15.6:0/2170519150 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f39d40c6e10 0x7f39d408c820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:17.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:17.782+0000 7f39d3400640 1 -- 172.21.15.6:0/2170519150 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f39d40c8390 msgr2=0x7f39d41538c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:23:17.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:17.782+0000 7f39d3400640 1 --2- 172.21.15.6:0/2170519150 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f39d40c8390 0x7f39d41538c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:17.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:17.782+0000 7f39d3400640 1 -- 172.21.15.6:0/2170519150 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39d42064d0 con 0x7f39d41593b0 2024-04-24T23:23:17.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:17.783+0000 7f39d3400640 1 --2- 172.21.15.6:0/2170519150 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f39d41593b0 0x7f39d4155e00 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f39cc00c840 tx=0x7f39cc00cd00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:23:17.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:17.783+0000 7f39caa00640 1 -- 172.21.15.6:0/2170519150 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39cc018070 con 0x7f39d41593b0 2024-04-24T23:23:17.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:17.783+0000 7f39d9c00640 1 -- 172.21.15.6:0/2170519150 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39d415d0a0 con 0x7f39d41593b0 2024-04-24T23:23:17.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:17.783+0000 7f39d9c00640 1 -- 172.21.15.6:0/2170519150 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39d4206760 con 0x7f39d41593b0 2024-04-24T23:23:17.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:17.784+0000 7f39caa00640 1 -- 172.21.15.6:0/2170519150 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f39cc00a640 con 0x7f39d41593b0 2024-04-24T23:23:17.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:17.784+0000 7f39caa00640 1 -- 172.21.15.6:0/2170519150 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39cc00aa30 con 0x7f39d41593b0 2024-04-24T23:23:17.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:17.785+0000 7f39d9c00640 1 -- 172.21.15.6:0/2170519150 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f39ac005290 con 0x7f39d41593b0 2024-04-24T23:23:17.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:17.790+0000 7f39caa00640 1 -- 172.21.15.6:0/2170519150 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f39cc00abd0 con 0x7f39d41593b0 2024-04-24T23:23:17.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:17.791+0000 7f39caa00640 1 --2- 172.21.15.6:0/2170519150 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f39b406be30 0x7f39b406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:23:17.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:17.791+0000 7f39caa00640 1 -- 172.21.15.6:0/2170519150 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6762+0+0 (secure 0 0 0) 0x7f39cc08b020 con 0x7f39d41593b0 2024-04-24T23:23:17.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:17.795+0000 7f39caa00640 1 -- 172.21.15.6:0/2170519150 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f39cc059510 con 0x7f39d41593b0 2024-04-24T23:23:17.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:17.795+0000 7f39d2a00640 1 --2- 172.21.15.6:0/2170519150 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f39b406be30 0x7f39b406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:23:17.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:17.801+0000 7f39d2a00640 1 --2- 172.21.15.6:0/2170519150 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f39b406be30 0x7f39b406e2f0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f39c40095f0 tx=0x7f39c4006b50 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:23:18.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:18.360+0000 7f39d9c00640 1 -- 172.21.15.6:0/2170519150 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f39ac005580 con 0x7f39d41593b0 2024-04-24T23:23:18.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:18.363+0000 7f39caa00640 1 -- 172.21.15.6:0/2170519150 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) ==== 94+0+6867 (secure 0 0 0) 0x7f39cc058e10 con 0x7f39d41593b0 2024-04-24T23:23:18.365 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:23:18.365 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":38,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5081,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/1190759626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1190759626},{"type":"v1","addr":"172.21.15.6:6834","nonce":1190759626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":5503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/3928731059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":3928731059},{"type":"v1","addr":"172.21.15.6:6837","nonce":3928731059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":35},{"gid":5551,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6838/3253046315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":3253046315},{"type":"v1","addr":"172.21.15.190:6838","nonce":3253046315}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":38}],"filesystems":[{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:23:08.146813+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1,2],"up":{"mds_0":4306,"mds_1":4282,"mds_2":4235},"failed":[],"damaged":[],"stopped":[3,4],"info":{"gid_4235":{"gid":4235,"name":"d","rank":2,"incarnation":13,"state":"up:stopping","state_seq":14,"addr":"172.21.15.190:6839/1572872451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":1572872451},{"type":"v1","addr":"172.21.15.190:6839","nonce":1572872451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4282":{"gid":4282,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":12,"addr":"172.21.15.190:6837/580315269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":580315269},{"type":"v1","addr":"172.21.15.190:6837","nonce":580315269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/489092393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":489092393},{"type":"v1","addr":"172.21.15.6:6835","nonce":489092393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4306,"qdb_cluster":[4306,4282]},"id":1}]} 2024-04-24T23:23:18.365 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 38 2024-04-24T23:23:18.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:18.368+0000 7f39d9c00640 1 -- 172.21.15.6:0/2170519150 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f39b406be30 msgr2=0x7f39b406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:23:18.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:18.368+0000 7f39d9c00640 1 --2- 172.21.15.6:0/2170519150 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f39b406be30 0x7f39b406e2f0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f39c40095f0 tx=0x7f39c4006b50 comp rx=0 tx=0).stop 2024-04-24T23:23:18.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:18.368+0000 7f39d9c00640 1 -- 172.21.15.6:0/2170519150 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f39d41593b0 msgr2=0x7f39d4155e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:23:18.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:18.368+0000 7f39d9c00640 1 --2- 172.21.15.6:0/2170519150 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f39d41593b0 0x7f39d4155e00 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f39cc00c840 tx=0x7f39cc00cd00 comp rx=0 tx=0).stop 2024-04-24T23:23:18.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:18.368+0000 7f39d9c00640 1 -- 172.21.15.6:0/2170519150 shutdown_connections 2024-04-24T23:23:18.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:18.368+0000 7f39d9c00640 1 --2- 172.21.15.6:0/2170519150 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f39b406be30 0x7f39b406e2f0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:18.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:18.368+0000 7f39d9c00640 1 --2- 172.21.15.6:0/2170519150 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f39d41593b0 0x7f39d4155e00 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:18.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:18.368+0000 7f39d9c00640 1 --2- 172.21.15.6:0/2170519150 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f39d40c8390 0x7f39d41538c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:18.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:18.368+0000 7f39d9c00640 1 --2- 172.21.15.6:0/2170519150 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f39d40c6e10 0x7f39d408c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:18.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:18.368+0000 7f39d9c00640 1 -- 172.21.15.6:0/2170519150 >> 172.21.15.6:0/2170519150 conn(0x7f39d406b0e0 msgr2=0x7f39d4162720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:23:18.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:18.369+0000 7f39d9c00640 1 -- 172.21.15.6:0/2170519150 shutdown_connections 2024-04-24T23:23:18.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:18.369+0000 7f39d9c00640 1 -- 172.21.15.6:0/2170519150 wait complete. 2024-04-24T23:23:18.412 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-24T23:23:20.413 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:23:20.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:20.739+0000 7ff925e00640 1 Processor -- start 2024-04-24T23:23:20.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:20.739+0000 7ff925e00640 1 -- start start 2024-04-24T23:23:20.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:20.742+0000 7ff925e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ff92006bbd0 0x7ff920150990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:23:20.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:20.742+0000 7ff925e00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff920157470 0x7ff9200c9110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:23:20.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:20.742+0000 7ff925e00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff9200c8690 con 0x7ff920073270 2024-04-24T23:23:20.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:20.742+0000 7ff925e00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7ff9200c8c50 con 0x7ff92006bbd0 2024-04-24T23:23:20.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:20.742+0000 7ff925e00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7ff9200c8970 con 0x7ff920157470 2024-04-24T23:23:20.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:20.742+0000 7ff91e000640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7ff920073270 0x7ff92015a980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:38940/0 (socket says 172.21.15.6:38940) 2024-04-24T23:23:20.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:20.742+0000 7ff91e000640 1 -- 172.21.15.6:0/3638772551 learned_addr learned my addr 172.21.15.6:0/3638772551 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:23:20.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:20.742+0000 7ff91f400640 1 --2- 172.21.15.6:0/3638772551 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff920157470 0x7ff9200c9110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:23:20.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:20.743+0000 7ff91ea00640 1 --2- 172.21.15.6:0/3638772551 >> v2:172.21.15.6:3300/0 conn(0x7ff92006bbd0 0x7ff920150990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:23:20.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:20.743+0000 7ff91f400640 1 -- 172.21.15.6:0/3638772551 >> v2:172.21.15.6:3300/0 conn(0x7ff92006bbd0 msgr2=0x7ff920150990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:23:20.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:20.744+0000 7ff91f400640 1 --2- 172.21.15.6:0/3638772551 >> v2:172.21.15.6:3300/0 conn(0x7ff92006bbd0 0x7ff920150990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:20.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:20.744+0000 7ff91f400640 1 -- 172.21.15.6:0/3638772551 >> v1:172.21.15.6:6789/0 conn(0x7ff920073270 legacy=0x7ff92015a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:23:20.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:20.744+0000 7ff91f400640 1 -- 172.21.15.6:0/3638772551 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff92015a180 con 0x7ff920157470 2024-04-24T23:23:20.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:20.744+0000 7ff91ea00640 1 --2- 172.21.15.6:0/3638772551 >> v2:172.21.15.6:3300/0 conn(0x7ff92006bbd0 0x7ff920150990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:23:20.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:20.747+0000 7ff91f400640 1 --2- 172.21.15.6:0/3638772551 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff920157470 0x7ff9200c9110 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7ff918009d90 tx=0x7ff91802a660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6f4373ac2db22e17 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:23:20.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:20.747+0000 7ff91d600640 1 -- 172.21.15.6:0/3638772551 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff918007740 con 0x7ff920157470 2024-04-24T23:23:20.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:20.748+0000 7ff91d600640 1 -- 172.21.15.6:0/3638772551 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff9180078e0 con 0x7ff920157470 2024-04-24T23:23:20.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:20.748+0000 7ff91d600640 1 -- 172.21.15.6:0/3638772551 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff918007cb0 con 0x7ff920157470 2024-04-24T23:23:20.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:20.749+0000 7ff925e00640 1 -- 172.21.15.6:0/3638772551 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff920157470 msgr2=0x7ff9200c9110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:23:20.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:20.749+0000 7ff925e00640 1 --2- 172.21.15.6:0/3638772551 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff920157470 0x7ff9200c9110 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7ff918009d90 tx=0x7ff91802a660 comp rx=0 tx=0).stop 2024-04-24T23:23:20.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:20.750+0000 7ff925e00640 1 -- 172.21.15.6:0/3638772551 shutdown_connections 2024-04-24T23:23:20.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:20.750+0000 7ff925e00640 1 --2- 172.21.15.6:0/3638772551 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff920157470 0x7ff9200c9110 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:20.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:20.750+0000 7ff925e00640 1 --2- 172.21.15.6:0/3638772551 >> v2:172.21.15.6:3300/0 conn(0x7ff92006bbd0 0x7ff920150990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:20.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:20.750+0000 7ff925e00640 1 -- 172.21.15.6:0/3638772551 >> 172.21.15.6:0/3638772551 conn(0x7ff92006b0e0 msgr2=0x7ff92006b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:23:20.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:20.750+0000 7ff925e00640 1 -- 172.21.15.6:0/3638772551 shutdown_connections 2024-04-24T23:23:20.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:20.750+0000 7ff925e00640 1 -- 172.21.15.6:0/3638772551 wait complete. 2024-04-24T23:23:20.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:20.755+0000 7ff925e00640 1 Processor -- start 2024-04-24T23:23:20.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:20.756+0000 7ff925e00640 1 -- start start 2024-04-24T23:23:20.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:20.757+0000 7ff925e00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff92006bbd0 0x7ff920159800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:23:20.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:20.757+0000 7ff925e00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff920073270 0x7ff9201588f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:23:20.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:20.758+0000 7ff925e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff920157470 0x7ff920141fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:23:20.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:20.759+0000 7ff925e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7ff9200c45f0 con 0x7ff920157470 2024-04-24T23:23:20.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:20.759+0000 7ff925e00640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7ff9200c7a70 con 0x7ff92006bbd0 2024-04-24T23:23:20.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:20.759+0000 7ff925e00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7ff9200c4770 con 0x7ff920073270 2024-04-24T23:23:20.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:20.759+0000 7ff91e000640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff920073270 0x7ff9201588f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:23:20.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:20.760+0000 7ff91ea00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff92006bbd0 0x7ff920159800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:23:20.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:20.760+0000 7ff91e000640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff920073270 0x7ff9201588f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.6:33884/0 (socket says 172.21.15.6:33884) 2024-04-24T23:23:20.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:20.760+0000 7ff91e000640 1 -- 172.21.15.6:0/3801855000 learned_addr learned my addr 172.21.15.6:0/3801855000 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:23:20.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:20.760+0000 7ff91f400640 1 --2- 172.21.15.6:0/3801855000 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff920157470 0x7ff920141fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:23:20.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:20.761+0000 7ff91ea00640 1 -- 172.21.15.6:0/3801855000 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff920073270 msgr2=0x7ff9201588f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:23:20.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:20.761+0000 7ff91ea00640 1 --2- 172.21.15.6:0/3801855000 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff920073270 0x7ff9201588f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:20.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:20.761+0000 7ff91ea00640 1 -- 172.21.15.6:0/3801855000 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff920157470 msgr2=0x7ff920141fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:23:20.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:20.761+0000 7ff91ea00640 1 --2- 172.21.15.6:0/3801855000 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff920157470 0x7ff920141fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:20.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:20.761+0000 7ff91ea00640 1 -- 172.21.15.6:0/3801855000 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9200cbf90 con 0x7ff92006bbd0 2024-04-24T23:23:20.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:20.761+0000 7ff91e000640 1 --2- 172.21.15.6:0/3801855000 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff920073270 0x7ff9201588f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:23:20.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:20.762+0000 7ff91ea00640 1 --2- 172.21.15.6:0/3801855000 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff92006bbd0 0x7ff920159800 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7ff91000e900 tx=0x7ff91000edc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:23:20.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:20.762+0000 7ff917400640 1 -- 172.21.15.6:0/3801855000 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff910017070 con 0x7ff92006bbd0 2024-04-24T23:23:20.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:20.762+0000 7ff917400640 1 -- 172.21.15.6:0/3801855000 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff91000ca00 con 0x7ff92006bbd0 2024-04-24T23:23:20.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:20.762+0000 7ff917400640 1 -- 172.21.15.6:0/3801855000 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff91000cdf0 con 0x7ff92006bbd0 2024-04-24T23:23:20.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:20.763+0000 7ff925e00640 1 -- 172.21.15.6:0/3801855000 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9201403d0 con 0x7ff92006bbd0 2024-04-24T23:23:20.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:20.763+0000 7ff925e00640 1 -- 172.21.15.6:0/3801855000 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9200cb9d0 con 0x7ff92006bbd0 2024-04-24T23:23:20.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:20.767+0000 7ff925e00640 1 -- 172.21.15.6:0/3801855000 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff9200767b0 con 0x7ff92006bbd0 2024-04-24T23:23:20.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:20.774+0000 7ff917400640 1 -- 172.21.15.6:0/3801855000 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7ff91001a070 con 0x7ff92006bbd0 2024-04-24T23:23:20.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:20.775+0000 7ff917400640 1 --2- 172.21.15.6:0/3801855000 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7ff90006bf00 0x7ff90006e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:23:20.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:20.775+0000 7ff917400640 1 -- 172.21.15.6:0/3801855000 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6762+0+0 (secure 0 0 0) 0x7ff91008b400 con 0x7ff92006bbd0 2024-04-24T23:23:20.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:20.777+0000 7ff91e000640 1 --2- 172.21.15.6:0/3801855000 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7ff90006bf00 0x7ff90006e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:23:20.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:20.778+0000 7ff91e000640 1 --2- 172.21.15.6:0/3801855000 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7ff90006bf00 0x7ff90006e3c0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7ff9201f4d30 tx=0x7ff90c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:23:20.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:20.782+0000 7ff917400640 1 -- 172.21.15.6:0/3801855000 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7ff9100593b0 con 0x7ff92006bbd0 2024-04-24T23:23:21.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:21.386+0000 7ff925e00640 1 -- 172.21.15.6:0/3801855000 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff920158650 con 0x7ff92006bbd0 2024-04-24T23:23:21.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:21.388+0000 7ff917400640 1 -- 172.21.15.6:0/3801855000 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) ==== 94+0+6867 (secure 0 0 0) 0x7ff91005d6b0 con 0x7ff92006bbd0 2024-04-24T23:23:21.394 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:23:21.397 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":38,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5081,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/1190759626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1190759626},{"type":"v1","addr":"172.21.15.6:6834","nonce":1190759626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":5503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/3928731059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":3928731059},{"type":"v1","addr":"172.21.15.6:6837","nonce":3928731059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":35},{"gid":5551,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6838/3253046315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":3253046315},{"type":"v1","addr":"172.21.15.190:6838","nonce":3253046315}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":38}],"filesystems":[{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:23:08.146813+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1,2],"up":{"mds_0":4306,"mds_1":4282,"mds_2":4235},"failed":[],"damaged":[],"stopped":[3,4],"info":{"gid_4235":{"gid":4235,"name":"d","rank":2,"incarnation":13,"state":"up:stopping","state_seq":14,"addr":"172.21.15.190:6839/1572872451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":1572872451},{"type":"v1","addr":"172.21.15.190:6839","nonce":1572872451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4282":{"gid":4282,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":12,"addr":"172.21.15.190:6837/580315269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":580315269},{"type":"v1","addr":"172.21.15.190:6837","nonce":580315269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/489092393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":489092393},{"type":"v1","addr":"172.21.15.6:6835","nonce":489092393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4306,"qdb_cluster":[4306,4282]},"id":1}]} 2024-04-24T23:23:21.397 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 38 2024-04-24T23:23:21.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:21.397+0000 7ff925e00640 1 -- 172.21.15.6:0/3801855000 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7ff90006bf00 msgr2=0x7ff90006e3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:23:21.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:21.398+0000 7ff925e00640 1 --2- 172.21.15.6:0/3801855000 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7ff90006bf00 0x7ff90006e3c0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7ff9201f4d30 tx=0x7ff90c009040 comp rx=0 tx=0).stop 2024-04-24T23:23:21.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:21.399+0000 7ff925e00640 1 -- 172.21.15.6:0/3801855000 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff92006bbd0 msgr2=0x7ff920159800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:23:21.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:21.399+0000 7ff925e00640 1 --2- 172.21.15.6:0/3801855000 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff92006bbd0 0x7ff920159800 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7ff91000e900 tx=0x7ff91000edc0 comp rx=0 tx=0).stop 2024-04-24T23:23:21.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:21.400+0000 7ff925e00640 1 -- 172.21.15.6:0/3801855000 shutdown_connections 2024-04-24T23:23:21.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:21.400+0000 7ff925e00640 1 --2- 172.21.15.6:0/3801855000 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7ff90006bf00 0x7ff90006e3c0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:21.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:21.400+0000 7ff925e00640 1 --2- 172.21.15.6:0/3801855000 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff920073270 0x7ff9201588f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:21.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:21.400+0000 7ff925e00640 1 --2- 172.21.15.6:0/3801855000 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff920157470 0x7ff920141fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:21.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:21.400+0000 7ff925e00640 1 --2- 172.21.15.6:0/3801855000 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff92006bbd0 0x7ff920159800 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:21.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:21.400+0000 7ff925e00640 1 -- 172.21.15.6:0/3801855000 >> 172.21.15.6:0/3801855000 conn(0x7ff92006b0e0 msgr2=0x7ff920070f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:23:21.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:21.401+0000 7ff925e00640 1 -- 172.21.15.6:0/3801855000 shutdown_connections 2024-04-24T23:23:21.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:21.404+0000 7ff925e00640 1 -- 172.21.15.6:0/3801855000 wait complete. 2024-04-24T23:23:21.460 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-24T23:23:23.461 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:23:23.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:23.852+0000 7fb3ef800640 1 Processor -- start 2024-04-24T23:23:23.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:23.853+0000 7fb3ef800640 1 -- start start 2024-04-24T23:23:23.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:23.855+0000 7fb3ef800640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fb3e815b510 0x7fb3e80c9da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:23:23.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:23.855+0000 7fb3ece00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fb3e8154a20 0x7fb3e80cabc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:38948/0 (socket says 172.21.15.6:38948) 2024-04-24T23:23:23.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:23.855+0000 7fb3ece00640 1 -- 172.21.15.6:0/164537711 learned_addr learned my addr 172.21.15.6:0/164537711 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:23:23.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:23.856+0000 7fb3ef800640 1 -- 172.21.15.6:0/164537711 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb3e80c8cc0 con 0x7fb3e8154a20 2024-04-24T23:23:23.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:23.856+0000 7fb3ef800640 1 -- 172.21.15.6:0/164537711 --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb3e80ca3e0 con 0x7fb3e806fdd0 2024-04-24T23:23:23.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:23.856+0000 7fb3ef800640 1 -- 172.21.15.6:0/164537711 --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7fb3e80ca9a0 con 0x7fb3e815b510 2024-04-24T23:23:23.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:23.856+0000 7fb3e7e00640 1 --2- 172.21.15.6:0/164537711 >> v2:172.21.15.190:3300/0 conn(0x7fb3e815b510 0x7fb3e80c9da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:23:23.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:23.859+0000 7fb3e7400640 1 -- 172.21.15.6:0/164537711 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 851399599 0 0) 0x7fb3e80ca3e0 con 0x7fb3e806fdd0 2024-04-24T23:23:23.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:23.859+0000 7fb3e7400640 1 -- 172.21.15.6:0/164537711 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb3d0003520 con 0x7fb3e806fdd0 2024-04-24T23:23:23.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:23.860+0000 7fb3e7400640 1 -- 172.21.15.6:0/164537711 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 458193628 0 0) 0x7fb3e80c8cc0 con 0x7fb3e8154a20 2024-04-24T23:23:23.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:23.860+0000 7fb3e7400640 1 -- 172.21.15.6:0/164537711 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb3e80ca3e0 con 0x7fb3e8154a20 2024-04-24T23:23:23.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:23.860+0000 7fb3e7400640 1 -- 172.21.15.6:0/164537711 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 322429665 0 0) 0x7fb3d0003520 con 0x7fb3e806fdd0 2024-04-24T23:23:23.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:23.860+0000 7fb3e7400640 1 -- 172.21.15.6:0/164537711 >> v2:172.21.15.190:3300/0 conn(0x7fb3e815b510 msgr2=0x7fb3e80c9da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:23:23.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:23.860+0000 7fb3e7400640 1 --2- 172.21.15.6:0/164537711 >> v2:172.21.15.190:3300/0 conn(0x7fb3e815b510 0x7fb3e80c9da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:23.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:23.860+0000 7fb3e7400640 1 -- 172.21.15.6:0/164537711 >> v1:172.21.15.6:6789/0 conn(0x7fb3e8154a20 legacy=0x7fb3e80cabc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:23:23.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:23.860+0000 7fb3e7400640 1 -- 172.21.15.6:0/164537711 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3e80ca6c0 con 0x7fb3e806fdd0 2024-04-24T23:23:23.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:23.860+0000 7fb3e7400640 1 -- 172.21.15.6:0/164537711 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3463268701 0 0) 0x7fb3e0001950 con 0x7fb3e806fdd0 2024-04-24T23:23:23.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:23.860+0000 7fb3e7400640 1 -- 172.21.15.6:0/164537711 >> v1:172.21.15.190:6789/0 conn(0x7fb3e806fdd0 legacy=0x7fb3e81651a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:23:23.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:23.860+0000 7fb3e7400640 1 --2- 172.21.15.6:0/164537711 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb3d0004bf0 0x7fb3d0004ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:23:23.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:23.860+0000 7fb3e7400640 1 --2- 172.21.15.6:0/164537711 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb3d00037a0 0x7fb3d0003c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:23:23.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:23.860+0000 7fb3e7400640 1 --2- 172.21.15.6:0/164537711 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb3d0004160 0x7fb3d000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:23:23.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:23.860+0000 7fb3e7400640 1 -- 172.21.15.6:0/164537711 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fb3e80ca9a0 con 0x7fb3d0004bf0 2024-04-24T23:23:23.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:23.860+0000 7fb3e7400640 1 -- 172.21.15.6:0/164537711 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fb3d000bfe0 con 0x7fb3d00037a0 2024-04-24T23:23:23.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:23.860+0000 7fb3e7400640 1 -- 172.21.15.6:0/164537711 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fb3d000c1e0 con 0x7fb3d0004160 2024-04-24T23:23:23.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:23.862+0000 7fb3ed800640 1 --2- 172.21.15.6:0/164537711 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb3d0004160 0x7fb3d000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:23:23.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:23.862+0000 7fb3e7e00640 1 --2- 172.21.15.6:0/164537711 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb3d00037a0 0x7fb3d0003c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:23:23.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:23.862+0000 7fb3ece00640 1 --2- 172.21.15.6:0/164537711 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb3d0004bf0 0x7fb3d0004ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:23:23.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:23.863+0000 7fb3ed800640 1 -- 172.21.15.6:0/164537711 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb3d00037a0 msgr2=0x7fb3d0003c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:23:23.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:23.863+0000 7fb3ed800640 1 --2- 172.21.15.6:0/164537711 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb3d00037a0 0x7fb3d0003c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:23.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:23.863+0000 7fb3ed800640 1 -- 172.21.15.6:0/164537711 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb3d0004bf0 msgr2=0x7fb3d0004ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:23:23.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:23.864+0000 7fb3ed800640 1 --2- 172.21.15.6:0/164537711 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb3d0004bf0 0x7fb3d0004ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:23.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:23.864+0000 7fb3ed800640 1 -- 172.21.15.6:0/164537711 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3e80ca6c0 con 0x7fb3d0004160 2024-04-24T23:23:23.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:23.864+0000 7fb3ece00640 1 --2- 172.21.15.6:0/164537711 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb3d0004bf0 0x7fb3d0004ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:23:23.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:23.866+0000 7fb3ed800640 1 -- 172.21.15.6:0/164537711 reap_dead start 2024-04-24T23:23:23.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:23.866+0000 7fb3ed800640 1 --2- 172.21.15.6:0/164537711 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb3d0004160 0x7fb3d000b9c0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fb3e000c510 tx=0x7fb3d0005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=887d03ea6d2678d5 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:23:23.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:23.867+0000 7fb3e7400640 1 -- 172.21.15.6:0/164537711 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3e000c0e0 con 0x7fb3d0004160 2024-04-24T23:23:23.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:23.867+0000 7fb3e7400640 1 -- 172.21.15.6:0/164537711 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb3e002e610 con 0x7fb3d0004160 2024-04-24T23:23:23.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:23.867+0000 7fb3e7400640 1 -- 172.21.15.6:0/164537711 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3e002ea00 con 0x7fb3d0004160 2024-04-24T23:23:23.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:23.868+0000 7fb3ef800640 1 -- 172.21.15.6:0/164537711 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb3d0004160 msgr2=0x7fb3d000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:23:23.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:23.868+0000 7fb3ef800640 1 --2- 172.21.15.6:0/164537711 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb3d0004160 0x7fb3d000b9c0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fb3e000c510 tx=0x7fb3d0005830 comp rx=0 tx=0).stop 2024-04-24T23:23:23.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:23.871+0000 7fb3ef800640 1 -- 172.21.15.6:0/164537711 shutdown_connections 2024-04-24T23:23:23.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:23.871+0000 7fb3ef800640 1 --2- 172.21.15.6:0/164537711 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb3d0004160 0x7fb3d000b9c0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:23.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:23.871+0000 7fb3ef800640 1 -- 172.21.15.6:0/164537711 >> 172.21.15.6:0/164537711 conn(0x7fb3e8070a60 msgr2=0x7fb3e8070e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:23:23.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:23.871+0000 7fb3ef800640 1 -- 172.21.15.6:0/164537711 shutdown_connections 2024-04-24T23:23:23.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:23.872+0000 7fb3ef800640 1 -- 172.21.15.6:0/164537711 wait complete. 2024-04-24T23:23:23.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:23.875+0000 7fb3ef800640 1 Processor -- start 2024-04-24T23:23:23.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:23.876+0000 7fb3ef800640 1 -- start start 2024-04-24T23:23:23.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:23.878+0000 7fb3ef800640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb3d0004160 0x7fb3e80cd5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:23:23.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:23.878+0000 7fb3ef800640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb3e814d5a0 0x7fb3e814da00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:23:23.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:23.878+0000 7fb3ece00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb3d0004160 0x7fb3e80cd5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:23:23.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:23.879+0000 7fb3ece00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb3d0004160 0x7fb3e80cd5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:44072/0 (socket says 172.21.15.6:44072) 2024-04-24T23:23:23.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:23.879+0000 7fb3ece00640 1 -- 172.21.15.6:0/3435450571 learned_addr learned my addr 172.21.15.6:0/3435450571 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:23:23.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:23.879+0000 7fb3ef800640 1 --2- 172.21.15.6:0/3435450571 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb3e81f0140 0x7fb3e81f05f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:23:23.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:23.879+0000 7fb3ef800640 1 -- 172.21.15.6:0/3435450571 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fb3e815ad10 con 0x7fb3e81f0140 2024-04-24T23:23:23.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:23.879+0000 7fb3ef800640 1 -- 172.21.15.6:0/3435450571 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fb3e815b080 con 0x7fb3d0004160 2024-04-24T23:23:23.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:23.880+0000 7fb3ef800640 1 -- 172.21.15.6:0/3435450571 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fb3e8154e00 con 0x7fb3e814d5a0 2024-04-24T23:23:23.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:23.880+0000 7fb3e7e00640 1 --2- 172.21.15.6:0/3435450571 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb3e814d5a0 0x7fb3e814da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:23:23.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:23.884+0000 7fb3ece00640 1 -- 172.21.15.6:0/3435450571 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb3e814d5a0 msgr2=0x7fb3e814da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:23:23.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:23.884+0000 7fb3ece00640 1 --2- 172.21.15.6:0/3435450571 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb3e814d5a0 0x7fb3e814da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:23.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:23.884+0000 7fb3ece00640 1 -- 172.21.15.6:0/3435450571 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb3e81f0140 msgr2=0x7fb3e81f05f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:23:23.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:23.884+0000 7fb3ed800640 1 --2- 172.21.15.6:0/3435450571 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb3e81f0140 0x7fb3e81f05f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:23:23.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:23.885+0000 7fb3ece00640 1 --2- 172.21.15.6:0/3435450571 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb3e81f0140 0x7fb3e81f05f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:23.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:23.885+0000 7fb3ece00640 1 -- 172.21.15.6:0/3435450571 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3e81f3670 con 0x7fb3d0004160 2024-04-24T23:23:23.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:23.885+0000 7fb3ed800640 1 --2- 172.21.15.6:0/3435450571 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb3e81f0140 0x7fb3e81f05f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:23:23.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:23.886+0000 7fb3ece00640 1 --2- 172.21.15.6:0/3435450571 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb3d0004160 0x7fb3e80cd5a0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fb3dc007290 tx=0x7fb3dc007750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:23:23.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:23.886+0000 7fb3e4c00640 1 -- 172.21.15.6:0/3435450571 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3dc0015c0 con 0x7fb3d0004160 2024-04-24T23:23:23.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:23.889+0000 7fb3e4c00640 1 -- 172.21.15.6:0/3435450571 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb3dc018070 con 0x7fb3d0004160 2024-04-24T23:23:23.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:23.890+0000 7fb3ef800640 1 -- 172.21.15.6:0/3435450571 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3e81f3950 con 0x7fb3d0004160 2024-04-24T23:23:23.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:23.890+0000 7fb3ef800640 1 -- 172.21.15.6:0/3435450571 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3e81f3f10 con 0x7fb3d0004160 2024-04-24T23:23:23.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:23.890+0000 7fb3e4c00640 1 -- 172.21.15.6:0/3435450571 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3dc00fb40 con 0x7fb3d0004160 2024-04-24T23:23:23.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:23.894+0000 7fb3da000640 1 -- 172.21.15.6:0/3435450571 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb3bc005290 con 0x7fb3d0004160 2024-04-24T23:23:23.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:23.898+0000 7fb3e4c00640 1 -- 172.21.15.6:0/3435450571 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fb3dc00fce0 con 0x7fb3d0004160 2024-04-24T23:23:23.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:23.899+0000 7fb3e4c00640 1 --2- 172.21.15.6:0/3435450571 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fb3c806be30 0x7fb3c806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:23:23.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:23.899+0000 7fb3e4c00640 1 -- 172.21.15.6:0/3435450571 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6762+0+0 (secure 0 0 0) 0x7fb3dc08b080 con 0x7fb3d0004160 2024-04-24T23:23:23.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:23.900+0000 7fb3e7e00640 1 --2- 172.21.15.6:0/3435450571 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fb3c806be30 0x7fb3c806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:23:23.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:23.902+0000 7fb3e7e00640 1 --2- 172.21.15.6:0/3435450571 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fb3c806be30 0x7fb3c806e2f0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fb3e8071190 tx=0x7fb3d4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:23:23.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:23.903+0000 7fb3e4c00640 1 -- 172.21.15.6:0/3435450571 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fb3dc058fb0 con 0x7fb3d0004160 2024-04-24T23:23:24.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:24.485+0000 7fb3da000640 1 -- 172.21.15.6:0/3435450571 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb3bc005580 con 0x7fb3d0004160 2024-04-24T23:23:24.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:24.490+0000 7fb3e4c00640 1 -- 172.21.15.6:0/3435450571 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) ==== 94+0+6867 (secure 0 0 0) 0x7fb3dc05d2b0 con 0x7fb3d0004160 2024-04-24T23:23:24.496 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:23:24.496 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":38,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5081,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/1190759626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1190759626},{"type":"v1","addr":"172.21.15.6:6834","nonce":1190759626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":5503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/3928731059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":3928731059},{"type":"v1","addr":"172.21.15.6:6837","nonce":3928731059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":35},{"gid":5551,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6838/3253046315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":3253046315},{"type":"v1","addr":"172.21.15.190:6838","nonce":3253046315}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":38}],"filesystems":[{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:23:08.146813+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1,2],"up":{"mds_0":4306,"mds_1":4282,"mds_2":4235},"failed":[],"damaged":[],"stopped":[3,4],"info":{"gid_4235":{"gid":4235,"name":"d","rank":2,"incarnation":13,"state":"up:stopping","state_seq":14,"addr":"172.21.15.190:6839/1572872451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":1572872451},{"type":"v1","addr":"172.21.15.190:6839","nonce":1572872451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4282":{"gid":4282,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":12,"addr":"172.21.15.190:6837/580315269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":580315269},{"type":"v1","addr":"172.21.15.190:6837","nonce":580315269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/489092393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":489092393},{"type":"v1","addr":"172.21.15.6:6835","nonce":489092393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4306,"qdb_cluster":[4306,4282]},"id":1}]} 2024-04-24T23:23:24.497 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 38 2024-04-24T23:23:24.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:24.497+0000 7fb3da000640 1 -- 172.21.15.6:0/3435450571 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fb3c806be30 msgr2=0x7fb3c806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:23:24.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:24.497+0000 7fb3da000640 1 --2- 172.21.15.6:0/3435450571 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fb3c806be30 0x7fb3c806e2f0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fb3e8071190 tx=0x7fb3d4008040 comp rx=0 tx=0).stop 2024-04-24T23:23:24.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:24.497+0000 7fb3da000640 1 -- 172.21.15.6:0/3435450571 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb3d0004160 msgr2=0x7fb3e80cd5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:23:24.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:24.497+0000 7fb3da000640 1 --2- 172.21.15.6:0/3435450571 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb3d0004160 0x7fb3e80cd5a0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fb3dc007290 tx=0x7fb3dc007750 comp rx=0 tx=0).stop 2024-04-24T23:23:24.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:24.498+0000 7fb3da000640 1 -- 172.21.15.6:0/3435450571 shutdown_connections 2024-04-24T23:23:24.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:24.498+0000 7fb3da000640 1 --2- 172.21.15.6:0/3435450571 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fb3c806be30 0x7fb3c806e2f0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:24.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:24.498+0000 7fb3da000640 1 --2- 172.21.15.6:0/3435450571 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb3e814d5a0 0x7fb3e814da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:24.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:24.498+0000 7fb3da000640 1 --2- 172.21.15.6:0/3435450571 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb3e81f0140 0x7fb3e81f05f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:24.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:24.498+0000 7fb3da000640 1 --2- 172.21.15.6:0/3435450571 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb3d0004160 0x7fb3e80cd5a0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:24.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:24.498+0000 7fb3da000640 1 -- 172.21.15.6:0/3435450571 >> 172.21.15.6:0/3435450571 conn(0x7fb3e8070a60 msgr2=0x7fb3e806f300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:23:24.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:24.500+0000 7fb3da000640 1 -- 172.21.15.6:0/3435450571 shutdown_connections 2024-04-24T23:23:24.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:24.500+0000 7fb3da000640 1 -- 172.21.15.6:0/3435450571 wait complete. 2024-04-24T23:23:24.537 INFO:tasks.mds_thrash.fs.[cephfs]:cluster is considered unstable while MDS are in up:stopping (!thrash_while_stopping) 2024-04-24T23:23:24.537 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 38, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-24T23:15:36.400716+0000", "damaged": [], "data_pools": [ 3, 4 ], "enabled": true, "epoch": 37, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_4235": { "addr": "172.21.15.190:6839/1572872451", "addrs": { "addrvec": [ { "addr": "172.21.15.190:6836", "nonce": 1572872451, "type": "v2" }, { "addr": "172.21.15.190:6839", "nonce": 1572872451, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4235, "incarnation": 13, "join_fscid": -1, "name": "d", "rank": 2, "state": "up:stopping", "state_seq": 14 }, "gid_4282": { "addr": "172.21.15.190:6837/580315269", "addrs": { "addrvec": [ { "addr": "172.21.15.190:6834", "nonce": 580315269, "type": "v2" }, { "addr": "172.21.15.190:6837", "nonce": 580315269, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4282, "incarnation": 9, "join_fscid": -1, "name": "f", "rank": 1, "state": "up:active", "state_seq": 12 }, "gid_4306": { "addr": "172.21.15.6:6835/489092393", "addrs": { "addrvec": [ { "addr": "172.21.15.6:6833", "nonce": 489092393, "type": "v2" }, { "addr": "172.21.15.6:6835", "nonce": 489092393, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4306, "incarnation": 4, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 4 } }, "last_failure": 0, "last_failure_osd_epoch": 28, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-04-24T23:23:08.146813+0000", "qdb_cluster": [ 4306, 4282 ], "qdb_leader": 4306, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 3, 4 ], "tableserver": 0, "up": { "mds_0": 4306, "mds_1": 4282, "mds_2": 4235 } } } ], "standbys": [ { "addr": "172.21.15.6:6834/1190759626", "addrs": { "addrvec": [ { "addr": "172.21.15.6:6832", "nonce": 1190759626, "type": "v2" }, { "addr": "172.21.15.6:6834", "nonce": 1190759626, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 30, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5081, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.6:6837/3928731059", "addrs": { "addrvec": [ { "addr": "172.21.15.6:6836", "nonce": 3928731059, "type": "v2" }, { "addr": "172.21.15.6:6837", "nonce": 3928731059, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 35, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5503, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.190:6838/3253046315", "addrs": { "addrvec": [ { "addr": "172.21.15.190:6835", "nonce": 3253046315, "type": "v2" }, { "addr": "172.21.15.190:6838", "nonce": 3253046315, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 38, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5551, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-04-24T23:23:26.538 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:23:26.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:26.854+0000 7f79e7c00640 1 Processor -- start 2024-04-24T23:23:26.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:26.854+0000 7f79e7c00640 1 -- start start 2024-04-24T23:23:26.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:26.856+0000 7f79e7c00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f79e0154b70 0x7f79e006f020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:23:26.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:26.856+0000 7f79e7c00640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f79e00c7080 0x7f79e0071000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:23:26.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:26.856+0000 7f79e7c00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f79e00c8570 con 0x7f79e00c80e0 2024-04-24T23:23:26.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:26.856+0000 7f79e7c00640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f79e0072a70 con 0x7f79e00c7080 2024-04-24T23:23:26.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:26.856+0000 7f79e7c00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f79e0072530 con 0x7f79e0154b70 2024-04-24T23:23:26.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:26.856+0000 7f79dfe00640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f79e00c7080 0x7f79e0071000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:23:26.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:26.857+0000 7f79e5200640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f79e0154b70 0x7f79e006f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:23:26.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:26.857+0000 7f79e5200640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f79e0154b70 0x7f79e006f020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.6:37620/0 (socket says 172.21.15.6:37620) 2024-04-24T23:23:26.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:26.857+0000 7f79e5200640 1 -- 172.21.15.6:0/833377239 learned_addr learned my addr 172.21.15.6:0/833377239 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:23:26.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:26.857+0000 7f79e5200640 1 -- 172.21.15.6:0/833377239 >> v2:172.21.15.190:3300/0 conn(0x7f79e00c7080 msgr2=0x7f79e0071000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:23:26.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:26.857+0000 7f79e5200640 1 --2- 172.21.15.6:0/833377239 >> v2:172.21.15.190:3300/0 conn(0x7f79e00c7080 0x7f79e0071000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:26.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:26.857+0000 7f79e5200640 1 -- 172.21.15.6:0/833377239 >> v1:172.21.15.6:6789/0 conn(0x7f79e00c80e0 legacy=0x7f79e015a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:23:26.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:26.857+0000 7f79e5200640 1 -- 172.21.15.6:0/833377239 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79e00727d0 con 0x7f79e0154b70 2024-04-24T23:23:26.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:26.859+0000 7f79e5200640 1 --2- 172.21.15.6:0/833377239 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f79e0154b70 0x7f79e006f020 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f79d4009870 tx=0x7f79d402a270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1c25eaec0a2d0abe server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:23:26.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:26.859+0000 7f79df400640 1 -- 172.21.15.6:0/833377239 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79d4037070 con 0x7f79e0154b70 2024-04-24T23:23:26.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:26.859+0000 7f79df400640 1 -- 172.21.15.6:0/833377239 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f79d4032440 con 0x7f79e0154b70 2024-04-24T23:23:26.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:26.859+0000 7f79df400640 1 -- 172.21.15.6:0/833377239 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79d40327f0 con 0x7f79e0154b70 2024-04-24T23:23:26.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:26.860+0000 7f79e7c00640 1 -- 172.21.15.6:0/833377239 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f79e0154b70 msgr2=0x7f79e006f020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:23:26.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:26.860+0000 7f79e7c00640 1 --2- 172.21.15.6:0/833377239 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f79e0154b70 0x7f79e006f020 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f79d4009870 tx=0x7f79d402a270 comp rx=0 tx=0).stop 2024-04-24T23:23:26.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:26.860+0000 7f79e7c00640 1 -- 172.21.15.6:0/833377239 shutdown_connections 2024-04-24T23:23:26.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:26.860+0000 7f79e7c00640 1 --2- 172.21.15.6:0/833377239 >> v2:172.21.15.190:3300/0 conn(0x7f79e00c7080 0x7f79e0071000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:26.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:26.860+0000 7f79e7c00640 1 --2- 172.21.15.6:0/833377239 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f79e0154b70 0x7f79e006f020 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:26.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:26.860+0000 7f79e7c00640 1 -- 172.21.15.6:0/833377239 >> 172.21.15.6:0/833377239 conn(0x7f79e006b0e0 msgr2=0x7f79e006b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:23:26.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:26.860+0000 7f79e7c00640 1 -- 172.21.15.6:0/833377239 shutdown_connections 2024-04-24T23:23:26.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:26.861+0000 7f79e7c00640 1 -- 172.21.15.6:0/833377239 wait complete. 2024-04-24T23:23:26.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:26.863+0000 7f79e7c00640 1 Processor -- start 2024-04-24T23:23:26.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:26.863+0000 7f79e7c00640 1 -- start start 2024-04-24T23:23:26.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:26.865+0000 7f79e7c00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f79e00c7080 0x7f79e00cb4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:23:26.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:26.865+0000 7f79e7c00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f79e00c80e0 0x7f79e01f1230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:23:26.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:26.866+0000 7f79e7c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f79e0154b70 0x7f79e015cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:23:26.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:26.866+0000 7f79e7c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f79e006eb00 con 0x7f79e0154b70 2024-04-24T23:23:26.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:26.866+0000 7f79e7c00640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f79e006e820 con 0x7f79e00c80e0 2024-04-24T23:23:26.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:26.866+0000 7f79e7c00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f79e0154f50 con 0x7f79e00c7080 2024-04-24T23:23:26.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:26.866+0000 7f79e5200640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f79e00c7080 0x7f79e00cb4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:23:26.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:26.866+0000 7f79e5200640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f79e00c7080 0x7f79e00cb4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.6:37628/0 (socket says 172.21.15.6:37628) 2024-04-24T23:23:26.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:26.866+0000 7f79e5200640 1 -- 172.21.15.6:0/3449132985 learned_addr learned my addr 172.21.15.6:0/3449132985 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:23:26.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:26.866+0000 7f79dfe00640 1 --2- 172.21.15.6:0/3449132985 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f79e00c80e0 0x7f79e01f1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:23:26.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:26.866+0000 7f79e5c00640 1 --2- 172.21.15.6:0/3449132985 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f79e0154b70 0x7f79e015cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:23:26.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:26.868+0000 7f79e5c00640 1 -- 172.21.15.6:0/3449132985 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f79e00c7080 msgr2=0x7f79e00cb4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:23:26.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:26.868+0000 7f79e5c00640 1 --2- 172.21.15.6:0/3449132985 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f79e00c7080 0x7f79e00cb4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:26.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:26.868+0000 7f79e5c00640 1 -- 172.21.15.6:0/3449132985 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f79e00c80e0 msgr2=0x7f79e01f1230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:23:26.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:26.868+0000 7f79e5c00640 1 --2- 172.21.15.6:0/3449132985 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f79e00c80e0 0x7f79e01f1230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:26.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:26.868+0000 7f79e5c00640 1 -- 172.21.15.6:0/3449132985 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79e00c4050 con 0x7f79e0154b70 2024-04-24T23:23:26.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:26.868+0000 7f79e5c00640 1 --2- 172.21.15.6:0/3449132985 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f79e0154b70 0x7f79e015cdc0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f79d8002180 tx=0x7f79d800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:23:26.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:26.869+0000 7f79dcc00640 1 -- 172.21.15.6:0/3449132985 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79d8006d50 con 0x7f79e0154b70 2024-04-24T23:23:26.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:26.869+0000 7f79dcc00640 1 -- 172.21.15.6:0/3449132985 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f79d8003970 con 0x7f79e0154b70 2024-04-24T23:23:26.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:26.869+0000 7f79dcc00640 1 -- 172.21.15.6:0/3449132985 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79d8003d60 con 0x7f79e0154b70 2024-04-24T23:23:26.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:26.869+0000 7f79e7c00640 1 -- 172.21.15.6:0/3449132985 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79e015bc30 con 0x7f79e0154b70 2024-04-24T23:23:26.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:26.870+0000 7f79e7c00640 1 -- 172.21.15.6:0/3449132985 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79e015bef0 con 0x7f79e0154b70 2024-04-24T23:23:26.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:26.871+0000 7f79e7c00640 1 -- 172.21.15.6:0/3449132985 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f79e0150ac0 con 0x7f79e0154b70 2024-04-24T23:23:26.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:26.874+0000 7f79dcc00640 1 -- 172.21.15.6:0/3449132985 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f79d8015960 con 0x7f79e0154b70 2024-04-24T23:23:26.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:26.878+0000 7f79dcc00640 1 --2- 172.21.15.6:0/3449132985 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f79c006beb0 0x7f79c006e370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:23:26.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:26.878+0000 7f79dcc00640 1 -- 172.21.15.6:0/3449132985 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6762+0+0 (secure 0 0 0) 0x7f79d80931c0 con 0x7f79e0154b70 2024-04-24T23:23:26.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:26.881+0000 7f79e5200640 1 --2- 172.21.15.6:0/3449132985 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f79c006beb0 0x7f79c006e370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:23:26.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:26.882+0000 7f79e5200640 1 --2- 172.21.15.6:0/3449132985 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f79c006beb0 0x7f79c006e370 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f79d4009870 tx=0x7f79d402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:23:26.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:26.885+0000 7f79dcc00640 1 -- 172.21.15.6:0/3449132985 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f79e0150ac0 con 0x7f79e0154b70 2024-04-24T23:23:27.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:27.429+0000 7f79e7c00640 1 -- 172.21.15.6:0/3449132985 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f79e00c7970 con 0x7f79e0154b70 2024-04-24T23:23:27.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:27.431+0000 7f79dcc00640 1 -- 172.21.15.6:0/3449132985 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) ==== 94+0+6867 (secure 0 0 0) 0x7f79d8061060 con 0x7f79e0154b70 2024-04-24T23:23:27.433 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:23:27.433 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":38,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5081,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/1190759626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1190759626},{"type":"v1","addr":"172.21.15.6:6834","nonce":1190759626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":5503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/3928731059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":3928731059},{"type":"v1","addr":"172.21.15.6:6837","nonce":3928731059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":35},{"gid":5551,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6838/3253046315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":3253046315},{"type":"v1","addr":"172.21.15.190:6838","nonce":3253046315}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":38}],"filesystems":[{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:23:08.146813+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1,2],"up":{"mds_0":4306,"mds_1":4282,"mds_2":4235},"failed":[],"damaged":[],"stopped":[3,4],"info":{"gid_4235":{"gid":4235,"name":"d","rank":2,"incarnation":13,"state":"up:stopping","state_seq":14,"addr":"172.21.15.190:6839/1572872451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":1572872451},{"type":"v1","addr":"172.21.15.190:6839","nonce":1572872451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4282":{"gid":4282,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":12,"addr":"172.21.15.190:6837/580315269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":580315269},{"type":"v1","addr":"172.21.15.190:6837","nonce":580315269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/489092393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":489092393},{"type":"v1","addr":"172.21.15.6:6835","nonce":489092393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4306,"qdb_cluster":[4306,4282]},"id":1}]} 2024-04-24T23:23:27.434 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 38 2024-04-24T23:23:27.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:27.433+0000 7f79e7c00640 1 -- 172.21.15.6:0/3449132985 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f79c006beb0 msgr2=0x7f79c006e370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:23:27.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:27.433+0000 7f79e7c00640 1 --2- 172.21.15.6:0/3449132985 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f79c006beb0 0x7f79c006e370 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f79d4009870 tx=0x7f79d402e000 comp rx=0 tx=0).stop 2024-04-24T23:23:27.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:27.433+0000 7f79e7c00640 1 -- 172.21.15.6:0/3449132985 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f79e0154b70 msgr2=0x7f79e015cdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:23:27.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:27.433+0000 7f79e7c00640 1 --2- 172.21.15.6:0/3449132985 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f79e0154b70 0x7f79e015cdc0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f79d8002180 tx=0x7f79d800cbf0 comp rx=0 tx=0).stop 2024-04-24T23:23:27.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:27.434+0000 7f79e7c00640 1 -- 172.21.15.6:0/3449132985 shutdown_connections 2024-04-24T23:23:27.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:27.434+0000 7f79e7c00640 1 --2- 172.21.15.6:0/3449132985 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f79c006beb0 0x7f79c006e370 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:27.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:27.434+0000 7f79e7c00640 1 --2- 172.21.15.6:0/3449132985 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f79e0154b70 0x7f79e015cdc0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:27.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:27.434+0000 7f79e7c00640 1 --2- 172.21.15.6:0/3449132985 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f79e00c80e0 0x7f79e01f1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:27.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:27.434+0000 7f79e7c00640 1 --2- 172.21.15.6:0/3449132985 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f79e00c7080 0x7f79e00cb4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:27.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:27.434+0000 7f79e7c00640 1 -- 172.21.15.6:0/3449132985 >> 172.21.15.6:0/3449132985 conn(0x7f79e006b0e0 msgr2=0x7f79e014ff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:23:27.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:27.435+0000 7f79e7c00640 1 -- 172.21.15.6:0/3449132985 shutdown_connections 2024-04-24T23:23:27.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:27.436+0000 7f79e7c00640 1 -- 172.21.15.6:0/3449132985 wait complete. 2024-04-24T23:23:27.488 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-24T23:23:29.489 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:23:29.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:29.990+0000 7f2751a00640 1 Processor -- start 2024-04-24T23:23:29.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:29.991+0000 7f2751a00640 1 -- start start 2024-04-24T23:23:29.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:29.992+0000 7f2751a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f274c152900 0x7f274c071fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:23:29.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:29.993+0000 7f2751a00640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f274c06ba60 0x7f274c158d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:23:29.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:29.994+0000 7f274be00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f274c152900 0x7f274c071fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:23:29.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:29.994+0000 7f274be00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f274c152900 0x7f274c071fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:49330/0 (socket says 172.21.15.6:49330) 2024-04-24T23:23:29.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:29.994+0000 7f274be00640 1 -- 172.21.15.6:0/2054671594 learned_addr learned my addr 172.21.15.6:0/2054671594 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:23:29.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:29.995+0000 7f2751000640 1 --2- 172.21.15.6:0/2054671594 >> v2:172.21.15.190:3300/0 conn(0x7f274c06ba60 0x7f274c158d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:23:29.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:29.993+0000 7f2751a00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f274c06c010 con 0x7f274c0c7d00 2024-04-24T23:23:29.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:29.996+0000 7f2751a00640 1 -- 172.21.15.6:0/2054671594 --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f274c158870 con 0x7f274c152900 2024-04-24T23:23:29.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:29.997+0000 7f2751a00640 1 -- 172.21.15.6:0/2054671594 --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f274c0732c0 con 0x7f274c06ba60 2024-04-24T23:23:29.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:29.997+0000 7f274aa00640 1 -- 172.21.15.6:0/2054671594 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4056750450 0 0) 0x7f273c001110 con 0x7f274c0c7d00 2024-04-24T23:23:29.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:29.997+0000 7f274aa00640 1 -- 172.21.15.6:0/2054671594 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2730003520 con 0x7f274c0c7d00 2024-04-24T23:23:30.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:30.000+0000 7f2751000640 1 -- 172.21.15.6:0/2054671594 >> v2:172.21.15.6:3300/0 conn(0x7f274c152900 msgr2=0x7f274c071fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:23:30.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:30.000+0000 7f2751000640 1 --2- 172.21.15.6:0/2054671594 >> v2:172.21.15.6:3300/0 conn(0x7f274c152900 0x7f274c071fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:30.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:30.000+0000 7f2751000640 1 -- 172.21.15.6:0/2054671594 >> v1:172.21.15.6:6789/0 conn(0x7f274c0c7d00 legacy=0x7f274c0c6a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:23:30.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:30.000+0000 7f2751000640 1 -- 172.21.15.6:0/2054671594 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f274c072a20 con 0x7f274c06ba60 2024-04-24T23:23:30.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:30.002+0000 7f2751000640 1 --2- 172.21.15.6:0/2054671594 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f274c06ba60 0x7f274c158d00 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f274c0c95d0 tx=0x7f274402ea40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7b30a2417f18320a server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:23:30.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:30.002+0000 7f274aa00640 1 -- 172.21.15.6:0/2054671594 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2744032800 con 0x7f274c06ba60 2024-04-24T23:23:30.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:30.003+0000 7f274aa00640 1 -- 172.21.15.6:0/2054671594 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f27440329a0 con 0x7f274c06ba60 2024-04-24T23:23:30.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:30.003+0000 7f274aa00640 1 -- 172.21.15.6:0/2054671594 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2744032d90 con 0x7f274c06ba60 2024-04-24T23:23:30.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:30.004+0000 7f2751a00640 1 -- 172.21.15.6:0/2054671594 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f274c06ba60 msgr2=0x7f274c158d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:23:30.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:30.004+0000 7f2751a00640 1 --2- 172.21.15.6:0/2054671594 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f274c06ba60 0x7f274c158d00 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f274c0c95d0 tx=0x7f274402ea40 comp rx=0 tx=0).stop 2024-04-24T23:23:30.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:30.004+0000 7f2751a00640 1 -- 172.21.15.6:0/2054671594 shutdown_connections 2024-04-24T23:23:30.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:30.004+0000 7f2751a00640 1 --2- 172.21.15.6:0/2054671594 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f274c06ba60 0x7f274c158d00 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:30.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:30.004+0000 7f2751a00640 1 --2- 172.21.15.6:0/2054671594 >> v2:172.21.15.6:3300/0 conn(0x7f274c152900 0x7f274c071fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:30.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:30.004+0000 7f2751a00640 1 -- 172.21.15.6:0/2054671594 >> 172.21.15.6:0/2054671594 conn(0x7f274c06e590 msgr2=0x7f274c0cd300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:23:30.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:30.006+0000 7f274be00640 1 --2- 172.21.15.6:0/2054671594 >> v2:172.21.15.6:3300/0 conn(0x7f274c152900 0x7f274c071fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:23:30.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:30.007+0000 7f2751a00640 1 -- 172.21.15.6:0/2054671594 shutdown_connections 2024-04-24T23:23:30.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:30.007+0000 7f2751a00640 1 -- 172.21.15.6:0/2054671594 wait complete. 2024-04-24T23:23:30.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:30.010+0000 7f2751a00640 1 Processor -- start 2024-04-24T23:23:30.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:30.011+0000 7f2751a00640 1 -- start start 2024-04-24T23:23:30.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:30.013+0000 7f2751a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f274c06ba60 0x7f274c1431d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:23:30.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:30.013+0000 7f2751a00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f274c0c7d00 0x7f274c140f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:23:30.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:30.013+0000 7f2751a00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f274c14d650 0x7f274c141490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:23:30.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:30.014+0000 7f274b400640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f274c0c7d00 0x7f274c140f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:23:30.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:30.014+0000 7f274b400640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f274c0c7d00 0x7f274c140f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:55142/0 (socket says 172.21.15.6:55142) 2024-04-24T23:23:30.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:30.014+0000 7f274be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f274c06ba60 0x7f274c1431d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:23:30.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:30.014+0000 7f274be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f274c06ba60 0x7f274c1431d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:49346/0 (socket says 172.21.15.6:49346) 2024-04-24T23:23:30.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:30.014+0000 7f274b400640 1 -- 172.21.15.6:0/1122138667 learned_addr learned my addr 172.21.15.6:0/1122138667 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:23:30.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:30.014+0000 7f2751a00640 1 -- 172.21.15.6:0/1122138667 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f274c152ce0 con 0x7f274c06ba60 2024-04-24T23:23:30.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:30.014+0000 7f2751a00640 1 -- 172.21.15.6:0/1122138667 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f274c072fe0 con 0x7f274c0c7d00 2024-04-24T23:23:30.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:30.014+0000 7f2751a00640 1 -- 172.21.15.6:0/1122138667 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f274c0c7120 con 0x7f274c14d650 2024-04-24T23:23:30.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:30.015+0000 7f2751000640 1 --2- 172.21.15.6:0/1122138667 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f274c14d650 0x7f274c141490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:23:30.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:30.016+0000 7f274be00640 1 -- 172.21.15.6:0/1122138667 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f274c14d650 msgr2=0x7f274c141490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:23:30.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:30.016+0000 7f274be00640 1 --2- 172.21.15.6:0/1122138667 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f274c14d650 0x7f274c141490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:30.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:30.016+0000 7f274be00640 1 -- 172.21.15.6:0/1122138667 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f274c0c7d00 msgr2=0x7f274c140f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:23:30.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:30.016+0000 7f274be00640 1 --2- 172.21.15.6:0/1122138667 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f274c0c7d00 0x7f274c140f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:30.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:30.016+0000 7f274be00640 1 -- 172.21.15.6:0/1122138667 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f274c15a4d0 con 0x7f274c06ba60 2024-04-24T23:23:30.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:30.016+0000 7f274be00640 1 --2- 172.21.15.6:0/1122138667 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f274c06ba60 0x7f274c1431d0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f274c06ebf0 tx=0x7f273800e9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:23:30.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:30.017+0000 7f2743e00640 1 -- 172.21.15.6:0/1122138667 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f273800f040 con 0x7f274c06ba60 2024-04-24T23:23:30.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:30.017+0000 7f2743e00640 1 -- 172.21.15.6:0/1122138667 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2738004230 con 0x7f274c06ba60 2024-04-24T23:23:30.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:30.017+0000 7f2743e00640 1 -- 172.21.15.6:0/1122138667 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2738003610 con 0x7f274c06ba60 2024-04-24T23:23:30.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:30.017+0000 7f2751a00640 1 -- 172.21.15.6:0/1122138667 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f274c15a7b0 con 0x7f274c06ba60 2024-04-24T23:23:30.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:30.017+0000 7f2751a00640 1 -- 172.21.15.6:0/1122138667 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f274c15ad70 con 0x7f274c06ba60 2024-04-24T23:23:30.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:30.027+0000 7f2743e00640 1 -- 172.21.15.6:0/1122138667 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f27380037b0 con 0x7f274c06ba60 2024-04-24T23:23:30.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:30.030+0000 7f2751a00640 1 -- 172.21.15.6:0/1122138667 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2724005290 con 0x7f274c06ba60 2024-04-24T23:23:30.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:30.030+0000 7f2743e00640 1 --2- 172.21.15.6:0/1122138667 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f272c06bf30 0x7f272c06e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:23:30.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:30.030+0000 7f2743e00640 1 -- 172.21.15.6:0/1122138667 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6762+0+0 (secure 0 0 0) 0x7f273808adc0 con 0x7f274c06ba60 2024-04-24T23:23:30.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:30.032+0000 7f274b400640 1 --2- 172.21.15.6:0/1122138667 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f272c06bf30 0x7f272c06e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:23:30.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:30.033+0000 7f274b400640 1 --2- 172.21.15.6:0/1122138667 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f272c06bf30 0x7f272c06e3f0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f274c13fd00 tx=0x7f273c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:23:30.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:30.049+0000 7f2743e00640 1 -- 172.21.15.6:0/1122138667 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f2738017020 con 0x7f274c06ba60 2024-04-24T23:23:30.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:30.617+0000 7f2751a00640 1 -- 172.21.15.6:0/1122138667 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2724005580 con 0x7f274c06ba60 2024-04-24T23:23:30.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:30.618+0000 7f2743e00640 1 -- 172.21.15.6:0/1122138667 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) ==== 94+0+6867 (secure 0 0 0) 0x7f27380045e0 con 0x7f274c06ba60 2024-04-24T23:23:30.619 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:23:30.619 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":38,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5081,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/1190759626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1190759626},{"type":"v1","addr":"172.21.15.6:6834","nonce":1190759626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":5503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/3928731059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":3928731059},{"type":"v1","addr":"172.21.15.6:6837","nonce":3928731059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":35},{"gid":5551,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6838/3253046315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":3253046315},{"type":"v1","addr":"172.21.15.190:6838","nonce":3253046315}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":38}],"filesystems":[{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:23:08.146813+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1,2],"up":{"mds_0":4306,"mds_1":4282,"mds_2":4235},"failed":[],"damaged":[],"stopped":[3,4],"info":{"gid_4235":{"gid":4235,"name":"d","rank":2,"incarnation":13,"state":"up:stopping","state_seq":14,"addr":"172.21.15.190:6839/1572872451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":1572872451},{"type":"v1","addr":"172.21.15.190:6839","nonce":1572872451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4282":{"gid":4282,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":12,"addr":"172.21.15.190:6837/580315269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":580315269},{"type":"v1","addr":"172.21.15.190:6837","nonce":580315269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/489092393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":489092393},{"type":"v1","addr":"172.21.15.6:6835","nonce":489092393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4306,"qdb_cluster":[4306,4282]},"id":1}]} 2024-04-24T23:23:30.620 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 38 2024-04-24T23:23:30.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:30.623+0000 7f2741600640 1 -- 172.21.15.6:0/1122138667 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f272c06bf30 msgr2=0x7f272c06e3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:23:30.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:30.623+0000 7f2741600640 1 --2- 172.21.15.6:0/1122138667 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f272c06bf30 0x7f272c06e3f0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f274c13fd00 tx=0x7f273c009040 comp rx=0 tx=0).stop 2024-04-24T23:23:30.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:30.623+0000 7f2741600640 1 -- 172.21.15.6:0/1122138667 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f274c06ba60 msgr2=0x7f274c1431d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:23:30.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:30.623+0000 7f2741600640 1 --2- 172.21.15.6:0/1122138667 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f274c06ba60 0x7f274c1431d0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f274c06ebf0 tx=0x7f273800e9d0 comp rx=0 tx=0).stop 2024-04-24T23:23:30.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:30.624+0000 7f2741600640 1 -- 172.21.15.6:0/1122138667 shutdown_connections 2024-04-24T23:23:30.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:30.624+0000 7f2741600640 1 --2- 172.21.15.6:0/1122138667 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f272c06bf30 0x7f272c06e3f0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:30.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:30.624+0000 7f2741600640 1 --2- 172.21.15.6:0/1122138667 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f274c14d650 0x7f274c141490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:30.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:30.624+0000 7f2741600640 1 --2- 172.21.15.6:0/1122138667 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f274c0c7d00 0x7f274c140f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:30.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:30.624+0000 7f2741600640 1 --2- 172.21.15.6:0/1122138667 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f274c06ba60 0x7f274c1431d0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:30.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:30.624+0000 7f2741600640 1 -- 172.21.15.6:0/1122138667 >> 172.21.15.6:0/1122138667 conn(0x7f274c06e590 msgr2=0x7f274c157a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:23:30.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:30.625+0000 7f2741600640 1 -- 172.21.15.6:0/1122138667 shutdown_connections 2024-04-24T23:23:30.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:30.625+0000 7f2741600640 1 -- 172.21.15.6:0/1122138667 wait complete. 2024-04-24T23:23:30.658 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-24T23:23:32.659 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:23:33.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:33.342+0000 7f33e7a00640 1 Processor -- start 2024-04-24T23:23:33.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:33.342+0000 7f33e7a00640 1 -- start start 2024-04-24T23:23:33.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:33.344+0000 7f33e7a00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f33e0071530 0x7f33e00c8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:23:33.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:33.344+0000 7f33e7a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f33e00c9230 0x7f33e0157950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:23:33.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:33.345+0000 7f33e5000640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f33e0151bb0 0x7f33e00c37f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:51832/0 (socket says 172.21.15.6:51832) 2024-04-24T23:23:33.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:33.347+0000 7f33dfe00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f33e0071530 0x7f33e00c8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:23:33.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:33.348+0000 7f33e5000640 1 -- 172.21.15.6:0/4269426459 learned_addr learned my addr 172.21.15.6:0/4269426459 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:23:33.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:33.348+0000 7f33e7a00640 1 -- 172.21.15.6:0/4269426459 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f33e00c8390 con 0x7f33e0151bb0 2024-04-24T23:23:33.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:33.348+0000 7f33e7a00640 1 -- 172.21.15.6:0/4269426459 --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f33e0071d10 con 0x7f33e00c9230 2024-04-24T23:23:33.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:33.348+0000 7f33e7a00640 1 -- 172.21.15.6:0/4269426459 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f33e0071ff0 con 0x7f33e0071530 2024-04-24T23:23:33.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:33.348+0000 7f33e5a00640 1 --2- 172.21.15.6:0/4269426459 >> v2:172.21.15.6:3300/0 conn(0x7f33e00c9230 0x7f33e0157950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:23:33.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:33.350+0000 7f33df400640 1 -- 172.21.15.6:0/4269426459 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1175212453 0 0) 0x7f33e00c8390 con 0x7f33e0151bb0 2024-04-24T23:23:33.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:33.350+0000 7f33df400640 1 -- 172.21.15.6:0/4269426459 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f33c4003520 con 0x7f33e0151bb0 2024-04-24T23:23:33.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:33.351+0000 7f33dfe00640 1 -- 172.21.15.6:0/4269426459 >> v2:172.21.15.6:3300/0 conn(0x7f33e00c9230 msgr2=0x7f33e0157950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:23:33.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:33.352+0000 7f33df400640 1 -- 172.21.15.6:0/4269426459 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 133570826 0 0) 0x7f33c4003520 con 0x7f33e0151bb0 2024-04-24T23:23:33.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:33.352+0000 7f33dfe00640 1 --2- 172.21.15.6:0/4269426459 >> v2:172.21.15.6:3300/0 conn(0x7f33e00c9230 0x7f33e0157950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:33.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:33.352+0000 7f33dfe00640 1 -- 172.21.15.6:0/4269426459 >> v1:172.21.15.6:6789/0 conn(0x7f33e0151bb0 legacy=0x7f33e00c37f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:23:33.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:33.352+0000 7f33dfe00640 1 -- 172.21.15.6:0/4269426459 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33e00722d0 con 0x7f33e0071530 2024-04-24T23:23:33.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:33.352+0000 7f33e5a00640 1 --2- 172.21.15.6:0/4269426459 >> v2:172.21.15.6:3300/0 conn(0x7f33e00c9230 0x7f33e0157950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:23:33.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:33.357+0000 7f33dfe00640 1 --2- 172.21.15.6:0/4269426459 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f33e0071530 0x7f33e00c8cf0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f33d00097a0 tx=0x7f33d002a230 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b09fe7a961c775dc server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:23:33.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:33.357+0000 7f33df400640 1 -- 172.21.15.6:0/4269426459 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33d0037070 con 0x7f33e0071530 2024-04-24T23:23:33.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:33.358+0000 7f33df400640 1 -- 172.21.15.6:0/4269426459 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f33d0032440 con 0x7f33e0071530 2024-04-24T23:23:33.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:33.358+0000 7f33df400640 1 -- 172.21.15.6:0/4269426459 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33d0032810 con 0x7f33e0071530 2024-04-24T23:23:33.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:33.360+0000 7f33e7a00640 1 -- 172.21.15.6:0/4269426459 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f33e0071530 msgr2=0x7f33e00c8cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:23:33.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:33.360+0000 7f33e7a00640 1 --2- 172.21.15.6:0/4269426459 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f33e0071530 0x7f33e00c8cf0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f33d00097a0 tx=0x7f33d002a230 comp rx=0 tx=0).stop 2024-04-24T23:23:33.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:33.360+0000 7f33e7a00640 1 -- 172.21.15.6:0/4269426459 shutdown_connections 2024-04-24T23:23:33.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:33.360+0000 7f33e7a00640 1 --2- 172.21.15.6:0/4269426459 >> v2:172.21.15.6:3300/0 conn(0x7f33e00c9230 0x7f33e0157950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:33.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:33.360+0000 7f33e7a00640 1 --2- 172.21.15.6:0/4269426459 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f33e0071530 0x7f33e00c8cf0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:33.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:33.360+0000 7f33e7a00640 1 -- 172.21.15.6:0/4269426459 >> 172.21.15.6:0/4269426459 conn(0x7f33e006b0e0 msgr2=0x7f33e006b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:23:33.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:33.361+0000 7f33e7a00640 1 -- 172.21.15.6:0/4269426459 shutdown_connections 2024-04-24T23:23:33.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:33.364+0000 7f33e7a00640 1 -- 172.21.15.6:0/4269426459 wait complete. 2024-04-24T23:23:33.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:33.367+0000 7f33e7a00640 1 Processor -- start 2024-04-24T23:23:33.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:33.372+0000 7f33e7a00640 1 -- start start 2024-04-24T23:23:33.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:33.374+0000 7f33e7a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f33e0071530 0x7f33e0142c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:23:33.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:33.374+0000 7f33e7a00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f33e00c9230 0x7f33e0143170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:23:33.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:33.374+0000 7f33e7a00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f33e0151bb0 0x7f33e0156440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:23:33.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:33.374+0000 7f33e7a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f33e00c80f0 con 0x7f33e0071530 2024-04-24T23:23:33.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:33.374+0000 7f33e7a00640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f33e00c7f70 con 0x7f33e0151bb0 2024-04-24T23:23:33.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:33.374+0000 7f33e7a00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f33e0157e90 con 0x7f33e00c9230 2024-04-24T23:23:33.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:33.375+0000 7f33dfe00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f33e00c9230 0x7f33e0143170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:23:33.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:33.375+0000 7f33dfe00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f33e00c9230 0x7f33e0143170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.6:37660/0 (socket says 172.21.15.6:37660) 2024-04-24T23:23:33.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:33.375+0000 7f33dfe00640 1 -- 172.21.15.6:0/3975974855 learned_addr learned my addr 172.21.15.6:0/3975974855 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:23:33.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:33.376+0000 7f33e5a00640 1 --2- 172.21.15.6:0/3975974855 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f33e0151bb0 0x7f33e0156440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:23:33.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:33.376+0000 7f33e5000640 1 --2- 172.21.15.6:0/3975974855 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f33e0071530 0x7f33e0142c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:23:33.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:33.377+0000 7f33dfe00640 1 -- 172.21.15.6:0/3975974855 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f33e0151bb0 msgr2=0x7f33e0156440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:23:33.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:33.377+0000 7f33dfe00640 1 --2- 172.21.15.6:0/3975974855 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f33e0151bb0 0x7f33e0156440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:33.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:33.377+0000 7f33dfe00640 1 -- 172.21.15.6:0/3975974855 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f33e0071530 msgr2=0x7f33e0142c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:23:33.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:33.377+0000 7f33dfe00640 1 --2- 172.21.15.6:0/3975974855 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f33e0071530 0x7f33e0142c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:33.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:33.377+0000 7f33dfe00640 1 -- 172.21.15.6:0/3975974855 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33e014ef80 con 0x7f33e00c9230 2024-04-24T23:23:33.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:33.380+0000 7f33dfe00640 1 --2- 172.21.15.6:0/3975974855 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f33e00c9230 0x7f33e0143170 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f33d00098c0 tx=0x7f33d00046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:23:33.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:33.382+0000 7f33dcc00640 1 -- 172.21.15.6:0/3975974855 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33d0041030 con 0x7f33e00c9230 2024-04-24T23:23:33.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:33.382+0000 7f33dcc00640 1 -- 172.21.15.6:0/3975974855 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f33d003d420 con 0x7f33e00c9230 2024-04-24T23:23:33.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:33.382+0000 7f33dcc00640 1 -- 172.21.15.6:0/3975974855 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33d0037040 con 0x7f33e00c9230 2024-04-24T23:23:33.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:33.382+0000 7f33e7a00640 1 -- 172.21.15.6:0/3975974855 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33e0155a10 con 0x7f33e00c9230 2024-04-24T23:23:33.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:33.382+0000 7f33e7a00640 1 -- 172.21.15.6:0/3975974855 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f33e01556f0 con 0x7f33e00c9230 2024-04-24T23:23:33.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:33.386+0000 7f33e7a00640 1 -- 172.21.15.6:0/3975974855 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f33e01ed480 con 0x7f33e00c9230 2024-04-24T23:23:33.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:33.394+0000 7f33dcc00640 1 -- 172.21.15.6:0/3975974855 <== mon.2 v2:172.21.15.190:3301/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f33d0044070 con 0x7f33e00c9230 2024-04-24T23:23:33.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:33.398+0000 7f33dcc00640 1 --2- 172.21.15.6:0/3975974855 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f33c006bf00 0x7f33c006e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:23:33.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:33.402+0000 7f33dcc00640 1 -- 172.21.15.6:0/3975974855 <== mon.2 v2:172.21.15.190:3301/0 5 ==== osd_map(28..28 src has 1..28) ==== 6762+0+0 (secure 0 0 0) 0x7f33d00ab710 con 0x7f33e00c9230 2024-04-24T23:23:33.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:33.405+0000 7f33e5000640 1 --2- 172.21.15.6:0/3975974855 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f33c006bf00 0x7f33c006e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:23:33.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:33.406+0000 7f33dcc00640 1 -- 172.21.15.6:0/3975974855 <== mon.2 v2:172.21.15.190:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f33e01ed480 con 0x7f33e00c9230 2024-04-24T23:23:33.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:33.412+0000 7f33e5000640 1 --2- 172.21.15.6:0/3975974855 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f33c006bf00 0x7f33c006e3c0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f33d80016e0 tx=0x7f33d8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:23:34.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:34.006+0000 7f33e7a00640 1 -- 172.21.15.6:0/3975974855 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f33e006e3b0 con 0x7f33e00c9230 2024-04-24T23:23:34.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:34.012+0000 7f33dcc00640 1 -- 172.21.15.6:0/3975974855 <== mon.2 v2:172.21.15.190:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) ==== 94+0+6867 (secure 0 0 0) 0x7f33d0044e40 con 0x7f33e00c9230 2024-04-24T23:23:34.013 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:23:34.018 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":38,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5081,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/1190759626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1190759626},{"type":"v1","addr":"172.21.15.6:6834","nonce":1190759626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":5503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/3928731059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":3928731059},{"type":"v1","addr":"172.21.15.6:6837","nonce":3928731059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":35},{"gid":5551,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6838/3253046315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":3253046315},{"type":"v1","addr":"172.21.15.190:6838","nonce":3253046315}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":38}],"filesystems":[{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:23:08.146813+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1,2],"up":{"mds_0":4306,"mds_1":4282,"mds_2":4235},"failed":[],"damaged":[],"stopped":[3,4],"info":{"gid_4235":{"gid":4235,"name":"d","rank":2,"incarnation":13,"state":"up:stopping","state_seq":14,"addr":"172.21.15.190:6839/1572872451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":1572872451},{"type":"v1","addr":"172.21.15.190:6839","nonce":1572872451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4282":{"gid":4282,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":12,"addr":"172.21.15.190:6837/580315269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":580315269},{"type":"v1","addr":"172.21.15.190:6837","nonce":580315269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/489092393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":489092393},{"type":"v1","addr":"172.21.15.6:6835","nonce":489092393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4306,"qdb_cluster":[4306,4282]},"id":1}]} 2024-04-24T23:23:34.018 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 38 2024-04-24T23:23:34.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:34.021+0000 7f33e7a00640 1 -- 172.21.15.6:0/3975974855 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f33c006bf00 msgr2=0x7f33c006e3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:23:34.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:34.021+0000 7f33e7a00640 1 --2- 172.21.15.6:0/3975974855 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f33c006bf00 0x7f33c006e3c0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f33d80016e0 tx=0x7f33d8009040 comp rx=0 tx=0).stop 2024-04-24T23:23:34.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:34.021+0000 7f33e7a00640 1 -- 172.21.15.6:0/3975974855 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f33e00c9230 msgr2=0x7f33e0143170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:23:34.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:34.021+0000 7f33e7a00640 1 --2- 172.21.15.6:0/3975974855 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f33e00c9230 0x7f33e0143170 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f33d00098c0 tx=0x7f33d00046c0 comp rx=0 tx=0).stop 2024-04-24T23:23:34.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:34.022+0000 7f33e7a00640 1 -- 172.21.15.6:0/3975974855 shutdown_connections 2024-04-24T23:23:34.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:34.022+0000 7f33e7a00640 1 --2- 172.21.15.6:0/3975974855 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f33c006bf00 0x7f33c006e3c0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:34.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:34.022+0000 7f33e7a00640 1 --2- 172.21.15.6:0/3975974855 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f33e00c9230 0x7f33e0143170 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:34.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:34.022+0000 7f33e7a00640 1 --2- 172.21.15.6:0/3975974855 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f33e0151bb0 0x7f33e0156440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:34.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:34.022+0000 7f33e7a00640 1 --2- 172.21.15.6:0/3975974855 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f33e0071530 0x7f33e0142c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:34.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:34.022+0000 7f33e7a00640 1 -- 172.21.15.6:0/3975974855 >> 172.21.15.6:0/3975974855 conn(0x7f33e006b0e0 msgr2=0x7f33e01470b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:23:34.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:34.024+0000 7f33e7a00640 1 -- 172.21.15.6:0/3975974855 shutdown_connections 2024-04-24T23:23:34.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:34.024+0000 7f33e7a00640 1 -- 172.21.15.6:0/3975974855 wait complete. 2024-04-24T23:23:34.103 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-24T23:23:36.104 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:23:36.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:36.311+0000 7f7bb0e00640 1 Processor -- start 2024-04-24T23:23:36.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:36.312+0000 7f7bb0e00640 1 -- start start 2024-04-24T23:23:36.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:36.319+0000 7f7ba9e00640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f7bac150990 0x7f7bac15a980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.6:46254/0 (socket says 172.21.15.6:46254) 2024-04-24T23:23:36.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:36.321+0000 7f7bb0e00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7bac157470 0x7f7bac162380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:23:36.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:36.321+0000 7f7bb0e00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7bac0c8690 con 0x7f7bac073270 2024-04-24T23:23:36.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:36.321+0000 7f7bb0e00640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7bac0c8970 con 0x7f7bac150990 2024-04-24T23:23:36.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:36.321+0000 7f7bb0e00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f7bac0c8c50 con 0x7f7bac157470 2024-04-24T23:23:36.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:36.321+0000 7f7ba9e00640 1 -- 172.21.15.6:0/2675626556 learned_addr learned my addr 172.21.15.6:0/2675626556 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:23:36.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:36.319+0000 7f7ba9400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f7bac073270 0x7f7bac0c9110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:60768/0 (socket says 172.21.15.6:60768) 2024-04-24T23:23:36.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:36.321+0000 7f7baa800640 1 --2- 172.21.15.6:0/2675626556 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7bac157470 0x7f7bac162380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:23:36.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:36.324+0000 7f7baa800640 1 -- 172.21.15.6:0/2675626556 >> v1:172.21.15.190:6789/0 conn(0x7f7bac150990 legacy=0x7f7bac15a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:23:36.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:36.324+0000 7f7baa800640 1 -- 172.21.15.6:0/2675626556 >> v1:172.21.15.6:6789/0 conn(0x7f7bac073270 legacy=0x7f7bac0c9110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:23:36.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:36.324+0000 7f7baa800640 1 -- 172.21.15.6:0/2675626556 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7bac15a180 con 0x7f7bac157470 2024-04-24T23:23:36.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:36.325+0000 7f7baa800640 1 --2- 172.21.15.6:0/2675626556 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7bac157470 0x7f7bac162380 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f7ba4009dc0 tx=0x7f7ba402a500 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7203b6a3db8a749a server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:23:36.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:36.338+0000 7f7ba8a00640 1 -- 172.21.15.6:0/2675626556 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ba4035020 con 0x7f7bac157470 2024-04-24T23:23:36.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:36.338+0000 7f7ba8a00640 1 -- 172.21.15.6:0/2675626556 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ba4007740 con 0x7f7bac157470 2024-04-24T23:23:36.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:36.338+0000 7f7ba8a00640 1 -- 172.21.15.6:0/2675626556 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ba4007b10 con 0x7f7bac157470 2024-04-24T23:23:36.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:36.340+0000 7f7bb0e00640 1 -- 172.21.15.6:0/2675626556 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7bac157470 msgr2=0x7f7bac162380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:23:36.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:36.340+0000 7f7bb0e00640 1 --2- 172.21.15.6:0/2675626556 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7bac157470 0x7f7bac162380 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f7ba4009dc0 tx=0x7f7ba402a500 comp rx=0 tx=0).stop 2024-04-24T23:23:36.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:36.342+0000 7f7bb0e00640 1 -- 172.21.15.6:0/2675626556 shutdown_connections 2024-04-24T23:23:36.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:36.342+0000 7f7bb0e00640 1 --2- 172.21.15.6:0/2675626556 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7bac157470 0x7f7bac162380 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:36.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:36.342+0000 7f7bb0e00640 1 -- 172.21.15.6:0/2675626556 >> 172.21.15.6:0/2675626556 conn(0x7f7bac06b0e0 msgr2=0x7f7bac06b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:23:36.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:36.346+0000 7f7bb0e00640 1 -- 172.21.15.6:0/2675626556 shutdown_connections 2024-04-24T23:23:36.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:36.346+0000 7f7bb0e00640 1 -- 172.21.15.6:0/2675626556 wait complete. 2024-04-24T23:23:36.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:36.356+0000 7f7bb0e00640 1 Processor -- start 2024-04-24T23:23:36.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:36.357+0000 7f7bb0e00640 1 -- start start 2024-04-24T23:23:36.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:36.361+0000 7f7bb0e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7bac073270 0x7f7bac151550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:23:36.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:36.362+0000 7f7ba9e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7bac073270 0x7f7bac151550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:23:36.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:36.363+0000 7f7ba9e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7bac073270 0x7f7bac151550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:49348/0 (socket says 172.21.15.6:49348) 2024-04-24T23:23:36.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:36.363+0000 7f7ba9e00640 1 -- 172.21.15.6:0/2880833435 learned_addr learned my addr 172.21.15.6:0/2880833435 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:23:36.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:36.363+0000 7f7bb0e00640 1 --2- 172.21.15.6:0/2880833435 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7bac150990 0x7f7bac151a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:23:36.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:36.364+0000 7f7bb0e00640 1 --2- 172.21.15.6:0/2880833435 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7bac157470 0x7f7bac167120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:23:36.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:36.364+0000 7f7bb0e00640 1 -- 172.21.15.6:0/2880833435 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f7bac150d70 con 0x7f7bac073270 2024-04-24T23:23:36.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:36.365+0000 7f7bb0e00640 1 -- 172.21.15.6:0/2880833435 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f7bac0c7a70 con 0x7f7bac157470 2024-04-24T23:23:36.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:36.365+0000 7f7ba9400640 1 --2- 172.21.15.6:0/2880833435 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7bac150990 0x7f7bac151a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:23:36.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:36.365+0000 7f7bb0e00640 1 -- 172.21.15.6:0/2880833435 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f7bac0c9820 con 0x7f7bac150990 2024-04-24T23:23:36.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:36.368+0000 7f7ba9e00640 1 -- 172.21.15.6:0/2880833435 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7bac150990 msgr2=0x7f7bac151a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:23:36.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:36.368+0000 7f7ba9e00640 1 --2- 172.21.15.6:0/2880833435 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7bac150990 0x7f7bac151a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:36.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:36.368+0000 7f7ba9e00640 1 -- 172.21.15.6:0/2880833435 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7bac157470 msgr2=0x7f7bac167120 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-24T23:23:36.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:36.368+0000 7f7ba9e00640 1 --2- 172.21.15.6:0/2880833435 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7bac157470 0x7f7bac167120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:36.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:36.368+0000 7f7ba9e00640 1 -- 172.21.15.6:0/2880833435 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7bac0cadb0 con 0x7f7bac073270 2024-04-24T23:23:36.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:36.370+0000 7f7ba9400640 1 --2- 172.21.15.6:0/2880833435 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7bac150990 0x7f7bac151a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:23:36.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:36.370+0000 7f7ba9e00640 1 --2- 172.21.15.6:0/2880833435 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7bac073270 0x7f7bac151550 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f7b9800c7c0 tx=0x7f7b9800cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:23:36.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:36.373+0000 7f7ba2000640 1 -- 172.21.15.6:0/2880833435 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b98015020 con 0x7f7bac073270 2024-04-24T23:23:36.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:36.374+0000 7f7ba2000640 1 -- 172.21.15.6:0/2880833435 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b98007a80 con 0x7f7bac073270 2024-04-24T23:23:36.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:36.374+0000 7f7bb0e00640 1 -- 172.21.15.6:0/2880833435 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7bac1f7f00 con 0x7f7bac073270 2024-04-24T23:23:36.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:36.374+0000 7f7bb0e00640 1 -- 172.21.15.6:0/2880833435 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7bac1f84c0 con 0x7f7bac073270 2024-04-24T23:23:36.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:36.378+0000 7f7ba2000640 1 -- 172.21.15.6:0/2880833435 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b98007e50 con 0x7f7bac073270 2024-04-24T23:23:36.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:36.381+0000 7f7bb0e00640 1 -- 172.21.15.6:0/2880833435 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7bac0767b0 con 0x7f7bac073270 2024-04-24T23:23:36.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:36.389+0000 7f7ba2000640 1 -- 172.21.15.6:0/2880833435 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f7b98018020 con 0x7f7bac073270 2024-04-24T23:23:36.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:36.391+0000 7f7ba2000640 1 --2- 172.21.15.6:0/2880833435 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f7b8c06c0d0 0x7f7b8c06e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:23:36.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:36.391+0000 7f7ba2000640 1 -- 172.21.15.6:0/2880833435 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6762+0+0 (secure 0 0 0) 0x7f7b9808b5c0 con 0x7f7bac073270 2024-04-24T23:23:36.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:36.396+0000 7f7ba9400640 1 --2- 172.21.15.6:0/2880833435 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f7b8c06c0d0 0x7f7b8c06e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:23:36.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:36.399+0000 7f7ba9400640 1 --2- 172.21.15.6:0/2880833435 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f7b8c06c0d0 0x7f7b8c06e590 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f7bac0ca730 tx=0x7f7b9c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:23:36.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:36.402+0000 7f7ba2000640 1 -- 172.21.15.6:0/2880833435 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f7b9801f020 con 0x7f7bac073270 2024-04-24T23:23:36.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:36.895+0000 7f7bb0e00640 1 -- 172.21.15.6:0/2880833435 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7bac0cd000 con 0x7f7bac073270 2024-04-24T23:23:36.905 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:23:36.905 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":38,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5081,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/1190759626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1190759626},{"type":"v1","addr":"172.21.15.6:6834","nonce":1190759626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":5503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/3928731059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":3928731059},{"type":"v1","addr":"172.21.15.6:6837","nonce":3928731059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":35},{"gid":5551,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6838/3253046315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":3253046315},{"type":"v1","addr":"172.21.15.190:6838","nonce":3253046315}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":38}],"filesystems":[{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:23:08.146813+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1,2],"up":{"mds_0":4306,"mds_1":4282,"mds_2":4235},"failed":[],"damaged":[],"stopped":[3,4],"info":{"gid_4235":{"gid":4235,"name":"d","rank":2,"incarnation":13,"state":"up:stopping","state_seq":14,"addr":"172.21.15.190:6839/1572872451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":1572872451},{"type":"v1","addr":"172.21.15.190:6839","nonce":1572872451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4282":{"gid":4282,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":12,"addr":"172.21.15.190:6837/580315269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":580315269},{"type":"v1","addr":"172.21.15.190:6837","nonce":580315269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/489092393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":489092393},{"type":"v1","addr":"172.21.15.6:6835","nonce":489092393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4306,"qdb_cluster":[4306,4282]},"id":1}]} 2024-04-24T23:23:36.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:36.901+0000 7f7ba2000640 1 -- 172.21.15.6:0/2880833435 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) ==== 94+0+6867 (secure 0 0 0) 0x7f7b98059bc0 con 0x7f7bac073270 2024-04-24T23:23:36.906 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 38 2024-04-24T23:23:36.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:36.905+0000 7f7bb0e00640 1 -- 172.21.15.6:0/2880833435 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f7b8c06c0d0 msgr2=0x7f7b8c06e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:23:36.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:36.905+0000 7f7bb0e00640 1 --2- 172.21.15.6:0/2880833435 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f7b8c06c0d0 0x7f7b8c06e590 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f7bac0ca730 tx=0x7f7b9c009040 comp rx=0 tx=0).stop 2024-04-24T23:23:36.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:36.905+0000 7f7bb0e00640 1 -- 172.21.15.6:0/2880833435 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7bac073270 msgr2=0x7f7bac151550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:23:36.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:36.905+0000 7f7bb0e00640 1 --2- 172.21.15.6:0/2880833435 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7bac073270 0x7f7bac151550 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f7b9800c7c0 tx=0x7f7b9800cbc0 comp rx=0 tx=0).stop 2024-04-24T23:23:36.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:36.907+0000 7f7bb0e00640 1 -- 172.21.15.6:0/2880833435 shutdown_connections 2024-04-24T23:23:36.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:36.907+0000 7f7bb0e00640 1 --2- 172.21.15.6:0/2880833435 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f7b8c06c0d0 0x7f7b8c06e590 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:36.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:36.907+0000 7f7bb0e00640 1 --2- 172.21.15.6:0/2880833435 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7bac150990 0x7f7bac151a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:36.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:36.907+0000 7f7bb0e00640 1 --2- 172.21.15.6:0/2880833435 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7bac157470 0x7f7bac167120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:36.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:36.907+0000 7f7bb0e00640 1 --2- 172.21.15.6:0/2880833435 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7bac073270 0x7f7bac151550 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:36.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:36.907+0000 7f7bb0e00640 1 -- 172.21.15.6:0/2880833435 >> 172.21.15.6:0/2880833435 conn(0x7f7bac06b0e0 msgr2=0x7f7bac0c9110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:23:36.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:36.907+0000 7f7bb0e00640 1 -- 172.21.15.6:0/2880833435 shutdown_connections 2024-04-24T23:23:36.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:36.908+0000 7f7bb0e00640 1 -- 172.21.15.6:0/2880833435 wait complete. 2024-04-24T23:23:36.952 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-24T23:23:38.654 INFO:tasks.ceph.mds.d.smithi190.stdout:starting mds.d at 2024-04-24T23:23:38.953 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:23:39.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:39.480+0000 7f1c30000640 1 Processor -- start 2024-04-24T23:23:39.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:39.481+0000 7f1c30000640 1 -- start start 2024-04-24T23:23:39.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:39.483+0000 7f1c30000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f1c28167180 0x7f1c280c7dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:23:39.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:39.483+0000 7f1c30000640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1c2816abf0 0x7f1c2816b060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:23:39.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:39.483+0000 7f1c30000640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1c28072970 con 0x7f1c28072530 2024-04-24T23:23:39.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:39.483+0000 7f1c30000640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f1c28168570 con 0x7f1c28167180 2024-04-24T23:23:39.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:39.483+0000 7f1c30000640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f1c280720a0 con 0x7f1c2816abf0 2024-04-24T23:23:39.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:39.483+0000 7f1c2cc00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f1c28072530 0x7f1c2806edb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:60778/0 (socket says 172.21.15.6:60778) 2024-04-24T23:23:39.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:39.483+0000 7f1c2cc00640 1 -- 172.21.15.6:0/2126431861 learned_addr learned my addr 172.21.15.6:0/2126431861 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:23:39.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:39.483+0000 7f1c2d600640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f1c28167180 0x7f1c280c7dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:23:39.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:39.484+0000 7f1c27e00640 1 -- 172.21.15.6:0/2126431861 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3326013296 0 0) 0x7f1c18001110 con 0x7f1c28072530 2024-04-24T23:23:39.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:39.531+0000 7f1c27e00640 1 -- 172.21.15.6:0/2126431861 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1c10003520 con 0x7f1c28072530 2024-04-24T23:23:39.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:39.490+0000 7f1c2e000640 1 --2- 172.21.15.6:0/2126431861 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1c2816abf0 0x7f1c2816b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:23:39.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:39.532+0000 7f1c27e00640 1 -- 172.21.15.6:0/2126431861 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3717988844 0 0) 0x7f1c10003520 con 0x7f1c28072530 2024-04-24T23:23:39.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:39.532+0000 7f1c27e00640 1 -- 172.21.15.6:0/2126431861 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1c2816abf0 msgr2=0x7f1c2816b060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:23:39.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:39.532+0000 7f1c27e00640 1 --2- 172.21.15.6:0/2126431861 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1c2816abf0 0x7f1c2816b060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:39.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:39.533+0000 7f1c27e00640 1 -- 172.21.15.6:0/2126431861 >> v2:172.21.15.6:3300/0 conn(0x7f1c28167180 msgr2=0x7f1c280c7dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:23:39.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:39.533+0000 7f1c27e00640 1 --2- 172.21.15.6:0/2126431861 >> v2:172.21.15.6:3300/0 conn(0x7f1c28167180 0x7f1c280c7dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:39.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:39.533+0000 7f1c27e00640 1 -- 172.21.15.6:0/2126431861 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c280c74c0 con 0x7f1c28072530 2024-04-24T23:23:39.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:39.533+0000 7f1c27e00640 1 -- 172.21.15.6:0/2126431861 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3463268701 0 0) 0x7f1c18001840 con 0x7f1c28072530 2024-04-24T23:23:39.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:39.533+0000 7f1c27e00640 1 -- 172.21.15.6:0/2126431861 >> v1:172.21.15.6:6789/0 conn(0x7f1c28072530 legacy=0x7f1c2806edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:23:39.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:39.533+0000 7f1c27e00640 1 --2- 172.21.15.6:0/2126431861 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1c10003b80 0x7f1c10005fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:23:39.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:39.533+0000 7f1c27e00640 1 --2- 172.21.15.6:0/2126431861 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1c100065d0 0x7f1c10008a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:23:39.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:39.533+0000 7f1c27e00640 1 --2- 172.21.15.6:0/2126431861 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1c2816abf0 0x7f1c1000b140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:23:39.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:39.533+0000 7f1c27e00640 1 -- 172.21.15.6:0/2126431861 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f1c280720a0 con 0x7f1c100065d0 2024-04-24T23:23:39.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:39.533+0000 7f1c27e00640 1 -- 172.21.15.6:0/2126431861 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f1c1000b910 con 0x7f1c10003b80 2024-04-24T23:23:39.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:39.533+0000 7f1c27e00640 1 -- 172.21.15.6:0/2126431861 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f1c1000bb10 con 0x7f1c2816abf0 2024-04-24T23:23:39.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:39.534+0000 7f1c2d600640 1 --2- 172.21.15.6:0/2126431861 >> v2:172.21.15.6:3300/0 conn(0x7f1c28167180 0x7f1c280c7dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:23:39.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:39.536+0000 7f1c2e000640 1 --2- 172.21.15.6:0/2126431861 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1c2816abf0 0x7f1c1000b140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:23:39.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:39.537+0000 7f1c2cc00640 1 --2- 172.21.15.6:0/2126431861 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1c100065d0 0x7f1c10008a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:23:39.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:39.538+0000 7f1c2cc00640 1 -- 172.21.15.6:0/2126431861 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1c2816abf0 msgr2=0x7f1c1000b140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:23:39.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:39.538+0000 7f1c2cc00640 1 --2- 172.21.15.6:0/2126431861 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1c2816abf0 0x7f1c1000b140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:39.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:39.538+0000 7f1c2cc00640 1 -- 172.21.15.6:0/2126431861 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1c10003b80 msgr2=0x7f1c10005fb0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-24T23:23:39.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:39.538+0000 7f1c2cc00640 1 --2- 172.21.15.6:0/2126431861 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1c10003b80 0x7f1c10005fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:39.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:39.538+0000 7f1c2cc00640 1 -- 172.21.15.6:0/2126431861 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c280c74c0 con 0x7f1c100065d0 2024-04-24T23:23:39.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:39.541+0000 7f1c2cc00640 1 --2- 172.21.15.6:0/2126431861 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1c100065d0 0x7f1c10008a60 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f1c1800c740 tx=0x7f1c10003810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=906905f27c692f45 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:23:39.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:39.541+0000 7f1c27e00640 1 -- 172.21.15.6:0/2126431861 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c1800c0e0 con 0x7f1c100065d0 2024-04-24T23:23:39.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:39.542+0000 7f1c27e00640 1 -- 172.21.15.6:0/2126431861 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1c1802f070 con 0x7f1c100065d0 2024-04-24T23:23:39.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:39.543+0000 7f1c30000640 1 -- 172.21.15.6:0/2126431861 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1c100065d0 msgr2=0x7f1c10008a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:23:39.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:39.543+0000 7f1c30000640 1 --2- 172.21.15.6:0/2126431861 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1c100065d0 0x7f1c10008a60 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f1c1800c740 tx=0x7f1c10003810 comp rx=0 tx=0).stop 2024-04-24T23:23:39.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:39.544+0000 7f1c2e000640 1 -- 172.21.15.6:0/2126431861 reap_dead start 2024-04-24T23:23:39.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:39.544+0000 7f1c30000640 1 -- 172.21.15.6:0/2126431861 shutdown_connections 2024-04-24T23:23:39.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:39.545+0000 7f1c30000640 1 -- 172.21.15.6:0/2126431861 >> 172.21.15.6:0/2126431861 conn(0x7f1c2806b0e0 msgr2=0x7f1c2806b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:23:39.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:39.545+0000 7f1c30000640 1 -- 172.21.15.6:0/2126431861 shutdown_connections 2024-04-24T23:23:39.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:39.545+0000 7f1c30000640 1 -- 172.21.15.6:0/2126431861 wait complete. 2024-04-24T23:23:39.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:39.552+0000 7f1c30000640 1 Processor -- start 2024-04-24T23:23:39.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:39.552+0000 7f1c30000640 1 -- start start 2024-04-24T23:23:39.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:39.554+0000 7f1c30000640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1c100065d0 0x7f1c2816b270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:23:39.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:39.554+0000 7f1c30000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1c280c6000 0x7f1c280c4200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:23:39.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:39.554+0000 7f1c30000640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1c2816a6a0 0x7f1c280c4740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:23:39.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:39.554+0000 7f1c30000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f1c28071a50 con 0x7f1c280c6000 2024-04-24T23:23:39.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:39.554+0000 7f1c30000640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f1c28071770 con 0x7f1c100065d0 2024-04-24T23:23:39.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:39.554+0000 7f1c30000640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f1c28071d30 con 0x7f1c2816a6a0 2024-04-24T23:23:39.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:39.554+0000 7f1c2d600640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1c100065d0 0x7f1c2816b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:23:39.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:39.554+0000 7f1c2d600640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1c100065d0 0x7f1c2816b270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:51708/0 (socket says 172.21.15.6:51708) 2024-04-24T23:23:39.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:39.554+0000 7f1c2d600640 1 -- 172.21.15.6:0/1885951502 learned_addr learned my addr 172.21.15.6:0/1885951502 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:23:39.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:39.555+0000 7f1c2cc00640 1 --2- 172.21.15.6:0/1885951502 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1c280c6000 0x7f1c280c4200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:23:39.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:39.555+0000 7f1c2e000640 1 --2- 172.21.15.6:0/1885951502 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1c2816a6a0 0x7f1c280c4740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:23:39.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:39.555+0000 7f1c2d600640 1 -- 172.21.15.6:0/1885951502 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1c2816a6a0 msgr2=0x7f1c280c4740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:23:39.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:39.555+0000 7f1c2d600640 1 --2- 172.21.15.6:0/1885951502 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1c2816a6a0 0x7f1c280c4740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:39.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:39.555+0000 7f1c2d600640 1 -- 172.21.15.6:0/1885951502 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1c280c6000 msgr2=0x7f1c280c4200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:23:39.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:39.555+0000 7f1c2d600640 1 --2- 172.21.15.6:0/1885951502 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1c280c6000 0x7f1c280c4200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:39.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:39.555+0000 7f1c2d600640 1 -- 172.21.15.6:0/1885951502 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c28169ee0 con 0x7f1c100065d0 2024-04-24T23:23:39.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:39.556+0000 7f1c2d600640 1 --2- 172.21.15.6:0/1885951502 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1c100065d0 0x7f1c2816b270 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f1c1c00c3a0 tx=0x7f1c1c00c860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:23:39.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:39.556+0000 7f1c25600640 1 -- 172.21.15.6:0/1885951502 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c1c004030 con 0x7f1c100065d0 2024-04-24T23:23:39.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:39.556+0000 7f1c30000640 1 -- 172.21.15.6:0/1885951502 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c2816a460 con 0x7f1c100065d0 2024-04-24T23:23:39.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:39.556+0000 7f1c25600640 1 -- 172.21.15.6:0/1885951502 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1c1c0041d0 con 0x7f1c100065d0 2024-04-24T23:23:39.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:39.556+0000 7f1c25600640 1 -- 172.21.15.6:0/1885951502 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c1c004030 con 0x7f1c100065d0 2024-04-24T23:23:39.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:39.556+0000 7f1c30000640 1 -- 172.21.15.6:0/1885951502 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c2816a180 con 0x7f1c100065d0 2024-04-24T23:23:39.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:39.558+0000 7f1c30000640 1 -- 172.21.15.6:0/1885951502 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1c280767b0 con 0x7f1c100065d0 2024-04-24T23:23:39.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:39.560+0000 7f1c25600640 1 -- 172.21.15.6:0/1885951502 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f1c1c004380 con 0x7f1c100065d0 2024-04-24T23:23:39.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:39.561+0000 7f1c25600640 1 --2- 172.21.15.6:0/1885951502 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f1c0c08e070 0x7f1c0c090530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:23:39.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:39.561+0000 7f1c25600640 1 -- 172.21.15.6:0/1885951502 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6762+0+0 (secure 0 0 0) 0x7f1c1c00f070 con 0x7f1c100065d0 2024-04-24T23:23:39.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:39.562+0000 7f1c2cc00640 1 --2- 172.21.15.6:0/1885951502 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f1c0c08e070 0x7f1c0c090530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:23:39.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:39.564+0000 7f1c2cc00640 1 --2- 172.21.15.6:0/1885951502 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f1c0c08e070 0x7f1c0c090530 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f1c18033000 tx=0x7f1c18007000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:23:39.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:39.565+0000 7f1c25600640 1 -- 172.21.15.6:0/1885951502 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f1c1c059060 con 0x7f1c100065d0 2024-04-24T23:23:40.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:40.108+0000 7f1c30000640 1 -- 172.21.15.6:0/1885951502 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1c280c9190 con 0x7f1c100065d0 2024-04-24T23:23:40.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:40.119+0000 7f1c25600640 1 -- 172.21.15.6:0/1885951502 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) ==== 94+0+6852 (secure 0 0 0) 0x7f1c1c058960 con 0x7f1c100065d0 2024-04-24T23:23:40.120 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:23:40.120 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":40,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5081,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/1190759626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1190759626},{"type":"v1","addr":"172.21.15.6:6834","nonce":1190759626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":5503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/3928731059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":3928731059},{"type":"v1","addr":"172.21.15.6:6837","nonce":3928731059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":35},{"gid":5551,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6838/3253046315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":3253046315},{"type":"v1","addr":"172.21.15.190:6838","nonce":3253046315}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":38},{"gid":5599,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6839/2643233757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":2643233757},{"type":"v1","addr":"172.21.15.190:6839","nonce":2643233757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40}],"filesystems":[{"mdsmap":{"epoch":39,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:23:38.162102+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":4306,"mds_1":4282},"failed":[],"damaged":[],"stopped":[2,3,4],"info":{"gid_4282":{"gid":4282,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":12,"addr":"172.21.15.190:6837/580315269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":580315269},{"type":"v1","addr":"172.21.15.190:6837","nonce":580315269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/489092393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":489092393},{"type":"v1","addr":"172.21.15.6:6835","nonce":489092393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4306,"qdb_cluster":[4306,4282]},"id":1}]} 2024-04-24T23:23:40.120 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 40 2024-04-24T23:23:40.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:40.125+0000 7f1c30000640 1 -- 172.21.15.6:0/1885951502 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f1c0c08e070 msgr2=0x7f1c0c090530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:23:40.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:40.125+0000 7f1c30000640 1 --2- 172.21.15.6:0/1885951502 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f1c0c08e070 0x7f1c0c090530 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f1c18033000 tx=0x7f1c18007000 comp rx=0 tx=0).stop 2024-04-24T23:23:40.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:40.125+0000 7f1c30000640 1 -- 172.21.15.6:0/1885951502 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1c100065d0 msgr2=0x7f1c2816b270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:23:40.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:40.125+0000 7f1c30000640 1 --2- 172.21.15.6:0/1885951502 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1c100065d0 0x7f1c2816b270 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f1c1c00c3a0 tx=0x7f1c1c00c860 comp rx=0 tx=0).stop 2024-04-24T23:23:40.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:40.126+0000 7f1c30000640 1 -- 172.21.15.6:0/1885951502 shutdown_connections 2024-04-24T23:23:40.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:40.126+0000 7f1c30000640 1 --2- 172.21.15.6:0/1885951502 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f1c0c08e070 0x7f1c0c090530 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:40.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:40.126+0000 7f1c30000640 1 --2- 172.21.15.6:0/1885951502 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1c2816a6a0 0x7f1c280c4740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:40.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:40.126+0000 7f1c30000640 1 --2- 172.21.15.6:0/1885951502 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1c280c6000 0x7f1c280c4200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:40.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:40.126+0000 7f1c30000640 1 --2- 172.21.15.6:0/1885951502 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1c100065d0 0x7f1c2816b270 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:40.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:40.126+0000 7f1c30000640 1 -- 172.21.15.6:0/1885951502 >> 172.21.15.6:0/1885951502 conn(0x7f1c2806b0e0 msgr2=0x7f1c2806ed40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:23:40.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:40.126+0000 7f1c30000640 1 -- 172.21.15.6:0/1885951502 shutdown_connections 2024-04-24T23:23:40.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:40.126+0000 7f1c30000640 1 -- 172.21.15.6:0/1885951502 wait complete. 2024-04-24T23:23:40.161 INFO:tasks.mds_thrash.fs.[cephfs]:mds cluster has 2 alive and active, now stable! 2024-04-24T23:23:40.161 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.b (rank=3) 2024-04-24T23:23:40.162 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-24T23:23:40.162 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T23:23:40.177 INFO:tasks.ceph.mds.b:Stopped 2024-04-24T23:23:40.177 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:23:40.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:40.815+0000 7feab6e00640 1 Processor -- start 2024-04-24T23:23:40.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:40.815+0000 7feab6e00640 1 -- start start 2024-04-24T23:23:40.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:40.818+0000 7feab6e00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feab01519b0 0x7feab00c8d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:23:40.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:40.819+0000 7feaafe00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feab01519b0 0x7feab00c8d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:23:40.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:40.819+0000 7feaafe00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feab01519b0 0x7feab00c8d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.6:36526/0 (socket says 172.21.15.6:36526) 2024-04-24T23:23:40.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:40.819+0000 7feaaf400640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7feab006c1f0 0x7feab00caed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.6:46260/0 (socket says 172.21.15.6:46260) 2024-04-24T23:23:40.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:40.820+0000 7feaafe00640 1 -- 172.21.15.6:0/2625448216 learned_addr learned my addr 172.21.15.6:0/2625448216 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:23:40.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:40.820+0000 7feab6e00640 1 -- 172.21.15.6:0/2625448216 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feab006ca80 con 0x7feab006c5d0 2024-04-24T23:23:40.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:40.821+0000 7feab6e00640 1 -- 172.21.15.6:0/2625448216 --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feab00c9260 con 0x7feab006c1f0 2024-04-24T23:23:40.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:40.821+0000 7feab6e00640 1 -- 172.21.15.6:0/2625448216 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7feab015ba10 con 0x7feab01519b0 2024-04-24T23:23:40.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:40.821+0000 7feaaea00640 1 -- 172.21.15.6:0/2625448216 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3380220322 0 0) 0x7feab00c9260 con 0x7feab006c1f0 2024-04-24T23:23:40.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:40.822+0000 7feaaea00640 1 -- 172.21.15.6:0/2625448216 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fea94003520 con 0x7feab006c1f0 2024-04-24T23:23:40.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:40.822+0000 7feaaea00640 1 -- 172.21.15.6:0/2625448216 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2369817252 0 0) 0x7feab006ca80 con 0x7feab006c5d0 2024-04-24T23:23:40.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:40.822+0000 7feaaea00640 1 -- 172.21.15.6:0/2625448216 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feab00c9260 con 0x7feab006c5d0 2024-04-24T23:23:40.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:40.823+0000 7feaaea00640 1 -- 172.21.15.6:0/2625448216 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3121241273 0 0) 0x7fea94003520 con 0x7feab006c1f0 2024-04-24T23:23:40.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:40.823+0000 7feaaea00640 1 -- 172.21.15.6:0/2625448216 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feab01519b0 msgr2=0x7feab00c8d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:23:40.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:40.823+0000 7feaaea00640 1 --2- 172.21.15.6:0/2625448216 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feab01519b0 0x7feab00c8d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:40.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:40.823+0000 7feaaea00640 1 -- 172.21.15.6:0/2625448216 >> v1:172.21.15.6:6789/0 conn(0x7feab006c5d0 legacy=0x7feab015e2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:23:40.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:40.823+0000 7feaaea00640 1 -- 172.21.15.6:0/2625448216 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feab015ae00 con 0x7feab006c1f0 2024-04-24T23:23:40.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:40.823+0000 7feaaea00640 1 -- 172.21.15.6:0/2625448216 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3463268701 0 0) 0x7fea9c001670 con 0x7feab006c1f0 2024-04-24T23:23:40.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:40.823+0000 7feaaea00640 1 -- 172.21.15.6:0/2625448216 >> v1:172.21.15.190:6789/0 conn(0x7feab006c1f0 legacy=0x7feab00caed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:23:40.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:40.823+0000 7feaaea00640 1 --2- 172.21.15.6:0/2625448216 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fea94004bd0 0x7fea94004fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:23:40.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:40.823+0000 7feaaea00640 1 --2- 172.21.15.6:0/2625448216 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fea940037a0 0x7fea94003c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:23:40.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:40.823+0000 7feaaea00640 1 --2- 172.21.15.6:0/2625448216 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fea94004160 0x7fea9400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:23:40.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:40.823+0000 7feaaea00640 1 -- 172.21.15.6:0/2625448216 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7feab015ba10 con 0x7fea940037a0 2024-04-24T23:23:40.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:40.823+0000 7feaaea00640 1 -- 172.21.15.6:0/2625448216 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fea9400bf30 con 0x7fea94004bd0 2024-04-24T23:23:40.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:40.823+0000 7feaaea00640 1 -- 172.21.15.6:0/2625448216 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fea9400c130 con 0x7fea94004160 2024-04-24T23:23:40.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:40.824+0000 7feaafe00640 1 --2- 172.21.15.6:0/2625448216 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fea94004bd0 0x7fea94004fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:23:40.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:40.824+0000 7feaaf400640 1 --2- 172.21.15.6:0/2625448216 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fea940037a0 0x7fea94003c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:23:40.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:40.825+0000 7feab4e00640 1 --2- 172.21.15.6:0/2625448216 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fea94004160 0x7fea9400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:23:40.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:40.825+0000 7feaafe00640 1 -- 172.21.15.6:0/2625448216 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fea94004160 msgr2=0x7fea9400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:23:40.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:40.825+0000 7feaafe00640 1 --2- 172.21.15.6:0/2625448216 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fea94004160 0x7fea9400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:40.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:40.825+0000 7feaafe00640 1 -- 172.21.15.6:0/2625448216 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fea940037a0 msgr2=0x7fea94003c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:23:40.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:40.825+0000 7feaafe00640 1 --2- 172.21.15.6:0/2625448216 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fea940037a0 0x7fea94003c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:40.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:40.825+0000 7feaafe00640 1 -- 172.21.15.6:0/2625448216 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feab015ae00 con 0x7fea94004bd0 2024-04-24T23:23:40.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:40.826+0000 7feaafe00640 1 --2- 172.21.15.6:0/2625448216 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fea94004bd0 0x7fea94004fb0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7feab00ce230 tx=0x7fea94005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d7a9f1774db360c0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:23:40.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:40.827+0000 7feaaea00640 1 -- 172.21.15.6:0/2625448216 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feaa0036070 con 0x7fea94004bd0 2024-04-24T23:23:40.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:40.827+0000 7feaaea00640 1 -- 172.21.15.6:0/2625448216 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feaa0031400 con 0x7fea94004bd0 2024-04-24T23:23:40.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:40.828+0000 7feaaea00640 1 -- 172.21.15.6:0/2625448216 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feaa0009720 con 0x7fea94004bd0 2024-04-24T23:23:40.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:40.830+0000 7feab6e00640 1 -- 172.21.15.6:0/2625448216 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fea94004bd0 msgr2=0x7fea94004fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:23:40.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:40.830+0000 7feab6e00640 1 --2- 172.21.15.6:0/2625448216 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fea94004bd0 0x7fea94004fb0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7feab00ce230 tx=0x7fea94005830 comp rx=0 tx=0).stop 2024-04-24T23:23:40.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:40.830+0000 7feab4e00640 1 -- 172.21.15.6:0/2625448216 reap_dead start 2024-04-24T23:23:40.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:40.831+0000 7feab6e00640 1 -- 172.21.15.6:0/2625448216 shutdown_connections 2024-04-24T23:23:40.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:40.831+0000 7feab6e00640 1 -- 172.21.15.6:0/2625448216 >> 172.21.15.6:0/2625448216 conn(0x7feab0070280 msgr2=0x7feab006f320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:23:40.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:40.831+0000 7feab6e00640 1 -- 172.21.15.6:0/2625448216 shutdown_connections 2024-04-24T23:23:40.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:40.831+0000 7feab6e00640 1 -- 172.21.15.6:0/2625448216 wait complete. 2024-04-24T23:23:40.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:40.834+0000 7feab6e00640 1 Processor -- start 2024-04-24T23:23:40.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:40.834+0000 7feab6e00640 1 -- start start 2024-04-24T23:23:40.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:40.836+0000 7feab6e00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fea94004bd0 0x7feab0157900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:23:40.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:40.836+0000 7feab6e00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feab0153120 0x7feab0153580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:23:40.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:40.836+0000 7feab6e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feab00cc120 0x7feab00cc5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:23:40.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:40.836+0000 7feab6e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7feab015b6a0 con 0x7feab00cc120 2024-04-24T23:23:40.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:40.836+0000 7feab6e00640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7feab015b3c0 con 0x7feab0153120 2024-04-24T23:23:40.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:40.836+0000 7feab6e00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7feab0151d90 con 0x7fea94004bd0 2024-04-24T23:23:40.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:40.837+0000 7feaaf400640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feab0153120 0x7feab0153580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:23:40.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:40.837+0000 7feaaf400640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feab0153120 0x7feab0153580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:51726/0 (socket says 172.21.15.6:51726) 2024-04-24T23:23:40.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:40.837+0000 7feaaf400640 1 -- 172.21.15.6:0/3184186402 learned_addr learned my addr 172.21.15.6:0/3184186402 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:23:40.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:40.837+0000 7feaaf400640 1 -- 172.21.15.6:0/3184186402 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fea94004bd0 msgr2=0x7feab0157900 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-24T23:23:40.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:40.837+0000 7feaaf400640 1 --2- 172.21.15.6:0/3184186402 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fea94004bd0 0x7feab0157900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:40.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:40.837+0000 7feaaf400640 1 -- 172.21.15.6:0/3184186402 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feab00cc120 msgr2=0x7feab00cc5a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-24T23:23:40.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:40.837+0000 7feaaf400640 1 --2- 172.21.15.6:0/3184186402 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feab00cc120 0x7feab00cc5a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:40.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:40.837+0000 7feaaf400640 1 -- 172.21.15.6:0/3184186402 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feab0153ac0 con 0x7feab0153120 2024-04-24T23:23:40.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:40.838+0000 7feaaf400640 1 --2- 172.21.15.6:0/3184186402 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feab0153120 0x7feab0153580 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fea9c0076f0 tx=0x7fea9c004e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:23:40.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:40.838+0000 7feaa7e00640 1 -- 172.21.15.6:0/3184186402 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea9c02b070 con 0x7feab0153120 2024-04-24T23:23:40.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:40.838+0000 7feaa7e00640 1 -- 172.21.15.6:0/3184186402 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea9c00fa20 con 0x7feab0153120 2024-04-24T23:23:40.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:40.838+0000 7feaa7e00640 1 -- 172.21.15.6:0/3184186402 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea9c00fe10 con 0x7feab0153120 2024-04-24T23:23:40.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:40.841+0000 7feab6e00640 1 -- 172.21.15.6:0/3184186402 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feab00cde70 con 0x7feab0153120 2024-04-24T23:23:40.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:40.842+0000 7feab6e00640 1 -- 172.21.15.6:0/3184186402 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feab00ce430 con 0x7feab0153120 2024-04-24T23:23:40.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:40.844+0000 7feab6e00640 1 -- 172.21.15.6:0/3184186402 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feab00ca0b0 con 0x7feab0153120 2024-04-24T23:23:40.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:40.848+0000 7feaa7e00640 1 -- 172.21.15.6:0/3184186402 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fea9c023070 con 0x7feab0153120 2024-04-24T23:23:40.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:40.850+0000 7feaa7e00640 1 --2- 172.21.15.6:0/3184186402 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fea9006be30 0x7fea9006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:23:40.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:40.852+0000 7feaa7e00640 1 -- 172.21.15.6:0/3184186402 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6762+0+0 (secure 0 0 0) 0x7fea9c067940 con 0x7feab0153120 2024-04-24T23:23:40.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:40.852+0000 7feaafe00640 1 --2- 172.21.15.6:0/3184186402 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fea9006be30 0x7fea9006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:23:40.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:40.855+0000 7feaafe00640 1 --2- 172.21.15.6:0/3184186402 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fea9006be30 0x7fea9006e2f0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7feaa0033000 tx=0x7feaa0008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:23:40.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:40.857+0000 7feaa7e00640 1 -- 172.21.15.6:0/3184186402 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7feab00ca0b0 con 0x7feab0153120 2024-04-24T23:23:41.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:41.355+0000 7feab6e00640 1 -- 172.21.15.6:0/3184186402 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7feab00710b0 con 0x7feab0153120 2024-04-24T23:23:41.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:41.357+0000 7feaa7e00640 1 -- 172.21.15.6:0/3184186402 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) ==== 94+0+6852 (secure 0 0 0) 0x7feab00710b0 con 0x7feab0153120 2024-04-24T23:23:41.359 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:23:41.359 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":40,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5081,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/1190759626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1190759626},{"type":"v1","addr":"172.21.15.6:6834","nonce":1190759626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":5503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/3928731059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":3928731059},{"type":"v1","addr":"172.21.15.6:6837","nonce":3928731059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":35},{"gid":5551,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6838/3253046315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":3253046315},{"type":"v1","addr":"172.21.15.190:6838","nonce":3253046315}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":38},{"gid":5599,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6839/2643233757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":2643233757},{"type":"v1","addr":"172.21.15.190:6839","nonce":2643233757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40}],"filesystems":[{"mdsmap":{"epoch":39,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:23:38.162102+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":4306,"mds_1":4282},"failed":[],"damaged":[],"stopped":[2,3,4],"info":{"gid_4282":{"gid":4282,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":12,"addr":"172.21.15.190:6837/580315269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":580315269},{"type":"v1","addr":"172.21.15.190:6837","nonce":580315269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/489092393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":489092393},{"type":"v1","addr":"172.21.15.6:6835","nonce":489092393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4306,"qdb_cluster":[4306,4282]},"id":1}]} 2024-04-24T23:23:41.359 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 40 2024-04-24T23:23:41.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:41.362+0000 7feab6e00640 1 -- 172.21.15.6:0/3184186402 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fea9006be30 msgr2=0x7fea9006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:23:41.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:41.362+0000 7feab6e00640 1 --2- 172.21.15.6:0/3184186402 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fea9006be30 0x7fea9006e2f0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7feaa0033000 tx=0x7feaa0008030 comp rx=0 tx=0).stop 2024-04-24T23:23:41.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:41.362+0000 7feab6e00640 1 -- 172.21.15.6:0/3184186402 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feab0153120 msgr2=0x7feab0153580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:23:41.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:41.362+0000 7feab6e00640 1 --2- 172.21.15.6:0/3184186402 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feab0153120 0x7feab0153580 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fea9c0076f0 tx=0x7fea9c004e30 comp rx=0 tx=0).stop 2024-04-24T23:23:41.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:41.362+0000 7feab6e00640 1 -- 172.21.15.6:0/3184186402 shutdown_connections 2024-04-24T23:23:41.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:41.362+0000 7feab6e00640 1 --2- 172.21.15.6:0/3184186402 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fea9006be30 0x7fea9006e2f0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:41.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:41.362+0000 7feab6e00640 1 --2- 172.21.15.6:0/3184186402 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feab00cc120 0x7feab00cc5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:41.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:41.362+0000 7feab6e00640 1 --2- 172.21.15.6:0/3184186402 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feab0153120 0x7feab0153580 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:41.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:41.362+0000 7feab6e00640 1 --2- 172.21.15.6:0/3184186402 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fea94004bd0 0x7feab0157900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:41.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:41.362+0000 7feab6e00640 1 -- 172.21.15.6:0/3184186402 >> 172.21.15.6:0/3184186402 conn(0x7feab0070280 msgr2=0x7feab00c3340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:23:41.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:41.363+0000 7feab6e00640 1 -- 172.21.15.6:0/3184186402 shutdown_connections 2024-04-24T23:23:41.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:41.363+0000 7feab6e00640 1 -- 172.21.15.6:0/3184186402 wait complete. 2024-04-24T23:23:41.401 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-04-24T23:23:43.402 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:23:44.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:44.249+0000 7f88d7e00640 1 Processor -- start 2024-04-24T23:23:44.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:44.249+0000 7f88d7e00640 1 -- start start 2024-04-24T23:23:44.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:44.251+0000 7f88d7e00640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f88d80cc870 0x7f88d8070e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:23:44.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:44.251+0000 7f88d7e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f88d8162ac0 0x7f88d8158a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:23:44.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:44.251+0000 7f88d7e00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f88d80715e0 con 0x7f88d80ccc50 2024-04-24T23:23:44.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:44.251+0000 7f88d7e00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f88d80713b0 con 0x7f88d8162ac0 2024-04-24T23:23:44.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:44.251+0000 7f88d7e00640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f88d8162350 con 0x7f88d80cc870 2024-04-24T23:23:44.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:44.253+0000 7f88d6a00640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f88d80cc870 0x7f88d8070e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:23:44.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:44.253+0000 7f88d6a00640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f88d80cc870 0x7f88d8070e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:51730/0 (socket says 172.21.15.6:51730) 2024-04-24T23:23:44.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:44.253+0000 7f88d6a00640 1 -- 172.21.15.6:0/816801073 learned_addr learned my addr 172.21.15.6:0/816801073 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:23:44.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:44.254+0000 7f88d6a00640 1 -- 172.21.15.6:0/816801073 >> v2:172.21.15.6:3300/0 conn(0x7f88d8162ac0 msgr2=0x7f88d8158a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:23:44.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:44.254+0000 7f88d6a00640 1 --2- 172.21.15.6:0/816801073 >> v2:172.21.15.6:3300/0 conn(0x7f88d8162ac0 0x7f88d8158a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:44.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:44.254+0000 7f88d6a00640 1 -- 172.21.15.6:0/816801073 >> v1:172.21.15.6:6789/0 conn(0x7f88d80ccc50 legacy=0x7f88d815f790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:23:44.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:44.254+0000 7f88d6a00640 1 -- 172.21.15.6:0/816801073 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88d806dcb0 con 0x7f88d80cc870 2024-04-24T23:23:44.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:44.256+0000 7f88d6a00640 1 --2- 172.21.15.6:0/816801073 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f88d80cc870 0x7f88d8070e70 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f88d806ce50 tx=0x7f88c802a230 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6edee20e1a1c3c04 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:23:44.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:44.257+0000 7f88d5600640 1 -- 172.21.15.6:0/816801073 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88c8037070 con 0x7f88d80cc870 2024-04-24T23:23:44.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:44.257+0000 7f88d5600640 1 -- 172.21.15.6:0/816801073 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f88c8032440 con 0x7f88d80cc870 2024-04-24T23:23:44.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:44.257+0000 7f88d5600640 1 -- 172.21.15.6:0/816801073 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88c8032810 con 0x7f88d80cc870 2024-04-24T23:23:44.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:44.258+0000 7f88d7e00640 1 -- 172.21.15.6:0/816801073 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f88d80cc870 msgr2=0x7f88d8070e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:23:44.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:44.258+0000 7f88d7e00640 1 --2- 172.21.15.6:0/816801073 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f88d80cc870 0x7f88d8070e70 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f88d806ce50 tx=0x7f88c802a230 comp rx=0 tx=0).stop 2024-04-24T23:23:44.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:44.260+0000 7f88d7e00640 1 -- 172.21.15.6:0/816801073 shutdown_connections 2024-04-24T23:23:44.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:44.260+0000 7f88d7e00640 1 --2- 172.21.15.6:0/816801073 >> v2:172.21.15.6:3300/0 conn(0x7f88d8162ac0 0x7f88d8158a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:44.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:44.260+0000 7f88d7e00640 1 --2- 172.21.15.6:0/816801073 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f88d80cc870 0x7f88d8070e70 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:44.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:44.260+0000 7f88d7e00640 1 -- 172.21.15.6:0/816801073 >> 172.21.15.6:0/816801073 conn(0x7f88d806ed80 msgr2=0x7f88d806f190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:23:44.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:44.260+0000 7f88d7e00640 1 -- 172.21.15.6:0/816801073 shutdown_connections 2024-04-24T23:23:44.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:44.260+0000 7f88d7e00640 1 -- 172.21.15.6:0/816801073 wait complete. 2024-04-24T23:23:44.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:44.264+0000 7f88d7e00640 1 Processor -- start 2024-04-24T23:23:44.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:44.267+0000 7f88d7e00640 1 -- start start 2024-04-24T23:23:44.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:44.269+0000 7f88d7e00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f88d80cc870 0x7f88d81f7fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:23:44.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:44.269+0000 7f88d7e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f88d80ccc50 0x7f88d81f5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:23:44.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:44.269+0000 7f88d7e00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f88d8162ac0 0x7f88d8151220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:23:44.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:44.270+0000 7f88d7e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f88d806df50 con 0x7f88d80ccc50 2024-04-24T23:23:44.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:44.270+0000 7f88d7e00640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f88d80724e0 con 0x7f88d8162ac0 2024-04-24T23:23:44.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:44.270+0000 7f88d7400640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f88d8162ac0 0x7f88d8151220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:23:44.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:44.270+0000 7f88d7400640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f88d8162ac0 0x7f88d8151220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:51742/0 (socket says 172.21.15.6:51742) 2024-04-24T23:23:44.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:44.270+0000 7f88d7400640 1 -- 172.21.15.6:0/810375798 learned_addr learned my addr 172.21.15.6:0/810375798 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:23:44.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:44.270+0000 7f88d7e00640 1 -- 172.21.15.6:0/810375798 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f88d806e230 con 0x7f88d80cc870 2024-04-24T23:23:44.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:44.271+0000 7f88d7400640 1 -- 172.21.15.6:0/810375798 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f88d80cc870 msgr2=0x7f88d81f7fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:23:44.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:44.271+0000 7f88d6a00640 1 --2- 172.21.15.6:0/810375798 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f88d80cc870 0x7f88d81f7fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:23:44.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:44.271+0000 7f88d7400640 1 --2- 172.21.15.6:0/810375798 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f88d80cc870 0x7f88d81f7fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:44.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:44.271+0000 7f88d7400640 1 -- 172.21.15.6:0/810375798 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f88d80ccc50 msgr2=0x7f88d81f5470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:23:44.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:44.271+0000 7f88d6000640 1 --2- 172.21.15.6:0/810375798 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f88d80ccc50 0x7f88d81f5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:23:44.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:44.271+0000 7f88d7400640 1 --2- 172.21.15.6:0/810375798 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f88d80ccc50 0x7f88d81f5470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:44.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:44.271+0000 7f88d7400640 1 -- 172.21.15.6:0/810375798 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88d8155670 con 0x7f88d8162ac0 2024-04-24T23:23:44.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:44.271+0000 7f88d6000640 1 --2- 172.21.15.6:0/810375798 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f88d80ccc50 0x7f88d81f5470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:23:44.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:44.273+0000 7f88d7400640 1 --2- 172.21.15.6:0/810375798 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f88d8162ac0 0x7f88d8151220 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f88d000b970 tx=0x7f88d000be30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:23:44.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:44.273+0000 7f88cf400640 1 -- 172.21.15.6:0/810375798 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88d0019070 con 0x7f88d8162ac0 2024-04-24T23:23:44.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:44.273+0000 7f88cf400640 1 -- 172.21.15.6:0/810375798 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f88d000cb60 con 0x7f88d8162ac0 2024-04-24T23:23:44.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:44.273+0000 7f88cf400640 1 -- 172.21.15.6:0/810375798 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88d0014440 con 0x7f88d8162ac0 2024-04-24T23:23:44.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:44.273+0000 7f88d7e00640 1 -- 172.21.15.6:0/810375798 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88d815ec50 con 0x7f88d8162ac0 2024-04-24T23:23:44.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:44.273+0000 7f88d7e00640 1 -- 172.21.15.6:0/810375798 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88d8155910 con 0x7f88d8162ac0 2024-04-24T23:23:44.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:44.276+0000 7f88d7e00640 1 -- 172.21.15.6:0/810375798 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f88d8075b30 con 0x7f88d8162ac0 2024-04-24T23:23:44.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:44.294+0000 7f88cf400640 1 -- 172.21.15.6:0/810375798 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f88d001c070 con 0x7f88d8162ac0 2024-04-24T23:23:44.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:44.304+0000 7f88cf400640 1 --2- 172.21.15.6:0/810375798 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f88b806be60 0x7f88b806e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:23:44.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:44.304+0000 7f88cf400640 1 -- 172.21.15.6:0/810375798 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6762+0+0 (secure 0 0 0) 0x7f88d008b780 con 0x7f88d8162ac0 2024-04-24T23:23:44.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:44.304+0000 7f88cf400640 1 -- 172.21.15.6:0/810375798 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f88d008bba0 con 0x7f88d8162ac0 2024-04-24T23:23:44.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:44.310+0000 7f88d6a00640 1 --2- 172.21.15.6:0/810375798 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f88b806be60 0x7f88b806e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:23:44.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:44.312+0000 7f88d6a00640 1 --2- 172.21.15.6:0/810375798 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f88b806be60 0x7f88b806e320 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f88c8034000 tx=0x7f88c802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:23:44.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:44.654+0000 7f88d7e00640 1 -- 172.21.15.6:0/810375798 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f88d8155200 con 0x7f88d8162ac0 2024-04-24T23:23:44.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:44.655+0000 7f88cf400640 1 -- 172.21.15.6:0/810375798 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) ==== 94+0+6852 (secure 0 0 0) 0x7f88d0059cf0 con 0x7f88d8162ac0 2024-04-24T23:23:44.659 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:23:44.659 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":40,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5081,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/1190759626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1190759626},{"type":"v1","addr":"172.21.15.6:6834","nonce":1190759626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":5503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/3928731059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":3928731059},{"type":"v1","addr":"172.21.15.6:6837","nonce":3928731059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":35},{"gid":5551,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6838/3253046315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":3253046315},{"type":"v1","addr":"172.21.15.190:6838","nonce":3253046315}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":38},{"gid":5599,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6839/2643233757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":2643233757},{"type":"v1","addr":"172.21.15.190:6839","nonce":2643233757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40}],"filesystems":[{"mdsmap":{"epoch":39,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:23:38.162102+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":4306,"mds_1":4282},"failed":[],"damaged":[],"stopped":[2,3,4],"info":{"gid_4282":{"gid":4282,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":12,"addr":"172.21.15.190:6837/580315269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":580315269},{"type":"v1","addr":"172.21.15.190:6837","nonce":580315269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/489092393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":489092393},{"type":"v1","addr":"172.21.15.6:6835","nonce":489092393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4306,"qdb_cluster":[4306,4282]},"id":1}]} 2024-04-24T23:23:44.659 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 40 2024-04-24T23:23:44.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:44.660+0000 7f88d7e00640 1 -- 172.21.15.6:0/810375798 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f88b806be60 msgr2=0x7f88b806e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:23:44.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:44.660+0000 7f88d7e00640 1 --2- 172.21.15.6:0/810375798 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f88b806be60 0x7f88b806e320 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f88c8034000 tx=0x7f88c802e000 comp rx=0 tx=0).stop 2024-04-24T23:23:44.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:44.660+0000 7f88d7e00640 1 -- 172.21.15.6:0/810375798 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f88d8162ac0 msgr2=0x7f88d8151220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:23:44.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:44.660+0000 7f88d7e00640 1 --2- 172.21.15.6:0/810375798 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f88d8162ac0 0x7f88d8151220 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f88d000b970 tx=0x7f88d000be30 comp rx=0 tx=0).stop 2024-04-24T23:23:44.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:44.660+0000 7f88d7e00640 1 -- 172.21.15.6:0/810375798 shutdown_connections 2024-04-24T23:23:44.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:44.660+0000 7f88d7e00640 1 --2- 172.21.15.6:0/810375798 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f88b806be60 0x7f88b806e320 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:44.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:44.661+0000 7f88d7e00640 1 --2- 172.21.15.6:0/810375798 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f88d8162ac0 0x7f88d8151220 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:44.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:44.661+0000 7f88d7e00640 1 --2- 172.21.15.6:0/810375798 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f88d80ccc50 0x7f88d81f5470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:44.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:44.661+0000 7f88d7e00640 1 --2- 172.21.15.6:0/810375798 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f88d80cc870 0x7f88d81f7fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:44.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:44.661+0000 7f88d7e00640 1 -- 172.21.15.6:0/810375798 >> 172.21.15.6:0/810375798 conn(0x7f88d806ed80 msgr2=0x7f88d81523a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:23:44.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:44.662+0000 7f88d7e00640 1 -- 172.21.15.6:0/810375798 shutdown_connections 2024-04-24T23:23:44.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:44.663+0000 7f88d7e00640 1 -- 172.21.15.6:0/810375798 wait complete. 2024-04-24T23:23:44.695 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-04-24T23:23:46.696 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:23:47.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:47.646+0000 7f9f08c00640 1 Processor -- start 2024-04-24T23:23:47.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:47.647+0000 7f9f08c00640 1 -- start start 2024-04-24T23:23:47.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:47.649+0000 7f9f08c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f9f04073270 0x7f9f040c9110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:23:47.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:47.649+0000 7f9f08c00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9f04157470 0x7f9f040c9650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:23:47.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:47.649+0000 7f9f08c00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9f040c8690 con 0x7f9f04150990 2024-04-24T23:23:47.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:47.649+0000 7f9f08c00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f9f040c8c50 con 0x7f9f04073270 2024-04-24T23:23:47.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:47.649+0000 7f9f08c00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f9f040c8970 con 0x7f9f04157470 2024-04-24T23:23:47.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:47.650+0000 7f9f01400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f9f04073270 0x7f9f040c9110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:23:47.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:47.650+0000 7f9f02800640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9f04157470 0x7f9f040c9650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:23:47.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:47.650+0000 7f9f02800640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9f04157470 0x7f9f040c9650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.6:36212/0 (socket says 172.21.15.6:36212) 2024-04-24T23:23:47.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:47.650+0000 7f9f02800640 1 -- 172.21.15.6:0/515083362 learned_addr learned my addr 172.21.15.6:0/515083362 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:23:47.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:47.651+0000 7f9f02800640 1 -- 172.21.15.6:0/515083362 >> v2:172.21.15.6:3300/0 conn(0x7f9f04073270 msgr2=0x7f9f040c9110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:23:47.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:47.651+0000 7f9f02800640 1 --2- 172.21.15.6:0/515083362 >> v2:172.21.15.6:3300/0 conn(0x7f9f04073270 0x7f9f040c9110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:47.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:47.651+0000 7f9f02800640 1 -- 172.21.15.6:0/515083362 >> v1:172.21.15.6:6789/0 conn(0x7f9f04150990 legacy=0x7f9f0415a980 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-24T23:23:47.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:47.651+0000 7f9f02800640 1 -- 172.21.15.6:0/515083362 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f0415a180 con 0x7f9f04157470 2024-04-24T23:23:47.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:47.653+0000 7f9f02800640 1 --2- 172.21.15.6:0/515083362 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9f04157470 0x7f9f040c9650 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f9eec009870 tx=0x7f9eec02c420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f8931daa22b8a3e6 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:23:47.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:47.653+0000 7f9f00a00640 1 -- 172.21.15.6:0/515083362 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9eec038070 con 0x7f9f04157470 2024-04-24T23:23:47.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:47.654+0000 7f9f00a00640 1 -- 172.21.15.6:0/515083362 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9eec031710 con 0x7f9f04157470 2024-04-24T23:23:47.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:47.654+0000 7f9f00a00640 1 -- 172.21.15.6:0/515083362 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9eec031ae0 con 0x7f9f04157470 2024-04-24T23:23:47.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:47.655+0000 7f9f08c00640 1 -- 172.21.15.6:0/515083362 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9f04157470 msgr2=0x7f9f040c9650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:23:47.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:47.655+0000 7f9f08c00640 1 --2- 172.21.15.6:0/515083362 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9f04157470 0x7f9f040c9650 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f9eec009870 tx=0x7f9eec02c420 comp rx=0 tx=0).stop 2024-04-24T23:23:47.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:47.662+0000 7f9f08c00640 1 -- 172.21.15.6:0/515083362 shutdown_connections 2024-04-24T23:23:47.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:47.662+0000 7f9f08c00640 1 --2- 172.21.15.6:0/515083362 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9f04157470 0x7f9f040c9650 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:47.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:47.662+0000 7f9f08c00640 1 --2- 172.21.15.6:0/515083362 >> v2:172.21.15.6:3300/0 conn(0x7f9f04073270 0x7f9f040c9110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:47.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:47.662+0000 7f9f08c00640 1 -- 172.21.15.6:0/515083362 >> 172.21.15.6:0/515083362 conn(0x7f9f0406b0e0 msgr2=0x7f9f0406b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:23:47.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:47.662+0000 7f9f08c00640 1 -- 172.21.15.6:0/515083362 shutdown_connections 2024-04-24T23:23:47.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:47.662+0000 7f9f08c00640 1 -- 172.21.15.6:0/515083362 wait complete. 2024-04-24T23:23:47.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:47.666+0000 7f9f08c00640 1 Processor -- start 2024-04-24T23:23:47.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:47.667+0000 7f9f08c00640 1 -- start start 2024-04-24T23:23:47.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:47.671+0000 7f9f08c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9f04073270 0x7f9f041536b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:23:47.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:47.671+0000 7f9f08c00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9f04150990 0x7f9f04156170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:23:47.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:47.671+0000 7f9f08c00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9f04157470 0x7f9f04158f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:23:47.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:47.671+0000 7f9f08c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f9f04150d70 con 0x7f9f04073270 2024-04-24T23:23:47.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:47.671+0000 7f9f08c00640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f9f040c7a70 con 0x7f9f04157470 2024-04-24T23:23:47.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:47.671+0000 7f9f08c00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f9f0406bbd0 con 0x7f9f04150990 2024-04-24T23:23:47.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:47.672+0000 7f9f01400640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9f04150990 0x7f9f04156170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:23:47.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:47.672+0000 7f9f01400640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9f04150990 0x7f9f04156170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.6:36218/0 (socket says 172.21.15.6:36218) 2024-04-24T23:23:47.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:47.672+0000 7f9f01400640 1 -- 172.21.15.6:0/311938563 learned_addr learned my addr 172.21.15.6:0/311938563 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:23:47.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:47.672+0000 7f9f02800640 1 --2- 172.21.15.6:0/311938563 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9f04157470 0x7f9f04158f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:23:47.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:47.672+0000 7f9f01e00640 1 --2- 172.21.15.6:0/311938563 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9f04073270 0x7f9f041536b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:23:47.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:47.672+0000 7f9f02800640 1 -- 172.21.15.6:0/311938563 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9f04150990 msgr2=0x7f9f04156170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:23:47.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:47.672+0000 7f9f02800640 1 --2- 172.21.15.6:0/311938563 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9f04150990 0x7f9f04156170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:47.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:47.672+0000 7f9f02800640 1 -- 172.21.15.6:0/311938563 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9f04073270 msgr2=0x7f9f041536b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:23:47.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:47.672+0000 7f9f02800640 1 --2- 172.21.15.6:0/311938563 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9f04073270 0x7f9f041536b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:47.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:47.672+0000 7f9f02800640 1 -- 172.21.15.6:0/311938563 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f041557c0 con 0x7f9f04157470 2024-04-24T23:23:47.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:47.672+0000 7f9f02800640 1 --2- 172.21.15.6:0/311938563 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9f04157470 0x7f9f04158f40 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f9eec033000 tx=0x7f9eec0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:23:47.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:47.672+0000 7f9efa000640 1 -- 172.21.15.6:0/311938563 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9eec038040 con 0x7f9f04157470 2024-04-24T23:23:47.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:47.672+0000 7f9efa000640 1 -- 172.21.15.6:0/311938563 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9eec053020 con 0x7f9f04157470 2024-04-24T23:23:47.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:47.672+0000 7f9efa000640 1 -- 172.21.15.6:0/311938563 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9eec0337b0 con 0x7f9f04157470 2024-04-24T23:23:47.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:47.672+0000 7f9f08c00640 1 -- 172.21.15.6:0/311938563 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f041f4de0 con 0x7f9f04157470 2024-04-24T23:23:47.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:47.672+0000 7f9f08c00640 1 -- 172.21.15.6:0/311938563 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f04154ce0 con 0x7f9f04157470 2024-04-24T23:23:47.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:47.672+0000 7f9f08c00640 1 -- 172.21.15.6:0/311938563 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9f040722a0 con 0x7f9f04157470 2024-04-24T23:23:47.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:47.679+0000 7f9efa000640 1 -- 172.21.15.6:0/311938563 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f9eec033970 con 0x7f9f04157470 2024-04-24T23:23:47.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:47.680+0000 7f9efa000640 1 --2- 172.21.15.6:0/311938563 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f9ee006be30 0x7f9ee006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:23:47.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:47.681+0000 7f9efa000640 1 -- 172.21.15.6:0/311938563 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6762+0+0 (secure 0 0 0) 0x7f9eec0bd180 con 0x7f9f04157470 2024-04-24T23:23:47.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:47.682+0000 7f9f01e00640 1 --2- 172.21.15.6:0/311938563 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f9ee006be30 0x7f9ee006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:23:47.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:47.682+0000 7f9f01e00640 1 --2- 172.21.15.6:0/311938563 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f9ee006be30 0x7f9ee006e2f0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f9efc009f40 tx=0x7f9efc0094d0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:23:47.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:47.687+0000 7f9efa000640 1 -- 172.21.15.6:0/311938563 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f9f040722a0 con 0x7f9f04157470 2024-04-24T23:23:48.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:48.253+0000 7f9f08c00640 1 -- 172.21.15.6:0/311938563 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9f0404f620 con 0x7f9f04157470 2024-04-24T23:23:48.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:48.257+0000 7f9efa000640 1 -- 172.21.15.6:0/311938563 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) ==== 94+0+6852 (secure 0 0 0) 0x7f9f0404f620 con 0x7f9f04157470 2024-04-24T23:23:48.261 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:23:48.261 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":40,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5081,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/1190759626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1190759626},{"type":"v1","addr":"172.21.15.6:6834","nonce":1190759626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":5503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/3928731059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":3928731059},{"type":"v1","addr":"172.21.15.6:6837","nonce":3928731059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":35},{"gid":5551,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6838/3253046315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":3253046315},{"type":"v1","addr":"172.21.15.190:6838","nonce":3253046315}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":38},{"gid":5599,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6839/2643233757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":2643233757},{"type":"v1","addr":"172.21.15.190:6839","nonce":2643233757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40}],"filesystems":[{"mdsmap":{"epoch":39,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:23:38.162102+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":4306,"mds_1":4282},"failed":[],"damaged":[],"stopped":[2,3,4],"info":{"gid_4282":{"gid":4282,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":12,"addr":"172.21.15.190:6837/580315269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":580315269},{"type":"v1","addr":"172.21.15.190:6837","nonce":580315269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/489092393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":489092393},{"type":"v1","addr":"172.21.15.6:6835","nonce":489092393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4306,"qdb_cluster":[4306,4282]},"id":1}]} 2024-04-24T23:23:48.262 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 40 2024-04-24T23:23:48.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:48.264+0000 7f9f08c00640 1 -- 172.21.15.6:0/311938563 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f9ee006be30 msgr2=0x7f9ee006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:23:48.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:48.264+0000 7f9f08c00640 1 --2- 172.21.15.6:0/311938563 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f9ee006be30 0x7f9ee006e2f0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f9efc009f40 tx=0x7f9efc0094d0 comp rx=0 tx=0).stop 2024-04-24T23:23:48.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:48.264+0000 7f9f08c00640 1 -- 172.21.15.6:0/311938563 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9f04157470 msgr2=0x7f9f04158f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:23:48.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:48.264+0000 7f9f08c00640 1 --2- 172.21.15.6:0/311938563 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9f04157470 0x7f9f04158f40 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f9eec033000 tx=0x7f9eec0046c0 comp rx=0 tx=0).stop 2024-04-24T23:23:48.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:48.265+0000 7f9f08c00640 1 -- 172.21.15.6:0/311938563 shutdown_connections 2024-04-24T23:23:48.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:48.265+0000 7f9f08c00640 1 --2- 172.21.15.6:0/311938563 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f9ee006be30 0x7f9ee006e2f0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:48.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:48.265+0000 7f9f08c00640 1 --2- 172.21.15.6:0/311938563 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9f04150990 0x7f9f04156170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:48.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:48.265+0000 7f9f08c00640 1 --2- 172.21.15.6:0/311938563 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9f04157470 0x7f9f04158f40 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:48.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:48.265+0000 7f9f08c00640 1 --2- 172.21.15.6:0/311938563 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9f04073270 0x7f9f041536b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:48.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:48.265+0000 7f9f08c00640 1 -- 172.21.15.6:0/311938563 >> 172.21.15.6:0/311938563 conn(0x7f9f0406b0e0 msgr2=0x7f9f040cb810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:23:48.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:48.266+0000 7f9f08c00640 1 -- 172.21.15.6:0/311938563 shutdown_connections 2024-04-24T23:23:48.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:48.266+0000 7f9f08c00640 1 -- 172.21.15.6:0/311938563 wait complete. 2024-04-24T23:23:48.329 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-04-24T23:23:49.341 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-24T23:23:49.358 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-24T23:23:50.330 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:23:50.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:50.725+0000 7f7e21a00640 1 Processor -- start 2024-04-24T23:23:50.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:50.725+0000 7f7e21a00640 1 -- start start 2024-04-24T23:23:50.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:50.727+0000 7f7e21a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f7e1c071000 0x7f7e1c06e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:23:50.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:50.727+0000 7f7e21a00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e1c0c8a50 con 0x7f7e1c06e8e0 2024-04-24T23:23:50.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:50.727+0000 7f7e21a00640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e1c15ba70 con 0x7f7e1c154db0 2024-04-24T23:23:50.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:50.727+0000 7f7e21a00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f7e1c15b5e0 con 0x7f7e1c071000 2024-04-24T23:23:50.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:50.728+0000 7f7e1b400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f7e1c06e8e0 0x7f7e1c0c9000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:55826/0 (socket says 172.21.15.6:55826) 2024-04-24T23:23:50.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:50.728+0000 7f7e1b400640 1 -- 172.21.15.6:0/3044607083 learned_addr learned my addr 172.21.15.6:0/3044607083 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:23:50.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:50.729+0000 7f7e19600640 1 -- 172.21.15.6:0/3044607083 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2176065526 0 0) 0x7f7e1c0c8a50 con 0x7f7e1c06e8e0 2024-04-24T23:23:50.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:50.729+0000 7f7e19600640 1 -- 172.21.15.6:0/3044607083 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7e00003520 con 0x7f7e1c06e8e0 2024-04-24T23:23:50.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:50.729+0000 7f7e1a000640 1 --2- 172.21.15.6:0/3044607083 >> v2:172.21.15.6:3300/0 conn(0x7f7e1c071000 0x7f7e1c06e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:23:50.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:50.729+0000 7f7e19600640 1 -- 172.21.15.6:0/3044607083 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 799383693 0 0) 0x7f7e1c15ba70 con 0x7f7e1c154db0 2024-04-24T23:23:50.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:50.729+0000 7f7e19600640 1 -- 172.21.15.6:0/3044607083 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7e1c0c8a50 con 0x7f7e1c154db0 2024-04-24T23:23:50.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:50.730+0000 7f7e19600640 1 -- 172.21.15.6:0/3044607083 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 123587399 0 0) 0x7f7e00003520 con 0x7f7e1c06e8e0 2024-04-24T23:23:50.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:50.730+0000 7f7e19600640 1 -- 172.21.15.6:0/3044607083 >> v2:172.21.15.6:3300/0 conn(0x7f7e1c071000 msgr2=0x7f7e1c06e3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:23:50.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:50.730+0000 7f7e19600640 1 --2- 172.21.15.6:0/3044607083 >> v2:172.21.15.6:3300/0 conn(0x7f7e1c071000 0x7f7e1c06e3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:50.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:50.730+0000 7f7e19600640 1 -- 172.21.15.6:0/3044607083 >> v1:172.21.15.190:6789/0 conn(0x7f7e1c154db0 legacy=0x7f7e1c0c8230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:23:50.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:50.730+0000 7f7e19600640 1 -- 172.21.15.6:0/3044607083 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e1c15acb0 con 0x7f7e1c06e8e0 2024-04-24T23:23:50.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:50.730+0000 7f7e1a000640 1 --2- 172.21.15.6:0/3044607083 >> v2:172.21.15.6:3300/0 conn(0x7f7e1c071000 0x7f7e1c06e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:23:50.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:50.730+0000 7f7e19600640 1 -- 172.21.15.6:0/3044607083 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3463268701 0 0) 0x7f7e14001970 con 0x7f7e1c06e8e0 2024-04-24T23:23:50.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:50.730+0000 7f7e19600640 1 -- 172.21.15.6:0/3044607083 >> v1:172.21.15.6:6789/0 conn(0x7f7e1c06e8e0 legacy=0x7f7e1c0c9000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:23:50.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:50.730+0000 7f7e19600640 1 --2- 172.21.15.6:0/3044607083 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7e00003b90 0x7f7e00003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:23:50.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:50.730+0000 7f7e19600640 1 --2- 172.21.15.6:0/3044607083 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7e00004800 0x7f7e00004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:23:50.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:50.730+0000 7f7e19600640 1 --2- 172.21.15.6:0/3044607083 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7e000052e0 0x7f7e0000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:23:50.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:50.733+0000 7f7e19600640 1 -- 172.21.15.6:0/3044607083 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f7e1c15b5e0 con 0x7f7e00003b90 2024-04-24T23:23:50.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:50.733+0000 7f7e19600640 1 -- 172.21.15.6:0/3044607083 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f7e0000c200 con 0x7f7e00004800 2024-04-24T23:23:50.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:50.733+0000 7f7e19600640 1 -- 172.21.15.6:0/3044607083 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f7e0000c400 con 0x7f7e000052e0 2024-04-24T23:23:50.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:50.733+0000 7f7e1a000640 1 --2- 172.21.15.6:0/3044607083 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7e00004800 0x7f7e00004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:23:50.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:50.734+0000 7f7e1a000640 1 -- 172.21.15.6:0/3044607083 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7e000052e0 msgr2=0x7f7e0000b9c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-24T23:23:50.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:50.734+0000 7f7e1a000640 1 --2- 172.21.15.6:0/3044607083 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7e000052e0 0x7f7e0000b9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:50.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:50.734+0000 7f7e1a000640 1 -- 172.21.15.6:0/3044607083 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7e00003b90 msgr2=0x7f7e00003f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:23:50.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:50.734+0000 7f7e1aa00640 1 --2- 172.21.15.6:0/3044607083 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7e00003b90 0x7f7e00003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:23:50.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:50.734+0000 7f7e1a000640 1 --2- 172.21.15.6:0/3044607083 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7e00003b90 0x7f7e00003f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:50.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:50.734+0000 7f7e1a000640 1 -- 172.21.15.6:0/3044607083 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e00003520 con 0x7f7e00004800 2024-04-24T23:23:50.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:50.734+0000 7f7e1aa00640 1 --2- 172.21.15.6:0/3044607083 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7e00003b90 0x7f7e00003f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:23:50.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:50.740+0000 7f7e1b400640 1 -- 172.21.15.6:0/3044607083 reap_dead start 2024-04-24T23:23:50.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:50.740+0000 7f7e1a000640 1 --2- 172.21.15.6:0/3044607083 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7e00004800 0x7f7e00004c80 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f7e0800aa80 tx=0x7f7e00005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e0e2d3139397b6f3 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:23:50.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:50.744+0000 7f7e19600640 1 -- 172.21.15.6:0/3044607083 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e08037070 con 0x7f7e00004800 2024-04-24T23:23:50.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:50.744+0000 7f7e19600640 1 -- 172.21.15.6:0/3044607083 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e0802ac70 con 0x7f7e00004800 2024-04-24T23:23:50.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:50.744+0000 7f7e19600640 1 -- 172.21.15.6:0/3044607083 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e0800a650 con 0x7f7e00004800 2024-04-24T23:23:50.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:50.746+0000 7f7e21a00640 1 -- 172.21.15.6:0/3044607083 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7e00004800 msgr2=0x7f7e00004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:23:50.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:50.747+0000 7f7e21a00640 1 --2- 172.21.15.6:0/3044607083 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7e00004800 0x7f7e00004c80 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f7e0800aa80 tx=0x7f7e00005830 comp rx=0 tx=0).stop 2024-04-24T23:23:50.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:50.747+0000 7f7e21a00640 1 -- 172.21.15.6:0/3044607083 shutdown_connections 2024-04-24T23:23:50.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:50.747+0000 7f7e21a00640 1 --2- 172.21.15.6:0/3044607083 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7e00004800 0x7f7e00004c80 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:50.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:50.747+0000 7f7e21a00640 1 -- 172.21.15.6:0/3044607083 >> 172.21.15.6:0/3044607083 conn(0x7f7e1c06b0e0 msgr2=0x7f7e1c06b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:23:50.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:50.748+0000 7f7e21a00640 1 -- 172.21.15.6:0/3044607083 shutdown_connections 2024-04-24T23:23:50.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:50.749+0000 7f7e21a00640 1 -- 172.21.15.6:0/3044607083 wait complete. 2024-04-24T23:23:50.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:50.754+0000 7f7e21a00640 1 Processor -- start 2024-04-24T23:23:50.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:50.755+0000 7f7e21a00640 1 -- start start 2024-04-24T23:23:50.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:50.756+0000 7f7e21a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7e00004800 0x7f7e1c0cb060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:23:50.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:50.757+0000 7f7e21a00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7e1c1e5060 0x7f7e1c0cc950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:23:50.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:50.757+0000 7f7e21a00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7e1c1e48f0 0x7f7e1c1f4740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:23:50.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:50.757+0000 7f7e21a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f7e1c071430 con 0x7f7e00004800 2024-04-24T23:23:50.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:50.757+0000 7f7e1a000640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7e1c1e5060 0x7f7e1c0cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:23:50.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:50.757+0000 7f7e1a000640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7e1c1e5060 0x7f7e1c0cc950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.6:36236/0 (socket says 172.21.15.6:36236) 2024-04-24T23:23:50.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:50.757+0000 7f7e1a000640 1 -- 172.21.15.6:0/3378883939 learned_addr learned my addr 172.21.15.6:0/3378883939 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:23:50.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:50.759+0000 7f7e1aa00640 1 --2- 172.21.15.6:0/3378883939 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7e00004800 0x7f7e1c0cb060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:23:50.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:50.759+0000 7f7e21a00640 1 -- 172.21.15.6:0/3378883939 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f7e1c15b270 con 0x7f7e1c1e48f0 2024-04-24T23:23:50.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:50.760+0000 7f7e21a00640 1 -- 172.21.15.6:0/3378883939 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f7e1c155190 con 0x7f7e1c1e5060 2024-04-24T23:23:50.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:50.760+0000 7f7e1b400640 1 --2- 172.21.15.6:0/3378883939 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7e1c1e48f0 0x7f7e1c1f4740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:23:50.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:50.763+0000 7f7e1a000640 1 -- 172.21.15.6:0/3378883939 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7e1c1e48f0 msgr2=0x7f7e1c1f4740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:23:50.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:50.763+0000 7f7e1a000640 1 --2- 172.21.15.6:0/3378883939 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7e1c1e48f0 0x7f7e1c1f4740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:50.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:50.763+0000 7f7e1a000640 1 -- 172.21.15.6:0/3378883939 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7e00004800 msgr2=0x7f7e1c0cb060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:23:50.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:50.764+0000 7f7e1a000640 1 --2- 172.21.15.6:0/3378883939 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7e00004800 0x7f7e1c0cb060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:50.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:50.764+0000 7f7e1a000640 1 -- 172.21.15.6:0/3378883939 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e1c1f4c80 con 0x7f7e1c1e5060 2024-04-24T23:23:50.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:50.764+0000 7f7e1aa00640 1 --2- 172.21.15.6:0/3378883939 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7e00004800 0x7f7e1c0cb060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:23:50.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:50.764+0000 7f7e1a000640 1 --2- 172.21.15.6:0/3378883939 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7e1c1e5060 0x7f7e1c0cc950 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f7e0800aba0 tx=0x7f7e08004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:23:50.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:50.765+0000 7f7e12a00640 1 -- 172.21.15.6:0/3378883939 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e08041020 con 0x7f7e1c1e5060 2024-04-24T23:23:50.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:50.765+0000 7f7e12a00640 1 -- 172.21.15.6:0/3378883939 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e0803d420 con 0x7f7e1c1e5060 2024-04-24T23:23:50.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:50.765+0000 7f7e12a00640 1 -- 172.21.15.6:0/3378883939 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e08037040 con 0x7f7e1c1e5060 2024-04-24T23:23:50.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:50.765+0000 7f7e21a00640 1 -- 172.21.15.6:0/3378883939 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e1c1e6780 con 0x7f7e1c1e5060 2024-04-24T23:23:50.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:50.766+0000 7f7e21a00640 1 -- 172.21.15.6:0/3378883939 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e1c1f3f40 con 0x7f7e1c1e5060 2024-04-24T23:23:50.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:50.768+0000 7f7e21a00640 1 -- 172.21.15.6:0/3378883939 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e1c1e6ca0 con 0x7f7e1c1e5060 2024-04-24T23:23:50.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:50.769+0000 7f7e12a00640 1 -- 172.21.15.6:0/3378883939 <== mon.2 v2:172.21.15.190:3301/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f7e08044020 con 0x7f7e1c1e5060 2024-04-24T23:23:50.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:50.771+0000 7f7e12a00640 1 --2- 172.21.15.6:0/3378883939 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f7dfc06bf00 0x7f7dfc06e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:23:50.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:50.771+0000 7f7e12a00640 1 -- 172.21.15.6:0/3378883939 <== mon.2 v2:172.21.15.190:3301/0 5 ==== osd_map(28..28 src has 1..28) ==== 6762+0+0 (secure 0 0 0) 0x7f7e080ab740 con 0x7f7e1c1e5060 2024-04-24T23:23:50.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:50.771+0000 7f7e1aa00640 1 --2- 172.21.15.6:0/3378883939 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f7dfc06bf00 0x7f7dfc06e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:23:50.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:50.772+0000 7f7e1aa00640 1 --2- 172.21.15.6:0/3378883939 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f7dfc06bf00 0x7f7dfc06e3c0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f7e0c001460 tx=0x7f7e0c0013f0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:23:50.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:50.776+0000 7f7e12a00640 1 -- 172.21.15.6:0/3378883939 <== mon.2 v2:172.21.15.190:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f7e1c1e6ca0 con 0x7f7e1c1e5060 2024-04-24T23:23:51.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:51.209+0000 7f7e21a00640 1 -- 172.21.15.6:0/3378883939 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7e1c0767b0 con 0x7f7e1c1e5060 2024-04-24T23:23:51.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:51.210+0000 7f7e12a00640 1 -- 172.21.15.6:0/3378883939 <== mon.2 v2:172.21.15.190:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) ==== 94+0+6852 (secure 0 0 0) 0x7f7e0804b030 con 0x7f7e1c1e5060 2024-04-24T23:23:51.212 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:23:51.212 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":40,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5081,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/1190759626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1190759626},{"type":"v1","addr":"172.21.15.6:6834","nonce":1190759626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":5503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/3928731059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":3928731059},{"type":"v1","addr":"172.21.15.6:6837","nonce":3928731059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":35},{"gid":5551,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6838/3253046315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":3253046315},{"type":"v1","addr":"172.21.15.190:6838","nonce":3253046315}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":38},{"gid":5599,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6839/2643233757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":2643233757},{"type":"v1","addr":"172.21.15.190:6839","nonce":2643233757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40}],"filesystems":[{"mdsmap":{"epoch":39,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:23:38.162102+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":4306,"mds_1":4282},"failed":[],"damaged":[],"stopped":[2,3,4],"info":{"gid_4282":{"gid":4282,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":12,"addr":"172.21.15.190:6837/580315269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":580315269},{"type":"v1","addr":"172.21.15.190:6837","nonce":580315269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/489092393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":489092393},{"type":"v1","addr":"172.21.15.6:6835","nonce":489092393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4306,"qdb_cluster":[4306,4282]},"id":1}]} 2024-04-24T23:23:51.212 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 40 2024-04-24T23:23:51.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:51.214+0000 7f7e21a00640 1 -- 172.21.15.6:0/3378883939 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f7dfc06bf00 msgr2=0x7f7dfc06e3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:23:51.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:51.214+0000 7f7e21a00640 1 --2- 172.21.15.6:0/3378883939 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f7dfc06bf00 0x7f7dfc06e3c0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f7e0c001460 tx=0x7f7e0c0013f0 comp rx=0 tx=0).stop 2024-04-24T23:23:51.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:51.214+0000 7f7e21a00640 1 -- 172.21.15.6:0/3378883939 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7e1c1e5060 msgr2=0x7f7e1c0cc950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:23:51.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:51.214+0000 7f7e21a00640 1 --2- 172.21.15.6:0/3378883939 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7e1c1e5060 0x7f7e1c0cc950 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f7e0800aba0 tx=0x7f7e08004730 comp rx=0 tx=0).stop 2024-04-24T23:23:51.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:51.214+0000 7f7e21a00640 1 -- 172.21.15.6:0/3378883939 shutdown_connections 2024-04-24T23:23:51.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:51.214+0000 7f7e21a00640 1 --2- 172.21.15.6:0/3378883939 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f7dfc06bf00 0x7f7dfc06e3c0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:51.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:51.214+0000 7f7e21a00640 1 --2- 172.21.15.6:0/3378883939 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7e1c1e5060 0x7f7e1c0cc950 secure :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f7e0800aba0 tx=0x7f7e08004730 comp rx=0 tx=0).stop 2024-04-24T23:23:51.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:51.214+0000 7f7e21a00640 1 --2- 172.21.15.6:0/3378883939 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7e1c1e48f0 0x7f7e1c1f4740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:51.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:51.214+0000 7f7e21a00640 1 --2- 172.21.15.6:0/3378883939 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7e00004800 0x7f7e1c0cb060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:51.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:51.214+0000 7f7e21a00640 1 -- 172.21.15.6:0/3378883939 >> 172.21.15.6:0/3378883939 conn(0x7f7e1c06b0e0 msgr2=0x7f7e1c06ecf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:23:51.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:51.214+0000 7f7e21a00640 1 -- 172.21.15.6:0/3378883939 shutdown_connections 2024-04-24T23:23:51.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:51.214+0000 7f7e21a00640 1 -- 172.21.15.6:0/3378883939 wait complete. 2024-04-24T23:23:51.247 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-04-24T23:23:53.248 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:23:53.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:53.822+0000 7f97b4800640 1 Processor -- start 2024-04-24T23:23:53.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:53.823+0000 7f97b4800640 1 -- start start 2024-04-24T23:23:53.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:53.825+0000 7f97b4800640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f97ac1593b0 0x7f97ac0c6e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:23:53.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:53.825+0000 7f97b4800640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f97ac06edb0 0x7f97ac0c7e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:23:53.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:53.825+0000 7f97b4800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f97ac0c8390 0x7f97ac162460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:23:53.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:53.825+0000 7f97b4800640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f97ac072470 con 0x7f97ac0c8390 2024-04-24T23:23:53.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:53.825+0000 7f97b4800640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f97ac071880 con 0x7f97ac06edb0 2024-04-24T23:23:53.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:53.825+0000 7f97b4800640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f97ac072900 con 0x7f97ac1593b0 2024-04-24T23:23:53.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:53.826+0000 7f97b1400640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f97ac06edb0 0x7f97ac0c7e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:23:53.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:53.826+0000 7f97b1400640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f97ac06edb0 0x7f97ac0c7e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:56950/0 (socket says 172.21.15.6:56950) 2024-04-24T23:23:53.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:53.826+0000 7f97b1400640 1 -- 172.21.15.6:0/2234137649 learned_addr learned my addr 172.21.15.6:0/2234137649 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:23:53.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:53.826+0000 7f97b1e00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f97ac1593b0 0x7f97ac0c6e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:23:53.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:53.826+0000 7f97b1400640 1 -- 172.21.15.6:0/2234137649 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f97ac1593b0 msgr2=0x7f97ac0c6e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:23:53.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:53.826+0000 7f97b1400640 1 --2- 172.21.15.6:0/2234137649 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f97ac1593b0 0x7f97ac0c6e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:53.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:53.826+0000 7f97b1400640 1 -- 172.21.15.6:0/2234137649 >> v2:172.21.15.6:3300/0 conn(0x7f97ac0c8390 msgr2=0x7f97ac162460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:23:53.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:53.826+0000 7f97b1400640 1 --2- 172.21.15.6:0/2234137649 >> v2:172.21.15.6:3300/0 conn(0x7f97ac0c8390 0x7f97ac162460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:53.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:53.826+0000 7f97b1400640 1 -- 172.21.15.6:0/2234137649 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97ac0c7410 con 0x7f97ac06edb0 2024-04-24T23:23:53.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:53.828+0000 7f97b1400640 1 --2- 172.21.15.6:0/2234137649 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f97ac06edb0 0x7f97ac0c7e50 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f97ac06bf90 tx=0x7f979c02a260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=831b6bc46e14c4ae server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:23:53.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:53.829+0000 7f97b0a00640 1 -- 172.21.15.6:0/2234137649 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f979c037070 con 0x7f97ac06edb0 2024-04-24T23:23:53.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:53.829+0000 7f97b0a00640 1 -- 172.21.15.6:0/2234137649 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f979c032440 con 0x7f97ac06edb0 2024-04-24T23:23:53.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:53.829+0000 7f97b0a00640 1 -- 172.21.15.6:0/2234137649 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f979c032810 con 0x7f97ac06edb0 2024-04-24T23:23:53.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:53.830+0000 7f97b4800640 1 -- 172.21.15.6:0/2234137649 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f97ac06edb0 msgr2=0x7f97ac0c7e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:23:53.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:53.830+0000 7f97b4800640 1 --2- 172.21.15.6:0/2234137649 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f97ac06edb0 0x7f97ac0c7e50 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f97ac06bf90 tx=0x7f979c02a260 comp rx=0 tx=0).stop 2024-04-24T23:23:53.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:53.831+0000 7f97b4800640 1 -- 172.21.15.6:0/2234137649 shutdown_connections 2024-04-24T23:23:53.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:53.831+0000 7f97b4800640 1 --2- 172.21.15.6:0/2234137649 >> v2:172.21.15.6:3300/0 conn(0x7f97ac0c8390 0x7f97ac162460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:53.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:53.831+0000 7f97b4800640 1 --2- 172.21.15.6:0/2234137649 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f97ac06edb0 0x7f97ac0c7e50 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:53.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:53.831+0000 7f97b4800640 1 --2- 172.21.15.6:0/2234137649 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f97ac1593b0 0x7f97ac0c6e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:53.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:53.831+0000 7f97b4800640 1 -- 172.21.15.6:0/2234137649 >> 172.21.15.6:0/2234137649 conn(0x7f97ac06b0e0 msgr2=0x7f97ac06b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:23:53.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:53.832+0000 7f97b4800640 1 -- 172.21.15.6:0/2234137649 shutdown_connections 2024-04-24T23:23:53.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:53.832+0000 7f97b4800640 1 -- 172.21.15.6:0/2234137649 wait complete. 2024-04-24T23:23:53.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:53.835+0000 7f97b4800640 1 Processor -- start 2024-04-24T23:23:53.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:53.835+0000 7f97b4800640 1 -- start start 2024-04-24T23:23:53.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:53.837+0000 7f97b4800640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f97ac06edb0 0x7f97ac1f8c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:23:53.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:53.837+0000 7f97b4800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f97ac0c8390 0x7f97ac0c4100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:23:53.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:53.837+0000 7f97b4800640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f97ac1593b0 0x7f97ac1e8280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:23:53.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:53.837+0000 7f97b4800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f97ac072100 con 0x7f97ac0c8390 2024-04-24T23:23:53.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:53.837+0000 7f97b4800640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f97ac071e20 con 0x7f97ac06edb0 2024-04-24T23:23:53.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:53.837+0000 7f97b4800640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f97ac163080 con 0x7f97ac1593b0 2024-04-24T23:23:53.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:53.838+0000 7f97b1e00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f97ac06edb0 0x7f97ac1f8c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:23:53.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:53.838+0000 7f97b1400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f97ac0c8390 0x7f97ac0c4100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:23:53.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:53.838+0000 7f97b1400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f97ac0c8390 0x7f97ac0c4100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57210/0 (socket says 172.21.15.6:57210) 2024-04-24T23:23:53.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:53.838+0000 7f97b1400640 1 -- 172.21.15.6:0/3413400382 learned_addr learned my addr 172.21.15.6:0/3413400382 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:23:53.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:53.839+0000 7f97b2800640 1 --2- 172.21.15.6:0/3413400382 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f97ac1593b0 0x7f97ac1e8280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:23:53.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:53.839+0000 7f97b1e00640 1 -- 172.21.15.6:0/3413400382 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f97ac1593b0 msgr2=0x7f97ac1e8280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:23:53.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:53.839+0000 7f97b1e00640 1 --2- 172.21.15.6:0/3413400382 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f97ac1593b0 0x7f97ac1e8280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:53.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:53.839+0000 7f97b1e00640 1 -- 172.21.15.6:0/3413400382 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f97ac0c8390 msgr2=0x7f97ac0c4100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:23:53.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:53.839+0000 7f97b1e00640 1 --2- 172.21.15.6:0/3413400382 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f97ac0c8390 0x7f97ac0c4100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:53.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:53.839+0000 7f97b1e00640 1 -- 172.21.15.6:0/3413400382 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97ac0c4640 con 0x7f97ac06edb0 2024-04-24T23:23:53.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:53.840+0000 7f97b1e00640 1 --2- 172.21.15.6:0/3413400382 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f97ac06edb0 0x7f97ac1f8c60 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f97a000e8a0 tx=0x7f97a000ed60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:23:53.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:53.840+0000 7f97a6000640 1 -- 172.21.15.6:0/3413400382 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97a0017070 con 0x7f97ac06edb0 2024-04-24T23:23:53.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:53.841+0000 7f97a6000640 1 -- 172.21.15.6:0/3413400382 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97a000c8e0 con 0x7f97ac06edb0 2024-04-24T23:23:53.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:53.841+0000 7f97a6000640 1 -- 172.21.15.6:0/3413400382 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97a0009e60 con 0x7f97ac06edb0 2024-04-24T23:23:53.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:53.842+0000 7f97b4800640 1 -- 172.21.15.6:0/3413400382 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97ac152ed0 con 0x7f97ac06edb0 2024-04-24T23:23:53.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:53.843+0000 7f97b4800640 1 -- 172.21.15.6:0/3413400382 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97ac1f4ed0 con 0x7f97ac06edb0 2024-04-24T23:23:53.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:53.844+0000 7f97b4800640 1 -- 172.21.15.6:0/3413400382 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97ac154e20 con 0x7f97ac06edb0 2024-04-24T23:23:53.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:53.846+0000 7f97a6000640 1 -- 172.21.15.6:0/3413400382 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f97a001a070 con 0x7f97ac06edb0 2024-04-24T23:23:53.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:53.847+0000 7f97a6000640 1 --2- 172.21.15.6:0/3413400382 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f979006be80 0x7f979006e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:23:53.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:53.848+0000 7f97a6000640 1 -- 172.21.15.6:0/3413400382 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6762+0+0 (secure 0 0 0) 0x7f97a0056f60 con 0x7f97ac06edb0 2024-04-24T23:23:53.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:53.848+0000 7f97b1400640 1 --2- 172.21.15.6:0/3413400382 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f979006be80 0x7f979006e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:23:53.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:53.849+0000 7f97b1400640 1 --2- 172.21.15.6:0/3413400382 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f979006be80 0x7f979006e340 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f97ac1ec3a0 tx=0x7f979c02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:23:53.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:53.855+0000 7f97a6000640 1 -- 172.21.15.6:0/3413400382 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f97ac154e20 con 0x7f97ac06edb0 2024-04-24T23:23:54.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:54.416+0000 7f97b4800640 1 -- 172.21.15.6:0/3413400382 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f97ac1f4ce0 con 0x7f97ac06edb0 2024-04-24T23:23:54.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:54.417+0000 7f97a6000640 1 -- 172.21.15.6:0/3413400382 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) ==== 94+0+6024 (secure 0 0 0) 0x7f97a0058ec0 con 0x7f97ac06edb0 2024-04-24T23:23:54.422 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:23:54.422 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":41,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5081,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/1190759626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1190759626},{"type":"v1","addr":"172.21.15.6:6834","nonce":1190759626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":5503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/3928731059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":3928731059},{"type":"v1","addr":"172.21.15.6:6837","nonce":3928731059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":35},{"gid":5599,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6839/2643233757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":2643233757},{"type":"v1","addr":"172.21.15.190:6839","nonce":2643233757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40}],"filesystems":[{"mdsmap":{"epoch":39,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:23:38.162102+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":4306,"mds_1":4282},"failed":[],"damaged":[],"stopped":[2,3,4],"info":{"gid_4282":{"gid":4282,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":12,"addr":"172.21.15.190:6837/580315269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":580315269},{"type":"v1","addr":"172.21.15.190:6837","nonce":580315269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/489092393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":489092393},{"type":"v1","addr":"172.21.15.6:6835","nonce":489092393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4306,"qdb_cluster":[4282,4306]},"id":1}]} 2024-04-24T23:23:54.422 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 41 2024-04-24T23:23:54.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:54.427+0000 7f97b4800640 1 -- 172.21.15.6:0/3413400382 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f979006be80 msgr2=0x7f979006e340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:23:54.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:54.427+0000 7f97b4800640 1 --2- 172.21.15.6:0/3413400382 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f979006be80 0x7f979006e340 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f97ac1ec3a0 tx=0x7f979c02e000 comp rx=0 tx=0).stop 2024-04-24T23:23:54.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:54.427+0000 7f97b4800640 1 -- 172.21.15.6:0/3413400382 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f97ac06edb0 msgr2=0x7f97ac1f8c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:23:54.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:54.427+0000 7f97b4800640 1 --2- 172.21.15.6:0/3413400382 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f97ac06edb0 0x7f97ac1f8c60 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f97a000e8a0 tx=0x7f97a000ed60 comp rx=0 tx=0).stop 2024-04-24T23:23:54.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:54.428+0000 7f97b4800640 1 -- 172.21.15.6:0/3413400382 shutdown_connections 2024-04-24T23:23:54.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:54.428+0000 7f97b4800640 1 --2- 172.21.15.6:0/3413400382 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f979006be80 0x7f979006e340 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:54.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:54.428+0000 7f97b4800640 1 --2- 172.21.15.6:0/3413400382 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f97ac1593b0 0x7f97ac1e8280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:54.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:54.428+0000 7f97b4800640 1 --2- 172.21.15.6:0/3413400382 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f97ac0c8390 0x7f97ac0c4100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:54.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:54.428+0000 7f97b4800640 1 --2- 172.21.15.6:0/3413400382 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f97ac06edb0 0x7f97ac1f8c60 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:54.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:54.428+0000 7f97b4800640 1 -- 172.21.15.6:0/3413400382 >> 172.21.15.6:0/3413400382 conn(0x7f97ac06b0e0 msgr2=0x7f97ac154710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:23:54.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:54.428+0000 7f97b4800640 1 -- 172.21.15.6:0/3413400382 shutdown_connections 2024-04-24T23:23:54.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:54.428+0000 7f97b4800640 1 -- 172.21.15.6:0/3413400382 wait complete. 2024-04-24T23:23:54.480 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b down, removed from mdsmap 2024-04-24T23:23:54.480 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2024-04-24T23:23:54.480 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:23:55.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:55.953+0000 7f6230400640 1 Processor -- start 2024-04-24T23:23:55.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:55.953+0000 7f6230400640 1 -- start start 2024-04-24T23:23:55.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:55.955+0000 7f6230400640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f6228159110 0x7f6228072420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:23:55.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:55.955+0000 7f6230400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f62280c7dc0 0x7f62281623a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:23:55.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:55.955+0000 7f6230400640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f62280c81c0 0x7f62281628e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:23:55.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:55.955+0000 7f6230400640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f62281594f0 con 0x7f62280c7dc0 2024-04-24T23:23:55.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:55.955+0000 7f6230400640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f62280c7230 con 0x7f6228159110 2024-04-24T23:23:55.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:55.955+0000 7f6230400640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f62280c74d0 con 0x7f62280c81c0 2024-04-24T23:23:55.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:55.956+0000 7f622d000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f62280c7dc0 0x7f62281623a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:23:55.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:55.956+0000 7f622d000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f62280c7dc0 0x7f62281623a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:34840/0 (socket says 172.21.15.6:34840) 2024-04-24T23:23:55.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:55.956+0000 7f622d000640 1 -- 172.21.15.6:0/691258885 learned_addr learned my addr 172.21.15.6:0/691258885 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:23:55.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:55.957+0000 7f622da00640 1 --2- 172.21.15.6:0/691258885 >> v2:172.21.15.190:3300/0 conn(0x7f6228159110 0x7f6228072420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:23:55.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:55.957+0000 7f622e400640 1 --2- 172.21.15.6:0/691258885 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f62280c81c0 0x7f62281628e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:23:55.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:55.959+0000 7f622da00640 1 -- 172.21.15.6:0/691258885 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f62280c81c0 msgr2=0x7f62281628e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:23:55.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:55.959+0000 7f622da00640 1 --2- 172.21.15.6:0/691258885 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f62280c81c0 0x7f62281628e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:55.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:55.959+0000 7f622da00640 1 -- 172.21.15.6:0/691258885 >> v2:172.21.15.6:3300/0 conn(0x7f62280c7dc0 msgr2=0x7f62281623a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:23:55.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:55.959+0000 7f622da00640 1 --2- 172.21.15.6:0/691258885 >> v2:172.21.15.6:3300/0 conn(0x7f62280c7dc0 0x7f62281623a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:55.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:55.959+0000 7f622da00640 1 -- 172.21.15.6:0/691258885 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f622806e440 con 0x7f6228159110 2024-04-24T23:23:55.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:55.961+0000 7f622da00640 1 --2- 172.21.15.6:0/691258885 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6228159110 0x7f6228072420 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f62280c9a30 tx=0x7f621c02a1e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=badf9c72f1b4df1e server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:23:55.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:55.962+0000 7f6227e00640 1 -- 172.21.15.6:0/691258885 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f621c037070 con 0x7f6228159110 2024-04-24T23:23:55.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:55.962+0000 7f6227e00640 1 -- 172.21.15.6:0/691258885 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f621c032440 con 0x7f6228159110 2024-04-24T23:23:55.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:55.962+0000 7f6227e00640 1 -- 172.21.15.6:0/691258885 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f621c032850 con 0x7f6228159110 2024-04-24T23:23:55.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:55.965+0000 7f6230400640 1 -- 172.21.15.6:0/691258885 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6228159110 msgr2=0x7f6228072420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:23:55.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:55.965+0000 7f6230400640 1 --2- 172.21.15.6:0/691258885 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6228159110 0x7f6228072420 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f62280c9a30 tx=0x7f621c02a1e0 comp rx=0 tx=0).stop 2024-04-24T23:23:55.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:55.966+0000 7f6230400640 1 -- 172.21.15.6:0/691258885 shutdown_connections 2024-04-24T23:23:55.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:55.966+0000 7f6230400640 1 --2- 172.21.15.6:0/691258885 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f62280c81c0 0x7f62281628e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:55.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:55.966+0000 7f6230400640 1 --2- 172.21.15.6:0/691258885 >> v2:172.21.15.6:3300/0 conn(0x7f62280c7dc0 0x7f62281623a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:55.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:55.966+0000 7f6230400640 1 --2- 172.21.15.6:0/691258885 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6228159110 0x7f6228072420 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:55.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:55.966+0000 7f6230400640 1 -- 172.21.15.6:0/691258885 >> 172.21.15.6:0/691258885 conn(0x7f622806b0e0 msgr2=0x7f622806b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:23:55.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:55.967+0000 7f6230400640 1 -- 172.21.15.6:0/691258885 shutdown_connections 2024-04-24T23:23:55.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:55.967+0000 7f6230400640 1 -- 172.21.15.6:0/691258885 wait complete. 2024-04-24T23:23:55.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:55.971+0000 7f6230400640 1 Processor -- start 2024-04-24T23:23:55.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:55.971+0000 7f6230400640 1 -- start start 2024-04-24T23:23:55.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:55.977+0000 7f6230400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f62280c7dc0 0x7f62281f5880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:23:55.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:55.977+0000 7f6230400640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f62280c81c0 0x7f62281e9af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:23:55.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:55.977+0000 7f6230400640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6228159110 0x7f62281625f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:23:55.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:55.977+0000 7f6230400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f622806ece0 con 0x7f62280c7dc0 2024-04-24T23:23:55.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:55.977+0000 7f6230400640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f622806ea00 con 0x7f62280c81c0 2024-04-24T23:23:55.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:55.977+0000 7f6230400640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f6228072960 con 0x7f6228159110 2024-04-24T23:23:55.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:55.977+0000 7f622e400640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6228159110 0x7f62281625f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:23:55.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:55.977+0000 7f622e400640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6228159110 0x7f62281625f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.6:60448/0 (socket says 172.21.15.6:60448) 2024-04-24T23:23:55.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:55.977+0000 7f622e400640 1 -- 172.21.15.6:0/3376042501 learned_addr learned my addr 172.21.15.6:0/3376042501 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:23:55.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:55.978+0000 7f622e400640 1 -- 172.21.15.6:0/3376042501 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f62280c81c0 msgr2=0x7f62281e9af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:23:55.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:55.978+0000 7f622e400640 1 --2- 172.21.15.6:0/3376042501 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f62280c81c0 0x7f62281e9af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:55.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:55.978+0000 7f622e400640 1 -- 172.21.15.6:0/3376042501 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f62280c7dc0 msgr2=0x7f62281f5880 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-24T23:23:55.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:55.978+0000 7f622e400640 1 --2- 172.21.15.6:0/3376042501 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f62280c7dc0 0x7f62281f5880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:55.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:55.978+0000 7f622e400640 1 -- 172.21.15.6:0/3376042501 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62281f8410 con 0x7f6228159110 2024-04-24T23:23:55.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:55.979+0000 7f622e400640 1 --2- 172.21.15.6:0/3376042501 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6228159110 0x7f62281625f0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f6220009dd0 tx=0x7f622000c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:23:55.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:55.980+0000 7f6225600640 1 -- 172.21.15.6:0/3376042501 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6220018020 con 0x7f6228159110 2024-04-24T23:23:55.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:55.980+0000 7f6225600640 1 -- 172.21.15.6:0/3376042501 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6220014480 con 0x7f6228159110 2024-04-24T23:23:55.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:55.980+0000 7f6225600640 1 -- 172.21.15.6:0/3376042501 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6220014870 con 0x7f6228159110 2024-04-24T23:23:55.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:55.980+0000 7f6230400640 1 -- 172.21.15.6:0/3376042501 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f622815abe0 con 0x7f6228159110 2024-04-24T23:23:55.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:55.981+0000 7f6230400640 1 -- 172.21.15.6:0/3376042501 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f62281f8680 con 0x7f6228159110 2024-04-24T23:23:55.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:55.984+0000 7f6230400640 1 -- 172.21.15.6:0/3376042501 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f62280767b0 con 0x7f6228159110 2024-04-24T23:23:55.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:55.985+0000 7f6225600640 1 -- 172.21.15.6:0/3376042501 <== mon.2 v2:172.21.15.190:3301/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f6220005d10 con 0x7f6228159110 2024-04-24T23:23:55.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:55.986+0000 7f6225600640 1 --2- 172.21.15.6:0/3376042501 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f620806be80 0x7f620806e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:23:55.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:55.986+0000 7f6225600640 1 -- 172.21.15.6:0/3376042501 <== mon.2 v2:172.21.15.190:3301/0 5 ==== osd_map(28..28 src has 1..28) ==== 6762+0+0 (secure 0 0 0) 0x7f622008b980 con 0x7f6228159110 2024-04-24T23:23:55.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:55.986+0000 7f622da00640 1 --2- 172.21.15.6:0/3376042501 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f620806be80 0x7f620806e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:23:55.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:55.987+0000 7f622da00640 1 --2- 172.21.15.6:0/3376042501 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f620806be80 0x7f620806e340 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f621c034000 tx=0x7f621c02c000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:23:55.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:55.991+0000 7f6225600640 1 -- 172.21.15.6:0/3376042501 <== mon.2 v2:172.21.15.190:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f6220010040 con 0x7f6228159110 2024-04-24T23:23:56.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:56.522+0000 7f6230400640 1 -- 172.21.15.6:0/3376042501 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6228068530 con 0x7f6228159110 2024-04-24T23:23:56.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:56.528+0000 7f6225600640 1 -- 172.21.15.6:0/3376042501 <== mon.2 v2:172.21.15.190:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) ==== 94+0+6024 (secure 0 0 0) 0x7f6220059f60 con 0x7f6228159110 2024-04-24T23:23:56.530 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:23:56.530 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":41,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5081,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/1190759626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1190759626},{"type":"v1","addr":"172.21.15.6:6834","nonce":1190759626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":5503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/3928731059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":3928731059},{"type":"v1","addr":"172.21.15.6:6837","nonce":3928731059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":35},{"gid":5599,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6839/2643233757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":2643233757},{"type":"v1","addr":"172.21.15.190:6839","nonce":2643233757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40}],"filesystems":[{"mdsmap":{"epoch":39,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:23:38.162102+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":4306,"mds_1":4282},"failed":[],"damaged":[],"stopped":[2,3,4],"info":{"gid_4282":{"gid":4282,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":12,"addr":"172.21.15.190:6837/580315269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":580315269},{"type":"v1","addr":"172.21.15.190:6837","nonce":580315269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/489092393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":489092393},{"type":"v1","addr":"172.21.15.6:6835","nonce":489092393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4306,"qdb_cluster":[4282,4306]},"id":1}]} 2024-04-24T23:23:56.531 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 41 2024-04-24T23:23:56.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:56.534+0000 7f6230400640 1 -- 172.21.15.6:0/3376042501 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f620806be80 msgr2=0x7f620806e340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:23:56.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:56.534+0000 7f6230400640 1 --2- 172.21.15.6:0/3376042501 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f620806be80 0x7f620806e340 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f621c034000 tx=0x7f621c02c000 comp rx=0 tx=0).stop 2024-04-24T23:23:56.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:56.534+0000 7f6230400640 1 -- 172.21.15.6:0/3376042501 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6228159110 msgr2=0x7f62281625f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:23:56.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:56.534+0000 7f6230400640 1 --2- 172.21.15.6:0/3376042501 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6228159110 0x7f62281625f0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f6220009dd0 tx=0x7f622000c490 comp rx=0 tx=0).stop 2024-04-24T23:23:56.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:56.534+0000 7f6230400640 1 -- 172.21.15.6:0/3376042501 shutdown_connections 2024-04-24T23:23:56.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:56.534+0000 7f6230400640 1 --2- 172.21.15.6:0/3376042501 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f620806be80 0x7f620806e340 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:56.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:56.534+0000 7f6230400640 1 --2- 172.21.15.6:0/3376042501 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6228159110 0x7f62281625f0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:56.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:56.534+0000 7f6230400640 1 --2- 172.21.15.6:0/3376042501 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f62280c81c0 0x7f62281e9af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:56.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:56.534+0000 7f6230400640 1 --2- 172.21.15.6:0/3376042501 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f62280c7dc0 0x7f62281f5880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:56.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:56.534+0000 7f6230400640 1 -- 172.21.15.6:0/3376042501 >> 172.21.15.6:0/3376042501 conn(0x7f622806b0e0 msgr2=0x7f6228074630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:23:56.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:56.534+0000 7f6230400640 1 -- 172.21.15.6:0/3376042501 shutdown_connections 2024-04-24T23:23:56.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:56.534+0000 7f6230400640 1 -- 172.21.15.6:0/3376042501 wait complete. 2024-04-24T23:23:56.577 INFO:tasks.mds_thrash.fs.[cephfs]:cluster has 2 actives (max_mds is 2), no MDS can replace rank 3 2024-04-24T23:23:56.577 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 0 secs before reviving mds.b 2024-04-24T23:23:56.577 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.b 2024-04-24T23:23:56.577 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-24T23:23:56.577 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-24T23:23:56.580 INFO:tasks.ceph.mds.b:Started 2024-04-24T23:23:56.580 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:23:56.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:56.935+0000 7fdaa3600640 1 Processor -- start 2024-04-24T23:23:56.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:56.935+0000 7fdaa3600640 1 -- start start 2024-04-24T23:23:56.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:56.938+0000 7fdaa3600640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fda9c0c7dc0 0x7fda9c1623a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:23:56.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:56.938+0000 7fdaa3600640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fda9c0c81c0 0x7fda9c1628e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:23:56.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:56.938+0000 7fdaa3600640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda9c0c7230 con 0x7fda9c159110 2024-04-24T23:23:56.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:56.938+0000 7fdaa3600640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7fda9c1594f0 con 0x7fda9c0c81c0 2024-04-24T23:23:56.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:56.938+0000 7fdaa3600640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7fda9c0c74d0 con 0x7fda9c0c7dc0 2024-04-24T23:23:56.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:56.940+0000 7fdaa0c00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fda9c159110 0x7fda9c072420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:59692/0 (socket says 172.21.15.6:59692) 2024-04-24T23:23:56.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:56.940+0000 7fdaa0c00640 1 -- 172.21.15.6:0/576015265 learned_addr learned my addr 172.21.15.6:0/576015265 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:23:56.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:56.940+0000 7fda9be00640 1 --2- 172.21.15.6:0/576015265 >> v2:172.21.15.190:3300/0 conn(0x7fda9c0c7dc0 0x7fda9c1623a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:23:56.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:56.940+0000 7fdaa1600640 1 --2- 172.21.15.6:0/576015265 >> v2:172.21.15.6:3300/0 conn(0x7fda9c0c81c0 0x7fda9c1628e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:23:56.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:56.941+0000 7fda9be00640 1 -- 172.21.15.6:0/576015265 >> v2:172.21.15.6:3300/0 conn(0x7fda9c0c81c0 msgr2=0x7fda9c1628e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:23:56.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:56.941+0000 7fda9be00640 1 --2- 172.21.15.6:0/576015265 >> v2:172.21.15.6:3300/0 conn(0x7fda9c0c81c0 0x7fda9c1628e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:56.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:56.941+0000 7fda9be00640 1 -- 172.21.15.6:0/576015265 >> v1:172.21.15.6:6789/0 conn(0x7fda9c159110 legacy=0x7fda9c072420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:23:56.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:56.941+0000 7fda9be00640 1 -- 172.21.15.6:0/576015265 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda9c06e440 con 0x7fda9c0c7dc0 2024-04-24T23:23:56.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:56.942+0000 7fda9be00640 1 --2- 172.21.15.6:0/576015265 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fda9c0c7dc0 0x7fda9c1623a0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fda9c0cd3b0 tx=0x7fda8802a210 comp rx=0 tx=0).ready entity=mon.1 client_cookie=19f54f7d08964d69 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:23:56.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:56.943+0000 7fda9b400640 1 -- 172.21.15.6:0/576015265 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda88037070 con 0x7fda9c0c7dc0 2024-04-24T23:23:56.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:56.943+0000 7fda9b400640 1 -- 172.21.15.6:0/576015265 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda88032440 con 0x7fda9c0c7dc0 2024-04-24T23:23:56.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:56.943+0000 7fda9b400640 1 -- 172.21.15.6:0/576015265 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda88032810 con 0x7fda9c0c7dc0 2024-04-24T23:23:56.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:56.944+0000 7fdaa3600640 1 -- 172.21.15.6:0/576015265 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fda9c0c7dc0 msgr2=0x7fda9c1623a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:23:56.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:56.944+0000 7fdaa3600640 1 --2- 172.21.15.6:0/576015265 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fda9c0c7dc0 0x7fda9c1623a0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fda9c0cd3b0 tx=0x7fda8802a210 comp rx=0 tx=0).stop 2024-04-24T23:23:56.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:56.945+0000 7fdaa3600640 1 -- 172.21.15.6:0/576015265 shutdown_connections 2024-04-24T23:23:56.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:56.945+0000 7fdaa3600640 1 --2- 172.21.15.6:0/576015265 >> v2:172.21.15.6:3300/0 conn(0x7fda9c0c81c0 0x7fda9c1628e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:56.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:56.945+0000 7fdaa3600640 1 --2- 172.21.15.6:0/576015265 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fda9c0c7dc0 0x7fda9c1623a0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:56.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:56.945+0000 7fdaa3600640 1 -- 172.21.15.6:0/576015265 >> 172.21.15.6:0/576015265 conn(0x7fda9c06b0e0 msgr2=0x7fda9c06b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:23:56.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:56.968+0000 7fdaa3600640 1 -- 172.21.15.6:0/576015265 shutdown_connections 2024-04-24T23:23:56.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:56.969+0000 7fdaa3600640 1 -- 172.21.15.6:0/576015265 wait complete. 2024-04-24T23:23:56.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:56.971+0000 7fdaa3600640 1 Processor -- start 2024-04-24T23:23:56.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:56.972+0000 7fdaa3600640 1 -- start start 2024-04-24T23:23:56.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:56.974+0000 7fdaa3600640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fda9c0c7dc0 0x7fda9c0cb680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:23:56.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:56.975+0000 7fdaa3600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fda9c0c81c0 0x7fda9c1524b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:23:56.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:56.975+0000 7fdaa3600640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fda9c0c5bb0 0x7fda9c1529f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:23:56.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:56.975+0000 7fdaa3600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fda9c06ece0 con 0x7fda9c0c81c0 2024-04-24T23:23:56.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:56.975+0000 7fdaa3600640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fda9c06ea00 con 0x7fda9c0c7dc0 2024-04-24T23:23:56.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:56.975+0000 7fdaa3600640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fda9c06f200 con 0x7fda9c0c5bb0 2024-04-24T23:23:56.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:56.976+0000 7fdaa1600640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fda9c0c5bb0 0x7fda9c1529f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:23:56.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:56.976+0000 7fdaa1600640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fda9c0c5bb0 0x7fda9c1529f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.6:60450/0 (socket says 172.21.15.6:60450) 2024-04-24T23:23:56.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:56.976+0000 7fdaa1600640 1 -- 172.21.15.6:0/4257487793 learned_addr learned my addr 172.21.15.6:0/4257487793 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:23:56.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:56.979+0000 7fdaa1600640 1 -- 172.21.15.6:0/4257487793 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fda9c0c7dc0 msgr2=0x7fda9c0cb680 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-24T23:23:56.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:56.979+0000 7fdaa1600640 1 --2- 172.21.15.6:0/4257487793 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fda9c0c7dc0 0x7fda9c0cb680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:56.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:56.979+0000 7fdaa1600640 1 -- 172.21.15.6:0/4257487793 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fda9c0c81c0 msgr2=0x7fda9c1524b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:23:56.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:56.979+0000 7fdaa1600640 1 --2- 172.21.15.6:0/4257487793 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fda9c0c81c0 0x7fda9c1524b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:56.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:56.979+0000 7fdaa1600640 1 -- 172.21.15.6:0/4257487793 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda9c0ca5e0 con 0x7fda9c0c5bb0 2024-04-24T23:23:56.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:56.982+0000 7fdaa1600640 1 --2- 172.21.15.6:0/4257487793 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fda9c0c5bb0 0x7fda9c1529f0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fda94007f00 tx=0x7fda940304c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:23:56.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:56.984+0000 7fda98c00640 1 -- 172.21.15.6:0/4257487793 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda94036430 con 0x7fda9c0c5bb0 2024-04-24T23:23:56.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:56.984+0000 7fda98c00640 1 -- 172.21.15.6:0/4257487793 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda9403b070 con 0x7fda9c0c5bb0 2024-04-24T23:23:56.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:56.984+0000 7fda98c00640 1 -- 172.21.15.6:0/4257487793 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda94036830 con 0x7fda9c0c5bb0 2024-04-24T23:23:56.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:56.984+0000 7fdaa3600640 1 -- 172.21.15.6:0/4257487793 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda9c0ca8c0 con 0x7fda9c0c5bb0 2024-04-24T23:23:56.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:56.985+0000 7fdaa3600640 1 -- 172.21.15.6:0/4257487793 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fda9c0cae80 con 0x7fda9c0c5bb0 2024-04-24T23:23:56.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:56.995+0000 7fdaa3600640 1 -- 172.21.15.6:0/4257487793 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda78005290 con 0x7fda9c0c5bb0 2024-04-24T23:23:57.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:56.999+0000 7fda98c00640 1 -- 172.21.15.6:0/4257487793 <== mon.2 v2:172.21.15.190:3301/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fda940369d0 con 0x7fda9c0c5bb0 2024-04-24T23:23:57.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:57.002+0000 7fda98c00640 1 --2- 172.21.15.6:0/4257487793 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fda7c06bf30 0x7fda7c06e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:23:57.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:57.002+0000 7fda98c00640 1 -- 172.21.15.6:0/4257487793 <== mon.2 v2:172.21.15.190:3301/0 5 ==== osd_map(28..28 src has 1..28) ==== 6762+0+0 (secure 0 0 0) 0x7fda940ad870 con 0x7fda9c0c5bb0 2024-04-24T23:23:57.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:57.007+0000 7fda98c00640 1 -- 172.21.15.6:0/4257487793 <== mon.2 v2:172.21.15.190:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fda9407bde0 con 0x7fda9c0c5bb0 2024-04-24T23:23:57.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:57.013+0000 7fdaa0c00640 1 --2- 172.21.15.6:0/4257487793 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fda7c06bf30 0x7fda7c06e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:23:57.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:57.014+0000 7fdaa0c00640 1 --2- 172.21.15.6:0/4257487793 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fda7c06bf30 0x7fda7c06e3f0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fda90007fd0 tx=0x7fda90007aa0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:23:57.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:57.639+0000 7fdaa3600640 1 -- 172.21.15.6:0/4257487793 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fda78005580 con 0x7fda9c0c5bb0 2024-04-24T23:23:57.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:57.641+0000 7fda98c00640 1 -- 172.21.15.6:0/4257487793 <== mon.2 v2:172.21.15.190:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) ==== 94+0+6024 (secure 0 0 0) 0x7fda9407b6e0 con 0x7fda9c0c5bb0 2024-04-24T23:23:57.644 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:23:57.645 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":41,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5081,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/1190759626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1190759626},{"type":"v1","addr":"172.21.15.6:6834","nonce":1190759626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":5503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/3928731059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":3928731059},{"type":"v1","addr":"172.21.15.6:6837","nonce":3928731059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":35},{"gid":5599,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6839/2643233757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":2643233757},{"type":"v1","addr":"172.21.15.190:6839","nonce":2643233757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40}],"filesystems":[{"mdsmap":{"epoch":39,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:23:38.162102+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":4306,"mds_1":4282},"failed":[],"damaged":[],"stopped":[2,3,4],"info":{"gid_4282":{"gid":4282,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":12,"addr":"172.21.15.190:6837/580315269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":580315269},{"type":"v1","addr":"172.21.15.190:6837","nonce":580315269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/489092393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":489092393},{"type":"v1","addr":"172.21.15.6:6835","nonce":489092393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4306,"qdb_cluster":[4282,4306]},"id":1}]} 2024-04-24T23:23:57.645 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 41 2024-04-24T23:23:57.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:57.649+0000 7fdaa3600640 1 -- 172.21.15.6:0/4257487793 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fda7c06bf30 msgr2=0x7fda7c06e3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:23:57.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:57.649+0000 7fdaa3600640 1 --2- 172.21.15.6:0/4257487793 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fda7c06bf30 0x7fda7c06e3f0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fda90007fd0 tx=0x7fda90007aa0 comp rx=0 tx=0).stop 2024-04-24T23:23:57.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:57.649+0000 7fdaa3600640 1 -- 172.21.15.6:0/4257487793 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fda9c0c5bb0 msgr2=0x7fda9c1529f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:23:57.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:57.649+0000 7fdaa3600640 1 --2- 172.21.15.6:0/4257487793 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fda9c0c5bb0 0x7fda9c1529f0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fda94007f00 tx=0x7fda940304c0 comp rx=0 tx=0).stop 2024-04-24T23:23:57.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:57.655+0000 7fdaa3600640 1 -- 172.21.15.6:0/4257487793 shutdown_connections 2024-04-24T23:23:57.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:57.655+0000 7fdaa3600640 1 --2- 172.21.15.6:0/4257487793 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fda7c06bf30 0x7fda7c06e3f0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:57.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:57.655+0000 7fdaa3600640 1 --2- 172.21.15.6:0/4257487793 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fda9c0c5bb0 0x7fda9c1529f0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:57.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:57.655+0000 7fdaa3600640 1 --2- 172.21.15.6:0/4257487793 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fda9c0c81c0 0x7fda9c1524b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:57.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:57.655+0000 7fdaa3600640 1 --2- 172.21.15.6:0/4257487793 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fda9c0c7dc0 0x7fda9c0cb680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:57.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:57.655+0000 7fdaa3600640 1 -- 172.21.15.6:0/4257487793 >> 172.21.15.6:0/4257487793 conn(0x7fda9c06b0e0 msgr2=0x7fda9c154db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:23:57.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:57.655+0000 7fdaa3600640 1 -- 172.21.15.6:0/4257487793 shutdown_connections 2024-04-24T23:23:57.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:57.655+0000 7fdaa3600640 1 -- 172.21.15.6:0/4257487793 wait complete. 2024-04-24T23:23:57.705 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2024-04-24T23:23:59.469 INFO:tasks.ceph.mds.b.smithi190.stdout:starting mds.b at 2024-04-24T23:23:59.706 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:23:59.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:59.927+0000 7f088c600640 1 Processor -- start 2024-04-24T23:23:59.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:59.927+0000 7f088c600640 1 -- start start 2024-04-24T23:23:59.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:59.928+0000 7f088c600640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f088406c1f0 0x7f08840caed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:23:59.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:59.928+0000 7f088c600640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f088406c5d0 0x7f08840cb410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:23:59.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:59.928+0000 7f088c600640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f088406ca80 con 0x7f08841519b0 2024-04-24T23:23:59.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:59.928+0000 7f088c600640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f088415ba10 con 0x7f088406c1f0 2024-04-24T23:23:59.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:59.928+0000 7f088c600640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f088415ae00 con 0x7f088406c5d0 2024-04-24T23:23:59.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:59.929+0000 7f0889200640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f088406c1f0 0x7f08840caed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:23:59.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:59.929+0000 7f088a600640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f088406c5d0 0x7f08840cb410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:23:59.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:59.929+0000 7f0889c00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f08841519b0 0x7f08840c8d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:59708/0 (socket says 172.21.15.6:59708) 2024-04-24T23:23:59.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:59.929+0000 7f0889c00640 1 -- 172.21.15.6:0/2086655958 learned_addr learned my addr 172.21.15.6:0/2086655958 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:23:59.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:59.929+0000 7f088a600640 1 -- 172.21.15.6:0/2086655958 >> v2:172.21.15.190:3300/0 conn(0x7f088406c1f0 msgr2=0x7f08840caed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:23:59.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:59.929+0000 7f088a600640 1 --2- 172.21.15.6:0/2086655958 >> v2:172.21.15.190:3300/0 conn(0x7f088406c1f0 0x7f08840caed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:59.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:59.929+0000 7f088a600640 1 -- 172.21.15.6:0/2086655958 >> v1:172.21.15.6:6789/0 conn(0x7f08841519b0 legacy=0x7f08840c8d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:23:59.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:59.929+0000 7f088a600640 1 -- 172.21.15.6:0/2086655958 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f088415b0e0 con 0x7f088406c5d0 2024-04-24T23:23:59.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:59.931+0000 7f088a600640 1 --2- 172.21.15.6:0/2086655958 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f088406c5d0 0x7f08840cb410 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f087c009d90 tx=0x7f087c02a660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fee7399a81548dc3 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:23:59.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:59.931+0000 7f0883e00640 1 -- 172.21.15.6:0/2086655958 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f087c007740 con 0x7f088406c5d0 2024-04-24T23:23:59.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:59.931+0000 7f0883e00640 1 -- 172.21.15.6:0/2086655958 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f087c0078e0 con 0x7f088406c5d0 2024-04-24T23:23:59.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:59.931+0000 7f0883e00640 1 -- 172.21.15.6:0/2086655958 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f087c007cb0 con 0x7f088406c5d0 2024-04-24T23:23:59.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:59.932+0000 7f088c600640 1 -- 172.21.15.6:0/2086655958 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f088406c5d0 msgr2=0x7f08840cb410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:23:59.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:59.932+0000 7f088c600640 1 --2- 172.21.15.6:0/2086655958 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f088406c5d0 0x7f08840cb410 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f087c009d90 tx=0x7f087c02a660 comp rx=0 tx=0).stop 2024-04-24T23:23:59.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:59.932+0000 7f088c600640 1 -- 172.21.15.6:0/2086655958 shutdown_connections 2024-04-24T23:23:59.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:59.932+0000 7f088c600640 1 --2- 172.21.15.6:0/2086655958 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f088406c5d0 0x7f08840cb410 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:59.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:59.932+0000 7f088c600640 1 --2- 172.21.15.6:0/2086655958 >> v2:172.21.15.190:3300/0 conn(0x7f088406c1f0 0x7f08840caed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:59.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:59.932+0000 7f088c600640 1 -- 172.21.15.6:0/2086655958 >> 172.21.15.6:0/2086655958 conn(0x7f0884070280 msgr2=0x7f088406f320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:23:59.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:59.932+0000 7f088c600640 1 -- 172.21.15.6:0/2086655958 shutdown_connections 2024-04-24T23:23:59.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:59.932+0000 7f088c600640 1 -- 172.21.15.6:0/2086655958 wait complete. 2024-04-24T23:23:59.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:59.934+0000 7f088c600640 1 Processor -- start 2024-04-24T23:23:59.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:59.934+0000 7f088c600640 1 -- start start 2024-04-24T23:23:59.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:59.935+0000 7f088c600640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f088406c1f0 0x7f08841f47f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:23:59.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:59.935+0000 7f088c600640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f088406c5d0 0x7f08840c8390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:23:59.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:59.935+0000 7f088c600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f08841519b0 0x7f08840ceb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:23:59.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:59.935+0000 7f088c600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f0884151d90 con 0x7f08841519b0 2024-04-24T23:23:59.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:59.935+0000 7f088c600640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f088415b6a0 con 0x7f088406c1f0 2024-04-24T23:23:59.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:59.935+0000 7f088c600640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f088415e2e0 con 0x7f088406c5d0 2024-04-24T23:23:59.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:59.936+0000 7f0889c00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f088406c1f0 0x7f08841f47f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:23:59.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:59.936+0000 7f0889200640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f088406c5d0 0x7f08840c8390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:23:59.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:59.936+0000 7f088a600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f08841519b0 0x7f08840ceb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:23:59.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:59.936+0000 7f0889200640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f088406c5d0 0x7f08840c8390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.6:60456/0 (socket says 172.21.15.6:60456) 2024-04-24T23:23:59.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:59.936+0000 7f0889c00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f088406c1f0 0x7f08841f47f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:57166/0 (socket says 172.21.15.6:57166) 2024-04-24T23:23:59.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:59.936+0000 7f0889200640 1 -- 172.21.15.6:0/1534510388 learned_addr learned my addr 172.21.15.6:0/1534510388 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:23:59.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:59.936+0000 7f0889c00640 1 -- 172.21.15.6:0/1534510388 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f088406c5d0 msgr2=0x7f08840c8390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:23:59.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:59.936+0000 7f0889c00640 1 --2- 172.21.15.6:0/1534510388 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f088406c5d0 0x7f08840c8390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:59.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:59.936+0000 7f0889c00640 1 -- 172.21.15.6:0/1534510388 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f08841519b0 msgr2=0x7f08840ceb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:23:59.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:59.936+0000 7f0889c00640 1 --2- 172.21.15.6:0/1534510388 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f08841519b0 0x7f08840ceb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:23:59.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:59.936+0000 7f0889c00640 1 -- 172.21.15.6:0/1534510388 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08840c58e0 con 0x7f088406c1f0 2024-04-24T23:23:59.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:59.936+0000 7f0889200640 1 --2- 172.21.15.6:0/1534510388 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f088406c5d0 0x7f08840c8390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:23:59.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:59.937+0000 7f0889c00640 1 --2- 172.21.15.6:0/1534510388 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f088406c1f0 0x7f08841f47f0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f0878000f60 tx=0x7f087800c760 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:23:59.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:59.937+0000 7f0881600640 1 -- 172.21.15.6:0/1534510388 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0878017070 con 0x7f088406c1f0 2024-04-24T23:23:59.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:59.937+0000 7f0881600640 1 -- 172.21.15.6:0/1534510388 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0878005ce0 con 0x7f088406c1f0 2024-04-24T23:23:59.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:59.937+0000 7f088c600640 1 -- 172.21.15.6:0/1534510388 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08840c5bc0 con 0x7f088406c1f0 2024-04-24T23:23:59.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:59.937+0000 7f0881600640 1 -- 172.21.15.6:0/1534510388 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0878007310 con 0x7f088406c1f0 2024-04-24T23:23:59.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:59.937+0000 7f088c600640 1 -- 172.21.15.6:0/1534510388 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08840c6180 con 0x7f088406c1f0 2024-04-24T23:23:59.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:59.938+0000 7f088c600640 1 -- 172.21.15.6:0/1534510388 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f08840767b0 con 0x7f088406c1f0 2024-04-24T23:23:59.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:59.939+0000 7f0881600640 1 -- 172.21.15.6:0/1534510388 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f087801a070 con 0x7f088406c1f0 2024-04-24T23:23:59.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:59.940+0000 7f0881600640 1 --2- 172.21.15.6:0/1534510388 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f086406be30 0x7f086406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:23:59.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:59.941+0000 7f0881600640 1 -- 172.21.15.6:0/1534510388 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6762+0+0 (secure 0 0 0) 0x7f087808a4b0 con 0x7f088406c1f0 2024-04-24T23:23:59.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:59.941+0000 7f0889200640 1 --2- 172.21.15.6:0/1534510388 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f086406be30 0x7f086406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:23:59.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:59.941+0000 7f0889200640 1 --2- 172.21.15.6:0/1534510388 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f086406be30 0x7f086406e2f0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f088415a850 tx=0x7f0870009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:23:59.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:23:59.945+0000 7f0881600640 1 -- 172.21.15.6:0/1534510388 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f08780583e0 con 0x7f088406c1f0 2024-04-24T23:24:00.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:00.236+0000 7f088c600640 1 -- 172.21.15.6:0/1534510388 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f08840cba10 con 0x7f088406c1f0 2024-04-24T23:24:00.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:00.239+0000 7f0881600640 1 -- 172.21.15.6:0/1534510388 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) ==== 94+0+6852 (secure 0 0 0) 0x7f087805c6e0 con 0x7f088406c1f0 2024-04-24T23:24:00.244 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:24:00.244 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":42,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5081,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/1190759626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1190759626},{"type":"v1","addr":"172.21.15.6:6834","nonce":1190759626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":5249,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6838/2416336040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2416336040},{"type":"v1","addr":"172.21.15.190:6838","nonce":2416336040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":42},{"gid":5503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/3928731059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":3928731059},{"type":"v1","addr":"172.21.15.6:6837","nonce":3928731059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":35},{"gid":5599,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6839/2643233757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":2643233757},{"type":"v1","addr":"172.21.15.190:6839","nonce":2643233757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40}],"filesystems":[{"mdsmap":{"epoch":39,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:23:38.162102+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":4306,"mds_1":4282},"failed":[],"damaged":[],"stopped":[2,3,4],"info":{"gid_4282":{"gid":4282,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":12,"addr":"172.21.15.190:6837/580315269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":580315269},{"type":"v1","addr":"172.21.15.190:6837","nonce":580315269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/489092393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":489092393},{"type":"v1","addr":"172.21.15.6:6835","nonce":489092393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4306,"qdb_cluster":[4306,4282]},"id":1}]} 2024-04-24T23:24:00.244 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 42 2024-04-24T23:24:00.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:00.245+0000 7f088c600640 1 -- 172.21.15.6:0/1534510388 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f086406be30 msgr2=0x7f086406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:24:00.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:00.246+0000 7f088c600640 1 --2- 172.21.15.6:0/1534510388 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f086406be30 0x7f086406e2f0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f088415a850 tx=0x7f0870009190 comp rx=0 tx=0).stop 2024-04-24T23:24:00.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:00.246+0000 7f088c600640 1 -- 172.21.15.6:0/1534510388 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f088406c1f0 msgr2=0x7f08841f47f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:24:00.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:00.246+0000 7f088c600640 1 --2- 172.21.15.6:0/1534510388 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f088406c1f0 0x7f08841f47f0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f0878000f60 tx=0x7f087800c760 comp rx=0 tx=0).stop 2024-04-24T23:24:00.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:00.248+0000 7f088c600640 1 -- 172.21.15.6:0/1534510388 shutdown_connections 2024-04-24T23:24:00.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:00.248+0000 7f088c600640 1 --2- 172.21.15.6:0/1534510388 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f086406be30 0x7f086406e2f0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:24:00.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:00.248+0000 7f088c600640 1 --2- 172.21.15.6:0/1534510388 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f088406c5d0 0x7f08840c8390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:24:00.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:00.248+0000 7f088c600640 1 --2- 172.21.15.6:0/1534510388 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f08841519b0 0x7f08840ceb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:24:00.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:00.248+0000 7f088c600640 1 --2- 172.21.15.6:0/1534510388 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f088406c1f0 0x7f08841f47f0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:24:00.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:00.248+0000 7f088c600640 1 -- 172.21.15.6:0/1534510388 >> 172.21.15.6:0/1534510388 conn(0x7f0884070280 msgr2=0x7f08840c3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:24:00.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:00.248+0000 7f088c600640 1 -- 172.21.15.6:0/1534510388 shutdown_connections 2024-04-24T23:24:00.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:00.248+0000 7f088c600640 1 -- 172.21.15.6:0/1534510388 wait complete. 2024-04-24T23:24:00.278 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b reported in up:standby state 2024-04-24T23:24:00.279 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 14 secs before thrashing 2024-04-24T23:24:14.280 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:24:14.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:14.951+0000 7fe09fe00640 1 Processor -- start 2024-04-24T23:24:14.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:14.951+0000 7fe09fe00640 1 -- start start 2024-04-24T23:24:14.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:14.956+0000 7fe09fe00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fe0a0150cd0 0x7fe0a00cc930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:24:14.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:14.958+0000 7fe09fe00640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fe0a00cce70 0x7fe0a00c8ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:24:14.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:14.958+0000 7fe09fe00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe0a006e580 con 0x7fe0a006dd40 2024-04-24T23:24:14.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:14.958+0000 7fe09fe00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7fe0a00c7720 con 0x7fe0a0150cd0 2024-04-24T23:24:14.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:14.958+0000 7fe09fe00640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7fe0a006e2a0 con 0x7fe0a00cce70 2024-04-24T23:24:14.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:14.958+0000 7fe09f400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fe0a006dd40 0x7fe0a0159e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:45090/0 (socket says 172.21.15.6:45090) 2024-04-24T23:24:14.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:14.958+0000 7fe09f400640 1 -- 172.21.15.6:0/1164483157 learned_addr learned my addr 172.21.15.6:0/1164483157 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:24:14.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:14.959+0000 7fe09e000640 1 --2- 172.21.15.6:0/1164483157 >> v2:172.21.15.190:3300/0 conn(0x7fe0a00cce70 0x7fe0a00c8ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:24:14.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:14.959+0000 7fe09d600640 1 -- 172.21.15.6:0/1164483157 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2591619514 0 0) 0x7fe0a006e580 con 0x7fe0a006dd40 2024-04-24T23:24:14.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:14.959+0000 7fe09d600640 1 -- 172.21.15.6:0/1164483157 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe084003520 con 0x7fe0a006dd40 2024-04-24T23:24:14.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:14.960+0000 7fe09d600640 1 -- 172.21.15.6:0/1164483157 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3228259418 0 0) 0x7fe084003520 con 0x7fe0a006dd40 2024-04-24T23:24:14.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:14.960+0000 7fe09d600640 1 -- 172.21.15.6:0/1164483157 >> v2:172.21.15.190:3300/0 conn(0x7fe0a00cce70 msgr2=0x7fe0a00c8ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:24:14.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:14.960+0000 7fe09d600640 1 --2- 172.21.15.6:0/1164483157 >> v2:172.21.15.190:3300/0 conn(0x7fe0a00cce70 0x7fe0a00c8ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:24:14.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:14.960+0000 7fe09d600640 1 -- 172.21.15.6:0/1164483157 >> v2:172.21.15.6:3300/0 conn(0x7fe0a0150cd0 msgr2=0x7fe0a00cc930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:24:14.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:14.957+0000 7fe09ea00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fe0a0150cd0 0x7fe0a00cc930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:24:14.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:14.961+0000 7fe09d600640 1 --2- 172.21.15.6:0/1164483157 >> v2:172.21.15.6:3300/0 conn(0x7fe0a0150cd0 0x7fe0a00cc930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:24:14.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:14.961+0000 7fe09d600640 1 -- 172.21.15.6:0/1164483157 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0a00c79c0 con 0x7fe0a006dd40 2024-04-24T23:24:14.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:14.961+0000 7fe09d600640 1 -- 172.21.15.6:0/1164483157 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3463268701 0 0) 0x7fe0980019f0 con 0x7fe0a006dd40 2024-04-24T23:24:14.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:14.961+0000 7fe09ea00640 1 --2- 172.21.15.6:0/1164483157 >> v2:172.21.15.6:3300/0 conn(0x7fe0a0150cd0 0x7fe0a00cc930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:24:14.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:14.961+0000 7fe09d600640 1 -- 172.21.15.6:0/1164483157 >> v1:172.21.15.6:6789/0 conn(0x7fe0a006dd40 legacy=0x7fe0a0159e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:24:14.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:14.961+0000 7fe09d600640 1 --2- 172.21.15.6:0/1164483157 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe084003a00 0x7fe084003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:24:14.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:14.962+0000 7fe09d600640 1 --2- 172.21.15.6:0/1164483157 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe084004800 0x7fe084004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:24:14.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:14.962+0000 7fe09d600640 1 --2- 172.21.15.6:0/1164483157 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe084005210 0x7fe0840079a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:24:14.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:14.962+0000 7fe09d600640 1 -- 172.21.15.6:0/1164483157 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fe0a00c7720 con 0x7fe084004800 2024-04-24T23:24:14.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:14.962+0000 7fe09d600640 1 -- 172.21.15.6:0/1164483157 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fe0a006e2a0 con 0x7fe084005210 2024-04-24T23:24:14.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:14.962+0000 7fe09d600640 1 -- 172.21.15.6:0/1164483157 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fe0840081c0 con 0x7fe084003a00 2024-04-24T23:24:14.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:14.962+0000 7fe09e000640 1 --2- 172.21.15.6:0/1164483157 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe084004800 0x7fe084004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:24:14.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:14.963+0000 7fe09e000640 1 -- 172.21.15.6:0/1164483157 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe084003a00 msgr2=0x7fe084003e20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-24T23:24:14.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:14.963+0000 7fe09e000640 1 --2- 172.21.15.6:0/1164483157 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe084003a00 0x7fe084003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:24:14.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:14.963+0000 7fe09e000640 1 -- 172.21.15.6:0/1164483157 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe084005210 msgr2=0x7fe0840079a0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-24T23:24:14.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:14.963+0000 7fe09e000640 1 --2- 172.21.15.6:0/1164483157 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe084005210 0x7fe0840079a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:24:14.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:14.963+0000 7fe09e000640 1 -- 172.21.15.6:0/1164483157 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe084003520 con 0x7fe084004800 2024-04-24T23:24:14.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:14.966+0000 7fe09e000640 1 --2- 172.21.15.6:0/1164483157 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe084004800 0x7fe084004c80 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fe088009a90 tx=0x7fe084005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=285e3e4de5209941 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:24:14.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:14.966+0000 7fe09d600640 1 -- 172.21.15.6:0/1164483157 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe088036070 con 0x7fe084004800 2024-04-24T23:24:14.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:14.967+0000 7fe09d600640 1 -- 172.21.15.6:0/1164483157 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe088031400 con 0x7fe084004800 2024-04-24T23:24:14.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:14.967+0000 7fe09d600640 1 -- 172.21.15.6:0/1164483157 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe088009690 con 0x7fe084004800 2024-04-24T23:24:14.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:14.967+0000 7fe09f400640 1 -- 172.21.15.6:0/1164483157 reap_dead start 2024-04-24T23:24:14.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:14.968+0000 7fe09fe00640 1 -- 172.21.15.6:0/1164483157 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe084004800 msgr2=0x7fe084004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:24:14.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:14.968+0000 7fe09fe00640 1 --2- 172.21.15.6:0/1164483157 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe084004800 0x7fe084004c80 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fe088009a90 tx=0x7fe084005830 comp rx=0 tx=0).stop 2024-04-24T23:24:14.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:14.980+0000 7fe09fe00640 1 -- 172.21.15.6:0/1164483157 shutdown_connections 2024-04-24T23:24:14.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:14.980+0000 7fe09fe00640 1 --2- 172.21.15.6:0/1164483157 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe084004800 0x7fe084004c80 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:24:14.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:14.980+0000 7fe09fe00640 1 -- 172.21.15.6:0/1164483157 >> 172.21.15.6:0/1164483157 conn(0x7fe0a006ed80 msgr2=0x7fe0a006f190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:24:14.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:14.980+0000 7fe09fe00640 1 -- 172.21.15.6:0/1164483157 shutdown_connections 2024-04-24T23:24:14.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:14.980+0000 7fe09fe00640 1 -- 172.21.15.6:0/1164483157 wait complete. 2024-04-24T23:24:14.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:14.983+0000 7fe09fe00640 1 Processor -- start 2024-04-24T23:24:14.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:14.990+0000 7fe09fe00640 1 -- start start 2024-04-24T23:24:14.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:14.997+0000 7fe09fe00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe084004800 0x7fe0a0153930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:24:15.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:14.998+0000 7fe09fe00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe0a01f0e90 0x7fe0a01512a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:24:15.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:14.999+0000 7fe09fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe0a01f09e0 0x7fe0a01517e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:24:15.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:15.000+0000 7fe09f400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe0a01f09e0 0x7fe0a01517e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:24:15.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:15.001+0000 7fe09f400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe0a01f09e0 0x7fe0a01517e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:41872/0 (socket says 172.21.15.6:41872) 2024-04-24T23:24:15.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:15.001+0000 7fe09f400640 1 -- 172.21.15.6:0/981829428 learned_addr learned my addr 172.21.15.6:0/981829428 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:24:15.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:15.003+0000 7fe09ea00640 1 --2- 172.21.15.6:0/981829428 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe084004800 0x7fe0a0153930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:24:15.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:15.003+0000 7fe09fe00640 1 -- 172.21.15.6:0/981829428 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fe0a00c7240 con 0x7fe0a01f09e0 2024-04-24T23:24:15.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:15.003+0000 7fe09e000640 1 --2- 172.21.15.6:0/981829428 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe0a01f0e90 0x7fe0a01512a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:24:15.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:15.003+0000 7fe09fe00640 1 -- 172.21.15.6:0/981829428 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fe0a00c7f80 con 0x7fe084004800 2024-04-24T23:24:15.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:15.003+0000 7fe09fe00640 1 -- 172.21.15.6:0/981829428 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fe0a00cc930 con 0x7fe0a01f0e90 2024-04-24T23:24:15.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:15.004+0000 7fe09ea00640 1 -- 172.21.15.6:0/981829428 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe0a01f0e90 msgr2=0x7fe0a01512a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:24:15.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:15.004+0000 7fe09ea00640 1 --2- 172.21.15.6:0/981829428 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe0a01f0e90 0x7fe0a01512a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:24:15.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:15.004+0000 7fe09ea00640 1 -- 172.21.15.6:0/981829428 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe0a01f09e0 msgr2=0x7fe0a01517e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:24:15.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:15.004+0000 7fe09ea00640 1 --2- 172.21.15.6:0/981829428 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe0a01f09e0 0x7fe0a01517e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:24:15.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:15.004+0000 7fe09ea00640 1 -- 172.21.15.6:0/981829428 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0a00ca400 con 0x7fe084004800 2024-04-24T23:24:15.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:15.006+0000 7fe09ea00640 1 --2- 172.21.15.6:0/981829428 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe084004800 0x7fe0a0153930 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fe09000c820 tx=0x7fe09000cce0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:24:15.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:15.007+0000 7fe096a00640 1 -- 172.21.15.6:0/981829428 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe090015070 con 0x7fe084004800 2024-04-24T23:24:15.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:15.007+0000 7fe09fe00640 1 -- 172.21.15.6:0/981829428 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0a0154970 con 0x7fe084004800 2024-04-24T23:24:15.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:15.007+0000 7fe09fe00640 1 -- 172.21.15.6:0/981829428 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0a0200660 con 0x7fe084004800 2024-04-24T23:24:15.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:15.009+0000 7fe096a00640 1 -- 172.21.15.6:0/981829428 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe0900079d0 con 0x7fe084004800 2024-04-24T23:24:15.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:15.009+0000 7fe096a00640 1 -- 172.21.15.6:0/981829428 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe090007dc0 con 0x7fe084004800 2024-04-24T23:24:15.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:15.011+0000 7fe09fe00640 1 -- 172.21.15.6:0/981829428 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe078005290 con 0x7fe084004800 2024-04-24T23:24:15.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:15.014+0000 7fe096a00640 1 -- 172.21.15.6:0/981829428 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fe090018070 con 0x7fe084004800 2024-04-24T23:24:15.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:15.017+0000 7fe096a00640 1 --2- 172.21.15.6:0/981829428 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fe07c06bf30 0x7fe07c06e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:24:15.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:15.018+0000 7fe096a00640 1 -- 172.21.15.6:0/981829428 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6762+0+0 (secure 0 0 0) 0x7fe09008af20 con 0x7fe084004800 2024-04-24T23:24:15.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:15.018+0000 7fe09e000640 1 --2- 172.21.15.6:0/981829428 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fe07c06bf30 0x7fe07c06e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:24:15.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:15.022+0000 7fe096a00640 1 -- 172.21.15.6:0/981829428 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fe090058ed0 con 0x7fe084004800 2024-04-24T23:24:15.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:15.022+0000 7fe09e000640 1 --2- 172.21.15.6:0/981829428 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fe07c06bf30 0x7fe07c06e3f0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fe088033000 tx=0x7fe088008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:24:15.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:15.630+0000 7fe09fe00640 1 -- 172.21.15.6:0/981829428 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe078005580 con 0x7fe084004800 2024-04-24T23:24:15.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:15.631+0000 7fe096a00640 1 -- 172.21.15.6:0/981829428 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) ==== 94+0+6852 (secure 0 0 0) 0x7fe09005d1d0 con 0x7fe084004800 2024-04-24T23:24:15.633 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:24:15.633 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":42,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5081,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/1190759626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1190759626},{"type":"v1","addr":"172.21.15.6:6834","nonce":1190759626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":5249,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6838/2416336040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2416336040},{"type":"v1","addr":"172.21.15.190:6838","nonce":2416336040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":42},{"gid":5503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/3928731059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":3928731059},{"type":"v1","addr":"172.21.15.6:6837","nonce":3928731059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":35},{"gid":5599,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6839/2643233757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":2643233757},{"type":"v1","addr":"172.21.15.190:6839","nonce":2643233757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40}],"filesystems":[{"mdsmap":{"epoch":39,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:23:38.162102+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":4306,"mds_1":4282},"failed":[],"damaged":[],"stopped":[2,3,4],"info":{"gid_4282":{"gid":4282,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":12,"addr":"172.21.15.190:6837/580315269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":580315269},{"type":"v1","addr":"172.21.15.190:6837","nonce":580315269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/489092393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":489092393},{"type":"v1","addr":"172.21.15.6:6835","nonce":489092393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4306,"qdb_cluster":[4306,4282]},"id":1}]} 2024-04-24T23:24:15.634 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 42 2024-04-24T23:24:15.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:15.635+0000 7fe09fe00640 1 -- 172.21.15.6:0/981829428 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fe07c06bf30 msgr2=0x7fe07c06e3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:24:15.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:15.636+0000 7fe09fe00640 1 --2- 172.21.15.6:0/981829428 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fe07c06bf30 0x7fe07c06e3f0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fe088033000 tx=0x7fe088008030 comp rx=0 tx=0).stop 2024-04-24T23:24:15.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:15.636+0000 7fe09fe00640 1 -- 172.21.15.6:0/981829428 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe084004800 msgr2=0x7fe0a0153930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:24:15.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:15.636+0000 7fe09fe00640 1 --2- 172.21.15.6:0/981829428 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe084004800 0x7fe0a0153930 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fe09000c820 tx=0x7fe09000cce0 comp rx=0 tx=0).stop 2024-04-24T23:24:15.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:15.636+0000 7fe09fe00640 1 -- 172.21.15.6:0/981829428 shutdown_connections 2024-04-24T23:24:15.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:15.636+0000 7fe09fe00640 1 --2- 172.21.15.6:0/981829428 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fe07c06bf30 0x7fe07c06e3f0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:24:15.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:15.636+0000 7fe09fe00640 1 --2- 172.21.15.6:0/981829428 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe0a01f0e90 0x7fe0a01512a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:24:15.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:15.636+0000 7fe09fe00640 1 --2- 172.21.15.6:0/981829428 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe0a01f09e0 0x7fe0a01517e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:24:15.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:15.636+0000 7fe09fe00640 1 --2- 172.21.15.6:0/981829428 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe084004800 0x7fe0a0153930 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:24:15.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:15.636+0000 7fe09fe00640 1 -- 172.21.15.6:0/981829428 >> 172.21.15.6:0/981829428 conn(0x7fe0a006ed80 msgr2=0x7fe0a00c8ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:24:15.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:15.640+0000 7fe09fe00640 1 -- 172.21.15.6:0/981829428 shutdown_connections 2024-04-24T23:24:15.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:15.640+0000 7fe09fe00640 1 -- 172.21.15.6:0/981829428 wait complete. 2024-04-24T23:24:15.690 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.f (rank=1) 2024-04-24T23:24:15.691 DEBUG:tasks.ceph.mds.f:waiting for process to exit 2024-04-24T23:24:15.691 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T23:24:21.714 INFO:tasks.ceph.mds.f:Stopped 2024-04-24T23:24:21.715 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:24:22.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:22.015+0000 7f8b72a00640 1 Processor -- start 2024-04-24T23:24:22.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:22.016+0000 7f8b72a00640 1 -- start start 2024-04-24T23:24:22.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:22.017+0000 7f8b72a00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8b6c073270 0x7f8b6c0c9110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:24:22.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:22.017+0000 7f8b72a00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8b6c0c8690 con 0x7f8b6c157470 2024-04-24T23:24:22.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:22.017+0000 7f8b72a00640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8b6c0c8970 con 0x7f8b6c150990 2024-04-24T23:24:22.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:22.017+0000 7f8b72a00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f8b6c0c8c50 con 0x7f8b6c073270 2024-04-24T23:24:22.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:22.018+0000 7f8b6be00640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f8b6c150990 0x7f8b6c15a980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.6:41058/0 (socket says 172.21.15.6:41058) 2024-04-24T23:24:22.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:22.018+0000 7f8b63e00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8b6c073270 0x7f8b6c0c9110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:24:22.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:22.018+0000 7f8b6be00640 1 -- 172.21.15.6:0/3808501412 learned_addr learned my addr 172.21.15.6:0/3808501412 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:24:22.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:22.018+0000 7f8b63e00640 1 -- 172.21.15.6:0/3808501412 >> v1:172.21.15.190:6789/0 conn(0x7f8b6c150990 legacy=0x7f8b6c15a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:24:22.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:22.018+0000 7f8b63e00640 1 -- 172.21.15.6:0/3808501412 >> v1:172.21.15.6:6789/0 conn(0x7f8b6c157470 legacy=0x7f8b6c161370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:24:22.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:22.018+0000 7f8b63e00640 1 -- 172.21.15.6:0/3808501412 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b6c15a180 con 0x7f8b6c073270 2024-04-24T23:24:22.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:22.019+0000 7f8b63e00640 1 --2- 172.21.15.6:0/3808501412 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8b6c073270 0x7f8b6c0c9110 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f8b5400c980 tx=0x7f8b5402c630 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e8baf941f39c93f0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:24:22.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:22.019+0000 7f8b6b400640 1 -- 172.21.15.6:0/3808501412 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b5400a530 con 0x7f8b6c073270 2024-04-24T23:24:22.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:22.020+0000 7f8b6b400640 1 -- 172.21.15.6:0/3808501412 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8b5402ce50 con 0x7f8b6c073270 2024-04-24T23:24:22.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:22.020+0000 7f8b6b400640 1 -- 172.21.15.6:0/3808501412 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b5400a930 con 0x7f8b6c073270 2024-04-24T23:24:22.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:22.020+0000 7f8b72a00640 1 -- 172.21.15.6:0/3808501412 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8b6c073270 msgr2=0x7f8b6c0c9110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:24:22.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:22.020+0000 7f8b72a00640 1 --2- 172.21.15.6:0/3808501412 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8b6c073270 0x7f8b6c0c9110 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f8b5400c980 tx=0x7f8b5402c630 comp rx=0 tx=0).stop 2024-04-24T23:24:22.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:22.020+0000 7f8b72a00640 1 -- 172.21.15.6:0/3808501412 shutdown_connections 2024-04-24T23:24:22.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:22.020+0000 7f8b72a00640 1 --2- 172.21.15.6:0/3808501412 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8b6c073270 0x7f8b6c0c9110 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:24:22.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:22.020+0000 7f8b72a00640 1 -- 172.21.15.6:0/3808501412 >> 172.21.15.6:0/3808501412 conn(0x7f8b6c06b0e0 msgr2=0x7f8b6c06b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:24:22.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:22.022+0000 7f8b72a00640 1 -- 172.21.15.6:0/3808501412 shutdown_connections 2024-04-24T23:24:22.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:22.023+0000 7f8b72a00640 1 -- 172.21.15.6:0/3808501412 wait complete. 2024-04-24T23:24:22.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:22.025+0000 7f8b72a00640 1 Processor -- start 2024-04-24T23:24:22.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:22.025+0000 7f8b72a00640 1 -- start start 2024-04-24T23:24:22.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:22.026+0000 7f8b72a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8b6c073270 0x7f8b6c201240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:24:22.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:22.027+0000 7f8b72a00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8b6c150990 0x7f8b6c1f42f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:24:22.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:22.027+0000 7f8b72a00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8b6c157470 0x7f8b6c1519f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:24:22.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:22.027+0000 7f8b72a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f8b6c150d70 con 0x7f8b6c073270 2024-04-24T23:24:22.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:22.027+0000 7f8b72a00640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f8b6c0c7a70 con 0x7f8b6c150990 2024-04-24T23:24:22.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:22.027+0000 7f8b72a00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f8b6c06bbd0 con 0x7f8b6c157470 2024-04-24T23:24:22.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:22.027+0000 7f8b6be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8b6c073270 0x7f8b6c201240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:24:22.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:22.027+0000 7f8b6be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8b6c073270 0x7f8b6c201240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:41874/0 (socket says 172.21.15.6:41874) 2024-04-24T23:24:22.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:22.027+0000 7f8b6be00640 1 -- 172.21.15.6:0/3064964089 learned_addr learned my addr 172.21.15.6:0/3064964089 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:24:22.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:22.027+0000 7f8b70a00640 1 --2- 172.21.15.6:0/3064964089 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8b6c157470 0x7f8b6c1519f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:24:22.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:22.028+0000 7f8b6be00640 1 -- 172.21.15.6:0/3064964089 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8b6c157470 msgr2=0x7f8b6c1519f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:24:22.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:22.028+0000 7f8b6be00640 1 --2- 172.21.15.6:0/3064964089 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8b6c157470 0x7f8b6c1519f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:24:22.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:22.028+0000 7f8b6be00640 1 -- 172.21.15.6:0/3064964089 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8b6c150990 msgr2=0x7f8b6c1f42f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:24:22.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:22.028+0000 7f8b6be00640 1 --2- 172.21.15.6:0/3064964089 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8b6c150990 0x7f8b6c1f42f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:24:22.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:22.028+0000 7f8b6be00640 1 -- 172.21.15.6:0/3064964089 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b6c1fd4d0 con 0x7f8b6c073270 2024-04-24T23:24:22.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:22.028+0000 7f8b6be00640 1 --2- 172.21.15.6:0/3064964089 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8b6c073270 0x7f8b6c201240 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f8b5c00c530 tx=0x7f8b5c00c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:24:22.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:22.028+0000 7f8b68c00640 1 -- 172.21.15.6:0/3064964089 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b5c016020 con 0x7f8b6c073270 2024-04-24T23:24:22.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:22.028+0000 7f8b68c00640 1 -- 172.21.15.6:0/3064964089 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8b5c012830 con 0x7f8b6c073270 2024-04-24T23:24:22.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:22.028+0000 7f8b68c00640 1 -- 172.21.15.6:0/3064964089 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b5c012c00 con 0x7f8b6c073270 2024-04-24T23:24:22.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:22.028+0000 7f8b72a00640 1 -- 172.21.15.6:0/3064964089 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b6c1fd770 con 0x7f8b6c073270 2024-04-24T23:24:22.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:22.029+0000 7f8b72a00640 1 -- 172.21.15.6:0/3064964089 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b6c1fdd30 con 0x7f8b6c073270 2024-04-24T23:24:22.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:22.031+0000 7f8b72a00640 1 -- 172.21.15.6:0/3064964089 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b6c0767b0 con 0x7f8b6c073270 2024-04-24T23:24:22.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:22.032+0000 7f8b70a00640 1 --2- 172.21.15.6:0/3064964089 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8b6c157470 0x7f8b6c1519f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:24:22.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:22.032+0000 7f8b68c00640 1 -- 172.21.15.6:0/3064964089 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f8b5c019020 con 0x7f8b6c073270 2024-04-24T23:24:22.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:22.033+0000 7f8b68c00640 1 --2- 172.21.15.6:0/3064964089 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f8b4806be80 0x7f8b4806e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:24:22.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:22.033+0000 7f8b68c00640 1 -- 172.21.15.6:0/3064964089 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6762+0+0 (secure 0 0 0) 0x7f8b5c08b300 con 0x7f8b6c073270 2024-04-24T23:24:22.036 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-24T23:24:22.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:22.036+0000 7f8b63e00640 1 --2- 172.21.15.6:0/3064964089 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f8b4806be80 0x7f8b4806e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:24:22.042 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-24T23:24:22.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:22.037+0000 7f8b63e00640 1 --2- 172.21.15.6:0/3064964089 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f8b4806be80 0x7f8b4806e340 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f8b54034000 tx=0x7f8b54030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:24:22.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:22.050+0000 7f8b68c00640 1 -- 172.21.15.6:0/3064964089 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f8b5c020020 con 0x7f8b6c073270 2024-04-24T23:24:22.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:22.496+0000 7f8b72a00640 1 -- 172.21.15.6:0/3064964089 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8b6c154310 con 0x7f8b6c073270 2024-04-24T23:24:22.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:22.497+0000 7f8b68c00640 1 -- 172.21.15.6:0/3064964089 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) ==== 94+0+6852 (secure 0 0 0) 0x7f8b5c0182c0 con 0x7f8b6c073270 2024-04-24T23:24:22.502 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:24:22.502 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":42,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5081,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/1190759626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1190759626},{"type":"v1","addr":"172.21.15.6:6834","nonce":1190759626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":5249,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6838/2416336040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2416336040},{"type":"v1","addr":"172.21.15.190:6838","nonce":2416336040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":42},{"gid":5503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/3928731059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":3928731059},{"type":"v1","addr":"172.21.15.6:6837","nonce":3928731059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":35},{"gid":5599,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6839/2643233757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":2643233757},{"type":"v1","addr":"172.21.15.190:6839","nonce":2643233757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40}],"filesystems":[{"mdsmap":{"epoch":39,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:23:38.162102+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":4306,"mds_1":4282},"failed":[],"damaged":[],"stopped":[2,3,4],"info":{"gid_4282":{"gid":4282,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":12,"addr":"172.21.15.190:6837/580315269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":580315269},{"type":"v1","addr":"172.21.15.190:6837","nonce":580315269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/489092393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":489092393},{"type":"v1","addr":"172.21.15.6:6835","nonce":489092393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4306,"qdb_cluster":[4306,4282]},"id":1}]} 2024-04-24T23:24:22.503 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 42 2024-04-24T23:24:22.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:22.501+0000 7f8b61600640 1 -- 172.21.15.6:0/3064964089 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f8b4806be80 msgr2=0x7f8b4806e340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:24:22.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:22.501+0000 7f8b61600640 1 --2- 172.21.15.6:0/3064964089 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f8b4806be80 0x7f8b4806e340 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f8b54034000 tx=0x7f8b54030000 comp rx=0 tx=0).stop 2024-04-24T23:24:22.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:22.501+0000 7f8b61600640 1 -- 172.21.15.6:0/3064964089 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8b6c073270 msgr2=0x7f8b6c201240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:24:22.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:22.501+0000 7f8b61600640 1 --2- 172.21.15.6:0/3064964089 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8b6c073270 0x7f8b6c201240 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f8b5c00c530 tx=0x7f8b5c00c9f0 comp rx=0 tx=0).stop 2024-04-24T23:24:22.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:22.502+0000 7f8b61600640 1 -- 172.21.15.6:0/3064964089 shutdown_connections 2024-04-24T23:24:22.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:22.502+0000 7f8b61600640 1 --2- 172.21.15.6:0/3064964089 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f8b4806be80 0x7f8b4806e340 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:24:22.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:22.502+0000 7f8b61600640 1 --2- 172.21.15.6:0/3064964089 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8b6c157470 0x7f8b6c1519f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:24:22.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:22.502+0000 7f8b61600640 1 --2- 172.21.15.6:0/3064964089 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8b6c150990 0x7f8b6c1f42f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:24:22.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:22.502+0000 7f8b61600640 1 --2- 172.21.15.6:0/3064964089 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8b6c073270 0x7f8b6c201240 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:24:22.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:22.502+0000 7f8b61600640 1 -- 172.21.15.6:0/3064964089 >> 172.21.15.6:0/3064964089 conn(0x7f8b6c06b0e0 msgr2=0x7f8b6c0c5140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:24:22.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:22.503+0000 7f8b61600640 1 -- 172.21.15.6:0/3064964089 shutdown_connections 2024-04-24T23:24:22.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:22.503+0000 7f8b61600640 1 -- 172.21.15.6:0/3064964089 wait complete. 2024-04-24T23:24:22.533 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2024-04-24T23:24:24.533 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:24:24.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:24.933+0000 7f26c2800640 1 Processor -- start 2024-04-24T23:24:24.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:24.933+0000 7f26c2800640 1 -- start start 2024-04-24T23:24:24.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:24.935+0000 7f26c2800640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f26bc167180 0x7f26bc0c7dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:24:24.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:24.935+0000 7f26c2800640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f26bc16abf0 0x7f26bc16b080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:24:24.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:24.935+0000 7f26c2800640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f26bc072970 con 0x7f26bc072530 2024-04-24T23:24:24.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:24.935+0000 7f26c2800640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f26bc168570 con 0x7f26bc16abf0 2024-04-24T23:24:24.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:24.935+0000 7f26c2800640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f26bc0720a0 con 0x7f26bc167180 2024-04-24T23:24:24.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:24.936+0000 7f26bb400640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f26bc167180 0x7f26bc0c7dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:24:24.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:24.936+0000 7f26baa00640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f26bc072530 0x7f26bc06edb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.6:49706/0 (socket says 172.21.15.6:49706) 2024-04-24T23:24:24.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:24.936+0000 7f26baa00640 1 -- 172.21.15.6:0/979489046 learned_addr learned my addr 172.21.15.6:0/979489046 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:24:24.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:24.937+0000 7f26ba000640 1 -- 172.21.15.6:0/979489046 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2589022917 0 0) 0x7f26a8001110 con 0x7f26bc072530 2024-04-24T23:24:24.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:24.936+0000 7f26bbe00640 1 --2- 172.21.15.6:0/979489046 >> v2:172.21.15.190:3300/0 conn(0x7f26bc16abf0 0x7f26bc16b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:24:24.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:24.963+0000 7f26ba000640 1 -- 172.21.15.6:0/979489046 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f26a0003500 con 0x7f26bc072530 2024-04-24T23:24:24.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:24.963+0000 7f26bb400640 1 -- 172.21.15.6:0/979489046 >> v2:172.21.15.190:3300/0 conn(0x7f26bc16abf0 msgr2=0x7f26bc16b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:24:24.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:24.963+0000 7f26bb400640 1 --2- 172.21.15.6:0/979489046 >> v2:172.21.15.190:3300/0 conn(0x7f26bc16abf0 0x7f26bc16b080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:24:24.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:24.963+0000 7f26bb400640 1 -- 172.21.15.6:0/979489046 >> v1:172.21.15.190:6789/0 conn(0x7f26bc072530 legacy=0x7f26bc06edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:24:24.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:24.963+0000 7f26bb400640 1 -- 172.21.15.6:0/979489046 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26bc0c74c0 con 0x7f26bc167180 2024-04-24T23:24:24.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:24.965+0000 7f26bb400640 1 --2- 172.21.15.6:0/979489046 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f26bc167180 0x7f26bc0c7dc0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f26ac008260 tx=0x7f26ac00ab30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c36c5cc570cef8e3 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:24:24.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:24.966+0000 7f26ba000640 1 -- 172.21.15.6:0/979489046 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26ac0089e0 con 0x7f26bc167180 2024-04-24T23:24:24.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:24.966+0000 7f26ba000640 1 -- 172.21.15.6:0/979489046 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f26ac008b80 con 0x7f26bc167180 2024-04-24T23:24:24.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:24.966+0000 7f26ba000640 1 -- 172.21.15.6:0/979489046 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26ac00a0d0 con 0x7f26bc167180 2024-04-24T23:24:24.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:24.967+0000 7f26c2800640 1 -- 172.21.15.6:0/979489046 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f26bc167180 msgr2=0x7f26bc0c7dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:24:24.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:24.967+0000 7f26c2800640 1 --2- 172.21.15.6:0/979489046 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f26bc167180 0x7f26bc0c7dc0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f26ac008260 tx=0x7f26ac00ab30 comp rx=0 tx=0).stop 2024-04-24T23:24:24.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:24.967+0000 7f26c2800640 1 -- 172.21.15.6:0/979489046 shutdown_connections 2024-04-24T23:24:24.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:24.967+0000 7f26c2800640 1 --2- 172.21.15.6:0/979489046 >> v2:172.21.15.190:3300/0 conn(0x7f26bc16abf0 0x7f26bc16b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:24:24.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:24.967+0000 7f26c2800640 1 --2- 172.21.15.6:0/979489046 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f26bc167180 0x7f26bc0c7dc0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:24:24.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:24.968+0000 7f26c2800640 1 -- 172.21.15.6:0/979489046 >> 172.21.15.6:0/979489046 conn(0x7f26bc06b0e0 msgr2=0x7f26bc06b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:24:24.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:24.969+0000 7f26c2800640 1 -- 172.21.15.6:0/979489046 shutdown_connections 2024-04-24T23:24:24.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:24.969+0000 7f26c2800640 1 -- 172.21.15.6:0/979489046 wait complete. 2024-04-24T23:24:24.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:24.974+0000 7f26c2800640 1 Processor -- start 2024-04-24T23:24:24.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:24.974+0000 7f26c2800640 1 -- start start 2024-04-24T23:24:24.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:24.976+0000 7f26c2800640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f26bc072530 0x7f26bc0c57a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:24:24.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:24.976+0000 7f26c2800640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f26bc167180 0x7f26bc1fe520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:24:24.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:24.976+0000 7f26c2800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f26bc16abf0 0x7f26bc1f4080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:24:24.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:24.977+0000 7f26c2800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f26bc071a50 con 0x7f26bc16abf0 2024-04-24T23:24:24.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:24.977+0000 7f26c2800640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f26bc071770 con 0x7f26bc167180 2024-04-24T23:24:24.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:24.977+0000 7f26bb400640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f26bc072530 0x7f26bc0c57a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:24:24.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:24.977+0000 7f26bb400640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f26bc072530 0x7f26bc0c57a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.6:49142/0 (socket says 172.21.15.6:49142) 2024-04-24T23:24:24.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:24.981+0000 7f26bb400640 1 -- 172.21.15.6:0/3555596822 learned_addr learned my addr 172.21.15.6:0/3555596822 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:24:24.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:24.981+0000 7f26c2800640 1 -- 172.21.15.6:0/3555596822 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f26bc071d30 con 0x7f26bc072530 2024-04-24T23:24:24.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:24.981+0000 7f26baa00640 1 --2- 172.21.15.6:0/3555596822 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f26bc167180 0x7f26bc1fe520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:24:24.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:24.983+0000 7f26bbe00640 1 --2- 172.21.15.6:0/3555596822 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f26bc16abf0 0x7f26bc1f4080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:24:24.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:24.986+0000 7f26bbe00640 1 -- 172.21.15.6:0/3555596822 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f26bc072530 msgr2=0x7f26bc0c57a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:24:24.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:24.986+0000 7f26bbe00640 1 --2- 172.21.15.6:0/3555596822 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f26bc072530 0x7f26bc0c57a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:24:24.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:24.986+0000 7f26bbe00640 1 -- 172.21.15.6:0/3555596822 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f26bc167180 msgr2=0x7f26bc1fe520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:24:24.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:24.986+0000 7f26bbe00640 1 --2- 172.21.15.6:0/3555596822 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f26bc167180 0x7f26bc1fe520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:24:24.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:24.986+0000 7f26bbe00640 1 -- 172.21.15.6:0/3555596822 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26bc1641e0 con 0x7f26bc16abf0 2024-04-24T23:24:24.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:24.987+0000 7f26bb400640 1 --2- 172.21.15.6:0/3555596822 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f26bc072530 0x7f26bc0c57a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:24:24.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:24.987+0000 7f26baa00640 1 --2- 172.21.15.6:0/3555596822 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f26bc167180 0x7f26bc1fe520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:24:24.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:24.987+0000 7f26bbe00640 1 --2- 172.21.15.6:0/3555596822 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f26bc16abf0 0x7f26bc1f4080 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f26b4012f40 tx=0x7f26b4010490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:24:24.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:24.988+0000 7f26b3400640 1 -- 172.21.15.6:0/3555596822 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26b4010b20 con 0x7f26bc16abf0 2024-04-24T23:24:24.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:24.988+0000 7f26c2800640 1 -- 172.21.15.6:0/3555596822 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26bc200750 con 0x7f26bc16abf0 2024-04-24T23:24:24.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:24.988+0000 7f26c2800640 1 -- 172.21.15.6:0/3555596822 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26bc163c20 con 0x7f26bc16abf0 2024-04-24T23:24:24.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:24.994+0000 7f26b3400640 1 -- 172.21.15.6:0/3555596822 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f26b4010cc0 con 0x7f26bc16abf0 2024-04-24T23:24:24.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:24.994+0000 7f26b3400640 1 -- 172.21.15.6:0/3555596822 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26b4010b20 con 0x7f26bc16abf0 2024-04-24T23:24:25.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:25.004+0000 7f26c2800640 1 -- 172.21.15.6:0/3555596822 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2694005290 con 0x7f26bc16abf0 2024-04-24T23:24:25.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:25.005+0000 7f26b3400640 1 -- 172.21.15.6:0/3555596822 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f26b401f030 con 0x7f26bc16abf0 2024-04-24T23:24:25.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:25.011+0000 7f26b3400640 1 --2- 172.21.15.6:0/3555596822 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f269c06c030 0x7f269c06e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:24:25.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:25.011+0000 7f26b3400640 1 -- 172.21.15.6:0/3555596822 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6762+0+0 (secure 0 0 0) 0x7f26b408ef60 con 0x7f26bc16abf0 2024-04-24T23:24:25.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:25.013+0000 7f26bb400640 1 --2- 172.21.15.6:0/3555596822 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f269c06c030 0x7f269c06e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:24:25.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:25.018+0000 7f26bb400640 1 --2- 172.21.15.6:0/3555596822 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f269c06c030 0x7f269c06e4f0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f26ac00ab70 tx=0x7f26ac02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:24:25.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:25.025+0000 7f26b3400640 1 -- 172.21.15.6:0/3555596822 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f26b4026020 con 0x7f26bc16abf0 2024-04-24T23:24:25.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:25.574+0000 7f26c2800640 1 -- 172.21.15.6:0/3555596822 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2694005850 con 0x7f26bc16abf0 2024-04-24T23:24:25.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:25.575+0000 7f26b3400640 1 -- 172.21.15.6:0/3555596822 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) ==== 94+0+6852 (secure 0 0 0) 0x7f26b4014040 con 0x7f26bc16abf0 2024-04-24T23:24:25.578 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:24:25.578 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":42,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5081,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/1190759626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1190759626},{"type":"v1","addr":"172.21.15.6:6834","nonce":1190759626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":5249,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6838/2416336040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2416336040},{"type":"v1","addr":"172.21.15.190:6838","nonce":2416336040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":42},{"gid":5503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/3928731059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":3928731059},{"type":"v1","addr":"172.21.15.6:6837","nonce":3928731059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":35},{"gid":5599,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6839/2643233757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":2643233757},{"type":"v1","addr":"172.21.15.190:6839","nonce":2643233757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40}],"filesystems":[{"mdsmap":{"epoch":39,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:23:38.162102+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":4306,"mds_1":4282},"failed":[],"damaged":[],"stopped":[2,3,4],"info":{"gid_4282":{"gid":4282,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":12,"addr":"172.21.15.190:6837/580315269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":580315269},{"type":"v1","addr":"172.21.15.190:6837","nonce":580315269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/489092393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":489092393},{"type":"v1","addr":"172.21.15.6:6835","nonce":489092393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4306,"qdb_cluster":[4306,4282]},"id":1}]} 2024-04-24T23:24:25.578 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 42 2024-04-24T23:24:25.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:25.584+0000 7f26c2800640 1 -- 172.21.15.6:0/3555596822 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f269c06c030 msgr2=0x7f269c06e4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:24:25.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:25.584+0000 7f26c2800640 1 --2- 172.21.15.6:0/3555596822 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f269c06c030 0x7f269c06e4f0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f26ac00ab70 tx=0x7f26ac02e000 comp rx=0 tx=0).stop 2024-04-24T23:24:25.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:25.584+0000 7f26c2800640 1 -- 172.21.15.6:0/3555596822 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f26bc16abf0 msgr2=0x7f26bc1f4080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:24:25.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:25.584+0000 7f26c2800640 1 --2- 172.21.15.6:0/3555596822 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f26bc16abf0 0x7f26bc1f4080 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f26b4012f40 tx=0x7f26b4010490 comp rx=0 tx=0).stop 2024-04-24T23:24:25.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:25.585+0000 7f26c2800640 1 -- 172.21.15.6:0/3555596822 shutdown_connections 2024-04-24T23:24:25.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:25.585+0000 7f26c2800640 1 --2- 172.21.15.6:0/3555596822 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f269c06c030 0x7f269c06e4f0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:24:25.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:25.585+0000 7f26c2800640 1 --2- 172.21.15.6:0/3555596822 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f26bc16abf0 0x7f26bc1f4080 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:24:25.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:25.585+0000 7f26c2800640 1 --2- 172.21.15.6:0/3555596822 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f26bc167180 0x7f26bc1fe520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:24:25.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:25.585+0000 7f26c2800640 1 --2- 172.21.15.6:0/3555596822 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f26bc072530 0x7f26bc0c57a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:24:25.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:25.585+0000 7f26c2800640 1 -- 172.21.15.6:0/3555596822 >> 172.21.15.6:0/3555596822 conn(0x7f26bc06b0e0 msgr2=0x7f26bc0cab30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:24:25.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:25.586+0000 7f26c2800640 1 -- 172.21.15.6:0/3555596822 shutdown_connections 2024-04-24T23:24:25.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:25.586+0000 7f26c2800640 1 -- 172.21.15.6:0/3555596822 wait complete. 2024-04-24T23:24:25.621 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2024-04-24T23:24:27.623 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:24:28.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:28.009+0000 7ff325e00640 1 Processor -- start 2024-04-24T23:24:28.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:28.010+0000 7ff325e00640 1 -- start start 2024-04-24T23:24:28.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:28.017+0000 7ff325e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ff320073270 0x7ff320150990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:24:28.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:28.021+0000 7ff31ea00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7ff32006bbd0 0x7ff32015a980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:46984/0 (socket says 172.21.15.6:46984) 2024-04-24T23:24:28.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:28.021+0000 7ff31ea00640 1 -- 172.21.15.6:0/2614257005 learned_addr learned my addr 172.21.15.6:0/2614257005 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:24:28.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:28.022+0000 7ff325e00640 1 -- 172.21.15.6:0/2614257005 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff3200c8690 con 0x7ff32006bbd0 2024-04-24T23:24:28.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:28.025+0000 7ff325e00640 1 -- 172.21.15.6:0/2614257005 --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff3200c8970 con 0x7ff320157470 2024-04-24T23:24:28.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:28.025+0000 7ff325e00640 1 -- 172.21.15.6:0/2614257005 --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7ff3200c8c50 con 0x7ff320073270 2024-04-24T23:24:28.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:28.026+0000 7ff31d600640 1 -- 172.21.15.6:0/2614257005 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4182283204 0 0) 0x7ff3200c8970 con 0x7ff320157470 2024-04-24T23:24:28.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:28.026+0000 7ff31d600640 1 -- 172.21.15.6:0/2614257005 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff304003520 con 0x7ff320157470 2024-04-24T23:24:28.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:28.027+0000 7ff31d600640 1 -- 172.21.15.6:0/2614257005 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3402348058 0 0) 0x7ff304003520 con 0x7ff320157470 2024-04-24T23:24:28.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:28.027+0000 7ff31d600640 1 -- 172.21.15.6:0/2614257005 >> v2:172.21.15.6:3300/0 conn(0x7ff320073270 msgr2=0x7ff320150990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:24:28.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:28.027+0000 7ff31d600640 1 --2- 172.21.15.6:0/2614257005 >> v2:172.21.15.6:3300/0 conn(0x7ff320073270 0x7ff320150990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:24:28.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:28.027+0000 7ff31d600640 1 -- 172.21.15.6:0/2614257005 >> v1:172.21.15.6:6789/0 conn(0x7ff32006bbd0 legacy=0x7ff32015a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:24:28.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:28.027+0000 7ff31d600640 1 -- 172.21.15.6:0/2614257005 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff32015a180 con 0x7ff320157470 2024-04-24T23:24:28.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:28.028+0000 7ff31d600640 1 -- 172.21.15.6:0/2614257005 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3463268701 0 0) 0x7ff3180019d0 con 0x7ff320157470 2024-04-24T23:24:28.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:28.029+0000 7ff31d600640 1 -- 172.21.15.6:0/2614257005 >> v1:172.21.15.190:6789/0 conn(0x7ff320157470 legacy=0x7ff3200c9110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:24:28.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:28.029+0000 7ff31d600640 1 --2- 172.21.15.6:0/2614257005 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff304003a10 0x7ff304003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:24:28.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:28.029+0000 7ff31d600640 1 --2- 172.21.15.6:0/2614257005 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff304004800 0x7ff304004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:24:28.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:28.029+0000 7ff31d600640 1 --2- 172.21.15.6:0/2614257005 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff304005370 0x7ff3040141f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:24:28.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:28.029+0000 7ff31d600640 1 -- 172.21.15.6:0/2614257005 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7ff3200c8c50 con 0x7ff304005370 2024-04-24T23:24:28.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:28.029+0000 7ff31d600640 1 -- 172.21.15.6:0/2614257005 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7ff304014ad0 con 0x7ff304003a10 2024-04-24T23:24:28.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:28.029+0000 7ff31d600640 1 -- 172.21.15.6:0/2614257005 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7ff304014cd0 con 0x7ff304004800 2024-04-24T23:24:28.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:28.031+0000 7ff31f400640 1 --2- 172.21.15.6:0/2614257005 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff304005370 0x7ff3040141f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:24:28.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:28.033+0000 7ff31f400640 1 -- 172.21.15.6:0/2614257005 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff304004800 msgr2=0x7ff304004c80 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-24T23:24:28.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:28.033+0000 7ff31f400640 1 --2- 172.21.15.6:0/2614257005 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff304004800 0x7ff304004c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:24:28.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:28.033+0000 7ff31f400640 1 -- 172.21.15.6:0/2614257005 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff304003a10 msgr2=0x7ff304003e50 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-24T23:24:28.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:28.033+0000 7ff31f400640 1 --2- 172.21.15.6:0/2614257005 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff304003a10 0x7ff304003e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:24:28.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:28.033+0000 7ff31f400640 1 -- 172.21.15.6:0/2614257005 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff32015a180 con 0x7ff304005370 2024-04-24T23:24:28.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:28.034+0000 7ff31f400640 1 -- 172.21.15.6:0/2614257005 reap_dead start 2024-04-24T23:24:28.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:28.040+0000 7ff31f400640 1 --2- 172.21.15.6:0/2614257005 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff304005370 0x7ff3040141f0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7ff31800c500 tx=0x7ff304005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6da62ab439e42cbf server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:24:28.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:28.040+0000 7ff31d600640 1 -- 172.21.15.6:0/2614257005 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff31800c100 con 0x7ff304005370 2024-04-24T23:24:28.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:28.041+0000 7ff31d600640 1 -- 172.21.15.6:0/2614257005 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff318006d50 con 0x7ff304005370 2024-04-24T23:24:28.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:28.042+0000 7ff325e00640 1 -- 172.21.15.6:0/2614257005 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff304005370 msgr2=0x7ff3040141f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:24:28.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:28.042+0000 7ff325e00640 1 --2- 172.21.15.6:0/2614257005 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff304005370 0x7ff3040141f0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7ff31800c500 tx=0x7ff304005830 comp rx=0 tx=0).stop 2024-04-24T23:24:28.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:28.044+0000 7ff325e00640 1 -- 172.21.15.6:0/2614257005 shutdown_connections 2024-04-24T23:24:28.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:28.044+0000 7ff325e00640 1 --2- 172.21.15.6:0/2614257005 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff304005370 0x7ff3040141f0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:24:28.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:28.044+0000 7ff325e00640 1 -- 172.21.15.6:0/2614257005 >> 172.21.15.6:0/2614257005 conn(0x7ff32006b0e0 msgr2=0x7ff32006b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:24:28.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:28.056+0000 7ff325e00640 1 -- 172.21.15.6:0/2614257005 shutdown_connections 2024-04-24T23:24:28.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:28.056+0000 7ff325e00640 1 -- 172.21.15.6:0/2614257005 wait complete. 2024-04-24T23:24:28.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:28.059+0000 7ff325e00640 1 Processor -- start 2024-04-24T23:24:28.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:28.063+0000 7ff325e00640 1 -- start start 2024-04-24T23:24:28.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:28.064+0000 7ff325e00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff3201ef6c0 0x7ff320155c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:24:28.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:28.064+0000 7ff325e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff32014ec00 0x7ff320145e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:24:28.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:28.064+0000 7ff325e00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff3201efaa0 0x7ff3201f0360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:24:28.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:28.066+0000 7ff31ea00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff3201ef6c0 0x7ff320155c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:24:28.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:28.066+0000 7ff31ea00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff3201ef6c0 0x7ff320155c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.6:49158/0 (socket says 172.21.15.6:49158) 2024-04-24T23:24:28.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:28.067+0000 7ff325e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7ff3200c45f0 con 0x7ff32014ec00 2024-04-24T23:24:28.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:28.067+0000 7ff325e00640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7ff3200c7a70 con 0x7ff3201efaa0 2024-04-24T23:24:28.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:28.067+0000 7ff325e00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7ff3200c4770 con 0x7ff3201ef6c0 2024-04-24T23:24:28.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:28.067+0000 7ff31f400640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff3201efaa0 0x7ff3201f0360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:24:28.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:28.067+0000 7ff31e000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff32014ec00 0x7ff320145e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:24:28.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:28.067+0000 7ff31f400640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff3201efaa0 0x7ff3201f0360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:35894/0 (socket says 172.21.15.6:35894) 2024-04-24T23:24:28.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:28.067+0000 7ff31f400640 1 -- 172.21.15.6:0/2950056233 learned_addr learned my addr 172.21.15.6:0/2950056233 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:24:28.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:28.068+0000 7ff31ea00640 1 -- 172.21.15.6:0/2950056233 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff3201efaa0 msgr2=0x7ff3201f0360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:24:28.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:28.068+0000 7ff31ea00640 1 --2- 172.21.15.6:0/2950056233 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff3201efaa0 0x7ff3201f0360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:24:28.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:28.068+0000 7ff31ea00640 1 -- 172.21.15.6:0/2950056233 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff32014ec00 msgr2=0x7ff320145e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:24:28.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:28.069+0000 7ff31ea00640 1 --2- 172.21.15.6:0/2950056233 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff32014ec00 0x7ff320145e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:24:28.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:28.069+0000 7ff31ea00640 1 -- 172.21.15.6:0/2950056233 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff32014f0d0 con 0x7ff3201ef6c0 2024-04-24T23:24:28.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:28.069+0000 7ff31e000640 1 --2- 172.21.15.6:0/2950056233 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff32014ec00 0x7ff320145e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:24:28.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:28.070+0000 7ff31ea00640 1 --2- 172.21.15.6:0/2950056233 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff3201ef6c0 0x7ff320155c00 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7ff30c00c390 tx=0x7ff30c00c850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:24:28.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:28.071+0000 7ff317400640 1 -- 172.21.15.6:0/2950056233 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff30c017070 con 0x7ff3201ef6c0 2024-04-24T23:24:28.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:28.071+0000 7ff317400640 1 -- 172.21.15.6:0/2950056233 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff30c005ce0 con 0x7ff3201ef6c0 2024-04-24T23:24:28.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:28.071+0000 7ff317400640 1 -- 172.21.15.6:0/2950056233 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff30c0072d0 con 0x7ff3201ef6c0 2024-04-24T23:24:28.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:28.073+0000 7ff325e00640 1 -- 172.21.15.6:0/2950056233 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff32014f390 con 0x7ff3201ef6c0 2024-04-24T23:24:28.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:28.074+0000 7ff325e00640 1 -- 172.21.15.6:0/2950056233 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff320155120 con 0x7ff3201ef6c0 2024-04-24T23:24:28.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:28.076+0000 7ff325e00640 1 -- 172.21.15.6:0/2950056233 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff320071df0 con 0x7ff3201ef6c0 2024-04-24T23:24:28.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:28.080+0000 7ff317400640 1 -- 172.21.15.6:0/2950056233 <== mon.2 v2:172.21.15.190:3301/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7ff30c01a070 con 0x7ff3201ef6c0 2024-04-24T23:24:28.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:28.082+0000 7ff317400640 1 --2- 172.21.15.6:0/2950056233 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7ff30006be30 0x7ff30006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:24:28.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:28.082+0000 7ff317400640 1 -- 172.21.15.6:0/2950056233 <== mon.2 v2:172.21.15.190:3301/0 5 ==== osd_map(28..28 src has 1..28) ==== 6762+0+0 (secure 0 0 0) 0x7ff30c08a980 con 0x7ff3201ef6c0 2024-04-24T23:24:28.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:28.083+0000 7ff31e000640 1 --2- 172.21.15.6:0/2950056233 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7ff30006be30 0x7ff30006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:24:28.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:28.084+0000 7ff31e000640 1 --2- 172.21.15.6:0/2950056233 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7ff30006be30 0x7ff30006e2f0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7ff320157110 tx=0x7ff310009110 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:24:28.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:28.092+0000 7ff317400640 1 -- 172.21.15.6:0/2950056233 <== mon.2 v2:172.21.15.190:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7ff320071df0 con 0x7ff3201ef6c0 2024-04-24T23:24:28.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:28.653+0000 7ff325e00640 1 -- 172.21.15.6:0/2950056233 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff320159b80 con 0x7ff3201ef6c0 2024-04-24T23:24:28.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:28.655+0000 7ff317400640 1 -- 172.21.15.6:0/2950056233 <== mon.2 v2:172.21.15.190:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) ==== 94+0+6852 (secure 0 0 0) 0x7ff320159b80 con 0x7ff3201ef6c0 2024-04-24T23:24:28.658 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:24:28.658 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":42,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5081,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/1190759626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1190759626},{"type":"v1","addr":"172.21.15.6:6834","nonce":1190759626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":5249,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6838/2416336040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2416336040},{"type":"v1","addr":"172.21.15.190:6838","nonce":2416336040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":42},{"gid":5503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/3928731059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":3928731059},{"type":"v1","addr":"172.21.15.6:6837","nonce":3928731059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":35},{"gid":5599,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6839/2643233757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":2643233757},{"type":"v1","addr":"172.21.15.190:6839","nonce":2643233757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40}],"filesystems":[{"mdsmap":{"epoch":39,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:23:38.162102+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":4306,"mds_1":4282},"failed":[],"damaged":[],"stopped":[2,3,4],"info":{"gid_4282":{"gid":4282,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":12,"addr":"172.21.15.190:6837/580315269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":580315269},{"type":"v1","addr":"172.21.15.190:6837","nonce":580315269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/489092393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":489092393},{"type":"v1","addr":"172.21.15.6:6835","nonce":489092393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4306,"qdb_cluster":[4306,4282]},"id":1}]} 2024-04-24T23:24:28.658 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 42 2024-04-24T23:24:28.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:28.663+0000 7ff325e00640 1 -- 172.21.15.6:0/2950056233 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7ff30006be30 msgr2=0x7ff30006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:24:28.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:28.663+0000 7ff325e00640 1 --2- 172.21.15.6:0/2950056233 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7ff30006be30 0x7ff30006e2f0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7ff320157110 tx=0x7ff310009110 comp rx=0 tx=0).stop 2024-04-24T23:24:28.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:28.663+0000 7ff325e00640 1 -- 172.21.15.6:0/2950056233 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff3201ef6c0 msgr2=0x7ff320155c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:24:28.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:28.663+0000 7ff325e00640 1 --2- 172.21.15.6:0/2950056233 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff3201ef6c0 0x7ff320155c00 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7ff30c00c390 tx=0x7ff30c00c850 comp rx=0 tx=0).stop 2024-04-24T23:24:28.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:28.663+0000 7ff325e00640 1 -- 172.21.15.6:0/2950056233 shutdown_connections 2024-04-24T23:24:28.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:28.663+0000 7ff325e00640 1 --2- 172.21.15.6:0/2950056233 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7ff30006be30 0x7ff30006e2f0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:24:28.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:28.663+0000 7ff325e00640 1 --2- 172.21.15.6:0/2950056233 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff3201efaa0 0x7ff3201f0360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:24:28.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:28.663+0000 7ff325e00640 1 --2- 172.21.15.6:0/2950056233 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff32014ec00 0x7ff320145e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:24:28.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:28.663+0000 7ff325e00640 1 --2- 172.21.15.6:0/2950056233 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff3201ef6c0 0x7ff320155c00 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:24:28.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:28.663+0000 7ff325e00640 1 -- 172.21.15.6:0/2950056233 >> 172.21.15.6:0/2950056233 conn(0x7ff32006b0e0 msgr2=0x7ff32006b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:24:28.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:28.664+0000 7ff325e00640 1 -- 172.21.15.6:0/2950056233 shutdown_connections 2024-04-24T23:24:28.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:28.664+0000 7ff325e00640 1 -- 172.21.15.6:0/2950056233 wait complete. 2024-04-24T23:24:28.717 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2024-04-24T23:24:30.719 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:24:31.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:31.000+0000 7f5d7ae00640 1 Processor -- start 2024-04-24T23:24:31.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:31.000+0000 7f5d7ae00640 1 -- start start 2024-04-24T23:24:31.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:31.003+0000 7f5d7ae00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f5d74167180 0x7f5d740c7dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:24:31.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:31.003+0000 7f5d7ae00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5d7416abf0 0x7f5d7416b060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:24:31.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:31.003+0000 7f5d7ae00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5d74072970 con 0x7f5d74072530 2024-04-24T23:24:31.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:31.003+0000 7f5d7ae00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f5d74168570 con 0x7f5d74167180 2024-04-24T23:24:31.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:31.003+0000 7f5d7ae00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f5d740720a0 con 0x7f5d7416abf0 2024-04-24T23:24:31.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:31.007+0000 7f5d78e00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5d7416abf0 0x7f5d7416b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:24:31.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:31.007+0000 7f5d78e00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5d7416abf0 0x7f5d7416b060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.6:49166/0 (socket says 172.21.15.6:49166) 2024-04-24T23:24:31.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:31.007+0000 7f5d78e00640 1 -- 172.21.15.6:0/125702630 learned_addr learned my addr 172.21.15.6:0/125702630 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:24:31.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:31.007+0000 7f5d73e00640 1 --2- 172.21.15.6:0/125702630 >> v2:172.21.15.6:3300/0 conn(0x7f5d74167180 0x7f5d740c7dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:24:31.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:31.011+0000 7f5d72a00640 1 -- 172.21.15.6:0/125702630 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3177765430 0 0) 0x7f5d60001110 con 0x7f5d74072530 2024-04-24T23:24:31.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:31.122+0000 7f5d72a00640 1 -- 172.21.15.6:0/125702630 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5d58003500 con 0x7f5d74072530 2024-04-24T23:24:31.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:31.122+0000 7f5d72a00640 1 -- 172.21.15.6:0/125702630 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1537850141 0 0) 0x7f5d58003500 con 0x7f5d74072530 2024-04-24T23:24:31.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:31.123+0000 7f5d72a00640 1 -- 172.21.15.6:0/125702630 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5d7416abf0 msgr2=0x7f5d7416b060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:24:31.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:31.123+0000 7f5d72a00640 1 --2- 172.21.15.6:0/125702630 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5d7416abf0 0x7f5d7416b060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:24:31.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:31.123+0000 7f5d72a00640 1 -- 172.21.15.6:0/125702630 >> v2:172.21.15.6:3300/0 conn(0x7f5d74167180 msgr2=0x7f5d740c7dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:24:31.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:31.123+0000 7f5d72a00640 1 --2- 172.21.15.6:0/125702630 >> v2:172.21.15.6:3300/0 conn(0x7f5d74167180 0x7f5d740c7dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:24:31.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:31.123+0000 7f5d72a00640 1 -- 172.21.15.6:0/125702630 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d740c74c0 con 0x7f5d74072530 2024-04-24T23:24:31.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:31.123+0000 7f5d72a00640 1 -- 172.21.15.6:0/125702630 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3463268701 0 0) 0x7f5d60001840 con 0x7f5d74072530 2024-04-24T23:24:31.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:31.123+0000 7f5d72a00640 1 -- 172.21.15.6:0/125702630 >> v1:172.21.15.6:6789/0 conn(0x7f5d74072530 legacy=0x7f5d7406edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:24:31.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:31.123+0000 7f5d72a00640 1 --2- 172.21.15.6:0/125702630 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5d58014ce0 0x7f5d580170d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:24:31.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:31.123+0000 7f5d72a00640 1 --2- 172.21.15.6:0/125702630 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5d58017610 0x7f5d58019a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:24:31.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:31.123+0000 7f5d72a00640 1 --2- 172.21.15.6:0/125702630 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5d5801a090 0x7f5d5801c520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:24:31.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:31.123+0000 7f5d72a00640 1 -- 172.21.15.6:0/125702630 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f5d740720a0 con 0x7f5d5801a090 2024-04-24T23:24:31.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:31.123+0000 7f5d72a00640 1 -- 172.21.15.6:0/125702630 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f5d5801cdf0 con 0x7f5d58017610 2024-04-24T23:24:31.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:31.123+0000 7f5d72a00640 1 -- 172.21.15.6:0/125702630 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f5d5801cff0 con 0x7f5d58014ce0 2024-04-24T23:24:31.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:31.127+0000 7f5d78e00640 1 --2- 172.21.15.6:0/125702630 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5d5801a090 0x7f5d5801c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:24:31.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:31.128+0000 7f5d73400640 1 --2- 172.21.15.6:0/125702630 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5d58017610 0x7f5d58019a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:24:31.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:31.128+0000 7f5d73e00640 1 --2- 172.21.15.6:0/125702630 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5d58014ce0 0x7f5d580170d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:24:31.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:31.128+0000 7f5d73400640 1 -- 172.21.15.6:0/125702630 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5d58014ce0 msgr2=0x7f5d580170d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:24:31.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:31.128+0000 7f5d73400640 1 --2- 172.21.15.6:0/125702630 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5d58014ce0 0x7f5d580170d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:24:31.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:31.128+0000 7f5d73400640 1 -- 172.21.15.6:0/125702630 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5d5801a090 msgr2=0x7f5d5801c520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:24:31.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:31.128+0000 7f5d73400640 1 --2- 172.21.15.6:0/125702630 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5d5801a090 0x7f5d5801c520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:24:31.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:31.128+0000 7f5d73400640 1 -- 172.21.15.6:0/125702630 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d740c74c0 con 0x7f5d58017610 2024-04-24T23:24:31.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:31.131+0000 7f5d73400640 1 --2- 172.21.15.6:0/125702630 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5d58017610 0x7f5d58019a20 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f5d6000c370 tx=0x7f5d580037f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=974ad8e388a182fe server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:24:31.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:31.131+0000 7f5d72a00640 1 -- 172.21.15.6:0/125702630 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d60007230 con 0x7f5d58017610 2024-04-24T23:24:31.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:31.131+0000 7f5d72a00640 1 -- 172.21.15.6:0/125702630 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5d60035070 con 0x7f5d58017610 2024-04-24T23:24:31.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:31.136+0000 7f5d72a00640 1 -- 172.21.15.6:0/125702630 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d60007630 con 0x7f5d58017610 2024-04-24T23:24:31.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:31.136+0000 7f5d7ae00640 1 -- 172.21.15.6:0/125702630 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5d58017610 msgr2=0x7f5d58019a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:24:31.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:31.136+0000 7f5d7ae00640 1 --2- 172.21.15.6:0/125702630 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5d58017610 0x7f5d58019a20 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f5d6000c370 tx=0x7f5d580037f0 comp rx=0 tx=0).stop 2024-04-24T23:24:31.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:31.136+0000 7f5d78e00640 1 -- 172.21.15.6:0/125702630 reap_dead start 2024-04-24T23:24:31.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:31.136+0000 7f5d7ae00640 1 -- 172.21.15.6:0/125702630 shutdown_connections 2024-04-24T23:24:31.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:31.136+0000 7f5d7ae00640 1 -- 172.21.15.6:0/125702630 >> 172.21.15.6:0/125702630 conn(0x7f5d7406b0e0 msgr2=0x7f5d7406b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:24:31.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:31.137+0000 7f5d7ae00640 1 -- 172.21.15.6:0/125702630 shutdown_connections 2024-04-24T23:24:31.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:31.137+0000 7f5d7ae00640 1 -- 172.21.15.6:0/125702630 wait complete. 2024-04-24T23:24:31.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:31.146+0000 7f5d7ae00640 1 Processor -- start 2024-04-24T23:24:31.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:31.146+0000 7f5d7ae00640 1 -- start start 2024-04-24T23:24:31.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:31.151+0000 7f5d7ae00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5d58017610 0x7f5d741febd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:24:31.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:31.151+0000 7f5d7ae00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5d740c31b0 0x7f5d741fe1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:24:31.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:31.151+0000 7f5d7ae00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5d74169510 0x7f5d740c54c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:24:31.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:31.156+0000 7f5d7ae00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f5d74071a50 con 0x7f5d740c31b0 2024-04-24T23:24:31.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:31.156+0000 7f5d73400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5d740c31b0 0x7f5d741fe1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:24:31.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:31.159+0000 7f5d7ae00640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f5d74071770 con 0x7f5d58017610 2024-04-24T23:24:31.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:31.159+0000 7f5d7ae00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f5d74071d30 con 0x7f5d74169510 2024-04-24T23:24:31.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:31.159+0000 7f5d73400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5d740c31b0 0x7f5d741fe1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53312/0 (socket says 172.21.15.6:53312) 2024-04-24T23:24:31.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:31.159+0000 7f5d73400640 1 -- 172.21.15.6:0/3191821553 learned_addr learned my addr 172.21.15.6:0/3191821553 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:24:31.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:31.160+0000 7f5d73400640 1 -- 172.21.15.6:0/3191821553 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5d74169510 msgr2=0x7f5d740c54c0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-24T23:24:31.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:31.160+0000 7f5d73400640 1 --2- 172.21.15.6:0/3191821553 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5d74169510 0x7f5d740c54c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:24:31.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:31.160+0000 7f5d73400640 1 -- 172.21.15.6:0/3191821553 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5d58017610 msgr2=0x7f5d741febd0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-24T23:24:31.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:31.160+0000 7f5d73400640 1 --2- 172.21.15.6:0/3191821553 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5d58017610 0x7f5d741febd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:24:31.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:31.160+0000 7f5d73400640 1 -- 172.21.15.6:0/3191821553 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d7416a540 con 0x7f5d740c31b0 2024-04-24T23:24:31.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:31.162+0000 7f5d73400640 1 --2- 172.21.15.6:0/3191821553 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5d740c31b0 0x7f5d741fe1e0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f5d6000c370 tx=0x7f5d60000fe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:24:31.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:31.163+0000 7f5d6be00640 1 -- 172.21.15.6:0/3191821553 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d60005310 con 0x7f5d740c31b0 2024-04-24T23:24:31.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:31.163+0000 7f5d7ae00640 1 -- 172.21.15.6:0/3191821553 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d7416a820 con 0x7f5d740c31b0 2024-04-24T23:24:31.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:31.163+0000 7f5d6be00640 1 -- 172.21.15.6:0/3191821553 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5d60007e50 con 0x7f5d740c31b0 2024-04-24T23:24:31.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:31.164+0000 7f5d7ae00640 1 -- 172.21.15.6:0/3191821553 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d74168d10 con 0x7f5d740c31b0 2024-04-24T23:24:31.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:31.165+0000 7f5d6be00640 1 -- 172.21.15.6:0/3191821553 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d60006a10 con 0x7f5d740c31b0 2024-04-24T23:24:31.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:31.170+0000 7f5d6be00640 1 -- 172.21.15.6:0/3191821553 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f5d60006bb0 con 0x7f5d740c31b0 2024-04-24T23:24:31.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:31.170+0000 7f5d7ae00640 1 -- 172.21.15.6:0/3191821553 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d7408ca50 con 0x7f5d740c31b0 2024-04-24T23:24:31.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:31.171+0000 7f5d6be00640 1 --2- 172.21.15.6:0/3191821553 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f5d5406be30 0x7f5d5406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:24:31.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:31.171+0000 7f5d6be00640 1 -- 172.21.15.6:0/3191821553 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6762+0+0 (secure 0 0 0) 0x7f5d600ab9b0 con 0x7f5d740c31b0 2024-04-24T23:24:31.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:31.177+0000 7f5d73e00640 1 --2- 172.21.15.6:0/3191821553 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f5d5406be30 0x7f5d5406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:24:31.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:31.178+0000 7f5d73e00640 1 --2- 172.21.15.6:0/3191821553 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f5d5406be30 0x7f5d5406e2f0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f5d6c00aa90 tx=0x7f5d6c00a520 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:24:31.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:31.181+0000 7f5d6be00640 1 -- 172.21.15.6:0/3191821553 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f5d7408ca50 con 0x7f5d740c31b0 2024-04-24T23:24:31.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:31.700+0000 7f5d7ae00640 1 -- 172.21.15.6:0/3191821553 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5d740c3590 con 0x7f5d740c31b0 2024-04-24T23:24:31.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:31.702+0000 7f5d6be00640 1 -- 172.21.15.6:0/3191821553 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) ==== 94+0+6852 (secure 0 0 0) 0x7f5d60040070 con 0x7f5d740c31b0 2024-04-24T23:24:31.704 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:24:31.707 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":42,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5081,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/1190759626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1190759626},{"type":"v1","addr":"172.21.15.6:6834","nonce":1190759626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":5249,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6838/2416336040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2416336040},{"type":"v1","addr":"172.21.15.190:6838","nonce":2416336040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":42},{"gid":5503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/3928731059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":3928731059},{"type":"v1","addr":"172.21.15.6:6837","nonce":3928731059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":35},{"gid":5599,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6839/2643233757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":2643233757},{"type":"v1","addr":"172.21.15.190:6839","nonce":2643233757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40}],"filesystems":[{"mdsmap":{"epoch":39,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:23:38.162102+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":4306,"mds_1":4282},"failed":[],"damaged":[],"stopped":[2,3,4],"info":{"gid_4282":{"gid":4282,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":12,"addr":"172.21.15.190:6837/580315269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":580315269},{"type":"v1","addr":"172.21.15.190:6837","nonce":580315269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/489092393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":489092393},{"type":"v1","addr":"172.21.15.6:6835","nonce":489092393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4306,"qdb_cluster":[4306,4282]},"id":1}]} 2024-04-24T23:24:31.707 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 42 2024-04-24T23:24:31.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:31.709+0000 7f5d7ae00640 1 -- 172.21.15.6:0/3191821553 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f5d5406be30 msgr2=0x7f5d5406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:24:31.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:31.709+0000 7f5d7ae00640 1 --2- 172.21.15.6:0/3191821553 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f5d5406be30 0x7f5d5406e2f0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f5d6c00aa90 tx=0x7f5d6c00a520 comp rx=0 tx=0).stop 2024-04-24T23:24:31.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:31.709+0000 7f5d7ae00640 1 -- 172.21.15.6:0/3191821553 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5d740c31b0 msgr2=0x7f5d741fe1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:24:31.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:31.709+0000 7f5d7ae00640 1 --2- 172.21.15.6:0/3191821553 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5d740c31b0 0x7f5d741fe1e0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f5d6000c370 tx=0x7f5d60000fe0 comp rx=0 tx=0).stop 2024-04-24T23:24:31.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:31.711+0000 7f5d7ae00640 1 -- 172.21.15.6:0/3191821553 shutdown_connections 2024-04-24T23:24:31.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:31.711+0000 7f5d7ae00640 1 --2- 172.21.15.6:0/3191821553 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f5d5406be30 0x7f5d5406e2f0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:24:31.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:31.711+0000 7f5d7ae00640 1 --2- 172.21.15.6:0/3191821553 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5d74169510 0x7f5d740c54c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:24:31.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:31.711+0000 7f5d7ae00640 1 --2- 172.21.15.6:0/3191821553 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5d740c31b0 0x7f5d741fe1e0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:24:31.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:31.711+0000 7f5d7ae00640 1 --2- 172.21.15.6:0/3191821553 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5d58017610 0x7f5d741febd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:24:31.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:31.711+0000 7f5d7ae00640 1 -- 172.21.15.6:0/3191821553 >> 172.21.15.6:0/3191821553 conn(0x7f5d7406b0e0 msgr2=0x7f5d7406ed40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:24:31.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:31.711+0000 7f5d7ae00640 1 -- 172.21.15.6:0/3191821553 shutdown_connections 2024-04-24T23:24:31.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:31.714+0000 7f5d7ae00640 1 -- 172.21.15.6:0/3191821553 wait complete. 2024-04-24T23:24:31.775 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2024-04-24T23:24:33.776 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:24:34.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:34.366+0000 7f58b1600640 1 Processor -- start 2024-04-24T23:24:34.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:34.371+0000 7f58b1600640 1 -- start start 2024-04-24T23:24:34.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:34.375+0000 7f58b1600640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f58a40455d0 0x7f58a40df820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:24:34.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:34.376+0000 7f58b1600640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f58a40459b0 0x7f58a4005f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:24:34.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:34.376+0000 7f58b1600640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f58a40dfd60 con 0x7f58a404c630 2024-04-24T23:24:34.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:34.377+0000 7f58b1600640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f58a404b4a0 con 0x7f58a40455d0 2024-04-24T23:24:34.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:34.377+0000 7f58b1600640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f58a4046000 con 0x7f58a40459b0 2024-04-24T23:24:34.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:34.378+0000 7f58aa000640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f58a40459b0 0x7f58a4005f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:24:34.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:34.378+0000 7f58aaa00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f58a40455d0 0x7f58a40df820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:24:34.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:34.378+0000 7f58aaa00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f58a40455d0 0x7f58a40df820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53318/0 (socket says 172.21.15.6:53318) 2024-04-24T23:24:34.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:34.378+0000 7f58aaa00640 1 -- 172.21.15.6:0/611710871 learned_addr learned my addr 172.21.15.6:0/611710871 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:24:34.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:34.379+0000 7f58aaa00640 1 -- 172.21.15.6:0/611710871 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f58a40459b0 msgr2=0x7f58a4005f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:24:34.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:34.380+0000 7f58aaa00640 1 --2- 172.21.15.6:0/611710871 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f58a40459b0 0x7f58a4005f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:24:34.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:34.380+0000 7f58aaa00640 1 -- 172.21.15.6:0/611710871 >> v1:172.21.15.190:6789/0 conn(0x7f58a404c630 legacy=0x7f58a40e2390 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-24T23:24:34.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:34.380+0000 7f58aaa00640 1 -- 172.21.15.6:0/611710871 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58a40462e0 con 0x7f58a40455d0 2024-04-24T23:24:34.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:34.380+0000 7f58aa000640 1 --2- 172.21.15.6:0/611710871 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f58a40459b0 0x7f58a4005f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:24:34.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:34.385+0000 7f58aaa00640 1 --2- 172.21.15.6:0/611710871 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f58a40455d0 0x7f58a40df820 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f589c009ad0 tx=0x7f589c02a470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f043700df5d956a2 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:24:34.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:34.386+0000 7f58a9600640 1 -- 172.21.15.6:0/611710871 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f589c036070 con 0x7f58a40455d0 2024-04-24T23:24:34.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:34.386+0000 7f58a9600640 1 -- 172.21.15.6:0/611710871 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f589c031600 con 0x7f58a40455d0 2024-04-24T23:24:34.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:34.387+0000 7f58a9600640 1 -- 172.21.15.6:0/611710871 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f589c0319d0 con 0x7f58a40455d0 2024-04-24T23:24:34.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:34.388+0000 7f58b1600640 1 -- 172.21.15.6:0/611710871 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f58a40455d0 msgr2=0x7f58a40df820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:24:34.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:34.388+0000 7f58b1600640 1 --2- 172.21.15.6:0/611710871 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f58a40455d0 0x7f58a40df820 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f589c009ad0 tx=0x7f589c02a470 comp rx=0 tx=0).stop 2024-04-24T23:24:34.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:34.389+0000 7f58b1600640 1 -- 172.21.15.6:0/611710871 shutdown_connections 2024-04-24T23:24:34.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:34.389+0000 7f58b1600640 1 --2- 172.21.15.6:0/611710871 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f58a40459b0 0x7f58a4005f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:24:34.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:34.389+0000 7f58b1600640 1 --2- 172.21.15.6:0/611710871 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f58a40455d0 0x7f58a40df820 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:24:34.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:34.389+0000 7f58b1600640 1 -- 172.21.15.6:0/611710871 >> 172.21.15.6:0/611710871 conn(0x7f58a40501b0 msgr2=0x7f58a40505c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:24:34.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:34.391+0000 7f58b1600640 1 -- 172.21.15.6:0/611710871 shutdown_connections 2024-04-24T23:24:34.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:34.391+0000 7f58b1600640 1 -- 172.21.15.6:0/611710871 wait complete. 2024-04-24T23:24:34.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:34.397+0000 7f58b1600640 1 Processor -- start 2024-04-24T23:24:34.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:34.398+0000 7f58b1600640 1 -- start start 2024-04-24T23:24:34.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:34.400+0000 7f58b1600640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f58a40455d0 0x7f58a4049e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:24:34.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:34.401+0000 7f58b1600640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f58a40459b0 0x7f58a404f270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:24:34.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:34.401+0000 7f58aaa00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f58a40455d0 0x7f58a4049e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:24:34.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:34.401+0000 7f58aaa00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f58a40455d0 0x7f58a4049e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.6:49212/0 (socket says 172.21.15.6:49212) 2024-04-24T23:24:34.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:34.402+0000 7f58aaa00640 1 -- 172.21.15.6:0/3088252383 learned_addr learned my addr 172.21.15.6:0/3088252383 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:24:34.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:34.402+0000 7f58b1600640 1 --2- 172.21.15.6:0/3088252383 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f58a404c630 0x7f58a40d5be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:24:34.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:34.403+0000 7f58b1600640 1 -- 172.21.15.6:0/3088252383 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f58a4006450 con 0x7f58a404c630 2024-04-24T23:24:34.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:34.403+0000 7f58b1600640 1 -- 172.21.15.6:0/3088252383 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f58a40468a0 con 0x7f58a40459b0 2024-04-24T23:24:34.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:34.403+0000 7f58b1600640 1 -- 172.21.15.6:0/3088252383 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f58a404b0c0 con 0x7f58a40455d0 2024-04-24T23:24:34.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:34.403+0000 7f58aa000640 1 --2- 172.21.15.6:0/3088252383 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f58a40459b0 0x7f58a404f270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:24:34.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:34.404+0000 7f58ab400640 1 --2- 172.21.15.6:0/3088252383 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f58a404c630 0x7f58a40d5be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:24:34.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:34.404+0000 7f58aaa00640 1 -- 172.21.15.6:0/3088252383 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f58a40459b0 msgr2=0x7f58a404f270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:24:34.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:34.404+0000 7f58aaa00640 1 --2- 172.21.15.6:0/3088252383 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f58a40459b0 0x7f58a404f270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:24:34.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:34.405+0000 7f58aaa00640 1 -- 172.21.15.6:0/3088252383 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f58a404c630 msgr2=0x7f58a40d5be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:24:34.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:34.405+0000 7f58aaa00640 1 --2- 172.21.15.6:0/3088252383 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f58a404c630 0x7f58a40d5be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:24:34.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:34.405+0000 7f58aaa00640 1 -- 172.21.15.6:0/3088252383 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58a41790d0 con 0x7f58a40455d0 2024-04-24T23:24:34.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:34.405+0000 7f58ab400640 1 --2- 172.21.15.6:0/3088252383 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f58a404c630 0x7f58a40d5be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:24:34.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:34.409+0000 7f58aaa00640 1 --2- 172.21.15.6:0/3088252383 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f58a40455d0 0x7f58a4049e50 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f589c009bf0 tx=0x7f589c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:24:34.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:34.409+0000 7f58a2a00640 1 -- 172.21.15.6:0/3088252383 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f589c041020 con 0x7f58a40455d0 2024-04-24T23:24:34.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:34.409+0000 7f58b1600640 1 -- 172.21.15.6:0/3088252383 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58a41793b0 con 0x7f58a40455d0 2024-04-24T23:24:34.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:34.410+0000 7f58b1600640 1 -- 172.21.15.6:0/3088252383 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f58a4179970 con 0x7f58a40455d0 2024-04-24T23:24:34.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:34.410+0000 7f58a2a00640 1 -- 172.21.15.6:0/3088252383 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f589c031a70 con 0x7f58a40455d0 2024-04-24T23:24:34.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:34.411+0000 7f58a2a00640 1 -- 172.21.15.6:0/3088252383 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f589c041020 con 0x7f58a40455d0 2024-04-24T23:24:34.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:34.414+0000 7f58a2a00640 1 -- 172.21.15.6:0/3088252383 <== mon.2 v2:172.21.15.190:3301/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f589c044020 con 0x7f58a40455d0 2024-04-24T23:24:34.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:34.416+0000 7f58a2a00640 1 --2- 172.21.15.6:0/3088252383 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f588c06bff0 0x7f588c06e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:24:34.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:34.416+0000 7f58a2a00640 1 -- 172.21.15.6:0/3088252383 <== mon.2 v2:172.21.15.190:3301/0 5 ==== osd_map(29..29 src has 1..29) ==== 6848+0+0 (secure 0 0 0) 0x7f589c0ab610 con 0x7f58a40455d0 2024-04-24T23:24:34.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:34.417+0000 7f58b1600640 1 -- 172.21.15.6:0/3088252383 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5884005290 con 0x7f58a40455d0 2024-04-24T23:24:34.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:34.420+0000 7f58aa000640 1 --2- 172.21.15.6:0/3088252383 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f588c06bff0 0x7f588c06e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:24:34.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:34.426+0000 7f58aa000640 1 --2- 172.21.15.6:0/3088252383 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f588c06bff0 0x7f588c06e4b0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f58a40cb140 tx=0x7f5898008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:24:34.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:34.432+0000 7f58a2a00640 1 -- 172.21.15.6:0/3088252383 <== mon.2 v2:172.21.15.190:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f589c04b030 con 0x7f58a40455d0 2024-04-24T23:24:34.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:34.879+0000 7f58b1600640 1 -- 172.21.15.6:0/3088252383 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5884005850 con 0x7f58a40455d0 2024-04-24T23:24:34.881 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:24:34.881 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":43,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5081,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/1190759626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1190759626},{"type":"v1","addr":"172.21.15.6:6834","nonce":1190759626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":5249,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6838/2416336040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2416336040},{"type":"v1","addr":"172.21.15.190:6838","nonce":2416336040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":42},{"gid":5503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/3928731059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":3928731059},{"type":"v1","addr":"172.21.15.6:6837","nonce":3928731059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":43,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:24:32.678603+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":4306,"mds_1":5599},"failed":[],"damaged":[],"stopped":[2,3,4],"info":{"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/489092393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":489092393},{"type":"v1","addr":"172.21.15.6:6835","nonce":489092393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5599":{"gid":5599,"name":"d","rank":1,"incarnation":43,"state":"up:replay","state_seq":1,"addr":"172.21.15.190:6839/2643233757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":2643233757},{"type":"v1","addr":"172.21.15.190:6839","nonce":2643233757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4306,"qdb_cluster":[4306]},"id":1}]} 2024-04-24T23:24:34.881 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 43 2024-04-24T23:24:34.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:34.879+0000 7f58a2a00640 1 -- 172.21.15.6:0/3088252383 <== mon.2 v2:172.21.15.190:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) ==== 94+0+6022 (secure 0 0 0) 0x7f58a40dfd60 con 0x7f58a40455d0 2024-04-24T23:24:34.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:34.882+0000 7f58b1600640 1 -- 172.21.15.6:0/3088252383 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f588c06bff0 msgr2=0x7f588c06e4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:24:34.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:34.882+0000 7f58b1600640 1 --2- 172.21.15.6:0/3088252383 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f588c06bff0 0x7f588c06e4b0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f58a40cb140 tx=0x7f5898008040 comp rx=0 tx=0).stop 2024-04-24T23:24:34.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:34.882+0000 7f58b1600640 1 -- 172.21.15.6:0/3088252383 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f58a40455d0 msgr2=0x7f58a4049e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:24:34.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:34.882+0000 7f58b1600640 1 --2- 172.21.15.6:0/3088252383 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f58a40455d0 0x7f58a4049e50 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f589c009bf0 tx=0x7f589c0046c0 comp rx=0 tx=0).stop 2024-04-24T23:24:34.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:34.882+0000 7f58b1600640 1 -- 172.21.15.6:0/3088252383 shutdown_connections 2024-04-24T23:24:34.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:34.882+0000 7f58b1600640 1 --2- 172.21.15.6:0/3088252383 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f588c06bff0 0x7f588c06e4b0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:24:34.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:34.882+0000 7f58b1600640 1 --2- 172.21.15.6:0/3088252383 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f58a404c630 0x7f58a40d5be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:24:34.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:34.882+0000 7f58b1600640 1 --2- 172.21.15.6:0/3088252383 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f58a40459b0 0x7f58a404f270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:24:34.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:34.882+0000 7f58b1600640 1 --2- 172.21.15.6:0/3088252383 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f58a40455d0 0x7f58a4049e50 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:24:34.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:34.882+0000 7f58b1600640 1 -- 172.21.15.6:0/3088252383 >> 172.21.15.6:0/3088252383 conn(0x7f58a40501b0 msgr2=0x7f58a4005500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:24:34.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:34.882+0000 7f58b1600640 1 -- 172.21.15.6:0/3088252383 shutdown_connections 2024-04-24T23:24:34.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:34.882+0000 7f58b1600640 1 -- 172.21.15.6:0/3088252383 wait complete. 2024-04-24T23:24:34.941 INFO:tasks.mds_thrash.fs.[cephfs]:mds.f down, removed from mdsmap 2024-04-24T23:24:34.941 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2024-04-24T23:24:34.941 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:24:35.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:35.241+0000 7f6902c00640 1 Processor -- start 2024-04-24T23:24:35.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:35.241+0000 7f6902c00640 1 -- start start 2024-04-24T23:24:35.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:35.243+0000 7f6902c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f68fc159110 0x7f68fc072420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:24:35.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:35.243+0000 7f6902c00640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f68fc0c81c0 0x7f68fc162ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:24:35.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:35.243+0000 7f6902c00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f68fc0c7230 con 0x7f68fc0c7dc0 2024-04-24T23:24:35.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:35.243+0000 7f6902c00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f68fc1594f0 con 0x7f68fc159110 2024-04-24T23:24:35.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:35.243+0000 7f6902c00640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f68fc0c74d0 con 0x7f68fc0c81c0 2024-04-24T23:24:35.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:35.244+0000 7f68fbe00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f68fc159110 0x7f68fc072420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:24:35.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:35.244+0000 7f68fb400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f68fc0c7dc0 0x7f68fc1623a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:50052/0 (socket says 172.21.15.6:50052) 2024-04-24T23:24:35.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:35.244+0000 7f68fb400640 1 -- 172.21.15.6:0/2186800893 learned_addr learned my addr 172.21.15.6:0/2186800893 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:24:35.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:35.244+0000 7f6900c00640 1 --2- 172.21.15.6:0/2186800893 >> v2:172.21.15.190:3300/0 conn(0x7f68fc0c81c0 0x7f68fc162ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:24:35.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:35.245+0000 7f68fbe00640 1 -- 172.21.15.6:0/2186800893 >> v2:172.21.15.190:3300/0 conn(0x7f68fc0c81c0 msgr2=0x7f68fc162ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:24:35.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:35.245+0000 7f68fbe00640 1 --2- 172.21.15.6:0/2186800893 >> v2:172.21.15.190:3300/0 conn(0x7f68fc0c81c0 0x7f68fc162ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:24:35.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:35.245+0000 7f68fbe00640 1 -- 172.21.15.6:0/2186800893 >> v1:172.21.15.6:6789/0 conn(0x7f68fc0c7dc0 legacy=0x7f68fc1623a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:24:35.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:35.245+0000 7f68faa00640 1 -- 172.21.15.6:0/2186800893 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1062115241 0 0) 0x7f68fc0c7230 con 0x7f68fc0c7dc0 2024-04-24T23:24:35.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:35.245+0000 7f68fbe00640 1 -- 172.21.15.6:0/2186800893 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68fc06e440 con 0x7f68fc159110 2024-04-24T23:24:35.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:35.246+0000 7f68fbe00640 1 --2- 172.21.15.6:0/2186800893 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f68fc159110 0x7f68fc072420 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f68e80097a0 tx=0x7f68e802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ae7b044762db734 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:24:35.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:35.246+0000 7f68faa00640 1 -- 172.21.15.6:0/2186800893 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68e8037070 con 0x7f68fc159110 2024-04-24T23:24:35.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:35.246+0000 7f68faa00640 1 -- 172.21.15.6:0/2186800893 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f68e8032440 con 0x7f68fc159110 2024-04-24T23:24:35.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:35.246+0000 7f68faa00640 1 -- 172.21.15.6:0/2186800893 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68e8032810 con 0x7f68fc159110 2024-04-24T23:24:35.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:35.247+0000 7f6902c00640 1 -- 172.21.15.6:0/2186800893 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f68fc159110 msgr2=0x7f68fc072420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:24:35.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:35.247+0000 7f6902c00640 1 --2- 172.21.15.6:0/2186800893 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f68fc159110 0x7f68fc072420 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f68e80097a0 tx=0x7f68e802a270 comp rx=0 tx=0).stop 2024-04-24T23:24:35.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:35.248+0000 7f6902c00640 1 -- 172.21.15.6:0/2186800893 shutdown_connections 2024-04-24T23:24:35.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:35.248+0000 7f6902c00640 1 --2- 172.21.15.6:0/2186800893 >> v2:172.21.15.190:3300/0 conn(0x7f68fc0c81c0 0x7f68fc162ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:24:35.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:35.248+0000 7f6902c00640 1 --2- 172.21.15.6:0/2186800893 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f68fc159110 0x7f68fc072420 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:24:35.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:35.248+0000 7f6902c00640 1 -- 172.21.15.6:0/2186800893 >> 172.21.15.6:0/2186800893 conn(0x7f68fc06b0e0 msgr2=0x7f68fc06b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:24:35.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:35.248+0000 7f6902c00640 1 -- 172.21.15.6:0/2186800893 shutdown_connections 2024-04-24T23:24:35.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:35.248+0000 7f6902c00640 1 -- 172.21.15.6:0/2186800893 wait complete. 2024-04-24T23:24:35.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:35.251+0000 7f6902c00640 1 Processor -- start 2024-04-24T23:24:35.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:35.251+0000 7f6902c00640 1 -- start start 2024-04-24T23:24:35.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:35.253+0000 7f6902c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f68fc0c7dc0 0x7f68fc1f5ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:24:35.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:35.253+0000 7f6902c00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f68fc0c81c0 0x7f68fc15ae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:24:35.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:35.253+0000 7f6902c00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f68fc159110 0x7f68fc1f8660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:24:35.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:35.253+0000 7f6902c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f68fc06ece0 con 0x7f68fc0c7dc0 2024-04-24T23:24:35.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:35.253+0000 7f6902c00640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f68fc06ea00 con 0x7f68fc0c81c0 2024-04-24T23:24:35.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:35.253+0000 7f6902c00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f68fc072960 con 0x7f68fc159110 2024-04-24T23:24:35.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:35.253+0000 7f68fb400640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f68fc0c81c0 0x7f68fc15ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:24:35.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:35.253+0000 7f68fb400640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f68fc0c81c0 0x7f68fc15ae60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:34438/0 (socket says 172.21.15.6:34438) 2024-04-24T23:24:35.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:35.253+0000 7f68fb400640 1 -- 172.21.15.6:0/1761883076 learned_addr learned my addr 172.21.15.6:0/1761883076 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:24:35.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:35.254+0000 7f6900c00640 1 --2- 172.21.15.6:0/1761883076 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f68fc159110 0x7f68fc1f8660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:24:35.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:35.254+0000 7f68fb400640 1 -- 172.21.15.6:0/1761883076 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f68fc159110 msgr2=0x7f68fc1f8660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:24:35.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:35.254+0000 7f68fb400640 1 --2- 172.21.15.6:0/1761883076 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f68fc159110 0x7f68fc1f8660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:24:35.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:35.254+0000 7f68fb400640 1 -- 172.21.15.6:0/1761883076 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f68fc0c7dc0 msgr2=0x7f68fc1f5ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:24:35.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:35.254+0000 7f68fb400640 1 --2- 172.21.15.6:0/1761883076 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f68fc0c7dc0 0x7f68fc1f5ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:24:35.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:35.254+0000 7f68fb400640 1 -- 172.21.15.6:0/1761883076 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68fc1f8e80 con 0x7f68fc0c81c0 2024-04-24T23:24:35.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:35.255+0000 7f68fb400640 1 --2- 172.21.15.6:0/1761883076 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f68fc0c81c0 0x7f68fc15ae60 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f68ec00c490 tx=0x7f68ec00c950 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:24:35.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:35.257+0000 7f68f3e00640 1 -- 172.21.15.6:0/1761883076 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68ec005ae0 con 0x7f68fc0c81c0 2024-04-24T23:24:35.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:35.258+0000 7f68f3e00640 1 -- 172.21.15.6:0/1761883076 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f68ec005c80 con 0x7f68fc0c81c0 2024-04-24T23:24:35.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:35.258+0000 7f68f3e00640 1 -- 172.21.15.6:0/1761883076 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68ec005ae0 con 0x7f68fc0c81c0 2024-04-24T23:24:35.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:35.258+0000 7f6902c00640 1 -- 172.21.15.6:0/1761883076 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68fc1f9160 con 0x7f68fc0c81c0 2024-04-24T23:24:35.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:35.258+0000 7f6902c00640 1 -- 172.21.15.6:0/1761883076 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68fc1f54e0 con 0x7f68fc0c81c0 2024-04-24T23:24:35.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:35.261+0000 7f6902c00640 1 -- 172.21.15.6:0/1761883076 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f68fc0716d0 con 0x7f68fc0c81c0 2024-04-24T23:24:35.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:35.267+0000 7f68f3e00640 1 -- 172.21.15.6:0/1761883076 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f68ec007070 con 0x7f68fc0c81c0 2024-04-24T23:24:35.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:35.278+0000 7f68f3e00640 1 --2- 172.21.15.6:0/1761883076 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f68dc06c030 0x7f68dc06e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:24:35.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:35.279+0000 7f68f3e00640 1 -- 172.21.15.6:0/1761883076 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6848+0+0 (secure 0 0 0) 0x7f68ec09bc80 con 0x7f68fc0c81c0 2024-04-24T23:24:35.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:35.279+0000 7f68f3e00640 1 -- 172.21.15.6:0/1761883076 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f68fc0716d0 con 0x7f68fc0c81c0 2024-04-24T23:24:35.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:35.283+0000 7f68fbe00640 1 --2- 172.21.15.6:0/1761883076 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f68dc06c030 0x7f68dc06e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:24:35.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:35.288+0000 7f68fbe00640 1 --2- 172.21.15.6:0/1761883076 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f68dc06c030 0x7f68dc06e4f0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f68e8034000 tx=0x7f68e802c000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:24:35.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:35.812+0000 7f6902c00640 1 -- 172.21.15.6:0/1761883076 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f68fc0767b0 con 0x7f68fc0c81c0 2024-04-24T23:24:35.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:35.816+0000 7f68f3e00640 1 -- 172.21.15.6:0/1761883076 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) ==== 94+0+6022 (secure 0 0 0) 0x7f68ec06a130 con 0x7f68fc0c81c0 2024-04-24T23:24:35.819 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:24:35.822 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":43,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5081,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/1190759626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1190759626},{"type":"v1","addr":"172.21.15.6:6834","nonce":1190759626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":5249,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6838/2416336040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2416336040},{"type":"v1","addr":"172.21.15.190:6838","nonce":2416336040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":42},{"gid":5503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/3928731059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":3928731059},{"type":"v1","addr":"172.21.15.6:6837","nonce":3928731059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":43,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:24:32.678603+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":4306,"mds_1":5599},"failed":[],"damaged":[],"stopped":[2,3,4],"info":{"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/489092393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":489092393},{"type":"v1","addr":"172.21.15.6:6835","nonce":489092393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5599":{"gid":5599,"name":"d","rank":1,"incarnation":43,"state":"up:replay","state_seq":1,"addr":"172.21.15.190:6839/2643233757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":2643233757},{"type":"v1","addr":"172.21.15.190:6839","nonce":2643233757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4306,"qdb_cluster":[4306]},"id":1}]} 2024-04-24T23:24:35.822 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 43 2024-04-24T23:24:35.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:35.826+0000 7f6902c00640 1 -- 172.21.15.6:0/1761883076 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f68dc06c030 msgr2=0x7f68dc06e4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:24:35.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:35.826+0000 7f6902c00640 1 --2- 172.21.15.6:0/1761883076 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f68dc06c030 0x7f68dc06e4f0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f68e8034000 tx=0x7f68e802c000 comp rx=0 tx=0).stop 2024-04-24T23:24:35.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:35.826+0000 7f6902c00640 1 -- 172.21.15.6:0/1761883076 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f68fc0c81c0 msgr2=0x7f68fc15ae60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:24:35.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:35.826+0000 7f6902c00640 1 --2- 172.21.15.6:0/1761883076 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f68fc0c81c0 0x7f68fc15ae60 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f68ec00c490 tx=0x7f68ec00c950 comp rx=0 tx=0).stop 2024-04-24T23:24:35.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:35.828+0000 7f6902c00640 1 -- 172.21.15.6:0/1761883076 shutdown_connections 2024-04-24T23:24:35.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:35.828+0000 7f6902c00640 1 --2- 172.21.15.6:0/1761883076 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f68dc06c030 0x7f68dc06e4f0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:24:35.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:35.828+0000 7f6902c00640 1 --2- 172.21.15.6:0/1761883076 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f68fc159110 0x7f68fc1f8660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:24:35.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:35.828+0000 7f6902c00640 1 --2- 172.21.15.6:0/1761883076 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f68fc0c81c0 0x7f68fc15ae60 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:24:35.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:35.828+0000 7f6902c00640 1 --2- 172.21.15.6:0/1761883076 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f68fc0c7dc0 0x7f68fc1f5ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:24:35.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:35.828+0000 7f6902c00640 1 -- 172.21.15.6:0/1761883076 >> 172.21.15.6:0/1761883076 conn(0x7f68fc06b0e0 msgr2=0x7f68fc06b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:24:35.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:35.828+0000 7f6902c00640 1 -- 172.21.15.6:0/1761883076 shutdown_connections 2024-04-24T23:24:35.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:35.828+0000 7f6902c00640 1 -- 172.21.15.6:0/1761883076 wait complete. 2024-04-24T23:24:35.886 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 43, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-24T23:15:36.400716+0000", "damaged": [], "data_pools": [ 3, 4 ], "enabled": true, "epoch": 43, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_4306": { "addr": "172.21.15.6:6835/489092393", "addrs": { "addrvec": [ { "addr": "172.21.15.6:6833", "nonce": 489092393, "type": "v2" }, { "addr": "172.21.15.6:6835", "nonce": 489092393, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4306, "incarnation": 4, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 4 }, "gid_5599": { "addr": "172.21.15.190:6839/2643233757", "addrs": { "addrvec": [ { "addr": "172.21.15.190:6836", "nonce": 2643233757, "type": "v2" }, { "addr": "172.21.15.190:6839", "nonce": 2643233757, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5599, "incarnation": 43, "join_fscid": -1, "name": "d", "rank": 1, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 29, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-04-24T23:24:32.678603+0000", "qdb_cluster": [ 4306 ], "qdb_leader": 4306, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 2, 3, 4 ], "tableserver": 0, "up": { "mds_0": 4306, "mds_1": 5599 } } } ], "standbys": [ { "addr": "172.21.15.6:6834/1190759626", "addrs": { "addrvec": [ { "addr": "172.21.15.6:6832", "nonce": 1190759626, "type": "v2" }, { "addr": "172.21.15.6:6834", "nonce": 1190759626, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 30, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5081, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.190:6838/2416336040", "addrs": { "addrvec": [ { "addr": "172.21.15.190:6835", "nonce": 2416336040, "type": "v2" }, { "addr": "172.21.15.190:6838", "nonce": 2416336040, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 42, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5249, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.6:6837/3928731059", "addrs": { "addrvec": [ { "addr": "172.21.15.6:6836", "nonce": 3928731059, "type": "v2" }, { "addr": "172.21.15.6:6837", "nonce": 3928731059, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 35, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5503, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-04-24T23:24:37.887 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:24:38.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:38.846+0000 7f529d200640 1 Processor -- start 2024-04-24T23:24:38.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:38.846+0000 7f529d200640 1 -- start start 2024-04-24T23:24:38.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:38.848+0000 7f529d200640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f52980cb090 0x7f52980d0820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:24:38.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:38.848+0000 7f529d200640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f52980d0d60 0x7f529815e000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:24:38.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:38.849+0000 7f5297e00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f52980d0d60 0x7f529815e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:24:38.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:38.849+0000 7f5297e00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f52980d0d60 0x7f529815e000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.6:58330/0 (socket says 172.21.15.6:58330) 2024-04-24T23:24:38.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:38.849+0000 7f5297e00640 1 -- 172.21.15.6:0/374160272 learned_addr learned my addr 172.21.15.6:0/374160272 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:24:38.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:38.851+0000 7f5296a00640 1 --2- 172.21.15.6:0/374160272 >> v2:172.21.15.190:3300/0 conn(0x7f52980cb090 0x7f52980d0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:24:38.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:38.856+0000 7f529d200640 1 -- 172.21.15.6:0/374160272 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5298155170 con 0x7f5298154d90 2024-04-24T23:24:38.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:38.856+0000 7f529d200640 1 -- 172.21.15.6:0/374160272 --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f52980cc9a0 con 0x7f52980cb090 2024-04-24T23:24:38.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:38.856+0000 7f529d200640 1 -- 172.21.15.6:0/374160272 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f529815ad60 con 0x7f52980d0d60 2024-04-24T23:24:38.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:38.858+0000 7f5297e00640 1 -- 172.21.15.6:0/374160272 >> v2:172.21.15.190:3300/0 conn(0x7f52980cb090 msgr2=0x7f52980d0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:24:38.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:38.858+0000 7f5297e00640 1 --2- 172.21.15.6:0/374160272 >> v2:172.21.15.190:3300/0 conn(0x7f52980cb090 0x7f52980d0820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:24:38.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:38.858+0000 7f5297e00640 1 -- 172.21.15.6:0/374160272 >> v1:172.21.15.6:6789/0 conn(0x7f5298154d90 legacy=0x7f52980cbe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:24:38.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:38.858+0000 7f5297e00640 1 -- 172.21.15.6:0/374160272 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f529815b040 con 0x7f52980d0d60 2024-04-24T23:24:38.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:38.859+0000 7f5297e00640 1 --2- 172.21.15.6:0/374160272 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f52980d0d60 0x7f529815e000 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f5288009870 tx=0x7f528802a2a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2e4cf86a455ebf6e server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:24:38.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:38.860+0000 7f5296000640 1 -- 172.21.15.6:0/374160272 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5288037070 con 0x7f52980d0d60 2024-04-24T23:24:38.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:38.862+0000 7f5296000640 1 -- 172.21.15.6:0/374160272 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5288032440 con 0x7f52980d0d60 2024-04-24T23:24:38.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:38.862+0000 7f5296000640 1 -- 172.21.15.6:0/374160272 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5288032810 con 0x7f52980d0d60 2024-04-24T23:24:38.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:38.863+0000 7f529d200640 1 -- 172.21.15.6:0/374160272 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f52980d0d60 msgr2=0x7f529815e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:24:38.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:38.863+0000 7f529d200640 1 --2- 172.21.15.6:0/374160272 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f52980d0d60 0x7f529815e000 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f5288009870 tx=0x7f528802a2a0 comp rx=0 tx=0).stop 2024-04-24T23:24:38.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:38.864+0000 7f529d200640 1 -- 172.21.15.6:0/374160272 shutdown_connections 2024-04-24T23:24:38.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:38.864+0000 7f529d200640 1 --2- 172.21.15.6:0/374160272 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f52980d0d60 0x7f529815e000 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:24:38.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:38.864+0000 7f529d200640 1 --2- 172.21.15.6:0/374160272 >> v2:172.21.15.190:3300/0 conn(0x7f52980cb090 0x7f52980d0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:24:38.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:38.864+0000 7f529d200640 1 -- 172.21.15.6:0/374160272 >> 172.21.15.6:0/374160272 conn(0x7f529806be10 msgr2=0x7f529806c1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:24:38.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:38.866+0000 7f529d200640 1 -- 172.21.15.6:0/374160272 shutdown_connections 2024-04-24T23:24:38.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:38.869+0000 7f529d200640 1 -- 172.21.15.6:0/374160272 wait complete. 2024-04-24T23:24:38.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:38.873+0000 7f529d200640 1 Processor -- start 2024-04-24T23:24:38.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:38.873+0000 7f529d200640 1 -- start start 2024-04-24T23:24:38.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:38.874+0000 7f529d200640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f52980cb090 0x7f529814a070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:24:38.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:38.874+0000 7f529d200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f52980d0d60 0x7f5298152530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:24:38.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:38.874+0000 7f529d200640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5298154d90 0x7f5298152a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:24:38.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:38.874+0000 7f529d200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f529815aa30 con 0x7f52980d0d60 2024-04-24T23:24:38.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:38.874+0000 7f529d200640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f529815b600 con 0x7f52980cb090 2024-04-24T23:24:38.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:38.874+0000 7f529d200640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f52980cc590 con 0x7f5298154d90 2024-04-24T23:24:38.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:38.876+0000 7f5297400640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f52980cb090 0x7f529814a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:24:38.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:38.876+0000 7f5297400640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f52980cb090 0x7f529814a070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:34452/0 (socket says 172.21.15.6:34452) 2024-04-24T23:24:38.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:38.876+0000 7f5297400640 1 -- 172.21.15.6:0/2788865641 learned_addr learned my addr 172.21.15.6:0/2788865641 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:24:38.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:38.877+0000 7f5297e00640 1 --2- 172.21.15.6:0/2788865641 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5298154d90 0x7f5298152a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:24:38.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:38.877+0000 7f5297400640 1 -- 172.21.15.6:0/2788865641 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5298154d90 msgr2=0x7f5298152a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:24:38.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:38.877+0000 7f5297400640 1 --2- 172.21.15.6:0/2788865641 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5298154d90 0x7f5298152a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:24:38.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:38.877+0000 7f5297400640 1 -- 172.21.15.6:0/2788865641 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f52980d0d60 msgr2=0x7f5298152530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:24:38.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:38.877+0000 7f5297400640 1 --2- 172.21.15.6:0/2788865641 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f52980d0d60 0x7f5298152530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:24:38.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:38.878+0000 7f5297400640 1 -- 172.21.15.6:0/2788865641 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52980ce2b0 con 0x7f52980cb090 2024-04-24T23:24:38.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:38.878+0000 7f5297400640 1 --2- 172.21.15.6:0/2788865641 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f52980cb090 0x7f529814a070 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f528400c430 tx=0x7f528400c8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:24:38.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:38.878+0000 7f528f400640 1 -- 172.21.15.6:0/2788865641 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5284011790 con 0x7f52980cb090 2024-04-24T23:24:38.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:38.879+0000 7f529d200640 1 -- 172.21.15.6:0/2788865641 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52980ce590 con 0x7f52980cb090 2024-04-24T23:24:38.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:38.879+0000 7f529d200640 1 -- 172.21.15.6:0/2788865641 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52980cff50 con 0x7f52980cb090 2024-04-24T23:24:38.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:38.879+0000 7f528f400640 1 -- 172.21.15.6:0/2788865641 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5284017070 con 0x7f52980cb090 2024-04-24T23:24:38.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:38.880+0000 7f528f400640 1 -- 172.21.15.6:0/2788865641 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5284011bb0 con 0x7f52980cb090 2024-04-24T23:24:38.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:38.885+0000 7f529d200640 1 -- 172.21.15.6:0/2788865641 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5270005290 con 0x7f52980cb090 2024-04-24T23:24:38.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:38.885+0000 7f528f400640 1 -- 172.21.15.6:0/2788865641 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f528401c460 con 0x7f52980cb090 2024-04-24T23:24:38.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:38.887+0000 7f528f400640 1 --2- 172.21.15.6:0/2788865641 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f527806be30 0x7f527806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:24:38.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:38.887+0000 7f528f400640 1 -- 172.21.15.6:0/2788865641 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6848+0+0 (secure 0 0 0) 0x7f528408b1a0 con 0x7f52980cb090 2024-04-24T23:24:38.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:38.888+0000 7f5296a00640 1 --2- 172.21.15.6:0/2788865641 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f527806be30 0x7f527806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:24:38.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:38.891+0000 7f5296a00640 1 --2- 172.21.15.6:0/2788865641 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f527806be30 0x7f527806e2f0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f52980cb7d0 tx=0x7f529000b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:24:38.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:38.896+0000 7f528f400640 1 -- 172.21.15.6:0/2788865641 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f52840596f0 con 0x7f52980cb090 2024-04-24T23:24:39.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:39.511+0000 7f529d200640 1 -- 172.21.15.6:0/2788865641 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5270005580 con 0x7f52980cb090 2024-04-24T23:24:39.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:39.514+0000 7f528f400640 1 -- 172.21.15.6:0/2788865641 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) ==== 94+0+6022 (secure 0 0 0) 0x7f5284058ff0 con 0x7f52980cb090 2024-04-24T23:24:39.516 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:24:39.519 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":43,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5081,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/1190759626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1190759626},{"type":"v1","addr":"172.21.15.6:6834","nonce":1190759626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":5249,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6838/2416336040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2416336040},{"type":"v1","addr":"172.21.15.190:6838","nonce":2416336040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":42},{"gid":5503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/3928731059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":3928731059},{"type":"v1","addr":"172.21.15.6:6837","nonce":3928731059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":43,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:24:32.678603+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":4306,"mds_1":5599},"failed":[],"damaged":[],"stopped":[2,3,4],"info":{"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/489092393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":489092393},{"type":"v1","addr":"172.21.15.6:6835","nonce":489092393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5599":{"gid":5599,"name":"d","rank":1,"incarnation":43,"state":"up:replay","state_seq":1,"addr":"172.21.15.190:6839/2643233757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":2643233757},{"type":"v1","addr":"172.21.15.190:6839","nonce":2643233757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4306,"qdb_cluster":[4306]},"id":1}]} 2024-04-24T23:24:39.519 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 43 2024-04-24T23:24:39.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:39.518+0000 7f529d200640 1 -- 172.21.15.6:0/2788865641 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f527806be30 msgr2=0x7f527806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:24:39.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:39.518+0000 7f529d200640 1 --2- 172.21.15.6:0/2788865641 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f527806be30 0x7f527806e2f0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f52980cb7d0 tx=0x7f529000b040 comp rx=0 tx=0).stop 2024-04-24T23:24:39.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:39.518+0000 7f529d200640 1 -- 172.21.15.6:0/2788865641 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f52980cb090 msgr2=0x7f529814a070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:24:39.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:39.518+0000 7f529d200640 1 --2- 172.21.15.6:0/2788865641 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f52980cb090 0x7f529814a070 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f528400c430 tx=0x7f528400c8f0 comp rx=0 tx=0).stop 2024-04-24T23:24:39.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:39.518+0000 7f529d200640 1 -- 172.21.15.6:0/2788865641 shutdown_connections 2024-04-24T23:24:39.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:39.518+0000 7f529d200640 1 --2- 172.21.15.6:0/2788865641 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f527806be30 0x7f527806e2f0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:24:39.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:39.518+0000 7f529d200640 1 --2- 172.21.15.6:0/2788865641 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5298154d90 0x7f5298152a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:24:39.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:39.518+0000 7f529d200640 1 --2- 172.21.15.6:0/2788865641 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f52980d0d60 0x7f5298152530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:24:39.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:39.518+0000 7f529d200640 1 --2- 172.21.15.6:0/2788865641 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f52980cb090 0x7f529814a070 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:24:39.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:39.519+0000 7f529d200640 1 -- 172.21.15.6:0/2788865641 >> 172.21.15.6:0/2788865641 conn(0x7f529806be10 msgr2=0x7f52980b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:24:39.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:39.519+0000 7f529d200640 1 -- 172.21.15.6:0/2788865641 shutdown_connections 2024-04-24T23:24:39.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:39.519+0000 7f529d200640 1 -- 172.21.15.6:0/2788865641 wait complete. 2024-04-24T23:24:39.552 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-24T23:24:41.553 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:24:42.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:41.936+0000 7f1993e00640 1 Processor -- start 2024-04-24T23:24:42.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:41.936+0000 7f1993e00640 1 -- start start 2024-04-24T23:24:42.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:41.938+0000 7f1993e00640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f199416b5f0 0x7f1994167ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:24:42.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:41.938+0000 7f1993e00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1994072370 con 0x7f19940ccc50 2024-04-24T23:24:42.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:41.938+0000 7f1993e00640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1994072650 con 0x7f19940cc870 2024-04-24T23:24:42.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:41.938+0000 7f1993e00640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f1994070e70 con 0x7f199416b5f0 2024-04-24T23:24:42.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:41.943+0000 7f1992a00640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f19940cc870 0x7f199406ddd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.6:51618/0 (socket says 172.21.15.6:51618) 2024-04-24T23:24:42.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:41.943+0000 7f1992a00640 1 -- 172.21.15.6:0/139278418 learned_addr learned my addr 172.21.15.6:0/139278418 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:24:42.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:41.945+0000 7f1993400640 1 --2- 172.21.15.6:0/139278418 >> v2:172.21.15.190:3300/0 conn(0x7f199416b5f0 0x7f1994167ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:24:42.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:41.948+0000 7f1991600640 1 -- 172.21.15.6:0/139278418 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1188156250 0 0) 0x7f1994072370 con 0x7f19940ccc50 2024-04-24T23:24:42.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:42.458+0000 7f1991600640 1 -- 172.21.15.6:0/139278418 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1978003500 con 0x7f19940ccc50 2024-04-24T23:24:42.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:42.458+0000 7f1991600640 1 -- 172.21.15.6:0/139278418 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 485706566 0 0) 0x7f1994072650 con 0x7f19940cc870 2024-04-24T23:24:42.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:42.458+0000 7f1991600640 1 -- 172.21.15.6:0/139278418 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1994072370 con 0x7f19940cc870 2024-04-24T23:24:42.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:42.458+0000 7f1991600640 1 -- 172.21.15.6:0/139278418 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2320127260 0 0) 0x7f1978003500 con 0x7f19940ccc50 2024-04-24T23:24:42.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:42.458+0000 7f1991600640 1 -- 172.21.15.6:0/139278418 >> v2:172.21.15.190:3300/0 conn(0x7f199416b5f0 msgr2=0x7f1994167ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:24:42.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:42.458+0000 7f1991600640 1 --2- 172.21.15.6:0/139278418 >> v2:172.21.15.190:3300/0 conn(0x7f199416b5f0 0x7f1994167ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:24:42.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:42.459+0000 7f1991600640 1 -- 172.21.15.6:0/139278418 >> v1:172.21.15.190:6789/0 conn(0x7f19940cc870 legacy=0x7f199406ddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:24:42.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:42.459+0000 7f1991600640 1 -- 172.21.15.6:0/139278418 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19940710c0 con 0x7f19940ccc50 2024-04-24T23:24:42.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:42.459+0000 7f1991600640 1 -- 172.21.15.6:0/139278418 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3463268701 0 0) 0x7f197c001670 con 0x7f19940ccc50 2024-04-24T23:24:42.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:42.459+0000 7f1991600640 1 -- 172.21.15.6:0/139278418 >> v1:172.21.15.6:6789/0 conn(0x7f19940ccc50 legacy=0x7f1994164a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:24:42.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:42.459+0000 7f1991600640 1 --2- 172.21.15.6:0/139278418 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1978003890 0x7f1978003c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:24:42.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:42.459+0000 7f1991600640 1 --2- 172.21.15.6:0/139278418 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1978004300 0x7f1978008d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:24:42.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:42.459+0000 7f1991600640 1 --2- 172.21.15.6:0/139278418 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f19780092f0 0x7f19780097a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:24:42.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:42.459+0000 7f1991600640 1 -- 172.21.15.6:0/139278418 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f1994070e70 con 0x7f1978004300 2024-04-24T23:24:42.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:42.459+0000 7f1991600640 1 -- 172.21.15.6:0/139278418 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f1978009fa0 con 0x7f19780092f0 2024-04-24T23:24:42.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:42.459+0000 7f1991600640 1 -- 172.21.15.6:0/139278418 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f197800a1a0 con 0x7f1978003890 2024-04-24T23:24:42.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:42.460+0000 7f1992a00640 1 --2- 172.21.15.6:0/139278418 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1978003890 0x7f1978003c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:24:42.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:42.460+0000 7f1993400640 1 --2- 172.21.15.6:0/139278418 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f19780092f0 0x7f19780097a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:24:42.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:42.460+0000 7f1992000640 1 --2- 172.21.15.6:0/139278418 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1978004300 0x7f1978008d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:24:42.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:42.461+0000 7f1993400640 1 -- 172.21.15.6:0/139278418 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1978003890 msgr2=0x7f1978003c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:24:42.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:42.461+0000 7f1993400640 1 --2- 172.21.15.6:0/139278418 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1978003890 0x7f1978003c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:24:42.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:42.461+0000 7f1993400640 1 -- 172.21.15.6:0/139278418 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1978004300 msgr2=0x7f1978008d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:24:42.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:42.461+0000 7f1993400640 1 --2- 172.21.15.6:0/139278418 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1978004300 0x7f1978008d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:24:42.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:42.461+0000 7f1993400640 1 -- 172.21.15.6:0/139278418 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19940710c0 con 0x7f19780092f0 2024-04-24T23:24:42.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:42.463+0000 7f1993400640 1 -- 172.21.15.6:0/139278418 reap_dead start 2024-04-24T23:24:42.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:42.463+0000 7f1993400640 1 --2- 172.21.15.6:0/139278418 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f19780092f0 0x7f19780097a0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f1984009410 tx=0x7f197800afc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eeae450a2bb7d9da server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:24:42.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:42.464+0000 7f1991600640 1 -- 172.21.15.6:0/139278418 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1984009060 con 0x7f19780092f0 2024-04-24T23:24:42.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:42.464+0000 7f1991600640 1 -- 172.21.15.6:0/139278418 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f198404ce30 con 0x7f19780092f0 2024-04-24T23:24:42.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:42.464+0000 7f1991600640 1 -- 172.21.15.6:0/139278418 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1984004220 con 0x7f19780092f0 2024-04-24T23:24:42.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:42.465+0000 7f1993e00640 1 -- 172.21.15.6:0/139278418 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f19780092f0 msgr2=0x7f19780097a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:24:42.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:42.465+0000 7f1993e00640 1 --2- 172.21.15.6:0/139278418 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f19780092f0 0x7f19780097a0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f1984009410 tx=0x7f197800afc0 comp rx=0 tx=0).stop 2024-04-24T23:24:42.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:42.467+0000 7f1993e00640 1 -- 172.21.15.6:0/139278418 shutdown_connections 2024-04-24T23:24:42.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:42.467+0000 7f1993e00640 1 --2- 172.21.15.6:0/139278418 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f19780092f0 0x7f19780097a0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:24:42.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:42.467+0000 7f1993e00640 1 -- 172.21.15.6:0/139278418 >> 172.21.15.6:0/139278418 conn(0x7f199406ed80 msgr2=0x7f199406f190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:24:42.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:42.470+0000 7f1993e00640 1 -- 172.21.15.6:0/139278418 shutdown_connections 2024-04-24T23:24:42.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:42.470+0000 7f1993e00640 1 -- 172.21.15.6:0/139278418 wait complete. 2024-04-24T23:24:42.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:42.473+0000 7f1993e00640 1 Processor -- start 2024-04-24T23:24:42.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:42.475+0000 7f1993e00640 1 -- start start 2024-04-24T23:24:42.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:42.478+0000 7f1993e00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f19780092f0 0x7f1994160fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:24:42.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:42.478+0000 7f1993e00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f19940c3e60 0x7f1994169150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:24:42.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:42.478+0000 7f1993e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f199415baa0 0x7f199415bf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:24:42.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:42.479+0000 7f1993e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f199416ba20 con 0x7f199415baa0 2024-04-24T23:24:42.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:42.479+0000 7f1993e00640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f19940c3ba0 con 0x7f19940c3e60 2024-04-24T23:24:42.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:42.479+0000 7f1993e00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f199406ddd0 con 0x7f19780092f0 2024-04-24T23:24:42.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:42.479+0000 7f1993400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f199415baa0 0x7f199415bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:24:42.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:42.480+0000 7f1993400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f199415baa0 0x7f199415bf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:40940/0 (socket says 172.21.15.6:40940) 2024-04-24T23:24:42.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:42.480+0000 7f1993400640 1 -- 172.21.15.6:0/3567668218 learned_addr learned my addr 172.21.15.6:0/3567668218 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:24:42.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:42.480+0000 7f1993400640 1 -- 172.21.15.6:0/3567668218 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f19780092f0 msgr2=0x7f1994160fa0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-24T23:24:42.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:42.480+0000 7f1993400640 1 --2- 172.21.15.6:0/3567668218 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f19780092f0 0x7f1994160fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:24:42.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:42.480+0000 7f1993400640 1 -- 172.21.15.6:0/3567668218 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f19940c3e60 msgr2=0x7f1994169150 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-24T23:24:42.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:42.480+0000 7f1993400640 1 --2- 172.21.15.6:0/3567668218 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f19940c3e60 0x7f1994169150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:24:42.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:42.480+0000 7f1993400640 1 -- 172.21.15.6:0/3567668218 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f199415c460 con 0x7f199415baa0 2024-04-24T23:24:42.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:42.481+0000 7f1993400640 1 --2- 172.21.15.6:0/3567668218 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f199415baa0 0x7f199415bf20 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f1984009500 tx=0x7f1984009a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:24:42.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:42.482+0000 7f198aa00640 1 -- 172.21.15.6:0/3567668218 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1984003e50 con 0x7f199415baa0 2024-04-24T23:24:42.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:42.482+0000 7f198aa00640 1 -- 172.21.15.6:0/3567668218 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f198405e440 con 0x7f199415baa0 2024-04-24T23:24:42.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:42.482+0000 7f198aa00640 1 -- 172.21.15.6:0/3567668218 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f198405e630 con 0x7f199415baa0 2024-04-24T23:24:42.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:42.483+0000 7f1993e00640 1 -- 172.21.15.6:0/3567668218 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1994200f10 con 0x7f199415baa0 2024-04-24T23:24:42.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:42.483+0000 7f1993e00640 1 -- 172.21.15.6:0/3567668218 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19942011b0 con 0x7f199415baa0 2024-04-24T23:24:42.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:42.486+0000 7f198aa00640 1 -- 172.21.15.6:0/3567668218 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f198405e7d0 con 0x7f199415baa0 2024-04-24T23:24:42.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:42.487+0000 7f1993e00640 1 -- 172.21.15.6:0/3567668218 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f199416abe0 con 0x7f199415baa0 2024-04-24T23:24:42.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:42.489+0000 7f198aa00640 1 --2- 172.21.15.6:0/3567668218 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f197006beb0 0x7f197006e370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:24:42.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:42.489+0000 7f198aa00640 1 -- 172.21.15.6:0/3567668218 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6848+0+0 (secure 0 0 0) 0x7f19840cc3a0 con 0x7f199415baa0 2024-04-24T23:24:42.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:42.492+0000 7f1992a00640 1 --2- 172.21.15.6:0/3567668218 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f197006beb0 0x7f197006e370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:24:42.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:42.493+0000 7f1992a00640 1 --2- 172.21.15.6:0/3567668218 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f197006beb0 0x7f197006e370 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f198c004d40 tx=0x7f198c001f40 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:24:42.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:42.497+0000 7f198aa00640 1 -- 172.21.15.6:0/3567668218 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f199416abe0 con 0x7f199415baa0 2024-04-24T23:24:43.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:43.019+0000 7f1993e00640 1 -- 172.21.15.6:0/3567668218 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1994075b30 con 0x7f199415baa0 2024-04-24T23:24:43.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:43.020+0000 7f198aa00640 1 -- 172.21.15.6:0/3567668218 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) ==== 94+0+6022 (secure 0 0 0) 0x7f198405a070 con 0x7f199415baa0 2024-04-24T23:24:43.022 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:24:43.023 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":43,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5081,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/1190759626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1190759626},{"type":"v1","addr":"172.21.15.6:6834","nonce":1190759626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":5249,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6838/2416336040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2416336040},{"type":"v1","addr":"172.21.15.190:6838","nonce":2416336040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":42},{"gid":5503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/3928731059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":3928731059},{"type":"v1","addr":"172.21.15.6:6837","nonce":3928731059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":43,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:24:32.678603+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":4306,"mds_1":5599},"failed":[],"damaged":[],"stopped":[2,3,4],"info":{"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/489092393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":489092393},{"type":"v1","addr":"172.21.15.6:6835","nonce":489092393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5599":{"gid":5599,"name":"d","rank":1,"incarnation":43,"state":"up:replay","state_seq":1,"addr":"172.21.15.190:6839/2643233757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":2643233757},{"type":"v1","addr":"172.21.15.190:6839","nonce":2643233757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4306,"qdb_cluster":[4306]},"id":1}]} 2024-04-24T23:24:43.023 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 43 2024-04-24T23:24:43.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:43.029+0000 7f1993e00640 1 -- 172.21.15.6:0/3567668218 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f197006beb0 msgr2=0x7f197006e370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:24:43.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:43.029+0000 7f1993e00640 1 --2- 172.21.15.6:0/3567668218 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f197006beb0 0x7f197006e370 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f198c004d40 tx=0x7f198c001f40 comp rx=0 tx=0).stop 2024-04-24T23:24:43.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:43.029+0000 7f1993e00640 1 -- 172.21.15.6:0/3567668218 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f199415baa0 msgr2=0x7f199415bf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:24:43.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:43.029+0000 7f1993e00640 1 --2- 172.21.15.6:0/3567668218 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f199415baa0 0x7f199415bf20 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f1984009500 tx=0x7f1984009a20 comp rx=0 tx=0).stop 2024-04-24T23:24:43.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:43.031+0000 7f1993e00640 1 -- 172.21.15.6:0/3567668218 shutdown_connections 2024-04-24T23:24:43.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:43.031+0000 7f1993e00640 1 --2- 172.21.15.6:0/3567668218 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f197006beb0 0x7f197006e370 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:24:43.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:43.031+0000 7f1993e00640 1 --2- 172.21.15.6:0/3567668218 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f199415baa0 0x7f199415bf20 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:24:43.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:43.031+0000 7f1993e00640 1 --2- 172.21.15.6:0/3567668218 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f19940c3e60 0x7f1994169150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:24:43.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:43.031+0000 7f1993e00640 1 --2- 172.21.15.6:0/3567668218 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f19780092f0 0x7f1994160fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:24:43.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:43.031+0000 7f1993e00640 1 -- 172.21.15.6:0/3567668218 >> 172.21.15.6:0/3567668218 conn(0x7f199406ed80 msgr2=0x7f1994162ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:24:43.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:43.032+0000 7f1993e00640 1 -- 172.21.15.6:0/3567668218 shutdown_connections 2024-04-24T23:24:43.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:43.032+0000 7f1993e00640 1 -- 172.21.15.6:0/3567668218 wait complete. 2024-04-24T23:24:43.070 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-24T23:24:45.071 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:24:45.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:45.441+0000 7f5480a00640 1 Processor -- start 2024-04-24T23:24:45.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:45.441+0000 7f5480a00640 1 -- start start 2024-04-24T23:24:45.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:45.443+0000 7f5480a00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f547c152bc0 0x7f547c159390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:24:45.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:45.443+0000 7f5480a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f547c0c8f50 0x7f547c15bf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:24:45.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:45.443+0000 7f5479e00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f547c152bc0 0x7f547c159390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:24:45.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:45.443+0000 7f5479e00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f547c152bc0 0x7f547c159390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.6:56138/0 (socket says 172.21.15.6:56138) 2024-04-24T23:24:45.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:45.444+0000 7f5479400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f547c0c8f50 0x7f547c15bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:24:45.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:45.444+0000 7f5479400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f547c0c8f50 0x7f547c15bf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42752/0 (socket says 172.21.15.6:42752) 2024-04-24T23:24:45.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:45.444+0000 7f5480a00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f547c0c6e70 con 0x7f547c0c9330 2024-04-24T23:24:45.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:45.444+0000 7f5480a00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f547c0c73d0 con 0x7f547c0c8f50 2024-04-24T23:24:45.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:45.444+0000 7f5480a00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f547c0c8ac0 con 0x7f547c152bc0 2024-04-24T23:24:45.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:45.444+0000 7f5479e00640 1 -- 172.21.15.6:0/858128666 learned_addr learned my addr 172.21.15.6:0/858128666 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:24:45.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:45.445+0000 7f5479400640 1 -- 172.21.15.6:0/858128666 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f547c152bc0 msgr2=0x7f547c159390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:24:45.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:45.445+0000 7f5479400640 1 --2- 172.21.15.6:0/858128666 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f547c152bc0 0x7f547c159390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:24:45.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:45.445+0000 7f5479400640 1 -- 172.21.15.6:0/858128666 >> v1:172.21.15.6:6789/0 conn(0x7f547c0c9330 legacy=0x7f547c15c440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:24:45.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:45.445+0000 7f5479400640 1 -- 172.21.15.6:0/858128666 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f547c1598d0 con 0x7f547c0c8f50 2024-04-24T23:24:45.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:45.447+0000 7f5479400640 1 --2- 172.21.15.6:0/858128666 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f547c0c8f50 0x7f547c15bf00 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f54680097a0 tx=0x7f546802a100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e9091d2f06ad9a6 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:24:45.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:45.447+0000 7f5478a00640 1 -- 172.21.15.6:0/858128666 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5468037070 con 0x7f547c0c8f50 2024-04-24T23:24:45.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:45.447+0000 7f5478a00640 1 -- 172.21.15.6:0/858128666 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f546802ae50 con 0x7f547c0c8f50 2024-04-24T23:24:45.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:45.447+0000 7f5478a00640 1 -- 172.21.15.6:0/858128666 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5468032660 con 0x7f547c0c8f50 2024-04-24T23:24:45.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:45.448+0000 7f5480a00640 1 -- 172.21.15.6:0/858128666 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f547c0c8f50 msgr2=0x7f547c15bf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:24:45.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:45.448+0000 7f5480a00640 1 --2- 172.21.15.6:0/858128666 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f547c0c8f50 0x7f547c15bf00 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f54680097a0 tx=0x7f546802a100 comp rx=0 tx=0).stop 2024-04-24T23:24:45.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:45.448+0000 7f5480a00640 1 -- 172.21.15.6:0/858128666 shutdown_connections 2024-04-24T23:24:45.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:45.448+0000 7f5480a00640 1 --2- 172.21.15.6:0/858128666 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f547c0c8f50 0x7f547c15bf00 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:24:45.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:45.448+0000 7f5480a00640 1 --2- 172.21.15.6:0/858128666 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f547c152bc0 0x7f547c159390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:24:45.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:45.448+0000 7f5480a00640 1 -- 172.21.15.6:0/858128666 >> 172.21.15.6:0/858128666 conn(0x7f547c06b0e0 msgr2=0x7f547c06b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:24:45.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:45.449+0000 7f5480a00640 1 -- 172.21.15.6:0/858128666 shutdown_connections 2024-04-24T23:24:45.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:45.449+0000 7f5480a00640 1 -- 172.21.15.6:0/858128666 wait complete. 2024-04-24T23:24:45.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:45.450+0000 7f5480a00640 1 Processor -- start 2024-04-24T23:24:45.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:45.451+0000 7f5480a00640 1 -- start start 2024-04-24T23:24:45.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:45.452+0000 7f5480a00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f547c0c8f50 0x7f547c15ae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:24:45.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:45.452+0000 7f5480a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f547c0c9330 0x7f547c1f9020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:24:45.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:45.452+0000 7f5480a00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f547c152bc0 0x7f547c143150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:24:45.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:45.452+0000 7f5480a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f547c0c8470 con 0x7f547c0c9330 2024-04-24T23:24:45.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:45.452+0000 7f5480a00640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f547c0c6700 con 0x7f547c152bc0 2024-04-24T23:24:45.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:45.452+0000 7f5480a00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f547c0c8750 con 0x7f547c0c8f50 2024-04-24T23:24:45.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:45.452+0000 7f5479e00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f547c0c8f50 0x7f547c15ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:24:45.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:45.452+0000 7f5479e00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f547c0c8f50 0x7f547c15ae00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.6:56144/0 (socket says 172.21.15.6:56144) 2024-04-24T23:24:45.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:45.453+0000 7f5479e00640 1 -- 172.21.15.6:0/2962877709 learned_addr learned my addr 172.21.15.6:0/2962877709 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:24:45.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:45.453+0000 7f5479400640 1 --2- 172.21.15.6:0/2962877709 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f547c0c9330 0x7f547c1f9020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:24:45.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:45.453+0000 7f547a800640 1 --2- 172.21.15.6:0/2962877709 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f547c152bc0 0x7f547c143150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:24:45.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:45.453+0000 7f547a800640 1 -- 172.21.15.6:0/2962877709 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f547c0c8f50 msgr2=0x7f547c15ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:24:45.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:45.453+0000 7f547a800640 1 --2- 172.21.15.6:0/2962877709 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f547c0c8f50 0x7f547c15ae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:24:45.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:45.453+0000 7f547a800640 1 -- 172.21.15.6:0/2962877709 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f547c0c9330 msgr2=0x7f547c1f9020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:24:45.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:45.453+0000 7f547a800640 1 --2- 172.21.15.6:0/2962877709 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f547c0c9330 0x7f547c1f9020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:24:45.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:45.453+0000 7f547a800640 1 -- 172.21.15.6:0/2962877709 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f547c143690 con 0x7f547c152bc0 2024-04-24T23:24:45.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:45.454+0000 7f547a800640 1 --2- 172.21.15.6:0/2962877709 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f547c152bc0 0x7f547c143150 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f546c00c390 tx=0x7f546c00c850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:24:45.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:45.454+0000 7f5472000640 1 -- 172.21.15.6:0/2962877709 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f546c017070 con 0x7f547c152bc0 2024-04-24T23:24:45.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:45.455+0000 7f5472000640 1 -- 172.21.15.6:0/2962877709 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f546c0052d0 con 0x7f547c152bc0 2024-04-24T23:24:45.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:45.455+0000 7f5472000640 1 -- 172.21.15.6:0/2962877709 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f546c003850 con 0x7f547c152bc0 2024-04-24T23:24:45.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:45.455+0000 7f5480a00640 1 -- 172.21.15.6:0/2962877709 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f547c072fb0 con 0x7f547c152bc0 2024-04-24T23:24:45.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:45.455+0000 7f5480a00640 1 -- 172.21.15.6:0/2962877709 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f547c158de0 con 0x7f547c152bc0 2024-04-24T23:24:45.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:45.461+0000 7f5472000640 1 -- 172.21.15.6:0/2962877709 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f546c01a070 con 0x7f547c152bc0 2024-04-24T23:24:45.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:45.465+0000 7f5480a00640 1 -- 172.21.15.6:0/2962877709 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f547c151050 con 0x7f547c152bc0 2024-04-24T23:24:45.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:45.467+0000 7f5472000640 1 --2- 172.21.15.6:0/2962877709 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f545c06bf30 0x7f545c06e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:24:45.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:45.467+0000 7f5472000640 1 -- 172.21.15.6:0/2962877709 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6848+0+0 (secure 0 0 0) 0x7f546c08a470 con 0x7f547c152bc0 2024-04-24T23:24:45.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:45.468+0000 7f5479e00640 1 --2- 172.21.15.6:0/2962877709 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f545c06bf30 0x7f545c06e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:24:45.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:45.472+0000 7f5479e00640 1 --2- 172.21.15.6:0/2962877709 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f545c06bf30 0x7f545c06e3f0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f5474011420 tx=0x7f547400b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:24:45.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:45.477+0000 7f5472000640 1 -- 172.21.15.6:0/2962877709 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f547c151050 con 0x7f547c152bc0 2024-04-24T23:24:46.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:46.006+0000 7f5480a00640 1 -- 172.21.15.6:0/2962877709 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f547c0cbff0 con 0x7f547c152bc0 2024-04-24T23:24:46.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:46.010+0000 7f5472000640 1 -- 172.21.15.6:0/2962877709 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) ==== 94+0+6022 (secure 0 0 0) 0x7f547c0cbff0 con 0x7f547c152bc0 2024-04-24T23:24:46.013 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:24:46.013 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":43,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5081,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/1190759626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1190759626},{"type":"v1","addr":"172.21.15.6:6834","nonce":1190759626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":5249,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6838/2416336040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2416336040},{"type":"v1","addr":"172.21.15.190:6838","nonce":2416336040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":42},{"gid":5503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/3928731059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":3928731059},{"type":"v1","addr":"172.21.15.6:6837","nonce":3928731059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":43,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:24:32.678603+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":4306,"mds_1":5599},"failed":[],"damaged":[],"stopped":[2,3,4],"info":{"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/489092393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":489092393},{"type":"v1","addr":"172.21.15.6:6835","nonce":489092393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5599":{"gid":5599,"name":"d","rank":1,"incarnation":43,"state":"up:replay","state_seq":1,"addr":"172.21.15.190:6839/2643233757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":2643233757},{"type":"v1","addr":"172.21.15.190:6839","nonce":2643233757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4306,"qdb_cluster":[4306]},"id":1}]} 2024-04-24T23:24:46.013 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 43 2024-04-24T23:24:46.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:46.019+0000 7f5480a00640 1 -- 172.21.15.6:0/2962877709 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f545c06bf30 msgr2=0x7f545c06e3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:24:46.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:46.019+0000 7f5480a00640 1 --2- 172.21.15.6:0/2962877709 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f545c06bf30 0x7f545c06e3f0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f5474011420 tx=0x7f547400b040 comp rx=0 tx=0).stop 2024-04-24T23:24:46.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:46.024+0000 7f5480a00640 1 -- 172.21.15.6:0/2962877709 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f547c152bc0 msgr2=0x7f547c143150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:24:46.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:46.024+0000 7f5480a00640 1 --2- 172.21.15.6:0/2962877709 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f547c152bc0 0x7f547c143150 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f546c00c390 tx=0x7f546c00c850 comp rx=0 tx=0).stop 2024-04-24T23:24:46.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:46.024+0000 7f5480a00640 1 -- 172.21.15.6:0/2962877709 shutdown_connections 2024-04-24T23:24:46.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:46.024+0000 7f5480a00640 1 --2- 172.21.15.6:0/2962877709 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f545c06bf30 0x7f545c06e3f0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:24:46.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:46.024+0000 7f5480a00640 1 --2- 172.21.15.6:0/2962877709 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f547c152bc0 0x7f547c143150 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:24:46.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:46.024+0000 7f5480a00640 1 --2- 172.21.15.6:0/2962877709 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f547c0c9330 0x7f547c1f9020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:24:46.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:46.024+0000 7f5480a00640 1 --2- 172.21.15.6:0/2962877709 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f547c0c8f50 0x7f547c15ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:24:46.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:46.024+0000 7f5480a00640 1 -- 172.21.15.6:0/2962877709 >> 172.21.15.6:0/2962877709 conn(0x7f547c06b0e0 msgr2=0x7f547c148780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:24:46.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:46.024+0000 7f5480a00640 1 -- 172.21.15.6:0/2962877709 shutdown_connections 2024-04-24T23:24:46.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:46.024+0000 7f5480a00640 1 -- 172.21.15.6:0/2962877709 wait complete. 2024-04-24T23:24:46.058 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-24T23:24:48.059 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:24:48.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:48.381+0000 7f891c400640 1 Processor -- start 2024-04-24T23:24:48.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:48.381+0000 7f891c400640 1 -- start start 2024-04-24T23:24:48.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:48.383+0000 7f891c400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f8914152d80 0x7f89140c8d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:24:48.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:48.383+0000 7f891c400640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f89141592a0 0x7f891415c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:24:48.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:48.383+0000 7f891c400640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f891415cb10 0x7f8914070c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:24:48.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:48.383+0000 7f891c400640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f89140c5500 con 0x7f8914152d80 2024-04-24T23:24:48.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:48.383+0000 7f891c400640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f891415c140 con 0x7f891415cb10 2024-04-24T23:24:48.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:48.383+0000 7f891c400640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f8914159770 con 0x7f89141592a0 2024-04-24T23:24:48.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:48.384+0000 7f8919a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f8914152d80 0x7f89140c8d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:24:48.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:48.384+0000 7f8919a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f8914152d80 0x7f89140c8d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42768/0 (socket says 172.21.15.6:42768) 2024-04-24T23:24:48.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:48.384+0000 7f8919a00640 1 -- 172.21.15.6:0/3597773059 learned_addr learned my addr 172.21.15.6:0/3597773059 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:24:48.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:48.385+0000 7f891a400640 1 --2- 172.21.15.6:0/3597773059 >> v2:172.21.15.190:3300/0 conn(0x7f891415cb10 0x7f8914070c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:24:48.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:48.385+0000 7f8919000640 1 --2- 172.21.15.6:0/3597773059 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f89141592a0 0x7f891415c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:24:48.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:48.386+0000 7f8919000640 1 -- 172.21.15.6:0/3597773059 >> v2:172.21.15.190:3300/0 conn(0x7f891415cb10 msgr2=0x7f8914070c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:24:48.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:48.386+0000 7f8919000640 1 --2- 172.21.15.6:0/3597773059 >> v2:172.21.15.190:3300/0 conn(0x7f891415cb10 0x7f8914070c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:24:48.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:48.386+0000 7f8919000640 1 -- 172.21.15.6:0/3597773059 >> v2:172.21.15.6:3300/0 conn(0x7f8914152d80 msgr2=0x7f89140c8d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:24:48.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:48.387+0000 7f8919000640 1 --2- 172.21.15.6:0/3597773059 >> v2:172.21.15.6:3300/0 conn(0x7f8914152d80 0x7f89140c8d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:24:48.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:48.387+0000 7f8919000640 1 -- 172.21.15.6:0/3597773059 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89140c3470 con 0x7f89141592a0 2024-04-24T23:24:48.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:48.388+0000 7f891a400640 1 --2- 172.21.15.6:0/3597773059 >> v2:172.21.15.190:3300/0 conn(0x7f891415cb10 0x7f8914070c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:24:48.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:48.388+0000 7f8919a00640 1 --2- 172.21.15.6:0/3597773059 >> v2:172.21.15.6:3300/0 conn(0x7f8914152d80 0x7f89140c8d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:24:48.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:48.389+0000 7f8919000640 1 --2- 172.21.15.6:0/3597773059 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f89141592a0 0x7f891415c5d0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f89140717c0 tx=0x7f890002a140 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2a3f1c9f01e45198 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:24:48.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:48.390+0000 7f8913e00640 1 -- 172.21.15.6:0/3597773059 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8900037070 con 0x7f89141592a0 2024-04-24T23:24:48.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:48.390+0000 7f8913e00640 1 -- 172.21.15.6:0/3597773059 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8900032440 con 0x7f89141592a0 2024-04-24T23:24:48.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:48.390+0000 7f8913e00640 1 -- 172.21.15.6:0/3597773059 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8900032850 con 0x7f89141592a0 2024-04-24T23:24:48.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:48.392+0000 7f891c400640 1 -- 172.21.15.6:0/3597773059 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f89141592a0 msgr2=0x7f891415c5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:24:48.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:48.392+0000 7f891c400640 1 --2- 172.21.15.6:0/3597773059 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f89141592a0 0x7f891415c5d0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f89140717c0 tx=0x7f890002a140 comp rx=0 tx=0).stop 2024-04-24T23:24:48.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:48.392+0000 7f891c400640 1 -- 172.21.15.6:0/3597773059 shutdown_connections 2024-04-24T23:24:48.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:48.392+0000 7f891c400640 1 --2- 172.21.15.6:0/3597773059 >> v2:172.21.15.190:3300/0 conn(0x7f891415cb10 0x7f8914070c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:24:48.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:48.392+0000 7f891c400640 1 --2- 172.21.15.6:0/3597773059 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f89141592a0 0x7f891415c5d0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:24:48.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:48.392+0000 7f891c400640 1 --2- 172.21.15.6:0/3597773059 >> v2:172.21.15.6:3300/0 conn(0x7f8914152d80 0x7f89140c8d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:24:48.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:48.392+0000 7f891c400640 1 -- 172.21.15.6:0/3597773059 >> 172.21.15.6:0/3597773059 conn(0x7f8914070280 msgr2=0x7f891406f320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:24:48.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:48.392+0000 7f891c400640 1 -- 172.21.15.6:0/3597773059 shutdown_connections 2024-04-24T23:24:48.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:48.393+0000 7f891c400640 1 -- 172.21.15.6:0/3597773059 wait complete. 2024-04-24T23:24:48.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:48.398+0000 7f891c400640 1 Processor -- start 2024-04-24T23:24:48.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:48.399+0000 7f891c400640 1 -- start start 2024-04-24T23:24:48.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:48.406+0000 7f891c400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8914152d80 0x7f89141f03a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:24:48.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:48.407+0000 7f8919a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8914152d80 0x7f89141f03a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:24:48.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:48.407+0000 7f8919a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8914152d80 0x7f89141f03a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42774/0 (socket says 172.21.15.6:42774) 2024-04-24T23:24:48.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:48.407+0000 7f8919a00640 1 -- 172.21.15.6:0/206642594 learned_addr learned my addr 172.21.15.6:0/206642594 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:24:48.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:48.409+0000 7f891c400640 1 --2- 172.21.15.6:0/206642594 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f89141592a0 0x7f891414f490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:24:48.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:48.409+0000 7f8919000640 1 --2- 172.21.15.6:0/206642594 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f89141592a0 0x7f891414f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:24:48.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:48.411+0000 7f891c400640 1 --2- 172.21.15.6:0/206642594 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f891415cb10 0x7f89141ecb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:24:48.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:48.412+0000 7f891a400640 1 --2- 172.21.15.6:0/206642594 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f891415cb10 0x7f89141ecb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:24:48.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:48.412+0000 7f891c400640 1 -- 172.21.15.6:0/206642594 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f891415bdd0 con 0x7f8914152d80 2024-04-24T23:24:48.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:48.412+0000 7f891c400640 1 -- 172.21.15.6:0/206642594 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f89140c3a30 con 0x7f89141592a0 2024-04-24T23:24:48.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:48.412+0000 7f891c400640 1 -- 172.21.15.6:0/206642594 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f8914159cd0 con 0x7f891415cb10 2024-04-24T23:24:48.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:48.414+0000 7f8919000640 1 -- 172.21.15.6:0/206642594 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f891415cb10 msgr2=0x7f89141ecb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:24:48.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:48.414+0000 7f8919000640 1 --2- 172.21.15.6:0/206642594 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f891415cb10 0x7f89141ecb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:24:48.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:48.414+0000 7f8919000640 1 -- 172.21.15.6:0/206642594 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8914152d80 msgr2=0x7f89141f03a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:24:48.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:48.416+0000 7f8919000640 1 --2- 172.21.15.6:0/206642594 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8914152d80 0x7f89141f03a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:24:48.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:48.416+0000 7f8919000640 1 -- 172.21.15.6:0/206642594 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89140c6210 con 0x7f89141592a0 2024-04-24T23:24:48.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:48.416+0000 7f8919000640 1 --2- 172.21.15.6:0/206642594 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f89141592a0 0x7f891414f490 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f8900031000 tx=0x7f89000046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:24:48.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:48.416+0000 7f8919a00640 1 --2- 172.21.15.6:0/206642594 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8914152d80 0x7f89141f03a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:24:48.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:48.417+0000 7f8911600640 1 -- 172.21.15.6:0/206642594 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8900041070 con 0x7f89141592a0 2024-04-24T23:24:48.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:48.417+0000 7f8911600640 1 -- 172.21.15.6:0/206642594 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f890003c890 con 0x7f89141592a0 2024-04-24T23:24:48.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:48.417+0000 7f8911600640 1 -- 172.21.15.6:0/206642594 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8900037040 con 0x7f89141592a0 2024-04-24T23:24:48.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:48.417+0000 7f891c400640 1 -- 172.21.15.6:0/206642594 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89140c64f0 con 0x7f89141592a0 2024-04-24T23:24:48.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:48.417+0000 7f891c400640 1 -- 172.21.15.6:0/206642594 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89140c6ab0 con 0x7f89141592a0 2024-04-24T23:24:48.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:48.418+0000 7f891a400640 1 --2- 172.21.15.6:0/206642594 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f891415cb10 0x7f89141ecb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:24:48.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:48.424+0000 7f8911600640 1 -- 172.21.15.6:0/206642594 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f8900044070 con 0x7f89141592a0 2024-04-24T23:24:48.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:48.426+0000 7f8911600640 1 --2- 172.21.15.6:0/206642594 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f88f406beb0 0x7f88f406e370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:24:48.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:48.426+0000 7f8911600640 1 -- 172.21.15.6:0/206642594 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6848+0+0 (secure 0 0 0) 0x7f89000abbd0 con 0x7f89141592a0 2024-04-24T23:24:48.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:48.429+0000 7f891c400640 1 -- 172.21.15.6:0/206642594 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f89140c6cf0 con 0x7f89141592a0 2024-04-24T23:24:48.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:48.432+0000 7f8919a00640 1 --2- 172.21.15.6:0/206642594 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f88f406beb0 0x7f88f406e370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:24:48.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:48.440+0000 7f8919a00640 1 --2- 172.21.15.6:0/206642594 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f88f406beb0 0x7f88f406e370 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f8908009630 tx=0x7f8908009240 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:24:48.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:48.447+0000 7f8911600640 1 -- 172.21.15.6:0/206642594 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f89140c6cf0 con 0x7f89141592a0 2024-04-24T23:24:48.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:48.974+0000 7f891c400640 1 -- 172.21.15.6:0/206642594 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f89140767b0 con 0x7f89141592a0 2024-04-24T23:24:48.993 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:24:48.993 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":43,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5081,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/1190759626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1190759626},{"type":"v1","addr":"172.21.15.6:6834","nonce":1190759626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":5249,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6838/2416336040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2416336040},{"type":"v1","addr":"172.21.15.190:6838","nonce":2416336040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":42},{"gid":5503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/3928731059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":3928731059},{"type":"v1","addr":"172.21.15.6:6837","nonce":3928731059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":43,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:24:32.678603+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":4306,"mds_1":5599},"failed":[],"damaged":[],"stopped":[2,3,4],"info":{"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/489092393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":489092393},{"type":"v1","addr":"172.21.15.6:6835","nonce":489092393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5599":{"gid":5599,"name":"d","rank":1,"incarnation":43,"state":"up:replay","state_seq":1,"addr":"172.21.15.190:6839/2643233757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":2643233757},{"type":"v1","addr":"172.21.15.190:6839","nonce":2643233757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4306,"qdb_cluster":[4306]},"id":1}]} 2024-04-24T23:24:48.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:48.988+0000 7f8911600640 1 -- 172.21.15.6:0/206642594 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) ==== 94+0+6022 (secure 0 0 0) 0x7f8900079ac0 con 0x7f89141592a0 2024-04-24T23:24:48.994 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 43 2024-04-24T23:24:48.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:48.993+0000 7f891c400640 1 -- 172.21.15.6:0/206642594 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f88f406beb0 msgr2=0x7f88f406e370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:24:48.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:48.993+0000 7f891c400640 1 --2- 172.21.15.6:0/206642594 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f88f406beb0 0x7f88f406e370 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f8908009630 tx=0x7f8908009240 comp rx=0 tx=0).stop 2024-04-24T23:24:48.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:48.993+0000 7f891c400640 1 -- 172.21.15.6:0/206642594 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f89141592a0 msgr2=0x7f891414f490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:24:48.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:48.993+0000 7f891c400640 1 --2- 172.21.15.6:0/206642594 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f89141592a0 0x7f891414f490 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f8900031000 tx=0x7f89000046c0 comp rx=0 tx=0).stop 2024-04-24T23:24:48.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:48.994+0000 7f891c400640 1 -- 172.21.15.6:0/206642594 shutdown_connections 2024-04-24T23:24:48.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:48.994+0000 7f891c400640 1 --2- 172.21.15.6:0/206642594 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f88f406beb0 0x7f88f406e370 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:24:48.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:48.994+0000 7f891c400640 1 --2- 172.21.15.6:0/206642594 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f891415cb10 0x7f89141ecb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:24:48.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:48.994+0000 7f891c400640 1 --2- 172.21.15.6:0/206642594 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f89141592a0 0x7f891414f490 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:24:48.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:48.994+0000 7f891c400640 1 --2- 172.21.15.6:0/206642594 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8914152d80 0x7f89141f03a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:24:48.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:48.994+0000 7f891c400640 1 -- 172.21.15.6:0/206642594 >> 172.21.15.6:0/206642594 conn(0x7f8914070280 msgr2=0x7f89140cd830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:24:48.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:48.994+0000 7f891c400640 1 -- 172.21.15.6:0/206642594 shutdown_connections 2024-04-24T23:24:48.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:48.995+0000 7f891c400640 1 -- 172.21.15.6:0/206642594 wait complete. 2024-04-24T23:24:49.032 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-24T23:24:51.034 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:24:51.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:51.380+0000 7fb2c9400640 1 Processor -- start 2024-04-24T23:24:51.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:51.381+0000 7fb2c9400640 1 -- start start 2024-04-24T23:24:51.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:51.382+0000 7fb2c9400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fb2c4154d90 0x7fb2c40cbe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:24:51.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:51.383+0000 7fb2c9400640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fb2c40cb090 0x7fb2c40cc3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:24:51.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:51.383+0000 7fb2c9400640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb2c40d0820 0x7fb2c40d0cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:24:51.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:51.383+0000 7fb2c9400640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7fb2c4155170 con 0x7fb2c4154d90 2024-04-24T23:24:51.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:51.383+0000 7fb2c9400640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7fb2c40cc9a0 con 0x7fb2c40cb090 2024-04-24T23:24:51.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:51.383+0000 7fb2c9400640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fb2c415ad60 con 0x7fb2c40d0820 2024-04-24T23:24:51.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:51.383+0000 7fb2c8a00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb2c40d0820 0x7fb2c40d0cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:24:51.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:51.383+0000 7fb2c8a00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb2c40d0820 0x7fb2c40d0cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.6:56184/0 (socket says 172.21.15.6:56184) 2024-04-24T23:24:51.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:51.383+0000 7fb2c8a00640 1 -- 172.21.15.6:0/1123119340 learned_addr learned my addr 172.21.15.6:0/1123119340 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:24:51.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:51.384+0000 7fb2c3e00640 1 --2- 172.21.15.6:0/1123119340 >> v2:172.21.15.6:3300/0 conn(0x7fb2c4154d90 0x7fb2c40cbe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:24:51.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:51.386+0000 7fb2c8a00640 1 -- 172.21.15.6:0/1123119340 >> v2:172.21.15.190:3300/0 conn(0x7fb2c40cb090 msgr2=0x7fb2c40cc3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:24:51.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:51.384+0000 7fb2c3400640 1 --2- 172.21.15.6:0/1123119340 >> v2:172.21.15.190:3300/0 conn(0x7fb2c40cb090 0x7fb2c40cc3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:24:51.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:51.386+0000 7fb2c8a00640 1 --2- 172.21.15.6:0/1123119340 >> v2:172.21.15.190:3300/0 conn(0x7fb2c40cb090 0x7fb2c40cc3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:24:51.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:51.386+0000 7fb2c8a00640 1 -- 172.21.15.6:0/1123119340 >> v2:172.21.15.6:3300/0 conn(0x7fb2c4154d90 msgr2=0x7fb2c40cbe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:24:51.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:51.386+0000 7fb2c8a00640 1 --2- 172.21.15.6:0/1123119340 >> v2:172.21.15.6:3300/0 conn(0x7fb2c4154d90 0x7fb2c40cbe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:24:51.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:51.386+0000 7fb2c8a00640 1 -- 172.21.15.6:0/1123119340 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2c415b040 con 0x7fb2c40d0820 2024-04-24T23:24:51.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:51.388+0000 7fb2c8a00640 1 --2- 172.21.15.6:0/1123119340 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb2c40d0820 0x7fb2c40d0cb0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fb2c406c9f0 tx=0x7fb2bc02a760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7a70ac820cda296a server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:24:51.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:51.389+0000 7fb2c3400640 1 --2- 172.21.15.6:0/1123119340 >> v2:172.21.15.190:3300/0 conn(0x7fb2c40cb090 0x7fb2c40cc3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:24:51.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:51.389+0000 7fb2c3e00640 1 --2- 172.21.15.6:0/1123119340 >> v2:172.21.15.6:3300/0 conn(0x7fb2c4154d90 0x7fb2c40cbe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:24:51.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:51.389+0000 7fb2c2a00640 1 -- 172.21.15.6:0/1123119340 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2bc036070 con 0x7fb2c40d0820 2024-04-24T23:24:51.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:51.389+0000 7fb2c2a00640 1 -- 172.21.15.6:0/1123119340 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb2bc031600 con 0x7fb2c40d0820 2024-04-24T23:24:51.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:51.389+0000 7fb2c2a00640 1 -- 172.21.15.6:0/1123119340 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2bc0319b0 con 0x7fb2c40d0820 2024-04-24T23:24:51.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:51.390+0000 7fb2c9400640 1 -- 172.21.15.6:0/1123119340 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb2c40d0820 msgr2=0x7fb2c40d0cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:24:51.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:51.390+0000 7fb2c9400640 1 --2- 172.21.15.6:0/1123119340 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb2c40d0820 0x7fb2c40d0cb0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fb2c406c9f0 tx=0x7fb2bc02a760 comp rx=0 tx=0).stop 2024-04-24T23:24:51.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:51.390+0000 7fb2c9400640 1 -- 172.21.15.6:0/1123119340 shutdown_connections 2024-04-24T23:24:51.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:51.390+0000 7fb2c9400640 1 --2- 172.21.15.6:0/1123119340 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb2c40d0820 0x7fb2c40d0cb0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:24:51.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:51.391+0000 7fb2c9400640 1 --2- 172.21.15.6:0/1123119340 >> v2:172.21.15.190:3300/0 conn(0x7fb2c40cb090 0x7fb2c40cc3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:24:51.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:51.391+0000 7fb2c9400640 1 --2- 172.21.15.6:0/1123119340 >> v2:172.21.15.6:3300/0 conn(0x7fb2c4154d90 0x7fb2c40cbe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:24:51.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:51.391+0000 7fb2c9400640 1 -- 172.21.15.6:0/1123119340 >> 172.21.15.6:0/1123119340 conn(0x7fb2c406be10 msgr2=0x7fb2c406c1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:24:51.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:51.393+0000 7fb2c9400640 1 -- 172.21.15.6:0/1123119340 shutdown_connections 2024-04-24T23:24:51.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:51.394+0000 7fb2c9400640 1 -- 172.21.15.6:0/1123119340 wait complete. 2024-04-24T23:24:51.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:51.396+0000 7fb2c9400640 1 Processor -- start 2024-04-24T23:24:51.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:51.397+0000 7fb2c9400640 1 -- start start 2024-04-24T23:24:51.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:51.398+0000 7fb2c9400640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb2c40cb090 0x7fb2c415a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:24:51.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:51.398+0000 7fb2c9400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb2c40d0820 0x7fb2c41f1530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:24:51.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:51.398+0000 7fb2c9400640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb2c4154d90 0x7fb2c414d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:24:51.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:51.398+0000 7fb2c9400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fb2c415aa30 con 0x7fb2c40d0820 2024-04-24T23:24:51.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:51.398+0000 7fb2c9400640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fb2c415b600 con 0x7fb2c4154d90 2024-04-24T23:24:51.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:51.398+0000 7fb2c9400640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fb2c415e000 con 0x7fb2c40cb090 2024-04-24T23:24:51.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:51.399+0000 7fb2c8a00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb2c4154d90 0x7fb2c414d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:24:51.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:51.399+0000 7fb2c8a00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb2c4154d90 0x7fb2c414d690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:51506/0 (socket says 172.21.15.6:51506) 2024-04-24T23:24:51.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:51.399+0000 7fb2c8a00640 1 -- 172.21.15.6:0/2069584048 learned_addr learned my addr 172.21.15.6:0/2069584048 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:24:51.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:51.400+0000 7fb2c3e00640 1 --2- 172.21.15.6:0/2069584048 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb2c40cb090 0x7fb2c415a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:24:51.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:51.400+0000 7fb2c3400640 1 --2- 172.21.15.6:0/2069584048 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb2c40d0820 0x7fb2c41f1530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:24:51.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:51.400+0000 7fb2c8a00640 1 -- 172.21.15.6:0/2069584048 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb2c40cb090 msgr2=0x7fb2c415a0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:24:51.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:51.400+0000 7fb2c8a00640 1 --2- 172.21.15.6:0/2069584048 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb2c40cb090 0x7fb2c415a0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:24:51.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:51.400+0000 7fb2c8a00640 1 -- 172.21.15.6:0/2069584048 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb2c40d0820 msgr2=0x7fb2c41f1530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:24:51.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:51.400+0000 7fb2c8a00640 1 --2- 172.21.15.6:0/2069584048 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb2c40d0820 0x7fb2c41f1530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:24:51.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:51.400+0000 7fb2c8a00640 1 -- 172.21.15.6:0/2069584048 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2c4152f30 con 0x7fb2c4154d90 2024-04-24T23:24:51.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:51.400+0000 7fb2c8a00640 1 --2- 172.21.15.6:0/2069584048 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb2c4154d90 0x7fb2c414d690 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fb2bc009f10 tx=0x7fb2bc004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:24:51.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:51.401+0000 7fb2bbe00640 1 -- 172.21.15.6:0/2069584048 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2bc03f030 con 0x7fb2c4154d90 2024-04-24T23:24:51.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:51.401+0000 7fb2bbe00640 1 -- 172.21.15.6:0/2069584048 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb2bc032e20 con 0x7fb2c4154d90 2024-04-24T23:24:51.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:51.402+0000 7fb2c9400640 1 -- 172.21.15.6:0/2069584048 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2c41536f0 con 0x7fb2c4154d90 2024-04-24T23:24:51.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:51.402+0000 7fb2bbe00640 1 -- 172.21.15.6:0/2069584048 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2bc036040 con 0x7fb2c4154d90 2024-04-24T23:24:51.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:51.403+0000 7fb2c9400640 1 -- 172.21.15.6:0/2069584048 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2c4154370 con 0x7fb2c4154d90 2024-04-24T23:24:51.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:51.404+0000 7fb2c9400640 1 -- 172.21.15.6:0/2069584048 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb2c4075b30 con 0x7fb2c4154d90 2024-04-24T23:24:51.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:51.410+0000 7fb2bbe00640 1 -- 172.21.15.6:0/2069584048 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fb2bc042070 con 0x7fb2c4154d90 2024-04-24T23:24:51.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:51.415+0000 7fb2bbe00640 1 --2- 172.21.15.6:0/2069584048 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fb2a406be80 0x7fb2a406e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:24:51.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:51.415+0000 7fb2bbe00640 1 -- 172.21.15.6:0/2069584048 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6848+0+0 (secure 0 0 0) 0x7fb2bc0ab560 con 0x7fb2c4154d90 2024-04-24T23:24:51.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:51.415+0000 7fb2c3e00640 1 --2- 172.21.15.6:0/2069584048 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fb2a406be80 0x7fb2a406e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:24:51.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:51.415+0000 7fb2bbe00640 1 -- 172.21.15.6:0/2069584048 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fb2bc079bf0 con 0x7fb2c4154d90 2024-04-24T23:24:51.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:51.416+0000 7fb2c3e00640 1 --2- 172.21.15.6:0/2069584048 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fb2a406be80 0x7fb2a406e340 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fb2b4006fd0 tx=0x7fb2b4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:24:51.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:51.944+0000 7fb2c9400640 1 -- 172.21.15.6:0/2069584048 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb2c40cbc60 con 0x7fb2c4154d90 2024-04-24T23:24:51.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:51.946+0000 7fb2bbe00640 1 -- 172.21.15.6:0/2069584048 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) ==== 94+0+6024 (secure 0 0 0) 0x7fb2bc0794f0 con 0x7fb2c4154d90 2024-04-24T23:24:51.948 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:24:51.948 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":44,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5081,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/1190759626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1190759626},{"type":"v1","addr":"172.21.15.6:6834","nonce":1190759626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":5249,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6838/2416336040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2416336040},{"type":"v1","addr":"172.21.15.190:6838","nonce":2416336040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":42},{"gid":5503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/3928731059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":3928731059},{"type":"v1","addr":"172.21.15.6:6837","nonce":3928731059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:24:48.478336+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":4306,"mds_1":5599},"failed":[],"damaged":[],"stopped":[2,3,4],"info":{"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/489092393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":489092393},{"type":"v1","addr":"172.21.15.6:6835","nonce":489092393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5599":{"gid":5599,"name":"d","rank":1,"incarnation":43,"state":"up:resolve","state_seq":19,"addr":"172.21.15.190:6839/2643233757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":2643233757},{"type":"v1","addr":"172.21.15.190:6839","nonce":2643233757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4306,"qdb_cluster":[4306]},"id":1}]} 2024-04-24T23:24:51.948 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 44 2024-04-24T23:24:51.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:51.954+0000 7fb2c9400640 1 -- 172.21.15.6:0/2069584048 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fb2a406be80 msgr2=0x7fb2a406e340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:24:51.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:51.954+0000 7fb2c9400640 1 --2- 172.21.15.6:0/2069584048 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fb2a406be80 0x7fb2a406e340 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fb2b4006fd0 tx=0x7fb2b4008040 comp rx=0 tx=0).stop 2024-04-24T23:24:51.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:51.954+0000 7fb2c9400640 1 -- 172.21.15.6:0/2069584048 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb2c4154d90 msgr2=0x7fb2c414d690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:24:51.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:51.954+0000 7fb2c9400640 1 --2- 172.21.15.6:0/2069584048 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb2c4154d90 0x7fb2c414d690 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fb2bc009f10 tx=0x7fb2bc004a40 comp rx=0 tx=0).stop 2024-04-24T23:24:51.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:51.955+0000 7fb2c9400640 1 -- 172.21.15.6:0/2069584048 shutdown_connections 2024-04-24T23:24:51.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:51.955+0000 7fb2c9400640 1 --2- 172.21.15.6:0/2069584048 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fb2a406be80 0x7fb2a406e340 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:24:51.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:51.955+0000 7fb2c9400640 1 --2- 172.21.15.6:0/2069584048 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb2c4154d90 0x7fb2c414d690 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:24:51.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:51.955+0000 7fb2c9400640 1 --2- 172.21.15.6:0/2069584048 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb2c40d0820 0x7fb2c41f1530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:24:51.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:51.955+0000 7fb2c9400640 1 --2- 172.21.15.6:0/2069584048 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb2c40cb090 0x7fb2c415a0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:24:51.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:51.955+0000 7fb2c9400640 1 -- 172.21.15.6:0/2069584048 >> 172.21.15.6:0/2069584048 conn(0x7fb2c406be10 msgr2=0x7fb2c4151af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:24:51.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:51.955+0000 7fb2c9400640 1 -- 172.21.15.6:0/2069584048 shutdown_connections 2024-04-24T23:24:51.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:51.955+0000 7fb2c9400640 1 -- 172.21.15.6:0/2069584048 wait complete. 2024-04-24T23:24:51.990 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 44, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-24T23:15:36.400716+0000", "damaged": [], "data_pools": [ 3, 4 ], "enabled": true, "epoch": 44, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_4306": { "addr": "172.21.15.6:6835/489092393", "addrs": { "addrvec": [ { "addr": "172.21.15.6:6833", "nonce": 489092393, "type": "v2" }, { "addr": "172.21.15.6:6835", "nonce": 489092393, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4306, "incarnation": 4, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 4 }, "gid_5599": { "addr": "172.21.15.190:6839/2643233757", "addrs": { "addrvec": [ { "addr": "172.21.15.190:6836", "nonce": 2643233757, "type": "v2" }, { "addr": "172.21.15.190:6839", "nonce": 2643233757, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5599, "incarnation": 43, "join_fscid": -1, "name": "d", "rank": 1, "state": "up:resolve", "state_seq": 19 } }, "last_failure": 0, "last_failure_osd_epoch": 29, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-04-24T23:24:48.478336+0000", "qdb_cluster": [ 4306 ], "qdb_leader": 4306, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 2, 3, 4 ], "tableserver": 0, "up": { "mds_0": 4306, "mds_1": 5599 } } } ], "standbys": [ { "addr": "172.21.15.6:6834/1190759626", "addrs": { "addrvec": [ { "addr": "172.21.15.6:6832", "nonce": 1190759626, "type": "v2" }, { "addr": "172.21.15.6:6834", "nonce": 1190759626, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 30, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5081, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.190:6838/2416336040", "addrs": { "addrvec": [ { "addr": "172.21.15.190:6835", "nonce": 2416336040, "type": "v2" }, { "addr": "172.21.15.190:6838", "nonce": 2416336040, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 42, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5249, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.6:6837/3928731059", "addrs": { "addrvec": [ { "addr": "172.21.15.6:6836", "nonce": 3928731059, "type": "v2" }, { "addr": "172.21.15.6:6837", "nonce": 3928731059, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 35, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5503, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-04-24T23:24:53.991 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:24:54.412 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-24T23:24:54.424 DEBUG:teuthology.orchestra.run.smithi190:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-24T23:24:54.535 INFO:tasks.ceph.osd.4.smithi190.stderr:2024-04-24T23:24:54.533+0000 7fc8e218d640 -1 received signal: Hangup from killall ceph-osd -1 (PID: 40938) UID: 0 2024-04-24T23:24:54.536 INFO:tasks.ceph.osd.7.smithi190.stderr:2024-04-24T23:24:54.534+0000 7f25d8cd7640 -1 received signal: Hangup from killall ceph-osd -1 (PID: 40938) UID: 0 2024-04-24T23:24:54.536 INFO:tasks.ceph.osd.5.smithi190.stderr:2024-04-24T23:24:54.535+0000 7f8147d41640 -1 received signal: Hangup from killall ceph-osd -1 (PID: 40938) UID: 0 2024-04-24T23:24:54.537 INFO:tasks.ceph.osd.6.smithi190.stderr:2024-04-24T23:24:54.535+0000 7fefcb958640 -1 received signal: Hangup from killall ceph-osd -1 (PID: 40938) UID: 0 2024-04-24T23:24:54.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:54.576+0000 7f785e000640 1 Processor -- start 2024-04-24T23:24:54.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:54.576+0000 7f785e000640 1 -- start start 2024-04-24T23:24:54.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:54.579+0000 7f785e000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f78580cc870 0x7f785816b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:24:54.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:54.579+0000 7f785e000640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f78580ccc50 0x7f785806ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:24:54.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:54.579+0000 7f785e000640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7858164a90 0x7f7858164f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:24:54.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:54.580+0000 7f785e000640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f785816bb30 con 0x7f78580cc870 2024-04-24T23:24:54.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:54.580+0000 7f785e000640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f7858070e70 con 0x7f78580ccc50 2024-04-24T23:24:54.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:54.580+0000 7f785e000640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f78580c3ba0 con 0x7f7858164a90 2024-04-24T23:24:54.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:54.580+0000 7f785d600640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7858164a90 0x7f7858164f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:24:54.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:54.580+0000 7f785d600640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7858164a90 0x7f7858164f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.6:56202/0 (socket says 172.21.15.6:56202) 2024-04-24T23:24:54.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:54.580+0000 7f785d600640 1 -- 172.21.15.6:0/2549698555 learned_addr learned my addr 172.21.15.6:0/2549698555 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:24:54.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:54.666+0000 7f785d600640 1 -- 172.21.15.6:0/2549698555 >> v2:172.21.15.190:3300/0 conn(0x7f78580ccc50 msgr2=0x7f785806ddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:24:54.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:54.580+0000 7f7857e00640 1 --2- 172.21.15.6:0/2549698555 >> v2:172.21.15.190:3300/0 conn(0x7f78580ccc50 0x7f785806ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:24:54.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:54.580+0000 7f785cc00640 1 --2- 172.21.15.6:0/2549698555 >> v2:172.21.15.6:3300/0 conn(0x7f78580cc870 0x7f785816b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:24:54.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:54.666+0000 7f785d600640 1 --2- 172.21.15.6:0/2549698555 >> v2:172.21.15.190:3300/0 conn(0x7f78580ccc50 0x7f785806ddd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:24:54.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:54.666+0000 7f785d600640 1 -- 172.21.15.6:0/2549698555 >> v2:172.21.15.6:3300/0 conn(0x7f78580cc870 msgr2=0x7f785816b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:24:54.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:54.666+0000 7f785d600640 1 --2- 172.21.15.6:0/2549698555 >> v2:172.21.15.6:3300/0 conn(0x7f78580cc870 0x7f785816b5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:24:54.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:54.666+0000 7f785d600640 1 -- 172.21.15.6:0/2549698555 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7858072370 con 0x7f7858164a90 2024-04-24T23:24:54.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:54.668+0000 7f785d600640 1 --2- 172.21.15.6:0/2549698555 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7858164a90 0x7f7858164f00 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f785806f950 tx=0x7f785002cf80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3f3460c192a80236 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:24:54.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:54.668+0000 7f7857400640 1 -- 172.21.15.6:0/2549698555 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7850033070 con 0x7f7858164a90 2024-04-24T23:24:54.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:54.668+0000 7f7857400640 1 -- 172.21.15.6:0/2549698555 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f785002dcd0 con 0x7f7858164a90 2024-04-24T23:24:54.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:54.668+0000 7f7857400640 1 -- 172.21.15.6:0/2549698555 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7850037620 con 0x7f7858164a90 2024-04-24T23:24:54.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:54.669+0000 7f785e000640 1 -- 172.21.15.6:0/2549698555 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7858164a90 msgr2=0x7f7858164f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:24:54.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:54.669+0000 7f785e000640 1 --2- 172.21.15.6:0/2549698555 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7858164a90 0x7f7858164f00 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f785806f950 tx=0x7f785002cf80 comp rx=0 tx=0).stop 2024-04-24T23:24:54.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:54.671+0000 7f785cc00640 1 --2- 172.21.15.6:0/2549698555 >> v2:172.21.15.6:3300/0 conn(0x7f78580cc870 0x7f785816b5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:24:54.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:54.671+0000 7f7857e00640 1 --2- 172.21.15.6:0/2549698555 >> v2:172.21.15.190:3300/0 conn(0x7f78580ccc50 0x7f785806ddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:24:54.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:54.675+0000 7f785e000640 1 -- 172.21.15.6:0/2549698555 shutdown_connections 2024-04-24T23:24:54.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:54.675+0000 7f785e000640 1 --2- 172.21.15.6:0/2549698555 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7858164a90 0x7f7858164f00 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:24:54.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:54.675+0000 7f785e000640 1 --2- 172.21.15.6:0/2549698555 >> v2:172.21.15.190:3300/0 conn(0x7f78580ccc50 0x7f785806ddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:24:54.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:54.675+0000 7f785e000640 1 --2- 172.21.15.6:0/2549698555 >> v2:172.21.15.6:3300/0 conn(0x7f78580cc870 0x7f785816b5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:24:54.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:54.675+0000 7f785e000640 1 -- 172.21.15.6:0/2549698555 >> 172.21.15.6:0/2549698555 conn(0x7f785806ed80 msgr2=0x7f785806f190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:24:54.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:54.675+0000 7f785e000640 1 -- 172.21.15.6:0/2549698555 shutdown_connections 2024-04-24T23:24:54.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:54.675+0000 7f785e000640 1 -- 172.21.15.6:0/2549698555 wait complete. 2024-04-24T23:24:54.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:54.678+0000 7f785e000640 1 Processor -- start 2024-04-24T23:24:54.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:54.688+0000 7f785e000640 1 -- start start 2024-04-24T23:24:54.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:54.690+0000 7f785e000640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f78580cc870 0x7f78580ca350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:24:54.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:54.691+0000 7f785e000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f78580ccc50 0x7f78581fd5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:24:54.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:54.691+0000 7f785cc00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f78580cc870 0x7f78580ca350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:24:54.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:54.692+0000 7f785cc00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f78580cc870 0x7f78580ca350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:51526/0 (socket says 172.21.15.6:51526) 2024-04-24T23:24:54.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:54.693+0000 7f785cc00640 1 -- 172.21.15.6:0/2684528200 learned_addr learned my addr 172.21.15.6:0/2684528200 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:24:54.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:54.694+0000 7f785e000640 1 --2- 172.21.15.6:0/2684528200 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7858164a90 0x7f78581fdb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:24:54.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:54.697+0000 7f785d600640 1 --2- 172.21.15.6:0/2684528200 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7858164a90 0x7f78581fdb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:24:54.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:54.697+0000 7f785e000640 1 -- 172.21.15.6:0/2684528200 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f78580713a0 con 0x7f78580ccc50 2024-04-24T23:24:54.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:54.697+0000 7f7857e00640 1 --2- 172.21.15.6:0/2684528200 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f78580ccc50 0x7f78581fd5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:24:54.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:54.699+0000 7f785e000640 1 -- 172.21.15.6:0/2684528200 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f78580710c0 con 0x7f78580cc870 2024-04-24T23:24:54.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:54.699+0000 7f785e000640 1 -- 172.21.15.6:0/2684528200 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f785806e310 con 0x7f7858164a90 2024-04-24T23:24:54.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:54.702+0000 7f785d600640 1 -- 172.21.15.6:0/2684528200 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f78580cc870 msgr2=0x7f78580ca350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:24:54.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:54.702+0000 7f785d600640 1 --2- 172.21.15.6:0/2684528200 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f78580cc870 0x7f78580ca350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:24:54.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:54.702+0000 7f785d600640 1 -- 172.21.15.6:0/2684528200 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f78580ccc50 msgr2=0x7f78581fd5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:24:54.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:54.703+0000 7f785d600640 1 --2- 172.21.15.6:0/2684528200 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f78580ccc50 0x7f78581fd5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:24:54.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:54.705+0000 7f785d600640 1 -- 172.21.15.6:0/2684528200 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7858168c10 con 0x7f7858164a90 2024-04-24T23:24:54.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:54.706+0000 7f785d600640 1 --2- 172.21.15.6:0/2684528200 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7858164a90 0x7f78581fdb10 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f7850035000 tx=0x7f7850004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:24:54.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:54.706+0000 7f7857e00640 1 --2- 172.21.15.6:0/2684528200 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f78580ccc50 0x7f78581fd5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:24:54.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:54.707+0000 7f7854c00640 1 -- 172.21.15.6:0/2684528200 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7850041610 con 0x7f7858164a90 2024-04-24T23:24:54.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:54.707+0000 7f7854c00640 1 -- 172.21.15.6:0/2684528200 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f78500417b0 con 0x7f7858164a90 2024-04-24T23:24:54.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:54.707+0000 7f7854c00640 1 -- 172.21.15.6:0/2684528200 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7850041ba0 con 0x7f7858164a90 2024-04-24T23:24:54.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:54.707+0000 7f785e000640 1 -- 172.21.15.6:0/2684528200 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7858168ef0 con 0x7f7858164a90 2024-04-24T23:24:54.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:54.709+0000 7f785e000640 1 -- 172.21.15.6:0/2684528200 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f78581694b0 con 0x7f7858164a90 2024-04-24T23:24:54.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:54.711+0000 7f7854c00640 1 -- 172.21.15.6:0/2684528200 <== mon.2 v2:172.21.15.190:3301/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f785003e070 con 0x7f7858164a90 2024-04-24T23:24:54.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:54.713+0000 7f7854c00640 1 --2- 172.21.15.6:0/2684528200 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f783806be80 0x7f783806e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:24:54.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:54.713+0000 7f7854c00640 1 -- 172.21.15.6:0/2684528200 <== mon.2 v2:172.21.15.190:3301/0 5 ==== osd_map(29..29 src has 1..29) ==== 6848+0+0 (secure 0 0 0) 0x7f78500ac9d0 con 0x7f7858164a90 2024-04-24T23:24:54.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:54.713+0000 7f785e000640 1 -- 172.21.15.6:0/2684528200 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7858075b30 con 0x7f7858164a90 2024-04-24T23:24:54.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:54.715+0000 7f785cc00640 1 --2- 172.21.15.6:0/2684528200 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f783806be80 0x7f783806e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:24:54.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:54.717+0000 7f785cc00640 1 --2- 172.21.15.6:0/2684528200 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f783806be80 0x7f783806e340 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f784400aba0 tx=0x7f7844008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:24:54.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:54.719+0000 7f7854c00640 1 -- 172.21.15.6:0/2684528200 <== mon.2 v2:172.21.15.190:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f785007af80 con 0x7f7858164a90 2024-04-24T23:24:54.783 INFO:tasks.ceph.osd.1.smithi006.stderr:2024-04-24T23:24:54.781+0000 7fea96200640 -1 received signal: Hangup from killall ceph-osd -1 (PID: 52785) UID: 0 2024-04-24T23:24:54.785 INFO:tasks.ceph.osd.2.smithi006.stderr:2024-04-24T23:24:54.781+0000 7f0513200640 -1 received signal: Hangup from killall ceph-osd -1 (PID: 52785) UID: 0 2024-04-24T23:24:54.795 INFO:tasks.ceph.osd.3.smithi006.stderr:2024-04-24T23:24:54.789+0000 7f8a80400640 -1 received signal: Hangup from killall ceph-osd -1 (PID: 52785) UID: 0 2024-04-24T23:24:54.795 INFO:tasks.ceph.osd.0.smithi006.stderr:2024-04-24T23:24:54.793+0000 7f0e34600640 -1 Fail to open '/proc/52785/cmdline' error = (2) No such file or directory 2024-04-24T23:24:54.795 INFO:tasks.ceph.osd.0.smithi006.stderr:2024-04-24T23:24:54.793+0000 7f0e34600640 -1 received signal: Hangup from (PID: 52785) UID: 0 2024-04-24T23:24:55.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:55.208+0000 7f785e000640 1 -- 172.21.15.6:0/2684528200 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f785816b640 con 0x7f7858164a90 2024-04-24T23:24:55.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:55.210+0000 7f7854c00640 1 -- 172.21.15.6:0/2684528200 <== mon.2 v2:172.21.15.190:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) ==== 94+0+6026 (secure 0 0 0) 0x7f785007a880 con 0x7f7858164a90 2024-04-24T23:24:55.211 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:24:55.212 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":45,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5081,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/1190759626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1190759626},{"type":"v1","addr":"172.21.15.6:6834","nonce":1190759626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":5249,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6838/2416336040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2416336040},{"type":"v1","addr":"172.21.15.190:6838","nonce":2416336040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":42},{"gid":5503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/3928731059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":3928731059},{"type":"v1","addr":"172.21.15.6:6837","nonce":3928731059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:24:54.115762+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":4306,"mds_1":5599},"failed":[],"damaged":[],"stopped":[2,3,4],"info":{"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/489092393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":489092393},{"type":"v1","addr":"172.21.15.6:6835","nonce":489092393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5599":{"gid":5599,"name":"d","rank":1,"incarnation":43,"state":"up:reconnect","state_seq":21,"addr":"172.21.15.190:6839/2643233757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":2643233757},{"type":"v1","addr":"172.21.15.190:6839","nonce":2643233757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4306,"qdb_cluster":[4306]},"id":1}]} 2024-04-24T23:24:55.212 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 45 2024-04-24T23:24:55.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:55.214+0000 7f785e000640 1 -- 172.21.15.6:0/2684528200 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f783806be80 msgr2=0x7f783806e340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:24:55.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:55.214+0000 7f785e000640 1 --2- 172.21.15.6:0/2684528200 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f783806be80 0x7f783806e340 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f784400aba0 tx=0x7f7844008040 comp rx=0 tx=0).stop 2024-04-24T23:24:55.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:55.214+0000 7f785e000640 1 -- 172.21.15.6:0/2684528200 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7858164a90 msgr2=0x7f78581fdb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:24:55.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:55.214+0000 7f785e000640 1 --2- 172.21.15.6:0/2684528200 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7858164a90 0x7f78581fdb10 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f7850035000 tx=0x7f7850004a40 comp rx=0 tx=0).stop 2024-04-24T23:24:55.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:55.214+0000 7f785e000640 1 -- 172.21.15.6:0/2684528200 shutdown_connections 2024-04-24T23:24:55.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:55.214+0000 7f785e000640 1 --2- 172.21.15.6:0/2684528200 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f783806be80 0x7f783806e340 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:24:55.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:55.214+0000 7f785e000640 1 --2- 172.21.15.6:0/2684528200 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7858164a90 0x7f78581fdb10 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:24:55.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:55.214+0000 7f785e000640 1 --2- 172.21.15.6:0/2684528200 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f78580ccc50 0x7f78581fd5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:24:55.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:55.214+0000 7f785e000640 1 --2- 172.21.15.6:0/2684528200 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f78580cc870 0x7f78580ca350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:24:55.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:55.214+0000 7f785e000640 1 -- 172.21.15.6:0/2684528200 >> 172.21.15.6:0/2684528200 conn(0x7f785806ed80 msgr2=0x7f78580cb9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:24:55.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:55.217+0000 7f785e000640 1 -- 172.21.15.6:0/2684528200 shutdown_connections 2024-04-24T23:24:55.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:55.218+0000 7f785e000640 1 -- 172.21.15.6:0/2684528200 wait complete. 2024-04-24T23:24:55.252 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-24T23:24:57.253 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:24:57.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:57.941+0000 7f3da5400640 1 Processor -- start 2024-04-24T23:24:57.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:57.941+0000 7f3da5400640 1 -- start start 2024-04-24T23:24:57.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:57.943+0000 7f3da5400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f3d980d7020 0x7f3d9804c470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:24:57.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:57.943+0000 7f3da5400640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3d980e0cd0 0x7f3d980dd1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:24:57.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:57.943+0000 7f3da5400640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d9804bb80 con 0x7f3d980e0890 2024-04-24T23:24:57.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:57.943+0000 7f3da5400640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f3d98034cd0 con 0x7f3d980d7020 2024-04-24T23:24:57.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:57.943+0000 7f3da5400640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f3d9804b930 con 0x7f3d980e0cd0 2024-04-24T23:24:57.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:57.944+0000 7f3d9e800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f3d980d7020 0x7f3d9804c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:24:57.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:57.944+0000 7f3d9de00640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f3d980e0890 0x7f3d980dca90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.6:54106/0 (socket says 172.21.15.6:54106) 2024-04-24T23:24:57.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:57.944+0000 7f3d9de00640 1 -- 172.21.15.6:0/2829299832 learned_addr learned my addr 172.21.15.6:0/2829299832 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:24:57.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:57.944+0000 7f3d9f200640 1 --2- 172.21.15.6:0/2829299832 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3d980e0cd0 0x7f3d980dd1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:24:57.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:57.944+0000 7f3d9e800640 1 -- 172.21.15.6:0/2829299832 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3d980e0cd0 msgr2=0x7f3d980dd1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:24:57.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:57.944+0000 7f3d9e800640 1 --2- 172.21.15.6:0/2829299832 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3d980e0cd0 0x7f3d980dd1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:24:57.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:57.944+0000 7f3d9e800640 1 -- 172.21.15.6:0/2829299832 >> v1:172.21.15.190:6789/0 conn(0x7f3d980e0890 legacy=0x7f3d980dca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:24:57.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:57.944+0000 7f3d9e800640 1 -- 172.21.15.6:0/2829299832 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d9804be60 con 0x7f3d980d7020 2024-04-24T23:24:57.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:57.946+0000 7f3d9e800640 1 --2- 172.21.15.6:0/2829299832 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3d980d7020 0x7f3d9804c470 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f3d8c0097a0 tx=0x7f3d8c02a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c49fa0910551f717 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:24:57.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:57.947+0000 7f3d9d400640 1 -- 172.21.15.6:0/2829299832 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d8c037070 con 0x7f3d980d7020 2024-04-24T23:24:57.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:57.947+0000 7f3d9d400640 1 -- 172.21.15.6:0/2829299832 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d8c032440 con 0x7f3d980d7020 2024-04-24T23:24:57.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:57.947+0000 7f3d9d400640 1 -- 172.21.15.6:0/2829299832 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d8c032810 con 0x7f3d980d7020 2024-04-24T23:24:57.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:57.948+0000 7f3da5400640 1 -- 172.21.15.6:0/2829299832 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3d980d7020 msgr2=0x7f3d9804c470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:24:57.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:57.948+0000 7f3da5400640 1 --2- 172.21.15.6:0/2829299832 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3d980d7020 0x7f3d9804c470 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f3d8c0097a0 tx=0x7f3d8c02a1c0 comp rx=0 tx=0).stop 2024-04-24T23:24:57.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:57.948+0000 7f3da5400640 1 -- 172.21.15.6:0/2829299832 shutdown_connections 2024-04-24T23:24:57.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:57.948+0000 7f3da5400640 1 --2- 172.21.15.6:0/2829299832 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3d980e0cd0 0x7f3d980dd1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:24:57.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:57.948+0000 7f3da5400640 1 --2- 172.21.15.6:0/2829299832 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3d980d7020 0x7f3d9804c470 secure :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f3d8c0097a0 tx=0x7f3d8c02a1c0 comp rx=0 tx=0).stop 2024-04-24T23:24:57.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:57.948+0000 7f3da5400640 1 -- 172.21.15.6:0/2829299832 >> 172.21.15.6:0/2829299832 conn(0x7f3d98050180 msgr2=0x7f3d98050590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:24:57.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:57.950+0000 7f3da5400640 1 -- 172.21.15.6:0/2829299832 shutdown_connections 2024-04-24T23:24:57.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:57.950+0000 7f3da5400640 1 -- 172.21.15.6:0/2829299832 wait complete. 2024-04-24T23:24:57.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:57.959+0000 7f3da5400640 1 Processor -- start 2024-04-24T23:24:57.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:57.959+0000 7f3da5400640 1 -- start start 2024-04-24T23:24:57.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:57.964+0000 7f3da5400640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3d980e0890 0x7f3d980d3eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:24:57.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:57.965+0000 7f3d9e800640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3d980e0890 0x7f3d980d3eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:24:57.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:57.965+0000 7f3d9e800640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3d980e0890 0x7f3d980d3eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:50984/0 (socket says 172.21.15.6:50984) 2024-04-24T23:24:57.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:57.965+0000 7f3d9e800640 1 -- 172.21.15.6:0/1788971649 learned_addr learned my addr 172.21.15.6:0/1788971649 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:24:57.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:57.965+0000 7f3da5400640 1 --2- 172.21.15.6:0/1788971649 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3d980e0cd0 0x7f3d98173260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:24:57.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:57.965+0000 7f3da5400640 1 --2- 172.21.15.6:0/1788971649 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3d980dd590 0x7f3d980cacf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:24:57.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:57.966+0000 7f3d9de00640 1 --2- 172.21.15.6:0/1788971649 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3d980e0cd0 0x7f3d98173260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:24:57.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:57.966+0000 7f3da5400640 1 -- 172.21.15.6:0/1788971649 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f3d980e0090 con 0x7f3d980dd590 2024-04-24T23:24:57.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:57.967+0000 7f3da5400640 1 -- 172.21.15.6:0/1788971649 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f3d980e0400 con 0x7f3d980e0890 2024-04-24T23:24:57.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:57.967+0000 7f3da5400640 1 -- 172.21.15.6:0/1788971649 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f3d98005ee0 con 0x7f3d980e0cd0 2024-04-24T23:24:57.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:57.967+0000 7f3d9f200640 1 --2- 172.21.15.6:0/1788971649 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3d980dd590 0x7f3d980cacf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:24:57.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:57.968+0000 7f3d9e800640 1 -- 172.21.15.6:0/1788971649 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3d980e0cd0 msgr2=0x7f3d98173260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:24:57.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:57.968+0000 7f3d9e800640 1 --2- 172.21.15.6:0/1788971649 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3d980e0cd0 0x7f3d98173260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:24:57.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:57.968+0000 7f3d9e800640 1 -- 172.21.15.6:0/1788971649 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3d980dd590 msgr2=0x7f3d980cacf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:24:57.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:57.969+0000 7f3d9e800640 1 --2- 172.21.15.6:0/1788971649 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3d980dd590 0x7f3d980cacf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:24:57.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:57.970+0000 7f3d9e800640 1 -- 172.21.15.6:0/1788971649 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d98172aa0 con 0x7f3d980e0890 2024-04-24T23:24:57.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:57.971+0000 7f3d9f200640 1 --2- 172.21.15.6:0/1788971649 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3d980dd590 0x7f3d980cacf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:24:57.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:57.972+0000 7f3d9e800640 1 --2- 172.21.15.6:0/1788971649 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3d980e0890 0x7f3d980d3eb0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f3d8c02a730 tx=0x7f3d8c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:24:57.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:57.976+0000 7f3d96a00640 1 -- 172.21.15.6:0/1788971649 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d8c041020 con 0x7f3d980e0890 2024-04-24T23:24:57.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:57.978+0000 7f3d96a00640 1 -- 172.21.15.6:0/1788971649 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d8c030740 con 0x7f3d980e0890 2024-04-24T23:24:57.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:57.978+0000 7f3da5400640 1 -- 172.21.15.6:0/1788971649 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d980d2230 con 0x7f3d980e0890 2024-04-24T23:24:57.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:57.979+0000 7f3da5400640 1 -- 172.21.15.6:0/1788971649 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d980dacb0 con 0x7f3d980e0890 2024-04-24T23:24:57.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:57.979+0000 7f3d96a00640 1 -- 172.21.15.6:0/1788971649 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d8c037040 con 0x7f3d980e0890 2024-04-24T23:24:57.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:57.987+0000 7f3d96a00640 1 -- 172.21.15.6:0/1788971649 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f3d8c044020 con 0x7f3d980e0890 2024-04-24T23:24:57.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:57.987+0000 7f3d8be00640 1 -- 172.21.15.6:0/1788971649 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d74005290 con 0x7f3d980e0890 2024-04-24T23:24:57.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:57.989+0000 7f3d96a00640 1 --2- 172.21.15.6:0/1788971649 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f3d8006be30 0x7f3d8006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:24:57.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:57.989+0000 7f3d96a00640 1 -- 172.21.15.6:0/1788971649 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6848+0+0 (secure 0 0 0) 0x7f3d8c0abf30 con 0x7f3d980e0890 2024-04-24T23:24:57.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:57.992+0000 7f3d9de00640 1 --2- 172.21.15.6:0/1788971649 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f3d8006be30 0x7f3d8006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:24:57.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:57.994+0000 7f3d9de00640 1 --2- 172.21.15.6:0/1788971649 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f3d8006be30 0x7f3d8006e2f0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f3d90007c90 tx=0x7f3d90007be0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:24:57.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:57.997+0000 7f3d96a00640 1 -- 172.21.15.6:0/1788971649 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f3d8c04b030 con 0x7f3d980e0890 2024-04-24T23:24:58.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:58.677+0000 7f3d8be00640 1 -- 172.21.15.6:0/1788971649 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3d74005580 con 0x7f3d980e0890 2024-04-24T23:24:58.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:58.686+0000 7f3d96a00640 1 -- 172.21.15.6:0/1788971649 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) ==== 94+0+6026 (secure 0 0 0) 0x7f3d8c07a3f0 con 0x7f3d980e0890 2024-04-24T23:24:58.688 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:24:58.688 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":45,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5081,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/1190759626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1190759626},{"type":"v1","addr":"172.21.15.6:6834","nonce":1190759626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":5249,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6838/2416336040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2416336040},{"type":"v1","addr":"172.21.15.190:6838","nonce":2416336040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":42},{"gid":5503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/3928731059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":3928731059},{"type":"v1","addr":"172.21.15.6:6837","nonce":3928731059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:24:54.115762+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":4306,"mds_1":5599},"failed":[],"damaged":[],"stopped":[2,3,4],"info":{"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/489092393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":489092393},{"type":"v1","addr":"172.21.15.6:6835","nonce":489092393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5599":{"gid":5599,"name":"d","rank":1,"incarnation":43,"state":"up:reconnect","state_seq":21,"addr":"172.21.15.190:6839/2643233757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":2643233757},{"type":"v1","addr":"172.21.15.190:6839","nonce":2643233757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4306,"qdb_cluster":[4306]},"id":1}]} 2024-04-24T23:24:58.688 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 45 2024-04-24T23:24:58.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:58.691+0000 7f3d8be00640 1 -- 172.21.15.6:0/1788971649 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f3d8006be30 msgr2=0x7f3d8006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:24:58.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:58.691+0000 7f3d8be00640 1 --2- 172.21.15.6:0/1788971649 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f3d8006be30 0x7f3d8006e2f0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f3d90007c90 tx=0x7f3d90007be0 comp rx=0 tx=0).stop 2024-04-24T23:24:58.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:58.691+0000 7f3d8be00640 1 -- 172.21.15.6:0/1788971649 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3d980e0890 msgr2=0x7f3d980d3eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:24:58.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:58.691+0000 7f3d8be00640 1 --2- 172.21.15.6:0/1788971649 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3d980e0890 0x7f3d980d3eb0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f3d8c02a730 tx=0x7f3d8c0046c0 comp rx=0 tx=0).stop 2024-04-24T23:24:58.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:58.692+0000 7f3d8be00640 1 -- 172.21.15.6:0/1788971649 shutdown_connections 2024-04-24T23:24:58.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:58.692+0000 7f3d8be00640 1 --2- 172.21.15.6:0/1788971649 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f3d8006be30 0x7f3d8006e2f0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:24:58.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:58.692+0000 7f3d8be00640 1 --2- 172.21.15.6:0/1788971649 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3d980e0cd0 0x7f3d98173260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:24:58.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:58.692+0000 7f3d8be00640 1 --2- 172.21.15.6:0/1788971649 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3d980dd590 0x7f3d980cacf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:24:58.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:58.692+0000 7f3d8be00640 1 --2- 172.21.15.6:0/1788971649 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3d980e0890 0x7f3d980d3eb0 secure :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f3d8c02a730 tx=0x7f3d8c0046c0 comp rx=0 tx=0).stop 2024-04-24T23:24:58.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:58.692+0000 7f3d8be00640 1 -- 172.21.15.6:0/1788971649 >> 172.21.15.6:0/1788971649 conn(0x7f3d98050180 msgr2=0x7f3d98006ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:24:58.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:58.696+0000 7f3d8be00640 1 -- 172.21.15.6:0/1788971649 shutdown_connections 2024-04-24T23:24:58.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:24:58.696+0000 7f3d8be00640 1 -- 172.21.15.6:0/1788971649 wait complete. 2024-04-24T23:24:58.724 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-24T23:25:00.725 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:25:01.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:01.103+0000 7fb835c00640 1 Processor -- start 2024-04-24T23:25:01.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:01.104+0000 7fb835c00640 1 -- start start 2024-04-24T23:25:01.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:01.108+0000 7fb835c00640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fb8301593b0 0x7fb8300c6e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:25:01.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:01.108+0000 7fb835c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fb83006eba0 0x7fb8300c8fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:25:01.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:01.108+0000 7fb835c00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb83006f010 0x7fb8300c9510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:25:01.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:01.109+0000 7fb82ea00640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fb8301593b0 0x7fb8300c6e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:25:01.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:01.109+0000 7fb82ea00640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fb8301593b0 0x7fb8300c6e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:50996/0 (socket says 172.21.15.6:50996) 2024-04-24T23:25:01.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:01.112+0000 7fb835c00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7fb83006e710 con 0x7fb83006eba0 2024-04-24T23:25:01.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:01.112+0000 7fb835c00640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7fb830071880 con 0x7fb8301593b0 2024-04-24T23:25:01.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:01.112+0000 7fb835c00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fb83015f200 con 0x7fb83006f010 2024-04-24T23:25:01.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:01.112+0000 7fb82ea00640 1 -- 172.21.15.6:0/3450091913 learned_addr learned my addr 172.21.15.6:0/3450091913 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:25:01.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:01.112+0000 7fb82e000640 1 --2- 172.21.15.6:0/3450091913 >> v2:172.21.15.6:3300/0 conn(0x7fb83006eba0 0x7fb8300c8fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:25:01.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:01.112+0000 7fb82f400640 1 --2- 172.21.15.6:0/3450091913 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb83006f010 0x7fb8300c9510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:25:01.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:01.113+0000 7fb82ea00640 1 -- 172.21.15.6:0/3450091913 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb83006f010 msgr2=0x7fb8300c9510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:25:01.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:01.113+0000 7fb82ea00640 1 --2- 172.21.15.6:0/3450091913 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb83006f010 0x7fb8300c9510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:01.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:01.113+0000 7fb82ea00640 1 -- 172.21.15.6:0/3450091913 >> v2:172.21.15.6:3300/0 conn(0x7fb83006eba0 msgr2=0x7fb8300c8fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:25:01.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:01.113+0000 7fb82ea00640 1 --2- 172.21.15.6:0/3450091913 >> v2:172.21.15.6:3300/0 conn(0x7fb83006eba0 0x7fb8300c8fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:01.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:01.113+0000 7fb82ea00640 1 -- 172.21.15.6:0/3450091913 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb83015f4c0 con 0x7fb8301593b0 2024-04-24T23:25:01.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:01.115+0000 7fb82ea00640 1 --2- 172.21.15.6:0/3450091913 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb8301593b0 0x7fb8300c6e10 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fb83006f540 tx=0x7fb81c02a280 comp rx=0 tx=0).ready entity=mon.1 client_cookie=369cd3e709c53117 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:25:01.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:01.116+0000 7fb82d600640 1 -- 172.21.15.6:0/3450091913 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb81c037070 con 0x7fb8301593b0 2024-04-24T23:25:01.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:01.116+0000 7fb82d600640 1 -- 172.21.15.6:0/3450091913 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb81c032440 con 0x7fb8301593b0 2024-04-24T23:25:01.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:01.116+0000 7fb82d600640 1 -- 172.21.15.6:0/3450091913 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb81c037070 con 0x7fb8301593b0 2024-04-24T23:25:01.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:01.117+0000 7fb835c00640 1 -- 172.21.15.6:0/3450091913 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb8301593b0 msgr2=0x7fb8300c6e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:25:01.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:01.117+0000 7fb835c00640 1 --2- 172.21.15.6:0/3450091913 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb8301593b0 0x7fb8300c6e10 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fb83006f540 tx=0x7fb81c02a280 comp rx=0 tx=0).stop 2024-04-24T23:25:01.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:01.118+0000 7fb835c00640 1 -- 172.21.15.6:0/3450091913 shutdown_connections 2024-04-24T23:25:01.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:01.118+0000 7fb835c00640 1 --2- 172.21.15.6:0/3450091913 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb83006f010 0x7fb8300c9510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:01.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:01.118+0000 7fb835c00640 1 --2- 172.21.15.6:0/3450091913 >> v2:172.21.15.6:3300/0 conn(0x7fb83006eba0 0x7fb8300c8fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:01.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:01.118+0000 7fb835c00640 1 --2- 172.21.15.6:0/3450091913 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb8301593b0 0x7fb8300c6e10 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:01.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:01.118+0000 7fb835c00640 1 -- 172.21.15.6:0/3450091913 >> 172.21.15.6:0/3450091913 conn(0x7fb83006b0e0 msgr2=0x7fb83006b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:25:01.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:01.118+0000 7fb835c00640 1 -- 172.21.15.6:0/3450091913 shutdown_connections 2024-04-24T23:25:01.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:01.118+0000 7fb835c00640 1 -- 172.21.15.6:0/3450091913 wait complete. 2024-04-24T23:25:01.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:01.122+0000 7fb835c00640 1 Processor -- start 2024-04-24T23:25:01.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:01.123+0000 7fb835c00640 1 -- start start 2024-04-24T23:25:01.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:01.124+0000 7fb835c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb83006eba0 0x7fb8301491a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:25:01.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:01.124+0000 7fb835c00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb83006f010 0x7fb83014d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:25:01.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:01.125+0000 7fb835c00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb8301593b0 0x7fb83014dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:25:01.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:01.125+0000 7fb82e000640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb83006f010 0x7fb83014d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:25:01.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:01.125+0000 7fb82e000640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb83006f010 0x7fb83014d680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:51004/0 (socket says 172.21.15.6:51004) 2024-04-24T23:25:01.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:01.125+0000 7fb82ea00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb83006eba0 0x7fb8301491a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:25:01.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:01.125+0000 7fb82ea00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb83006eba0 0x7fb8301491a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:33816/0 (socket says 172.21.15.6:33816) 2024-04-24T23:25:01.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:01.125+0000 7fb82f400640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb8301593b0 0x7fb83014dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:25:01.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:01.125+0000 7fb82f400640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb8301593b0 0x7fb83014dbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.6:60426/0 (socket says 172.21.15.6:60426) 2024-04-24T23:25:01.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:01.126+0000 7fb82e000640 1 -- 172.21.15.6:0/2080461389 learned_addr learned my addr 172.21.15.6:0/2080461389 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:25:01.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:01.126+0000 7fb835c00640 1 -- 172.21.15.6:0/2080461389 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fb83015fd60 con 0x7fb83006eba0 2024-04-24T23:25:01.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:01.126+0000 7fb835c00640 1 -- 172.21.15.6:0/2080461389 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fb83015fa80 con 0x7fb83006f010 2024-04-24T23:25:01.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:01.126+0000 7fb835c00640 1 -- 172.21.15.6:0/2080461389 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fb83006e3a0 con 0x7fb8301593b0 2024-04-24T23:25:01.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:01.127+0000 7fb82e000640 1 -- 172.21.15.6:0/2080461389 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb8301593b0 msgr2=0x7fb83014dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:25:01.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:01.127+0000 7fb82e000640 1 --2- 172.21.15.6:0/2080461389 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb8301593b0 0x7fb83014dbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:01.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:01.127+0000 7fb82e000640 1 -- 172.21.15.6:0/2080461389 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb83006eba0 msgr2=0x7fb8301491a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:25:01.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:01.127+0000 7fb82e000640 1 --2- 172.21.15.6:0/2080461389 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb83006eba0 0x7fb8301491a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:01.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:01.127+0000 7fb82e000640 1 -- 172.21.15.6:0/2080461389 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8300c4330 con 0x7fb83006f010 2024-04-24T23:25:01.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:01.127+0000 7fb82f400640 1 --2- 172.21.15.6:0/2080461389 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb8301593b0 0x7fb83014dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:25:01.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:01.127+0000 7fb82ea00640 1 --2- 172.21.15.6:0/2080461389 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb83006eba0 0x7fb8301491a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:25:01.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:01.128+0000 7fb82e000640 1 --2- 172.21.15.6:0/2080461389 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb83006f010 0x7fb83014d680 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fb82000e870 tx=0x7fb82000ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:25:01.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:01.128+0000 7fb826a00640 1 -- 172.21.15.6:0/2080461389 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb820017070 con 0x7fb83006f010 2024-04-24T23:25:01.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:01.128+0000 7fb826a00640 1 -- 172.21.15.6:0/2080461389 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb82000ee60 con 0x7fb83006f010 2024-04-24T23:25:01.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:01.128+0000 7fb826a00640 1 -- 172.21.15.6:0/2080461389 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb82000cb80 con 0x7fb83006f010 2024-04-24T23:25:01.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:01.129+0000 7fb835c00640 1 -- 172.21.15.6:0/2080461389 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8300c45d0 con 0x7fb83006f010 2024-04-24T23:25:01.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:01.129+0000 7fb835c00640 1 -- 172.21.15.6:0/2080461389 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8300c4b90 con 0x7fb83006f010 2024-04-24T23:25:01.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:01.131+0000 7fb835c00640 1 -- 172.21.15.6:0/2080461389 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb8300767b0 con 0x7fb83006f010 2024-04-24T23:25:01.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:01.132+0000 7fb826a00640 1 -- 172.21.15.6:0/2080461389 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fb82001a070 con 0x7fb83006f010 2024-04-24T23:25:01.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:01.133+0000 7fb826a00640 1 --2- 172.21.15.6:0/2080461389 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fb81006be30 0x7fb81006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:25:01.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:01.133+0000 7fb826a00640 1 -- 172.21.15.6:0/2080461389 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6848+0+0 (secure 0 0 0) 0x7fb82008b3e0 con 0x7fb83006f010 2024-04-24T23:25:01.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:01.134+0000 7fb82ea00640 1 --2- 172.21.15.6:0/2080461389 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fb81006be30 0x7fb81006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:25:01.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:01.135+0000 7fb82ea00640 1 --2- 172.21.15.6:0/2080461389 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fb81006be30 0x7fb81006e2f0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fb81c034000 tx=0x7fb81c02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:25:01.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:01.142+0000 7fb826a00640 1 -- 172.21.15.6:0/2080461389 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fb8200592e0 con 0x7fb83006f010 2024-04-24T23:25:01.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:01.726+0000 7fb835c00640 1 -- 172.21.15.6:0/2080461389 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb8300c8130 con 0x7fb83006f010 2024-04-24T23:25:01.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:01.727+0000 7fb826a00640 1 -- 172.21.15.6:0/2080461389 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 46 v46) ==== 94+0+6023 (secure 0 0 0) 0x7fb82005d5e0 con 0x7fb83006f010 2024-04-24T23:25:01.728 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:25:01.728 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":46,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5081,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/1190759626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1190759626},{"type":"v1","addr":"172.21.15.6:6834","nonce":1190759626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":5249,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6838/2416336040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2416336040},{"type":"v1","addr":"172.21.15.190:6838","nonce":2416336040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":42},{"gid":5503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/3928731059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":3928731059},{"type":"v1","addr":"172.21.15.6:6837","nonce":3928731059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":46,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:24:59.153387+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":4306,"mds_1":5599},"failed":[],"damaged":[],"stopped":[2,3,4],"info":{"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/489092393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":489092393},{"type":"v1","addr":"172.21.15.6:6835","nonce":489092393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5599":{"gid":5599,"name":"d","rank":1,"incarnation":43,"state":"up:rejoin","state_seq":23,"addr":"172.21.15.190:6839/2643233757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":2643233757},{"type":"v1","addr":"172.21.15.190:6839","nonce":2643233757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4306,"qdb_cluster":[4306]},"id":1}]} 2024-04-24T23:25:01.728 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 46 2024-04-24T23:25:01.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:01.740+0000 7fb835c00640 1 -- 172.21.15.6:0/2080461389 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fb81006be30 msgr2=0x7fb81006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:25:01.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:01.740+0000 7fb835c00640 1 --2- 172.21.15.6:0/2080461389 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fb81006be30 0x7fb81006e2f0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fb81c034000 tx=0x7fb81c02e000 comp rx=0 tx=0).stop 2024-04-24T23:25:01.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:01.740+0000 7fb835c00640 1 -- 172.21.15.6:0/2080461389 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb83006f010 msgr2=0x7fb83014d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:25:01.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:01.740+0000 7fb835c00640 1 --2- 172.21.15.6:0/2080461389 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb83006f010 0x7fb83014d680 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fb82000e870 tx=0x7fb82000ed30 comp rx=0 tx=0).stop 2024-04-24T23:25:01.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:01.740+0000 7fb835c00640 1 -- 172.21.15.6:0/2080461389 shutdown_connections 2024-04-24T23:25:01.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:01.740+0000 7fb835c00640 1 --2- 172.21.15.6:0/2080461389 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fb81006be30 0x7fb81006e2f0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:01.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:01.740+0000 7fb835c00640 1 --2- 172.21.15.6:0/2080461389 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb8301593b0 0x7fb83014dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:01.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:01.740+0000 7fb835c00640 1 --2- 172.21.15.6:0/2080461389 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb83006f010 0x7fb83014d680 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:01.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:01.740+0000 7fb835c00640 1 --2- 172.21.15.6:0/2080461389 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb83006eba0 0x7fb8301491a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:01.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:01.740+0000 7fb835c00640 1 -- 172.21.15.6:0/2080461389 >> 172.21.15.6:0/2080461389 conn(0x7fb83006b0e0 msgr2=0x7fb83014b860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:25:01.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:01.741+0000 7fb835c00640 1 -- 172.21.15.6:0/2080461389 shutdown_connections 2024-04-24T23:25:01.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:01.741+0000 7fb835c00640 1 -- 172.21.15.6:0/2080461389 wait complete. 2024-04-24T23:25:01.790 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-24T23:25:03.791 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:25:04.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:04.021+0000 7fe521400640 1 Processor -- start 2024-04-24T23:25:04.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:04.021+0000 7fe521400640 1 -- start start 2024-04-24T23:25:04.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:04.022+0000 7fe521400640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe51c0cc870 0x7fe51c16b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:25:04.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:04.022+0000 7fe521400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fe51c0ccc50 0x7fe51c06ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:25:04.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:04.022+0000 7fe521400640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe51c072370 con 0x7fe51c164a90 2024-04-24T23:25:04.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:04.022+0000 7fe521400640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7fe51c16bb30 con 0x7fe51c0ccc50 2024-04-24T23:25:04.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:04.022+0000 7fe521400640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fe51c070e70 con 0x7fe51c0cc870 2024-04-24T23:25:04.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:04.022+0000 7fe51be00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe51c0cc870 0x7fe51c16b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:25:04.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:04.022+0000 7fe51be00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe51c0cc870 0x7fe51c16b5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.6:60432/0 (socket says 172.21.15.6:60432) 2024-04-24T23:25:04.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:04.022+0000 7fe51be00640 1 -- 172.21.15.6:0/604721645 learned_addr learned my addr 172.21.15.6:0/604721645 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:25:04.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:04.023+0000 7fe51aa00640 1 -- 172.21.15.6:0/604721645 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 906440222 0 0) 0x7fe51c072370 con 0x7fe51c164a90 2024-04-24T23:25:04.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:04.252+0000 7fe51aa00640 1 -- 172.21.15.6:0/604721645 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe500003520 con 0x7fe51c164a90 2024-04-24T23:25:04.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:04.253+0000 7fe51be00640 1 -- 172.21.15.6:0/604721645 >> v2:172.21.15.6:3300/0 conn(0x7fe51c0ccc50 msgr2=0x7fe51c06ddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:25:04.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:04.023+0000 7fe51b400640 1 --2- 172.21.15.6:0/604721645 >> v2:172.21.15.6:3300/0 conn(0x7fe51c0ccc50 0x7fe51c06ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:25:04.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:04.254+0000 7fe51be00640 1 --2- 172.21.15.6:0/604721645 >> v2:172.21.15.6:3300/0 conn(0x7fe51c0ccc50 0x7fe51c06ddd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:04.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:04.254+0000 7fe51be00640 1 -- 172.21.15.6:0/604721645 >> v1:172.21.15.6:6789/0 conn(0x7fe51c164a90 legacy=0x7fe51c167ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:25:04.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:04.254+0000 7fe51be00640 1 -- 172.21.15.6:0/604721645 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe51c072650 con 0x7fe51c0cc870 2024-04-24T23:25:04.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:04.254+0000 7fe51b400640 1 --2- 172.21.15.6:0/604721645 >> v2:172.21.15.6:3300/0 conn(0x7fe51c0ccc50 0x7fe51c06ddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:25:04.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:04.260+0000 7fe51be00640 1 --2- 172.21.15.6:0/604721645 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe51c0cc870 0x7fe51c16b5f0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fe50c007860 tx=0x7fe50c02b3c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5ba26ce89ab01af2 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:25:04.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:04.260+0000 7fe51aa00640 1 -- 172.21.15.6:0/604721645 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe50c0335f0 con 0x7fe51c0cc870 2024-04-24T23:25:04.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:04.260+0000 7fe51aa00640 1 -- 172.21.15.6:0/604721645 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe50c00ae50 con 0x7fe51c0cc870 2024-04-24T23:25:04.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:04.260+0000 7fe51aa00640 1 -- 172.21.15.6:0/604721645 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe50c0339b0 con 0x7fe51c0cc870 2024-04-24T23:25:04.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:04.262+0000 7fe521400640 1 -- 172.21.15.6:0/604721645 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe51c0cc870 msgr2=0x7fe51c16b5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:25:04.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:04.262+0000 7fe521400640 1 --2- 172.21.15.6:0/604721645 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe51c0cc870 0x7fe51c16b5f0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fe50c007860 tx=0x7fe50c02b3c0 comp rx=0 tx=0).stop 2024-04-24T23:25:04.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:04.263+0000 7fe521400640 1 -- 172.21.15.6:0/604721645 shutdown_connections 2024-04-24T23:25:04.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:04.263+0000 7fe521400640 1 --2- 172.21.15.6:0/604721645 >> v2:172.21.15.6:3300/0 conn(0x7fe51c0ccc50 0x7fe51c06ddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:04.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:04.263+0000 7fe521400640 1 --2- 172.21.15.6:0/604721645 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe51c0cc870 0x7fe51c16b5f0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:04.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:04.264+0000 7fe521400640 1 -- 172.21.15.6:0/604721645 >> 172.21.15.6:0/604721645 conn(0x7fe51c06ed80 msgr2=0x7fe51c06f190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:25:04.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:04.270+0000 7fe521400640 1 -- 172.21.15.6:0/604721645 shutdown_connections 2024-04-24T23:25:04.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:04.270+0000 7fe521400640 1 -- 172.21.15.6:0/604721645 wait complete. 2024-04-24T23:25:04.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:04.282+0000 7fe521400640 1 Processor -- start 2024-04-24T23:25:04.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:04.282+0000 7fe521400640 1 -- start start 2024-04-24T23:25:04.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:04.294+0000 7fe521400640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe51c0cc870 0x7fe51c1fcaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:25:04.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:04.295+0000 7fe521400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe51c0ccc50 0x7fe51c0bab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:25:04.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:04.295+0000 7fe521400640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe51c164a90 0x7fe51c162210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:25:04.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:04.295+0000 7fe521400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fe51c0713a0 con 0x7fe51c0ccc50 2024-04-24T23:25:04.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:04.295+0000 7fe521400640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fe51c0c3ba0 con 0x7fe51c0cc870 2024-04-24T23:25:04.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:04.296+0000 7fe521400640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fe51c06e310 con 0x7fe51c164a90 2024-04-24T23:25:04.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:04.299+0000 7fe520a00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe51c164a90 0x7fe51c162210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:25:04.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:04.299+0000 7fe51be00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe51c0cc870 0x7fe51c1fcaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:25:04.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:04.300+0000 7fe51be00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe51c0cc870 0x7fe51c1fcaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:51016/0 (socket says 172.21.15.6:51016) 2024-04-24T23:25:04.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:04.300+0000 7fe51be00640 1 -- 172.21.15.6:0/4026324770 learned_addr learned my addr 172.21.15.6:0/4026324770 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:25:04.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:04.299+0000 7fe51b400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe51c0ccc50 0x7fe51c0bab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:25:04.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:04.302+0000 7fe51be00640 1 -- 172.21.15.6:0/4026324770 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe51c164a90 msgr2=0x7fe51c162210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:25:04.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:04.302+0000 7fe51be00640 1 --2- 172.21.15.6:0/4026324770 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe51c164a90 0x7fe51c162210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:04.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:04.302+0000 7fe51be00640 1 -- 172.21.15.6:0/4026324770 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe51c0ccc50 msgr2=0x7fe51c0bab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:25:04.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:04.302+0000 7fe51be00640 1 --2- 172.21.15.6:0/4026324770 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe51c0ccc50 0x7fe51c0bab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:04.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:04.302+0000 7fe51be00640 1 -- 172.21.15.6:0/4026324770 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe51c0b9ec0 con 0x7fe51c0cc870 2024-04-24T23:25:04.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:04.303+0000 7fe51be00640 1 --2- 172.21.15.6:0/4026324770 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe51c0cc870 0x7fe51c1fcaf0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fe50c007860 tx=0x7fe50c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:25:04.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:04.304+0000 7fe513e00640 1 -- 172.21.15.6:0/4026324770 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe50c042020 con 0x7fe51c0cc870 2024-04-24T23:25:04.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:04.304+0000 7fe513e00640 1 -- 172.21.15.6:0/4026324770 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe50c03d660 con 0x7fe51c0cc870 2024-04-24T23:25:04.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:04.304+0000 7fe513e00640 1 -- 172.21.15.6:0/4026324770 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe50c03da30 con 0x7fe51c0cc870 2024-04-24T23:25:04.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:04.304+0000 7fe521400640 1 -- 172.21.15.6:0/4026324770 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe51c0b9340 con 0x7fe51c0cc870 2024-04-24T23:25:04.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:04.308+0000 7fe521400640 1 -- 172.21.15.6:0/4026324770 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe51c0b9900 con 0x7fe51c0cc870 2024-04-24T23:25:04.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:04.311+0000 7fe521400640 1 -- 172.21.15.6:0/4026324770 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe51c075b30 con 0x7fe51c0cc870 2024-04-24T23:25:04.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:04.331+0000 7fe513e00640 1 -- 172.21.15.6:0/4026324770 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fe50c045020 con 0x7fe51c0cc870 2024-04-24T23:25:04.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:04.337+0000 7fe513e00640 1 --2- 172.21.15.6:0/4026324770 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fe4fc06bf00 0x7fe4fc06e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:25:04.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:04.337+0000 7fe513e00640 1 -- 172.21.15.6:0/4026324770 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6848+0+0 (secure 0 0 0) 0x7fe50c0abe60 con 0x7fe51c0cc870 2024-04-24T23:25:04.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:04.338+0000 7fe513e00640 1 -- 172.21.15.6:0/4026324770 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fe50c04c030 con 0x7fe51c0cc870 2024-04-24T23:25:04.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:04.343+0000 7fe51b400640 1 --2- 172.21.15.6:0/4026324770 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fe4fc06bf00 0x7fe4fc06e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:25:04.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:04.345+0000 7fe51b400640 1 --2- 172.21.15.6:0/4026324770 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fe4fc06bf00 0x7fe4fc06e3c0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fe51c071a80 tx=0x7fe508008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:25:04.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:04.846+0000 7fe521400640 1 -- 172.21.15.6:0/4026324770 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe51c0cbc20 con 0x7fe51c0cc870 2024-04-24T23:25:04.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:04.850+0000 7fe513e00640 1 -- 172.21.15.6:0/4026324770 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 46 v46) ==== 94+0+6023 (secure 0 0 0) 0x7fe51c0b9ec0 con 0x7fe51c0cc870 2024-04-24T23:25:04.851 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:25:04.852 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":46,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5081,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/1190759626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1190759626},{"type":"v1","addr":"172.21.15.6:6834","nonce":1190759626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":5249,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6838/2416336040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2416336040},{"type":"v1","addr":"172.21.15.190:6838","nonce":2416336040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":42},{"gid":5503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/3928731059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":3928731059},{"type":"v1","addr":"172.21.15.6:6837","nonce":3928731059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":46,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:24:59.153387+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":4306,"mds_1":5599},"failed":[],"damaged":[],"stopped":[2,3,4],"info":{"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/489092393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":489092393},{"type":"v1","addr":"172.21.15.6:6835","nonce":489092393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5599":{"gid":5599,"name":"d","rank":1,"incarnation":43,"state":"up:rejoin","state_seq":23,"addr":"172.21.15.190:6839/2643233757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":2643233757},{"type":"v1","addr":"172.21.15.190:6839","nonce":2643233757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4306,"qdb_cluster":[4306]},"id":1}]} 2024-04-24T23:25:04.852 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 46 2024-04-24T23:25:04.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:04.855+0000 7fe521400640 1 -- 172.21.15.6:0/4026324770 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fe4fc06bf00 msgr2=0x7fe4fc06e3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:25:04.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:04.855+0000 7fe521400640 1 --2- 172.21.15.6:0/4026324770 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fe4fc06bf00 0x7fe4fc06e3c0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fe51c071a80 tx=0x7fe508008040 comp rx=0 tx=0).stop 2024-04-24T23:25:04.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:04.855+0000 7fe521400640 1 -- 172.21.15.6:0/4026324770 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe51c0cc870 msgr2=0x7fe51c1fcaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:25:04.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:04.855+0000 7fe521400640 1 --2- 172.21.15.6:0/4026324770 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe51c0cc870 0x7fe51c1fcaf0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fe50c007860 tx=0x7fe50c0046c0 comp rx=0 tx=0).stop 2024-04-24T23:25:04.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:04.855+0000 7fe521400640 1 -- 172.21.15.6:0/4026324770 shutdown_connections 2024-04-24T23:25:04.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:04.855+0000 7fe521400640 1 --2- 172.21.15.6:0/4026324770 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fe4fc06bf00 0x7fe4fc06e3c0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:04.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:04.855+0000 7fe521400640 1 --2- 172.21.15.6:0/4026324770 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe51c164a90 0x7fe51c162210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:04.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:04.855+0000 7fe521400640 1 --2- 172.21.15.6:0/4026324770 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe51c0ccc50 0x7fe51c0bab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:04.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:04.855+0000 7fe521400640 1 --2- 172.21.15.6:0/4026324770 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe51c0cc870 0x7fe51c1fcaf0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:04.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:04.855+0000 7fe521400640 1 -- 172.21.15.6:0/4026324770 >> 172.21.15.6:0/4026324770 conn(0x7fe51c06ed80 msgr2=0x7fe51c167ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:25:04.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:04.856+0000 7fe521400640 1 -- 172.21.15.6:0/4026324770 shutdown_connections 2024-04-24T23:25:04.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:04.856+0000 7fe521400640 1 -- 172.21.15.6:0/4026324770 wait complete. 2024-04-24T23:25:04.884 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-24T23:25:06.884 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:25:06.978 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-04-24T23:25:06.977+0000 7fef1ce00640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-04-24T23:25:07.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:07.146+0000 7fde30e00640 1 Processor -- start 2024-04-24T23:25:07.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:07.147+0000 7fde30e00640 1 -- start start 2024-04-24T23:25:07.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:07.148+0000 7fde30e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fde2c152bc0 0x7fde2c159390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:25:07.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:07.148+0000 7fde30e00640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fde2c0c8f50 0x7fde2c15bf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:25:07.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:07.148+0000 7fde30e00640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fde2c0c6e70 con 0x7fde2c0c9360 2024-04-24T23:25:07.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:07.148+0000 7fde30e00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7fde2c0c73d0 con 0x7fde2c152bc0 2024-04-24T23:25:07.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:07.148+0000 7fde30e00640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7fde2c0c8ac0 con 0x7fde2c0c8f50 2024-04-24T23:25:07.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:07.149+0000 7fde29e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fde2c152bc0 0x7fde2c159390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:25:07.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:07.149+0000 7fde29e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fde2c152bc0 0x7fde2c159390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:43236/0 (socket says 172.21.15.6:43236) 2024-04-24T23:25:07.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:07.149+0000 7fde29e00640 1 -- 172.21.15.6:0/1414334991 learned_addr learned my addr 172.21.15.6:0/1414334991 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:25:07.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:07.149+0000 7fde29400640 1 --2- 172.21.15.6:0/1414334991 >> v2:172.21.15.190:3300/0 conn(0x7fde2c0c8f50 0x7fde2c15bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:25:07.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:07.150+0000 7fde29e00640 1 -- 172.21.15.6:0/1414334991 >> v2:172.21.15.190:3300/0 conn(0x7fde2c0c8f50 msgr2=0x7fde2c15bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:25:07.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:07.150+0000 7fde29e00640 1 --2- 172.21.15.6:0/1414334991 >> v2:172.21.15.190:3300/0 conn(0x7fde2c0c8f50 0x7fde2c15bf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:07.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:07.150+0000 7fde29e00640 1 -- 172.21.15.6:0/1414334991 >> v1:172.21.15.190:6789/0 conn(0x7fde2c0c9360 legacy=0x7fde2c15c440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:25:07.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:07.150+0000 7fde29e00640 1 -- 172.21.15.6:0/1414334991 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde2c1598d0 con 0x7fde2c152bc0 2024-04-24T23:25:07.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:07.151+0000 7fde29e00640 1 --2- 172.21.15.6:0/1414334991 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fde2c152bc0 0x7fde2c159390 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fde2c0c7550 tx=0x7fde1c02a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73fda924d4e292eb server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:25:07.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:07.153+0000 7fde28a00640 1 -- 172.21.15.6:0/1414334991 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde1c037070 con 0x7fde2c152bc0 2024-04-24T23:25:07.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:07.153+0000 7fde28a00640 1 -- 172.21.15.6:0/1414334991 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde1c032440 con 0x7fde2c152bc0 2024-04-24T23:25:07.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:07.153+0000 7fde28a00640 1 -- 172.21.15.6:0/1414334991 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde1c032810 con 0x7fde2c152bc0 2024-04-24T23:25:07.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:07.155+0000 7fde30e00640 1 -- 172.21.15.6:0/1414334991 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fde2c152bc0 msgr2=0x7fde2c159390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:25:07.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:07.155+0000 7fde30e00640 1 --2- 172.21.15.6:0/1414334991 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fde2c152bc0 0x7fde2c159390 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fde2c0c7550 tx=0x7fde1c02a180 comp rx=0 tx=0).stop 2024-04-24T23:25:07.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:07.156+0000 7fde30e00640 1 -- 172.21.15.6:0/1414334991 shutdown_connections 2024-04-24T23:25:07.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:07.156+0000 7fde30e00640 1 --2- 172.21.15.6:0/1414334991 >> v2:172.21.15.190:3300/0 conn(0x7fde2c0c8f50 0x7fde2c15bf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:07.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:07.156+0000 7fde30e00640 1 --2- 172.21.15.6:0/1414334991 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fde2c152bc0 0x7fde2c159390 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:07.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:07.156+0000 7fde30e00640 1 -- 172.21.15.6:0/1414334991 >> 172.21.15.6:0/1414334991 conn(0x7fde2c06b0e0 msgr2=0x7fde2c06b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:25:07.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:07.157+0000 7fde30e00640 1 -- 172.21.15.6:0/1414334991 shutdown_connections 2024-04-24T23:25:07.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:07.157+0000 7fde30e00640 1 -- 172.21.15.6:0/1414334991 wait complete. 2024-04-24T23:25:07.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:07.160+0000 7fde30e00640 1 Processor -- start 2024-04-24T23:25:07.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:07.161+0000 7fde30e00640 1 -- start start 2024-04-24T23:25:07.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:07.162+0000 7fde30e00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fde2c0c8f50 0x7fde2c15b590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:25:07.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:07.162+0000 7fde30e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fde2c0c9360 0x7fde2c0cb3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:25:07.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:07.162+0000 7fde30e00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fde2c152bc0 0x7fde2c1f43d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:25:07.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:07.162+0000 7fde30e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fde2c0c8470 con 0x7fde2c0c9360 2024-04-24T23:25:07.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:07.162+0000 7fde30e00640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fde2c0c6700 con 0x7fde2c152bc0 2024-04-24T23:25:07.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:07.162+0000 7fde30e00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fde2c0c8750 con 0x7fde2c0c8f50 2024-04-24T23:25:07.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:07.163+0000 7fde29400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fde2c0c9360 0x7fde2c0cb3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:25:07.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:07.163+0000 7fde29400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fde2c0c9360 0x7fde2c0cb3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:43238/0 (socket says 172.21.15.6:43238) 2024-04-24T23:25:07.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:07.163+0000 7fde29400640 1 -- 172.21.15.6:0/1294423360 learned_addr learned my addr 172.21.15.6:0/1294423360 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:25:07.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:07.164+0000 7fde29e00640 1 --2- 172.21.15.6:0/1294423360 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fde2c0c8f50 0x7fde2c15b590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:25:07.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:07.164+0000 7fde2a800640 1 --2- 172.21.15.6:0/1294423360 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fde2c152bc0 0x7fde2c1f43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:25:07.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:07.164+0000 7fde29400640 1 -- 172.21.15.6:0/1294423360 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fde2c0c8f50 msgr2=0x7fde2c15b590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:25:07.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:07.164+0000 7fde29400640 1 --2- 172.21.15.6:0/1294423360 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fde2c0c8f50 0x7fde2c15b590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:07.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:07.164+0000 7fde29400640 1 -- 172.21.15.6:0/1294423360 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fde2c152bc0 msgr2=0x7fde2c1f43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:25:07.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:07.165+0000 7fde29400640 1 --2- 172.21.15.6:0/1294423360 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fde2c152bc0 0x7fde2c1f43d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:07.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:07.165+0000 7fde29400640 1 -- 172.21.15.6:0/1294423360 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde2c15bad0 con 0x7fde2c0c9360 2024-04-24T23:25:07.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:07.165+0000 7fde2a800640 1 --2- 172.21.15.6:0/1294423360 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fde2c152bc0 0x7fde2c1f43d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:25:07.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:07.165+0000 7fde29400640 1 --2- 172.21.15.6:0/1294423360 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fde2c0c9360 0x7fde2c0cb3b0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fde1800d5f0 tx=0x7fde1800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:25:07.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:07.166+0000 7fde22000640 1 -- 172.21.15.6:0/1294423360 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde180125f0 con 0x7fde2c0c9360 2024-04-24T23:25:07.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:07.166+0000 7fde22000640 1 -- 172.21.15.6:0/1294423360 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde18019070 con 0x7fde2c0c9360 2024-04-24T23:25:07.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:07.166+0000 7fde22000640 1 -- 172.21.15.6:0/1294423360 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde180129d0 con 0x7fde2c0c9360 2024-04-24T23:25:07.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:07.167+0000 7fde30e00640 1 -- 172.21.15.6:0/1294423360 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde2c15bd60 con 0x7fde2c0c9360 2024-04-24T23:25:07.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:07.167+0000 7fde30e00640 1 -- 172.21.15.6:0/1294423360 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde2c15c630 con 0x7fde2c0c9360 2024-04-24T23:25:07.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:07.169+0000 7fde30e00640 1 -- 172.21.15.6:0/1294423360 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde2c14ee70 con 0x7fde2c0c9360 2024-04-24T23:25:07.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:07.171+0000 7fde22000640 1 -- 172.21.15.6:0/1294423360 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fde18012b70 con 0x7fde2c0c9360 2024-04-24T23:25:07.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:07.172+0000 7fde22000640 1 --2- 172.21.15.6:0/1294423360 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fde0c06be80 0x7fde0c06e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:25:07.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:07.172+0000 7fde22000640 1 -- 172.21.15.6:0/1294423360 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6848+0+0 (secure 0 0 0) 0x7fde1808b580 con 0x7fde2c0c9360 2024-04-24T23:25:07.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:07.173+0000 7fde29e00640 1 --2- 172.21.15.6:0/1294423360 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fde0c06be80 0x7fde0c06e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:25:07.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:07.174+0000 7fde29e00640 1 --2- 172.21.15.6:0/1294423360 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fde0c06be80 0x7fde0c06e340 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fde1c034000 tx=0x7fde1c02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:25:07.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:07.180+0000 7fde22000640 1 -- 172.21.15.6:0/1294423360 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fde2c14ee70 con 0x7fde2c0c9360 2024-04-24T23:25:07.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:07.691+0000 7fde30e00640 1 -- 172.21.15.6:0/1294423360 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fde2c0c75e0 con 0x7fde2c0c9360 2024-04-24T23:25:07.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:07.694+0000 7fde22000640 1 -- 172.21.15.6:0/1294423360 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 47 v47) ==== 94+0+6028 (secure 0 0 0) 0x7fde18059b30 con 0x7fde2c0c9360 2024-04-24T23:25:07.698 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:25:07.698 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":47,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5081,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/1190759626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1190759626},{"type":"v1","addr":"172.21.15.6:6834","nonce":1190759626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":5249,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6838/2416336040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2416336040},{"type":"v1","addr":"172.21.15.190:6838","nonce":2416336040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":42},{"gid":5503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/3928731059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":3928731059},{"type":"v1","addr":"172.21.15.6:6837","nonce":3928731059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":47,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:25:05.197173+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":4306,"mds_1":5599},"failed":[],"damaged":[],"stopped":[2,3,4],"info":{"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/489092393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":489092393},{"type":"v1","addr":"172.21.15.6:6835","nonce":489092393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5599":{"gid":5599,"name":"d","rank":1,"incarnation":43,"state":"up:active","state_seq":25,"addr":"172.21.15.190:6839/2643233757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":2643233757},{"type":"v1","addr":"172.21.15.190:6839","nonce":2643233757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4306,"qdb_cluster":[4306,5599]},"id":1}]} 2024-04-24T23:25:07.698 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 47 2024-04-24T23:25:07.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:07.700+0000 7fde30e00640 1 -- 172.21.15.6:0/1294423360 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fde0c06be80 msgr2=0x7fde0c06e340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:25:07.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:07.700+0000 7fde30e00640 1 --2- 172.21.15.6:0/1294423360 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fde0c06be80 0x7fde0c06e340 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fde1c034000 tx=0x7fde1c02e000 comp rx=0 tx=0).stop 2024-04-24T23:25:07.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:07.700+0000 7fde30e00640 1 -- 172.21.15.6:0/1294423360 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fde2c0c9360 msgr2=0x7fde2c0cb3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:25:07.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:07.700+0000 7fde30e00640 1 --2- 172.21.15.6:0/1294423360 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fde2c0c9360 0x7fde2c0cb3b0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fde1800d5f0 tx=0x7fde1800dab0 comp rx=0 tx=0).stop 2024-04-24T23:25:07.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:07.700+0000 7fde30e00640 1 -- 172.21.15.6:0/1294423360 shutdown_connections 2024-04-24T23:25:07.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:07.700+0000 7fde30e00640 1 --2- 172.21.15.6:0/1294423360 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fde0c06be80 0x7fde0c06e340 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:07.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:07.700+0000 7fde30e00640 1 --2- 172.21.15.6:0/1294423360 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fde2c152bc0 0x7fde2c1f43d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:07.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:07.700+0000 7fde30e00640 1 --2- 172.21.15.6:0/1294423360 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fde2c0c9360 0x7fde2c0cb3b0 secure :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fde1800d5f0 tx=0x7fde1800dab0 comp rx=0 tx=0).stop 2024-04-24T23:25:07.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:07.700+0000 7fde30e00640 1 --2- 172.21.15.6:0/1294423360 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fde2c0c8f50 0x7fde2c15b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:07.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:07.700+0000 7fde30e00640 1 -- 172.21.15.6:0/1294423360 >> 172.21.15.6:0/1294423360 conn(0x7fde2c06b0e0 msgr2=0x7fde2c14ff10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:25:07.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:07.702+0000 7fde30e00640 1 -- 172.21.15.6:0/1294423360 shutdown_connections 2024-04-24T23:25:07.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:07.702+0000 7fde30e00640 1 -- 172.21.15.6:0/1294423360 wait complete. 2024-04-24T23:25:07.743 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d has gained rank=1, replacing gid=4282 2024-04-24T23:25:07.743 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 1 secs before reviving mds.f 2024-04-24T23:25:08.743 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.f 2024-04-24T23:25:08.744 INFO:tasks.ceph.mds.f:Restarting daemon 2024-04-24T23:25:08.744 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i f 2024-04-24T23:25:08.746 INFO:tasks.ceph.mds.f:Started 2024-04-24T23:25:08.747 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:25:08.824 INFO:tasks.ceph.mds.f.smithi190.stdout:starting mds.f at 2024-04-24T23:25:09.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:09.061+0000 7fcd0e000640 1 Processor -- start 2024-04-24T23:25:09.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:09.061+0000 7fcd0e000640 1 -- start start 2024-04-24T23:25:09.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:09.063+0000 7fcd0e000640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fcd08159110 0x7fcd08072420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:25:09.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:09.064+0000 7fcd0e000640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcd080c7dc0 0x7fcd081623a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:25:09.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:09.064+0000 7fcd0e000640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcd080c7230 con 0x7fcd080c81c0 2024-04-24T23:25:09.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:09.064+0000 7fcd0e000640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7fcd081594f0 con 0x7fcd08159110 2024-04-24T23:25:09.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:09.064+0000 7fcd0e000640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fcd080c74d0 con 0x7fcd080c7dc0 2024-04-24T23:25:09.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:09.064+0000 7fcd06a00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcd080c7dc0 0x7fcd081623a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:25:09.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:09.065+0000 7fcd06a00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcd080c7dc0 0x7fcd081623a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.6:50822/0 (socket says 172.21.15.6:50822) 2024-04-24T23:25:09.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:09.065+0000 7fcd06a00640 1 -- 172.21.15.6:0/1849435244 learned_addr learned my addr 172.21.15.6:0/1849435244 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:25:09.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:09.064+0000 7fcd07e00640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7fcd080c81c0 0x7fcd081628e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.6:42484/0 (socket says 172.21.15.6:42484) 2024-04-24T23:25:09.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:09.065+0000 7fcd07400640 1 --2- 172.21.15.6:0/1849435244 >> v2:172.21.15.190:3300/0 conn(0x7fcd08159110 0x7fcd08072420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:25:09.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:09.066+0000 7fcd07400640 1 -- 172.21.15.6:0/1849435244 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcd080c7dc0 msgr2=0x7fcd081623a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:25:09.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:09.066+0000 7fcd07400640 1 --2- 172.21.15.6:0/1849435244 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcd080c7dc0 0x7fcd081623a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:09.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:09.066+0000 7fcd07400640 1 -- 172.21.15.6:0/1849435244 >> v1:172.21.15.190:6789/0 conn(0x7fcd080c81c0 legacy=0x7fcd081628e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:25:09.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:09.066+0000 7fcd07400640 1 -- 172.21.15.6:0/1849435244 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd0806e440 con 0x7fcd08159110 2024-04-24T23:25:09.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:09.066+0000 7fcd06a00640 1 --2- 172.21.15.6:0/1849435244 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcd080c7dc0 0x7fcd081623a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:25:09.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:09.068+0000 7fcd07400640 1 --2- 172.21.15.6:0/1849435244 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcd08159110 0x7fcd08072420 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fccf40097a0 tx=0x7fccf402a270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9622a866a23386ac server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:25:09.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:09.069+0000 7fcd06000640 1 -- 172.21.15.6:0/1849435244 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccf4037070 con 0x7fcd08159110 2024-04-24T23:25:09.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:09.070+0000 7fcd06000640 1 -- 172.21.15.6:0/1849435244 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fccf4032440 con 0x7fcd08159110 2024-04-24T23:25:09.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:09.070+0000 7fcd06000640 1 -- 172.21.15.6:0/1849435244 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccf4032810 con 0x7fcd08159110 2024-04-24T23:25:09.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:09.071+0000 7fcd0e000640 1 -- 172.21.15.6:0/1849435244 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcd08159110 msgr2=0x7fcd08072420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:25:09.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:09.071+0000 7fcd0e000640 1 --2- 172.21.15.6:0/1849435244 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcd08159110 0x7fcd08072420 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fccf40097a0 tx=0x7fccf402a270 comp rx=0 tx=0).stop 2024-04-24T23:25:09.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:09.072+0000 7fcd0e000640 1 -- 172.21.15.6:0/1849435244 shutdown_connections 2024-04-24T23:25:09.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:09.072+0000 7fcd0e000640 1 --2- 172.21.15.6:0/1849435244 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcd080c7dc0 0x7fcd081623a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:09.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:09.072+0000 7fcd0e000640 1 --2- 172.21.15.6:0/1849435244 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcd08159110 0x7fcd08072420 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:09.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:09.072+0000 7fcd0e000640 1 -- 172.21.15.6:0/1849435244 >> 172.21.15.6:0/1849435244 conn(0x7fcd0806b0e0 msgr2=0x7fcd0806b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:25:09.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:09.072+0000 7fcd0e000640 1 -- 172.21.15.6:0/1849435244 shutdown_connections 2024-04-24T23:25:09.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:09.076+0000 7fcd0e000640 1 -- 172.21.15.6:0/1849435244 wait complete. 2024-04-24T23:25:09.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:09.078+0000 7fcd0e000640 1 Processor -- start 2024-04-24T23:25:09.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:09.080+0000 7fcd0e000640 1 -- start start 2024-04-24T23:25:09.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:09.081+0000 7fcd0e000640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcd080c7dc0 0x7fcd0815a020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:25:09.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:09.081+0000 7fcd0e000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcd080c81c0 0x7fcd0815a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:25:09.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:09.081+0000 7fcd0e000640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcd08159110 0x7fcd08130850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:25:09.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:09.082+0000 7fcd0e000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fcd0806ece0 con 0x7fcd080c81c0 2024-04-24T23:25:09.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:09.082+0000 7fcd0e000640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fcd0806ea00 con 0x7fcd08159110 2024-04-24T23:25:09.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:09.082+0000 7fcd0e000640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fcd08072960 con 0x7fcd080c7dc0 2024-04-24T23:25:09.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:09.082+0000 7fcd07e00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcd08159110 0x7fcd08130850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:25:09.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:09.082+0000 7fcd07e00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcd08159110 0x7fcd08130850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:52890/0 (socket says 172.21.15.6:52890) 2024-04-24T23:25:09.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:09.082+0000 7fcd07e00640 1 -- 172.21.15.6:0/187668732 learned_addr learned my addr 172.21.15.6:0/187668732 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:25:09.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:09.082+0000 7fcd07400640 1 --2- 172.21.15.6:0/187668732 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcd080c7dc0 0x7fcd0815a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:25:09.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:09.082+0000 7fcd06a00640 1 --2- 172.21.15.6:0/187668732 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcd080c81c0 0x7fcd0815a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:25:09.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:09.084+0000 7fcd06a00640 1 -- 172.21.15.6:0/187668732 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcd080c7dc0 msgr2=0x7fcd0815a020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:25:09.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:09.084+0000 7fcd06a00640 1 --2- 172.21.15.6:0/187668732 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcd080c7dc0 0x7fcd0815a020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:09.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:09.085+0000 7fcd06a00640 1 -- 172.21.15.6:0/187668732 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcd08159110 msgr2=0x7fcd08130850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:25:09.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:09.085+0000 7fcd06a00640 1 --2- 172.21.15.6:0/187668732 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcd08159110 0x7fcd08130850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:09.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:09.085+0000 7fcd06a00640 1 -- 172.21.15.6:0/187668732 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd0815a560 con 0x7fcd080c81c0 2024-04-24T23:25:09.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:09.085+0000 7fcd07e00640 1 --2- 172.21.15.6:0/187668732 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcd08159110 0x7fcd08130850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:25:09.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:09.085+0000 7fcd07400640 1 --2- 172.21.15.6:0/187668732 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcd080c7dc0 0x7fcd0815a020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:25:09.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:09.085+0000 7fcd06a00640 1 --2- 172.21.15.6:0/187668732 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcd080c81c0 0x7fcd0815a970 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fccf800e900 tx=0x7fccf800edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:25:09.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:09.086+0000 7fccff400640 1 -- 172.21.15.6:0/187668732 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccf800ca00 con 0x7fcd080c81c0 2024-04-24T23:25:09.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:09.086+0000 7fccff400640 1 -- 172.21.15.6:0/187668732 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fccf800cba0 con 0x7fcd080c81c0 2024-04-24T23:25:09.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:09.086+0000 7fccff400640 1 -- 172.21.15.6:0/187668732 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccf8004030 con 0x7fcd080c81c0 2024-04-24T23:25:09.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:09.086+0000 7fcd0e000640 1 -- 172.21.15.6:0/187668732 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd080c6290 con 0x7fcd080c81c0 2024-04-24T23:25:09.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:09.086+0000 7fcd0e000640 1 -- 172.21.15.6:0/187668732 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd081614e0 con 0x7fcd080c81c0 2024-04-24T23:25:09.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:09.089+0000 7fccff400640 1 -- 172.21.15.6:0/187668732 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fccf80041d0 con 0x7fcd080c81c0 2024-04-24T23:25:09.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:09.091+0000 7fccff400640 1 --2- 172.21.15.6:0/187668732 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fcce806be60 0x7fcce806e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:25:09.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:09.091+0000 7fccff400640 1 -- 172.21.15.6:0/187668732 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6848+0+0 (secure 0 0 0) 0x7fccf808b240 con 0x7fcd080c81c0 2024-04-24T23:25:09.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:09.091+0000 7fcd0e000640 1 -- 172.21.15.6:0/187668732 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcd080c6790 con 0x7fcd080c81c0 2024-04-24T23:25:09.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:09.092+0000 7fcd07400640 1 --2- 172.21.15.6:0/187668732 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fcce806be60 0x7fcce806e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:25:09.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:09.093+0000 7fcd07400640 1 --2- 172.21.15.6:0/187668732 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fcce806be60 0x7fcce806e320 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fccf4034000 tx=0x7fccf402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:25:09.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:09.100+0000 7fccff400640 1 -- 172.21.15.6:0/187668732 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fcd080c6790 con 0x7fcd080c81c0 2024-04-24T23:25:09.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:09.697+0000 7fcd0e000640 1 -- 172.21.15.6:0/187668732 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcd080c7970 con 0x7fcd080c81c0 2024-04-24T23:25:09.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:09.698+0000 7fccff400640 1 -- 172.21.15.6:0/187668732 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) ==== 94+0+6856 (secure 0 0 0) 0x7fccf8017020 con 0x7fcd080c81c0 2024-04-24T23:25:09.700 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:25:09.700 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":48,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5081,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/1190759626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1190759626},{"type":"v1","addr":"172.21.15.6:6834","nonce":1190759626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":5249,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6838/2416336040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2416336040},{"type":"v1","addr":"172.21.15.190:6838","nonce":2416336040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":42},{"gid":5503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/3928731059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":3928731059},{"type":"v1","addr":"172.21.15.6:6837","nonce":3928731059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":35},{"gid":5725,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6837/1151527362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":1151527362},{"type":"v1","addr":"172.21.15.190:6837","nonce":1151527362}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":48}],"filesystems":[{"mdsmap":{"epoch":47,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:25:05.197173+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":4306,"mds_1":5599},"failed":[],"damaged":[],"stopped":[2,3,4],"info":{"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/489092393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":489092393},{"type":"v1","addr":"172.21.15.6:6835","nonce":489092393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5599":{"gid":5599,"name":"d","rank":1,"incarnation":43,"state":"up:active","state_seq":25,"addr":"172.21.15.190:6839/2643233757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":2643233757},{"type":"v1","addr":"172.21.15.190:6839","nonce":2643233757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4306,"qdb_cluster":[5599,4306]},"id":1}]} 2024-04-24T23:25:09.700 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 48 2024-04-24T23:25:09.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:09.702+0000 7fcd0e000640 1 -- 172.21.15.6:0/187668732 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fcce806be60 msgr2=0x7fcce806e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:25:09.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:09.702+0000 7fcd0e000640 1 --2- 172.21.15.6:0/187668732 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fcce806be60 0x7fcce806e320 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fccf4034000 tx=0x7fccf402e000 comp rx=0 tx=0).stop 2024-04-24T23:25:09.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:09.702+0000 7fcd0e000640 1 -- 172.21.15.6:0/187668732 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcd080c81c0 msgr2=0x7fcd0815a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:25:09.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:09.702+0000 7fcd0e000640 1 --2- 172.21.15.6:0/187668732 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcd080c81c0 0x7fcd0815a970 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fccf800e900 tx=0x7fccf800edc0 comp rx=0 tx=0).stop 2024-04-24T23:25:09.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:09.703+0000 7fcd0e000640 1 -- 172.21.15.6:0/187668732 shutdown_connections 2024-04-24T23:25:09.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:09.703+0000 7fcd0e000640 1 --2- 172.21.15.6:0/187668732 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fcce806be60 0x7fcce806e320 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:09.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:09.703+0000 7fcd0e000640 1 --2- 172.21.15.6:0/187668732 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcd08159110 0x7fcd08130850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:09.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:09.704+0000 7fcd0e000640 1 --2- 172.21.15.6:0/187668732 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcd080c81c0 0x7fcd0815a970 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:09.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:09.704+0000 7fcd0e000640 1 --2- 172.21.15.6:0/187668732 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcd080c7dc0 0x7fcd0815a020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:09.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:09.704+0000 7fcd0e000640 1 -- 172.21.15.6:0/187668732 >> 172.21.15.6:0/187668732 conn(0x7fcd0806b0e0 msgr2=0x7fcd08157440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:25:09.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:09.704+0000 7fcd0e000640 1 -- 172.21.15.6:0/187668732 shutdown_connections 2024-04-24T23:25:09.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:09.704+0000 7fcd0e000640 1 -- 172.21.15.6:0/187668732 wait complete. 2024-04-24T23:25:09.734 INFO:tasks.mds_thrash.fs.[cephfs]:mds.f reported in up:standby state 2024-04-24T23:25:09.734 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 17 secs before thrashing 2024-04-24T23:25:26.736 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:25:27.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:27.085+0000 7f2a91c00640 1 Processor -- start 2024-04-24T23:25:27.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:27.085+0000 7f2a91c00640 1 -- start start 2024-04-24T23:25:27.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:27.087+0000 7f2a91c00640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f2a8c15c690 0x7f2a8c06fc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:25:27.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:27.087+0000 7f2a91c00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2a8c06b390 con 0x7f2a8c0c9860 2024-04-24T23:25:27.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:27.087+0000 7f2a91c00640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2a8c0704a0 con 0x7f2a8c152c50 2024-04-24T23:25:27.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:27.087+0000 7f2a91c00640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f2a8c070760 con 0x7f2a8c15c690 2024-04-24T23:25:27.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:27.089+0000 7f2a8b400640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f2a8c15c690 0x7f2a8c06fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:25:27.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:27.089+0000 7f2a8b400640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f2a8c15c690 0x7f2a8c06fc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:43686/0 (socket says 172.21.15.6:43686) 2024-04-24T23:25:27.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:27.089+0000 7f2a8b400640 1 -- 172.21.15.6:0/3472725161 learned_addr learned my addr 172.21.15.6:0/3472725161 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:25:27.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:27.089+0000 7f2a8b400640 1 -- 172.21.15.6:0/3472725161 >> v1:172.21.15.190:6789/0 conn(0x7f2a8c152c50 legacy=0x7f2a8c0c6a10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-24T23:25:27.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:27.089+0000 7f2a8b400640 1 -- 172.21.15.6:0/3472725161 >> v1:172.21.15.6:6789/0 conn(0x7f2a8c0c9860 legacy=0x7f2a8c15bf80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-24T23:25:27.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:27.089+0000 7f2a8b400640 1 -- 172.21.15.6:0/3472725161 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a8c070a40 con 0x7f2a8c15c690 2024-04-24T23:25:27.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:27.090+0000 7f2a8b400640 1 --2- 172.21.15.6:0/3472725161 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2a8c15c690 0x7f2a8c06fc00 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f2a84009e50 tx=0x7f2a8402a7a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b0e2c40eabe2da52 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:25:27.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:27.091+0000 7f2a89600640 1 -- 172.21.15.6:0/3472725161 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a84036070 con 0x7f2a8c15c690 2024-04-24T23:25:27.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:27.091+0000 7f2a89600640 1 -- 172.21.15.6:0/3472725161 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a84031600 con 0x7f2a8c15c690 2024-04-24T23:25:27.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:27.091+0000 7f2a89600640 1 -- 172.21.15.6:0/3472725161 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a840319d0 con 0x7f2a8c15c690 2024-04-24T23:25:27.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:27.097+0000 7f2a91c00640 1 -- 172.21.15.6:0/3472725161 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2a8c15c690 msgr2=0x7f2a8c06fc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:25:27.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:27.097+0000 7f2a91c00640 1 --2- 172.21.15.6:0/3472725161 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2a8c15c690 0x7f2a8c06fc00 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f2a84009e50 tx=0x7f2a8402a7a0 comp rx=0 tx=0).stop 2024-04-24T23:25:27.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:27.097+0000 7f2a91c00640 1 -- 172.21.15.6:0/3472725161 shutdown_connections 2024-04-24T23:25:27.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:27.097+0000 7f2a91c00640 1 --2- 172.21.15.6:0/3472725161 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2a8c15c690 0x7f2a8c06fc00 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:27.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:27.097+0000 7f2a91c00640 1 -- 172.21.15.6:0/3472725161 >> 172.21.15.6:0/3472725161 conn(0x7f2a8c06c1f0 msgr2=0x7f2a8c06c600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:25:27.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:27.099+0000 7f2a91c00640 1 -- 172.21.15.6:0/3472725161 shutdown_connections 2024-04-24T23:25:27.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:27.099+0000 7f2a91c00640 1 -- 172.21.15.6:0/3472725161 wait complete. 2024-04-24T23:25:27.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:27.105+0000 7f2a91c00640 1 Processor -- start 2024-04-24T23:25:27.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:27.105+0000 7f2a91c00640 1 -- start start 2024-04-24T23:25:27.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:27.107+0000 7f2a91c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2a8c0c9860 0x7f2a8c156e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:25:27.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:27.107+0000 7f2a91c00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2a8c152c50 0x7f2a8c1fd780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:25:27.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:27.109+0000 7f2a91c00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2a8c15c690 0x7f2a8c0cb1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:25:27.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:27.109+0000 7f2a91c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f2a8c0c7120 con 0x7f2a8c0c9860 2024-04-24T23:25:27.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:27.109+0000 7f2a91c00640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f2a8c06b910 con 0x7f2a8c152c50 2024-04-24T23:25:27.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:27.109+0000 7f2a91c00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f2a8c0c72a0 con 0x7f2a8c15c690 2024-04-24T23:25:27.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:27.113+0000 7f2a8a000640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2a8c152c50 0x7f2a8c1fd780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:25:27.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:27.113+0000 7f2a8a000640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2a8c152c50 0x7f2a8c1fd780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:43700/0 (socket says 172.21.15.6:43700) 2024-04-24T23:25:27.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:27.113+0000 7f2a8a000640 1 -- 172.21.15.6:0/2469138802 learned_addr learned my addr 172.21.15.6:0/2469138802 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:25:27.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:27.113+0000 7f2a8b400640 1 --2- 172.21.15.6:0/2469138802 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2a8c15c690 0x7f2a8c0cb1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:25:27.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:27.114+0000 7f2a8b400640 1 -- 172.21.15.6:0/2469138802 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2a8c152c50 msgr2=0x7f2a8c1fd780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:25:27.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:27.114+0000 7f2a8b400640 1 --2- 172.21.15.6:0/2469138802 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2a8c152c50 0x7f2a8c1fd780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:27.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:27.114+0000 7f2a8b400640 1 -- 172.21.15.6:0/2469138802 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2a8c0c9860 msgr2=0x7f2a8c156e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:25:27.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:27.114+0000 7f2a8b400640 1 --2- 172.21.15.6:0/2469138802 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2a8c0c9860 0x7f2a8c156e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:27.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:27.114+0000 7f2a8b400640 1 -- 172.21.15.6:0/2469138802 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a8c0cce20 con 0x7f2a8c15c690 2024-04-24T23:25:27.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:27.119+0000 7f2a8b400640 1 --2- 172.21.15.6:0/2469138802 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2a8c15c690 0x7f2a8c0cb1e0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f2a84030000 tx=0x7f2a84004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:25:27.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:27.120+0000 7f2a82a00640 1 -- 172.21.15.6:0/2469138802 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a84036040 con 0x7f2a8c15c690 2024-04-24T23:25:27.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:27.121+0000 7f2a82a00640 1 -- 172.21.15.6:0/2469138802 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a84003990 con 0x7f2a8c15c690 2024-04-24T23:25:27.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:27.121+0000 7f2a82a00640 1 -- 172.21.15.6:0/2469138802 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a84009510 con 0x7f2a8c15c690 2024-04-24T23:25:27.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:27.121+0000 7f2a91c00640 1 -- 172.21.15.6:0/2469138802 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a8c0cd100 con 0x7f2a8c15c690 2024-04-24T23:25:27.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:27.121+0000 7f2a91c00640 1 -- 172.21.15.6:0/2469138802 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a8c0cd6c0 con 0x7f2a8c15c690 2024-04-24T23:25:27.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:27.129+0000 7f2a91c00640 1 -- 172.21.15.6:0/2469138802 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2a64005290 con 0x7f2a8c15c690 2024-04-24T23:25:27.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:27.129+0000 7f2a82a00640 1 -- 172.21.15.6:0/2469138802 <== mon.2 v2:172.21.15.190:3301/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f2a840077a0 con 0x7f2a8c15c690 2024-04-24T23:25:27.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:27.130+0000 7f2a82a00640 1 --2- 172.21.15.6:0/2469138802 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f2a6c06bf00 0x7f2a6c06e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:25:27.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:27.131+0000 7f2a82a00640 1 -- 172.21.15.6:0/2469138802 <== mon.2 v2:172.21.15.190:3301/0 5 ==== osd_map(29..29 src has 1..29) ==== 6848+0+0 (secure 0 0 0) 0x7f2a840ac500 con 0x7f2a8c15c690 2024-04-24T23:25:27.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:27.132+0000 7f2a8aa00640 1 --2- 172.21.15.6:0/2469138802 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f2a6c06bf00 0x7f2a6c06e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:25:27.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:27.134+0000 7f2a8aa00640 1 --2- 172.21.15.6:0/2469138802 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f2a6c06bf00 0x7f2a6c06e3c0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f2a7c0096f0 tx=0x7f2a7c006c40 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:25:27.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:27.137+0000 7f2a82a00640 1 -- 172.21.15.6:0/2469138802 <== mon.2 v2:172.21.15.190:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f2a8407aad0 con 0x7f2a8c15c690 2024-04-24T23:25:27.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:27.714+0000 7f2a91c00640 1 -- 172.21.15.6:0/2469138802 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2a64005580 con 0x7f2a8c15c690 2024-04-24T23:25:27.719 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:25:27.719 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":48,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5081,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/1190759626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1190759626},{"type":"v1","addr":"172.21.15.6:6834","nonce":1190759626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":5249,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6838/2416336040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2416336040},{"type":"v1","addr":"172.21.15.190:6838","nonce":2416336040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":42},{"gid":5503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/3928731059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":3928731059},{"type":"v1","addr":"172.21.15.6:6837","nonce":3928731059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":35},{"gid":5725,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6837/1151527362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":1151527362},{"type":"v1","addr":"172.21.15.190:6837","nonce":1151527362}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":48}],"filesystems":[{"mdsmap":{"epoch":47,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:25:05.197173+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":4306,"mds_1":5599},"failed":[],"damaged":[],"stopped":[2,3,4],"info":{"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/489092393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":489092393},{"type":"v1","addr":"172.21.15.6:6835","nonce":489092393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5599":{"gid":5599,"name":"d","rank":1,"incarnation":43,"state":"up:active","state_seq":25,"addr":"172.21.15.190:6839/2643233757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":2643233757},{"type":"v1","addr":"172.21.15.190:6839","nonce":2643233757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4306,"qdb_cluster":[5599,4306]},"id":1}]} 2024-04-24T23:25:27.719 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 48 2024-04-24T23:25:27.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:27.718+0000 7f2a82a00640 1 -- 172.21.15.6:0/2469138802 <== mon.2 v2:172.21.15.190:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) ==== 94+0+6856 (secure 0 0 0) 0x7f2a8407a3d0 con 0x7f2a8c15c690 2024-04-24T23:25:27.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:27.724+0000 7f2a91c00640 1 -- 172.21.15.6:0/2469138802 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f2a6c06bf00 msgr2=0x7f2a6c06e3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:25:27.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:27.724+0000 7f2a91c00640 1 --2- 172.21.15.6:0/2469138802 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f2a6c06bf00 0x7f2a6c06e3c0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f2a7c0096f0 tx=0x7f2a7c006c40 comp rx=0 tx=0).stop 2024-04-24T23:25:27.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:27.724+0000 7f2a91c00640 1 -- 172.21.15.6:0/2469138802 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2a8c15c690 msgr2=0x7f2a8c0cb1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:25:27.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:27.724+0000 7f2a91c00640 1 --2- 172.21.15.6:0/2469138802 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2a8c15c690 0x7f2a8c0cb1e0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f2a84030000 tx=0x7f2a84004a40 comp rx=0 tx=0).stop 2024-04-24T23:25:27.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:27.725+0000 7f2a91c00640 1 -- 172.21.15.6:0/2469138802 shutdown_connections 2024-04-24T23:25:27.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:27.725+0000 7f2a91c00640 1 --2- 172.21.15.6:0/2469138802 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f2a6c06bf00 0x7f2a6c06e3c0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:27.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:27.725+0000 7f2a91c00640 1 --2- 172.21.15.6:0/2469138802 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2a8c15c690 0x7f2a8c0cb1e0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:27.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:27.725+0000 7f2a91c00640 1 --2- 172.21.15.6:0/2469138802 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2a8c152c50 0x7f2a8c1fd780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:27.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:27.725+0000 7f2a91c00640 1 --2- 172.21.15.6:0/2469138802 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2a8c0c9860 0x7f2a8c156e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:27.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:27.725+0000 7f2a91c00640 1 -- 172.21.15.6:0/2469138802 >> 172.21.15.6:0/2469138802 conn(0x7f2a8c06c1f0 msgr2=0x7f2a8c14d990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:25:27.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:27.725+0000 7f2a91c00640 1 -- 172.21.15.6:0/2469138802 shutdown_connections 2024-04-24T23:25:27.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:27.725+0000 7f2a91c00640 1 -- 172.21.15.6:0/2469138802 wait complete. 2024-04-24T23:25:27.758 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.c (rank=0) 2024-04-24T23:25:27.759 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-24T23:25:27.759 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T23:25:27.805 INFO:tasks.ceph.mds.c:Stopped 2024-04-24T23:25:27.805 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:25:28.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:28.108+0000 7fa495000640 1 Processor -- start 2024-04-24T23:25:28.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:28.109+0000 7fa495000640 1 -- start start 2024-04-24T23:25:28.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:28.111+0000 7fa495000640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa490154c40 0x7fa4900c6710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:25:28.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:28.111+0000 7fa495000640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fa4900c53d0 0x7fa4900cb0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:25:28.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:28.111+0000 7fa495000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa4900cb5f0 0x7fa49015e020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:25:28.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:28.111+0000 7fa495000640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7fa49015ef20 con 0x7fa4900cb5f0 2024-04-24T23:25:28.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:28.111+0000 7fa495000640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7fa49015ec40 con 0x7fa4900c53d0 2024-04-24T23:25:28.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:28.111+0000 7fa495000640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fa4900c5950 con 0x7fa490154c40 2024-04-24T23:25:28.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:28.111+0000 7fa48fe00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa4900cb5f0 0x7fa49015e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:25:28.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:28.111+0000 7fa48fe00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa4900cb5f0 0x7fa49015e020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:47288/0 (socket says 172.21.15.6:47288) 2024-04-24T23:25:28.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:28.111+0000 7fa48fe00640 1 -- 172.21.15.6:0/1893371290 learned_addr learned my addr 172.21.15.6:0/1893371290 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:25:28.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:28.111+0000 7fa48f400640 1 --2- 172.21.15.6:0/1893371290 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa490154c40 0x7fa4900c6710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:25:28.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:28.112+0000 7fa48f400640 1 -- 172.21.15.6:0/1893371290 >> v2:172.21.15.190:3300/0 conn(0x7fa4900c53d0 msgr2=0x7fa4900cb0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:25:28.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:28.112+0000 7fa48f400640 1 --2- 172.21.15.6:0/1893371290 >> v2:172.21.15.190:3300/0 conn(0x7fa4900c53d0 0x7fa4900cb0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:28.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:28.112+0000 7fa48f400640 1 -- 172.21.15.6:0/1893371290 >> v2:172.21.15.6:3300/0 conn(0x7fa4900cb5f0 msgr2=0x7fa49015e020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:25:28.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:28.112+0000 7fa48f400640 1 --2- 172.21.15.6:0/1893371290 >> v2:172.21.15.6:3300/0 conn(0x7fa4900cb5f0 0x7fa49015e020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:28.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:28.112+0000 7fa48f400640 1 -- 172.21.15.6:0/1893371290 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4900c5c30 con 0x7fa490154c40 2024-04-24T23:25:28.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:28.114+0000 7fa48f400640 1 --2- 172.21.15.6:0/1893371290 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa490154c40 0x7fa4900c6710 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fa49006d450 tx=0x7fa47802a1c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=88cd33910cb672c8 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:25:28.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:28.114+0000 7fa48e000640 1 -- 172.21.15.6:0/1893371290 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa478037070 con 0x7fa490154c40 2024-04-24T23:25:28.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:28.114+0000 7fa48e000640 1 -- 172.21.15.6:0/1893371290 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa478032440 con 0x7fa490154c40 2024-04-24T23:25:28.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:28.114+0000 7fa48e000640 1 -- 172.21.15.6:0/1893371290 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4780327f0 con 0x7fa490154c40 2024-04-24T23:25:28.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:28.115+0000 7fa495000640 1 -- 172.21.15.6:0/1893371290 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa490154c40 msgr2=0x7fa4900c6710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:25:28.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:28.115+0000 7fa495000640 1 --2- 172.21.15.6:0/1893371290 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa490154c40 0x7fa4900c6710 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fa49006d450 tx=0x7fa47802a1c0 comp rx=0 tx=0).stop 2024-04-24T23:25:28.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:28.115+0000 7fa495000640 1 -- 172.21.15.6:0/1893371290 shutdown_connections 2024-04-24T23:25:28.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:28.115+0000 7fa495000640 1 --2- 172.21.15.6:0/1893371290 >> v2:172.21.15.6:3300/0 conn(0x7fa4900cb5f0 0x7fa49015e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:28.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:28.115+0000 7fa495000640 1 --2- 172.21.15.6:0/1893371290 >> v2:172.21.15.190:3300/0 conn(0x7fa4900c53d0 0x7fa4900cb0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:28.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:28.115+0000 7fa495000640 1 --2- 172.21.15.6:0/1893371290 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa490154c40 0x7fa4900c6710 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:28.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:28.115+0000 7fa495000640 1 -- 172.21.15.6:0/1893371290 >> 172.21.15.6:0/1893371290 conn(0x7fa49006cd70 msgr2=0x7fa49006d870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:25:28.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:28.116+0000 7fa495000640 1 -- 172.21.15.6:0/1893371290 shutdown_connections 2024-04-24T23:25:28.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:28.116+0000 7fa495000640 1 -- 172.21.15.6:0/1893371290 wait complete. 2024-04-24T23:25:28.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:28.118+0000 7fa495000640 1 Processor -- start 2024-04-24T23:25:28.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:28.118+0000 7fa495000640 1 -- start start 2024-04-24T23:25:28.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:28.120+0000 7fa495000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa4900c53d0 0x7fa490157920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:25:28.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:28.120+0000 7fa495000640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa4900cb5f0 0x7fa4900c8a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:25:28.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:28.120+0000 7fa495000640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa490154c40 0x7fa4901f3eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:25:28.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:28.120+0000 7fa495000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fa4900c6c50 con 0x7fa4900c53d0 2024-04-24T23:25:28.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:28.120+0000 7fa495000640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fa4900cc470 con 0x7fa490154c40 2024-04-24T23:25:28.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:28.120+0000 7fa495000640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fa4900c6dd0 con 0x7fa4900cb5f0 2024-04-24T23:25:28.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:28.120+0000 7fa48fe00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa490154c40 0x7fa4901f3eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:25:28.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:28.120+0000 7fa48fe00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa490154c40 0x7fa4901f3eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:43724/0 (socket says 172.21.15.6:43724) 2024-04-24T23:25:28.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:28.120+0000 7fa48fe00640 1 -- 172.21.15.6:0/603863526 learned_addr learned my addr 172.21.15.6:0/603863526 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:25:28.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:28.121+0000 7fa48f400640 1 --2- 172.21.15.6:0/603863526 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa4900c53d0 0x7fa490157920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:25:28.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:28.121+0000 7fa48fe00640 1 -- 172.21.15.6:0/603863526 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa4900cb5f0 msgr2=0x7fa4900c8a30 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-24T23:25:28.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:28.121+0000 7fa48fe00640 1 --2- 172.21.15.6:0/603863526 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa4900cb5f0 0x7fa4900c8a30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:28.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:28.121+0000 7fa48fe00640 1 -- 172.21.15.6:0/603863526 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa4900c53d0 msgr2=0x7fa490157920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:25:28.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:28.121+0000 7fa48fe00640 1 --2- 172.21.15.6:0/603863526 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa4900c53d0 0x7fa490157920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:28.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:28.121+0000 7fa48fe00640 1 -- 172.21.15.6:0/603863526 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4900cf930 con 0x7fa490154c40 2024-04-24T23:25:28.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:28.123+0000 7fa48fe00640 1 --2- 172.21.15.6:0/603863526 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa490154c40 0x7fa4901f3eb0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fa488009cf0 tx=0x7fa48800c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:25:28.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:28.124+0000 7fa487400640 1 -- 172.21.15.6:0/603863526 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa48800cd20 con 0x7fa490154c40 2024-04-24T23:25:28.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:28.124+0000 7fa487400640 1 -- 172.21.15.6:0/603863526 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa488003990 con 0x7fa490154c40 2024-04-24T23:25:28.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:28.124+0000 7fa487400640 1 -- 172.21.15.6:0/603863526 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa488003d80 con 0x7fa490154c40 2024-04-24T23:25:28.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:28.124+0000 7fa495000640 1 -- 172.21.15.6:0/603863526 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4900cfba0 con 0x7fa490154c40 2024-04-24T23:25:28.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:28.124+0000 7fa495000640 1 -- 172.21.15.6:0/603863526 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa490156e40 con 0x7fa490154c40 2024-04-24T23:25:28.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:28.126+0000 7fa495000640 1 -- 172.21.15.6:0/603863526 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa490157450 con 0x7fa490154c40 2024-04-24T23:25:28.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:28.131+0000 7fa487400640 1 -- 172.21.15.6:0/603863526 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fa488005d10 con 0x7fa490154c40 2024-04-24T23:25:28.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:28.133+0000 7fa487400640 1 --2- 172.21.15.6:0/603863526 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fa46c06bf00 0x7fa46c06e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:25:28.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:28.134+0000 7fa487400640 1 -- 172.21.15.6:0/603863526 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6848+0+0 (secure 0 0 0) 0x7fa48808b7e0 con 0x7fa490154c40 2024-04-24T23:25:28.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:28.134+0000 7fa48f400640 1 --2- 172.21.15.6:0/603863526 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fa46c06bf00 0x7fa46c06e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:25:28.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:28.135+0000 7fa487400640 1 -- 172.21.15.6:0/603863526 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fa488022020 con 0x7fa490154c40 2024-04-24T23:25:28.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:28.136+0000 7fa48f400640 1 --2- 172.21.15.6:0/603863526 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fa46c06bf00 0x7fa46c06e3c0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fa478034000 tx=0x7fa47802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:25:28.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:28.590+0000 7fa495000640 1 -- 172.21.15.6:0/603863526 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa490068b40 con 0x7fa490154c40 2024-04-24T23:25:28.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:28.591+0000 7fa487400640 1 -- 172.21.15.6:0/603863526 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) ==== 94+0+6856 (secure 0 0 0) 0x7fa490068b40 con 0x7fa490154c40 2024-04-24T23:25:28.592 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:25:28.592 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":48,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5081,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/1190759626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1190759626},{"type":"v1","addr":"172.21.15.6:6834","nonce":1190759626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":5249,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6838/2416336040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2416336040},{"type":"v1","addr":"172.21.15.190:6838","nonce":2416336040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":42},{"gid":5503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/3928731059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":3928731059},{"type":"v1","addr":"172.21.15.6:6837","nonce":3928731059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":35},{"gid":5725,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6837/1151527362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":1151527362},{"type":"v1","addr":"172.21.15.190:6837","nonce":1151527362}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":48}],"filesystems":[{"mdsmap":{"epoch":47,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:25:05.197173+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":4306,"mds_1":5599},"failed":[],"damaged":[],"stopped":[2,3,4],"info":{"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/489092393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":489092393},{"type":"v1","addr":"172.21.15.6:6835","nonce":489092393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5599":{"gid":5599,"name":"d","rank":1,"incarnation":43,"state":"up:active","state_seq":25,"addr":"172.21.15.190:6839/2643233757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":2643233757},{"type":"v1","addr":"172.21.15.190:6839","nonce":2643233757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4306,"qdb_cluster":[5599,4306]},"id":1}]} 2024-04-24T23:25:28.592 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 48 2024-04-24T23:25:28.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:28.594+0000 7fa495000640 1 -- 172.21.15.6:0/603863526 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fa46c06bf00 msgr2=0x7fa46c06e3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:25:28.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:28.594+0000 7fa495000640 1 --2- 172.21.15.6:0/603863526 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fa46c06bf00 0x7fa46c06e3c0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fa478034000 tx=0x7fa47802e000 comp rx=0 tx=0).stop 2024-04-24T23:25:28.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:28.594+0000 7fa495000640 1 -- 172.21.15.6:0/603863526 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa490154c40 msgr2=0x7fa4901f3eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:25:28.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:28.594+0000 7fa495000640 1 --2- 172.21.15.6:0/603863526 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa490154c40 0x7fa4901f3eb0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fa488009cf0 tx=0x7fa48800c490 comp rx=0 tx=0).stop 2024-04-24T23:25:28.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:28.595+0000 7fa495000640 1 -- 172.21.15.6:0/603863526 shutdown_connections 2024-04-24T23:25:28.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:28.595+0000 7fa495000640 1 --2- 172.21.15.6:0/603863526 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fa46c06bf00 0x7fa46c06e3c0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:28.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:28.595+0000 7fa495000640 1 --2- 172.21.15.6:0/603863526 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa4900cb5f0 0x7fa4900c8a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:28.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:28.595+0000 7fa495000640 1 --2- 172.21.15.6:0/603863526 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa490154c40 0x7fa4901f3eb0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:28.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:28.595+0000 7fa495000640 1 --2- 172.21.15.6:0/603863526 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa4900c53d0 0x7fa490157920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:28.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:28.595+0000 7fa495000640 1 -- 172.21.15.6:0/603863526 >> 172.21.15.6:0/603863526 conn(0x7fa49006cd70 msgr2=0x7fa490153980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:25:28.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:28.595+0000 7fa495000640 1 -- 172.21.15.6:0/603863526 shutdown_connections 2024-04-24T23:25:28.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:28.595+0000 7fa495000640 1 -- 172.21.15.6:0/603863526 wait complete. 2024-04-24T23:25:28.621 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-24T23:25:30.622 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:25:31.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:31.044+0000 7fee13a00640 1 Processor -- start 2024-04-24T23:25:31.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:31.045+0000 7fee13a00640 1 -- start start 2024-04-24T23:25:31.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:31.049+0000 7fee13a00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fee0c06bbd0 0x7fee0c150990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:25:31.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:31.049+0000 7fee13a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fee0c157470 0x7fee0c161370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:25:31.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:31.049+0000 7fee13a00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fee0c0c99a0 con 0x7fee0c073270 2024-04-24T23:25:31.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:31.050+0000 7fee13a00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7fee0c0c8c50 con 0x7fee0c157470 2024-04-24T23:25:31.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:31.050+0000 7fee13a00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fee0c0c8690 con 0x7fee0c06bbd0 2024-04-24T23:25:31.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:31.051+0000 7fee0be00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fee0c073270 0x7fee0c15a980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:41744/0 (socket says 172.21.15.6:41744) 2024-04-24T23:25:31.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:31.051+0000 7fee0be00640 1 -- 172.21.15.6:0/2958667895 learned_addr learned my addr 172.21.15.6:0/2958667895 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:25:31.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:31.051+0000 7fee11a00640 1 --2- 172.21.15.6:0/2958667895 >> v2:172.21.15.6:3300/0 conn(0x7fee0c157470 0x7fee0c161370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:25:31.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:31.052+0000 7fee11a00640 1 -- 172.21.15.6:0/2958667895 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fee0c06bbd0 msgr2=0x7fee0c150990 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-24T23:25:31.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:31.052+0000 7fee11a00640 1 --2- 172.21.15.6:0/2958667895 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fee0c06bbd0 0x7fee0c150990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:31.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:31.052+0000 7fee11a00640 1 -- 172.21.15.6:0/2958667895 >> v1:172.21.15.6:6789/0 conn(0x7fee0c073270 legacy=0x7fee0c15a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:25:31.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:31.052+0000 7fee11a00640 1 -- 172.21.15.6:0/2958667895 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee0c0c8970 con 0x7fee0c157470 2024-04-24T23:25:31.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:31.054+0000 7fee11a00640 1 --2- 172.21.15.6:0/2958667895 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fee0c157470 0x7fee0c161370 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fee0400aea0 tx=0x7fee0402b7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb39a5601e4cfca2 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:25:31.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:31.055+0000 7fee0b400640 1 -- 172.21.15.6:0/2958667895 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee04037070 con 0x7fee0c157470 2024-04-24T23:25:31.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:31.055+0000 7fee0b400640 1 -- 172.21.15.6:0/2958667895 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee04008640 con 0x7fee0c157470 2024-04-24T23:25:31.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:31.056+0000 7fee13a00640 1 -- 172.21.15.6:0/2958667895 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fee0c157470 msgr2=0x7fee0c161370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:25:31.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:31.056+0000 7fee13a00640 1 --2- 172.21.15.6:0/2958667895 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fee0c157470 0x7fee0c161370 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fee0400aea0 tx=0x7fee0402b7d0 comp rx=0 tx=0).stop 2024-04-24T23:25:31.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:31.056+0000 7fee13a00640 1 -- 172.21.15.6:0/2958667895 shutdown_connections 2024-04-24T23:25:31.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:31.056+0000 7fee13a00640 1 --2- 172.21.15.6:0/2958667895 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fee0c157470 0x7fee0c161370 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:31.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:31.056+0000 7fee13a00640 1 --2- 172.21.15.6:0/2958667895 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fee0c06bbd0 0x7fee0c150990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:31.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:31.056+0000 7fee13a00640 1 -- 172.21.15.6:0/2958667895 >> 172.21.15.6:0/2958667895 conn(0x7fee0c06b0e0 msgr2=0x7fee0c06b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:25:31.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:31.056+0000 7fee13a00640 1 -- 172.21.15.6:0/2958667895 shutdown_connections 2024-04-24T23:25:31.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:31.056+0000 7fee13a00640 1 -- 172.21.15.6:0/2958667895 wait complete. 2024-04-24T23:25:31.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:31.062+0000 7fee13a00640 1 Processor -- start 2024-04-24T23:25:31.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:31.063+0000 7fee13a00640 1 -- start start 2024-04-24T23:25:31.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:31.064+0000 7fee13a00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fee0c06bbd0 0x7fee0c14c020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:25:31.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:31.064+0000 7fee13a00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fee0c073270 0x7fee0c158b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:25:31.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:31.064+0000 7fee13a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fee0c157470 0x7fee0c0c4f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:25:31.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:31.065+0000 7fee0be00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fee0c073270 0x7fee0c158b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:25:31.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:31.065+0000 7fee0be00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fee0c073270 0x7fee0c158b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.6:48192/0 (socket says 172.21.15.6:48192) 2024-04-24T23:25:31.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:31.065+0000 7fee11a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fee0c157470 0x7fee0c0c4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:25:31.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:31.065+0000 7fee11a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fee0c157470 0x7fee0c0c4f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:47324/0 (socket says 172.21.15.6:47324) 2024-04-24T23:25:31.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:31.065+0000 7fee11000640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fee0c06bbd0 0x7fee0c14c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:25:31.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:31.066+0000 7fee13a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fee0c0c7a70 con 0x7fee0c157470 2024-04-24T23:25:31.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:31.066+0000 7fee13a00640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fee0c15a460 con 0x7fee0c06bbd0 2024-04-24T23:25:31.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:31.066+0000 7fee13a00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fee0c0c45f0 con 0x7fee0c073270 2024-04-24T23:25:31.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:31.066+0000 7fee11000640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fee0c06bbd0 0x7fee0c14c020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:43732/0 (socket says 172.21.15.6:43732) 2024-04-24T23:25:31.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:31.066+0000 7fee11000640 1 -- 172.21.15.6:0/4249415625 learned_addr learned my addr 172.21.15.6:0/4249415625 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:25:31.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:31.066+0000 7fee11a00640 1 -- 172.21.15.6:0/4249415625 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fee0c073270 msgr2=0x7fee0c158b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:25:31.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:31.066+0000 7fee11a00640 1 --2- 172.21.15.6:0/4249415625 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fee0c073270 0x7fee0c158b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:31.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:31.067+0000 7fee11a00640 1 -- 172.21.15.6:0/4249415625 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fee0c06bbd0 msgr2=0x7fee0c14c020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:25:31.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:31.067+0000 7fee11a00640 1 --2- 172.21.15.6:0/4249415625 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fee0c06bbd0 0x7fee0c14c020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:31.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:31.067+0000 7fee11a00640 1 -- 172.21.15.6:0/4249415625 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee0c14c560 con 0x7fee0c157470 2024-04-24T23:25:31.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:31.067+0000 7fee0be00640 1 --2- 172.21.15.6:0/4249415625 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fee0c073270 0x7fee0c158b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:25:31.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:31.067+0000 7fee11000640 1 --2- 172.21.15.6:0/4249415625 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fee0c06bbd0 0x7fee0c14c020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello state changed while learned_addr, mark_down or replacing must be happened just now 2024-04-24T23:25:31.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:31.067+0000 7fee11a00640 1 --2- 172.21.15.6:0/4249415625 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fee0c157470 0x7fee0c0c4f70 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fee04031000 tx=0x7fee04004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:25:31.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:31.068+0000 7fee08c00640 1 -- 172.21.15.6:0/4249415625 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee04040030 con 0x7fee0c157470 2024-04-24T23:25:31.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:31.068+0000 7fee13a00640 1 -- 172.21.15.6:0/4249415625 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee0c1f4760 con 0x7fee0c157470 2024-04-24T23:25:31.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:31.068+0000 7fee13a00640 1 -- 172.21.15.6:0/4249415625 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee0c158540 con 0x7fee0c157470 2024-04-24T23:25:31.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:31.069+0000 7fee08c00640 1 -- 172.21.15.6:0/4249415625 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee040082d0 con 0x7fee0c157470 2024-04-24T23:25:31.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:31.069+0000 7fee08c00640 1 -- 172.21.15.6:0/4249415625 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee04040030 con 0x7fee0c157470 2024-04-24T23:25:31.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:31.070+0000 7fee13a00640 1 -- 172.21.15.6:0/4249415625 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fede8005290 con 0x7fee0c157470 2024-04-24T23:25:31.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:31.076+0000 7fee08c00640 1 -- 172.21.15.6:0/4249415625 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fee04043070 con 0x7fee0c157470 2024-04-24T23:25:31.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:31.079+0000 7fee08c00640 1 --2- 172.21.15.6:0/4249415625 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fedec06c230 0x7fedec06e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:25:31.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:31.079+0000 7fee08c00640 1 -- 172.21.15.6:0/4249415625 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6848+0+0 (secure 0 0 0) 0x7fee04041070 con 0x7fee0c157470 2024-04-24T23:25:31.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:31.080+0000 7fee11000640 1 --2- 172.21.15.6:0/4249415625 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fedec06c230 0x7fedec06e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:25:31.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:31.083+0000 7fee08c00640 1 -- 172.21.15.6:0/4249415625 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fee0407a460 con 0x7fee0c157470 2024-04-24T23:25:31.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:31.083+0000 7fee11000640 1 --2- 172.21.15.6:0/4249415625 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fedec06c230 0x7fedec06e6f0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fedf800a8f0 tx=0x7fedf8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:25:31.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:31.623+0000 7fee13a00640 1 -- 172.21.15.6:0/4249415625 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fede8005580 con 0x7fee0c157470 2024-04-24T23:25:31.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:31.627+0000 7fee08c00640 1 -- 172.21.15.6:0/4249415625 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) ==== 94+0+6856 (secure 0 0 0) 0x7fee04079d60 con 0x7fee0c157470 2024-04-24T23:25:31.629 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:25:31.629 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":48,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5081,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/1190759626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1190759626},{"type":"v1","addr":"172.21.15.6:6834","nonce":1190759626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":5249,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6838/2416336040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2416336040},{"type":"v1","addr":"172.21.15.190:6838","nonce":2416336040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":42},{"gid":5503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/3928731059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":3928731059},{"type":"v1","addr":"172.21.15.6:6837","nonce":3928731059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":35},{"gid":5725,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6837/1151527362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":1151527362},{"type":"v1","addr":"172.21.15.190:6837","nonce":1151527362}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":48}],"filesystems":[{"mdsmap":{"epoch":47,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:25:05.197173+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":4306,"mds_1":5599},"failed":[],"damaged":[],"stopped":[2,3,4],"info":{"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/489092393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":489092393},{"type":"v1","addr":"172.21.15.6:6835","nonce":489092393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5599":{"gid":5599,"name":"d","rank":1,"incarnation":43,"state":"up:active","state_seq":25,"addr":"172.21.15.190:6839/2643233757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":2643233757},{"type":"v1","addr":"172.21.15.190:6839","nonce":2643233757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4306,"qdb_cluster":[5599,4306]},"id":1}]} 2024-04-24T23:25:31.629 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 48 2024-04-24T23:25:31.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:31.636+0000 7fee13a00640 1 -- 172.21.15.6:0/4249415625 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fedec06c230 msgr2=0x7fedec06e6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:25:31.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:31.636+0000 7fee13a00640 1 --2- 172.21.15.6:0/4249415625 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fedec06c230 0x7fedec06e6f0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fedf800a8f0 tx=0x7fedf8008040 comp rx=0 tx=0).stop 2024-04-24T23:25:31.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:31.636+0000 7fee13a00640 1 -- 172.21.15.6:0/4249415625 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fee0c157470 msgr2=0x7fee0c0c4f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:25:31.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:31.636+0000 7fee13a00640 1 --2- 172.21.15.6:0/4249415625 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fee0c157470 0x7fee0c0c4f70 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fee04031000 tx=0x7fee04004a40 comp rx=0 tx=0).stop 2024-04-24T23:25:31.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:31.637+0000 7fee13a00640 1 -- 172.21.15.6:0/4249415625 shutdown_connections 2024-04-24T23:25:31.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:31.637+0000 7fee13a00640 1 --2- 172.21.15.6:0/4249415625 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fedec06c230 0x7fedec06e6f0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:31.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:31.637+0000 7fee13a00640 1 --2- 172.21.15.6:0/4249415625 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fee0c073270 0x7fee0c158b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:31.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:31.637+0000 7fee13a00640 1 --2- 172.21.15.6:0/4249415625 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fee0c157470 0x7fee0c0c4f70 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:31.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:31.637+0000 7fee13a00640 1 --2- 172.21.15.6:0/4249415625 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fee0c06bbd0 0x7fee0c14c020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:31.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:31.637+0000 7fee13a00640 1 -- 172.21.15.6:0/4249415625 >> 172.21.15.6:0/4249415625 conn(0x7fee0c06b0e0 msgr2=0x7fee0c06b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:25:31.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:31.638+0000 7fee13a00640 1 -- 172.21.15.6:0/4249415625 shutdown_connections 2024-04-24T23:25:31.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:31.641+0000 7fee13a00640 1 -- 172.21.15.6:0/4249415625 wait complete. 2024-04-24T23:25:31.698 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-24T23:25:33.700 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:25:34.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:34.135+0000 7f824ee00640 1 Processor -- start 2024-04-24T23:25:34.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:34.135+0000 7f824ee00640 1 -- start start 2024-04-24T23:25:34.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:34.141+0000 7f824ee00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f82480c92c0 0x7f82480cc9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:25:34.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:34.141+0000 7f824ee00640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f824806d630 0x7f824806dac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:25:34.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:34.143+0000 7f824ee00640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f82480cd080 con 0x7f82480c8ee0 2024-04-24T23:25:34.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:34.143+0000 7f824da00640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f82480c8ee0 0x7f8248162c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.6:40376/0 (socket says 172.21.15.6:40376) 2024-04-24T23:25:34.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:34.144+0000 7f824da00640 1 -- 172.21.15.6:0/1203266259 learned_addr learned my addr 172.21.15.6:0/1203266259 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:25:34.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:34.144+0000 7f824ee00640 1 -- 172.21.15.6:0/1203266259 --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f824806d1a0 con 0x7f824806d630 2024-04-24T23:25:34.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:34.144+0000 7f824ee00640 1 -- 172.21.15.6:0/1203266259 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f8248070f10 con 0x7f82480c92c0 2024-04-24T23:25:34.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:34.143+0000 7f824e400640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f824806d630 0x7f824806dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:25:34.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:34.143+0000 7f824d000640 1 --2- 172.21.15.6:0/1203266259 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f82480c92c0 0x7f82480cc9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:25:34.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:34.146+0000 7f8247e00640 1 -- 172.21.15.6:0/1203266259 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1830805759 0 0) 0x7f82480cd080 con 0x7f82480c8ee0 2024-04-24T23:25:34.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:34.146+0000 7f8247e00640 1 -- 172.21.15.6:0/1203266259 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8230003520 con 0x7f82480c8ee0 2024-04-24T23:25:34.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:34.147+0000 7f824e400640 1 -- 172.21.15.6:0/1203266259 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f82480c92c0 msgr2=0x7f82480cc9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:25:34.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:34.147+0000 7f824e400640 1 --2- 172.21.15.6:0/1203266259 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f82480c92c0 0x7f82480cc9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:34.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:34.147+0000 7f824e400640 1 -- 172.21.15.6:0/1203266259 >> v1:172.21.15.190:6789/0 conn(0x7f82480c8ee0 legacy=0x7f8248162c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:25:34.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:34.147+0000 7f824e400640 1 -- 172.21.15.6:0/1203266259 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82480711f0 con 0x7f824806d630 2024-04-24T23:25:34.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:34.148+0000 7f824d000640 1 --2- 172.21.15.6:0/1203266259 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f82480c92c0 0x7f82480cc9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:25:34.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:34.152+0000 7f824e400640 1 --2- 172.21.15.6:0/1203266259 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f824806d630 0x7f824806dac0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f82340097a0 tx=0x7f823402a1a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=92a15d3af55c8601 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:25:34.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:34.156+0000 7f8247e00640 1 -- 172.21.15.6:0/1203266259 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8234037070 con 0x7f824806d630 2024-04-24T23:25:34.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:34.156+0000 7f8247e00640 1 -- 172.21.15.6:0/1203266259 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f823402ae30 con 0x7f824806d630 2024-04-24T23:25:34.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:34.156+0000 7f8247e00640 1 -- 172.21.15.6:0/1203266259 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8234032640 con 0x7f824806d630 2024-04-24T23:25:34.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:34.157+0000 7f824ee00640 1 -- 172.21.15.6:0/1203266259 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f824806d630 msgr2=0x7f824806dac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:25:34.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:34.157+0000 7f824ee00640 1 --2- 172.21.15.6:0/1203266259 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f824806d630 0x7f824806dac0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f82340097a0 tx=0x7f823402a1a0 comp rx=0 tx=0).stop 2024-04-24T23:25:34.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:34.159+0000 7f824ee00640 1 -- 172.21.15.6:0/1203266259 shutdown_connections 2024-04-24T23:25:34.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:34.159+0000 7f824ee00640 1 --2- 172.21.15.6:0/1203266259 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f824806d630 0x7f824806dac0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:34.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:34.159+0000 7f824ee00640 1 --2- 172.21.15.6:0/1203266259 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f82480c92c0 0x7f82480cc9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:34.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:34.159+0000 7f824ee00640 1 -- 172.21.15.6:0/1203266259 >> 172.21.15.6:0/1203266259 conn(0x7f824806ed80 msgr2=0x7f824806f190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:25:34.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:34.161+0000 7f824ee00640 1 -- 172.21.15.6:0/1203266259 shutdown_connections 2024-04-24T23:25:34.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:34.161+0000 7f824ee00640 1 -- 172.21.15.6:0/1203266259 wait complete. 2024-04-24T23:25:34.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:34.168+0000 7f824ee00640 1 Processor -- start 2024-04-24T23:25:34.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:34.170+0000 7f824ee00640 1 -- start start 2024-04-24T23:25:34.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:34.172+0000 7f824ee00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f824806d630 0x7f82480c4990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:25:34.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:34.172+0000 7f824ee00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f82480c8ee0 0x7f8248143060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:25:34.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:34.172+0000 7f824ee00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f82480c92c0 0x7f824813efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:25:34.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:34.172+0000 7f824ee00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f8248072420 con 0x7f82480c8ee0 2024-04-24T23:25:34.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:34.172+0000 7f824ee00640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f824806ce30 con 0x7f824806d630 2024-04-24T23:25:34.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:34.172+0000 7f824ee00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f824813ebd0 con 0x7f82480c92c0 2024-04-24T23:25:34.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:34.172+0000 7f824da00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f824806d630 0x7f82480c4990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:25:34.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:34.174+0000 7f824d000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f82480c8ee0 0x7f8248143060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:25:34.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:34.174+0000 7f824da00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f824806d630 0x7f82480c4990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:43762/0 (socket says 172.21.15.6:43762) 2024-04-24T23:25:34.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:34.174+0000 7f824da00640 1 -- 172.21.15.6:0/1636498673 learned_addr learned my addr 172.21.15.6:0/1636498673 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:25:34.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:34.175+0000 7f824d000640 1 -- 172.21.15.6:0/1636498673 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f82480c92c0 msgr2=0x7f824813efa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-24T23:25:34.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:34.175+0000 7f824d000640 1 --2- 172.21.15.6:0/1636498673 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f82480c92c0 0x7f824813efa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:34.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:34.175+0000 7f824d000640 1 -- 172.21.15.6:0/1636498673 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f824806d630 msgr2=0x7f82480c4990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:25:34.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:34.175+0000 7f824d000640 1 --2- 172.21.15.6:0/1636498673 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f824806d630 0x7f82480c4990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:34.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:34.175+0000 7f824d000640 1 -- 172.21.15.6:0/1636498673 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8248150680 con 0x7f82480c8ee0 2024-04-24T23:25:34.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:34.176+0000 7f824d000640 1 --2- 172.21.15.6:0/1636498673 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f82480c8ee0 0x7f8248143060 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f823c00e660 tx=0x7f823c00eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:25:34.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:34.177+0000 7f8245600640 1 -- 172.21.15.6:0/1636498673 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f823c039070 con 0x7f82480c8ee0 2024-04-24T23:25:34.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:34.177+0000 7f824ee00640 1 -- 172.21.15.6:0/1636498673 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82481514c0 con 0x7f82480c8ee0 2024-04-24T23:25:34.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:34.178+0000 7f824ee00640 1 -- 172.21.15.6:0/1636498673 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f824814fba0 con 0x7f82480c8ee0 2024-04-24T23:25:34.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:34.183+0000 7f823aa00640 1 -- 172.21.15.6:0/1636498673 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f82480c3a60 con 0x7f82480c8ee0 2024-04-24T23:25:34.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:34.186+0000 7f8245600640 1 -- 172.21.15.6:0/1636498673 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f823c00c8d0 con 0x7f82480c8ee0 2024-04-24T23:25:34.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:34.187+0000 7f8245600640 1 -- 172.21.15.6:0/1636498673 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f823c043060 con 0x7f82480c8ee0 2024-04-24T23:25:34.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:34.201+0000 7f8245600640 1 -- 172.21.15.6:0/1636498673 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f823c03c070 con 0x7f82480c8ee0 2024-04-24T23:25:34.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:34.206+0000 7f8245600640 1 --2- 172.21.15.6:0/1636498673 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f822806c230 0x7f822806e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:25:34.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:34.207+0000 7f8245600640 1 -- 172.21.15.6:0/1636498673 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6848+0+0 (secure 0 0 0) 0x7f823c00c6d0 con 0x7f82480c8ee0 2024-04-24T23:25:34.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:34.207+0000 7f824da00640 1 --2- 172.21.15.6:0/1636498673 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f822806c230 0x7f822806e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:25:34.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:34.213+0000 7f824da00640 1 --2- 172.21.15.6:0/1636498673 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f822806c230 0x7f822806e6f0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f82400012e0 tx=0x7f8240009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:25:34.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:34.216+0000 7f8245600640 1 -- 172.21.15.6:0/1636498673 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f82480c3a60 con 0x7f82480c8ee0 2024-04-24T23:25:34.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:34.876+0000 7f823aa00640 1 -- 172.21.15.6:0/1636498673 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8248075b30 con 0x7f82480c8ee0 2024-04-24T23:25:34.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:34.881+0000 7f8245600640 1 -- 172.21.15.6:0/1636498673 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) ==== 94+0+6856 (secure 0 0 0) 0x7f823c07ac70 con 0x7f82480c8ee0 2024-04-24T23:25:34.882 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:25:34.882 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":48,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5081,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/1190759626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1190759626},{"type":"v1","addr":"172.21.15.6:6834","nonce":1190759626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":5249,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6838/2416336040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2416336040},{"type":"v1","addr":"172.21.15.190:6838","nonce":2416336040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":42},{"gid":5503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/3928731059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":3928731059},{"type":"v1","addr":"172.21.15.6:6837","nonce":3928731059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":35},{"gid":5725,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6837/1151527362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":1151527362},{"type":"v1","addr":"172.21.15.190:6837","nonce":1151527362}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":48}],"filesystems":[{"mdsmap":{"epoch":47,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:25:05.197173+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":4306,"mds_1":5599},"failed":[],"damaged":[],"stopped":[2,3,4],"info":{"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/489092393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":489092393},{"type":"v1","addr":"172.21.15.6:6835","nonce":489092393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5599":{"gid":5599,"name":"d","rank":1,"incarnation":43,"state":"up:active","state_seq":25,"addr":"172.21.15.190:6839/2643233757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":2643233757},{"type":"v1","addr":"172.21.15.190:6839","nonce":2643233757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4306,"qdb_cluster":[5599,4306]},"id":1}]} 2024-04-24T23:25:34.883 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 48 2024-04-24T23:25:34.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:34.894+0000 7f823aa00640 1 -- 172.21.15.6:0/1636498673 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f822806c230 msgr2=0x7f822806e6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:25:34.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:34.894+0000 7f823aa00640 1 --2- 172.21.15.6:0/1636498673 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f822806c230 0x7f822806e6f0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f82400012e0 tx=0x7f8240009040 comp rx=0 tx=0).stop 2024-04-24T23:25:34.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:34.894+0000 7f823aa00640 1 -- 172.21.15.6:0/1636498673 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f82480c8ee0 msgr2=0x7f8248143060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:25:34.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:34.894+0000 7f823aa00640 1 --2- 172.21.15.6:0/1636498673 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f82480c8ee0 0x7f8248143060 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f823c00e660 tx=0x7f823c00eb20 comp rx=0 tx=0).stop 2024-04-24T23:25:34.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:34.894+0000 7f823aa00640 1 -- 172.21.15.6:0/1636498673 shutdown_connections 2024-04-24T23:25:34.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:34.894+0000 7f823aa00640 1 --2- 172.21.15.6:0/1636498673 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f822806c230 0x7f822806e6f0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:34.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:34.894+0000 7f823aa00640 1 --2- 172.21.15.6:0/1636498673 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f82480c92c0 0x7f824813efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:34.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:34.894+0000 7f823aa00640 1 --2- 172.21.15.6:0/1636498673 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f82480c8ee0 0x7f8248143060 secure :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f823c00e660 tx=0x7f823c00eb20 comp rx=0 tx=0).stop 2024-04-24T23:25:34.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:34.894+0000 7f823aa00640 1 --2- 172.21.15.6:0/1636498673 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f824806d630 0x7f82480c4990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:34.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:34.894+0000 7f823aa00640 1 -- 172.21.15.6:0/1636498673 >> 172.21.15.6:0/1636498673 conn(0x7f824806ed80 msgr2=0x7f82481576b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:25:34.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:34.895+0000 7f823aa00640 1 -- 172.21.15.6:0/1636498673 shutdown_connections 2024-04-24T23:25:34.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:34.895+0000 7f823aa00640 1 -- 172.21.15.6:0/1636498673 wait complete. 2024-04-24T23:25:34.933 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-24T23:25:36.934 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:25:37.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:37.316+0000 7f0731400640 1 Processor -- start 2024-04-24T23:25:37.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:37.318+0000 7f0731400640 1 -- start start 2024-04-24T23:25:37.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:37.325+0000 7f0731400640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f072c151390 0x7f072c157a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:25:37.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:37.325+0000 7f0731400640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f072c0caea0 0x7f072c0cbc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:25:37.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:37.325+0000 7f0731400640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f072c157f90 con 0x7f072c0cc1b0 2024-04-24T23:25:37.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:37.326+0000 7f0731400640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f072c151770 con 0x7f072c151390 2024-04-24T23:25:37.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:37.326+0000 7f0731400640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f072c15a760 con 0x7f072c0caea0 2024-04-24T23:25:37.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:37.326+0000 7f072b200640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f072c0cc1b0 0x7f072c0c56f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:51308/0 (socket says 172.21.15.6:51308) 2024-04-24T23:25:37.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:37.326+0000 7f072b200640 1 -- 172.21.15.6:0/3908773789 learned_addr learned my addr 172.21.15.6:0/3908773789 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:25:37.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:37.327+0000 7f0729e00640 1 --2- 172.21.15.6:0/3908773789 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f072c0caea0 0x7f072c0cbc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:25:37.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:37.327+0000 7f0729e00640 1 -- 172.21.15.6:0/3908773789 >> v2:172.21.15.190:3300/0 conn(0x7f072c151390 msgr2=0x7f072c157a50 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-24T23:25:37.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:37.328+0000 7f0729e00640 1 --2- 172.21.15.6:0/3908773789 >> v2:172.21.15.190:3300/0 conn(0x7f072c151390 0x7f072c157a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:37.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:37.328+0000 7f0729e00640 1 -- 172.21.15.6:0/3908773789 >> v1:172.21.15.6:6789/0 conn(0x7f072c0cc1b0 legacy=0x7f072c0c56f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:25:37.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:37.328+0000 7f0729e00640 1 -- 172.21.15.6:0/3908773789 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f072c15aa40 con 0x7f072c0caea0 2024-04-24T23:25:37.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:37.330+0000 7f0729e00640 1 --2- 172.21.15.6:0/3908773789 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f072c0caea0 0x7f072c0cbc70 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f071800ba40 tx=0x7f071802c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9cccfbd6b8223d88 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:25:37.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:37.331+0000 7f0729400640 1 -- 172.21.15.6:0/3908773789 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f071803a070 con 0x7f072c0caea0 2024-04-24T23:25:37.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:37.331+0000 7f0729400640 1 -- 172.21.15.6:0/3908773789 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0718032480 con 0x7f072c0caea0 2024-04-24T23:25:37.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:37.331+0000 7f0729400640 1 -- 172.21.15.6:0/3908773789 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f071803a070 con 0x7f072c0caea0 2024-04-24T23:25:37.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:37.332+0000 7f0731400640 1 -- 172.21.15.6:0/3908773789 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f072c0caea0 msgr2=0x7f072c0cbc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:25:37.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:37.332+0000 7f0731400640 1 --2- 172.21.15.6:0/3908773789 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f072c0caea0 0x7f072c0cbc70 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f071800ba40 tx=0x7f071802c650 comp rx=0 tx=0).stop 2024-04-24T23:25:37.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:37.339+0000 7f0731400640 1 -- 172.21.15.6:0/3908773789 shutdown_connections 2024-04-24T23:25:37.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:37.339+0000 7f0731400640 1 --2- 172.21.15.6:0/3908773789 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f072c0caea0 0x7f072c0cbc70 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:37.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:37.339+0000 7f0731400640 1 --2- 172.21.15.6:0/3908773789 >> v2:172.21.15.190:3300/0 conn(0x7f072c151390 0x7f072c157a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:37.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:37.339+0000 7f0731400640 1 -- 172.21.15.6:0/3908773789 >> 172.21.15.6:0/3908773789 conn(0x7f072c0cc9a0 msgr2=0x7f072c0ccd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:25:37.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:37.341+0000 7f0731400640 1 -- 172.21.15.6:0/3908773789 shutdown_connections 2024-04-24T23:25:37.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:37.342+0000 7f0731400640 1 -- 172.21.15.6:0/3908773789 wait complete. 2024-04-24T23:25:37.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:37.344+0000 7f0731400640 1 Processor -- start 2024-04-24T23:25:37.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:37.347+0000 7f0731400640 1 -- start start 2024-04-24T23:25:37.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:37.349+0000 7f0731400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f072c0caea0 0x7f072c072a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:25:37.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:37.350+0000 7f0731400640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f072c0cc1b0 0x7f072c0c93f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:25:37.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:37.350+0000 7f072a800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f072c0caea0 0x7f072c072a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:25:37.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:37.351+0000 7f0731400640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f072c151390 0x7f072c1f0170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:25:37.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:37.351+0000 7f0731400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f072c15b2e0 con 0x7f072c0caea0 2024-04-24T23:25:37.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:37.351+0000 7f072b200640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f072c151390 0x7f072c1f0170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:25:37.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:37.351+0000 7f072b200640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f072c151390 0x7f072c1f0170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.6:57234/0 (socket says 172.21.15.6:57234) 2024-04-24T23:25:37.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:37.351+0000 7f072b200640 1 -- 172.21.15.6:0/2582187041 learned_addr learned my addr 172.21.15.6:0/2582187041 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:25:37.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:37.351+0000 7f0731400640 1 -- 172.21.15.6:0/2582187041 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f072c15b000 con 0x7f072c0cc1b0 2024-04-24T23:25:37.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:37.351+0000 7f0731400640 1 -- 172.21.15.6:0/2582187041 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f072c0cb2e0 con 0x7f072c151390 2024-04-24T23:25:37.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:37.352+0000 7f0729e00640 1 --2- 172.21.15.6:0/2582187041 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f072c0cc1b0 0x7f072c0c93f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:25:37.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:37.353+0000 7f072b200640 1 -- 172.21.15.6:0/2582187041 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f072c0cc1b0 msgr2=0x7f072c0c93f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:25:37.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:37.353+0000 7f072b200640 1 --2- 172.21.15.6:0/2582187041 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f072c0cc1b0 0x7f072c0c93f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:37.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:37.354+0000 7f072b200640 1 -- 172.21.15.6:0/2582187041 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f072c0caea0 msgr2=0x7f072c072a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:25:37.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:37.354+0000 7f072b200640 1 --2- 172.21.15.6:0/2582187041 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f072c0caea0 0x7f072c072a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:37.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:37.354+0000 7f072b200640 1 -- 172.21.15.6:0/2582187041 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f072c155ed0 con 0x7f072c151390 2024-04-24T23:25:37.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:37.354+0000 7f072a800640 1 --2- 172.21.15.6:0/2582187041 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f072c0caea0 0x7f072c072a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:25:37.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:37.362+0000 7f072b200640 1 --2- 172.21.15.6:0/2582187041 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f072c151390 0x7f072c1f0170 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f072400ca80 tx=0x7f072400cf40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:25:37.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:37.362+0000 7f0722a00640 1 -- 172.21.15.6:0/2582187041 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0724017070 con 0x7f072c151390 2024-04-24T23:25:37.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:37.362+0000 7f0731400640 1 -- 172.21.15.6:0/2582187041 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f072c0c3970 con 0x7f072c151390 2024-04-24T23:25:37.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:37.363+0000 7f0731400640 1 -- 172.21.15.6:0/2582187041 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f072c150c00 con 0x7f072c151390 2024-04-24T23:25:37.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:37.363+0000 7f0722a00640 1 -- 172.21.15.6:0/2582187041 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f072400ac30 con 0x7f072c151390 2024-04-24T23:25:37.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:37.365+0000 7f0722a00640 1 -- 172.21.15.6:0/2582187041 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0724012600 con 0x7f072c151390 2024-04-24T23:25:37.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:37.367+0000 7f0731400640 1 -- 172.21.15.6:0/2582187041 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0704005290 con 0x7f072c151390 2024-04-24T23:25:37.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:37.372+0000 7f0722a00640 1 -- 172.21.15.6:0/2582187041 <== mon.2 v2:172.21.15.190:3301/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f072401a070 con 0x7f072c151390 2024-04-24T23:25:37.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:37.374+0000 7f0722a00640 1 --2- 172.21.15.6:0/2582187041 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f070c06be30 0x7f070c06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:25:37.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:37.374+0000 7f0722a00640 1 -- 172.21.15.6:0/2582187041 <== mon.2 v2:172.21.15.190:3301/0 5 ==== osd_map(29..29 src has 1..29) ==== 6848+0+0 (secure 0 0 0) 0x7f072408af70 con 0x7f072c151390 2024-04-24T23:25:37.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:37.380+0000 7f072a800640 1 --2- 172.21.15.6:0/2582187041 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f070c06be30 0x7f070c06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:25:37.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:37.382+0000 7f0722a00640 1 -- 172.21.15.6:0/2582187041 <== mon.2 v2:172.21.15.190:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f072400ae20 con 0x7f072c151390 2024-04-24T23:25:37.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:37.386+0000 7f072a800640 1 --2- 172.21.15.6:0/2582187041 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f070c06be30 0x7f070c06e2f0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f071c006fd0 tx=0x7f071c0090e0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:25:37.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:37.928+0000 7f0731400640 1 -- 172.21.15.6:0/2582187041 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0704005580 con 0x7f072c151390 2024-04-24T23:25:37.931 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:25:37.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:37.930+0000 7f0722a00640 1 -- 172.21.15.6:0/2582187041 <== mon.2 v2:172.21.15.190:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) ==== 94+0+6856 (secure 0 0 0) 0x7f0724058d50 con 0x7f072c151390 2024-04-24T23:25:37.932 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":48,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5081,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/1190759626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1190759626},{"type":"v1","addr":"172.21.15.6:6834","nonce":1190759626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":5249,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6838/2416336040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2416336040},{"type":"v1","addr":"172.21.15.190:6838","nonce":2416336040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":42},{"gid":5503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/3928731059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":3928731059},{"type":"v1","addr":"172.21.15.6:6837","nonce":3928731059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":35},{"gid":5725,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6837/1151527362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":1151527362},{"type":"v1","addr":"172.21.15.190:6837","nonce":1151527362}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":48}],"filesystems":[{"mdsmap":{"epoch":47,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:25:05.197173+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":4306,"mds_1":5599},"failed":[],"damaged":[],"stopped":[2,3,4],"info":{"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/489092393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":489092393},{"type":"v1","addr":"172.21.15.6:6835","nonce":489092393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5599":{"gid":5599,"name":"d","rank":1,"incarnation":43,"state":"up:active","state_seq":25,"addr":"172.21.15.190:6839/2643233757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":2643233757},{"type":"v1","addr":"172.21.15.190:6839","nonce":2643233757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4306,"qdb_cluster":[5599,4306]},"id":1}]} 2024-04-24T23:25:37.932 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 48 2024-04-24T23:25:37.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:37.934+0000 7f0731400640 1 -- 172.21.15.6:0/2582187041 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f070c06be30 msgr2=0x7f070c06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:25:37.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:37.934+0000 7f0731400640 1 --2- 172.21.15.6:0/2582187041 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f070c06be30 0x7f070c06e2f0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f071c006fd0 tx=0x7f071c0090e0 comp rx=0 tx=0).stop 2024-04-24T23:25:37.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:37.935+0000 7f0731400640 1 -- 172.21.15.6:0/2582187041 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f072c151390 msgr2=0x7f072c1f0170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:25:37.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:37.935+0000 7f0731400640 1 --2- 172.21.15.6:0/2582187041 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f072c151390 0x7f072c1f0170 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f072400ca80 tx=0x7f072400cf40 comp rx=0 tx=0).stop 2024-04-24T23:25:37.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:37.935+0000 7f0731400640 1 -- 172.21.15.6:0/2582187041 shutdown_connections 2024-04-24T23:25:37.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:37.935+0000 7f0731400640 1 --2- 172.21.15.6:0/2582187041 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f070c06be30 0x7f070c06e2f0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:37.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:37.935+0000 7f0731400640 1 --2- 172.21.15.6:0/2582187041 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f072c151390 0x7f072c1f0170 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:37.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:37.935+0000 7f0731400640 1 --2- 172.21.15.6:0/2582187041 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f072c0cc1b0 0x7f072c0c93f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:37.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:37.935+0000 7f0731400640 1 --2- 172.21.15.6:0/2582187041 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f072c0caea0 0x7f072c072a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:37.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:37.935+0000 7f0731400640 1 -- 172.21.15.6:0/2582187041 >> 172.21.15.6:0/2582187041 conn(0x7f072c0cc9a0 msgr2=0x7f072c0721a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:25:37.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:37.936+0000 7f0731400640 1 -- 172.21.15.6:0/2582187041 shutdown_connections 2024-04-24T23:25:37.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:37.936+0000 7f0731400640 1 -- 172.21.15.6:0/2582187041 wait complete. 2024-04-24T23:25:37.973 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-24T23:25:39.974 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:25:40.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:40.319+0000 7f65bf200640 1 Processor -- start 2024-04-24T23:25:40.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:40.320+0000 7f65bf200640 1 -- start start 2024-04-24T23:25:40.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:40.324+0000 7f65bf200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f65b004b500 0x7f65b00dac60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:25:40.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:40.324+0000 7f65bf200640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f65b004b8e0 0x7f65b0005ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:25:40.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:40.325+0000 7f65b7400640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f65b004b8e0 0x7f65b0005ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:25:40.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:40.325+0000 7f65b7400640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f65b004b8e0 0x7f65b0005ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:53404/0 (socket says 172.21.15.6:53404) 2024-04-24T23:25:40.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:40.325+0000 7f65b7400640 1 -- 172.21.15.6:0/2860907792 learned_addr learned my addr 172.21.15.6:0/2860907792 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:25:40.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:40.325+0000 7f65bf200640 1 -- 172.21.15.6:0/2860907792 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f65b0007760 con 0x7f65b0006420 2024-04-24T23:25:40.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:40.325+0000 7f65bf200640 1 -- 172.21.15.6:0/2860907792 --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f65b0006900 con 0x7f65b004b500 2024-04-24T23:25:40.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:40.325+0000 7f65bf200640 1 -- 172.21.15.6:0/2860907792 --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f65b0006be0 con 0x7f65b004b8e0 2024-04-24T23:25:40.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:40.326+0000 7f65b6a00640 1 -- 172.21.15.6:0/2860907792 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3776865245 0 0) 0x7f65b0007760 con 0x7f65b0006420 2024-04-24T23:25:40.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:40.326+0000 7f65b6a00640 1 -- 172.21.15.6:0/2860907792 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f65a0003520 con 0x7f65b0006420 2024-04-24T23:25:40.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:40.326+0000 7f65b7400640 1 -- 172.21.15.6:0/2860907792 >> v2:172.21.15.6:3300/0 conn(0x7f65b004b500 msgr2=0x7f65b00dac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:25:40.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:40.326+0000 7f65b7400640 1 --2- 172.21.15.6:0/2860907792 >> v2:172.21.15.6:3300/0 conn(0x7f65b004b500 0x7f65b00dac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:40.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:40.326+0000 7f65b7400640 1 -- 172.21.15.6:0/2860907792 >> v1:172.21.15.6:6789/0 conn(0x7f65b0006420 legacy=0x7f65b0034a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:25:40.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:40.326+0000 7f65b7400640 1 -- 172.21.15.6:0/2860907792 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65b0006ec0 con 0x7f65b004b8e0 2024-04-24T23:25:40.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:40.329+0000 7f65b7400640 1 --2- 172.21.15.6:0/2860907792 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f65b004b8e0 0x7f65b0005ee0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f65b00d5520 tx=0x7f65a402a330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ed3e43ed70c5f0a4 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:25:40.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:40.331+0000 7f65b6a00640 1 -- 172.21.15.6:0/2860907792 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65a4037070 con 0x7f65b004b8e0 2024-04-24T23:25:40.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:40.331+0000 7f65b6a00640 1 -- 172.21.15.6:0/2860907792 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f65a4032640 con 0x7f65b004b8e0 2024-04-24T23:25:40.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:40.331+0000 7f65b6a00640 1 -- 172.21.15.6:0/2860907792 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65a4032a10 con 0x7f65b004b8e0 2024-04-24T23:25:40.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:40.332+0000 7f65bf200640 1 -- 172.21.15.6:0/2860907792 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f65b004b8e0 msgr2=0x7f65b0005ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:25:40.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:40.332+0000 7f65bf200640 1 --2- 172.21.15.6:0/2860907792 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f65b004b8e0 0x7f65b0005ee0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f65b00d5520 tx=0x7f65a402a330 comp rx=0 tx=0).stop 2024-04-24T23:25:40.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:40.333+0000 7f65bf200640 1 -- 172.21.15.6:0/2860907792 shutdown_connections 2024-04-24T23:25:40.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:40.333+0000 7f65bf200640 1 --2- 172.21.15.6:0/2860907792 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f65b004b8e0 0x7f65b0005ee0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:40.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:40.333+0000 7f65bf200640 1 --2- 172.21.15.6:0/2860907792 >> v2:172.21.15.6:3300/0 conn(0x7f65b004b500 0x7f65b00dac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:40.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:40.333+0000 7f65bf200640 1 -- 172.21.15.6:0/2860907792 >> 172.21.15.6:0/2860907792 conn(0x7f65b0050180 msgr2=0x7f65b0050590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:25:40.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:40.337+0000 7f65bf200640 1 -- 172.21.15.6:0/2860907792 shutdown_connections 2024-04-24T23:25:40.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:40.339+0000 7f65bf200640 1 -- 172.21.15.6:0/2860907792 wait complete. 2024-04-24T23:25:40.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:40.343+0000 7f65bf200640 1 Processor -- start 2024-04-24T23:25:40.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:40.343+0000 7f65bf200640 1 -- start start 2024-04-24T23:25:40.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:40.345+0000 7f65bf200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f65b0006420 0x7f65b00cce40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:25:40.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:40.345+0000 7f65bf200640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f65b004b8e0 0x7f65b00d0400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:25:40.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:40.345+0000 7f65bf200640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f65b00caf80 0x7f65b00cb430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:25:40.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:40.345+0000 7f65bf200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f65b004bd00 con 0x7f65b0006420 2024-04-24T23:25:40.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:40.345+0000 7f65bf200640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f65b0007480 con 0x7f65b004b8e0 2024-04-24T23:25:40.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:40.345+0000 7f65bf200640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f65b00d5000 con 0x7f65b00caf80 2024-04-24T23:25:40.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:40.346+0000 7f65b7e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f65b0006420 0x7f65b00cce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:25:40.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:40.346+0000 7f65bd200640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f65b00caf80 0x7f65b00cb430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:25:40.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:40.346+0000 7f65bd200640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f65b00caf80 0x7f65b00cb430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.6:57248/0 (socket says 172.21.15.6:57248) 2024-04-24T23:25:40.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:40.346+0000 7f65bd200640 1 -- 172.21.15.6:0/930355341 learned_addr learned my addr 172.21.15.6:0/930355341 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:25:40.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:40.347+0000 7f65b7400640 1 --2- 172.21.15.6:0/930355341 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f65b004b8e0 0x7f65b00d0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:25:40.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:40.347+0000 7f65bd200640 1 -- 172.21.15.6:0/930355341 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f65b004b8e0 msgr2=0x7f65b00d0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:25:40.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:40.347+0000 7f65bd200640 1 --2- 172.21.15.6:0/930355341 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f65b004b8e0 0x7f65b00d0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:40.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:40.347+0000 7f65bd200640 1 -- 172.21.15.6:0/930355341 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f65b0006420 msgr2=0x7f65b00cce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:25:40.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:40.347+0000 7f65bd200640 1 --2- 172.21.15.6:0/930355341 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f65b0006420 0x7f65b00cce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:40.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:40.347+0000 7f65bd200640 1 -- 172.21.15.6:0/930355341 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65b0160d40 con 0x7f65b00caf80 2024-04-24T23:25:40.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:40.348+0000 7f65b7e00640 1 --2- 172.21.15.6:0/930355341 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f65b0006420 0x7f65b00cce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:25:40.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:40.348+0000 7f65bd200640 1 --2- 172.21.15.6:0/930355341 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f65b00caf80 0x7f65b00cb430 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f65b8064c60 tx=0x7f65b8076b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:25:40.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:40.348+0000 7f65abe00640 1 -- 172.21.15.6:0/930355341 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65b8068b40 con 0x7f65b00caf80 2024-04-24T23:25:40.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:40.349+0000 7f65abe00640 1 -- 172.21.15.6:0/930355341 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f65b8090030 con 0x7f65b00caf80 2024-04-24T23:25:40.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:40.349+0000 7f65abe00640 1 -- 172.21.15.6:0/930355341 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65b8068f00 con 0x7f65b00caf80 2024-04-24T23:25:40.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:40.350+0000 7f65bf200640 1 -- 172.21.15.6:0/930355341 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65b00cd380 con 0x7f65b00caf80 2024-04-24T23:25:40.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:40.350+0000 7f65bf200640 1 -- 172.21.15.6:0/930355341 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f65b0160fc0 con 0x7f65b00caf80 2024-04-24T23:25:40.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:40.354+0000 7f65abe00640 1 -- 172.21.15.6:0/930355341 <== mon.2 v2:172.21.15.190:3301/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f65b8095420 con 0x7f65b00caf80 2024-04-24T23:25:40.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:40.355+0000 7f65abe00640 1 --2- 172.21.15.6:0/930355341 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f659806be60 0x7f659806e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:25:40.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:40.356+0000 7f65abe00640 1 -- 172.21.15.6:0/930355341 <== mon.2 v2:172.21.15.190:3301/0 5 ==== osd_map(29..29 src has 1..29) ==== 6848+0+0 (secure 0 0 0) 0x7f65b80d3910 con 0x7f65b00caf80 2024-04-24T23:25:40.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:40.356+0000 7f65b7e00640 1 --2- 172.21.15.6:0/930355341 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f659806be60 0x7f659806e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:25:40.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:40.357+0000 7f65bf200640 1 -- 172.21.15.6:0/930355341 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f65b00052c0 con 0x7f65b00caf80 2024-04-24T23:25:40.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:40.357+0000 7f65b7e00640 1 --2- 172.21.15.6:0/930355341 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f659806be60 0x7f659806e320 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f65ac006fd0 tx=0x7f65ac008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:25:40.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:40.365+0000 7f65abe00640 1 -- 172.21.15.6:0/930355341 <== mon.2 v2:172.21.15.190:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f65b0160d40 con 0x7f65b00caf80 2024-04-24T23:25:40.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:40.943+0000 7f65bf200640 1 -- 172.21.15.6:0/930355341 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f65b0045120 con 0x7f65b00caf80 2024-04-24T23:25:40.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:40.946+0000 7f65abe00640 1 -- 172.21.15.6:0/930355341 <== mon.2 v2:172.21.15.190:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) ==== 94+0+6856 (secure 0 0 0) 0x7f65b80d5870 con 0x7f65b00caf80 2024-04-24T23:25:40.950 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:25:40.950 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":48,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5081,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/1190759626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1190759626},{"type":"v1","addr":"172.21.15.6:6834","nonce":1190759626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":5249,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6838/2416336040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2416336040},{"type":"v1","addr":"172.21.15.190:6838","nonce":2416336040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":42},{"gid":5503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/3928731059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":3928731059},{"type":"v1","addr":"172.21.15.6:6837","nonce":3928731059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":35},{"gid":5725,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6837/1151527362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":1151527362},{"type":"v1","addr":"172.21.15.190:6837","nonce":1151527362}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":48}],"filesystems":[{"mdsmap":{"epoch":47,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:25:05.197173+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":4306,"mds_1":5599},"failed":[],"damaged":[],"stopped":[2,3,4],"info":{"gid_4306":{"gid":4306,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/489092393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":489092393},{"type":"v1","addr":"172.21.15.6:6835","nonce":489092393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5599":{"gid":5599,"name":"d","rank":1,"incarnation":43,"state":"up:active","state_seq":25,"addr":"172.21.15.190:6839/2643233757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":2643233757},{"type":"v1","addr":"172.21.15.190:6839","nonce":2643233757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4306,"qdb_cluster":[5599,4306]},"id":1}]} 2024-04-24T23:25:40.950 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 48 2024-04-24T23:25:40.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:40.954+0000 7f65bf200640 1 -- 172.21.15.6:0/930355341 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f659806be60 msgr2=0x7f659806e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:25:40.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:40.954+0000 7f65bf200640 1 --2- 172.21.15.6:0/930355341 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f659806be60 0x7f659806e320 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f65ac006fd0 tx=0x7f65ac008040 comp rx=0 tx=0).stop 2024-04-24T23:25:40.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:40.954+0000 7f65bf200640 1 -- 172.21.15.6:0/930355341 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f65b00caf80 msgr2=0x7f65b00cb430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:25:40.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:40.954+0000 7f65bf200640 1 --2- 172.21.15.6:0/930355341 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f65b00caf80 0x7f65b00cb430 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f65b8064c60 tx=0x7f65b8076b80 comp rx=0 tx=0).stop 2024-04-24T23:25:40.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:40.955+0000 7f65bf200640 1 -- 172.21.15.6:0/930355341 shutdown_connections 2024-04-24T23:25:40.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:40.955+0000 7f65bf200640 1 --2- 172.21.15.6:0/930355341 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f659806be60 0x7f659806e320 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:40.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:40.955+0000 7f65bf200640 1 --2- 172.21.15.6:0/930355341 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f65b00caf80 0x7f65b00cb430 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:40.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:40.955+0000 7f65bf200640 1 --2- 172.21.15.6:0/930355341 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f65b004b8e0 0x7f65b00d0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:40.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:40.955+0000 7f65bf200640 1 --2- 172.21.15.6:0/930355341 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f65b0006420 0x7f65b00cce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:40.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:40.955+0000 7f65bf200640 1 -- 172.21.15.6:0/930355341 >> 172.21.15.6:0/930355341 conn(0x7f65b0050180 msgr2=0x7f65b004cb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:25:40.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:40.956+0000 7f65bf200640 1 -- 172.21.15.6:0/930355341 shutdown_connections 2024-04-24T23:25:40.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:40.956+0000 7f65bf200640 1 -- 172.21.15.6:0/930355341 wait complete. 2024-04-24T23:25:40.992 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-24T23:25:42.993 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:25:43.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:43.355+0000 7f9c6c800640 1 Processor -- start 2024-04-24T23:25:43.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:43.356+0000 7f9c6c800640 1 -- start start 2024-04-24T23:25:43.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:43.358+0000 7f9c6c800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f9c641593b0 0x7f9c640c6e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:25:43.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:43.358+0000 7f9c6c800640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9c6406eba0 0x7f9c6406efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:25:43.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:43.358+0000 7f9c6c800640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9c64071880 con 0x7f9c640c8fd0 2024-04-24T23:25:43.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:43.358+0000 7f9c6c800640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f9c6406e710 con 0x7f9c641593b0 2024-04-24T23:25:43.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:43.358+0000 7f9c6c800640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f9c6415f200 con 0x7f9c6406eba0 2024-04-24T23:25:43.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:43.360+0000 7f9c69400640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9c6406eba0 0x7f9c6406efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:25:43.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:43.361+0000 7f9c69e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f9c641593b0 0x7f9c640c6e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:25:43.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:43.364+0000 7f9c6a800640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f9c640c8fd0 0x7f9c64162460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:51328/0 (socket says 172.21.15.6:51328) 2024-04-24T23:25:43.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:43.364+0000 7f9c6a800640 1 -- 172.21.15.6:0/3603571341 learned_addr learned my addr 172.21.15.6:0/3603571341 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:25:43.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:43.367+0000 7f9c68a00640 1 -- 172.21.15.6:0/3603571341 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1442808464 0 0) 0x7f9c64071880 con 0x7f9c640c8fd0 2024-04-24T23:25:43.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:43.369+0000 7f9c68a00640 1 -- 172.21.15.6:0/3603571341 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9c4c003520 con 0x7f9c640c8fd0 2024-04-24T23:25:43.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:43.370+0000 7f9c68a00640 1 -- 172.21.15.6:0/3603571341 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2686032668 0 0) 0x7f9c4c003520 con 0x7f9c640c8fd0 2024-04-24T23:25:43.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:43.370+0000 7f9c68a00640 1 -- 172.21.15.6:0/3603571341 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9c6406eba0 msgr2=0x7f9c6406efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:25:43.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:43.370+0000 7f9c68a00640 1 --2- 172.21.15.6:0/3603571341 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9c6406eba0 0x7f9c6406efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:43.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:43.370+0000 7f9c68a00640 1 -- 172.21.15.6:0/3603571341 >> v2:172.21.15.6:3300/0 conn(0x7f9c641593b0 msgr2=0x7f9c640c6e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:25:43.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:43.370+0000 7f9c68a00640 1 --2- 172.21.15.6:0/3603571341 >> v2:172.21.15.6:3300/0 conn(0x7f9c641593b0 0x7f9c640c6e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:43.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:43.370+0000 7f9c68a00640 1 -- 172.21.15.6:0/3603571341 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c6415f4c0 con 0x7f9c640c8fd0 2024-04-24T23:25:43.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:43.370+0000 7f9c68a00640 1 -- 172.21.15.6:0/3603571341 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3463268701 0 0) 0x7f9c58001670 con 0x7f9c640c8fd0 2024-04-24T23:25:43.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:43.370+0000 7f9c68a00640 1 -- 172.21.15.6:0/3603571341 >> v1:172.21.15.6:6789/0 conn(0x7f9c640c8fd0 legacy=0x7f9c64162460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:25:43.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:43.370+0000 7f9c68a00640 1 --2- 172.21.15.6:0/3603571341 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9c4c0039b0 0x7f9c4c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:25:43.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:43.371+0000 7f9c68a00640 1 --2- 172.21.15.6:0/3603571341 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9c4c004800 0x7f9c4c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:25:43.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:43.371+0000 7f9c68a00640 1 --2- 172.21.15.6:0/3603571341 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9c4c0051c0 0x7f9c4c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:25:43.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:43.371+0000 7f9c68a00640 1 -- 172.21.15.6:0/3603571341 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f9c6406e710 con 0x7f9c4c004800 2024-04-24T23:25:43.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:43.371+0000 7f9c68a00640 1 -- 172.21.15.6:0/3603571341 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f9c6415f200 con 0x7f9c4c0039b0 2024-04-24T23:25:43.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:43.371+0000 7f9c68a00640 1 -- 172.21.15.6:0/3603571341 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f9c4c00bf00 con 0x7f9c4c0051c0 2024-04-24T23:25:43.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:43.372+0000 7f9c69400640 1 --2- 172.21.15.6:0/3603571341 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9c6406eba0 0x7f9c6406efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:25:43.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:43.373+0000 7f9c69400640 1 --2- 172.21.15.6:0/3603571341 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9c4c004800 0x7f9c4c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:25:43.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:43.374+0000 7f9c69e00640 1 --2- 172.21.15.6:0/3603571341 >> v2:172.21.15.6:3300/0 conn(0x7f9c641593b0 0x7f9c640c6e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:25:43.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:43.374+0000 7f9c6a800640 1 --2- 172.21.15.6:0/3603571341 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9c4c0051c0 0x7f9c4c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:25:43.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:43.376+0000 7f9c69400640 1 -- 172.21.15.6:0/3603571341 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9c4c0051c0 msgr2=0x7f9c4c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:25:43.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:43.376+0000 7f9c69400640 1 --2- 172.21.15.6:0/3603571341 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9c4c0051c0 0x7f9c4c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:43.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:43.376+0000 7f9c69400640 1 -- 172.21.15.6:0/3603571341 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9c4c0039b0 msgr2=0x7f9c4c003dd0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-24T23:25:43.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:43.376+0000 7f9c69400640 1 --2- 172.21.15.6:0/3603571341 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9c4c0039b0 0x7f9c4c003dd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:43.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:43.376+0000 7f9c69400640 1 -- 172.21.15.6:0/3603571341 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c4c003520 con 0x7f9c4c004800 2024-04-24T23:25:43.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:43.378+0000 7f9c6a800640 1 --2- 172.21.15.6:0/3603571341 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9c4c0051c0 0x7f9c4c00b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:25:43.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:43.379+0000 7f9c69400640 1 --2- 172.21.15.6:0/3603571341 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9c4c004800 0x7f9c4c004c80 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f9c54009b00 tx=0x7f9c4c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78eba18d07c99e09 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:25:43.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:43.380+0000 7f9c68a00640 1 -- 172.21.15.6:0/3603571341 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c54036070 con 0x7f9c4c004800 2024-04-24T23:25:43.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:43.380+0000 7f9c68a00640 1 -- 172.21.15.6:0/3603571341 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9c54004030 con 0x7f9c4c004800 2024-04-24T23:25:43.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:43.381+0000 7f9c6c800640 1 -- 172.21.15.6:0/3603571341 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9c4c004800 msgr2=0x7f9c4c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:25:43.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:43.381+0000 7f9c68a00640 1 -- 172.21.15.6:0/3603571341 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c54036070 con 0x7f9c4c004800 2024-04-24T23:25:43.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:43.381+0000 7f9c6c800640 1 --2- 172.21.15.6:0/3603571341 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9c4c004800 0x7f9c4c004c80 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f9c54009b00 tx=0x7f9c4c005830 comp rx=0 tx=0).stop 2024-04-24T23:25:43.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:43.382+0000 7f9c6a800640 1 -- 172.21.15.6:0/3603571341 reap_dead start 2024-04-24T23:25:43.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:43.382+0000 7f9c6c800640 1 -- 172.21.15.6:0/3603571341 shutdown_connections 2024-04-24T23:25:43.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:43.382+0000 7f9c6c800640 1 -- 172.21.15.6:0/3603571341 >> 172.21.15.6:0/3603571341 conn(0x7f9c6406b0e0 msgr2=0x7f9c6406b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:25:43.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:43.384+0000 7f9c6c800640 1 -- 172.21.15.6:0/3603571341 shutdown_connections 2024-04-24T23:25:43.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:43.386+0000 7f9c6c800640 1 -- 172.21.15.6:0/3603571341 wait complete. 2024-04-24T23:25:43.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:43.389+0000 7f9c6c800640 1 Processor -- start 2024-04-24T23:25:43.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:43.389+0000 7f9c6c800640 1 -- start start 2024-04-24T23:25:43.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:43.390+0000 7f9c6c800640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9c640cb4e0 0x7f9c6415b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:25:43.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:43.391+0000 7f9c6c800640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9c6415c500 0x7f9c6415b970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:25:43.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:43.391+0000 7f9c6c800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9c640cb8c0 0x7f9c6415cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:25:43.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:43.391+0000 7f9c6c800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f9c6415fd60 con 0x7f9c640cb8c0 2024-04-24T23:25:43.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:43.391+0000 7f9c6c800640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f9c6415fa80 con 0x7f9c6415c500 2024-04-24T23:25:43.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:43.391+0000 7f9c6c800640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f9c6406e3a0 con 0x7f9c640cb4e0 2024-04-24T23:25:43.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:43.394+0000 7f9c69400640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9c6415c500 0x7f9c6415b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:25:43.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:43.395+0000 7f9c69e00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9c640cb4e0 0x7f9c6415b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:25:43.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:43.395+0000 7f9c69e00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9c640cb4e0 0x7f9c6415b430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.6:57264/0 (socket says 172.21.15.6:57264) 2024-04-24T23:25:43.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:43.395+0000 7f9c69e00640 1 -- 172.21.15.6:0/218676293 learned_addr learned my addr 172.21.15.6:0/218676293 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:25:43.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:43.395+0000 7f9c6a800640 1 --2- 172.21.15.6:0/218676293 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9c640cb8c0 0x7f9c6415cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:25:43.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:43.396+0000 7f9c69e00640 1 -- 172.21.15.6:0/218676293 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9c6415c500 msgr2=0x7f9c6415b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:25:43.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:43.396+0000 7f9c69e00640 1 --2- 172.21.15.6:0/218676293 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9c6415c500 0x7f9c6415b970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:43.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:43.397+0000 7f9c69e00640 1 -- 172.21.15.6:0/218676293 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9c640cb8c0 msgr2=0x7f9c6415cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:25:43.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:43.397+0000 7f9c69e00640 1 --2- 172.21.15.6:0/218676293 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9c640cb8c0 0x7f9c6415cd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:43.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:43.397+0000 7f9c69e00640 1 -- 172.21.15.6:0/218676293 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c6415eb40 con 0x7f9c640cb4e0 2024-04-24T23:25:43.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:43.397+0000 7f9c6a800640 1 --2- 172.21.15.6:0/218676293 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9c640cb8c0 0x7f9c6415cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:25:43.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:43.400+0000 7f9c69e00640 1 --2- 172.21.15.6:0/218676293 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9c640cb4e0 0x7f9c6415b430 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f9c60009f00 tx=0x7f9c6000c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:25:43.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:43.401+0000 7f9c5e000640 1 -- 172.21.15.6:0/218676293 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c60018030 con 0x7f9c640cb4e0 2024-04-24T23:25:43.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:43.401+0000 7f9c5e000640 1 -- 172.21.15.6:0/218676293 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9c6000ce50 con 0x7f9c640cb4e0 2024-04-24T23:25:43.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:43.401+0000 7f9c5e000640 1 -- 172.21.15.6:0/218676293 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c60018030 con 0x7f9c640cb4e0 2024-04-24T23:25:43.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:43.402+0000 7f9c6c800640 1 -- 172.21.15.6:0/218676293 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c6413f630 con 0x7f9c640cb4e0 2024-04-24T23:25:43.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:43.403+0000 7f9c6c800640 1 -- 172.21.15.6:0/218676293 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c6415a820 con 0x7f9c640cb4e0 2024-04-24T23:25:43.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:43.405+0000 7f9c6c800640 1 -- 172.21.15.6:0/218676293 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9c6415a610 con 0x7f9c640cb4e0 2024-04-24T23:25:43.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:43.413+0000 7f9c5e000640 1 -- 172.21.15.6:0/218676293 <== mon.2 v2:172.21.15.190:3301/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f9c60005d10 con 0x7f9c640cb4e0 2024-04-24T23:25:43.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:43.415+0000 7f9c5e000640 1 --2- 172.21.15.6:0/218676293 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f9c4806c060 0x7f9c4806e520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:25:43.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:43.416+0000 7f9c5e000640 1 -- 172.21.15.6:0/218676293 <== mon.2 v2:172.21.15.190:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6934+0+0 (secure 0 0 0) 0x7f9c6008bcd0 con 0x7f9c640cb4e0 2024-04-24T23:25:43.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:43.420+0000 7f9c69400640 1 --2- 172.21.15.6:0/218676293 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f9c4806c060 0x7f9c4806e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:25:43.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:43.422+0000 7f9c69400640 1 --2- 172.21.15.6:0/218676293 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f9c4806c060 0x7f9c4806e520 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f9c54033000 tx=0x7f9c54008070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:25:43.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:43.422+0000 7f9c5e000640 1 -- 172.21.15.6:0/218676293 <== mon.2 v2:172.21.15.190:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f9c60022070 con 0x7f9c640cb4e0 2024-04-24T23:25:44.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:44.005+0000 7f9c6c800640 1 -- 172.21.15.6:0/218676293 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9c640767b0 con 0x7f9c640cb4e0 2024-04-24T23:25:44.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:44.010+0000 7f9c5e000640 1 -- 172.21.15.6:0/218676293 <== mon.2 v2:172.21.15.190:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6033 (secure 0 0 0) 0x7f9c60089e20 con 0x7f9c640cb4e0 2024-04-24T23:25:44.013 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:25:44.013 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5081,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/1190759626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1190759626},{"type":"v1","addr":"172.21.15.6:6834","nonce":1190759626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":5249,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6838/2416336040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2416336040},{"type":"v1","addr":"172.21.15.190:6838","nonce":2416336040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":42},{"gid":5503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/3928731059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":3928731059},{"type":"v1","addr":"172.21.15.6:6837","nonce":3928731059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:25:42.695130+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5725,"mds_1":5599},"failed":[],"damaged":[],"stopped":[2,3,4],"info":{"gid_5599":{"gid":5599,"name":"d","rank":1,"incarnation":43,"state":"up:active","state_seq":25,"addr":"172.21.15.190:6839/2643233757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":2643233757},{"type":"v1","addr":"172.21.15.190:6839","nonce":2643233757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5725":{"gid":5725,"name":"f","rank":0,"incarnation":49,"state":"up:replay","state_seq":1,"addr":"172.21.15.190:6837/1151527362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":1151527362},{"type":"v1","addr":"172.21.15.190:6837","nonce":1151527362}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5599,"qdb_cluster":[5599]},"id":1}]} 2024-04-24T23:25:44.013 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 49 2024-04-24T23:25:44.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:44.015+0000 7f9c6c800640 1 -- 172.21.15.6:0/218676293 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f9c4806c060 msgr2=0x7f9c4806e520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:25:44.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:44.015+0000 7f9c6c800640 1 --2- 172.21.15.6:0/218676293 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f9c4806c060 0x7f9c4806e520 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f9c54033000 tx=0x7f9c54008070 comp rx=0 tx=0).stop 2024-04-24T23:25:44.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:44.015+0000 7f9c6c800640 1 -- 172.21.15.6:0/218676293 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9c640cb4e0 msgr2=0x7f9c6415b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:25:44.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:44.015+0000 7f9c6c800640 1 --2- 172.21.15.6:0/218676293 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9c640cb4e0 0x7f9c6415b430 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f9c60009f00 tx=0x7f9c6000c490 comp rx=0 tx=0).stop 2024-04-24T23:25:44.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:44.016+0000 7f9c6c800640 1 -- 172.21.15.6:0/218676293 shutdown_connections 2024-04-24T23:25:44.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:44.016+0000 7f9c6c800640 1 --2- 172.21.15.6:0/218676293 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f9c4806c060 0x7f9c4806e520 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:44.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:44.016+0000 7f9c6c800640 1 --2- 172.21.15.6:0/218676293 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9c640cb8c0 0x7f9c6415cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:44.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:44.016+0000 7f9c6c800640 1 --2- 172.21.15.6:0/218676293 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9c6415c500 0x7f9c6415b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:44.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:44.016+0000 7f9c6c800640 1 --2- 172.21.15.6:0/218676293 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9c640cb4e0 0x7f9c6415b430 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:44.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:44.016+0000 7f9c6c800640 1 -- 172.21.15.6:0/218676293 >> 172.21.15.6:0/218676293 conn(0x7f9c6406b0e0 msgr2=0x7f9c641582b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:25:44.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:44.016+0000 7f9c6c800640 1 -- 172.21.15.6:0/218676293 shutdown_connections 2024-04-24T23:25:44.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:44.016+0000 7f9c6c800640 1 -- 172.21.15.6:0/218676293 wait complete. 2024-04-24T23:25:44.059 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c down, removed from mdsmap 2024-04-24T23:25:44.059 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2024-04-24T23:25:44.059 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:25:44.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:44.420+0000 7f197aa00640 1 Processor -- start 2024-04-24T23:25:44.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:44.421+0000 7f197aa00640 1 -- start start 2024-04-24T23:25:44.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:44.425+0000 7f1979600640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f19740ccfb0 0x7f197415b480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.6:43190/0 (socket says 172.21.15.6:43190) 2024-04-24T23:25:44.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:44.425+0000 7f197aa00640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f19740cd390 0x7f19740cacf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:25:44.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:44.426+0000 7f1978c00640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f19740cd390 0x7f19740cacf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:25:44.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:44.426+0000 7f1978c00640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f19740cd390 0x7f19740cacf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:53440/0 (socket says 172.21.15.6:53440) 2024-04-24T23:25:44.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:44.426+0000 7f1978c00640 1 -- 172.21.15.6:0/2661744855 learned_addr learned my addr 172.21.15.6:0/2661744855 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:25:44.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:44.430+0000 7f197aa00640 1 --2- 172.21.15.6:0/2661744855 >> v2:172.21.15.6:3300/0 conn(0x7f19741548a0 0x7f1974154cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:25:44.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:44.430+0000 7f197a000640 1 --2- 172.21.15.6:0/2661744855 >> v2:172.21.15.6:3300/0 conn(0x7f19741548a0 0x7f1974154cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:25:44.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:44.430+0000 7f197aa00640 1 -- 172.21.15.6:0/2661744855 --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f197406be70 con 0x7f19740ccfb0 2024-04-24T23:25:44.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:44.430+0000 7f197aa00640 1 -- 172.21.15.6:0/2661744855 --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f197406ccd0 con 0x7f19741548a0 2024-04-24T23:25:44.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:44.430+0000 7f197aa00640 1 -- 172.21.15.6:0/2661744855 --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f19740cb230 con 0x7f19740cd390 2024-04-24T23:25:44.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:44.431+0000 7f1973e00640 1 -- 172.21.15.6:0/2661744855 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3043943953 0 0) 0x7f197406be70 con 0x7f19740ccfb0 2024-04-24T23:25:44.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:44.431+0000 7f1973e00640 1 -- 172.21.15.6:0/2661744855 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f195c003520 con 0x7f19740ccfb0 2024-04-24T23:25:44.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:44.431+0000 7f197a000640 1 -- 172.21.15.6:0/2661744855 >> v2:172.21.15.190:3300/0 conn(0x7f19740cd390 msgr2=0x7f19740cacf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:25:44.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:44.431+0000 7f197a000640 1 --2- 172.21.15.6:0/2661744855 >> v2:172.21.15.190:3300/0 conn(0x7f19740cd390 0x7f19740cacf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:44.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:44.431+0000 7f197a000640 1 -- 172.21.15.6:0/2661744855 >> v1:172.21.15.190:6789/0 conn(0x7f19740ccfb0 legacy=0x7f197415b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:25:44.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:44.431+0000 7f197a000640 1 -- 172.21.15.6:0/2661744855 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f197406c150 con 0x7f19741548a0 2024-04-24T23:25:44.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:44.433+0000 7f197a000640 1 --2- 172.21.15.6:0/2661744855 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f19741548a0 0x7f1974154cb0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f19740cd8f0 tx=0x7f196c02a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aaa2f5a5b6d16e8f server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:25:44.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:44.436+0000 7f1973e00640 1 -- 172.21.15.6:0/2661744855 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f196c007740 con 0x7f19741548a0 2024-04-24T23:25:44.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:44.436+0000 7f1973e00640 1 -- 172.21.15.6:0/2661744855 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f196c0078e0 con 0x7f19741548a0 2024-04-24T23:25:44.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:44.436+0000 7f1973e00640 1 -- 172.21.15.6:0/2661744855 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f196c007cb0 con 0x7f19741548a0 2024-04-24T23:25:44.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:44.437+0000 7f197aa00640 1 -- 172.21.15.6:0/2661744855 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f19741548a0 msgr2=0x7f1974154cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:25:44.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:44.437+0000 7f197aa00640 1 --2- 172.21.15.6:0/2661744855 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f19741548a0 0x7f1974154cb0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f19740cd8f0 tx=0x7f196c02a710 comp rx=0 tx=0).stop 2024-04-24T23:25:44.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:44.439+0000 7f197aa00640 1 -- 172.21.15.6:0/2661744855 shutdown_connections 2024-04-24T23:25:44.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:44.439+0000 7f197aa00640 1 --2- 172.21.15.6:0/2661744855 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f19741548a0 0x7f1974154cb0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:44.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:44.439+0000 7f197aa00640 1 --2- 172.21.15.6:0/2661744855 >> v2:172.21.15.190:3300/0 conn(0x7f19740cd390 0x7f19740cacf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:44.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:44.439+0000 7f197aa00640 1 -- 172.21.15.6:0/2661744855 >> 172.21.15.6:0/2661744855 conn(0x7f197406d9f0 msgr2=0x7f1974066ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:25:44.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:44.440+0000 7f197aa00640 1 -- 172.21.15.6:0/2661744855 shutdown_connections 2024-04-24T23:25:44.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:44.442+0000 7f197aa00640 1 -- 172.21.15.6:0/2661744855 wait complete. 2024-04-24T23:25:44.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:44.448+0000 7f197aa00640 1 Processor -- start 2024-04-24T23:25:44.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:44.449+0000 7f197aa00640 1 -- start start 2024-04-24T23:25:44.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:44.454+0000 7f197aa00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f19740ccfb0 0x7f19740b4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:25:44.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:44.455+0000 7f197aa00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f19740cd390 0x7f1974147f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:25:44.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:44.455+0000 7f197aa00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f19741548a0 0x7f1974148480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:25:44.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:44.455+0000 7f197aa00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f197406c9f0 con 0x7f19740ccfb0 2024-04-24T23:25:44.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:44.455+0000 7f197aa00640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f197406c710 con 0x7f19741548a0 2024-04-24T23:25:44.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:44.455+0000 7f197aa00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f1974067bb0 con 0x7f19740cd390 2024-04-24T23:25:44.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:44.456+0000 7f1978c00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f19740cd390 0x7f1974147f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:25:44.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:44.460+0000 7f197a000640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f19741548a0 0x7f1974148480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:25:44.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:44.461+0000 7f1978c00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f19740cd390 0x7f1974147f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.6:57272/0 (socket says 172.21.15.6:57272) 2024-04-24T23:25:44.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:44.461+0000 7f1978c00640 1 -- 172.21.15.6:0/969778829 learned_addr learned my addr 172.21.15.6:0/969778829 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:25:44.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:44.461+0000 7f1979600640 1 --2- 172.21.15.6:0/969778829 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f19740ccfb0 0x7f19740b4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:25:44.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:44.462+0000 7f1979600640 1 -- 172.21.15.6:0/969778829 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f19740cd390 msgr2=0x7f1974147f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:25:44.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:44.463+0000 7f1979600640 1 --2- 172.21.15.6:0/969778829 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f19740cd390 0x7f1974147f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:44.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:44.463+0000 7f1979600640 1 -- 172.21.15.6:0/969778829 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f19741548a0 msgr2=0x7f1974148480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:25:44.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:44.463+0000 7f1979600640 1 --2- 172.21.15.6:0/969778829 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f19741548a0 0x7f1974148480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:44.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:44.463+0000 7f1979600640 1 -- 172.21.15.6:0/969778829 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19740c76b0 con 0x7f19740ccfb0 2024-04-24T23:25:44.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:44.463+0000 7f1978c00640 1 --2- 172.21.15.6:0/969778829 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f19740cd390 0x7f1974147f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:25:44.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:44.463+0000 7f197a000640 1 --2- 172.21.15.6:0/969778829 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f19741548a0 0x7f1974148480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:25:44.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:44.464+0000 7f1979600640 1 --2- 172.21.15.6:0/969778829 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f19740ccfb0 0x7f19740b4bf0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f196800c410 tx=0x7f196800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:25:44.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:44.465+0000 7f1971600640 1 -- 172.21.15.6:0/969778829 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19680074e0 con 0x7f19740ccfb0 2024-04-24T23:25:44.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:44.465+0000 7f1971600640 1 -- 172.21.15.6:0/969778829 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f19680047e0 con 0x7f19740ccfb0 2024-04-24T23:25:44.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:44.467+0000 7f1971600640 1 -- 172.21.15.6:0/969778829 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19680074e0 con 0x7f19740ccfb0 2024-04-24T23:25:44.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:44.467+0000 7f197aa00640 1 -- 172.21.15.6:0/969778829 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19740c7970 con 0x7f19740ccfb0 2024-04-24T23:25:44.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:44.467+0000 7f197aa00640 1 -- 172.21.15.6:0/969778829 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1974157c90 con 0x7f19740ccfb0 2024-04-24T23:25:44.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:44.478+0000 7f197aa00640 1 -- 172.21.15.6:0/969778829 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f19740b2c70 con 0x7f19740ccfb0 2024-04-24T23:25:44.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:44.479+0000 7f1971600640 1 -- 172.21.15.6:0/969778829 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f1968011070 con 0x7f19740ccfb0 2024-04-24T23:25:44.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:44.484+0000 7f1971600640 1 --2- 172.21.15.6:0/969778829 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f195406c060 0x7f195406e520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:25:44.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:44.484+0000 7f1971600640 1 -- 172.21.15.6:0/969778829 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6934+0+0 (secure 0 0 0) 0x7f196808a9a0 con 0x7f19740ccfb0 2024-04-24T23:25:44.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:44.486+0000 7f1978c00640 1 --2- 172.21.15.6:0/969778829 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f195406c060 0x7f195406e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:25:44.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:44.487+0000 7f1978c00640 1 --2- 172.21.15.6:0/969778829 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f195406c060 0x7f195406e520 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f19740b58f0 tx=0x7f1960009240 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:25:44.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:44.500+0000 7f1971600640 1 -- 172.21.15.6:0/969778829 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f1968058830 con 0x7f19740ccfb0 2024-04-24T23:25:45.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:45.005+0000 7f197aa00640 1 -- 172.21.15.6:0/969778829 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1974067990 con 0x7f19740ccfb0 2024-04-24T23:25:45.009 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:25:45.009 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5081,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/1190759626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1190759626},{"type":"v1","addr":"172.21.15.6:6834","nonce":1190759626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":5249,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6838/2416336040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2416336040},{"type":"v1","addr":"172.21.15.190:6838","nonce":2416336040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":42},{"gid":5503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/3928731059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":3928731059},{"type":"v1","addr":"172.21.15.6:6837","nonce":3928731059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:25:42.695130+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5725,"mds_1":5599},"failed":[],"damaged":[],"stopped":[2,3,4],"info":{"gid_5599":{"gid":5599,"name":"d","rank":1,"incarnation":43,"state":"up:active","state_seq":25,"addr":"172.21.15.190:6839/2643233757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":2643233757},{"type":"v1","addr":"172.21.15.190:6839","nonce":2643233757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5725":{"gid":5725,"name":"f","rank":0,"incarnation":49,"state":"up:replay","state_seq":1,"addr":"172.21.15.190:6837/1151527362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":1151527362},{"type":"v1","addr":"172.21.15.190:6837","nonce":1151527362}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5599,"qdb_cluster":[5599]},"id":1}]} 2024-04-24T23:25:45.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:45.007+0000 7f1971600640 1 -- 172.21.15.6:0/969778829 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6033 (secure 0 0 0) 0x7f1968088e20 con 0x7f19740ccfb0 2024-04-24T23:25:45.009 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 49 2024-04-24T23:25:45.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:45.010+0000 7f197aa00640 1 -- 172.21.15.6:0/969778829 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f195406c060 msgr2=0x7f195406e520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:25:45.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:45.011+0000 7f197aa00640 1 --2- 172.21.15.6:0/969778829 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f195406c060 0x7f195406e520 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f19740b58f0 tx=0x7f1960009240 comp rx=0 tx=0).stop 2024-04-24T23:25:45.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:45.011+0000 7f197aa00640 1 -- 172.21.15.6:0/969778829 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f19740ccfb0 msgr2=0x7f19740b4bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:25:45.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:45.011+0000 7f197aa00640 1 --2- 172.21.15.6:0/969778829 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f19740ccfb0 0x7f19740b4bf0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f196800c410 tx=0x7f196800c8d0 comp rx=0 tx=0).stop 2024-04-24T23:25:45.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:45.012+0000 7f197aa00640 1 -- 172.21.15.6:0/969778829 shutdown_connections 2024-04-24T23:25:45.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:45.012+0000 7f197aa00640 1 --2- 172.21.15.6:0/969778829 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f195406c060 0x7f195406e520 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:45.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:45.012+0000 7f197aa00640 1 --2- 172.21.15.6:0/969778829 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f19740cd390 0x7f1974147f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:45.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:45.012+0000 7f197aa00640 1 --2- 172.21.15.6:0/969778829 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f19741548a0 0x7f1974148480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:45.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:45.012+0000 7f197aa00640 1 --2- 172.21.15.6:0/969778829 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f19740ccfb0 0x7f19740b4bf0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:45.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:45.012+0000 7f197aa00640 1 -- 172.21.15.6:0/969778829 >> 172.21.15.6:0/969778829 conn(0x7f197406d9f0 msgr2=0x7f1974074630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:25:45.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:45.012+0000 7f197aa00640 1 -- 172.21.15.6:0/969778829 shutdown_connections 2024-04-24T23:25:45.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:45.012+0000 7f197aa00640 1 -- 172.21.15.6:0/969778829 wait complete. 2024-04-24T23:25:45.043 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 49, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-24T23:15:36.400716+0000", "damaged": [], "data_pools": [ 3, 4 ], "enabled": true, "epoch": 49, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_5599": { "addr": "172.21.15.190:6839/2643233757", "addrs": { "addrvec": [ { "addr": "172.21.15.190:6836", "nonce": 2643233757, "type": "v2" }, { "addr": "172.21.15.190:6839", "nonce": 2643233757, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5599, "incarnation": 43, "join_fscid": -1, "name": "d", "rank": 1, "state": "up:active", "state_seq": 25 }, "gid_5725": { "addr": "172.21.15.190:6837/1151527362", "addrs": { "addrvec": [ { "addr": "172.21.15.190:6834", "nonce": 1151527362, "type": "v2" }, { "addr": "172.21.15.190:6837", "nonce": 1151527362, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5725, "incarnation": 49, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 30, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-04-24T23:25:42.695130+0000", "qdb_cluster": [ 5599 ], "qdb_leader": 5599, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 2, 3, 4 ], "tableserver": 0, "up": { "mds_0": 5725, "mds_1": 5599 } } } ], "standbys": [ { "addr": "172.21.15.6:6834/1190759626", "addrs": { "addrvec": [ { "addr": "172.21.15.6:6832", "nonce": 1190759626, "type": "v2" }, { "addr": "172.21.15.6:6834", "nonce": 1190759626, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 30, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5081, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.190:6838/2416336040", "addrs": { "addrvec": [ { "addr": "172.21.15.190:6835", "nonce": 2416336040, "type": "v2" }, { "addr": "172.21.15.190:6838", "nonce": 2416336040, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 42, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5249, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.6:6837/3928731059", "addrs": { "addrvec": [ { "addr": "172.21.15.6:6836", "nonce": 3928731059, "type": "v2" }, { "addr": "172.21.15.6:6837", "nonce": 3928731059, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 35, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5503, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-04-24T23:25:47.044 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:25:47.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:47.363+0000 7fe867c00640 1 Processor -- start 2024-04-24T23:25:47.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:47.363+0000 7fe867c00640 1 -- start start 2024-04-24T23:25:47.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:47.367+0000 7fe867c00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe8600c6e10 0x7fe8600c7e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:25:47.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:47.368+0000 7fe867c00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe860071880 con 0x7fe8601593b0 2024-04-24T23:25:47.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:47.369+0000 7fe867c00640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe860072900 con 0x7fe8600c8390 2024-04-24T23:25:47.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:47.369+0000 7fe865c00640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7fe8600c8390 0x7fe860162460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.6:36004/0 (socket says 172.21.15.6:36004) 2024-04-24T23:25:47.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:47.370+0000 7fe865c00640 1 -- 172.21.15.6:0/3976563727 learned_addr learned my addr 172.21.15.6:0/3976563727 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:25:47.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:47.370+0000 7fe85fe00640 1 --2- 172.21.15.6:0/3976563727 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe8600c6e10 0x7fe8600c7e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:25:47.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:47.375+0000 7fe867c00640 1 -- 172.21.15.6:0/3976563727 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fe860072470 con 0x7fe8600c6e10 2024-04-24T23:25:47.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:47.378+0000 7fe85f400640 1 -- 172.21.15.6:0/3976563727 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2878629923 0 0) 0x7fe860072900 con 0x7fe8600c8390 2024-04-24T23:25:47.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:47.378+0000 7fe85f400640 1 -- 172.21.15.6:0/3976563727 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe844003520 con 0x7fe8600c8390 2024-04-24T23:25:47.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:47.378+0000 7fe85f400640 1 -- 172.21.15.6:0/3976563727 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2425709925 0 0) 0x7fe860071880 con 0x7fe8601593b0 2024-04-24T23:25:47.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:47.378+0000 7fe85f400640 1 -- 172.21.15.6:0/3976563727 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe860072900 con 0x7fe8601593b0 2024-04-24T23:25:47.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:47.379+0000 7fe85f400640 1 -- 172.21.15.6:0/3976563727 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3326980979 0 0) 0x7fe844003520 con 0x7fe8600c8390 2024-04-24T23:25:47.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:47.379+0000 7fe85f400640 1 -- 172.21.15.6:0/3976563727 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe8600c6e10 msgr2=0x7fe8600c7e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:25:47.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:47.379+0000 7fe85f400640 1 --2- 172.21.15.6:0/3976563727 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe8600c6e10 0x7fe8600c7e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:47.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:47.379+0000 7fe85f400640 1 -- 172.21.15.6:0/3976563727 >> v1:172.21.15.6:6789/0 conn(0x7fe8601593b0 legacy=0x7fe86006edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:25:47.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:47.380+0000 7fe85f400640 1 -- 172.21.15.6:0/3976563727 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8600c7410 con 0x7fe8600c8390 2024-04-24T23:25:47.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:47.383+0000 7fe85f400640 1 -- 172.21.15.6:0/3976563727 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3463268701 0 0) 0x7fe84c001670 con 0x7fe8600c8390 2024-04-24T23:25:47.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:47.383+0000 7fe85f400640 1 -- 172.21.15.6:0/3976563727 >> v1:172.21.15.190:6789/0 conn(0x7fe8600c8390 legacy=0x7fe860162460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:25:47.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:47.383+0000 7fe85f400640 1 --2- 172.21.15.6:0/3976563727 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe8600c6e10 0x7fe844003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:25:47.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:47.383+0000 7fe85f400640 1 --2- 172.21.15.6:0/3976563727 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe8440040b0 0x7fe844004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:25:47.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:47.384+0000 7fe865200640 1 --2- 172.21.15.6:0/3976563727 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe8600c6e10 0x7fe844003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:25:47.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:47.384+0000 7fe85fe00640 1 --2- 172.21.15.6:0/3976563727 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe8440040b0 0x7fe844004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:25:47.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:47.384+0000 7fe85f400640 1 --2- 172.21.15.6:0/3976563727 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe844004d40 0x7fe8440051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:25:47.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:47.384+0000 7fe85f400640 1 -- 172.21.15.6:0/3976563727 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fe860072470 con 0x7fe844004d40 2024-04-24T23:25:47.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:47.384+0000 7fe85f400640 1 -- 172.21.15.6:0/3976563727 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fe84400ba70 con 0x7fe8440040b0 2024-04-24T23:25:47.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:47.385+0000 7fe85f400640 1 -- 172.21.15.6:0/3976563727 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fe84400bc70 con 0x7fe8600c6e10 2024-04-24T23:25:47.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:47.385+0000 7fe865c00640 1 --2- 172.21.15.6:0/3976563727 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe844004d40 0x7fe8440051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:25:47.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:47.389+0000 7fe865200640 1 -- 172.21.15.6:0/3976563727 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe8440040b0 msgr2=0x7fe844004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:25:47.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:47.389+0000 7fe865200640 1 --2- 172.21.15.6:0/3976563727 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe8440040b0 0x7fe844004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:47.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:47.389+0000 7fe865200640 1 -- 172.21.15.6:0/3976563727 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe844004d40 msgr2=0x7fe8440051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:25:47.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:47.389+0000 7fe865200640 1 --2- 172.21.15.6:0/3976563727 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe844004d40 0x7fe8440051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:47.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:47.389+0000 7fe865200640 1 -- 172.21.15.6:0/3976563727 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe844003520 con 0x7fe8600c6e10 2024-04-24T23:25:47.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:47.391+0000 7fe865c00640 1 --2- 172.21.15.6:0/3976563727 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe844004d40 0x7fe8440051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:25:47.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:47.392+0000 7fe865200640 1 --2- 172.21.15.6:0/3976563727 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe8600c6e10 0x7fe844003b70 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fe85800c3a0 tx=0x7fe844005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9864ecebb26ba815 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:25:47.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:47.394+0000 7fe85f400640 1 -- 172.21.15.6:0/3976563727 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe858003970 con 0x7fe8600c6e10 2024-04-24T23:25:47.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:47.395+0000 7fe85f400640 1 -- 172.21.15.6:0/3976563727 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe858034070 con 0x7fe8600c6e10 2024-04-24T23:25:47.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:47.396+0000 7fe867c00640 1 -- 172.21.15.6:0/3976563727 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe8600c6e10 msgr2=0x7fe844003b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:25:47.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:47.396+0000 7fe867c00640 1 --2- 172.21.15.6:0/3976563727 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe8600c6e10 0x7fe844003b70 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fe85800c3a0 tx=0x7fe844005830 comp rx=0 tx=0).stop 2024-04-24T23:25:47.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:47.398+0000 7fe865c00640 1 -- 172.21.15.6:0/3976563727 reap_dead start 2024-04-24T23:25:47.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:47.398+0000 7fe867c00640 1 -- 172.21.15.6:0/3976563727 shutdown_connections 2024-04-24T23:25:47.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:47.398+0000 7fe867c00640 1 -- 172.21.15.6:0/3976563727 >> 172.21.15.6:0/3976563727 conn(0x7fe86006b0e0 msgr2=0x7fe86006b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:25:47.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:47.399+0000 7fe867c00640 1 -- 172.21.15.6:0/3976563727 shutdown_connections 2024-04-24T23:25:47.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:47.399+0000 7fe867c00640 1 -- 172.21.15.6:0/3976563727 wait complete. 2024-04-24T23:25:47.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:47.412+0000 7fe867c00640 1 Processor -- start 2024-04-24T23:25:47.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:47.412+0000 7fe867c00640 1 -- start start 2024-04-24T23:25:47.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:47.413+0000 7fe867c00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe8600c6e10 0x7fe86014c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:25:47.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:47.413+0000 7fe867c00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe8601f8690 0x7fe8601f4760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:25:47.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:47.416+0000 7fe867c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe86014cc10 0x7fe8601f4ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:25:47.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:47.416+0000 7fe867c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fe860072100 con 0x7fe86014cc10 2024-04-24T23:25:47.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:47.416+0000 7fe867c00640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fe860071e20 con 0x7fe8600c6e10 2024-04-24T23:25:47.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:47.416+0000 7fe867c00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fe860163080 con 0x7fe8601f8690 2024-04-24T23:25:47.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:47.417+0000 7fe865200640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe8600c6e10 0x7fe86014c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:25:47.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:47.417+0000 7fe85fe00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe8601f8690 0x7fe8601f4760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:25:47.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:47.417+0000 7fe865200640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe8600c6e10 0x7fe86014c0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:50896/0 (socket says 172.21.15.6:50896) 2024-04-24T23:25:47.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:47.417+0000 7fe865200640 1 -- 172.21.15.6:0/1792799495 learned_addr learned my addr 172.21.15.6:0/1792799495 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:25:47.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:47.418+0000 7fe865200640 1 -- 172.21.15.6:0/1792799495 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe8601f8690 msgr2=0x7fe8601f4760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:25:47.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:47.418+0000 7fe865200640 1 --2- 172.21.15.6:0/1792799495 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe8601f8690 0x7fe8601f4760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:47.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:47.418+0000 7fe865200640 1 -- 172.21.15.6:0/1792799495 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe86014cc10 msgr2=0x7fe8601f4ca0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-24T23:25:47.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:47.418+0000 7fe865200640 1 --2- 172.21.15.6:0/1792799495 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe86014cc10 0x7fe8601f4ca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:47.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:47.418+0000 7fe865200640 1 -- 172.21.15.6:0/1792799495 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe86014d260 con 0x7fe8600c6e10 2024-04-24T23:25:47.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:47.418+0000 7fe85fe00640 1 --2- 172.21.15.6:0/1792799495 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe8601f8690 0x7fe8601f4760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:25:47.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:47.419+0000 7fe865200640 1 --2- 172.21.15.6:0/1792799495 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe8600c6e10 0x7fe86014c0d0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fe85800c340 tx=0x7fe858001b10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:25:47.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:47.419+0000 7fe85cc00640 1 -- 172.21.15.6:0/1792799495 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe858006b10 con 0x7fe8600c6e10 2024-04-24T23:25:47.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:47.420+0000 7fe85cc00640 1 -- 172.21.15.6:0/1792799495 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe8580020b0 con 0x7fe8600c6e10 2024-04-24T23:25:47.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:47.420+0000 7fe85cc00640 1 -- 172.21.15.6:0/1792799495 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8580054a0 con 0x7fe8600c6e10 2024-04-24T23:25:47.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:47.420+0000 7fe867c00640 1 -- 172.21.15.6:0/1792799495 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8601f8b20 con 0x7fe8600c6e10 2024-04-24T23:25:47.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:47.421+0000 7fe867c00640 1 -- 172.21.15.6:0/1792799495 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe86014a580 con 0x7fe8600c6e10 2024-04-24T23:25:47.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:47.422+0000 7fe867c00640 1 -- 172.21.15.6:0/1792799495 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe8600cd1a0 con 0x7fe8600c6e10 2024-04-24T23:25:47.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:47.423+0000 7fe85cc00640 1 -- 172.21.15.6:0/1792799495 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fe858005640 con 0x7fe8600c6e10 2024-04-24T23:25:47.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:47.424+0000 7fe85cc00640 1 --2- 172.21.15.6:0/1792799495 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fe84006be60 0x7fe84006e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:25:47.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:47.424+0000 7fe85cc00640 1 -- 172.21.15.6:0/1792799495 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6934+0+0 (secure 0 0 0) 0x7fe85802c020 con 0x7fe8600c6e10 2024-04-24T23:25:47.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:47.425+0000 7fe85fe00640 1 --2- 172.21.15.6:0/1792799495 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fe84006be60 0x7fe84006e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:25:47.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:47.428+0000 7fe85fe00640 1 --2- 172.21.15.6:0/1792799495 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fe84006be60 0x7fe84006e320 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fe8600c60e0 tx=0x7fe850008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:25:47.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:47.433+0000 7fe85cc00640 1 -- 172.21.15.6:0/1792799495 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fe8600cd1a0 con 0x7fe8600c6e10 2024-04-24T23:25:47.891 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:25:47.891 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5081,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/1190759626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1190759626},{"type":"v1","addr":"172.21.15.6:6834","nonce":1190759626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":5249,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6838/2416336040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2416336040},{"type":"v1","addr":"172.21.15.190:6838","nonce":2416336040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":42},{"gid":5503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/3928731059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":3928731059},{"type":"v1","addr":"172.21.15.6:6837","nonce":3928731059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:25:42.695130+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5725,"mds_1":5599},"failed":[],"damaged":[],"stopped":[2,3,4],"info":{"gid_5599":{"gid":5599,"name":"d","rank":1,"incarnation":43,"state":"up:active","state_seq":25,"addr":"172.21.15.190:6839/2643233757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":2643233757},{"type":"v1","addr":"172.21.15.190:6839","nonce":2643233757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5725":{"gid":5725,"name":"f","rank":0,"incarnation":49,"state":"up:replay","state_seq":1,"addr":"172.21.15.190:6837/1151527362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":1151527362},{"type":"v1","addr":"172.21.15.190:6837","nonce":1151527362}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5599,"qdb_cluster":[5599]},"id":1}]} 2024-04-24T23:25:47.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:47.887+0000 7fe867c00640 1 -- 172.21.15.6:0/1792799495 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe8600767b0 con 0x7fe8600c6e10 2024-04-24T23:25:47.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:47.888+0000 7fe85cc00640 1 -- 172.21.15.6:0/1792799495 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6033 (secure 0 0 0) 0x7fe858079df0 con 0x7fe8600c6e10 2024-04-24T23:25:47.892 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 49 2024-04-24T23:25:47.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:47.893+0000 7fe867c00640 1 -- 172.21.15.6:0/1792799495 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fe84006be60 msgr2=0x7fe84006e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:25:47.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:47.893+0000 7fe867c00640 1 --2- 172.21.15.6:0/1792799495 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fe84006be60 0x7fe84006e320 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fe8600c60e0 tx=0x7fe850008040 comp rx=0 tx=0).stop 2024-04-24T23:25:47.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:47.893+0000 7fe867c00640 1 -- 172.21.15.6:0/1792799495 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe8600c6e10 msgr2=0x7fe86014c0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:25:47.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:47.893+0000 7fe867c00640 1 --2- 172.21.15.6:0/1792799495 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe8600c6e10 0x7fe86014c0d0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fe85800c340 tx=0x7fe858001b10 comp rx=0 tx=0).stop 2024-04-24T23:25:47.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:47.893+0000 7fe867c00640 1 -- 172.21.15.6:0/1792799495 shutdown_connections 2024-04-24T23:25:47.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:47.893+0000 7fe867c00640 1 --2- 172.21.15.6:0/1792799495 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fe84006be60 0x7fe84006e320 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:47.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:47.893+0000 7fe867c00640 1 --2- 172.21.15.6:0/1792799495 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe8601f8690 0x7fe8601f4760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:47.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:47.893+0000 7fe867c00640 1 --2- 172.21.15.6:0/1792799495 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe86014cc10 0x7fe8601f4ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:47.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:47.893+0000 7fe867c00640 1 --2- 172.21.15.6:0/1792799495 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe8600c6e10 0x7fe86014c0d0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:47.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:47.893+0000 7fe867c00640 1 -- 172.21.15.6:0/1792799495 >> 172.21.15.6:0/1792799495 conn(0x7fe86006b0e0 msgr2=0x7fe860156bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:25:47.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:47.893+0000 7fe867c00640 1 -- 172.21.15.6:0/1792799495 shutdown_connections 2024-04-24T23:25:47.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:47.893+0000 7fe867c00640 1 -- 172.21.15.6:0/1792799495 wait complete. 2024-04-24T23:25:47.920 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-24T23:25:49.921 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:25:50.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:50.234+0000 7fef36000640 1 Processor -- start 2024-04-24T23:25:50.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:50.235+0000 7fef36000640 1 -- start start 2024-04-24T23:25:50.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:50.236+0000 7fef36000640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fef300c8d50 0x7fef30156a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:25:50.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:50.236+0000 7fef36000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fef3006ce30 0x7fef300c7770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:25:50.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:50.236+0000 7fef36000640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fef3006e0b0 0x7fef300c7cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:25:50.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:50.236+0000 7fef36000640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7fef3006def0 con 0x7fef3006ce30 2024-04-24T23:25:50.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:50.236+0000 7fef36000640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7fef3006d370 con 0x7fef3006e0b0 2024-04-24T23:25:50.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:50.236+0000 7fef36000640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fef3006d650 con 0x7fef300c8d50 2024-04-24T23:25:50.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:50.237+0000 7fef35600640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fef3006e0b0 0x7fef300c7cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:25:50.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:50.237+0000 7fef35600640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fef3006e0b0 0x7fef300c7cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:50908/0 (socket says 172.21.15.6:50908) 2024-04-24T23:25:50.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:50.237+0000 7fef35600640 1 -- 172.21.15.6:0/3849343337 learned_addr learned my addr 172.21.15.6:0/3849343337 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:25:50.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:50.238+0000 7fef34c00640 1 --2- 172.21.15.6:0/3849343337 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fef300c8d50 0x7fef30156a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:25:50.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:50.239+0000 7fef2fe00640 1 --2- 172.21.15.6:0/3849343337 >> v2:172.21.15.6:3300/0 conn(0x7fef3006ce30 0x7fef300c7770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:25:50.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:50.239+0000 7fef34c00640 1 -- 172.21.15.6:0/3849343337 >> v2:172.21.15.190:3300/0 conn(0x7fef3006e0b0 msgr2=0x7fef300c7cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:25:50.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:50.239+0000 7fef34c00640 1 --2- 172.21.15.6:0/3849343337 >> v2:172.21.15.190:3300/0 conn(0x7fef3006e0b0 0x7fef300c7cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:50.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:50.239+0000 7fef34c00640 1 -- 172.21.15.6:0/3849343337 >> v2:172.21.15.6:3300/0 conn(0x7fef3006ce30 msgr2=0x7fef300c7770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:25:50.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:50.239+0000 7fef34c00640 1 --2- 172.21.15.6:0/3849343337 >> v2:172.21.15.6:3300/0 conn(0x7fef3006ce30 0x7fef300c7770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:50.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:50.239+0000 7fef34c00640 1 -- 172.21.15.6:0/3849343337 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef3006d930 con 0x7fef300c8d50 2024-04-24T23:25:50.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:50.239+0000 7fef35600640 1 --2- 172.21.15.6:0/3849343337 >> v2:172.21.15.190:3300/0 conn(0x7fef3006e0b0 0x7fef300c7cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:25:50.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:50.243+0000 7fef34c00640 1 --2- 172.21.15.6:0/3849343337 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fef300c8d50 0x7fef30156a20 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fef30154560 tx=0x7fef2802cb10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2ff66df0f77ecf29 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:25:50.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:50.244+0000 7fef2f400640 1 -- 172.21.15.6:0/3849343337 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef2802e870 con 0x7fef300c8d50 2024-04-24T23:25:50.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:50.245+0000 7fef2f400640 1 -- 172.21.15.6:0/3849343337 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fef2802ea10 con 0x7fef300c8d50 2024-04-24T23:25:50.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:50.245+0000 7fef2f400640 1 -- 172.21.15.6:0/3849343337 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef2802ede0 con 0x7fef300c8d50 2024-04-24T23:25:50.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:50.247+0000 7fef36000640 1 -- 172.21.15.6:0/3849343337 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fef300c8d50 msgr2=0x7fef30156a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:25:50.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:50.247+0000 7fef36000640 1 --2- 172.21.15.6:0/3849343337 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fef300c8d50 0x7fef30156a20 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fef30154560 tx=0x7fef2802cb10 comp rx=0 tx=0).stop 2024-04-24T23:25:50.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:50.250+0000 7fef36000640 1 -- 172.21.15.6:0/3849343337 shutdown_connections 2024-04-24T23:25:50.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:50.250+0000 7fef36000640 1 --2- 172.21.15.6:0/3849343337 >> v2:172.21.15.190:3300/0 conn(0x7fef3006e0b0 0x7fef300c7cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:50.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:50.250+0000 7fef36000640 1 --2- 172.21.15.6:0/3849343337 >> v2:172.21.15.6:3300/0 conn(0x7fef3006ce30 0x7fef300c7770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:50.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:50.250+0000 7fef36000640 1 --2- 172.21.15.6:0/3849343337 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fef300c8d50 0x7fef30156a20 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:50.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:50.250+0000 7fef36000640 1 -- 172.21.15.6:0/3849343337 >> 172.21.15.6:0/3849343337 conn(0x7fef3006ed80 msgr2=0x7fef3006f190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:25:50.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:50.250+0000 7fef36000640 1 -- 172.21.15.6:0/3849343337 shutdown_connections 2024-04-24T23:25:50.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:50.250+0000 7fef36000640 1 -- 172.21.15.6:0/3849343337 wait complete. 2024-04-24T23:25:50.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:50.253+0000 7fef36000640 1 Processor -- start 2024-04-24T23:25:50.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:50.259+0000 7fef36000640 1 -- start start 2024-04-24T23:25:50.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:50.260+0000 7fef36000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fef3006ce30 0x7fef300c93a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:25:50.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:50.260+0000 7fef36000640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fef3006e0b0 0x7fef30159a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:25:50.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:50.260+0000 7fef36000640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fef300c8d50 0x7fef30159f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:25:50.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:50.260+0000 7fef36000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fef300cc1e0 con 0x7fef3006ce30 2024-04-24T23:25:50.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:50.260+0000 7fef36000640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fef30071020 con 0x7fef3006e0b0 2024-04-24T23:25:50.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:50.260+0000 7fef36000640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fef300cc360 con 0x7fef300c8d50 2024-04-24T23:25:50.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:50.261+0000 7fef34c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fef3006ce30 0x7fef300c93a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:25:50.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:50.261+0000 7fef34c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fef3006ce30 0x7fef300c93a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:56308/0 (socket says 172.21.15.6:56308) 2024-04-24T23:25:50.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:50.261+0000 7fef34c00640 1 -- 172.21.15.6:0/748069044 learned_addr learned my addr 172.21.15.6:0/748069044 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:25:50.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:50.261+0000 7fef2fe00640 1 --2- 172.21.15.6:0/748069044 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fef3006e0b0 0x7fef30159a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:25:50.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:50.262+0000 7fef35600640 1 --2- 172.21.15.6:0/748069044 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fef300c8d50 0x7fef30159f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:25:50.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:50.262+0000 7fef2fe00640 1 -- 172.21.15.6:0/748069044 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fef300c8d50 msgr2=0x7fef30159f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:25:50.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:50.262+0000 7fef2fe00640 1 --2- 172.21.15.6:0/748069044 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fef300c8d50 0x7fef30159f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:50.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:50.262+0000 7fef2fe00640 1 -- 172.21.15.6:0/748069044 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fef3006ce30 msgr2=0x7fef300c93a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:25:50.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:50.263+0000 7fef2fe00640 1 --2- 172.21.15.6:0/748069044 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fef3006ce30 0x7fef300c93a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:50.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:50.263+0000 7fef2fe00640 1 -- 172.21.15.6:0/748069044 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef301ece60 con 0x7fef3006e0b0 2024-04-24T23:25:50.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:50.263+0000 7fef35600640 1 --2- 172.21.15.6:0/748069044 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fef300c8d50 0x7fef30159f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:25:50.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:50.263+0000 7fef2fe00640 1 --2- 172.21.15.6:0/748069044 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fef3006e0b0 0x7fef30159a10 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fef1c00d5b0 tx=0x7fef1c00da70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:25:50.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:50.263+0000 7fef34c00640 1 --2- 172.21.15.6:0/748069044 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fef3006ce30 0x7fef300c93a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:25:50.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:50.263+0000 7fef2cc00640 1 -- 172.21.15.6:0/748069044 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef1c018020 con 0x7fef3006e0b0 2024-04-24T23:25:50.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:50.264+0000 7fef36000640 1 -- 172.21.15.6:0/748069044 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef30158660 con 0x7fef3006e0b0 2024-04-24T23:25:50.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:50.264+0000 7fef2cc00640 1 -- 172.21.15.6:0/748069044 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fef1c014620 con 0x7fef3006e0b0 2024-04-24T23:25:50.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:50.264+0000 7fef2cc00640 1 -- 172.21.15.6:0/748069044 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef1c0149d0 con 0x7fef3006e0b0 2024-04-24T23:25:50.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:50.264+0000 7fef36000640 1 -- 172.21.15.6:0/748069044 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef301ecaf0 con 0x7fef3006e0b0 2024-04-24T23:25:50.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:50.266+0000 7fef36000640 1 -- 172.21.15.6:0/748069044 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef301578a0 con 0x7fef3006e0b0 2024-04-24T23:25:50.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:50.267+0000 7fef2cc00640 1 -- 172.21.15.6:0/748069044 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fef1c01b020 con 0x7fef3006e0b0 2024-04-24T23:25:50.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:50.268+0000 7fef2cc00640 1 --2- 172.21.15.6:0/748069044 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fef1006be60 0x7fef1006e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:25:50.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:50.268+0000 7fef2cc00640 1 -- 172.21.15.6:0/748069044 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6934+0+0 (secure 0 0 0) 0x7fef1c08b890 con 0x7fef3006e0b0 2024-04-24T23:25:50.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:50.269+0000 7fef34c00640 1 --2- 172.21.15.6:0/748069044 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fef1006be60 0x7fef1006e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:25:50.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:50.270+0000 7fef34c00640 1 --2- 172.21.15.6:0/748069044 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fef1006be60 0x7fef1006e320 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fef28009bd0 tx=0x7fef28032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:25:50.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:50.273+0000 7fef2cc00640 1 -- 172.21.15.6:0/748069044 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fef301578a0 con 0x7fef3006e0b0 2024-04-24T23:25:50.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:50.830+0000 7fef36000640 1 -- 172.21.15.6:0/748069044 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fef30075b30 con 0x7fef3006e0b0 2024-04-24T23:25:50.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:50.833+0000 7fef2cc00640 1 -- 172.21.15.6:0/748069044 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6033 (secure 0 0 0) 0x7fef1c022020 con 0x7fef3006e0b0 2024-04-24T23:25:50.838 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:25:50.838 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5081,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/1190759626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1190759626},{"type":"v1","addr":"172.21.15.6:6834","nonce":1190759626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":5249,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6838/2416336040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2416336040},{"type":"v1","addr":"172.21.15.190:6838","nonce":2416336040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":42},{"gid":5503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/3928731059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":3928731059},{"type":"v1","addr":"172.21.15.6:6837","nonce":3928731059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:25:42.695130+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5725,"mds_1":5599},"failed":[],"damaged":[],"stopped":[2,3,4],"info":{"gid_5599":{"gid":5599,"name":"d","rank":1,"incarnation":43,"state":"up:active","state_seq":25,"addr":"172.21.15.190:6839/2643233757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":2643233757},{"type":"v1","addr":"172.21.15.190:6839","nonce":2643233757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5725":{"gid":5725,"name":"f","rank":0,"incarnation":49,"state":"up:replay","state_seq":1,"addr":"172.21.15.190:6837/1151527362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":1151527362},{"type":"v1","addr":"172.21.15.190:6837","nonce":1151527362}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5599,"qdb_cluster":[5599]},"id":1}]} 2024-04-24T23:25:50.838 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 49 2024-04-24T23:25:50.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:50.837+0000 7fef36000640 1 -- 172.21.15.6:0/748069044 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fef1006be60 msgr2=0x7fef1006e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:25:50.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:50.837+0000 7fef36000640 1 --2- 172.21.15.6:0/748069044 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fef1006be60 0x7fef1006e320 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fef28009bd0 tx=0x7fef28032000 comp rx=0 tx=0).stop 2024-04-24T23:25:50.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:50.837+0000 7fef36000640 1 -- 172.21.15.6:0/748069044 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fef3006e0b0 msgr2=0x7fef30159a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:25:50.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:50.837+0000 7fef36000640 1 --2- 172.21.15.6:0/748069044 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fef3006e0b0 0x7fef30159a10 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fef1c00d5b0 tx=0x7fef1c00da70 comp rx=0 tx=0).stop 2024-04-24T23:25:50.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:50.838+0000 7fef36000640 1 -- 172.21.15.6:0/748069044 shutdown_connections 2024-04-24T23:25:50.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:50.838+0000 7fef36000640 1 --2- 172.21.15.6:0/748069044 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fef1006be60 0x7fef1006e320 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:50.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:50.838+0000 7fef36000640 1 --2- 172.21.15.6:0/748069044 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fef300c8d50 0x7fef30159f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:50.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:50.838+0000 7fef36000640 1 --2- 172.21.15.6:0/748069044 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fef3006e0b0 0x7fef30159a10 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:50.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:50.838+0000 7fef36000640 1 --2- 172.21.15.6:0/748069044 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fef3006ce30 0x7fef300c93a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:50.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:50.838+0000 7fef36000640 1 -- 172.21.15.6:0/748069044 >> 172.21.15.6:0/748069044 conn(0x7fef3006ed80 msgr2=0x7fef300ca660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:25:50.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:50.838+0000 7fef36000640 1 -- 172.21.15.6:0/748069044 shutdown_connections 2024-04-24T23:25:50.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:50.838+0000 7fef36000640 1 -- 172.21.15.6:0/748069044 wait complete. 2024-04-24T23:25:50.904 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-24T23:25:52.905 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:25:53.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:53.256+0000 7f9efcc00640 1 Processor -- start 2024-04-24T23:25:53.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:53.256+0000 7f9efcc00640 1 -- start start 2024-04-24T23:25:53.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:53.259+0000 7f9efcc00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9ef8150d50 0x7f9ef815aab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:25:53.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:53.259+0000 7f9efcc00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f9ef806e3a0 0x7f9ef80c6e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:25:53.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:53.259+0000 7f9efcc00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9ef806ec50 con 0x7f9ef806e7c0 2024-04-24T23:25:53.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:53.259+0000 7f9efcc00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f9ef815aff0 con 0x7f9ef806e3a0 2024-04-24T23:25:53.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:53.259+0000 7f9efcc00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f9ef806ef30 con 0x7f9ef8150d50 2024-04-24T23:25:53.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:53.262+0000 7f9ef5400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f9ef806e3a0 0x7f9ef80c6e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:25:53.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:53.262+0000 7f9ef5e00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9ef8150d50 0x7f9ef815aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:25:53.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:53.262+0000 7f9ef5400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f9ef806e3a0 0x7f9ef80c6e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:56316/0 (socket says 172.21.15.6:56316) 2024-04-24T23:25:53.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:53.262+0000 7f9ef5400640 1 -- 172.21.15.6:0/2398961578 learned_addr learned my addr 172.21.15.6:0/2398961578 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:25:53.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:53.263+0000 7f9ef5400640 1 -- 172.21.15.6:0/2398961578 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9ef8150d50 msgr2=0x7f9ef815aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:25:53.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:53.263+0000 7f9ef5400640 1 --2- 172.21.15.6:0/2398961578 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9ef8150d50 0x7f9ef815aab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:53.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:53.263+0000 7f9ef5400640 1 -- 172.21.15.6:0/2398961578 >> v1:172.21.15.6:6789/0 conn(0x7f9ef806e7c0 legacy=0x7f9ef80c8d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:25:53.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:53.263+0000 7f9ef5400640 1 -- 172.21.15.6:0/2398961578 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ef806f210 con 0x7f9ef806e3a0 2024-04-24T23:25:53.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:53.282+0000 7f9ef5400640 1 --2- 172.21.15.6:0/2398961578 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9ef806e3a0 0x7f9ef80c6e10 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f9ee4009870 tx=0x7f9ee402a2a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da3ea99126e5e8ea server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:25:53.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:53.285+0000 7f9ef4a00640 1 -- 172.21.15.6:0/2398961578 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ee4037070 con 0x7f9ef806e3a0 2024-04-24T23:25:53.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:53.285+0000 7f9ef4a00640 1 -- 172.21.15.6:0/2398961578 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9ee4032440 con 0x7f9ef806e3a0 2024-04-24T23:25:53.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:53.285+0000 7f9ef4a00640 1 -- 172.21.15.6:0/2398961578 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ee4037070 con 0x7f9ef806e3a0 2024-04-24T23:25:53.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:53.287+0000 7f9efcc00640 1 -- 172.21.15.6:0/2398961578 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9ef806e3a0 msgr2=0x7f9ef80c6e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:25:53.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:53.287+0000 7f9efcc00640 1 --2- 172.21.15.6:0/2398961578 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9ef806e3a0 0x7f9ef80c6e10 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f9ee4009870 tx=0x7f9ee402a2a0 comp rx=0 tx=0).stop 2024-04-24T23:25:53.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:53.288+0000 7f9efcc00640 1 -- 172.21.15.6:0/2398961578 shutdown_connections 2024-04-24T23:25:53.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:53.288+0000 7f9efcc00640 1 --2- 172.21.15.6:0/2398961578 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9ef806e3a0 0x7f9ef80c6e10 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:53.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:53.288+0000 7f9efcc00640 1 --2- 172.21.15.6:0/2398961578 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9ef8150d50 0x7f9ef815aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:53.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:53.288+0000 7f9efcc00640 1 -- 172.21.15.6:0/2398961578 >> 172.21.15.6:0/2398961578 conn(0x7f9ef806b0e0 msgr2=0x7f9ef806b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:25:53.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:53.290+0000 7f9efcc00640 1 -- 172.21.15.6:0/2398961578 shutdown_connections 2024-04-24T23:25:53.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:53.290+0000 7f9efcc00640 1 -- 172.21.15.6:0/2398961578 wait complete. 2024-04-24T23:25:53.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:53.296+0000 7f9efcc00640 1 Processor -- start 2024-04-24T23:25:53.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:53.297+0000 7f9efcc00640 1 -- start start 2024-04-24T23:25:53.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:53.300+0000 7f9efcc00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9ef806e3a0 0x7f9ef81f87a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:25:53.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:53.301+0000 7f9efcc00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9ef8150d50 0x7f9ef814ae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:25:53.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:53.301+0000 7f9efcc00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9ef814b390 0x7f9ef81f4ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:25:53.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:53.301+0000 7f9efcc00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f9ef80c8810 con 0x7f9ef806e3a0 2024-04-24T23:25:53.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:53.302+0000 7f9efcc00640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f9ef80c8530 con 0x7f9ef8150d50 2024-04-24T23:25:53.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:53.302+0000 7f9ef5e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9ef806e3a0 0x7f9ef81f87a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:25:53.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:53.302+0000 7f9ef5e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9ef806e3a0 0x7f9ef81f87a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:56330/0 (socket says 172.21.15.6:56330) 2024-04-24T23:25:53.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:53.303+0000 7f9ef5e00640 1 -- 172.21.15.6:0/1121291650 learned_addr learned my addr 172.21.15.6:0/1121291650 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:25:53.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:53.304+0000 7f9ef5400640 1 --2- 172.21.15.6:0/1121291650 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9ef8150d50 0x7f9ef814ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:25:53.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:53.304+0000 7f9efcc00640 1 -- 172.21.15.6:0/1121291650 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f9ef8151130 con 0x7f9ef814b390 2024-04-24T23:25:53.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:53.304+0000 7f9ef6800640 1 --2- 172.21.15.6:0/1121291650 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9ef814b390 0x7f9ef81f4ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:25:53.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:53.309+0000 7f9ef5e00640 1 -- 172.21.15.6:0/1121291650 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9ef814b390 msgr2=0x7f9ef81f4ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:25:53.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:53.309+0000 7f9ef5e00640 1 --2- 172.21.15.6:0/1121291650 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9ef814b390 0x7f9ef81f4ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:53.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:53.309+0000 7f9ef5e00640 1 -- 172.21.15.6:0/1121291650 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9ef8150d50 msgr2=0x7f9ef814ae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:25:53.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:53.309+0000 7f9ef5e00640 1 --2- 172.21.15.6:0/1121291650 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9ef8150d50 0x7f9ef814ae50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:53.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:53.310+0000 7f9ef5e00640 1 -- 172.21.15.6:0/1121291650 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ef81f8ce0 con 0x7f9ef806e3a0 2024-04-24T23:25:53.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:53.310+0000 7f9ef5400640 1 --2- 172.21.15.6:0/1121291650 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9ef8150d50 0x7f9ef814ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:25:53.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:53.311+0000 7f9ef5e00640 1 --2- 172.21.15.6:0/1121291650 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9ef806e3a0 0x7f9ef81f87a0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f9ee800c820 tx=0x7f9ee800cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:25:53.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:53.311+0000 7f9eee000640 1 -- 172.21.15.6:0/1121291650 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ee8015070 con 0x7f9ef806e3a0 2024-04-24T23:25:53.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:53.312+0000 7f9efcc00640 1 -- 172.21.15.6:0/1121291650 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ef81513a0 con 0x7f9ef806e3a0 2024-04-24T23:25:53.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:53.312+0000 7f9efcc00640 1 -- 172.21.15.6:0/1121291650 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ef8153a60 con 0x7f9ef806e3a0 2024-04-24T23:25:53.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:53.312+0000 7f9eee000640 1 -- 172.21.15.6:0/1121291650 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9ee8004020 con 0x7f9ef806e3a0 2024-04-24T23:25:53.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:53.312+0000 7f9eee000640 1 -- 172.21.15.6:0/1121291650 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ee80043f0 con 0x7f9ef806e3a0 2024-04-24T23:25:53.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:53.319+0000 7f9eee000640 1 -- 172.21.15.6:0/1121291650 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f9ee8007830 con 0x7f9ef806e3a0 2024-04-24T23:25:53.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:53.321+0000 7f9efcc00640 1 -- 172.21.15.6:0/1121291650 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ed0005290 con 0x7f9ef806e3a0 2024-04-24T23:25:53.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:53.322+0000 7f9eee000640 1 --2- 172.21.15.6:0/1121291650 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f9ed806bf30 0x7f9ed806e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:25:53.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:53.326+0000 7f9eee000640 1 -- 172.21.15.6:0/1121291650 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6934+0+0 (secure 0 0 0) 0x7f9ee808af70 con 0x7f9ef806e3a0 2024-04-24T23:25:53.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:53.326+0000 7f9ef5400640 1 --2- 172.21.15.6:0/1121291650 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f9ed806bf30 0x7f9ed806e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:25:53.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:53.327+0000 7f9ef5400640 1 --2- 172.21.15.6:0/1121291650 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f9ed806bf30 0x7f9ed806e3f0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f9ee4034000 tx=0x7f9ee402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:25:53.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:53.334+0000 7f9eee000640 1 -- 172.21.15.6:0/1121291650 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f9ef81513a0 con 0x7f9ef806e3a0 2024-04-24T23:25:53.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:53.842+0000 7f9efcc00640 1 -- 172.21.15.6:0/1121291650 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9ed0005580 con 0x7f9ef806e3a0 2024-04-24T23:25:53.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:53.845+0000 7f9eee000640 1 -- 172.21.15.6:0/1121291650 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6033 (secure 0 0 0) 0x7f9ee8058e70 con 0x7f9ef806e3a0 2024-04-24T23:25:53.849 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:25:53.849 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5081,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/1190759626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1190759626},{"type":"v1","addr":"172.21.15.6:6834","nonce":1190759626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":5249,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6838/2416336040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2416336040},{"type":"v1","addr":"172.21.15.190:6838","nonce":2416336040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":42},{"gid":5503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/3928731059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":3928731059},{"type":"v1","addr":"172.21.15.6:6837","nonce":3928731059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:25:42.695130+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5725,"mds_1":5599},"failed":[],"damaged":[],"stopped":[2,3,4],"info":{"gid_5599":{"gid":5599,"name":"d","rank":1,"incarnation":43,"state":"up:active","state_seq":25,"addr":"172.21.15.190:6839/2643233757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":2643233757},{"type":"v1","addr":"172.21.15.190:6839","nonce":2643233757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5725":{"gid":5725,"name":"f","rank":0,"incarnation":49,"state":"up:replay","state_seq":1,"addr":"172.21.15.190:6837/1151527362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":1151527362},{"type":"v1","addr":"172.21.15.190:6837","nonce":1151527362}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5599,"qdb_cluster":[5599]},"id":1}]} 2024-04-24T23:25:53.849 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 49 2024-04-24T23:25:53.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:53.852+0000 7f9efcc00640 1 -- 172.21.15.6:0/1121291650 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f9ed806bf30 msgr2=0x7f9ed806e3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:25:53.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:53.852+0000 7f9efcc00640 1 --2- 172.21.15.6:0/1121291650 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f9ed806bf30 0x7f9ed806e3f0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f9ee4034000 tx=0x7f9ee402e000 comp rx=0 tx=0).stop 2024-04-24T23:25:53.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:53.852+0000 7f9efcc00640 1 -- 172.21.15.6:0/1121291650 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9ef806e3a0 msgr2=0x7f9ef81f87a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:25:53.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:53.852+0000 7f9efcc00640 1 --2- 172.21.15.6:0/1121291650 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9ef806e3a0 0x7f9ef81f87a0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f9ee800c820 tx=0x7f9ee800cce0 comp rx=0 tx=0).stop 2024-04-24T23:25:53.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:53.856+0000 7f9efcc00640 1 -- 172.21.15.6:0/1121291650 shutdown_connections 2024-04-24T23:25:53.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:53.856+0000 7f9efcc00640 1 --2- 172.21.15.6:0/1121291650 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f9ed806bf30 0x7f9ed806e3f0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:53.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:53.856+0000 7f9efcc00640 1 --2- 172.21.15.6:0/1121291650 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9ef814b390 0x7f9ef81f4ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:53.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:53.856+0000 7f9efcc00640 1 --2- 172.21.15.6:0/1121291650 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9ef8150d50 0x7f9ef814ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:53.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:53.856+0000 7f9efcc00640 1 --2- 172.21.15.6:0/1121291650 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9ef806e3a0 0x7f9ef81f87a0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:53.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:53.856+0000 7f9efcc00640 1 -- 172.21.15.6:0/1121291650 >> 172.21.15.6:0/1121291650 conn(0x7f9ef806b0e0 msgr2=0x7f9ef80ca5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:25:53.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:53.856+0000 7f9efcc00640 1 -- 172.21.15.6:0/1121291650 shutdown_connections 2024-04-24T23:25:53.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:53.856+0000 7f9efcc00640 1 -- 172.21.15.6:0/1121291650 wait complete. 2024-04-24T23:25:53.898 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-24T23:25:55.899 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:25:56.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:56.270+0000 7f651b200640 1 Processor -- start 2024-04-24T23:25:56.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:56.271+0000 7f651b200640 1 -- start start 2024-04-24T23:25:56.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:56.273+0000 7f651b200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f651404b500 0x7f65140dac60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:25:56.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:56.273+0000 7f651b200640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f651404b8e0 0x7f6514005ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:25:56.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:56.273+0000 7f651b200640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6514007760 con 0x7f6514006420 2024-04-24T23:25:56.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:56.273+0000 7f651b200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f6514006900 con 0x7f651404b500 2024-04-24T23:25:56.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:56.273+0000 7f651b200640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f6514006be0 con 0x7f651404b8e0 2024-04-24T23:25:56.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:56.277+0000 7f651a800640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f6514006420 0x7f6514034a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.6:47420/0 (socket says 172.21.15.6:47420) 2024-04-24T23:25:56.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:56.277+0000 7f651a800640 1 -- 172.21.15.6:0/1608708315 learned_addr learned my addr 172.21.15.6:0/1608708315 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:25:56.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:56.277+0000 7f6519400640 1 --2- 172.21.15.6:0/1608708315 >> v2:172.21.15.190:3300/0 conn(0x7f651404b8e0 0x7f6514005ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:25:56.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:56.278+0000 7f6519e00640 1 --2- 172.21.15.6:0/1608708315 >> v2:172.21.15.6:3300/0 conn(0x7f651404b500 0x7f65140dac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:25:56.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:56.279+0000 7f6519400640 1 -- 172.21.15.6:0/1608708315 >> v2:172.21.15.6:3300/0 conn(0x7f651404b500 msgr2=0x7f65140dac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:25:56.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:56.279+0000 7f6519400640 1 --2- 172.21.15.6:0/1608708315 >> v2:172.21.15.6:3300/0 conn(0x7f651404b500 0x7f65140dac60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:56.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:56.279+0000 7f6519400640 1 -- 172.21.15.6:0/1608708315 >> v1:172.21.15.190:6789/0 conn(0x7f6514006420 legacy=0x7f6514034a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:25:56.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:56.279+0000 7f6519400640 1 -- 172.21.15.6:0/1608708315 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6514006ec0 con 0x7f651404b8e0 2024-04-24T23:25:56.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:56.279+0000 7f6518a00640 1 -- 172.21.15.6:0/1608708315 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2268178359 0 0) 0x7f6514007760 con 0x7f6514006420 2024-04-24T23:25:56.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:56.283+0000 7f6519400640 1 --2- 172.21.15.6:0/1608708315 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f651404b8e0 0x7f6514005ee0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f651c04fa20 tx=0x7f651c0a64f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c5054cfea9f1d7a6 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:25:56.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:56.284+0000 7f6518a00640 1 -- 172.21.15.6:0/1608708315 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f651c0ad600 con 0x7f651404b8e0 2024-04-24T23:25:56.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:56.285+0000 7f6518a00640 1 -- 172.21.15.6:0/1608708315 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f651c0ad7a0 con 0x7f651404b8e0 2024-04-24T23:25:56.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:56.285+0000 7f6518a00640 1 -- 172.21.15.6:0/1608708315 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f651c0adc10 con 0x7f651404b8e0 2024-04-24T23:25:56.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:56.287+0000 7f651b200640 1 -- 172.21.15.6:0/1608708315 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f651404b8e0 msgr2=0x7f6514005ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:25:56.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:56.287+0000 7f651b200640 1 --2- 172.21.15.6:0/1608708315 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f651404b8e0 0x7f6514005ee0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f651c04fa20 tx=0x7f651c0a64f0 comp rx=0 tx=0).stop 2024-04-24T23:25:56.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:56.287+0000 7f651b200640 1 -- 172.21.15.6:0/1608708315 shutdown_connections 2024-04-24T23:25:56.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:56.287+0000 7f651b200640 1 --2- 172.21.15.6:0/1608708315 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f651404b8e0 0x7f6514005ee0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:56.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:56.287+0000 7f651b200640 1 --2- 172.21.15.6:0/1608708315 >> v2:172.21.15.6:3300/0 conn(0x7f651404b500 0x7f65140dac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:56.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:56.287+0000 7f651b200640 1 -- 172.21.15.6:0/1608708315 >> 172.21.15.6:0/1608708315 conn(0x7f6514050180 msgr2=0x7f6514050590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:25:56.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:56.295+0000 7f651b200640 1 -- 172.21.15.6:0/1608708315 shutdown_connections 2024-04-24T23:25:56.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:56.296+0000 7f651b200640 1 -- 172.21.15.6:0/1608708315 wait complete. 2024-04-24T23:25:56.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:56.298+0000 7f651b200640 1 Processor -- start 2024-04-24T23:25:56.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:56.299+0000 7f651b200640 1 -- start start 2024-04-24T23:25:56.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:56.302+0000 7f651b200640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6514006420 0x7f65141828f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:25:56.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:56.302+0000 7f651b200640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f651404b500 0x7f6514184c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:25:56.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:56.302+0000 7f651b200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f651404b8e0 0x7f65140d11b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:25:56.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:56.302+0000 7f651b200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f651404bd00 con 0x7f651404b8e0 2024-04-24T23:25:56.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:56.302+0000 7f651b200640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f6514007480 con 0x7f651404b500 2024-04-24T23:25:56.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:56.302+0000 7f651b200640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f65140d5000 con 0x7f6514006420 2024-04-24T23:25:56.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:56.303+0000 7f651a800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f651404b8e0 0x7f65140d11b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:25:56.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:56.303+0000 7f651a800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f651404b8e0 0x7f65140d11b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:34182/0 (socket says 172.21.15.6:34182) 2024-04-24T23:25:56.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:56.303+0000 7f651a800640 1 -- 172.21.15.6:0/348307597 learned_addr learned my addr 172.21.15.6:0/348307597 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:25:56.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:56.304+0000 7f651a800640 1 -- 172.21.15.6:0/348307597 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6514006420 msgr2=0x7f65141828f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-24T23:25:56.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:56.304+0000 7f651a800640 1 --2- 172.21.15.6:0/348307597 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6514006420 0x7f65141828f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:56.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:56.304+0000 7f651a800640 1 -- 172.21.15.6:0/348307597 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f651404b500 msgr2=0x7f6514184c60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-24T23:25:56.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:56.304+0000 7f651a800640 1 --2- 172.21.15.6:0/348307597 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f651404b500 0x7f6514184c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:56.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:56.304+0000 7f651a800640 1 -- 172.21.15.6:0/348307597 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6514182e30 con 0x7f651404b8e0 2024-04-24T23:25:56.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:56.304+0000 7f651a800640 1 --2- 172.21.15.6:0/348307597 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f651404b8e0 0x7f65140d11b0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f650c00b930 tx=0x7f650c00bdf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:25:56.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:56.306+0000 7f6512000640 1 -- 172.21.15.6:0/348307597 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f650c00cc30 con 0x7f651404b8e0 2024-04-24T23:25:56.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:56.306+0000 7f6512000640 1 -- 172.21.15.6:0/348307597 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f650c010070 con 0x7f651404b8e0 2024-04-24T23:25:56.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:56.306+0000 7f6512000640 1 -- 172.21.15.6:0/348307597 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f650c015470 con 0x7f651404b8e0 2024-04-24T23:25:56.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:56.306+0000 7f651b200640 1 -- 172.21.15.6:0/348307597 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6514185590 con 0x7f651404b8e0 2024-04-24T23:25:56.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:56.306+0000 7f651b200640 1 -- 172.21.15.6:0/348307597 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6514049850 con 0x7f651404b8e0 2024-04-24T23:25:56.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:56.309+0000 7f651b200640 1 -- 172.21.15.6:0/348307597 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f64f4005290 con 0x7f651404b8e0 2024-04-24T23:25:56.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:56.312+0000 7f6512000640 1 -- 172.21.15.6:0/348307597 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f650c015630 con 0x7f651404b8e0 2024-04-24T23:25:56.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:56.313+0000 7f6512000640 1 --2- 172.21.15.6:0/348307597 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f64fc06bf00 0x7f64fc06e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:25:56.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:56.316+0000 7f6512000640 1 -- 172.21.15.6:0/348307597 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6934+0+0 (secure 0 0 0) 0x7f650c08b450 con 0x7f651404b8e0 2024-04-24T23:25:56.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:56.317+0000 7f6519e00640 1 --2- 172.21.15.6:0/348307597 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f64fc06bf00 0x7f64fc06e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:25:56.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:56.319+0000 7f6519e00640 1 --2- 172.21.15.6:0/348307597 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f64fc06bf00 0x7f64fc06e3c0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f6508009870 tx=0x7f6508008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:25:56.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:56.324+0000 7f6512000640 1 -- 172.21.15.6:0/348307597 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f650c059910 con 0x7f651404b8e0 2024-04-24T23:25:56.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:56.763+0000 7f651b200640 1 -- 172.21.15.6:0/348307597 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f64f4005580 con 0x7f651404b8e0 2024-04-24T23:25:56.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:56.766+0000 7f6512000640 1 -- 172.21.15.6:0/348307597 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6033 (secure 0 0 0) 0x7f650c059210 con 0x7f651404b8e0 2024-04-24T23:25:56.768 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:25:56.768 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5081,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/1190759626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1190759626},{"type":"v1","addr":"172.21.15.6:6834","nonce":1190759626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":5249,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6838/2416336040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2416336040},{"type":"v1","addr":"172.21.15.190:6838","nonce":2416336040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":42},{"gid":5503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/3928731059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":3928731059},{"type":"v1","addr":"172.21.15.6:6837","nonce":3928731059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:25:42.695130+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5725,"mds_1":5599},"failed":[],"damaged":[],"stopped":[2,3,4],"info":{"gid_5599":{"gid":5599,"name":"d","rank":1,"incarnation":43,"state":"up:active","state_seq":25,"addr":"172.21.15.190:6839/2643233757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":2643233757},{"type":"v1","addr":"172.21.15.190:6839","nonce":2643233757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5725":{"gid":5725,"name":"f","rank":0,"incarnation":49,"state":"up:replay","state_seq":1,"addr":"172.21.15.190:6837/1151527362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":1151527362},{"type":"v1","addr":"172.21.15.190:6837","nonce":1151527362}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5599,"qdb_cluster":[5599]},"id":1}]} 2024-04-24T23:25:56.768 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 49 2024-04-24T23:25:56.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:56.772+0000 7f651b200640 1 -- 172.21.15.6:0/348307597 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f64fc06bf00 msgr2=0x7f64fc06e3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:25:56.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:56.772+0000 7f651b200640 1 --2- 172.21.15.6:0/348307597 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f64fc06bf00 0x7f64fc06e3c0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f6508009870 tx=0x7f6508008040 comp rx=0 tx=0).stop 2024-04-24T23:25:56.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:56.773+0000 7f651b200640 1 -- 172.21.15.6:0/348307597 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f651404b8e0 msgr2=0x7f65140d11b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:25:56.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:56.773+0000 7f651b200640 1 --2- 172.21.15.6:0/348307597 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f651404b8e0 0x7f65140d11b0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f650c00b930 tx=0x7f650c00bdf0 comp rx=0 tx=0).stop 2024-04-24T23:25:56.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:56.775+0000 7f651b200640 1 -- 172.21.15.6:0/348307597 shutdown_connections 2024-04-24T23:25:56.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:56.775+0000 7f651b200640 1 --2- 172.21.15.6:0/348307597 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f64fc06bf00 0x7f64fc06e3c0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:56.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:56.775+0000 7f651b200640 1 --2- 172.21.15.6:0/348307597 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f651404b8e0 0x7f65140d11b0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:56.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:56.775+0000 7f651b200640 1 --2- 172.21.15.6:0/348307597 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f651404b500 0x7f6514184c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:56.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:56.775+0000 7f651b200640 1 --2- 172.21.15.6:0/348307597 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6514006420 0x7f65141828f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:56.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:56.775+0000 7f651b200640 1 -- 172.21.15.6:0/348307597 >> 172.21.15.6:0/348307597 conn(0x7f6514050180 msgr2=0x7f651404cb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:25:56.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:56.777+0000 7f651b200640 1 -- 172.21.15.6:0/348307597 shutdown_connections 2024-04-24T23:25:56.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:56.777+0000 7f651b200640 1 -- 172.21.15.6:0/348307597 wait complete. 2024-04-24T23:25:56.815 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-24T23:25:58.816 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:25:59.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:59.148+0000 7f35ad000640 1 Processor -- start 2024-04-24T23:25:59.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:59.148+0000 7f35ad000640 1 -- start start 2024-04-24T23:25:59.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:59.150+0000 7f35ad000640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f35a8159070 0x7f35a8162d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:25:59.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:59.150+0000 7f35ad000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f35a8072220 0x7f35a80c8c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:25:59.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:59.150+0000 7f35ad000640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f35a8159450 con 0x7f35a80c6be0 2024-04-24T23:25:59.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:59.150+0000 7f35ad000640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f35a8071550 con 0x7f35a8072220 2024-04-24T23:25:59.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:59.151+0000 7f35ad000640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f35a80cc860 con 0x7f35a8159070 2024-04-24T23:25:59.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:59.151+0000 7f35a7400640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f35a8159070 0x7f35a8162d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:25:59.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:59.151+0000 7f35a7400640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f35a8159070 0x7f35a8162d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.6:38262/0 (socket says 172.21.15.6:38262) 2024-04-24T23:25:59.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:59.151+0000 7f35a7400640 1 -- 172.21.15.6:0/2093339630 learned_addr learned my addr 172.21.15.6:0/2093339630 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:25:59.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:59.152+0000 7f35a7e00640 1 --2- 172.21.15.6:0/2093339630 >> v2:172.21.15.6:3300/0 conn(0x7f35a8072220 0x7f35a80c8c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:25:59.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:59.153+0000 7f35a7e00640 1 -- 172.21.15.6:0/2093339630 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f35a8159070 msgr2=0x7f35a8162d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:25:59.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:59.153+0000 7f35a7e00640 1 --2- 172.21.15.6:0/2093339630 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f35a8159070 0x7f35a8162d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:59.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:59.153+0000 7f35a7e00640 1 -- 172.21.15.6:0/2093339630 >> v1:172.21.15.6:6789/0 conn(0x7f35a80c6be0 legacy=0x7f35a806da40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:25:59.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:59.153+0000 7f35a6000640 1 -- 172.21.15.6:0/2093339630 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2607420340 0 0) 0x7f35a8159450 con 0x7f35a80c6be0 2024-04-24T23:25:59.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:59.153+0000 7f35a7e00640 1 -- 172.21.15.6:0/2093339630 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35a80cc2a0 con 0x7f35a8072220 2024-04-24T23:25:59.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:59.155+0000 7f35a7e00640 1 --2- 172.21.15.6:0/2093339630 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f35a8072220 0x7f35a80c8c70 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f35940097a0 tx=0x7f359402a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4791a5cf37f42ad8 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:25:59.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:59.155+0000 7f35a6000640 1 -- 172.21.15.6:0/2093339630 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3594037070 con 0x7f35a8072220 2024-04-24T23:25:59.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:59.155+0000 7f35a6000640 1 -- 172.21.15.6:0/2093339630 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3594032440 con 0x7f35a8072220 2024-04-24T23:25:59.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:59.155+0000 7f35a6000640 1 -- 172.21.15.6:0/2093339630 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f359402ae60 con 0x7f35a8072220 2024-04-24T23:25:59.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:59.156+0000 7f35ad000640 1 -- 172.21.15.6:0/2093339630 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f35a8072220 msgr2=0x7f35a80c8c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:25:59.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:59.156+0000 7f35ad000640 1 --2- 172.21.15.6:0/2093339630 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f35a8072220 0x7f35a80c8c70 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f35940097a0 tx=0x7f359402a110 comp rx=0 tx=0).stop 2024-04-24T23:25:59.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:59.157+0000 7f35ad000640 1 -- 172.21.15.6:0/2093339630 shutdown_connections 2024-04-24T23:25:59.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:59.157+0000 7f35ad000640 1 --2- 172.21.15.6:0/2093339630 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f35a8072220 0x7f35a80c8c70 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:59.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:59.157+0000 7f35ad000640 1 --2- 172.21.15.6:0/2093339630 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f35a8159070 0x7f35a8162d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:59.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:59.157+0000 7f35ad000640 1 -- 172.21.15.6:0/2093339630 >> 172.21.15.6:0/2093339630 conn(0x7f35a806ed80 msgr2=0x7f35a806f190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:25:59.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:59.157+0000 7f35ad000640 1 -- 172.21.15.6:0/2093339630 shutdown_connections 2024-04-24T23:25:59.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:59.157+0000 7f35ad000640 1 -- 172.21.15.6:0/2093339630 wait complete. 2024-04-24T23:25:59.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:59.159+0000 7f35ad000640 1 Processor -- start 2024-04-24T23:25:59.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:59.159+0000 7f35ad000640 1 -- start start 2024-04-24T23:25:59.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:59.161+0000 7f35ad000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f35a8072220 0x7f35a80cb3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:25:59.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:59.161+0000 7f35ad000640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f35a80c6be0 0x7f35a813ee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:25:59.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:59.161+0000 7f35ad000640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f35a8159070 0x7f35a81402e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:25:59.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:59.161+0000 7f35ad000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f35a806cf60 con 0x7f35a8072220 2024-04-24T23:25:59.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:59.161+0000 7f35ad000640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f35a806d5b0 con 0x7f35a8159070 2024-04-24T23:25:59.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:59.161+0000 7f35ad000640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f35a806d240 con 0x7f35a80c6be0 2024-04-24T23:25:59.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:59.162+0000 7f35a7400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f35a8072220 0x7f35a80cb3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:25:59.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:59.162+0000 7f35a6a00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f35a80c6be0 0x7f35a813ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:25:59.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:59.162+0000 7f35a6a00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f35a80c6be0 0x7f35a813ee90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.6:38268/0 (socket says 172.21.15.6:38268) 2024-04-24T23:25:59.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:59.162+0000 7f35a6a00640 1 -- 172.21.15.6:0/597264813 learned_addr learned my addr 172.21.15.6:0/597264813 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:25:59.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:59.162+0000 7f35a7e00640 1 --2- 172.21.15.6:0/597264813 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f35a8159070 0x7f35a81402e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:25:59.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:59.163+0000 7f35a7400640 1 -- 172.21.15.6:0/597264813 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f35a80c6be0 msgr2=0x7f35a813ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:25:59.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:59.163+0000 7f35a7400640 1 --2- 172.21.15.6:0/597264813 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f35a80c6be0 0x7f35a813ee90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:59.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:59.163+0000 7f35a7400640 1 -- 172.21.15.6:0/597264813 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f35a8159070 msgr2=0x7f35a81402e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:25:59.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:59.163+0000 7f35a7400640 1 --2- 172.21.15.6:0/597264813 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f35a8159070 0x7f35a81402e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:59.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:59.164+0000 7f35a7400640 1 -- 172.21.15.6:0/597264813 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35a80cba80 con 0x7f35a8072220 2024-04-24T23:25:59.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:59.164+0000 7f35a7400640 1 --2- 172.21.15.6:0/597264813 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f35a8072220 0x7f35a80cb3a0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f35a000efc0 tx=0x7f35a000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:25:59.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:59.165+0000 7f359f400640 1 -- 172.21.15.6:0/597264813 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35a000cda0 con 0x7f35a8072220 2024-04-24T23:25:59.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:59.165+0000 7f35ad000640 1 -- 172.21.15.6:0/597264813 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35a80cbd60 con 0x7f35a8072220 2024-04-24T23:25:59.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:59.165+0000 7f35ad000640 1 -- 172.21.15.6:0/597264813 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35a813fdc0 con 0x7f35a8072220 2024-04-24T23:25:59.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:59.166+0000 7f359f400640 1 -- 172.21.15.6:0/597264813 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f35a0019070 con 0x7f35a8072220 2024-04-24T23:25:59.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:59.166+0000 7f359f400640 1 -- 172.21.15.6:0/597264813 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35a000cda0 con 0x7f35a8072220 2024-04-24T23:25:59.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:59.167+0000 7f35ad000640 1 -- 172.21.15.6:0/597264813 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3580005290 con 0x7f35a8072220 2024-04-24T23:25:59.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:59.170+0000 7f359f400640 1 -- 172.21.15.6:0/597264813 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f35a0005d10 con 0x7f35a8072220 2024-04-24T23:25:59.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:59.172+0000 7f359f400640 1 --2- 172.21.15.6:0/597264813 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f358806c030 0x7f358806e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:25:59.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:59.172+0000 7f359f400640 1 -- 172.21.15.6:0/597264813 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6934+0+0 (secure 0 0 0) 0x7f35a008b3e0 con 0x7f35a8072220 2024-04-24T23:25:59.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:59.172+0000 7f35a6a00640 1 --2- 172.21.15.6:0/597264813 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f358806c030 0x7f358806e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:25:59.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:59.173+0000 7f35a6a00640 1 --2- 172.21.15.6:0/597264813 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f358806c030 0x7f358806e4f0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f35a80c8790 tx=0x7f3598009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:25:59.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:59.183+0000 7f359f400640 1 -- 172.21.15.6:0/597264813 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f35a0010040 con 0x7f35a8072220 2024-04-24T23:25:59.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:59.702+0000 7f35ad000640 1 -- 172.21.15.6:0/597264813 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3580005580 con 0x7f35a8072220 2024-04-24T23:25:59.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:59.703+0000 7f359f400640 1 -- 172.21.15.6:0/597264813 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6033 (secure 0 0 0) 0x7f35a00591d0 con 0x7f35a8072220 2024-04-24T23:25:59.705 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:25:59.705 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5081,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/1190759626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1190759626},{"type":"v1","addr":"172.21.15.6:6834","nonce":1190759626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":5249,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6838/2416336040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2416336040},{"type":"v1","addr":"172.21.15.190:6838","nonce":2416336040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":42},{"gid":5503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/3928731059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":3928731059},{"type":"v1","addr":"172.21.15.6:6837","nonce":3928731059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:25:42.695130+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5725,"mds_1":5599},"failed":[],"damaged":[],"stopped":[2,3,4],"info":{"gid_5599":{"gid":5599,"name":"d","rank":1,"incarnation":43,"state":"up:active","state_seq":25,"addr":"172.21.15.190:6839/2643233757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":2643233757},{"type":"v1","addr":"172.21.15.190:6839","nonce":2643233757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5725":{"gid":5725,"name":"f","rank":0,"incarnation":49,"state":"up:replay","state_seq":1,"addr":"172.21.15.190:6837/1151527362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":1151527362},{"type":"v1","addr":"172.21.15.190:6837","nonce":1151527362}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5599,"qdb_cluster":[5599]},"id":1}]} 2024-04-24T23:25:59.705 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 49 2024-04-24T23:25:59.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:59.707+0000 7f35ad000640 1 -- 172.21.15.6:0/597264813 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f358806c030 msgr2=0x7f358806e4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:25:59.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:59.707+0000 7f35ad000640 1 --2- 172.21.15.6:0/597264813 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f358806c030 0x7f358806e4f0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f35a80c8790 tx=0x7f3598009040 comp rx=0 tx=0).stop 2024-04-24T23:25:59.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:59.707+0000 7f35ad000640 1 -- 172.21.15.6:0/597264813 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f35a8072220 msgr2=0x7f35a80cb3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:25:59.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:59.707+0000 7f35ad000640 1 --2- 172.21.15.6:0/597264813 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f35a8072220 0x7f35a80cb3a0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f35a000efc0 tx=0x7f35a000c490 comp rx=0 tx=0).stop 2024-04-24T23:25:59.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:59.708+0000 7f35ad000640 1 -- 172.21.15.6:0/597264813 shutdown_connections 2024-04-24T23:25:59.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:59.708+0000 7f35ad000640 1 --2- 172.21.15.6:0/597264813 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f358806c030 0x7f358806e4f0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:59.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:59.708+0000 7f35ad000640 1 --2- 172.21.15.6:0/597264813 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f35a80c6be0 0x7f35a813ee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:59.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:59.708+0000 7f35ad000640 1 --2- 172.21.15.6:0/597264813 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f35a8159070 0x7f35a81402e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:59.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:59.708+0000 7f35ad000640 1 --2- 172.21.15.6:0/597264813 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f35a8072220 0x7f35a80cb3a0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:25:59.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:59.708+0000 7f35ad000640 1 -- 172.21.15.6:0/597264813 >> 172.21.15.6:0/597264813 conn(0x7f35a806ed80 msgr2=0x7f35a8155d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:25:59.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:59.709+0000 7f35ad000640 1 -- 172.21.15.6:0/597264813 shutdown_connections 2024-04-24T23:25:59.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:25:59.709+0000 7f35ad000640 1 -- 172.21.15.6:0/597264813 wait complete. 2024-04-24T23:25:59.753 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 49, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-24T23:15:36.400716+0000", "damaged": [], "data_pools": [ 3, 4 ], "enabled": true, "epoch": 49, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_5599": { "addr": "172.21.15.190:6839/2643233757", "addrs": { "addrvec": [ { "addr": "172.21.15.190:6836", "nonce": 2643233757, "type": "v2" }, { "addr": "172.21.15.190:6839", "nonce": 2643233757, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5599, "incarnation": 43, "join_fscid": -1, "name": "d", "rank": 1, "state": "up:active", "state_seq": 25 }, "gid_5725": { "addr": "172.21.15.190:6837/1151527362", "addrs": { "addrvec": [ { "addr": "172.21.15.190:6834", "nonce": 1151527362, "type": "v2" }, { "addr": "172.21.15.190:6837", "nonce": 1151527362, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5725, "incarnation": 49, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 30, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-04-24T23:25:42.695130+0000", "qdb_cluster": [ 5599 ], "qdb_leader": 5599, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 2, 3, 4 ], "tableserver": 0, "up": { "mds_0": 5725, "mds_1": 5599 } } } ], "standbys": [ { "addr": "172.21.15.6:6834/1190759626", "addrs": { "addrvec": [ { "addr": "172.21.15.6:6832", "nonce": 1190759626, "type": "v2" }, { "addr": "172.21.15.6:6834", "nonce": 1190759626, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 30, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5081, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.190:6838/2416336040", "addrs": { "addrvec": [ { "addr": "172.21.15.190:6835", "nonce": 2416336040, "type": "v2" }, { "addr": "172.21.15.190:6838", "nonce": 2416336040, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 42, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5249, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.6:6837/3928731059", "addrs": { "addrvec": [ { "addr": "172.21.15.6:6836", "nonce": 3928731059, "type": "v2" }, { "addr": "172.21.15.6:6837", "nonce": 3928731059, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 35, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5503, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-04-24T23:26:01.754 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:26:14.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:14.863+0000 7f0fa3400640 1 Processor -- start 2024-04-24T23:26:14.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:14.864+0000 7f0fa3400640 1 -- start start 2024-04-24T23:26:14.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:14.866+0000 7f0fa3400640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f0f9c0c7dc0 0x7f0f9c1623a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:26:14.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:14.867+0000 7f0fa3400640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0f9c0c81c0 0x7f0f9c1628e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:26:14.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:14.867+0000 7f0fa3400640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0f9c0c7230 con 0x7f0f9c159110 2024-04-24T23:26:14.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:14.867+0000 7f0fa3400640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f0f9c1594f0 con 0x7f0f9c0c7dc0 2024-04-24T23:26:14.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:14.867+0000 7f0fa3400640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f0f9c0c74d0 con 0x7f0f9c0c81c0 2024-04-24T23:26:14.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:14.867+0000 7f0fa0a00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f0f9c159110 0x7f0f9c072420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:44680/0 (socket says 172.21.15.6:44680) 2024-04-24T23:26:14.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:14.867+0000 7f0fa0a00640 1 -- 172.21.15.6:0/4258578098 learned_addr learned my addr 172.21.15.6:0/4258578098 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:26:14.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:14.867+0000 7f0f9be00640 1 --2- 172.21.15.6:0/4258578098 >> v2:172.21.15.190:3300/0 conn(0x7f0f9c0c7dc0 0x7f0f9c1623a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:26:14.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:14.867+0000 7f0fa1400640 1 --2- 172.21.15.6:0/4258578098 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0f9c0c81c0 0x7f0f9c1628e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:26:14.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:14.868+0000 7f0fa1400640 1 -- 172.21.15.6:0/4258578098 >> v2:172.21.15.190:3300/0 conn(0x7f0f9c0c7dc0 msgr2=0x7f0f9c1623a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:26:14.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:14.868+0000 7f0fa1400640 1 --2- 172.21.15.6:0/4258578098 >> v2:172.21.15.190:3300/0 conn(0x7f0f9c0c7dc0 0x7f0f9c1623a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:26:14.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:14.868+0000 7f0fa1400640 1 -- 172.21.15.6:0/4258578098 >> v1:172.21.15.6:6789/0 conn(0x7f0f9c159110 legacy=0x7f0f9c072420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:26:14.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:14.868+0000 7f0fa1400640 1 -- 172.21.15.6:0/4258578098 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f9c06e440 con 0x7f0f9c0c81c0 2024-04-24T23:26:14.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:14.870+0000 7f0fa1400640 1 --2- 172.21.15.6:0/4258578098 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0f9c0c81c0 0x7f0f9c1628e0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f0f94009d90 tx=0x7f0f9402a5f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d67631697976b178 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:26:14.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:14.872+0000 7f0f9b400640 1 -- 172.21.15.6:0/4258578098 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f94007740 con 0x7f0f9c0c81c0 2024-04-24T23:26:14.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:14.872+0000 7f0f9b400640 1 -- 172.21.15.6:0/4258578098 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0f940078e0 con 0x7f0f9c0c81c0 2024-04-24T23:26:14.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:14.872+0000 7f0f9b400640 1 -- 172.21.15.6:0/4258578098 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f94007cb0 con 0x7f0f9c0c81c0 2024-04-24T23:26:14.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:14.873+0000 7f0fa3400640 1 -- 172.21.15.6:0/4258578098 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0f9c0c81c0 msgr2=0x7f0f9c1628e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:26:14.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:14.873+0000 7f0fa3400640 1 --2- 172.21.15.6:0/4258578098 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0f9c0c81c0 0x7f0f9c1628e0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f0f94009d90 tx=0x7f0f9402a5f0 comp rx=0 tx=0).stop 2024-04-24T23:26:14.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:14.874+0000 7f0fa3400640 1 -- 172.21.15.6:0/4258578098 shutdown_connections 2024-04-24T23:26:14.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:14.874+0000 7f0fa3400640 1 --2- 172.21.15.6:0/4258578098 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0f9c0c81c0 0x7f0f9c1628e0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:26:14.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:14.874+0000 7f0fa3400640 1 --2- 172.21.15.6:0/4258578098 >> v2:172.21.15.190:3300/0 conn(0x7f0f9c0c7dc0 0x7f0f9c1623a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:26:14.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:14.874+0000 7f0fa3400640 1 -- 172.21.15.6:0/4258578098 >> 172.21.15.6:0/4258578098 conn(0x7f0f9c06b0e0 msgr2=0x7f0f9c06b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:26:14.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:14.874+0000 7f0fa3400640 1 -- 172.21.15.6:0/4258578098 shutdown_connections 2024-04-24T23:26:14.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:14.874+0000 7f0fa3400640 1 -- 172.21.15.6:0/4258578098 wait complete. 2024-04-24T23:26:14.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:14.877+0000 7f0fa3400640 1 Processor -- start 2024-04-24T23:26:14.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:14.877+0000 7f0fa3400640 1 -- start start 2024-04-24T23:26:14.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:14.879+0000 7f0fa3400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0f9c0c7dc0 0x7f0f9c0c8e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:26:14.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:14.879+0000 7f0fa3400640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0f9c0c81c0 0x7f0f9c1f8650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:26:14.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:14.879+0000 7f0fa3400640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0f9c159110 0x7f0f9c1f8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:26:14.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:14.880+0000 7f0fa0a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0f9c0c7dc0 0x7f0f9c0c8e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:26:14.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:14.880+0000 7f0fa1400640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0f9c159110 0x7f0f9c1f8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:26:14.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:14.880+0000 7f0fa0a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0f9c0c7dc0 0x7f0f9c0c8e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:60116/0 (socket says 172.21.15.6:60116) 2024-04-24T23:26:14.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:14.880+0000 7f0fa0a00640 1 -- 172.21.15.6:0/1738560888 learned_addr learned my addr 172.21.15.6:0/1738560888 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:26:14.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:14.880+0000 7f0fa3400640 1 -- 172.21.15.6:0/1738560888 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f0f9c06ece0 con 0x7f0f9c0c7dc0 2024-04-24T23:26:14.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:14.880+0000 7f0fa3400640 1 -- 172.21.15.6:0/1738560888 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f0f9c06ea00 con 0x7f0f9c159110 2024-04-24T23:26:14.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:14.880+0000 7f0fa3400640 1 -- 172.21.15.6:0/1738560888 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f0f9c06f200 con 0x7f0f9c0c81c0 2024-04-24T23:26:14.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:14.880+0000 7f0f9be00640 1 --2- 172.21.15.6:0/1738560888 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0f9c0c81c0 0x7f0f9c1f8650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:26:14.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:14.881+0000 7f0f9be00640 1 -- 172.21.15.6:0/1738560888 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0f9c159110 msgr2=0x7f0f9c1f8b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:26:14.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:14.881+0000 7f0f9be00640 1 --2- 172.21.15.6:0/1738560888 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0f9c159110 0x7f0f9c1f8b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:26:14.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:14.881+0000 7f0f9be00640 1 -- 172.21.15.6:0/1738560888 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0f9c0c7dc0 msgr2=0x7f0f9c0c8e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:26:14.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:14.881+0000 7f0f9be00640 1 --2- 172.21.15.6:0/1738560888 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0f9c0c7dc0 0x7f0f9c0c8e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:26:14.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:14.881+0000 7f0f9be00640 1 -- 172.21.15.6:0/1738560888 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f9c15ff70 con 0x7f0f9c0c81c0 2024-04-24T23:26:14.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:14.881+0000 7f0f9be00640 1 --2- 172.21.15.6:0/1738560888 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0f9c0c81c0 0x7f0f9c1f8650 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f0f8800e850 tx=0x7f0f8800ed10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:26:14.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:14.882+0000 7f0f98c00640 1 -- 172.21.15.6:0/1738560888 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f88017070 con 0x7f0f9c0c81c0 2024-04-24T23:26:14.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:14.882+0000 7f0f98c00640 1 -- 172.21.15.6:0/1738560888 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0f8800c940 con 0x7f0f9c0c81c0 2024-04-24T23:26:14.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:14.882+0000 7f0f98c00640 1 -- 172.21.15.6:0/1738560888 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f8800cd10 con 0x7f0f9c0c81c0 2024-04-24T23:26:14.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:14.882+0000 7f0fa3400640 1 -- 172.21.15.6:0/1738560888 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f9c1f91b0 con 0x7f0f9c0c81c0 2024-04-24T23:26:14.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:14.882+0000 7f0fa3400640 1 -- 172.21.15.6:0/1738560888 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f9c162a70 con 0x7f0f9c0c81c0 2024-04-24T23:26:14.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:14.885+0000 7f0fa3400640 1 -- 172.21.15.6:0/1738560888 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0f9c0767b0 con 0x7f0f9c0c81c0 2024-04-24T23:26:14.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:14.885+0000 7f0f98c00640 1 -- 172.21.15.6:0/1738560888 <== mon.2 v2:172.21.15.190:3301/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f0f8801a070 con 0x7f0f9c0c81c0 2024-04-24T23:26:14.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:14.886+0000 7f0f98c00640 1 --2- 172.21.15.6:0/1738560888 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f0f7c06bf30 0x7f0f7c06e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:26:14.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:14.887+0000 7f0f98c00640 1 -- 172.21.15.6:0/1738560888 <== mon.2 v2:172.21.15.190:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6934+0+0 (secure 0 0 0) 0x7f0f8808b4e0 con 0x7f0f9c0c81c0 2024-04-24T23:26:14.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:14.887+0000 7f0fa0a00640 1 --2- 172.21.15.6:0/1738560888 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f0f7c06bf30 0x7f0f7c06e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:26:14.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:14.890+0000 7f0fa0a00640 1 --2- 172.21.15.6:0/1738560888 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f0f7c06bf30 0x7f0f7c06e3f0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f0f8c000f60 tx=0x7f0f8c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:26:14.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:14.894+0000 7f0f98c00640 1 -- 172.21.15.6:0/1738560888 <== mon.2 v2:172.21.15.190:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f0f880593e0 con 0x7f0f9c0c81c0 2024-04-24T23:26:15.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:15.394+0000 7f0fa3400640 1 -- 172.21.15.6:0/1738560888 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0f9c162c40 con 0x7f0f9c0c81c0 2024-04-24T23:26:15.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:15.395+0000 7f0f98c00640 1 -- 172.21.15.6:0/1738560888 <== mon.2 v2:172.21.15.190:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6033 (secure 0 0 0) 0x7f0f8805d6e0 con 0x7f0f9c0c81c0 2024-04-24T23:26:15.399 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:26:15.399 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5081,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/1190759626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1190759626},{"type":"v1","addr":"172.21.15.6:6834","nonce":1190759626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":5249,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6838/2416336040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2416336040},{"type":"v1","addr":"172.21.15.190:6838","nonce":2416336040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":42},{"gid":5503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/3928731059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":3928731059},{"type":"v1","addr":"172.21.15.6:6837","nonce":3928731059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:25:42.695130+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5725,"mds_1":5599},"failed":[],"damaged":[],"stopped":[2,3,4],"info":{"gid_5599":{"gid":5599,"name":"d","rank":1,"incarnation":43,"state":"up:active","state_seq":25,"addr":"172.21.15.190:6839/2643233757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":2643233757},{"type":"v1","addr":"172.21.15.190:6839","nonce":2643233757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5725":{"gid":5725,"name":"f","rank":0,"incarnation":49,"state":"up:replay","state_seq":1,"addr":"172.21.15.190:6837/1151527362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":1151527362},{"type":"v1","addr":"172.21.15.190:6837","nonce":1151527362}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5599,"qdb_cluster":[5599]},"id":1}]} 2024-04-24T23:26:15.399 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 49 2024-04-24T23:26:15.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:15.402+0000 7f0fa3400640 1 -- 172.21.15.6:0/1738560888 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f0f7c06bf30 msgr2=0x7f0f7c06e3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:26:15.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:15.402+0000 7f0fa3400640 1 --2- 172.21.15.6:0/1738560888 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f0f7c06bf30 0x7f0f7c06e3f0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f0f8c000f60 tx=0x7f0f8c009040 comp rx=0 tx=0).stop 2024-04-24T23:26:15.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:15.402+0000 7f0fa3400640 1 -- 172.21.15.6:0/1738560888 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0f9c0c81c0 msgr2=0x7f0f9c1f8650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:26:15.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:15.402+0000 7f0fa3400640 1 --2- 172.21.15.6:0/1738560888 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0f9c0c81c0 0x7f0f9c1f8650 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f0f8800e850 tx=0x7f0f8800ed10 comp rx=0 tx=0).stop 2024-04-24T23:26:15.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:15.403+0000 7f0fa3400640 1 -- 172.21.15.6:0/1738560888 shutdown_connections 2024-04-24T23:26:15.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:15.403+0000 7f0fa3400640 1 --2- 172.21.15.6:0/1738560888 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f0f7c06bf30 0x7f0f7c06e3f0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:26:15.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:15.403+0000 7f0fa3400640 1 --2- 172.21.15.6:0/1738560888 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0f9c0c81c0 0x7f0f9c1f8650 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:26:15.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:15.403+0000 7f0fa3400640 1 --2- 172.21.15.6:0/1738560888 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0f9c159110 0x7f0f9c1f8b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:26:15.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:15.403+0000 7f0fa3400640 1 --2- 172.21.15.6:0/1738560888 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0f9c0c7dc0 0x7f0f9c0c8e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:26:15.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:15.403+0000 7f0fa3400640 1 -- 172.21.15.6:0/1738560888 >> 172.21.15.6:0/1738560888 conn(0x7f0f9c06b0e0 msgr2=0x7f0f9c157440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:26:15.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:15.404+0000 7f0fa3400640 1 -- 172.21.15.6:0/1738560888 shutdown_connections 2024-04-24T23:26:15.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:15.404+0000 7f0fa3400640 1 -- 172.21.15.6:0/1738560888 wait complete. 2024-04-24T23:26:15.782 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-24T23:26:17.783 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:26:18.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:18.241+0000 7f52d2a00640 1 Processor -- start 2024-04-24T23:26:18.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:18.243+0000 7f52d2a00640 1 -- start start 2024-04-24T23:26:18.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:18.245+0000 7f52d2a00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f52cc0c63c0 0x7f52cc070180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:26:18.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:18.245+0000 7f52d2a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f52cc0c46f0 0x7f52cc157a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:26:18.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:18.245+0000 7f52d2a00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f52cc0c9ac0 con 0x7f52cc0c67a0 2024-04-24T23:26:18.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:18.245+0000 7f52d2a00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f52cc15bb10 con 0x7f52cc0c46f0 2024-04-24T23:26:18.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:18.245+0000 7f52d2a00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f52cc15af00 con 0x7f52cc0c63c0 2024-04-24T23:26:18.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:18.247+0000 7f52cbe00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f52cc0c63c0 0x7f52cc070180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:26:18.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:18.247+0000 7f52cbe00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f52cc0c63c0 0x7f52cc070180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.6:59930/0 (socket says 172.21.15.6:59930) 2024-04-24T23:26:18.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:18.247+0000 7f52cbe00640 1 -- 172.21.15.6:0/1887610115 learned_addr learned my addr 172.21.15.6:0/1887610115 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:26:18.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:18.247+0000 7f52d0a00640 1 --2- 172.21.15.6:0/1887610115 >> v2:172.21.15.6:3300/0 conn(0x7f52cc0c46f0 0x7f52cc157a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:26:18.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:18.248+0000 7f52cbe00640 1 -- 172.21.15.6:0/1887610115 >> v2:172.21.15.6:3300/0 conn(0x7f52cc0c46f0 msgr2=0x7f52cc157a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:26:18.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:18.248+0000 7f52cbe00640 1 --2- 172.21.15.6:0/1887610115 >> v2:172.21.15.6:3300/0 conn(0x7f52cc0c46f0 0x7f52cc157a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:26:18.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:18.248+0000 7f52cbe00640 1 -- 172.21.15.6:0/1887610115 >> v1:172.21.15.6:6789/0 conn(0x7f52cc0c67a0 legacy=0x7f52cc1516a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:26:18.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:18.248+0000 7f52cbe00640 1 -- 172.21.15.6:0/1887610115 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52cc15b1e0 con 0x7f52cc0c63c0 2024-04-24T23:26:18.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:18.255+0000 7f52cbe00640 1 --2- 172.21.15.6:0/1887610115 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f52cc0c63c0 0x7f52cc070180 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f52c000bb10 tx=0x7f52c002c9b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c6bf2e7bc80ffe8d server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:26:18.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:18.262+0000 7f52caa00640 1 -- 172.21.15.6:0/1887610115 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52c0035070 con 0x7f52cc0c63c0 2024-04-24T23:26:18.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:18.262+0000 7f52caa00640 1 -- 172.21.15.6:0/1887610115 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f52c0031610 con 0x7f52cc0c63c0 2024-04-24T23:26:18.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:18.262+0000 7f52caa00640 1 -- 172.21.15.6:0/1887610115 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52c00319e0 con 0x7f52cc0c63c0 2024-04-24T23:26:18.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:18.264+0000 7f52d2a00640 1 -- 172.21.15.6:0/1887610115 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f52cc0c63c0 msgr2=0x7f52cc070180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:26:18.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:18.264+0000 7f52d2a00640 1 --2- 172.21.15.6:0/1887610115 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f52cc0c63c0 0x7f52cc070180 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f52c000bb10 tx=0x7f52c002c9b0 comp rx=0 tx=0).stop 2024-04-24T23:26:18.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:18.264+0000 7f52d2a00640 1 -- 172.21.15.6:0/1887610115 shutdown_connections 2024-04-24T23:26:18.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:18.265+0000 7f52d2a00640 1 --2- 172.21.15.6:0/1887610115 >> v2:172.21.15.6:3300/0 conn(0x7f52cc0c46f0 0x7f52cc157a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:26:18.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:18.265+0000 7f52d2a00640 1 --2- 172.21.15.6:0/1887610115 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f52cc0c63c0 0x7f52cc070180 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:26:18.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:18.265+0000 7f52d2a00640 1 -- 172.21.15.6:0/1887610115 >> 172.21.15.6:0/1887610115 conn(0x7f52cc0cd1f0 msgr2=0x7f52cc0cd600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:26:18.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:18.265+0000 7f52d2a00640 1 -- 172.21.15.6:0/1887610115 shutdown_connections 2024-04-24T23:26:18.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:18.265+0000 7f52d2a00640 1 -- 172.21.15.6:0/1887610115 wait complete. 2024-04-24T23:26:18.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:18.272+0000 7f52d2a00640 1 Processor -- start 2024-04-24T23:26:18.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:18.276+0000 7f52d2a00640 1 -- start start 2024-04-24T23:26:18.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:18.281+0000 7f52d2a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f52cc0c46f0 0x7f52cc0caff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:26:18.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:18.282+0000 7f52d2a00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f52cc0c63c0 0x7f52cc0748a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:26:18.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:18.282+0000 7f52d2a00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f52cc0c67a0 0x7f52cc155900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:26:18.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:18.283+0000 7f52d0a00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f52cc0c67a0 0x7f52cc155900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:26:18.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:18.283+0000 7f52d0a00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f52cc0c67a0 0x7f52cc155900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:51960/0 (socket says 172.21.15.6:51960) 2024-04-24T23:26:18.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:18.283+0000 7f52d0a00640 1 -- 172.21.15.6:0/759871976 learned_addr learned my addr 172.21.15.6:0/759871976 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:26:18.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:18.283+0000 7f52d2a00640 1 -- 172.21.15.6:0/759871976 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f52cc0c74e0 con 0x7f52cc0c46f0 2024-04-24T23:26:18.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:18.283+0000 7f52d2a00640 1 -- 172.21.15.6:0/759871976 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f52cc15b7a0 con 0x7f52cc0c67a0 2024-04-24T23:26:18.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:18.283+0000 7f52d2a00640 1 -- 172.21.15.6:0/759871976 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f52cc0c7660 con 0x7f52cc0c63c0 2024-04-24T23:26:18.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:18.283+0000 7f52cb400640 1 --2- 172.21.15.6:0/759871976 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f52cc0c63c0 0x7f52cc0748a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:26:18.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:18.283+0000 7f52cbe00640 1 --2- 172.21.15.6:0/759871976 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f52cc0c46f0 0x7f52cc0caff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:26:18.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:18.285+0000 7f52d0a00640 1 -- 172.21.15.6:0/759871976 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f52cc0c63c0 msgr2=0x7f52cc0748a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:26:18.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:18.285+0000 7f52d0a00640 1 --2- 172.21.15.6:0/759871976 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f52cc0c63c0 0x7f52cc0748a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:26:18.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:18.285+0000 7f52d0a00640 1 -- 172.21.15.6:0/759871976 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f52cc0c46f0 msgr2=0x7f52cc0caff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:26:18.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:18.285+0000 7f52d0a00640 1 --2- 172.21.15.6:0/759871976 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f52cc0c46f0 0x7f52cc0caff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:26:18.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:18.285+0000 7f52d0a00640 1 -- 172.21.15.6:0/759871976 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52cc148cd0 con 0x7f52cc0c67a0 2024-04-24T23:26:18.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:18.285+0000 7f52cbe00640 1 --2- 172.21.15.6:0/759871976 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f52cc0c46f0 0x7f52cc0caff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:26:18.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:18.286+0000 7f52d0a00640 1 --2- 172.21.15.6:0/759871976 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f52cc0c67a0 0x7f52cc155900 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f52c4007e40 tx=0x7f52c400e4b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:26:18.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:18.289+0000 7f52bfe00640 1 -- 172.21.15.6:0/759871976 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52c4014620 con 0x7f52cc0c67a0 2024-04-24T23:26:18.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:18.290+0000 7f52d2a00640 1 -- 172.21.15.6:0/759871976 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52cc0cb530 con 0x7f52cc0c67a0 2024-04-24T23:26:18.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:18.290+0000 7f52d2a00640 1 -- 172.21.15.6:0/759871976 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52cc1483a0 con 0x7f52cc0c67a0 2024-04-24T23:26:18.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:18.290+0000 7f52bfe00640 1 -- 172.21.15.6:0/759871976 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f52c4019070 con 0x7f52cc0c67a0 2024-04-24T23:26:18.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:18.290+0000 7f52bfe00640 1 -- 172.21.15.6:0/759871976 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52c4014620 con 0x7f52cc0c67a0 2024-04-24T23:26:18.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:18.300+0000 7f52bd600640 1 -- 172.21.15.6:0/759871976 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f52cc075b30 con 0x7f52cc0c67a0 2024-04-24T23:26:18.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:18.300+0000 7f52bfe00640 1 -- 172.21.15.6:0/759871976 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f52c40147c0 con 0x7f52cc0c67a0 2024-04-24T23:26:18.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:18.304+0000 7f52bfe00640 1 --2- 172.21.15.6:0/759871976 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f52ac06c030 0x7f52ac06e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:26:18.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:18.304+0000 7f52bfe00640 1 -- 172.21.15.6:0/759871976 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6934+0+0 (secure 0 0 0) 0x7f52c408b940 con 0x7f52cc0c67a0 2024-04-24T23:26:18.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:18.305+0000 7f52cbe00640 1 --2- 172.21.15.6:0/759871976 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f52ac06c030 0x7f52ac06e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:26:18.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:18.314+0000 7f52cbe00640 1 --2- 172.21.15.6:0/759871976 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f52ac06c030 0x7f52ac06e4f0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f52c0037000 tx=0x7f52c0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:26:18.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:18.314+0000 7f52bfe00640 1 -- 172.21.15.6:0/759871976 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f52c4059cf0 con 0x7f52cc0c67a0 2024-04-24T23:26:18.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:18.821+0000 7f52bd600640 1 -- 172.21.15.6:0/759871976 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f52cc155660 con 0x7f52cc0c67a0 2024-04-24T23:26:18.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:18.823+0000 7f52bfe00640 1 -- 172.21.15.6:0/759871976 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6033 (secure 0 0 0) 0x7f52c40595f0 con 0x7f52cc0c67a0 2024-04-24T23:26:18.826 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:26:18.826 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5081,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/1190759626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1190759626},{"type":"v1","addr":"172.21.15.6:6834","nonce":1190759626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":5249,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6838/2416336040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2416336040},{"type":"v1","addr":"172.21.15.190:6838","nonce":2416336040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":42},{"gid":5503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/3928731059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":3928731059},{"type":"v1","addr":"172.21.15.6:6837","nonce":3928731059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:25:42.695130+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5725,"mds_1":5599},"failed":[],"damaged":[],"stopped":[2,3,4],"info":{"gid_5599":{"gid":5599,"name":"d","rank":1,"incarnation":43,"state":"up:active","state_seq":25,"addr":"172.21.15.190:6839/2643233757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":2643233757},{"type":"v1","addr":"172.21.15.190:6839","nonce":2643233757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5725":{"gid":5725,"name":"f","rank":0,"incarnation":49,"state":"up:replay","state_seq":1,"addr":"172.21.15.190:6837/1151527362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":1151527362},{"type":"v1","addr":"172.21.15.190:6837","nonce":1151527362}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5599,"qdb_cluster":[5599]},"id":1}]} 2024-04-24T23:26:18.826 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 49 2024-04-24T23:26:18.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:18.832+0000 7f52bd600640 1 -- 172.21.15.6:0/759871976 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f52ac06c030 msgr2=0x7f52ac06e4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:26:18.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:18.832+0000 7f52bd600640 1 --2- 172.21.15.6:0/759871976 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f52ac06c030 0x7f52ac06e4f0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f52c0037000 tx=0x7f52c0008040 comp rx=0 tx=0).stop 2024-04-24T23:26:18.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:18.833+0000 7f52bd600640 1 -- 172.21.15.6:0/759871976 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f52cc0c67a0 msgr2=0x7f52cc155900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:26:18.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:18.833+0000 7f52bd600640 1 --2- 172.21.15.6:0/759871976 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f52cc0c67a0 0x7f52cc155900 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f52c4007e40 tx=0x7f52c400e4b0 comp rx=0 tx=0).stop 2024-04-24T23:26:18.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:18.833+0000 7f52bd600640 1 -- 172.21.15.6:0/759871976 shutdown_connections 2024-04-24T23:26:18.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:18.833+0000 7f52bd600640 1 --2- 172.21.15.6:0/759871976 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f52ac06c030 0x7f52ac06e4f0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:26:18.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:18.833+0000 7f52bd600640 1 --2- 172.21.15.6:0/759871976 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f52cc0c63c0 0x7f52cc0748a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:26:18.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:18.834+0000 7f52bd600640 1 --2- 172.21.15.6:0/759871976 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f52cc0c67a0 0x7f52cc155900 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:26:18.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:18.834+0000 7f52bd600640 1 --2- 172.21.15.6:0/759871976 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f52cc0c46f0 0x7f52cc0caff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:26:18.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:18.834+0000 7f52bd600640 1 -- 172.21.15.6:0/759871976 >> 172.21.15.6:0/759871976 conn(0x7f52cc0cd1f0 msgr2=0x7f52cc06b0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:26:18.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:18.834+0000 7f52bd600640 1 -- 172.21.15.6:0/759871976 shutdown_connections 2024-04-24T23:26:18.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:18.834+0000 7f52bd600640 1 -- 172.21.15.6:0/759871976 wait complete. 2024-04-24T23:26:18.867 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-24T23:26:20.868 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:26:21.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:21.165+0000 7f7b6a200640 1 Processor -- start 2024-04-24T23:26:21.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:21.165+0000 7f7b6a200640 1 -- start start 2024-04-24T23:26:21.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:21.166+0000 7f7b6a200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f7b64159110 0x7f7b64072420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:26:21.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:21.166+0000 7f7b6a200640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7b640c7dc0 0x7f7b641623a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:26:21.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:21.166+0000 7f7b6a200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7b640c7230 con 0x7f7b640c81c0 2024-04-24T23:26:21.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:21.167+0000 7f7b6a200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f7b641594f0 con 0x7f7b64159110 2024-04-24T23:26:21.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:21.167+0000 7f7b6a200640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f7b640c74d0 con 0x7f7b640c7dc0 2024-04-24T23:26:21.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:21.167+0000 7f7b63400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f7b64159110 0x7f7b64072420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:26:21.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:21.167+0000 7f7b63400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f7b64159110 0x7f7b64072420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:60132/0 (socket says 172.21.15.6:60132) 2024-04-24T23:26:21.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:21.167+0000 7f7b63400640 1 -- 172.21.15.6:0/2489594224 learned_addr learned my addr 172.21.15.6:0/2489594224 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:26:21.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:21.167+0000 7f7b62a00640 1 --2- 172.21.15.6:0/2489594224 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7b640c7dc0 0x7f7b641623a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:26:21.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:21.168+0000 7f7b62a00640 1 -- 172.21.15.6:0/2489594224 >> v2:172.21.15.6:3300/0 conn(0x7f7b64159110 msgr2=0x7f7b64072420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:26:21.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:21.168+0000 7f7b62a00640 1 --2- 172.21.15.6:0/2489594224 >> v2:172.21.15.6:3300/0 conn(0x7f7b64159110 0x7f7b64072420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:26:21.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:21.168+0000 7f7b62a00640 1 -- 172.21.15.6:0/2489594224 >> v1:172.21.15.6:6789/0 conn(0x7f7b640c81c0 legacy=0x7f7b641628e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:26:21.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:21.168+0000 7f7b62a00640 1 -- 172.21.15.6:0/2489594224 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b6406e440 con 0x7f7b640c7dc0 2024-04-24T23:26:21.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:21.171+0000 7f7b62a00640 1 --2- 172.21.15.6:0/2489594224 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7b640c7dc0 0x7f7b641623a0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f7b500097a0 tx=0x7f7b5002a250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=16533ae60b331109 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:26:21.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:21.172+0000 7f7b62000640 1 -- 172.21.15.6:0/2489594224 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b50037070 con 0x7f7b640c7dc0 2024-04-24T23:26:21.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:21.172+0000 7f7b62000640 1 -- 172.21.15.6:0/2489594224 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b50032440 con 0x7f7b640c7dc0 2024-04-24T23:26:21.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:21.172+0000 7f7b62000640 1 -- 172.21.15.6:0/2489594224 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b50032810 con 0x7f7b640c7dc0 2024-04-24T23:26:21.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:21.173+0000 7f7b6a200640 1 -- 172.21.15.6:0/2489594224 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7b640c7dc0 msgr2=0x7f7b641623a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:26:21.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:21.173+0000 7f7b6a200640 1 --2- 172.21.15.6:0/2489594224 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7b640c7dc0 0x7f7b641623a0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f7b500097a0 tx=0x7f7b5002a250 comp rx=0 tx=0).stop 2024-04-24T23:26:21.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:21.173+0000 7f7b6a200640 1 -- 172.21.15.6:0/2489594224 shutdown_connections 2024-04-24T23:26:21.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:21.173+0000 7f7b6a200640 1 --2- 172.21.15.6:0/2489594224 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7b640c7dc0 0x7f7b641623a0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:26:21.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:21.173+0000 7f7b6a200640 1 --2- 172.21.15.6:0/2489594224 >> v2:172.21.15.6:3300/0 conn(0x7f7b64159110 0x7f7b64072420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:26:21.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:21.173+0000 7f7b6a200640 1 -- 172.21.15.6:0/2489594224 >> 172.21.15.6:0/2489594224 conn(0x7f7b6406b0e0 msgr2=0x7f7b6406b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:26:21.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:21.174+0000 7f7b6a200640 1 -- 172.21.15.6:0/2489594224 shutdown_connections 2024-04-24T23:26:21.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:21.174+0000 7f7b6a200640 1 -- 172.21.15.6:0/2489594224 wait complete. 2024-04-24T23:26:21.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:21.392+0000 7f7b6a200640 1 Processor -- start 2024-04-24T23:26:21.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:21.396+0000 7f7b6a200640 1 -- start start 2024-04-24T23:26:21.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:21.397+0000 7f7b6a200640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7b640c7dc0 0x7f7b640c4130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:26:21.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:21.397+0000 7f7b6a200640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7b640c81c0 0x7f7b6415f640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:26:21.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:21.397+0000 7f7b6a200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7b641ed430 0x7f7b641614b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:26:21.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:21.397+0000 7f7b6a200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f7b6406ece0 con 0x7f7b641ed430 2024-04-24T23:26:21.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:21.397+0000 7f7b6a200640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f7b6406ea00 con 0x7f7b640c81c0 2024-04-24T23:26:21.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:21.397+0000 7f7b6a200640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f7b64072960 con 0x7f7b640c7dc0 2024-04-24T23:26:21.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:21.397+0000 7f7b63400640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7b640c7dc0 0x7f7b640c4130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:26:21.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:21.397+0000 7f7b63400640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7b640c7dc0 0x7f7b640c4130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.6:59968/0 (socket says 172.21.15.6:59968) 2024-04-24T23:26:21.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:21.398+0000 7f7b63400640 1 -- 172.21.15.6:0/3207476767 learned_addr learned my addr 172.21.15.6:0/3207476767 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:26:21.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:21.398+0000 7f7b62a00640 1 --2- 172.21.15.6:0/3207476767 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7b640c81c0 0x7f7b6415f640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:26:21.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:21.398+0000 7f7b63400640 1 -- 172.21.15.6:0/3207476767 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7b640c81c0 msgr2=0x7f7b6415f640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:26:21.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:21.398+0000 7f7b63400640 1 --2- 172.21.15.6:0/3207476767 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7b640c81c0 0x7f7b6415f640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:26:21.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:21.398+0000 7f7b63400640 1 -- 172.21.15.6:0/3207476767 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7b641ed430 msgr2=0x7f7b641614b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:26:21.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:21.398+0000 7f7b63400640 1 --2- 172.21.15.6:0/3207476767 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7b641ed430 0x7f7b641614b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:26:21.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:21.399+0000 7f7b63400640 1 -- 172.21.15.6:0/3207476767 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b641ec990 con 0x7f7b640c7dc0 2024-04-24T23:26:21.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:21.399+0000 7f7b63400640 1 --2- 172.21.15.6:0/3207476767 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7b640c7dc0 0x7f7b640c4130 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f7b5400e5b0 tx=0x7f7b5400ea70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:26:21.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:21.400+0000 7f7b5b400640 1 -- 172.21.15.6:0/3207476767 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b54017070 con 0x7f7b640c7dc0 2024-04-24T23:26:21.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:21.400+0000 7f7b5b400640 1 -- 172.21.15.6:0/3207476767 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b54013610 con 0x7f7b640c7dc0 2024-04-24T23:26:21.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:21.400+0000 7f7b5b400640 1 -- 172.21.15.6:0/3207476767 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b5400f040 con 0x7f7b640c7dc0 2024-04-24T23:26:21.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:21.401+0000 7f7b6a200640 1 -- 172.21.15.6:0/3207476767 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b64162480 con 0x7f7b640c7dc0 2024-04-24T23:26:21.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:21.401+0000 7f7b6a200640 1 -- 172.21.15.6:0/3207476767 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b641ecc30 con 0x7f7b640c7dc0 2024-04-24T23:26:21.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:21.411+0000 7f7b6a200640 1 -- 172.21.15.6:0/3207476767 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7b640767b0 con 0x7f7b640c7dc0 2024-04-24T23:26:21.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:21.416+0000 7f7b5b400640 1 -- 172.21.15.6:0/3207476767 <== mon.2 v2:172.21.15.190:3301/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f7b5401a070 con 0x7f7b640c7dc0 2024-04-24T23:26:21.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:21.417+0000 7f7b5b400640 1 --2- 172.21.15.6:0/3207476767 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f7b4406be60 0x7f7b4406e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:26:21.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:21.418+0000 7f7b5b400640 1 -- 172.21.15.6:0/3207476767 <== mon.2 v2:172.21.15.190:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6934+0+0 (secure 0 0 0) 0x7f7b5408ae30 con 0x7f7b640c7dc0 2024-04-24T23:26:21.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:21.422+0000 7f7b5b400640 1 -- 172.21.15.6:0/3207476767 <== mon.2 v2:172.21.15.190:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f7b54059310 con 0x7f7b640c7dc0 2024-04-24T23:26:21.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:21.422+0000 7f7b62a00640 1 --2- 172.21.15.6:0/3207476767 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f7b4406be60 0x7f7b4406e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:26:21.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:21.423+0000 7f7b62a00640 1 --2- 172.21.15.6:0/3207476767 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f7b4406be60 0x7f7b4406e320 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f7b50034000 tx=0x7f7b5002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:26:21.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:21.858+0000 7f7b6a200640 1 -- 172.21.15.6:0/3207476767 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7b640c7970 con 0x7f7b640c7dc0 2024-04-24T23:26:21.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:21.861+0000 7f7b5b400640 1 -- 172.21.15.6:0/3207476767 <== mon.2 v2:172.21.15.190:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6033 (secure 0 0 0) 0x7f7b54058c10 con 0x7f7b640c7dc0 2024-04-24T23:26:21.868 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:26:21.868 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5081,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/1190759626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1190759626},{"type":"v1","addr":"172.21.15.6:6834","nonce":1190759626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":5249,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6838/2416336040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2416336040},{"type":"v1","addr":"172.21.15.190:6838","nonce":2416336040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":42},{"gid":5503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/3928731059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":3928731059},{"type":"v1","addr":"172.21.15.6:6837","nonce":3928731059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:25:42.695130+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5725,"mds_1":5599},"failed":[],"damaged":[],"stopped":[2,3,4],"info":{"gid_5599":{"gid":5599,"name":"d","rank":1,"incarnation":43,"state":"up:active","state_seq":25,"addr":"172.21.15.190:6839/2643233757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":2643233757},{"type":"v1","addr":"172.21.15.190:6839","nonce":2643233757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5725":{"gid":5725,"name":"f","rank":0,"incarnation":49,"state":"up:replay","state_seq":1,"addr":"172.21.15.190:6837/1151527362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":1151527362},{"type":"v1","addr":"172.21.15.190:6837","nonce":1151527362}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5599,"qdb_cluster":[5599]},"id":1}]} 2024-04-24T23:26:21.868 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 49 2024-04-24T23:26:21.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:21.868+0000 7f7b6a200640 1 -- 172.21.15.6:0/3207476767 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f7b4406be60 msgr2=0x7f7b4406e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:26:21.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:21.868+0000 7f7b6a200640 1 --2- 172.21.15.6:0/3207476767 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f7b4406be60 0x7f7b4406e320 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f7b50034000 tx=0x7f7b5002e000 comp rx=0 tx=0).stop 2024-04-24T23:26:21.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:21.868+0000 7f7b6a200640 1 -- 172.21.15.6:0/3207476767 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7b640c7dc0 msgr2=0x7f7b640c4130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:26:21.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:21.868+0000 7f7b6a200640 1 --2- 172.21.15.6:0/3207476767 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7b640c7dc0 0x7f7b640c4130 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f7b5400e5b0 tx=0x7f7b5400ea70 comp rx=0 tx=0).stop 2024-04-24T23:26:21.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:21.869+0000 7f7b6a200640 1 -- 172.21.15.6:0/3207476767 shutdown_connections 2024-04-24T23:26:21.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:21.869+0000 7f7b6a200640 1 --2- 172.21.15.6:0/3207476767 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f7b4406be60 0x7f7b4406e320 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:26:21.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:21.869+0000 7f7b6a200640 1 --2- 172.21.15.6:0/3207476767 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7b641ed430 0x7f7b641614b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:26:21.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:21.869+0000 7f7b6a200640 1 --2- 172.21.15.6:0/3207476767 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7b640c81c0 0x7f7b6415f640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:26:21.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:21.869+0000 7f7b6a200640 1 --2- 172.21.15.6:0/3207476767 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7b640c7dc0 0x7f7b640c4130 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:26:21.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:21.869+0000 7f7b6a200640 1 -- 172.21.15.6:0/3207476767 >> 172.21.15.6:0/3207476767 conn(0x7f7b6406b0e0 msgr2=0x7f7b64153620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:26:21.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:21.869+0000 7f7b6a200640 1 -- 172.21.15.6:0/3207476767 shutdown_connections 2024-04-24T23:26:21.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:21.869+0000 7f7b6a200640 1 -- 172.21.15.6:0/3207476767 wait complete. 2024-04-24T23:26:21.898 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-24T23:26:23.899 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:26:24.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:24.199+0000 7fa621400640 1 Processor -- start 2024-04-24T23:26:24.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:24.199+0000 7fa621400640 1 -- start start 2024-04-24T23:26:24.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:24.202+0000 7fa621400640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa61c159110 0x7fa61c072420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:26:24.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:24.202+0000 7fa621400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa61c0c7dc0 0x7fa61c1623a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:26:24.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:24.202+0000 7fa621400640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa61c0c7230 con 0x7fa61c0c81c0 2024-04-24T23:26:24.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:24.202+0000 7fa621400640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7fa61c1594f0 con 0x7fa61c0c7dc0 2024-04-24T23:26:24.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:24.202+0000 7fa621400640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fa61c0c74d0 con 0x7fa61c159110 2024-04-24T23:26:24.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:24.202+0000 7fa61b200640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7fa61c0c81c0 0x7fa61c1628e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.6:41600/0 (socket says 172.21.15.6:41600) 2024-04-24T23:26:24.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:24.202+0000 7fa61b200640 1 -- 172.21.15.6:0/1506300368 learned_addr learned my addr 172.21.15.6:0/1506300368 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:26:24.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:24.203+0000 7fa61a800640 1 --2- 172.21.15.6:0/1506300368 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa61c159110 0x7fa61c072420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:26:24.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:24.203+0000 7fa619e00640 1 --2- 172.21.15.6:0/1506300368 >> v2:172.21.15.6:3300/0 conn(0x7fa61c0c7dc0 0x7fa61c1623a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:26:24.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:24.204+0000 7fa61a800640 1 -- 172.21.15.6:0/1506300368 >> v2:172.21.15.6:3300/0 conn(0x7fa61c0c7dc0 msgr2=0x7fa61c1623a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:26:24.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:24.204+0000 7fa61a800640 1 --2- 172.21.15.6:0/1506300368 >> v2:172.21.15.6:3300/0 conn(0x7fa61c0c7dc0 0x7fa61c1623a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:26:24.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:24.204+0000 7fa61a800640 1 -- 172.21.15.6:0/1506300368 >> v1:172.21.15.190:6789/0 conn(0x7fa61c0c81c0 legacy=0x7fa61c1628e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:26:24.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:24.204+0000 7fa61a800640 1 -- 172.21.15.6:0/1506300368 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa61c06e440 con 0x7fa61c159110 2024-04-24T23:26:24.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:24.207+0000 7fa61a800640 1 --2- 172.21.15.6:0/1506300368 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa61c159110 0x7fa61c072420 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fa6040097a0 tx=0x7fa60402a250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1c17ed4d5d599f5f server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:26:24.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:24.209+0000 7fa619400640 1 -- 172.21.15.6:0/1506300368 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa604037070 con 0x7fa61c159110 2024-04-24T23:26:24.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:24.209+0000 7fa619400640 1 -- 172.21.15.6:0/1506300368 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa604032440 con 0x7fa61c159110 2024-04-24T23:26:24.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:24.209+0000 7fa619400640 1 -- 172.21.15.6:0/1506300368 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa604032850 con 0x7fa61c159110 2024-04-24T23:26:24.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:24.210+0000 7fa621400640 1 -- 172.21.15.6:0/1506300368 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa61c159110 msgr2=0x7fa61c072420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:26:24.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:24.210+0000 7fa621400640 1 --2- 172.21.15.6:0/1506300368 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa61c159110 0x7fa61c072420 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fa6040097a0 tx=0x7fa60402a250 comp rx=0 tx=0).stop 2024-04-24T23:26:24.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:24.211+0000 7fa621400640 1 -- 172.21.15.6:0/1506300368 shutdown_connections 2024-04-24T23:26:24.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:24.211+0000 7fa621400640 1 --2- 172.21.15.6:0/1506300368 >> v2:172.21.15.6:3300/0 conn(0x7fa61c0c7dc0 0x7fa61c1623a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:26:24.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:24.211+0000 7fa621400640 1 --2- 172.21.15.6:0/1506300368 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa61c159110 0x7fa61c072420 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:26:24.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:24.212+0000 7fa621400640 1 -- 172.21.15.6:0/1506300368 >> 172.21.15.6:0/1506300368 conn(0x7fa61c06b0e0 msgr2=0x7fa61c06b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:26:24.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:24.212+0000 7fa621400640 1 -- 172.21.15.6:0/1506300368 shutdown_connections 2024-04-24T23:26:24.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:24.212+0000 7fa621400640 1 -- 172.21.15.6:0/1506300368 wait complete. 2024-04-24T23:26:24.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:24.221+0000 7fa621400640 1 Processor -- start 2024-04-24T23:26:24.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:24.221+0000 7fa621400640 1 -- start start 2024-04-24T23:26:24.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:24.227+0000 7fa621400640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa61c0c7dc0 0x7fa61c0c55c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:26:24.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:24.227+0000 7fa621400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa61c0c81c0 0x7fa61c155540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:26:24.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:24.227+0000 7fa621400640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa61c159110 0x7fa61c155a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:26:24.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:24.227+0000 7fa621400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fa61c06ece0 con 0x7fa61c0c81c0 2024-04-24T23:26:24.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:24.227+0000 7fa621400640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fa61c06ea00 con 0x7fa61c159110 2024-04-24T23:26:24.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:24.227+0000 7fa621400640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fa61c072960 con 0x7fa61c0c7dc0 2024-04-24T23:26:24.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:24.228+0000 7fa619e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa61c0c81c0 0x7fa61c155540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:26:24.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:24.228+0000 7fa61b200640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa61c159110 0x7fa61c155a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:26:24.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:24.228+0000 7fa61b200640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa61c159110 0x7fa61c155a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:51972/0 (socket says 172.21.15.6:51972) 2024-04-24T23:26:24.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:24.228+0000 7fa61b200640 1 -- 172.21.15.6:0/2854185935 learned_addr learned my addr 172.21.15.6:0/2854185935 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:26:24.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:24.229+0000 7fa61b200640 1 -- 172.21.15.6:0/2854185935 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa61c0c7dc0 msgr2=0x7fa61c0c55c0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-24T23:26:24.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:24.229+0000 7fa61b200640 1 --2- 172.21.15.6:0/2854185935 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa61c0c7dc0 0x7fa61c0c55c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:26:24.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:24.229+0000 7fa61b200640 1 -- 172.21.15.6:0/2854185935 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa61c0c81c0 msgr2=0x7fa61c155540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:26:24.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:24.229+0000 7fa61b200640 1 --2- 172.21.15.6:0/2854185935 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa61c0c81c0 0x7fa61c155540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:26:24.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:24.229+0000 7fa61b200640 1 -- 172.21.15.6:0/2854185935 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa61c0c42a0 con 0x7fa61c159110 2024-04-24T23:26:24.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:24.229+0000 7fa61b200640 1 --2- 172.21.15.6:0/2854185935 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa61c159110 0x7fa61c155a80 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fa61400cb40 tx=0x7fa61400a060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:26:24.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:24.229+0000 7fa612a00640 1 -- 172.21.15.6:0/2854185935 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa614017070 con 0x7fa61c159110 2024-04-24T23:26:24.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:24.229+0000 7fa612a00640 1 -- 172.21.15.6:0/2854185935 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa61400aca0 con 0x7fa61c159110 2024-04-24T23:26:24.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:24.229+0000 7fa612a00640 1 -- 172.21.15.6:0/2854185935 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa614012600 con 0x7fa61c159110 2024-04-24T23:26:24.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:24.230+0000 7fa621400640 1 -- 172.21.15.6:0/2854185935 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa61c1f4ef0 con 0x7fa61c159110 2024-04-24T23:26:24.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:24.230+0000 7fa621400640 1 -- 172.21.15.6:0/2854185935 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa61c0c4800 con 0x7fa61c159110 2024-04-24T23:26:24.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:24.232+0000 7fa612a00640 1 -- 172.21.15.6:0/2854185935 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fa61401a070 con 0x7fa61c159110 2024-04-24T23:26:24.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:24.233+0000 7fa621400640 1 -- 172.21.15.6:0/2854185935 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa61c0767b0 con 0x7fa61c159110 2024-04-24T23:26:24.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:24.233+0000 7fa612a00640 1 --2- 172.21.15.6:0/2854185935 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fa5f806bf00 0x7fa5f806e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:26:24.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:24.234+0000 7fa612a00640 1 -- 172.21.15.6:0/2854185935 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6934+0+0 (secure 0 0 0) 0x7fa61408b090 con 0x7fa61c159110 2024-04-24T23:26:24.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:24.237+0000 7fa61a800640 1 --2- 172.21.15.6:0/2854185935 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fa5f806bf00 0x7fa5f806e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:26:24.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:24.238+0000 7fa61a800640 1 --2- 172.21.15.6:0/2854185935 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fa5f806bf00 0x7fa5f806e3c0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fa604034000 tx=0x7fa60402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:26:24.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:24.240+0000 7fa612a00640 1 -- 172.21.15.6:0/2854185935 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fa61c0c42a0 con 0x7fa61c159110 2024-04-24T23:26:24.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:24.686+0000 7fa621400640 1 -- 172.21.15.6:0/2854185935 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa61c0c7970 con 0x7fa61c159110 2024-04-24T23:26:24.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:24.687+0000 7fa612a00640 1 -- 172.21.15.6:0/2854185935 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6033 (secure 0 0 0) 0x7fa614058e70 con 0x7fa61c159110 2024-04-24T23:26:24.689 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:26:24.689 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5081,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/1190759626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1190759626},{"type":"v1","addr":"172.21.15.6:6834","nonce":1190759626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":5249,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6838/2416336040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2416336040},{"type":"v1","addr":"172.21.15.190:6838","nonce":2416336040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":42},{"gid":5503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/3928731059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":3928731059},{"type":"v1","addr":"172.21.15.6:6837","nonce":3928731059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:25:42.695130+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5725,"mds_1":5599},"failed":[],"damaged":[],"stopped":[2,3,4],"info":{"gid_5599":{"gid":5599,"name":"d","rank":1,"incarnation":43,"state":"up:active","state_seq":25,"addr":"172.21.15.190:6839/2643233757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":2643233757},{"type":"v1","addr":"172.21.15.190:6839","nonce":2643233757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5725":{"gid":5725,"name":"f","rank":0,"incarnation":49,"state":"up:replay","state_seq":1,"addr":"172.21.15.190:6837/1151527362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":1151527362},{"type":"v1","addr":"172.21.15.190:6837","nonce":1151527362}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5599,"qdb_cluster":[5599]},"id":1}]} 2024-04-24T23:26:24.689 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 49 2024-04-24T23:26:24.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:24.690+0000 7fa621400640 1 -- 172.21.15.6:0/2854185935 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fa5f806bf00 msgr2=0x7fa5f806e3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:26:24.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:24.690+0000 7fa621400640 1 --2- 172.21.15.6:0/2854185935 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fa5f806bf00 0x7fa5f806e3c0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fa604034000 tx=0x7fa60402e000 comp rx=0 tx=0).stop 2024-04-24T23:26:24.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:24.690+0000 7fa621400640 1 -- 172.21.15.6:0/2854185935 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa61c159110 msgr2=0x7fa61c155a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:26:24.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:24.690+0000 7fa621400640 1 --2- 172.21.15.6:0/2854185935 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa61c159110 0x7fa61c155a80 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fa61400cb40 tx=0x7fa61400a060 comp rx=0 tx=0).stop 2024-04-24T23:26:24.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:24.693+0000 7fa621400640 1 -- 172.21.15.6:0/2854185935 shutdown_connections 2024-04-24T23:26:24.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:24.693+0000 7fa621400640 1 --2- 172.21.15.6:0/2854185935 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fa5f806bf00 0x7fa5f806e3c0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:26:24.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:24.693+0000 7fa621400640 1 --2- 172.21.15.6:0/2854185935 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa61c159110 0x7fa61c155a80 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:26:24.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:24.693+0000 7fa621400640 1 --2- 172.21.15.6:0/2854185935 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa61c0c81c0 0x7fa61c155540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:26:24.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:24.693+0000 7fa621400640 1 --2- 172.21.15.6:0/2854185935 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa61c0c7dc0 0x7fa61c0c55c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:26:24.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:24.693+0000 7fa621400640 1 -- 172.21.15.6:0/2854185935 >> 172.21.15.6:0/2854185935 conn(0x7fa61c06b0e0 msgr2=0x7fa61c1533b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:26:24.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:24.693+0000 7fa621400640 1 -- 172.21.15.6:0/2854185935 shutdown_connections 2024-04-24T23:26:24.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:24.694+0000 7fa621400640 1 -- 172.21.15.6:0/2854185935 wait complete. 2024-04-24T23:26:24.733 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-24T23:26:26.735 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:26:27.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:27.068+0000 7f9fad000640 1 Processor -- start 2024-04-24T23:26:27.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:27.068+0000 7f9fad000640 1 -- start start 2024-04-24T23:26:27.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:27.070+0000 7f9fad000640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9fa806ce30 0x7f9fa8150dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:26:27.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:27.070+0000 7f9fad000640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9fa806d370 con 0x7f9fa80c8d50 2024-04-24T23:26:27.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:27.070+0000 7f9fad000640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9fa806d650 con 0x7f9fa80c7e80 2024-04-24T23:26:27.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:27.070+0000 7f9fad000640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f9fa806def0 con 0x7f9fa806ce30 2024-04-24T23:26:27.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:27.071+0000 7f9fa7e00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9fa806ce30 0x7f9fa8150dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:26:27.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:27.072+0000 7f9fa7400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f9fa80c8d50 0x7f9fa80c7770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:46052/0 (socket says 172.21.15.6:46052) 2024-04-24T23:26:27.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:27.072+0000 7f9fa7400640 1 -- 172.21.15.6:0/1709940489 learned_addr learned my addr 172.21.15.6:0/1709940489 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:26:27.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:27.073+0000 7f9fa6000640 1 -- 172.21.15.6:0/1709940489 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1046660118 0 0) 0x7f9fa806d370 con 0x7f9fa80c8d50 2024-04-24T23:26:27.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:27.073+0000 7f9fa6000640 1 -- 172.21.15.6:0/1709940489 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9f8c003520 con 0x7f9fa80c8d50 2024-04-24T23:26:27.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:27.073+0000 7f9fa6000640 1 -- 172.21.15.6:0/1709940489 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2175565053 0 0) 0x7f9f8c003520 con 0x7f9fa80c8d50 2024-04-24T23:26:27.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:27.073+0000 7f9fa6000640 1 -- 172.21.15.6:0/1709940489 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9fa806ce30 msgr2=0x7f9fa8150dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:26:27.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:27.075+0000 7f9fa6000640 1 --2- 172.21.15.6:0/1709940489 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9fa806ce30 0x7f9fa8150dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:26:27.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:27.075+0000 7f9fa6000640 1 -- 172.21.15.6:0/1709940489 >> v1:172.21.15.190:6789/0 conn(0x7f9fa80c7e80 legacy=0x7f9fa81506b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-24T23:26:27.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:27.075+0000 7f9fa6000640 1 -- 172.21.15.6:0/1709940489 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9fa806d930 con 0x7f9fa80c8d50 2024-04-24T23:26:27.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:27.075+0000 7f9fa6000640 1 -- 172.21.15.6:0/1709940489 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3463268701 0 0) 0x7f9f94001670 con 0x7f9fa80c8d50 2024-04-24T23:26:27.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:27.075+0000 7f9fa6000640 1 -- 172.21.15.6:0/1709940489 >> v1:172.21.15.6:6789/0 conn(0x7f9fa80c8d50 legacy=0x7f9fa80c7770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:26:27.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:27.075+0000 7f9fa6000640 1 --2- 172.21.15.6:0/1709940489 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9f8c004270 0x7f9f8c0039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:26:27.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:27.075+0000 7f9fa6000640 1 --2- 172.21.15.6:0/1709940489 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9f8c004800 0x7f9f8c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:26:27.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:27.075+0000 7f9fa6000640 1 --2- 172.21.15.6:0/1709940489 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9f8c0051c0 0x7f9f8c005990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:26:27.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:27.075+0000 7f9fa6000640 1 -- 172.21.15.6:0/1709940489 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f9fa806def0 con 0x7f9f8c004270 2024-04-24T23:26:27.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:27.075+0000 7f9fa6000640 1 -- 172.21.15.6:0/1709940489 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f9f8c006090 con 0x7f9f8c0051c0 2024-04-24T23:26:27.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:27.075+0000 7f9fa6000640 1 -- 172.21.15.6:0/1709940489 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f9f8c006290 con 0x7f9f8c004800 2024-04-24T23:26:27.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:27.075+0000 7f9fa7400640 1 --2- 172.21.15.6:0/1709940489 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9f8c004270 0x7f9f8c0039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:26:27.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:27.076+0000 7f9fa7400640 1 -- 172.21.15.6:0/1709940489 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9f8c004800 msgr2=0x7f9f8c004c80 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-24T23:26:27.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:27.076+0000 7f9fa7400640 1 --2- 172.21.15.6:0/1709940489 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9f8c004800 0x7f9f8c004c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:26:27.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:27.076+0000 7f9fa7400640 1 -- 172.21.15.6:0/1709940489 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9f8c0051c0 msgr2=0x7f9f8c005990 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-24T23:26:27.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:27.076+0000 7f9fa7400640 1 --2- 172.21.15.6:0/1709940489 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9f8c0051c0 0x7f9f8c005990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:26:27.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:27.076+0000 7f9fa7400640 1 -- 172.21.15.6:0/1709940489 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9fa806d930 con 0x7f9f8c004270 2024-04-24T23:26:27.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:27.078+0000 7f9fa7e00640 1 --2- 172.21.15.6:0/1709940489 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9fa806ce30 0x7f9fa8150dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:26:27.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:27.079+0000 7f9fa7400640 1 --2- 172.21.15.6:0/1709940489 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9f8c004270 0x7f9f8c0039b0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f9f9400c5a0 tx=0x7f9f8c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=afbb150d87dcbfd9 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:26:27.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:27.080+0000 7f9fa6000640 1 -- 172.21.15.6:0/1709940489 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f9402fb60 con 0x7f9f8c004270 2024-04-24T23:26:27.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:27.080+0000 7f9fa6000640 1 -- 172.21.15.6:0/1709940489 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f94033070 con 0x7f9f8c004270 2024-04-24T23:26:27.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:27.080+0000 7f9fa6000640 1 -- 172.21.15.6:0/1709940489 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f940058d0 con 0x7f9f8c004270 2024-04-24T23:26:27.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:27.081+0000 7f9fad000640 1 -- 172.21.15.6:0/1709940489 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9f8c004270 msgr2=0x7f9f8c0039b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:26:27.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:27.081+0000 7f9fad000640 1 --2- 172.21.15.6:0/1709940489 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9f8c004270 0x7f9f8c0039b0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f9f9400c5a0 tx=0x7f9f8c005830 comp rx=0 tx=0).stop 2024-04-24T23:26:27.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:27.081+0000 7f9fa7e00640 1 -- 172.21.15.6:0/1709940489 reap_dead start 2024-04-24T23:26:27.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:27.085+0000 7f9fad000640 1 -- 172.21.15.6:0/1709940489 shutdown_connections 2024-04-24T23:26:27.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:27.085+0000 7f9fad000640 1 -- 172.21.15.6:0/1709940489 >> 172.21.15.6:0/1709940489 conn(0x7f9fa806ed80 msgr2=0x7f9fa806f190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:26:27.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:27.085+0000 7f9fad000640 1 -- 172.21.15.6:0/1709940489 shutdown_connections 2024-04-24T23:26:27.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:27.086+0000 7f9fad000640 1 -- 172.21.15.6:0/1709940489 wait complete. 2024-04-24T23:26:27.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:27.091+0000 7f9fad000640 1 Processor -- start 2024-04-24T23:26:27.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:27.094+0000 7f9fad000640 1 -- start start 2024-04-24T23:26:27.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:27.095+0000 7f9fad000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9fa80cb2f0 0x7f9fa81ec880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:26:27.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:27.096+0000 7f9fad000640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9fa80cb6d0 0x7f9fa8152400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:26:27.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:27.096+0000 7f9fad000640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9fa81468c0 0x7f9fa8146d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:26:27.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:27.096+0000 7f9fad000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f9fa80cc1e0 con 0x7f9fa80cb2f0 2024-04-24T23:26:27.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:27.096+0000 7f9fad000640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f9fa8071020 con 0x7f9fa81468c0 2024-04-24T23:26:27.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:27.096+0000 7f9fad000640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f9fa80cc360 con 0x7f9fa80cb6d0 2024-04-24T23:26:27.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:27.099+0000 7f9fa6a00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9fa80cb6d0 0x7f9fa8152400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:26:27.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:27.099+0000 7f9fa6a00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9fa80cb6d0 0x7f9fa8152400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.6:49834/0 (socket says 172.21.15.6:49834) 2024-04-24T23:26:27.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:27.099+0000 7f9fa6a00640 1 -- 172.21.15.6:0/3321519751 learned_addr learned my addr 172.21.15.6:0/3321519751 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:26:27.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:27.099+0000 7f9fa6a00640 1 -- 172.21.15.6:0/3321519751 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9fa81468c0 msgr2=0x7f9fa8146d70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-24T23:26:27.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:27.099+0000 7f9fa6a00640 1 --2- 172.21.15.6:0/3321519751 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9fa81468c0 0x7f9fa8146d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:26:27.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:27.099+0000 7f9fa6a00640 1 -- 172.21.15.6:0/3321519751 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9fa80cb2f0 msgr2=0x7f9fa81ec880 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-24T23:26:27.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:27.099+0000 7f9fa6a00640 1 --2- 172.21.15.6:0/3321519751 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9fa80cb2f0 0x7f9fa81ec880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:26:27.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:27.099+0000 7f9fa6a00640 1 -- 172.21.15.6:0/3321519751 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9fa80c6250 con 0x7f9fa80cb6d0 2024-04-24T23:26:27.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:27.100+0000 7f9fa6a00640 1 --2- 172.21.15.6:0/3321519751 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9fa80cb6d0 0x7f9fa8152400 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f9f9800b420 tx=0x7f9f9800b8e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:26:27.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:27.100+0000 7f9f9f400640 1 -- 172.21.15.6:0/3321519751 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f98022640 con 0x7f9fa80cb6d0 2024-04-24T23:26:27.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:27.100+0000 7f9f9f400640 1 -- 172.21.15.6:0/3321519751 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f98027070 con 0x7f9fa80cb6d0 2024-04-24T23:26:27.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:27.100+0000 7f9f9f400640 1 -- 172.21.15.6:0/3321519751 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f98022a20 con 0x7f9fa80cb6d0 2024-04-24T23:26:27.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:27.102+0000 7f9fad000640 1 -- 172.21.15.6:0/3321519751 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9fa80c6530 con 0x7f9fa80cb6d0 2024-04-24T23:26:27.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:27.102+0000 7f9fad000640 1 -- 172.21.15.6:0/3321519751 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f9fa80c46c0 con 0x7f9fa80cb6d0 2024-04-24T23:26:27.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:27.106+0000 7f9fad000640 1 -- 172.21.15.6:0/3321519751 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9fa8075b30 con 0x7f9fa80cb6d0 2024-04-24T23:26:27.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:27.108+0000 7f9f9f400640 1 -- 172.21.15.6:0/3321519751 <== mon.2 v2:172.21.15.190:3301/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f9f98022bc0 con 0x7f9fa80cb6d0 2024-04-24T23:26:27.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:27.115+0000 7f9f9f400640 1 --2- 172.21.15.6:0/3321519751 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f9f8806be60 0x7f9f8806e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:26:27.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:27.116+0000 7f9f9f400640 1 -- 172.21.15.6:0/3321519751 <== mon.2 v2:172.21.15.190:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6934+0+0 (secure 0 0 0) 0x7f9f9809b8c0 con 0x7f9fa80cb6d0 2024-04-24T23:26:27.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:27.116+0000 7f9fa7400640 1 --2- 172.21.15.6:0/3321519751 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f9f8806be60 0x7f9f8806e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:26:27.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:27.120+0000 7f9f9f400640 1 -- 172.21.15.6:0/3321519751 <== mon.2 v2:172.21.15.190:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f9f98030020 con 0x7f9fa80cb6d0 2024-04-24T23:26:27.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:27.135+0000 7f9fa7400640 1 --2- 172.21.15.6:0/3321519751 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f9f8806be60 0x7f9f8806e320 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f9f94037000 tx=0x7f9f94009030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:26:27.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:27.677+0000 7f9fad000640 1 -- 172.21.15.6:0/3321519751 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9fa8154340 con 0x7f9fa80cb6d0 2024-04-24T23:26:27.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:27.678+0000 7f9f9f400640 1 -- 172.21.15.6:0/3321519751 <== mon.2 v2:172.21.15.190:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6033 (secure 0 0 0) 0x7f9f98069d00 con 0x7f9fa80cb6d0 2024-04-24T23:26:27.683 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:26:27.683 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5081,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/1190759626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1190759626},{"type":"v1","addr":"172.21.15.6:6834","nonce":1190759626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":5249,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6838/2416336040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2416336040},{"type":"v1","addr":"172.21.15.190:6838","nonce":2416336040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":42},{"gid":5503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/3928731059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":3928731059},{"type":"v1","addr":"172.21.15.6:6837","nonce":3928731059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:25:42.695130+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5725,"mds_1":5599},"failed":[],"damaged":[],"stopped":[2,3,4],"info":{"gid_5599":{"gid":5599,"name":"d","rank":1,"incarnation":43,"state":"up:active","state_seq":25,"addr":"172.21.15.190:6839/2643233757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":2643233757},{"type":"v1","addr":"172.21.15.190:6839","nonce":2643233757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5725":{"gid":5725,"name":"f","rank":0,"incarnation":49,"state":"up:replay","state_seq":1,"addr":"172.21.15.190:6837/1151527362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":1151527362},{"type":"v1","addr":"172.21.15.190:6837","nonce":1151527362}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5599,"qdb_cluster":[5599]},"id":1}]} 2024-04-24T23:26:27.683 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 49 2024-04-24T23:26:27.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:27.685+0000 7f9fad000640 1 -- 172.21.15.6:0/3321519751 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f9f8806be60 msgr2=0x7f9f8806e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:26:27.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:27.685+0000 7f9fad000640 1 --2- 172.21.15.6:0/3321519751 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f9f8806be60 0x7f9f8806e320 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f9f94037000 tx=0x7f9f94009030 comp rx=0 tx=0).stop 2024-04-24T23:26:27.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:27.685+0000 7f9fad000640 1 -- 172.21.15.6:0/3321519751 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9fa80cb6d0 msgr2=0x7f9fa8152400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:26:27.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:27.685+0000 7f9fad000640 1 --2- 172.21.15.6:0/3321519751 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9fa80cb6d0 0x7f9fa8152400 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f9f9800b420 tx=0x7f9f9800b8e0 comp rx=0 tx=0).stop 2024-04-24T23:26:27.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:27.686+0000 7f9fad000640 1 -- 172.21.15.6:0/3321519751 shutdown_connections 2024-04-24T23:26:27.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:27.686+0000 7f9fad000640 1 --2- 172.21.15.6:0/3321519751 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f9f8806be60 0x7f9f8806e320 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:26:27.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:27.686+0000 7f9fad000640 1 --2- 172.21.15.6:0/3321519751 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9fa80cb6d0 0x7f9fa8152400 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:26:27.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:27.686+0000 7f9fad000640 1 --2- 172.21.15.6:0/3321519751 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9fa81468c0 0x7f9fa8146d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:26:27.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:27.686+0000 7f9fad000640 1 --2- 172.21.15.6:0/3321519751 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9fa80cb2f0 0x7f9fa81ec880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:26:27.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:27.686+0000 7f9fad000640 1 -- 172.21.15.6:0/3321519751 >> 172.21.15.6:0/3321519751 conn(0x7f9fa806ed80 msgr2=0x7f9fa80c68b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:26:27.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:27.687+0000 7f9fad000640 1 -- 172.21.15.6:0/3321519751 shutdown_connections 2024-04-24T23:26:27.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:27.688+0000 7f9fad000640 1 -- 172.21.15.6:0/3321519751 wait complete. 2024-04-24T23:26:27.725 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 49, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-24T23:15:36.400716+0000", "damaged": [], "data_pools": [ 3, 4 ], "enabled": true, "epoch": 49, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_5599": { "addr": "172.21.15.190:6839/2643233757", "addrs": { "addrvec": [ { "addr": "172.21.15.190:6836", "nonce": 2643233757, "type": "v2" }, { "addr": "172.21.15.190:6839", "nonce": 2643233757, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5599, "incarnation": 43, "join_fscid": -1, "name": "d", "rank": 1, "state": "up:active", "state_seq": 25 }, "gid_5725": { "addr": "172.21.15.190:6837/1151527362", "addrs": { "addrvec": [ { "addr": "172.21.15.190:6834", "nonce": 1151527362, "type": "v2" }, { "addr": "172.21.15.190:6837", "nonce": 1151527362, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5725, "incarnation": 49, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 30, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-04-24T23:25:42.695130+0000", "qdb_cluster": [ 5599 ], "qdb_leader": 5599, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 2, 3, 4 ], "tableserver": 0, "up": { "mds_0": 5725, "mds_1": 5599 } } } ], "standbys": [ { "addr": "172.21.15.6:6834/1190759626", "addrs": { "addrvec": [ { "addr": "172.21.15.6:6832", "nonce": 1190759626, "type": "v2" }, { "addr": "172.21.15.6:6834", "nonce": 1190759626, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 30, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5081, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.190:6838/2416336040", "addrs": { "addrvec": [ { "addr": "172.21.15.190:6835", "nonce": 2416336040, "type": "v2" }, { "addr": "172.21.15.190:6838", "nonce": 2416336040, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 42, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5249, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.6:6837/3928731059", "addrs": { "addrvec": [ { "addr": "172.21.15.6:6836", "nonce": 3928731059, "type": "v2" }, { "addr": "172.21.15.6:6837", "nonce": 3928731059, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 35, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5503, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-04-24T23:26:29.726 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:26:30.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:30.101+0000 7fc61f000640 1 Processor -- start 2024-04-24T23:26:30.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:30.101+0000 7fc61f000640 1 -- start start 2024-04-24T23:26:30.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:30.103+0000 7fc61f000640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc6100d90c0 0x7fc6100dedd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:26:30.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:30.103+0000 7fc61f000640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fc6100501b0 0x7fc6100505f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:26:30.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:30.103+0000 7fc61f000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc610037190 0x7fc610037600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:26:30.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:30.103+0000 7fc61f000640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7fc6100380c0 con 0x7fc610037190 2024-04-24T23:26:30.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:30.103+0000 7fc61f000640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7fc6100df4d0 con 0x7fc6100501b0 2024-04-24T23:26:30.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:30.103+0000 7fc61f000640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fc6100df7b0 con 0x7fc6100d90c0 2024-04-24T23:26:30.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:30.108+0000 7fc617400640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fc6100501b0 0x7fc6100505f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:26:30.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:30.108+0000 7fc617400640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fc6100501b0 0x7fc6100505f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:32866/0 (socket says 172.21.15.6:32866) 2024-04-24T23:26:30.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:30.108+0000 7fc617400640 1 -- 172.21.15.6:0/3037485933 learned_addr learned my addr 172.21.15.6:0/3037485933 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:26:30.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:30.108+0000 7fc617e00640 1 --2- 172.21.15.6:0/3037485933 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc6100d90c0 0x7fc6100dedd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:26:30.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:30.109+0000 7fc61d000640 1 --2- 172.21.15.6:0/3037485933 >> v2:172.21.15.6:3300/0 conn(0x7fc610037190 0x7fc610037600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:26:30.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:30.109+0000 7fc617400640 1 -- 172.21.15.6:0/3037485933 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc6100d90c0 msgr2=0x7fc6100dedd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:26:30.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:30.109+0000 7fc617400640 1 --2- 172.21.15.6:0/3037485933 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc6100d90c0 0x7fc6100dedd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:26:30.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:30.109+0000 7fc617400640 1 -- 172.21.15.6:0/3037485933 >> v2:172.21.15.6:3300/0 conn(0x7fc610037190 msgr2=0x7fc610037600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:26:30.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:30.109+0000 7fc617400640 1 --2- 172.21.15.6:0/3037485933 >> v2:172.21.15.6:3300/0 conn(0x7fc610037190 0x7fc610037600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:26:30.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:30.109+0000 7fc617400640 1 -- 172.21.15.6:0/3037485933 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6100dfa90 con 0x7fc6100501b0 2024-04-24T23:26:30.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:30.114+0000 7fc617400640 1 --2- 172.21.15.6:0/3037485933 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc6100501b0 0x7fc6100505f0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fc610011ce0 tx=0x7fc60402a1f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=72f5599947c73109 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:26:30.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:30.115+0000 7fc616a00640 1 -- 172.21.15.6:0/3037485933 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc604037070 con 0x7fc6100501b0 2024-04-24T23:26:30.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:30.115+0000 7fc616a00640 1 -- 172.21.15.6:0/3037485933 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc604032440 con 0x7fc6100501b0 2024-04-24T23:26:30.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:30.115+0000 7fc616a00640 1 -- 172.21.15.6:0/3037485933 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc604037070 con 0x7fc6100501b0 2024-04-24T23:26:30.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:30.116+0000 7fc61f000640 1 -- 172.21.15.6:0/3037485933 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc6100501b0 msgr2=0x7fc6100505f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:26:30.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:30.116+0000 7fc61f000640 1 --2- 172.21.15.6:0/3037485933 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc6100501b0 0x7fc6100505f0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fc610011ce0 tx=0x7fc60402a1f0 comp rx=0 tx=0).stop 2024-04-24T23:26:30.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:30.118+0000 7fc61f000640 1 -- 172.21.15.6:0/3037485933 shutdown_connections 2024-04-24T23:26:30.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:30.118+0000 7fc61f000640 1 --2- 172.21.15.6:0/3037485933 >> v2:172.21.15.6:3300/0 conn(0x7fc610037190 0x7fc610037600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:26:30.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:30.118+0000 7fc61f000640 1 --2- 172.21.15.6:0/3037485933 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc6100501b0 0x7fc6100505f0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:26:30.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:30.118+0000 7fc61f000640 1 --2- 172.21.15.6:0/3037485933 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc6100d90c0 0x7fc6100dedd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:26:30.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:30.118+0000 7fc61f000640 1 -- 172.21.15.6:0/3037485933 >> 172.21.15.6:0/3037485933 conn(0x7fc610054640 msgr2=0x7fc610054a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:26:30.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:30.120+0000 7fc61f000640 1 -- 172.21.15.6:0/3037485933 shutdown_connections 2024-04-24T23:26:30.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:30.120+0000 7fc61f000640 1 -- 172.21.15.6:0/3037485933 wait complete. 2024-04-24T23:26:30.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:30.132+0000 7fc61f000640 1 Processor -- start 2024-04-24T23:26:30.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:30.132+0000 7fc61f000640 1 -- start start 2024-04-24T23:26:30.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:30.138+0000 7fc61f000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc610037190 0x7fc61004e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:26:30.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:30.138+0000 7fc61f000640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc6100501b0 0x7fc61004bed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:26:30.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:30.138+0000 7fc61f000640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc6100d90c0 0x7fc610179670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:26:30.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:30.138+0000 7fc61f000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fc610038550 con 0x7fc610037190 2024-04-24T23:26:30.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:30.138+0000 7fc61f000640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fc6100d94a0 con 0x7fc6100d90c0 2024-04-24T23:26:30.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:30.138+0000 7fc61f000640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fc6100386d0 con 0x7fc6100501b0 2024-04-24T23:26:30.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:30.138+0000 7fc61d000640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc6100d90c0 0x7fc610179670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:26:30.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:30.138+0000 7fc61d000640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc6100d90c0 0x7fc610179670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:32870/0 (socket says 172.21.15.6:32870) 2024-04-24T23:26:30.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:30.138+0000 7fc61d000640 1 -- 172.21.15.6:0/3445494969 learned_addr learned my addr 172.21.15.6:0/3445494969 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:26:30.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:30.138+0000 7fc617400640 1 --2- 172.21.15.6:0/3445494969 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc6100501b0 0x7fc61004bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:26:30.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:30.139+0000 7fc617e00640 1 --2- 172.21.15.6:0/3445494969 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc610037190 0x7fc61004e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:26:30.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:30.139+0000 7fc617400640 1 -- 172.21.15.6:0/3445494969 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc6100d90c0 msgr2=0x7fc610179670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:26:30.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:30.139+0000 7fc617400640 1 --2- 172.21.15.6:0/3445494969 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc6100d90c0 0x7fc610179670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:26:30.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:30.139+0000 7fc617400640 1 -- 172.21.15.6:0/3445494969 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc610037190 msgr2=0x7fc61004e8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:26:30.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:30.139+0000 7fc617400640 1 --2- 172.21.15.6:0/3445494969 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc610037190 0x7fc61004e8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:26:30.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:30.139+0000 7fc617400640 1 -- 172.21.15.6:0/3445494969 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc610178eb0 con 0x7fc6100501b0 2024-04-24T23:26:30.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:30.140+0000 7fc617e00640 1 --2- 172.21.15.6:0/3445494969 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc610037190 0x7fc61004e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:26:30.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:30.140+0000 7fc61d000640 1 --2- 172.21.15.6:0/3445494969 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc6100d90c0 0x7fc610179670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:26:30.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:30.142+0000 7fc617400640 1 --2- 172.21.15.6:0/3445494969 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc6100501b0 0x7fc61004bed0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fc604031000 tx=0x7fc6040046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:26:30.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:30.142+0000 7fc60be00640 1 -- 172.21.15.6:0/3445494969 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc604037040 con 0x7fc6100501b0 2024-04-24T23:26:30.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:30.143+0000 7fc60be00640 1 -- 172.21.15.6:0/3445494969 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc604042070 con 0x7fc6100501b0 2024-04-24T23:26:30.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:30.143+0000 7fc60be00640 1 -- 172.21.15.6:0/3445494969 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc604032810 con 0x7fc6100501b0 2024-04-24T23:26:30.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:30.143+0000 7fc61f000640 1 -- 172.21.15.6:0/3445494969 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc61004c4f0 con 0x7fc6100501b0 2024-04-24T23:26:30.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:30.143+0000 7fc61f000640 1 -- 172.21.15.6:0/3445494969 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc61004cab0 con 0x7fc6100501b0 2024-04-24T23:26:30.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:30.147+0000 7fc61f000640 1 -- 172.21.15.6:0/3445494969 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc5f4005290 con 0x7fc6100501b0 2024-04-24T23:26:30.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:30.148+0000 7fc60be00640 1 -- 172.21.15.6:0/3445494969 <== mon.2 v2:172.21.15.190:3301/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fc6040329b0 con 0x7fc6100501b0 2024-04-24T23:26:30.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:30.149+0000 7fc60be00640 1 --2- 172.21.15.6:0/3445494969 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fc5f806bf00 0x7fc5f806e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:26:30.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:30.150+0000 7fc60be00640 1 -- 172.21.15.6:0/3445494969 <== mon.2 v2:172.21.15.190:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6934+0+0 (secure 0 0 0) 0x7fc6040abb40 con 0x7fc6100501b0 2024-04-24T23:26:30.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:30.151+0000 7fc617e00640 1 --2- 172.21.15.6:0/3445494969 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fc5f806bf00 0x7fc5f806e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:26:30.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:30.153+0000 7fc617e00640 1 --2- 172.21.15.6:0/3445494969 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fc5f806bf00 0x7fc5f806e3c0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fc60c009ae0 tx=0x7fc60c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:26:30.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:30.156+0000 7fc60be00640 1 -- 172.21.15.6:0/3445494969 <== mon.2 v2:172.21.15.190:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fc604079f70 con 0x7fc6100501b0 2024-04-24T23:26:30.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:30.707+0000 7fc61f000640 1 -- 172.21.15.6:0/3445494969 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc5f4005580 con 0x7fc6100501b0 2024-04-24T23:26:30.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:30.712+0000 7fc60be00640 1 -- 172.21.15.6:0/3445494969 <== mon.2 v2:172.21.15.190:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6033 (secure 0 0 0) 0x7fc604079f70 con 0x7fc6100501b0 2024-04-24T23:26:30.713 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:26:30.713 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5081,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/1190759626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1190759626},{"type":"v1","addr":"172.21.15.6:6834","nonce":1190759626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":5249,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6838/2416336040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2416336040},{"type":"v1","addr":"172.21.15.190:6838","nonce":2416336040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":42},{"gid":5503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/3928731059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":3928731059},{"type":"v1","addr":"172.21.15.6:6837","nonce":3928731059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:25:42.695130+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5725,"mds_1":5599},"failed":[],"damaged":[],"stopped":[2,3,4],"info":{"gid_5599":{"gid":5599,"name":"d","rank":1,"incarnation":43,"state":"up:active","state_seq":25,"addr":"172.21.15.190:6839/2643233757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":2643233757},{"type":"v1","addr":"172.21.15.190:6839","nonce":2643233757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5725":{"gid":5725,"name":"f","rank":0,"incarnation":49,"state":"up:replay","state_seq":1,"addr":"172.21.15.190:6837/1151527362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":1151527362},{"type":"v1","addr":"172.21.15.190:6837","nonce":1151527362}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5599,"qdb_cluster":[5599]},"id":1}]} 2024-04-24T23:26:30.713 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 49 2024-04-24T23:26:30.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:30.719+0000 7fc609600640 1 -- 172.21.15.6:0/3445494969 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fc5f806bf00 msgr2=0x7fc5f806e3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:26:30.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:30.719+0000 7fc609600640 1 --2- 172.21.15.6:0/3445494969 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fc5f806bf00 0x7fc5f806e3c0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fc60c009ae0 tx=0x7fc60c008040 comp rx=0 tx=0).stop 2024-04-24T23:26:30.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:30.719+0000 7fc609600640 1 -- 172.21.15.6:0/3445494969 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc6100501b0 msgr2=0x7fc61004bed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:26:30.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:30.719+0000 7fc609600640 1 --2- 172.21.15.6:0/3445494969 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc6100501b0 0x7fc61004bed0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fc604031000 tx=0x7fc6040046c0 comp rx=0 tx=0).stop 2024-04-24T23:26:30.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:30.720+0000 7fc609600640 1 -- 172.21.15.6:0/3445494969 shutdown_connections 2024-04-24T23:26:30.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:30.720+0000 7fc609600640 1 --2- 172.21.15.6:0/3445494969 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fc5f806bf00 0x7fc5f806e3c0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:26:30.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:30.720+0000 7fc609600640 1 --2- 172.21.15.6:0/3445494969 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc6100501b0 0x7fc61004bed0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:26:30.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:30.720+0000 7fc609600640 1 --2- 172.21.15.6:0/3445494969 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc6100d90c0 0x7fc610179670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:26:30.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:30.720+0000 7fc609600640 1 --2- 172.21.15.6:0/3445494969 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc610037190 0x7fc61004e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:26:30.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:30.720+0000 7fc609600640 1 -- 172.21.15.6:0/3445494969 >> 172.21.15.6:0/3445494969 conn(0x7fc610054640 msgr2=0x7fc6100d3a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:26:30.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:30.721+0000 7fc609600640 1 -- 172.21.15.6:0/3445494969 shutdown_connections 2024-04-24T23:26:30.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:30.721+0000 7fc609600640 1 -- 172.21.15.6:0/3445494969 wait complete. 2024-04-24T23:26:30.779 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-24T23:26:32.780 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:26:33.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:33.089+0000 7f733d200640 1 Processor -- start 2024-04-24T23:26:33.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:33.089+0000 7f733d200640 1 -- start start 2024-04-24T23:26:33.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:33.091+0000 7f733d200640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f73380cd390 0x7f73380cacf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:26:33.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:33.091+0000 7f733d200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f73381548a0 0x7f7338154cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:26:33.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:33.098+0000 7f732ea00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f73380cd390 0x7f73380cacf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:26:33.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:33.098+0000 7f732ea00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f73380cd390 0x7f73380cacf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.6:49862/0 (socket says 172.21.15.6:49862) 2024-04-24T23:26:33.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:33.098+0000 7f7337400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f73380ccfb0 0x7f733815b480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:46064/0 (socket says 172.21.15.6:46064) 2024-04-24T23:26:33.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:33.098+0000 7f7337400640 1 -- 172.21.15.6:0/1145138118 learned_addr learned my addr 172.21.15.6:0/1145138118 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:26:33.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:33.099+0000 7f7337e00640 1 --2- 172.21.15.6:0/1145138118 >> v2:172.21.15.6:3300/0 conn(0x7f73381548a0 0x7f7338154cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:26:33.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:33.100+0000 7f733d200640 1 -- 172.21.15.6:0/1145138118 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f733806be70 con 0x7f73380ccfb0 2024-04-24T23:26:33.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:33.100+0000 7f733d200640 1 -- 172.21.15.6:0/1145138118 --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f733806ccd0 con 0x7f73381548a0 2024-04-24T23:26:33.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:33.100+0000 7f733d200640 1 -- 172.21.15.6:0/1145138118 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f73380cb230 con 0x7f73380cd390 2024-04-24T23:26:33.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:33.101+0000 7f7336a00640 1 -- 172.21.15.6:0/1145138118 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1942597852 0 0) 0x7f733806be70 con 0x7f73380ccfb0 2024-04-24T23:26:33.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:33.103+0000 7f7336a00640 1 -- 172.21.15.6:0/1145138118 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f731c003520 con 0x7f73380ccfb0 2024-04-24T23:26:33.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:33.104+0000 7f7336a00640 1 -- 172.21.15.6:0/1145138118 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1276047510 0 0) 0x7f731c003520 con 0x7f73380ccfb0 2024-04-24T23:26:33.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:33.104+0000 7f732ea00640 1 -- 172.21.15.6:0/1145138118 >> v2:172.21.15.6:3300/0 conn(0x7f73381548a0 msgr2=0x7f7338154cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:26:33.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:33.104+0000 7f732ea00640 1 --2- 172.21.15.6:0/1145138118 >> v2:172.21.15.6:3300/0 conn(0x7f73381548a0 0x7f7338154cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:26:33.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:33.104+0000 7f732ea00640 1 -- 172.21.15.6:0/1145138118 >> v1:172.21.15.6:6789/0 conn(0x7f73380ccfb0 legacy=0x7f733815b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:26:33.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:33.104+0000 7f732ea00640 1 -- 172.21.15.6:0/1145138118 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f733806c150 con 0x7f73380cd390 2024-04-24T23:26:33.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:33.106+0000 7f732ea00640 1 --2- 172.21.15.6:0/1145138118 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f73380cd390 0x7f73380cacf0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f73200097a0 tx=0x7f732002a1c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b5e0fc9eb4512ab7 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:26:33.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:33.106+0000 7f7337e00640 1 --2- 172.21.15.6:0/1145138118 >> v2:172.21.15.6:3300/0 conn(0x7f73381548a0 0x7f7338154cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:26:33.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:33.107+0000 7f7336a00640 1 -- 172.21.15.6:0/1145138118 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7320037070 con 0x7f73380cd390 2024-04-24T23:26:33.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:33.107+0000 7f7336a00640 1 -- 172.21.15.6:0/1145138118 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f732002ac50 con 0x7f73380cd390 2024-04-24T23:26:33.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:33.107+0000 7f7336a00640 1 -- 172.21.15.6:0/1145138118 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7328001670 con 0x7f73380cd390 2024-04-24T23:26:33.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:33.108+0000 7f733d200640 1 -- 172.21.15.6:0/1145138118 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f73380cd390 msgr2=0x7f73380cacf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:26:33.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:33.108+0000 7f733d200640 1 --2- 172.21.15.6:0/1145138118 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f73380cd390 0x7f73380cacf0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f73200097a0 tx=0x7f732002a1c0 comp rx=0 tx=0).stop 2024-04-24T23:26:33.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:33.109+0000 7f733d200640 1 -- 172.21.15.6:0/1145138118 shutdown_connections 2024-04-24T23:26:33.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:33.109+0000 7f733d200640 1 --2- 172.21.15.6:0/1145138118 >> v2:172.21.15.6:3300/0 conn(0x7f73381548a0 0x7f7338154cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:26:33.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:33.109+0000 7f733d200640 1 --2- 172.21.15.6:0/1145138118 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f73380cd390 0x7f73380cacf0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:26:33.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:33.109+0000 7f733d200640 1 -- 172.21.15.6:0/1145138118 >> 172.21.15.6:0/1145138118 conn(0x7f733806d9f0 msgr2=0x7f7338066ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:26:33.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:33.110+0000 7f733d200640 1 -- 172.21.15.6:0/1145138118 shutdown_connections 2024-04-24T23:26:33.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:33.110+0000 7f733d200640 1 -- 172.21.15.6:0/1145138118 wait complete. 2024-04-24T23:26:33.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:33.121+0000 7f733d200640 1 Processor -- start 2024-04-24T23:26:33.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:33.121+0000 7f733d200640 1 -- start start 2024-04-24T23:26:33.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:33.124+0000 7f733d200640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f73380ccfb0 0x7f733815c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:26:33.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:33.125+0000 7f733d200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f73380cd390 0x7f73381f4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:26:33.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:33.126+0000 7f733d200640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f73381548a0 0x7f733814cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:26:33.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:33.125+0000 7f7337400640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f73380ccfb0 0x7f733815c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:26:33.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:33.126+0000 7f7337400640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f73380ccfb0 0x7f733815c600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:32880/0 (socket says 172.21.15.6:32880) 2024-04-24T23:26:33.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:33.128+0000 7f7337400640 1 -- 172.21.15.6:0/2834606956 learned_addr learned my addr 172.21.15.6:0/2834606956 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:26:33.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:33.128+0000 7f7337e00640 1 --2- 172.21.15.6:0/2834606956 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f73381548a0 0x7f733814cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:26:33.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:33.128+0000 7f733d200640 1 -- 172.21.15.6:0/2834606956 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f733806c9f0 con 0x7f73380cd390 2024-04-24T23:26:33.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:33.128+0000 7f733d200640 1 -- 172.21.15.6:0/2834606956 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f733806c710 con 0x7f73380ccfb0 2024-04-24T23:26:33.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:33.128+0000 7f733d200640 1 -- 172.21.15.6:0/2834606956 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f7338067bb0 con 0x7f73381548a0 2024-04-24T23:26:33.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:33.130+0000 7f732ea00640 1 --2- 172.21.15.6:0/2834606956 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f73380cd390 0x7f73381f4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:26:33.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:33.132+0000 7f7337e00640 1 -- 172.21.15.6:0/2834606956 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f73380ccfb0 msgr2=0x7f733815c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:26:33.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:33.132+0000 7f7337e00640 1 --2- 172.21.15.6:0/2834606956 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f73380ccfb0 0x7f733815c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:26:33.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:33.132+0000 7f7337e00640 1 -- 172.21.15.6:0/2834606956 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f73380cd390 msgr2=0x7f73381f4dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:26:33.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:33.133+0000 7f7337e00640 1 --2- 172.21.15.6:0/2834606956 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f73380cd390 0x7f73381f4dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:26:33.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:33.133+0000 7f7337e00640 1 -- 172.21.15.6:0/2834606956 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73380ce8b0 con 0x7f73381548a0 2024-04-24T23:26:33.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:33.135+0000 7f7337400640 1 --2- 172.21.15.6:0/2834606956 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f73380ccfb0 0x7f733815c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:26:33.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:33.135+0000 7f732ea00640 1 --2- 172.21.15.6:0/2834606956 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f73380cd390 0x7f73381f4dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:26:33.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:33.136+0000 7f7337e00640 1 --2- 172.21.15.6:0/2834606956 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f73381548a0 0x7f733814cef0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f733000efc0 tx=0x7f733000c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:26:33.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:33.138+0000 7f732fe00640 1 -- 172.21.15.6:0/2834606956 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7330018030 con 0x7f73381548a0 2024-04-24T23:26:33.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:33.138+0000 7f733d200640 1 -- 172.21.15.6:0/2834606956 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73380ceb90 con 0x7f73381548a0 2024-04-24T23:26:33.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:33.138+0000 7f733d200640 1 -- 172.21.15.6:0/2834606956 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f73381f45c0 con 0x7f73381548a0 2024-04-24T23:26:33.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:33.140+0000 7f732fe00640 1 -- 172.21.15.6:0/2834606956 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f733000cd30 con 0x7f73381548a0 2024-04-24T23:26:33.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:33.140+0000 7f732fe00640 1 -- 172.21.15.6:0/2834606956 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7330003bb0 con 0x7f73381548a0 2024-04-24T23:26:33.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:33.150+0000 7f733d200640 1 -- 172.21.15.6:0/2834606956 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7310005290 con 0x7f73381548a0 2024-04-24T23:26:33.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:33.152+0000 7f732fe00640 1 -- 172.21.15.6:0/2834606956 <== mon.2 v2:172.21.15.190:3301/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f733001b070 con 0x7f73381548a0 2024-04-24T23:26:33.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:33.155+0000 7f732fe00640 1 --2- 172.21.15.6:0/2834606956 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f731406be30 0x7f731406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:26:33.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:33.156+0000 7f732fe00640 1 -- 172.21.15.6:0/2834606956 <== mon.2 v2:172.21.15.190:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6934+0+0 (secure 0 0 0) 0x7f733008bbf0 con 0x7f73381548a0 2024-04-24T23:26:33.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:33.156+0000 7f7337400640 1 --2- 172.21.15.6:0/2834606956 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f731406be30 0x7f731406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:26:33.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:33.159+0000 7f732fe00640 1 -- 172.21.15.6:0/2834606956 <== mon.2 v2:172.21.15.190:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f7330010040 con 0x7f73381548a0 2024-04-24T23:26:33.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:33.160+0000 7f7337400640 1 --2- 172.21.15.6:0/2834606956 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f731406be30 0x7f731406e2f0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f7328001eb0 tx=0x7f7328009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:26:33.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:33.602+0000 7f733d200640 1 -- 172.21.15.6:0/2834606956 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7310005580 con 0x7f73381548a0 2024-04-24T23:26:33.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:33.605+0000 7f732fe00640 1 -- 172.21.15.6:0/2834606956 <== mon.2 v2:172.21.15.190:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6033 (secure 0 0 0) 0x7f7330059fa0 con 0x7f73381548a0 2024-04-24T23:26:33.605 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:26:33.605 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5081,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/1190759626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1190759626},{"type":"v1","addr":"172.21.15.6:6834","nonce":1190759626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":5249,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6838/2416336040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2416336040},{"type":"v1","addr":"172.21.15.190:6838","nonce":2416336040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":42},{"gid":5503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/3928731059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":3928731059},{"type":"v1","addr":"172.21.15.6:6837","nonce":3928731059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:25:42.695130+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5725,"mds_1":5599},"failed":[],"damaged":[],"stopped":[2,3,4],"info":{"gid_5599":{"gid":5599,"name":"d","rank":1,"incarnation":43,"state":"up:active","state_seq":25,"addr":"172.21.15.190:6839/2643233757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":2643233757},{"type":"v1","addr":"172.21.15.190:6839","nonce":2643233757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5725":{"gid":5725,"name":"f","rank":0,"incarnation":49,"state":"up:replay","state_seq":1,"addr":"172.21.15.190:6837/1151527362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":1151527362},{"type":"v1","addr":"172.21.15.190:6837","nonce":1151527362}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5599,"qdb_cluster":[5599]},"id":1}]} 2024-04-24T23:26:33.605 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 49 2024-04-24T23:26:33.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:33.610+0000 7f733d200640 1 -- 172.21.15.6:0/2834606956 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f731406be30 msgr2=0x7f731406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:26:33.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:33.610+0000 7f733d200640 1 --2- 172.21.15.6:0/2834606956 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f731406be30 0x7f731406e2f0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f7328001eb0 tx=0x7f7328009040 comp rx=0 tx=0).stop 2024-04-24T23:26:33.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:33.610+0000 7f733d200640 1 -- 172.21.15.6:0/2834606956 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f73381548a0 msgr2=0x7f733814cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:26:33.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:33.610+0000 7f733d200640 1 --2- 172.21.15.6:0/2834606956 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f73381548a0 0x7f733814cef0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f733000efc0 tx=0x7f733000c490 comp rx=0 tx=0).stop 2024-04-24T23:26:33.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:33.611+0000 7f733d200640 1 -- 172.21.15.6:0/2834606956 shutdown_connections 2024-04-24T23:26:33.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:33.611+0000 7f733d200640 1 --2- 172.21.15.6:0/2834606956 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f731406be30 0x7f731406e2f0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:26:33.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:33.611+0000 7f733d200640 1 --2- 172.21.15.6:0/2834606956 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f73381548a0 0x7f733814cef0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:26:33.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:33.611+0000 7f733d200640 1 --2- 172.21.15.6:0/2834606956 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f73380cd390 0x7f73381f4dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:26:33.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:33.611+0000 7f733d200640 1 --2- 172.21.15.6:0/2834606956 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f73380ccfb0 0x7f733815c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:26:33.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:33.611+0000 7f733d200640 1 -- 172.21.15.6:0/2834606956 >> 172.21.15.6:0/2834606956 conn(0x7f733806d9f0 msgr2=0x7f7338152e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:26:33.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:33.611+0000 7f733d200640 1 -- 172.21.15.6:0/2834606956 shutdown_connections 2024-04-24T23:26:33.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:33.613+0000 7f733d200640 1 -- 172.21.15.6:0/2834606956 wait complete. 2024-04-24T23:26:33.669 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-24T23:26:35.670 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:26:35.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:35.986+0000 7fc5e3e00640 1 Processor -- start 2024-04-24T23:26:35.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:35.987+0000 7fc5e3e00640 1 -- start start 2024-04-24T23:26:35.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:35.989+0000 7fc5e3e00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc5dc04b500 0x7fc5dc0dac60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:26:35.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:35.989+0000 7fc5e3e00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc5dc007760 con 0x7fc5dc034a40 2024-04-24T23:26:35.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:35.989+0000 7fc5e3e00640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc5dc006900 con 0x7fc5dc04b8e0 2024-04-24T23:26:35.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:35.989+0000 7fc5e3e00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fc5dc006be0 con 0x7fc5dc04b500 2024-04-24T23:26:35.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:35.995+0000 7fc5e2000640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7fc5dc04b8e0 0x7fc5dc005ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.6:38102/0 (socket says 172.21.15.6:38102) 2024-04-24T23:26:35.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:35.996+0000 7fc5e2a00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc5dc04b500 0x7fc5dc0dac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:26:35.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:35.996+0000 7fc5e2a00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc5dc04b500 0x7fc5dc0dac60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.6:51584/0 (socket says 172.21.15.6:51584) 2024-04-24T23:26:35.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:35.996+0000 7fc5e2a00640 1 -- 172.21.15.6:0/3567677971 learned_addr learned my addr 172.21.15.6:0/3567677971 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:26:36.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:36.001+0000 7fc5e2a00640 1 -- 172.21.15.6:0/3567677971 >> v1:172.21.15.190:6789/0 conn(0x7fc5dc04b8e0 legacy=0x7fc5dc005ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:26:36.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:36.001+0000 7fc5e2a00640 1 -- 172.21.15.6:0/3567677971 >> v1:172.21.15.6:6789/0 conn(0x7fc5dc034a40 legacy=0x7fc5dc034ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:26:36.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:36.001+0000 7fc5e2a00640 1 -- 172.21.15.6:0/3567677971 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5dc006ec0 con 0x7fc5dc04b500 2024-04-24T23:26:36.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:36.007+0000 7fc5e2000640 1 --1- 172.21.15.6:0/3567677971 >> v1:172.21.15.190:6789/0 conn(0x7fc5dc04b8e0 0x7fc5dc005ee0 :-1 s=CLOSED pgs=0 cs=0 l=0).handle_server_banner_and_identify state changed while learned_addr, mark_down or replacing must be happened just now 2024-04-24T23:26:36.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:36.008+0000 7fc5e2a00640 1 --2- 172.21.15.6:0/3567677971 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc5dc04b500 0x7fc5dc0dac60 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fc5d40098a0 tx=0x7fc5d402a330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=83b5049ab3b0b6cd server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:26:36.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:36.013+0000 7fc5e1600640 1 -- 172.21.15.6:0/3567677971 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5d4037070 con 0x7fc5dc04b500 2024-04-24T23:26:36.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:36.013+0000 7fc5e1600640 1 -- 172.21.15.6:0/3567677971 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5d4032640 con 0x7fc5dc04b500 2024-04-24T23:26:36.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:36.013+0000 7fc5e1600640 1 -- 172.21.15.6:0/3567677971 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5d4032a10 con 0x7fc5dc04b500 2024-04-24T23:26:36.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:36.019+0000 7fc5e3e00640 1 -- 172.21.15.6:0/3567677971 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc5dc04b500 msgr2=0x7fc5dc0dac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:26:36.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:36.019+0000 7fc5e3e00640 1 --2- 172.21.15.6:0/3567677971 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc5dc04b500 0x7fc5dc0dac60 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fc5d40098a0 tx=0x7fc5d402a330 comp rx=0 tx=0).stop 2024-04-24T23:26:36.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:36.024+0000 7fc5e3e00640 1 -- 172.21.15.6:0/3567677971 shutdown_connections 2024-04-24T23:26:36.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:36.024+0000 7fc5e3e00640 1 --2- 172.21.15.6:0/3567677971 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc5dc04b500 0x7fc5dc0dac60 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:26:36.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:36.024+0000 7fc5e3e00640 1 -- 172.21.15.6:0/3567677971 >> 172.21.15.6:0/3567677971 conn(0x7fc5dc050180 msgr2=0x7fc5dc050590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:26:36.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:36.030+0000 7fc5e3e00640 1 -- 172.21.15.6:0/3567677971 shutdown_connections 2024-04-24T23:26:36.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:36.031+0000 7fc5e3e00640 1 -- 172.21.15.6:0/3567677971 wait complete. 2024-04-24T23:26:36.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:36.036+0000 7fc5e3e00640 1 Processor -- start 2024-04-24T23:26:36.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:36.036+0000 7fc5e3e00640 1 -- start start 2024-04-24T23:26:36.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:36.038+0000 7fc5e3e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc5dc034a40 0x7fc5dc17d490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:26:36.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:36.038+0000 7fc5e3e00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc5dc04b500 0x7fc5dc0cc1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:26:36.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:36.038+0000 7fc5e3e00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc5dc04b8e0 0x7fc5dc0cffd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:26:36.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:36.039+0000 7fc5e3400640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc5dc04b8e0 0x7fc5dc0cffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:26:36.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:36.039+0000 7fc5e3400640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc5dc04b8e0 0x7fc5dc0cffd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:50866/0 (socket says 172.21.15.6:50866) 2024-04-24T23:26:36.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:36.039+0000 7fc5e3e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fc5dc0065f0 con 0x7fc5dc034a40 2024-04-24T23:26:36.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:36.039+0000 7fc5e3e00640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fc5dc007480 con 0x7fc5dc04b8e0 2024-04-24T23:26:36.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:36.039+0000 7fc5e3e00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fc5dc04bd00 con 0x7fc5dc04b500 2024-04-24T23:26:36.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:36.039+0000 7fc5e3400640 1 -- 172.21.15.6:0/3743399281 learned_addr learned my addr 172.21.15.6:0/3743399281 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:26:36.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:36.040+0000 7fc5e2a00640 1 --2- 172.21.15.6:0/3743399281 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc5dc034a40 0x7fc5dc17d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:26:36.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:36.040+0000 7fc5e2000640 1 --2- 172.21.15.6:0/3743399281 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc5dc04b500 0x7fc5dc0cc1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:26:36.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:36.041+0000 7fc5e3400640 1 -- 172.21.15.6:0/3743399281 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc5dc04b500 msgr2=0x7fc5dc0cc1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:26:36.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:36.041+0000 7fc5e3400640 1 --2- 172.21.15.6:0/3743399281 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc5dc04b500 0x7fc5dc0cc1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:26:36.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:36.041+0000 7fc5e3400640 1 -- 172.21.15.6:0/3743399281 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc5dc034a40 msgr2=0x7fc5dc17d490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:26:36.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:36.041+0000 7fc5e3400640 1 --2- 172.21.15.6:0/3743399281 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc5dc034a40 0x7fc5dc17d490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:26:36.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:36.044+0000 7fc5e3400640 1 -- 172.21.15.6:0/3743399281 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5dc17d9d0 con 0x7fc5dc04b8e0 2024-04-24T23:26:36.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:36.047+0000 7fc5e3400640 1 --2- 172.21.15.6:0/3743399281 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc5dc04b8e0 0x7fc5dc0cffd0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fc5e4000b60 tx=0x7fc5e4076b80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:26:36.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:36.047+0000 7fc5db400640 1 -- 172.21.15.6:0/3743399281 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5e4068b40 con 0x7fc5dc04b8e0 2024-04-24T23:26:36.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:36.047+0000 7fc5e3e00640 1 -- 172.21.15.6:0/3743399281 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5dc0cc720 con 0x7fc5dc04b8e0 2024-04-24T23:26:36.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:36.047+0000 7fc5e3e00640 1 -- 172.21.15.6:0/3743399281 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5dc0cd830 con 0x7fc5dc04b8e0 2024-04-24T23:26:36.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:36.048+0000 7fc5db400640 1 -- 172.21.15.6:0/3743399281 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5e4090030 con 0x7fc5dc04b8e0 2024-04-24T23:26:36.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:36.048+0000 7fc5db400640 1 -- 172.21.15.6:0/3743399281 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5e4068ce0 con 0x7fc5dc04b8e0 2024-04-24T23:26:36.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:36.052+0000 7fc5db400640 1 -- 172.21.15.6:0/3743399281 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fc5e4061b80 con 0x7fc5dc04b8e0 2024-04-24T23:26:36.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:36.054+0000 7fc5db400640 1 --2- 172.21.15.6:0/3743399281 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fc5c406be30 0x7fc5c406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:26:36.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:36.054+0000 7fc5db400640 1 -- 172.21.15.6:0/3743399281 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6934+0+0 (secure 0 0 0) 0x7fc5e4107650 con 0x7fc5dc04b8e0 2024-04-24T23:26:36.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:36.055+0000 7fc5e2a00640 1 --2- 172.21.15.6:0/3743399281 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fc5c406be30 0x7fc5c406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:26:36.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:36.061+0000 7fc5e3e00640 1 -- 172.21.15.6:0/3743399281 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc5bc005290 con 0x7fc5dc04b8e0 2024-04-24T23:26:36.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:36.056+0000 7fc5e2a00640 1 --2- 172.21.15.6:0/3743399281 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fc5c406be30 0x7fc5c406e2f0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fc5d4034000 tx=0x7fc5d402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:26:36.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:36.073+0000 7fc5db400640 1 -- 172.21.15.6:0/3743399281 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fc5dc17d9d0 con 0x7fc5dc04b8e0 2024-04-24T23:26:36.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:36.378+0000 7fc5e3e00640 1 -- 172.21.15.6:0/3743399281 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc5bc005580 con 0x7fc5dc04b8e0 2024-04-24T23:26:36.380 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:26:36.380 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5081,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/1190759626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1190759626},{"type":"v1","addr":"172.21.15.6:6834","nonce":1190759626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":5249,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6838/2416336040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2416336040},{"type":"v1","addr":"172.21.15.190:6838","nonce":2416336040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":42},{"gid":5503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/3928731059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":3928731059},{"type":"v1","addr":"172.21.15.6:6837","nonce":3928731059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:25:42.695130+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5725,"mds_1":5599},"failed":[],"damaged":[],"stopped":[2,3,4],"info":{"gid_5599":{"gid":5599,"name":"d","rank":1,"incarnation":43,"state":"up:active","state_seq":25,"addr":"172.21.15.190:6839/2643233757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":2643233757},{"type":"v1","addr":"172.21.15.190:6839","nonce":2643233757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5725":{"gid":5725,"name":"f","rank":0,"incarnation":49,"state":"up:replay","state_seq":1,"addr":"172.21.15.190:6837/1151527362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":1151527362},{"type":"v1","addr":"172.21.15.190:6837","nonce":1151527362}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5599,"qdb_cluster":[5599]},"id":1}]} 2024-04-24T23:26:36.380 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 49 2024-04-24T23:26:36.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:36.379+0000 7fc5db400640 1 -- 172.21.15.6:0/3743399281 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6033 (secure 0 0 0) 0x7fc5e40d53b0 con 0x7fc5dc04b8e0 2024-04-24T23:26:36.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:36.381+0000 7fc5e3e00640 1 -- 172.21.15.6:0/3743399281 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fc5c406be30 msgr2=0x7fc5c406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:26:36.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:36.381+0000 7fc5e3e00640 1 --2- 172.21.15.6:0/3743399281 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fc5c406be30 0x7fc5c406e2f0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fc5d4034000 tx=0x7fc5d402e000 comp rx=0 tx=0).stop 2024-04-24T23:26:36.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:36.382+0000 7fc5e3e00640 1 -- 172.21.15.6:0/3743399281 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc5dc04b8e0 msgr2=0x7fc5dc0cffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:26:36.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:36.382+0000 7fc5e3e00640 1 --2- 172.21.15.6:0/3743399281 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc5dc04b8e0 0x7fc5dc0cffd0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fc5e4000b60 tx=0x7fc5e4076b80 comp rx=0 tx=0).stop 2024-04-24T23:26:36.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:36.382+0000 7fc5e3e00640 1 -- 172.21.15.6:0/3743399281 shutdown_connections 2024-04-24T23:26:36.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:36.382+0000 7fc5e3e00640 1 --2- 172.21.15.6:0/3743399281 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fc5c406be30 0x7fc5c406e2f0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:26:36.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:36.382+0000 7fc5e3e00640 1 --2- 172.21.15.6:0/3743399281 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc5dc04b500 0x7fc5dc0cc1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:26:36.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:36.382+0000 7fc5e3e00640 1 --2- 172.21.15.6:0/3743399281 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc5dc04b8e0 0x7fc5dc0cffd0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:26:36.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:36.382+0000 7fc5e3e00640 1 --2- 172.21.15.6:0/3743399281 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc5dc034a40 0x7fc5dc17d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:26:36.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:36.382+0000 7fc5e3e00640 1 -- 172.21.15.6:0/3743399281 >> 172.21.15.6:0/3743399281 conn(0x7fc5dc050180 msgr2=0x7fc5dc04f2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:26:36.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:36.382+0000 7fc5e3e00640 1 -- 172.21.15.6:0/3743399281 shutdown_connections 2024-04-24T23:26:36.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:36.382+0000 7fc5e3e00640 1 -- 172.21.15.6:0/3743399281 wait complete. 2024-04-24T23:26:36.549 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-24T23:26:38.550 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:26:38.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:38.855+0000 7f464be00640 1 Processor -- start 2024-04-24T23:26:38.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:38.856+0000 7f464be00640 1 -- start start 2024-04-24T23:26:38.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:38.858+0000 7f464be00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4644159110 0x7f4644072420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:26:38.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:38.859+0000 7f464be00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f46440c81c0 0x7f4644162ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:26:38.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:38.859+0000 7f464be00640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f46440c7230 con 0x7f46440c7dc0 2024-04-24T23:26:38.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:38.859+0000 7f464be00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f46441594f0 con 0x7f46440c81c0 2024-04-24T23:26:38.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:38.859+0000 7f464be00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f46440c74d0 con 0x7f4644159110 2024-04-24T23:26:38.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:38.859+0000 7f4649400640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4644159110 0x7f4644072420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:26:38.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:38.859+0000 7f4649400640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4644159110 0x7f4644072420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.6:51594/0 (socket says 172.21.15.6:51594) 2024-04-24T23:26:38.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:38.859+0000 7f4649400640 1 -- 172.21.15.6:0/2818357857 learned_addr learned my addr 172.21.15.6:0/2818357857 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:26:38.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:38.859+0000 7f4649400640 1 -- 172.21.15.6:0/2818357857 >> v2:172.21.15.6:3300/0 conn(0x7f46440c81c0 msgr2=0x7f4644162ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:26:38.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:38.859+0000 7f4649400640 1 --2- 172.21.15.6:0/2818357857 >> v2:172.21.15.6:3300/0 conn(0x7f46440c81c0 0x7f4644162ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:26:38.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:38.859+0000 7f4649400640 1 -- 172.21.15.6:0/2818357857 >> v1:172.21.15.190:6789/0 conn(0x7f46440c7dc0 legacy=0x7f46441623a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-24T23:26:38.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:38.859+0000 7f4649400640 1 -- 172.21.15.6:0/2818357857 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f464406e440 con 0x7f4644159110 2024-04-24T23:26:38.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:38.861+0000 7f4649400640 1 --2- 172.21.15.6:0/2818357857 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4644159110 0x7f4644072420 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f46440c3610 tx=0x7f463802a280 comp rx=0 tx=0).ready entity=mon.2 client_cookie=14bceb112ae5fcd6 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:26:38.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:38.861+0000 7f4643e00640 1 -- 172.21.15.6:0/2818357857 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4638037070 con 0x7f4644159110 2024-04-24T23:26:38.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:38.862+0000 7f4643e00640 1 -- 172.21.15.6:0/2818357857 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4638032440 con 0x7f4644159110 2024-04-24T23:26:38.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:38.862+0000 7f4643e00640 1 -- 172.21.15.6:0/2818357857 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4638032810 con 0x7f4644159110 2024-04-24T23:26:38.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:38.862+0000 7f464be00640 1 -- 172.21.15.6:0/2818357857 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4644159110 msgr2=0x7f4644072420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:26:38.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:38.862+0000 7f464be00640 1 --2- 172.21.15.6:0/2818357857 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4644159110 0x7f4644072420 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f46440c3610 tx=0x7f463802a280 comp rx=0 tx=0).stop 2024-04-24T23:26:38.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:38.865+0000 7f464be00640 1 -- 172.21.15.6:0/2818357857 shutdown_connections 2024-04-24T23:26:38.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:38.865+0000 7f464be00640 1 --2- 172.21.15.6:0/2818357857 >> v2:172.21.15.6:3300/0 conn(0x7f46440c81c0 0x7f4644162ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:26:38.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:38.865+0000 7f464be00640 1 --2- 172.21.15.6:0/2818357857 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4644159110 0x7f4644072420 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:26:38.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:38.865+0000 7f464be00640 1 -- 172.21.15.6:0/2818357857 >> 172.21.15.6:0/2818357857 conn(0x7f464406b0e0 msgr2=0x7f464406b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:26:38.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:38.866+0000 7f464be00640 1 -- 172.21.15.6:0/2818357857 shutdown_connections 2024-04-24T23:26:38.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:38.867+0000 7f464be00640 1 -- 172.21.15.6:0/2818357857 wait complete. 2024-04-24T23:26:38.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:38.870+0000 7f464be00640 1 Processor -- start 2024-04-24T23:26:38.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:38.870+0000 7f464be00640 1 -- start start 2024-04-24T23:26:38.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:38.872+0000 7f464be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f46440c7dc0 0x7f464415c100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:26:38.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:38.872+0000 7f464be00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f46440c81c0 0x7f46441f8030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:26:38.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:38.873+0000 7f464be00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4644159110 0x7f46441ea7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:26:38.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:38.873+0000 7f4648a00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f46440c81c0 0x7f46441f8030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:26:38.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:38.873+0000 7f4649e00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4644159110 0x7f46441ea7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:26:38.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:38.873+0000 7f4649e00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4644159110 0x7f46441ea7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:50880/0 (socket says 172.21.15.6:50880) 2024-04-24T23:26:38.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:38.873+0000 7f4649e00640 1 -- 172.21.15.6:0/617895136 learned_addr learned my addr 172.21.15.6:0/617895136 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:26:38.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:38.874+0000 7f464be00640 1 -- 172.21.15.6:0/617895136 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f464406ece0 con 0x7f46440c7dc0 2024-04-24T23:26:38.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:38.874+0000 7f464be00640 1 -- 172.21.15.6:0/617895136 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f464406ea00 con 0x7f4644159110 2024-04-24T23:26:38.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:38.874+0000 7f464be00640 1 -- 172.21.15.6:0/617895136 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f4644072960 con 0x7f46440c81c0 2024-04-24T23:26:38.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:38.873+0000 7f4649400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f46440c7dc0 0x7f464415c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:26:38.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:38.874+0000 7f4648a00640 1 -- 172.21.15.6:0/617895136 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4644159110 msgr2=0x7f46441ea7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:26:38.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:38.874+0000 7f4648a00640 1 --2- 172.21.15.6:0/617895136 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4644159110 0x7f46441ea7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:26:38.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:38.874+0000 7f4648a00640 1 -- 172.21.15.6:0/617895136 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f46440c7dc0 msgr2=0x7f464415c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:26:38.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:38.875+0000 7f4648a00640 1 --2- 172.21.15.6:0/617895136 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f46440c7dc0 0x7f464415c100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:26:38.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:38.875+0000 7f4648a00640 1 -- 172.21.15.6:0/617895136 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f464415be60 con 0x7f46440c81c0 2024-04-24T23:26:38.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:38.875+0000 7f4649400640 1 --2- 172.21.15.6:0/617895136 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f46440c7dc0 0x7f464415c100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:26:38.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:38.875+0000 7f4648a00640 1 --2- 172.21.15.6:0/617895136 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f46440c81c0 0x7f46441f8030 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f463000b410 tx=0x7f463000b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:26:38.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:38.878+0000 7f4641600640 1 -- 172.21.15.6:0/617895136 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4630015020 con 0x7f46440c81c0 2024-04-24T23:26:38.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:38.879+0000 7f464be00640 1 -- 172.21.15.6:0/617895136 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f464415d850 con 0x7f46440c81c0 2024-04-24T23:26:38.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:38.879+0000 7f464be00640 1 -- 172.21.15.6:0/617895136 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f46441f88f0 con 0x7f46440c81c0 2024-04-24T23:26:38.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:38.879+0000 7f4641600640 1 -- 172.21.15.6:0/617895136 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4630011870 con 0x7f46440c81c0 2024-04-24T23:26:38.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:38.879+0000 7f4641600640 1 -- 172.21.15.6:0/617895136 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4630011c60 con 0x7f46440c81c0 2024-04-24T23:26:38.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:38.881+0000 7f464be00640 1 -- 172.21.15.6:0/617895136 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4620005290 con 0x7f46440c81c0 2024-04-24T23:26:38.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:38.883+0000 7f4641600640 1 -- 172.21.15.6:0/617895136 <== mon.2 v2:172.21.15.190:3301/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f4630018020 con 0x7f46440c81c0 2024-04-24T23:26:38.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:38.885+0000 7f4641600640 1 --2- 172.21.15.6:0/617895136 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f462406be30 0x7f462406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:26:38.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:38.885+0000 7f4641600640 1 -- 172.21.15.6:0/617895136 <== mon.2 v2:172.21.15.190:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6934+0+0 (secure 0 0 0) 0x7f463008ab70 con 0x7f46440c81c0 2024-04-24T23:26:38.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:38.886+0000 7f4649400640 1 --2- 172.21.15.6:0/617895136 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f462406be30 0x7f462406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:26:38.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:38.890+0000 7f4649400640 1 --2- 172.21.15.6:0/617895136 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f462406be30 0x7f462406e2f0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f4638034000 tx=0x7f463802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:26:38.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:38.895+0000 7f4641600640 1 -- 172.21.15.6:0/617895136 <== mon.2 v2:172.21.15.190:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f46300588d0 con 0x7f46440c81c0 2024-04-24T23:26:39.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:39.425+0000 7f464be00640 1 -- 172.21.15.6:0/617895136 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4620005580 con 0x7f46440c81c0 2024-04-24T23:26:39.430 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:26:39.430 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5081,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/1190759626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1190759626},{"type":"v1","addr":"172.21.15.6:6834","nonce":1190759626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":5249,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6838/2416336040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2416336040},{"type":"v1","addr":"172.21.15.190:6838","nonce":2416336040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":42},{"gid":5503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/3928731059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":3928731059},{"type":"v1","addr":"172.21.15.6:6837","nonce":3928731059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:25:42.695130+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5725,"mds_1":5599},"failed":[],"damaged":[],"stopped":[2,3,4],"info":{"gid_5599":{"gid":5599,"name":"d","rank":1,"incarnation":43,"state":"up:active","state_seq":25,"addr":"172.21.15.190:6839/2643233757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":2643233757},{"type":"v1","addr":"172.21.15.190:6839","nonce":2643233757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5725":{"gid":5725,"name":"f","rank":0,"incarnation":49,"state":"up:replay","state_seq":1,"addr":"172.21.15.190:6837/1151527362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":1151527362},{"type":"v1","addr":"172.21.15.190:6837","nonce":1151527362}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5599,"qdb_cluster":[5599]},"id":1}]} 2024-04-24T23:26:39.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:39.429+0000 7f4641600640 1 -- 172.21.15.6:0/617895136 <== mon.2 v2:172.21.15.190:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6033 (secure 0 0 0) 0x7f463005cbd0 con 0x7f46440c81c0 2024-04-24T23:26:39.431 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 49 2024-04-24T23:26:39.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:39.433+0000 7f464be00640 1 -- 172.21.15.6:0/617895136 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f462406be30 msgr2=0x7f462406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:26:39.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:39.433+0000 7f464be00640 1 --2- 172.21.15.6:0/617895136 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f462406be30 0x7f462406e2f0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f4638034000 tx=0x7f463802e000 comp rx=0 tx=0).stop 2024-04-24T23:26:39.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:39.433+0000 7f464be00640 1 -- 172.21.15.6:0/617895136 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f46440c81c0 msgr2=0x7f46441f8030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:26:39.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:39.433+0000 7f464be00640 1 --2- 172.21.15.6:0/617895136 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f46440c81c0 0x7f46441f8030 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f463000b410 tx=0x7f463000b8d0 comp rx=0 tx=0).stop 2024-04-24T23:26:39.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:39.434+0000 7f464be00640 1 -- 172.21.15.6:0/617895136 shutdown_connections 2024-04-24T23:26:39.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:39.434+0000 7f464be00640 1 --2- 172.21.15.6:0/617895136 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f462406be30 0x7f462406e2f0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:26:39.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:39.434+0000 7f464be00640 1 --2- 172.21.15.6:0/617895136 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f46440c81c0 0x7f46441f8030 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:26:39.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:39.434+0000 7f464be00640 1 --2- 172.21.15.6:0/617895136 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4644159110 0x7f46441ea7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:26:39.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:39.434+0000 7f464be00640 1 --2- 172.21.15.6:0/617895136 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f46440c7dc0 0x7f464415c100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:26:39.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:39.434+0000 7f464be00640 1 -- 172.21.15.6:0/617895136 >> 172.21.15.6:0/617895136 conn(0x7f464406b0e0 msgr2=0x7f4644074630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:26:39.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:39.434+0000 7f464be00640 1 -- 172.21.15.6:0/617895136 shutdown_connections 2024-04-24T23:26:39.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:39.434+0000 7f464be00640 1 -- 172.21.15.6:0/617895136 wait complete. 2024-04-24T23:26:39.465 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-24T23:26:41.466 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:26:41.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:41.800+0000 7f4f26200640 1 Processor -- start 2024-04-24T23:26:41.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:41.800+0000 7f4f26200640 1 -- start start 2024-04-24T23:26:41.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:41.801+0000 7f4f26200640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f4f200c8070 0x7f4f20157900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:26:41.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:41.801+0000 7f4f26200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f4f2006e3a0 0x7f4f2006e830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:26:41.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:41.801+0000 7f4f26200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f200c84c0 con 0x7f4f201511a0 2024-04-24T23:26:41.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:41.801+0000 7f4f26200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f4f200c97d0 con 0x7f4f2006e3a0 2024-04-24T23:26:41.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:41.801+0000 7f4f26200640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f4f20157e40 con 0x7f4f200c8070 2024-04-24T23:26:41.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:41.804+0000 7f4f1f400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f4f201511a0 0x7f4f200c90c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:55216/0 (socket says 172.21.15.6:55216) 2024-04-24T23:26:41.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:41.804+0000 7f4f1f400640 1 -- 172.21.15.6:0/201548974 learned_addr learned my addr 172.21.15.6:0/201548974 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:26:41.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:41.805+0000 7f4f1fe00640 1 --2- 172.21.15.6:0/201548974 >> v2:172.21.15.6:3300/0 conn(0x7f4f2006e3a0 0x7f4f2006e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:26:41.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:41.805+0000 7f4f1e000640 1 -- 172.21.15.6:0/201548974 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1320827385 0 0) 0x7f4f10001490 con 0x7f4f201511a0 2024-04-24T23:26:41.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:41.805+0000 7f4f1e000640 1 -- 172.21.15.6:0/201548974 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4f04003520 con 0x7f4f201511a0 2024-04-24T23:26:41.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:41.806+0000 7f4f1ea00640 1 --2- 172.21.15.6:0/201548974 >> v2:172.21.15.190:3300/0 conn(0x7f4f200c8070 0x7f4f20157900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:26:41.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:41.806+0000 7f4f1e000640 1 -- 172.21.15.6:0/201548974 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2771580657 0 0) 0x7f4f04003520 con 0x7f4f201511a0 2024-04-24T23:26:41.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:41.806+0000 7f4f1e000640 1 -- 172.21.15.6:0/201548974 >> v2:172.21.15.190:3300/0 conn(0x7f4f200c8070 msgr2=0x7f4f20157900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:26:41.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:41.806+0000 7f4f1e000640 1 --2- 172.21.15.6:0/201548974 >> v2:172.21.15.190:3300/0 conn(0x7f4f200c8070 0x7f4f20157900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:26:41.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:41.806+0000 7f4f1e000640 1 -- 172.21.15.6:0/201548974 >> v2:172.21.15.6:3300/0 conn(0x7f4f2006e3a0 msgr2=0x7f4f2006e830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:26:41.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:41.806+0000 7f4f1e000640 1 --2- 172.21.15.6:0/201548974 >> v2:172.21.15.6:3300/0 conn(0x7f4f2006e3a0 0x7f4f2006e830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:26:41.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:41.806+0000 7f4f1e000640 1 -- 172.21.15.6:0/201548974 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f200c6d90 con 0x7f4f201511a0 2024-04-24T23:26:41.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:41.807+0000 7f4f1e000640 1 -- 172.21.15.6:0/201548974 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3463268701 0 0) 0x7f4f10001bc0 con 0x7f4f201511a0 2024-04-24T23:26:41.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:41.807+0000 7f4f1e000640 1 -- 172.21.15.6:0/201548974 >> v1:172.21.15.6:6789/0 conn(0x7f4f201511a0 legacy=0x7f4f200c90c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:26:41.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:41.807+0000 7f4f1e000640 1 --2- 172.21.15.6:0/201548974 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4f04003a30 0x7f4f04003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:26:41.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:41.807+0000 7f4f1e000640 1 --2- 172.21.15.6:0/201548974 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4f04004800 0x7f4f04004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:26:41.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:41.807+0000 7f4f1e000640 1 --2- 172.21.15.6:0/201548974 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4f040051c0 0x7f4f0400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:26:41.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:41.807+0000 7f4f1e000640 1 -- 172.21.15.6:0/201548974 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f4f200c97d0 con 0x7f4f04003a30 2024-04-24T23:26:41.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:41.807+0000 7f4f1e000640 1 -- 172.21.15.6:0/201548974 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f4f20157e40 con 0x7f4f040051c0 2024-04-24T23:26:41.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:41.807+0000 7f4f1e000640 1 -- 172.21.15.6:0/201548974 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f4f0400c090 con 0x7f4f04004800 2024-04-24T23:26:41.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:41.809+0000 7f4f1fe00640 1 --2- 172.21.15.6:0/201548974 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4f040051c0 0x7f4f0400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:26:41.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:41.809+0000 7f4f1f400640 1 --2- 172.21.15.6:0/201548974 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4f04003a30 0x7f4f04003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:26:41.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:41.810+0000 7f4f1fe00640 1 -- 172.21.15.6:0/201548974 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4f04004800 msgr2=0x7f4f04004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:26:41.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:41.810+0000 7f4f1fe00640 1 --2- 172.21.15.6:0/201548974 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4f04004800 0x7f4f04004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:26:41.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:41.810+0000 7f4f1fe00640 1 -- 172.21.15.6:0/201548974 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4f04003a30 msgr2=0x7f4f04003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:26:41.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:41.810+0000 7f4f1fe00640 1 --2- 172.21.15.6:0/201548974 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4f04003a30 0x7f4f04003e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:26:41.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:41.810+0000 7f4f1fe00640 1 -- 172.21.15.6:0/201548974 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f04003520 con 0x7f4f040051c0 2024-04-24T23:26:41.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:41.810+0000 7f4f1f400640 1 --2- 172.21.15.6:0/201548974 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4f04003a30 0x7f4f04003e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:26:41.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:41.811+0000 7f4f1fe00640 1 -- 172.21.15.6:0/201548974 reap_dead start 2024-04-24T23:26:41.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:41.812+0000 7f4f1fe00640 1 --2- 172.21.15.6:0/201548974 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4f040051c0 0x7f4f0400b9c0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f4f1800ba30 tx=0x7f4f04005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fa371cc687abb7ac server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:26:41.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:41.812+0000 7f4f1e000640 1 -- 172.21.15.6:0/201548974 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f18030070 con 0x7f4f040051c0 2024-04-24T23:26:41.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:41.812+0000 7f4f1e000640 1 -- 172.21.15.6:0/201548974 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f18004020 con 0x7f4f040051c0 2024-04-24T23:26:41.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:41.812+0000 7f4f1e000640 1 -- 172.21.15.6:0/201548974 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f1800b600 con 0x7f4f040051c0 2024-04-24T23:26:41.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:41.813+0000 7f4f26200640 1 -- 172.21.15.6:0/201548974 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4f040051c0 msgr2=0x7f4f0400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:26:41.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:41.813+0000 7f4f26200640 1 --2- 172.21.15.6:0/201548974 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4f040051c0 0x7f4f0400b9c0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f4f1800ba30 tx=0x7f4f04005830 comp rx=0 tx=0).stop 2024-04-24T23:26:41.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:41.814+0000 7f4f26200640 1 -- 172.21.15.6:0/201548974 shutdown_connections 2024-04-24T23:26:41.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:41.814+0000 7f4f26200640 1 --2- 172.21.15.6:0/201548974 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4f040051c0 0x7f4f0400b9c0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:26:41.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:41.814+0000 7f4f26200640 1 -- 172.21.15.6:0/201548974 >> 172.21.15.6:0/201548974 conn(0x7f4f2006b0e0 msgr2=0x7f4f2006b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:26:41.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:41.814+0000 7f4f26200640 1 -- 172.21.15.6:0/201548974 shutdown_connections 2024-04-24T23:26:41.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:41.814+0000 7f4f26200640 1 -- 172.21.15.6:0/201548974 wait complete. 2024-04-24T23:26:41.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:41.817+0000 7f4f26200640 1 Processor -- start 2024-04-24T23:26:41.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:41.818+0000 7f4f26200640 1 -- start start 2024-04-24T23:26:41.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:41.822+0000 7f4f26200640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4f040051c0 0x7f4f20159cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:26:41.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:41.822+0000 7f4f26200640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4f20155840 0x7f4f201f2b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:26:41.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:41.822+0000 7f4f1f400640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4f040051c0 0x7f4f20159cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:26:41.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:41.823+0000 7f4f1f400640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4f040051c0 0x7f4f20159cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:50926/0 (socket says 172.21.15.6:50926) 2024-04-24T23:26:41.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:41.823+0000 7f4f1f400640 1 -- 172.21.15.6:0/1012815178 learned_addr learned my addr 172.21.15.6:0/1012815178 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:26:41.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:41.823+0000 7f4f1ea00640 1 --2- 172.21.15.6:0/1012815178 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4f20155840 0x7f4f201f2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:26:41.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:41.823+0000 7f4f26200640 1 --2- 172.21.15.6:0/1012815178 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4f201f3040 0x7f4f20153ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:26:41.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:41.824+0000 7f4f26200640 1 -- 172.21.15.6:0/1012815178 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f4f200c7630 con 0x7f4f201f3040 2024-04-24T23:26:41.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:41.824+0000 7f4f26200640 1 -- 172.21.15.6:0/1012815178 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f4f200c7350 con 0x7f4f040051c0 2024-04-24T23:26:41.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:41.824+0000 7f4f26200640 1 -- 172.21.15.6:0/1012815178 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f4f200c7910 con 0x7f4f20155840 2024-04-24T23:26:41.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:41.829+0000 7f4f1fe00640 1 --2- 172.21.15.6:0/1012815178 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4f201f3040 0x7f4f20153ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:26:41.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:41.829+0000 7f4f1ea00640 1 -- 172.21.15.6:0/1012815178 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4f040051c0 msgr2=0x7f4f20159cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:26:41.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:41.829+0000 7f4f1ea00640 1 --2- 172.21.15.6:0/1012815178 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4f040051c0 0x7f4f20159cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:26:41.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:41.829+0000 7f4f1ea00640 1 -- 172.21.15.6:0/1012815178 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4f201f3040 msgr2=0x7f4f20153ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:26:41.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:41.829+0000 7f4f1ea00640 1 --2- 172.21.15.6:0/1012815178 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4f201f3040 0x7f4f20153ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:26:41.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:41.829+0000 7f4f1ea00640 1 -- 172.21.15.6:0/1012815178 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f201f34c0 con 0x7f4f20155840 2024-04-24T23:26:41.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:41.830+0000 7f4f1f400640 1 --2- 172.21.15.6:0/1012815178 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4f040051c0 0x7f4f20159cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:26:41.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:41.833+0000 7f4f1ea00640 1 --2- 172.21.15.6:0/1012815178 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4f20155840 0x7f4f201f2b00 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f4f0c00e770 tx=0x7f4f0c00ec30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:26:41.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:41.836+0000 7f4f17400640 1 -- 172.21.15.6:0/1012815178 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f0c017070 con 0x7f4f20155840 2024-04-24T23:26:41.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:41.836+0000 7f4f17400640 1 -- 172.21.15.6:0/1012815178 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f0c00c730 con 0x7f4f20155840 2024-04-24T23:26:41.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:41.836+0000 7f4f17400640 1 -- 172.21.15.6:0/1012815178 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f0c00cb20 con 0x7f4f20155840 2024-04-24T23:26:41.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:41.837+0000 7f4f26200640 1 -- 172.21.15.6:0/1012815178 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f200c6350 con 0x7f4f20155840 2024-04-24T23:26:41.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:41.837+0000 7f4f26200640 1 -- 172.21.15.6:0/1012815178 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f2015b1b0 con 0x7f4f20155840 2024-04-24T23:26:41.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:41.840+0000 7f4f26200640 1 -- 172.21.15.6:0/1012815178 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f200767b0 con 0x7f4f20155840 2024-04-24T23:26:41.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:41.847+0000 7f4f17400640 1 -- 172.21.15.6:0/1012815178 <== mon.2 v2:172.21.15.190:3301/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f4f0c01a070 con 0x7f4f20155840 2024-04-24T23:26:41.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:41.851+0000 7f4f17400640 1 --2- 172.21.15.6:0/1012815178 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f4f0006bf00 0x7f4f0006e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:26:41.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:41.851+0000 7f4f17400640 1 -- 172.21.15.6:0/1012815178 <== mon.2 v2:172.21.15.190:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6934+0+0 (secure 0 0 0) 0x7f4f0c08b360 con 0x7f4f20155840 2024-04-24T23:26:41.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:41.852+0000 7f4f17400640 1 -- 172.21.15.6:0/1012815178 <== mon.2 v2:172.21.15.190:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f4f201f34c0 con 0x7f4f20155840 2024-04-24T23:26:41.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:41.855+0000 7f4f1f400640 1 --2- 172.21.15.6:0/1012815178 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f4f0006bf00 0x7f4f0006e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:26:41.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:41.856+0000 7f4f1f400640 1 --2- 172.21.15.6:0/1012815178 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f4f0006bf00 0x7f4f0006e3c0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f4f100013d0 tx=0x7f4f10001a00 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:26:42.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:42.418+0000 7f4f26200640 1 -- 172.21.15.6:0/1012815178 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4f20071000 con 0x7f4f20155840 2024-04-24T23:26:42.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:42.422+0000 7f4f17400640 1 -- 172.21.15.6:0/1012815178 <== mon.2 v2:172.21.15.190:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6033 (secure 0 0 0) 0x7f4f0c059260 con 0x7f4f20155840 2024-04-24T23:26:42.434 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:26:42.434 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5081,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/1190759626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1190759626},{"type":"v1","addr":"172.21.15.6:6834","nonce":1190759626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":5249,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6838/2416336040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2416336040},{"type":"v1","addr":"172.21.15.190:6838","nonce":2416336040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":42},{"gid":5503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/3928731059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":3928731059},{"type":"v1","addr":"172.21.15.6:6837","nonce":3928731059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:25:42.695130+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5725,"mds_1":5599},"failed":[],"damaged":[],"stopped":[2,3,4],"info":{"gid_5599":{"gid":5599,"name":"d","rank":1,"incarnation":43,"state":"up:active","state_seq":25,"addr":"172.21.15.190:6839/2643233757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":2643233757},{"type":"v1","addr":"172.21.15.190:6839","nonce":2643233757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5725":{"gid":5725,"name":"f","rank":0,"incarnation":49,"state":"up:replay","state_seq":1,"addr":"172.21.15.190:6837/1151527362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":1151527362},{"type":"v1","addr":"172.21.15.190:6837","nonce":1151527362}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5599,"qdb_cluster":[5599]},"id":1}]} 2024-04-24T23:26:42.434 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 49 2024-04-24T23:26:42.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:42.436+0000 7f4f26200640 1 -- 172.21.15.6:0/1012815178 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f4f0006bf00 msgr2=0x7f4f0006e3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:26:42.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:42.436+0000 7f4f26200640 1 --2- 172.21.15.6:0/1012815178 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f4f0006bf00 0x7f4f0006e3c0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f4f100013d0 tx=0x7f4f10001a00 comp rx=0 tx=0).stop 2024-04-24T23:26:42.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:42.436+0000 7f4f26200640 1 -- 172.21.15.6:0/1012815178 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4f20155840 msgr2=0x7f4f201f2b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:26:42.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:42.436+0000 7f4f26200640 1 --2- 172.21.15.6:0/1012815178 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4f20155840 0x7f4f201f2b00 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f4f0c00e770 tx=0x7f4f0c00ec30 comp rx=0 tx=0).stop 2024-04-24T23:26:42.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:42.441+0000 7f4f26200640 1 -- 172.21.15.6:0/1012815178 shutdown_connections 2024-04-24T23:26:42.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:42.441+0000 7f4f26200640 1 --2- 172.21.15.6:0/1012815178 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f4f0006bf00 0x7f4f0006e3c0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:26:42.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:42.441+0000 7f4f26200640 1 --2- 172.21.15.6:0/1012815178 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4f20155840 0x7f4f201f2b00 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:26:42.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:42.441+0000 7f4f26200640 1 --2- 172.21.15.6:0/1012815178 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4f201f3040 0x7f4f20153ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:26:42.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:42.441+0000 7f4f26200640 1 --2- 172.21.15.6:0/1012815178 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4f040051c0 0x7f4f20159cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:26:42.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:42.441+0000 7f4f26200640 1 -- 172.21.15.6:0/1012815178 >> 172.21.15.6:0/1012815178 conn(0x7f4f2006b0e0 msgr2=0x7f4f2006f310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:26:42.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:42.442+0000 7f4f26200640 1 -- 172.21.15.6:0/1012815178 shutdown_connections 2024-04-24T23:26:42.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:42.444+0000 7f4f26200640 1 -- 172.21.15.6:0/1012815178 wait complete. 2024-04-24T23:26:42.496 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 49, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-24T23:15:36.400716+0000", "damaged": [], "data_pools": [ 3, 4 ], "enabled": true, "epoch": 49, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_5599": { "addr": "172.21.15.190:6839/2643233757", "addrs": { "addrvec": [ { "addr": "172.21.15.190:6836", "nonce": 2643233757, "type": "v2" }, { "addr": "172.21.15.190:6839", "nonce": 2643233757, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5599, "incarnation": 43, "join_fscid": -1, "name": "d", "rank": 1, "state": "up:active", "state_seq": 25 }, "gid_5725": { "addr": "172.21.15.190:6837/1151527362", "addrs": { "addrvec": [ { "addr": "172.21.15.190:6834", "nonce": 1151527362, "type": "v2" }, { "addr": "172.21.15.190:6837", "nonce": 1151527362, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5725, "incarnation": 49, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 30, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-04-24T23:25:42.695130+0000", "qdb_cluster": [ 5599 ], "qdb_leader": 5599, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 2, 3, 4 ], "tableserver": 0, "up": { "mds_0": 5725, "mds_1": 5599 } } } ], "standbys": [ { "addr": "172.21.15.6:6834/1190759626", "addrs": { "addrvec": [ { "addr": "172.21.15.6:6832", "nonce": 1190759626, "type": "v2" }, { "addr": "172.21.15.6:6834", "nonce": 1190759626, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 30, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5081, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.190:6838/2416336040", "addrs": { "addrvec": [ { "addr": "172.21.15.190:6835", "nonce": 2416336040, "type": "v2" }, { "addr": "172.21.15.190:6838", "nonce": 2416336040, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 42, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5249, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.6:6837/3928731059", "addrs": { "addrvec": [ { "addr": "172.21.15.6:6836", "nonce": 3928731059, "type": "v2" }, { "addr": "172.21.15.6:6837", "nonce": 3928731059, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 35, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5503, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-04-24T23:26:44.497 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:26:44.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:44.773+0000 7fd01a600640 1 Processor -- start 2024-04-24T23:26:44.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:44.773+0000 7fd01a600640 1 -- start start 2024-04-24T23:26:44.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:44.775+0000 7fd01a600640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd014152d80 0x7fd0140c8d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:26:44.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:44.775+0000 7fd01a600640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd014159770 con 0x7fd0140c8500 2024-04-24T23:26:44.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:44.775+0000 7fd01a600640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd0140c3470 con 0x7fd0141592a0 2024-04-24T23:26:44.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:44.775+0000 7fd01a600640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fd0140c5500 con 0x7fd014152d80 2024-04-24T23:26:44.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:44.776+0000 7fd012a00640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7fd0141592a0 0x7fd01415c5d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.6:38112/0 (socket says 172.21.15.6:38112) 2024-04-24T23:26:44.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:44.776+0000 7fd012a00640 1 -- 172.21.15.6:0/528040682 learned_addr learned my addr 172.21.15.6:0/528040682 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:26:44.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:44.777+0000 7fd012000640 1 -- 172.21.15.6:0/528040682 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4192026304 0 0) 0x7fd0140c3470 con 0x7fd0141592a0 2024-04-24T23:26:44.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:44.777+0000 7fd012000640 1 -- 172.21.15.6:0/528040682 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcff8003520 con 0x7fd0141592a0 2024-04-24T23:26:44.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:44.777+0000 7fd013400640 1 --2- 172.21.15.6:0/528040682 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd014152d80 0x7fd0140c8d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:26:44.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:44.777+0000 7fd012000640 1 -- 172.21.15.6:0/528040682 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1004958848 0 0) 0x7fcff8003520 con 0x7fd0141592a0 2024-04-24T23:26:44.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:44.777+0000 7fd012000640 1 -- 172.21.15.6:0/528040682 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd014152d80 msgr2=0x7fd0140c8d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:26:44.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:44.777+0000 7fd012000640 1 --2- 172.21.15.6:0/528040682 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd014152d80 0x7fd0140c8d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:26:44.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:44.777+0000 7fd012000640 1 -- 172.21.15.6:0/528040682 >> v1:172.21.15.6:6789/0 conn(0x7fd0140c8500 legacy=0x7fd014070c10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-24T23:26:44.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:44.778+0000 7fd012000640 1 -- 172.21.15.6:0/528040682 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0140c3750 con 0x7fd0141592a0 2024-04-24T23:26:44.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:44.778+0000 7fd012000640 1 -- 172.21.15.6:0/528040682 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3463268701 0 0) 0x7fd000001670 con 0x7fd0141592a0 2024-04-24T23:26:44.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:44.778+0000 7fd012000640 1 -- 172.21.15.6:0/528040682 >> v1:172.21.15.190:6789/0 conn(0x7fd0141592a0 legacy=0x7fd01415c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:26:44.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:44.778+0000 7fd012000640 1 --2- 172.21.15.6:0/528040682 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcff80039b0 0x7fcff8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:26:44.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:44.778+0000 7fd012000640 1 --2- 172.21.15.6:0/528040682 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcff8004800 0x7fcff8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:26:44.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:44.778+0000 7fd012000640 1 --2- 172.21.15.6:0/528040682 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcff80052e0 0x7fcff800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:26:44.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:44.778+0000 7fd012000640 1 -- 172.21.15.6:0/528040682 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fd0140c5500 con 0x7fcff80052e0 2024-04-24T23:26:44.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:44.778+0000 7fd012000640 1 -- 172.21.15.6:0/528040682 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fcff80105d0 con 0x7fcff8004800 2024-04-24T23:26:44.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:44.778+0000 7fd012000640 1 -- 172.21.15.6:0/528040682 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fcff80107d0 con 0x7fcff80039b0 2024-04-24T23:26:44.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:44.778+0000 7fd012a00640 1 --2- 172.21.15.6:0/528040682 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcff8004800 0x7fcff8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:26:44.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:44.779+0000 7fd013400640 1 --2- 172.21.15.6:0/528040682 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd014152d80 0x7fd0140c8d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:26:44.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:44.779+0000 7fd012a00640 1 -- 172.21.15.6:0/528040682 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcff80039b0 msgr2=0x7fcff8003dd0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-24T23:26:44.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:44.779+0000 7fd012a00640 1 --2- 172.21.15.6:0/528040682 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcff80039b0 0x7fcff8003dd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:26:44.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:44.779+0000 7fd012a00640 1 -- 172.21.15.6:0/528040682 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcff80052e0 msgr2=0x7fcff800b9c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-24T23:26:44.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:44.779+0000 7fd012a00640 1 --2- 172.21.15.6:0/528040682 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcff80052e0 0x7fcff800b9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:26:44.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:44.779+0000 7fd012a00640 1 -- 172.21.15.6:0/528040682 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0140c3750 con 0x7fcff8004800 2024-04-24T23:26:44.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:44.780+0000 7fd012a00640 1 --2- 172.21.15.6:0/528040682 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcff8004800 0x7fcff8004c80 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fd00000c260 tx=0x7fcff8005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2876ee3a0e493792 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:26:44.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:44.781+0000 7fd012000640 1 -- 172.21.15.6:0/528040682 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd000007d70 con 0x7fcff8004800 2024-04-24T23:26:44.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:44.781+0000 7fd012000640 1 -- 172.21.15.6:0/528040682 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd00002e070 con 0x7fcff8004800 2024-04-24T23:26:44.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:44.781+0000 7fd012000640 1 -- 172.21.15.6:0/528040682 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd000005d10 con 0x7fcff8004800 2024-04-24T23:26:44.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:44.783+0000 7fd01a600640 1 -- 172.21.15.6:0/528040682 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcff8004800 msgr2=0x7fcff8004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:26:44.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:44.783+0000 7fd01a600640 1 --2- 172.21.15.6:0/528040682 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcff8004800 0x7fcff8004c80 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fd00000c260 tx=0x7fcff8005830 comp rx=0 tx=0).stop 2024-04-24T23:26:44.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:44.784+0000 7fd013e00640 1 -- 172.21.15.6:0/528040682 reap_dead start 2024-04-24T23:26:44.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:44.784+0000 7fd01a600640 1 -- 172.21.15.6:0/528040682 shutdown_connections 2024-04-24T23:26:44.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:44.784+0000 7fd01a600640 1 -- 172.21.15.6:0/528040682 >> 172.21.15.6:0/528040682 conn(0x7fd014070280 msgr2=0x7fd01406f320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:26:44.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:44.784+0000 7fd01a600640 1 -- 172.21.15.6:0/528040682 shutdown_connections 2024-04-24T23:26:44.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:44.784+0000 7fd01a600640 1 -- 172.21.15.6:0/528040682 wait complete. 2024-04-24T23:26:44.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:44.787+0000 7fd01a600640 1 Processor -- start 2024-04-24T23:26:44.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:44.787+0000 7fd01a600640 1 -- start start 2024-04-24T23:26:44.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:44.789+0000 7fd01a600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd0140ce9f0 0x7fd0140b3410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:26:44.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:44.789+0000 7fd01a600640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd0140cedd0 0x7fd0141f85f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:26:44.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:44.789+0000 7fd01a600640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd01414aec0 0x7fd01414c0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:26:44.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:44.790+0000 7fd01a600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fd01415bdd0 con 0x7fd0140ce9f0 2024-04-24T23:26:44.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:44.790+0000 7fd013e00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd01414aec0 0x7fd01414c0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:26:44.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:44.790+0000 7fd013e00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd01414aec0 0x7fd01414c0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.6:51650/0 (socket says 172.21.15.6:51650) 2024-04-24T23:26:44.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:44.790+0000 7fd013e00640 1 -- 172.21.15.6:0/997299276 learned_addr learned my addr 172.21.15.6:0/997299276 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:26:44.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:44.790+0000 7fd01a600640 1 -- 172.21.15.6:0/997299276 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fd01415c140 con 0x7fd0140cedd0 2024-04-24T23:26:44.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:44.790+0000 7fd01a600640 1 -- 172.21.15.6:0/997299276 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fd01415cce0 con 0x7fd01414aec0 2024-04-24T23:26:44.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:44.790+0000 7fd012a00640 1 --2- 172.21.15.6:0/997299276 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd0140cedd0 0x7fd0141f85f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:26:44.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:44.791+0000 7fd013400640 1 --2- 172.21.15.6:0/997299276 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd0140ce9f0 0x7fd0140b3410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:26:44.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:44.791+0000 7fd013e00640 1 -- 172.21.15.6:0/997299276 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd0140cedd0 msgr2=0x7fd0141f85f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:26:44.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:44.791+0000 7fd013e00640 1 --2- 172.21.15.6:0/997299276 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd0140cedd0 0x7fd0141f85f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:26:44.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:44.791+0000 7fd013e00640 1 -- 172.21.15.6:0/997299276 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd0140ce9f0 msgr2=0x7fd0140b3410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:26:44.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:44.791+0000 7fd013e00640 1 --2- 172.21.15.6:0/997299276 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd0140ce9f0 0x7fd0140b3410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:26:44.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:44.791+0000 7fd013e00640 1 -- 172.21.15.6:0/997299276 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0140cd600 con 0x7fd01414aec0 2024-04-24T23:26:44.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:44.792+0000 7fd013400640 1 --2- 172.21.15.6:0/997299276 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd0140ce9f0 0x7fd0140b3410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:26:44.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:44.793+0000 7fd013e00640 1 --2- 172.21.15.6:0/997299276 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd01414aec0 0x7fd01414c0b0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fd014071b20 tx=0x7fd00c00c8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:26:44.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:44.798+0000 7fd00b400640 1 -- 172.21.15.6:0/997299276 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd00c003970 con 0x7fd01414aec0 2024-04-24T23:26:44.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:44.798+0000 7fd00b400640 1 -- 172.21.15.6:0/997299276 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd00c003b10 con 0x7fd01414aec0 2024-04-24T23:26:44.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:44.798+0000 7fd00b400640 1 -- 172.21.15.6:0/997299276 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd00c004020 con 0x7fd01414aec0 2024-04-24T23:26:44.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:44.799+0000 7fd01a600640 1 -- 172.21.15.6:0/997299276 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0140cd8e0 con 0x7fd01414aec0 2024-04-24T23:26:44.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:44.799+0000 7fd01a600640 1 -- 172.21.15.6:0/997299276 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0140cdea0 con 0x7fd01414aec0 2024-04-24T23:26:44.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:44.801+0000 7fd01a600640 1 -- 172.21.15.6:0/997299276 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd0140767b0 con 0x7fd01414aec0 2024-04-24T23:26:44.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:44.804+0000 7fd00b400640 1 -- 172.21.15.6:0/997299276 <== mon.2 v2:172.21.15.190:3301/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fd00c0041c0 con 0x7fd01414aec0 2024-04-24T23:26:44.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:44.806+0000 7fd00b400640 1 --2- 172.21.15.6:0/997299276 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fcff406bf30 0x7fcff406e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:26:44.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:44.806+0000 7fd00b400640 1 -- 172.21.15.6:0/997299276 <== mon.2 v2:172.21.15.190:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6934+0+0 (secure 0 0 0) 0x7fd00c08abb0 con 0x7fd01414aec0 2024-04-24T23:26:44.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:44.808+0000 7fd013400640 1 --2- 172.21.15.6:0/997299276 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fcff406bf30 0x7fcff406e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:26:44.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:44.808+0000 7fd013400640 1 --2- 172.21.15.6:0/997299276 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fcff406bf30 0x7fcff406e3f0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fd004006fd0 tx=0x7fd004008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:26:44.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:44.811+0000 7fd00b400640 1 -- 172.21.15.6:0/997299276 <== mon.2 v2:172.21.15.190:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fd00c059070 con 0x7fd01414aec0 2024-04-24T23:26:45.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:45.278+0000 7fd01a600640 1 -- 172.21.15.6:0/997299276 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd0140ccbc0 con 0x7fd01414aec0 2024-04-24T23:26:45.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:45.279+0000 7fd00b400640 1 -- 172.21.15.6:0/997299276 <== mon.2 v2:172.21.15.190:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6033 (secure 0 0 0) 0x7fd00c058970 con 0x7fd01414aec0 2024-04-24T23:26:45.280 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:26:45.280 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5081,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/1190759626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1190759626},{"type":"v1","addr":"172.21.15.6:6834","nonce":1190759626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":5249,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6838/2416336040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2416336040},{"type":"v1","addr":"172.21.15.190:6838","nonce":2416336040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":42},{"gid":5503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/3928731059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":3928731059},{"type":"v1","addr":"172.21.15.6:6837","nonce":3928731059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:25:42.695130+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5725,"mds_1":5599},"failed":[],"damaged":[],"stopped":[2,3,4],"info":{"gid_5599":{"gid":5599,"name":"d","rank":1,"incarnation":43,"state":"up:active","state_seq":25,"addr":"172.21.15.190:6839/2643233757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":2643233757},{"type":"v1","addr":"172.21.15.190:6839","nonce":2643233757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5725":{"gid":5725,"name":"f","rank":0,"incarnation":49,"state":"up:replay","state_seq":1,"addr":"172.21.15.190:6837/1151527362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":1151527362},{"type":"v1","addr":"172.21.15.190:6837","nonce":1151527362}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5599,"qdb_cluster":[5599]},"id":1}]} 2024-04-24T23:26:45.280 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 49 2024-04-24T23:26:45.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:45.285+0000 7fd01a600640 1 -- 172.21.15.6:0/997299276 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fcff406bf30 msgr2=0x7fcff406e3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:26:45.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:45.285+0000 7fd01a600640 1 --2- 172.21.15.6:0/997299276 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fcff406bf30 0x7fcff406e3f0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fd004006fd0 tx=0x7fd004008040 comp rx=0 tx=0).stop 2024-04-24T23:26:45.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:45.285+0000 7fd01a600640 1 -- 172.21.15.6:0/997299276 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd01414aec0 msgr2=0x7fd01414c0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:26:45.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:45.285+0000 7fd01a600640 1 --2- 172.21.15.6:0/997299276 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd01414aec0 0x7fd01414c0b0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fd014071b20 tx=0x7fd00c00c8a0 comp rx=0 tx=0).stop 2024-04-24T23:26:45.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:45.286+0000 7fd01a600640 1 -- 172.21.15.6:0/997299276 shutdown_connections 2024-04-24T23:26:45.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:45.286+0000 7fd01a600640 1 --2- 172.21.15.6:0/997299276 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fcff406bf30 0x7fcff406e3f0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:26:45.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:45.286+0000 7fd01a600640 1 --2- 172.21.15.6:0/997299276 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd01414aec0 0x7fd01414c0b0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:26:45.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:45.286+0000 7fd01a600640 1 --2- 172.21.15.6:0/997299276 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd0140cedd0 0x7fd0141f85f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:26:45.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:45.286+0000 7fd01a600640 1 --2- 172.21.15.6:0/997299276 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd0140ce9f0 0x7fd0140b3410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:26:45.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:45.286+0000 7fd01a600640 1 -- 172.21.15.6:0/997299276 >> 172.21.15.6:0/997299276 conn(0x7fd014070280 msgr2=0x7fd01415a920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:26:45.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:45.287+0000 7fd01a600640 1 -- 172.21.15.6:0/997299276 shutdown_connections 2024-04-24T23:26:45.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:45.287+0000 7fd01a600640 1 -- 172.21.15.6:0/997299276 wait complete. 2024-04-24T23:26:45.344 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-24T23:26:47.345 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:26:47.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:47.673+0000 7fbc65a00640 1 Processor -- start 2024-04-24T23:26:47.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:47.674+0000 7fbc65a00640 1 -- start start 2024-04-24T23:26:47.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:47.676+0000 7fbc65a00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbc60159110 0x7fbc60072420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:26:47.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:47.676+0000 7fbc65a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fbc600c81c0 0x7fbc60162ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:26:47.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:47.676+0000 7fbc65a00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbc600c7230 con 0x7fbc600c7dc0 2024-04-24T23:26:47.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:47.676+0000 7fbc65a00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7fbc601594f0 con 0x7fbc600c81c0 2024-04-24T23:26:47.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:47.676+0000 7fbc65a00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fbc600c74d0 con 0x7fbc60159110 2024-04-24T23:26:47.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:47.676+0000 7fbc5ea00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbc60159110 0x7fbc60072420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:26:47.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:47.676+0000 7fbc5ea00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbc60159110 0x7fbc60072420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.6:41150/0 (socket says 172.21.15.6:41150) 2024-04-24T23:26:47.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:47.676+0000 7fbc5ea00640 1 -- 172.21.15.6:0/3587727818 learned_addr learned my addr 172.21.15.6:0/3587727818 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:26:47.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:47.676+0000 7fbc5f400640 1 --2- 172.21.15.6:0/3587727818 >> v2:172.21.15.6:3300/0 conn(0x7fbc600c81c0 0x7fbc60162ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:26:47.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:47.677+0000 7fbc5ea00640 1 -- 172.21.15.6:0/3587727818 >> v2:172.21.15.6:3300/0 conn(0x7fbc600c81c0 msgr2=0x7fbc60162ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:26:47.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:47.677+0000 7fbc5ea00640 1 --2- 172.21.15.6:0/3587727818 >> v2:172.21.15.6:3300/0 conn(0x7fbc600c81c0 0x7fbc60162ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:26:47.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:47.677+0000 7fbc5ea00640 1 -- 172.21.15.6:0/3587727818 >> v1:172.21.15.6:6789/0 conn(0x7fbc600c7dc0 legacy=0x7fbc601623a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-24T23:26:47.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:47.679+0000 7fbc5ea00640 1 -- 172.21.15.6:0/3587727818 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc6006e440 con 0x7fbc60159110 2024-04-24T23:26:47.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:47.680+0000 7fbc5ea00640 1 --2- 172.21.15.6:0/3587727818 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbc60159110 0x7fbc60072420 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fbc600c3610 tx=0x7fbc5002a270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=649bd042509da885 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:26:47.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:47.681+0000 7fbc5d600640 1 -- 172.21.15.6:0/3587727818 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc50037070 con 0x7fbc60159110 2024-04-24T23:26:47.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:47.681+0000 7fbc5d600640 1 -- 172.21.15.6:0/3587727818 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc50032440 con 0x7fbc60159110 2024-04-24T23:26:47.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:47.683+0000 7fbc5d600640 1 -- 172.21.15.6:0/3587727818 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc50032810 con 0x7fbc60159110 2024-04-24T23:26:47.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:47.684+0000 7fbc65a00640 1 -- 172.21.15.6:0/3587727818 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbc60159110 msgr2=0x7fbc60072420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:26:47.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:47.684+0000 7fbc65a00640 1 --2- 172.21.15.6:0/3587727818 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbc60159110 0x7fbc60072420 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fbc600c3610 tx=0x7fbc5002a270 comp rx=0 tx=0).stop 2024-04-24T23:26:47.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:47.688+0000 7fbc65a00640 1 -- 172.21.15.6:0/3587727818 shutdown_connections 2024-04-24T23:26:47.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:47.688+0000 7fbc65a00640 1 --2- 172.21.15.6:0/3587727818 >> v2:172.21.15.6:3300/0 conn(0x7fbc600c81c0 0x7fbc60162ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:26:47.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:47.688+0000 7fbc65a00640 1 --2- 172.21.15.6:0/3587727818 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbc60159110 0x7fbc60072420 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:26:47.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:47.688+0000 7fbc65a00640 1 -- 172.21.15.6:0/3587727818 >> 172.21.15.6:0/3587727818 conn(0x7fbc6006b0e0 msgr2=0x7fbc6006b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:26:47.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:47.690+0000 7fbc65a00640 1 -- 172.21.15.6:0/3587727818 shutdown_connections 2024-04-24T23:26:47.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:47.690+0000 7fbc65a00640 1 -- 172.21.15.6:0/3587727818 wait complete. 2024-04-24T23:26:47.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:47.694+0000 7fbc65a00640 1 Processor -- start 2024-04-24T23:26:47.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:47.694+0000 7fbc65a00640 1 -- start start 2024-04-24T23:26:47.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:47.696+0000 7fbc65a00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbc600c7dc0 0x7fbc60162c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:26:47.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:47.697+0000 7fbc65a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbc600c81c0 0x7fbc600c4200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:26:47.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:47.697+0000 7fbc65a00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbc60159110 0x7fbc601f5490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:26:47.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:47.697+0000 7fbc65a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fbc6006ece0 con 0x7fbc600c81c0 2024-04-24T23:26:47.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:47.697+0000 7fbc65a00640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fbc6006ea00 con 0x7fbc600c7dc0 2024-04-24T23:26:47.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:47.697+0000 7fbc65a00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fbc60072960 con 0x7fbc60159110 2024-04-24T23:26:47.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:47.697+0000 7fbc5f400640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbc60159110 0x7fbc601f5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:26:47.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:47.697+0000 7fbc5f400640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbc60159110 0x7fbc601f5490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.6:41164/0 (socket says 172.21.15.6:41164) 2024-04-24T23:26:47.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:47.697+0000 7fbc5f400640 1 -- 172.21.15.6:0/3866994559 learned_addr learned my addr 172.21.15.6:0/3866994559 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:26:47.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:47.698+0000 7fbc5f400640 1 -- 172.21.15.6:0/3866994559 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbc600c7dc0 msgr2=0x7fbc60162c60 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-24T23:26:47.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:47.698+0000 7fbc5f400640 1 --2- 172.21.15.6:0/3866994559 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbc600c7dc0 0x7fbc60162c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:26:47.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:47.698+0000 7fbc5f400640 1 -- 172.21.15.6:0/3866994559 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbc600c81c0 msgr2=0x7fbc600c4200 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-24T23:26:47.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:47.698+0000 7fbc5e000640 1 --2- 172.21.15.6:0/3866994559 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbc600c81c0 0x7fbc600c4200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:26:47.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:47.699+0000 7fbc5f400640 1 --2- 172.21.15.6:0/3866994559 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbc600c81c0 0x7fbc600c4200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:26:47.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:47.699+0000 7fbc5f400640 1 -- 172.21.15.6:0/3866994559 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc600c5840 con 0x7fbc60159110 2024-04-24T23:26:47.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:47.699+0000 7fbc5e000640 1 --2- 172.21.15.6:0/3866994559 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbc600c81c0 0x7fbc600c4200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:26:47.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:47.700+0000 7fbc5f400640 1 --2- 172.21.15.6:0/3866994559 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbc60159110 0x7fbc601f5490 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fbc58007e70 tx=0x7fbc5800e4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:26:47.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:47.700+0000 7fbc56a00640 1 -- 172.21.15.6:0/3866994559 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc58014620 con 0x7fbc60159110 2024-04-24T23:26:47.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:47.701+0000 7fbc65a00640 1 -- 172.21.15.6:0/3866994559 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc600c5b20 con 0x7fbc60159110 2024-04-24T23:26:47.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:47.701+0000 7fbc65a00640 1 -- 172.21.15.6:0/3866994559 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc60161460 con 0x7fbc60159110 2024-04-24T23:26:47.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:47.701+0000 7fbc56a00640 1 -- 172.21.15.6:0/3866994559 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc58019070 con 0x7fbc60159110 2024-04-24T23:26:47.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:47.701+0000 7fbc56a00640 1 -- 172.21.15.6:0/3866994559 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc58014a20 con 0x7fbc60159110 2024-04-24T23:26:47.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:47.705+0000 7fbc56a00640 1 -- 172.21.15.6:0/3866994559 <== mon.2 v2:172.21.15.190:3301/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fbc58014bc0 con 0x7fbc60159110 2024-04-24T23:26:47.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:47.706+0000 7fbc56a00640 1 --2- 172.21.15.6:0/3866994559 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fbc4006be30 0x7fbc4006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:26:47.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:47.707+0000 7fbc56a00640 1 -- 172.21.15.6:0/3866994559 <== mon.2 v2:172.21.15.190:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6934+0+0 (secure 0 0 0) 0x7fbc5808bc30 con 0x7fbc60159110 2024-04-24T23:26:47.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:47.707+0000 7fbc5ea00640 1 --2- 172.21.15.6:0/3866994559 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fbc4006be30 0x7fbc4006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:26:47.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:47.708+0000 7fbc5ea00640 1 --2- 172.21.15.6:0/3866994559 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fbc4006be30 0x7fbc4006e2f0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fbc50034000 tx=0x7fbc5002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:26:47.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:47.709+0000 7fbc65a00640 1 -- 172.21.15.6:0/3866994559 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc38005290 con 0x7fbc60159110 2024-04-24T23:26:47.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:47.717+0000 7fbc56a00640 1 -- 172.21.15.6:0/3866994559 <== mon.2 v2:172.21.15.190:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fbc5805a070 con 0x7fbc60159110 2024-04-24T23:26:48.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:48.247+0000 7fbc65a00640 1 -- 172.21.15.6:0/3866994559 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbc38005580 con 0x7fbc60159110 2024-04-24T23:26:48.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:48.251+0000 7fbc56a00640 1 -- 172.21.15.6:0/3866994559 <== mon.2 v2:172.21.15.190:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6033 (secure 0 0 0) 0x7fbc58059970 con 0x7fbc60159110 2024-04-24T23:26:48.254 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:26:48.254 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5081,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/1190759626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1190759626},{"type":"v1","addr":"172.21.15.6:6834","nonce":1190759626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":5249,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6838/2416336040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2416336040},{"type":"v1","addr":"172.21.15.190:6838","nonce":2416336040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":42},{"gid":5503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/3928731059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":3928731059},{"type":"v1","addr":"172.21.15.6:6837","nonce":3928731059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:25:42.695130+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5725,"mds_1":5599},"failed":[],"damaged":[],"stopped":[2,3,4],"info":{"gid_5599":{"gid":5599,"name":"d","rank":1,"incarnation":43,"state":"up:active","state_seq":25,"addr":"172.21.15.190:6839/2643233757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":2643233757},{"type":"v1","addr":"172.21.15.190:6839","nonce":2643233757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5725":{"gid":5725,"name":"f","rank":0,"incarnation":49,"state":"up:replay","state_seq":1,"addr":"172.21.15.190:6837/1151527362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":1151527362},{"type":"v1","addr":"172.21.15.190:6837","nonce":1151527362}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5599,"qdb_cluster":[5599]},"id":1}]} 2024-04-24T23:26:48.254 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 49 2024-04-24T23:26:48.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:48.256+0000 7fbc65a00640 1 -- 172.21.15.6:0/3866994559 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fbc4006be30 msgr2=0x7fbc4006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:26:48.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:48.256+0000 7fbc65a00640 1 --2- 172.21.15.6:0/3866994559 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fbc4006be30 0x7fbc4006e2f0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fbc50034000 tx=0x7fbc5002e000 comp rx=0 tx=0).stop 2024-04-24T23:26:48.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:48.256+0000 7fbc65a00640 1 -- 172.21.15.6:0/3866994559 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbc60159110 msgr2=0x7fbc601f5490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:26:48.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:48.256+0000 7fbc65a00640 1 --2- 172.21.15.6:0/3866994559 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbc60159110 0x7fbc601f5490 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fbc58007e70 tx=0x7fbc5800e4b0 comp rx=0 tx=0).stop 2024-04-24T23:26:48.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:48.257+0000 7fbc65a00640 1 -- 172.21.15.6:0/3866994559 shutdown_connections 2024-04-24T23:26:48.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:48.257+0000 7fbc65a00640 1 --2- 172.21.15.6:0/3866994559 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fbc4006be30 0x7fbc4006e2f0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:26:48.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:48.257+0000 7fbc65a00640 1 --2- 172.21.15.6:0/3866994559 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbc60159110 0x7fbc601f5490 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:26:48.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:48.257+0000 7fbc65a00640 1 --2- 172.21.15.6:0/3866994559 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbc600c81c0 0x7fbc600c4200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:26:48.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:48.257+0000 7fbc65a00640 1 --2- 172.21.15.6:0/3866994559 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbc600c7dc0 0x7fbc60162c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:26:48.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:48.257+0000 7fbc65a00640 1 -- 172.21.15.6:0/3866994559 >> 172.21.15.6:0/3866994559 conn(0x7fbc6006b0e0 msgr2=0x7fbc601e8200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:26:48.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:48.257+0000 7fbc65a00640 1 -- 172.21.15.6:0/3866994559 shutdown_connections 2024-04-24T23:26:48.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:48.257+0000 7fbc65a00640 1 -- 172.21.15.6:0/3866994559 wait complete. 2024-04-24T23:26:48.292 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-24T23:26:50.294 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:26:50.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:50.604+0000 7f5178e00640 1 Processor -- start 2024-04-24T23:26:50.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:50.604+0000 7f5178e00640 1 -- start start 2024-04-24T23:26:50.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:50.612+0000 7f5178e00640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f51740c7dc0 0x7f51741623a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:26:50.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:50.613+0000 7f5178e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f51740c81c0 0x7f51741628e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:26:50.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:50.613+0000 7f5178e00640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f51740c7230 con 0x7f5174159110 2024-04-24T23:26:50.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:50.613+0000 7f5178e00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f51741594f0 con 0x7f51740c81c0 2024-04-24T23:26:50.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:50.613+0000 7f5178e00640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f51740c74d0 con 0x7f51740c7dc0 2024-04-24T23:26:50.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:50.614+0000 7f5171e00640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f5174159110 0x7f5174072420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.6:33226/0 (socket says 172.21.15.6:33226) 2024-04-24T23:26:50.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:50.614+0000 7f5171e00640 1 -- 172.21.15.6:0/1051025447 learned_addr learned my addr 172.21.15.6:0/1051025447 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:26:50.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:50.614+0000 7f5171400640 1 --2- 172.21.15.6:0/1051025447 >> v2:172.21.15.190:3300/0 conn(0x7f51740c7dc0 0x7f51741623a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:26:50.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:50.615+0000 7f5170a00640 1 -- 172.21.15.6:0/1051025447 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2141480167 0 0) 0x7f51740c7230 con 0x7f5174159110 2024-04-24T23:26:50.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:50.615+0000 7f5170a00640 1 -- 172.21.15.6:0/1051025447 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5158003520 con 0x7f5174159110 2024-04-24T23:26:50.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:50.616+0000 7f5171400640 1 -- 172.21.15.6:0/1051025447 >> v2:172.21.15.6:3300/0 conn(0x7f51740c81c0 msgr2=0x7f51741628e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:26:50.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:50.616+0000 7f5171400640 1 --2- 172.21.15.6:0/1051025447 >> v2:172.21.15.6:3300/0 conn(0x7f51740c81c0 0x7f51741628e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:26:50.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:50.616+0000 7f5171400640 1 -- 172.21.15.6:0/1051025447 >> v1:172.21.15.190:6789/0 conn(0x7f5174159110 legacy=0x7f5174072420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:26:50.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:50.616+0000 7f5171400640 1 -- 172.21.15.6:0/1051025447 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f517406e440 con 0x7f51740c7dc0 2024-04-24T23:26:50.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:50.624+0000 7f5171400640 1 --2- 172.21.15.6:0/1051025447 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f51740c7dc0 0x7f51741623a0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f51640097a0 tx=0x7f516402a2a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a25c6fa63bdcebb7 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:26:50.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:50.625+0000 7f5170a00640 1 -- 172.21.15.6:0/1051025447 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5164037070 con 0x7f51740c7dc0 2024-04-24T23:26:50.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:50.625+0000 7f5170a00640 1 -- 172.21.15.6:0/1051025447 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5164032440 con 0x7f51740c7dc0 2024-04-24T23:26:50.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:50.625+0000 7f5170a00640 1 -- 172.21.15.6:0/1051025447 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5164032830 con 0x7f51740c7dc0 2024-04-24T23:26:50.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:50.626+0000 7f5178e00640 1 -- 172.21.15.6:0/1051025447 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f51740c7dc0 msgr2=0x7f51741623a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:26:50.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:50.626+0000 7f5178e00640 1 --2- 172.21.15.6:0/1051025447 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f51740c7dc0 0x7f51741623a0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f51640097a0 tx=0x7f516402a2a0 comp rx=0 tx=0).stop 2024-04-24T23:26:50.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:50.627+0000 7f5178e00640 1 -- 172.21.15.6:0/1051025447 shutdown_connections 2024-04-24T23:26:50.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:50.627+0000 7f5178e00640 1 --2- 172.21.15.6:0/1051025447 >> v2:172.21.15.6:3300/0 conn(0x7f51740c81c0 0x7f51741628e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:26:50.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:50.627+0000 7f5178e00640 1 --2- 172.21.15.6:0/1051025447 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f51740c7dc0 0x7f51741623a0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:26:50.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:50.628+0000 7f5178e00640 1 -- 172.21.15.6:0/1051025447 >> 172.21.15.6:0/1051025447 conn(0x7f517406b0e0 msgr2=0x7f517406b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:26:50.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:50.628+0000 7f5178e00640 1 -- 172.21.15.6:0/1051025447 shutdown_connections 2024-04-24T23:26:50.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:50.628+0000 7f5178e00640 1 -- 172.21.15.6:0/1051025447 wait complete. 2024-04-24T23:26:50.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:50.633+0000 7f5178e00640 1 Processor -- start 2024-04-24T23:26:50.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:50.634+0000 7f5178e00640 1 -- start start 2024-04-24T23:26:50.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:50.640+0000 7f5178e00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f51740c7dc0 0x7f51740cb3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:26:50.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:50.640+0000 7f5178e00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f51740c81c0 0x7f51741f5550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:26:50.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:50.641+0000 7f5178e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5174159110 0x7f51740c5820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:26:50.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:50.642+0000 7f5171400640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f51740c81c0 0x7f51741f5550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:26:50.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:50.642+0000 7f5171400640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f51740c81c0 0x7f51741f5550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.6:41176/0 (socket says 172.21.15.6:41176) 2024-04-24T23:26:50.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:50.642+0000 7f5171e00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f51740c7dc0 0x7f51740cb3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:26:50.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:50.643+0000 7f5172800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5174159110 0x7f51740c5820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:26:50.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:50.643+0000 7f5171400640 1 -- 172.21.15.6:0/2824613249 learned_addr learned my addr 172.21.15.6:0/2824613249 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:26:50.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:50.647+0000 7f5178e00640 1 -- 172.21.15.6:0/2824613249 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f517406ece0 con 0x7f5174159110 2024-04-24T23:26:50.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:50.647+0000 7f5178e00640 1 -- 172.21.15.6:0/2824613249 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f517406ea00 con 0x7f51740c7dc0 2024-04-24T23:26:50.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:50.647+0000 7f5178e00640 1 -- 172.21.15.6:0/2824613249 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f517406f200 con 0x7f51740c81c0 2024-04-24T23:26:50.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:50.648+0000 7f5171e00640 1 -- 172.21.15.6:0/2824613249 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f51740c81c0 msgr2=0x7f51741f5550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:26:50.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:50.648+0000 7f5171e00640 1 --2- 172.21.15.6:0/2824613249 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f51740c81c0 0x7f51741f5550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:26:50.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:50.648+0000 7f5171e00640 1 -- 172.21.15.6:0/2824613249 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5174159110 msgr2=0x7f51740c5820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:26:50.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:50.648+0000 7f5171e00640 1 --2- 172.21.15.6:0/2824613249 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5174159110 0x7f51740c5820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:26:50.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:50.648+0000 7f5171e00640 1 -- 172.21.15.6:0/2824613249 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51741e8430 con 0x7f51740c7dc0 2024-04-24T23:26:50.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:50.650+0000 7f5172800640 1 --2- 172.21.15.6:0/2824613249 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5174159110 0x7f51740c5820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:26:50.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:50.656+0000 7f5171e00640 1 --2- 172.21.15.6:0/2824613249 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f51740c7dc0 0x7f51740cb3b0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f5160000f60 tx=0x7f516000c890 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:26:50.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:50.656+0000 7f516a000640 1 -- 172.21.15.6:0/2824613249 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5160017070 con 0x7f51740c7dc0 2024-04-24T23:26:50.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:50.656+0000 7f516a000640 1 -- 172.21.15.6:0/2824613249 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5160005ce0 con 0x7f51740c7dc0 2024-04-24T23:26:50.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:50.656+0000 7f516a000640 1 -- 172.21.15.6:0/2824613249 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5160007310 con 0x7f51740c7dc0 2024-04-24T23:26:50.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:50.656+0000 7f5178e00640 1 -- 172.21.15.6:0/2824613249 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51741e8710 con 0x7f51740c7dc0 2024-04-24T23:26:50.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:50.657+0000 7f5178e00640 1 -- 172.21.15.6:0/2824613249 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51741e8cd0 con 0x7f51740c7dc0 2024-04-24T23:26:50.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:50.659+0000 7f5178e00640 1 -- 172.21.15.6:0/2824613249 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f51740767b0 con 0x7f51740c7dc0 2024-04-24T23:26:50.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:50.677+0000 7f516a000640 1 -- 172.21.15.6:0/2824613249 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f516001a070 con 0x7f51740c7dc0 2024-04-24T23:26:50.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:50.680+0000 7f516a000640 1 --2- 172.21.15.6:0/2824613249 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f515406be30 0x7f515406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:26:50.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:50.680+0000 7f516a000640 1 -- 172.21.15.6:0/2824613249 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6934+0+0 (secure 0 0 0) 0x7f516008a7d0 con 0x7f51740c7dc0 2024-04-24T23:26:50.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:50.680+0000 7f516a000640 1 -- 172.21.15.6:0/2824613249 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f5160037080 con 0x7f51740c7dc0 2024-04-24T23:26:50.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:50.682+0000 7f5171400640 1 --2- 172.21.15.6:0/2824613249 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f515406be30 0x7f515406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:26:50.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:50.685+0000 7f5171400640 1 --2- 172.21.15.6:0/2824613249 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f515406be30 0x7f515406e2f0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f516402a2e0 tx=0x7f516402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:26:51.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:51.165+0000 7f5178e00640 1 -- 172.21.15.6:0/2824613249 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f51740c8a40 con 0x7f51740c7dc0 2024-04-24T23:26:51.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:51.167+0000 7f516a000640 1 -- 172.21.15.6:0/2824613249 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6033 (secure 0 0 0) 0x7f51600586d0 con 0x7f51740c7dc0 2024-04-24T23:26:51.168 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:26:51.168 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5081,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/1190759626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1190759626},{"type":"v1","addr":"172.21.15.6:6834","nonce":1190759626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":5249,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6838/2416336040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2416336040},{"type":"v1","addr":"172.21.15.190:6838","nonce":2416336040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":42},{"gid":5503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/3928731059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":3928731059},{"type":"v1","addr":"172.21.15.6:6837","nonce":3928731059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:25:42.695130+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5725,"mds_1":5599},"failed":[],"damaged":[],"stopped":[2,3,4],"info":{"gid_5599":{"gid":5599,"name":"d","rank":1,"incarnation":43,"state":"up:active","state_seq":25,"addr":"172.21.15.190:6839/2643233757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":2643233757},{"type":"v1","addr":"172.21.15.190:6839","nonce":2643233757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5725":{"gid":5725,"name":"f","rank":0,"incarnation":49,"state":"up:replay","state_seq":1,"addr":"172.21.15.190:6837/1151527362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":1151527362},{"type":"v1","addr":"172.21.15.190:6837","nonce":1151527362}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5599,"qdb_cluster":[5599]},"id":1}]} 2024-04-24T23:26:51.168 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 49 2024-04-24T23:26:51.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:51.171+0000 7f5178e00640 1 -- 172.21.15.6:0/2824613249 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f515406be30 msgr2=0x7f515406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:26:51.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:51.171+0000 7f5178e00640 1 --2- 172.21.15.6:0/2824613249 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f515406be30 0x7f515406e2f0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f516402a2e0 tx=0x7f516402e000 comp rx=0 tx=0).stop 2024-04-24T23:26:51.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:51.171+0000 7f5178e00640 1 -- 172.21.15.6:0/2824613249 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f51740c7dc0 msgr2=0x7f51740cb3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:26:51.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:51.171+0000 7f5178e00640 1 --2- 172.21.15.6:0/2824613249 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f51740c7dc0 0x7f51740cb3b0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f5160000f60 tx=0x7f516000c890 comp rx=0 tx=0).stop 2024-04-24T23:26:51.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:51.172+0000 7f5178e00640 1 -- 172.21.15.6:0/2824613249 shutdown_connections 2024-04-24T23:26:51.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:51.172+0000 7f5178e00640 1 --2- 172.21.15.6:0/2824613249 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f515406be30 0x7f515406e2f0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:26:51.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:51.172+0000 7f5178e00640 1 --2- 172.21.15.6:0/2824613249 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f51740c81c0 0x7f51741f5550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:26:51.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:51.172+0000 7f5178e00640 1 --2- 172.21.15.6:0/2824613249 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5174159110 0x7f51740c5820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:26:51.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:51.172+0000 7f5178e00640 1 --2- 172.21.15.6:0/2824613249 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f51740c7dc0 0x7f51740cb3b0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:26:51.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:51.172+0000 7f5178e00640 1 -- 172.21.15.6:0/2824613249 >> 172.21.15.6:0/2824613249 conn(0x7f517406b0e0 msgr2=0x7f5174072180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:26:51.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:51.173+0000 7f5178e00640 1 -- 172.21.15.6:0/2824613249 shutdown_connections 2024-04-24T23:26:51.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:26:51.173+0000 7f5178e00640 1 -- 172.21.15.6:0/2824613249 wait complete. 2024-04-24T23:26:51.204 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-24T23:26:53.205 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:27:00.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:00.088+0000 7f16d8400640 1 Processor -- start 2024-04-24T23:27:00.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:00.088+0000 7f16d8400640 1 -- start start 2024-04-24T23:27:00.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:00.090+0000 7f16d8400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f16d00c9330 0x7f16d006eab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:00.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:00.090+0000 7f16d8400640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f16d00c6e70 con 0x7f16d00c8f50 2024-04-24T23:27:00.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:00.090+0000 7f16d8400640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f16d00c7110 con 0x7f16d0152bc0 2024-04-24T23:27:00.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:00.090+0000 7f16d8400640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f16d00c73d0 con 0x7f16d00c9330 2024-04-24T23:27:00.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:00.090+0000 7f16d5a00640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f16d0152bc0 0x7f16d0159390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.6:33850/0 (socket says 172.21.15.6:33850) 2024-04-24T23:27:00.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:00.090+0000 7f16d5a00640 1 -- 172.21.15.6:0/56157522 learned_addr learned my addr 172.21.15.6:0/56157522 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:27:00.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:00.091+0000 7f16d6400640 1 --2- 172.21.15.6:0/56157522 >> v2:172.21.15.6:3300/0 conn(0x7f16d00c9330 0x7f16d006eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:00.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:00.091+0000 7f16cfe00640 1 -- 172.21.15.6:0/56157522 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2227245304 0 0) 0x7f16d00c7110 con 0x7f16d0152bc0 2024-04-24T23:27:00.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:00.091+0000 7f16cfe00640 1 -- 172.21.15.6:0/56157522 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f16b4003520 con 0x7f16d0152bc0 2024-04-24T23:27:00.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:00.092+0000 7f16cfe00640 1 -- 172.21.15.6:0/56157522 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2084569187 0 0) 0x7f16b4003520 con 0x7f16d0152bc0 2024-04-24T23:27:00.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:00.092+0000 7f16cfe00640 1 -- 172.21.15.6:0/56157522 >> v2:172.21.15.6:3300/0 conn(0x7f16d00c9330 msgr2=0x7f16d006eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:27:00.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:00.092+0000 7f16cfe00640 1 --2- 172.21.15.6:0/56157522 >> v2:172.21.15.6:3300/0 conn(0x7f16d00c9330 0x7f16d006eab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:00.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:00.092+0000 7f16cfe00640 1 -- 172.21.15.6:0/56157522 >> v1:172.21.15.6:6789/0 conn(0x7f16d00c8f50 legacy=0x7f16d006e3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:00.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:00.093+0000 7f16cfe00640 1 -- 172.21.15.6:0/56157522 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16d00c6700 con 0x7f16d0152bc0 2024-04-24T23:27:00.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:00.093+0000 7f16cfe00640 1 -- 172.21.15.6:0/56157522 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3463268701 0 0) 0x7f16c8001670 con 0x7f16d0152bc0 2024-04-24T23:27:00.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:00.093+0000 7f16cfe00640 1 -- 172.21.15.6:0/56157522 >> v1:172.21.15.190:6789/0 conn(0x7f16d0152bc0 legacy=0x7f16d0159390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:00.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:00.093+0000 7f16cfe00640 1 --2- 172.21.15.6:0/56157522 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f16b40039b0 0x7f16b4003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:00.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:00.093+0000 7f16cfe00640 1 --2- 172.21.15.6:0/56157522 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f16b40043b0 0x7f16b4004880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:00.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:00.093+0000 7f16cfe00640 1 --2- 172.21.15.6:0/56157522 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f16b4005120 0x7f16b400fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:00.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:00.093+0000 7f16cfe00640 1 -- 172.21.15.6:0/56157522 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f16d00c73d0 con 0x7f16b4005120 2024-04-24T23:27:00.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:00.093+0000 7f16cfe00640 1 -- 172.21.15.6:0/56157522 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f16b40059c0 con 0x7f16b40039b0 2024-04-24T23:27:00.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:00.093+0000 7f16cfe00640 1 -- 172.21.15.6:0/56157522 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f16b4005bc0 con 0x7f16b40043b0 2024-04-24T23:27:00.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:00.093+0000 7f16d6400640 1 --2- 172.21.15.6:0/56157522 >> v2:172.21.15.6:3300/0 conn(0x7f16d00c9330 0x7f16d006eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:27:00.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:00.093+0000 7f16d5a00640 1 --2- 172.21.15.6:0/56157522 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f16b40039b0 0x7f16b4003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:00.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:00.094+0000 7f16d6400640 1 --2- 172.21.15.6:0/56157522 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f16b4005120 0x7f16b400fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:00.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:00.094+0000 7f16d5a00640 1 -- 172.21.15.6:0/56157522 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f16b40043b0 msgr2=0x7f16b4004880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:27:00.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:00.094+0000 7f16d5a00640 1 --2- 172.21.15.6:0/56157522 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f16b40043b0 0x7f16b4004880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:00.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:00.094+0000 7f16d5a00640 1 -- 172.21.15.6:0/56157522 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f16b4005120 msgr2=0x7f16b400fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:27:00.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:00.094+0000 7f16d5a00640 1 --2- 172.21.15.6:0/56157522 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f16b4005120 0x7f16b400fdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:00.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:00.094+0000 7f16d5a00640 1 -- 172.21.15.6:0/56157522 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16d00c6700 con 0x7f16b40039b0 2024-04-24T23:27:00.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:00.094+0000 7f16d6400640 1 --2- 172.21.15.6:0/56157522 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f16b4005120 0x7f16b400fdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:27:00.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:00.094+0000 7f16d6400640 1 -- 172.21.15.6:0/56157522 reap_dead start 2024-04-24T23:27:00.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:00.095+0000 7f16d5a00640 1 --2- 172.21.15.6:0/56157522 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f16b40039b0 0x7f16b4003d90 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f16c800c290 tx=0x7f16b4005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=77ce985359ac85c6 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:27:00.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:00.095+0000 7f16cfe00640 1 -- 172.21.15.6:0/56157522 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16c8007d10 con 0x7f16b40039b0 2024-04-24T23:27:00.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:00.095+0000 7f16cfe00640 1 -- 172.21.15.6:0/56157522 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16c8005ad0 con 0x7f16b40039b0 2024-04-24T23:27:00.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:00.095+0000 7f16cfe00640 1 -- 172.21.15.6:0/56157522 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16c8007070 con 0x7f16b40039b0 2024-04-24T23:27:00.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:00.096+0000 7f16d8400640 1 -- 172.21.15.6:0/56157522 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f16b40039b0 msgr2=0x7f16b4003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:00.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:00.096+0000 7f16d8400640 1 --2- 172.21.15.6:0/56157522 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f16b40039b0 0x7f16b4003d90 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f16c800c290 tx=0x7f16b4005830 comp rx=0 tx=0).stop 2024-04-24T23:27:00.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:00.097+0000 7f16d8400640 1 -- 172.21.15.6:0/56157522 shutdown_connections 2024-04-24T23:27:00.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:00.097+0000 7f16d8400640 1 --2- 172.21.15.6:0/56157522 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f16b40039b0 0x7f16b4003d90 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:00.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:00.097+0000 7f16d8400640 1 -- 172.21.15.6:0/56157522 >> 172.21.15.6:0/56157522 conn(0x7f16d006b0e0 msgr2=0x7f16d006b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:27:00.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:00.097+0000 7f16d8400640 1 -- 172.21.15.6:0/56157522 shutdown_connections 2024-04-24T23:27:00.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:00.097+0000 7f16d8400640 1 -- 172.21.15.6:0/56157522 wait complete. 2024-04-24T23:27:00.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:00.098+0000 7f16d8400640 1 Processor -- start 2024-04-24T23:27:00.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:00.099+0000 7f16d8400640 1 -- start start 2024-04-24T23:27:00.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:00.101+0000 7f16d8400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f16b40039b0 0x7f16d0150850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:00.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:00.101+0000 7f16d8400640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f16d0155ab0 0x7f16d0148830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:00.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:00.101+0000 7f16d8400640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f16d00721c0 0x7f16d01ef580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:00.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:00.101+0000 7f16d8400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f16d00c8750 con 0x7f16b40039b0 2024-04-24T23:27:00.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:00.101+0000 7f16d8400640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f16d00c8ac0 con 0x7f16d0155ab0 2024-04-24T23:27:00.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:00.101+0000 7f16d8400640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f16d0152fa0 con 0x7f16d00721c0 2024-04-24T23:27:00.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:00.101+0000 7f16d6400640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f16d00721c0 0x7f16d01ef580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:00.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:00.101+0000 7f16d5000640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f16d0155ab0 0x7f16d0148830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:00.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:00.101+0000 7f16d5a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f16b40039b0 0x7f16d0150850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:00.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:00.101+0000 7f16d6400640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f16d00721c0 0x7f16d01ef580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.6:39352/0 (socket says 172.21.15.6:39352) 2024-04-24T23:27:00.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:00.101+0000 7f16d6400640 1 -- 172.21.15.6:0/3707540456 learned_addr learned my addr 172.21.15.6:0/3707540456 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:27:00.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:00.102+0000 7f16d5a00640 1 -- 172.21.15.6:0/3707540456 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f16d00721c0 msgr2=0x7f16d01ef580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:00.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:00.102+0000 7f16d5a00640 1 --2- 172.21.15.6:0/3707540456 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f16d00721c0 0x7f16d01ef580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:00.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:00.102+0000 7f16d5a00640 1 -- 172.21.15.6:0/3707540456 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f16d0155ab0 msgr2=0x7f16d0148830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:00.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:00.102+0000 7f16d5a00640 1 --2- 172.21.15.6:0/3707540456 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f16d0155ab0 0x7f16d0148830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:00.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:00.102+0000 7f16d5a00640 1 -- 172.21.15.6:0/3707540456 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16d0154a50 con 0x7f16b40039b0 2024-04-24T23:27:00.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:00.102+0000 7f16d5a00640 1 --2- 172.21.15.6:0/3707540456 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f16b40039b0 0x7f16d0150850 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f16c8030000 tx=0x7f16c8001970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:27:00.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:00.103+0000 7f16cd600640 1 -- 172.21.15.6:0/3707540456 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16c8042020 con 0x7f16b40039b0 2024-04-24T23:27:00.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:00.103+0000 7f16cd600640 1 -- 172.21.15.6:0/3707540456 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16c8004bf0 con 0x7f16b40039b0 2024-04-24T23:27:00.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:00.103+0000 7f16cd600640 1 -- 172.21.15.6:0/3707540456 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16c8007d10 con 0x7f16b40039b0 2024-04-24T23:27:00.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:00.103+0000 7f16d8400640 1 -- 172.21.15.6:0/3707540456 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16d01552f0 con 0x7f16b40039b0 2024-04-24T23:27:00.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:00.103+0000 7f16d8400640 1 -- 172.21.15.6:0/3707540456 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16d01521e0 con 0x7f16b40039b0 2024-04-24T23:27:00.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:00.105+0000 7f16d8400640 1 -- 172.21.15.6:0/3707540456 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f16d00767b0 con 0x7f16b40039b0 2024-04-24T23:27:00.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:00.107+0000 7f16cd600640 1 -- 172.21.15.6:0/3707540456 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f16c8045020 con 0x7f16b40039b0 2024-04-24T23:27:00.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:00.108+0000 7f16cd600640 1 --2- 172.21.15.6:0/3707540456 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f16b006bf00 0x7f16b006e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:00.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:00.109+0000 7f16cd600640 1 -- 172.21.15.6:0/3707540456 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6934+0+0 (secure 0 0 0) 0x7f16c80ab810 con 0x7f16b40039b0 2024-04-24T23:27:00.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:00.109+0000 7f16d5000640 1 --2- 172.21.15.6:0/3707540456 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f16b006bf00 0x7f16b006e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:00.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:00.110+0000 7f16d5000640 1 --2- 172.21.15.6:0/3707540456 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f16b006bf00 0x7f16b006e3c0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f16d0156730 tx=0x7f16bc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:27:00.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:00.117+0000 7f16cd600640 1 -- 172.21.15.6:0/3707540456 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f16c804c070 con 0x7f16b40039b0 2024-04-24T23:27:00.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:00.494+0000 7f16d8400640 1 -- 172.21.15.6:0/3707540456 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f16d00c7110 con 0x7f16b40039b0 2024-04-24T23:27:00.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:00.495+0000 7f16cd600640 1 -- 172.21.15.6:0/3707540456 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6033 (secure 0 0 0) 0x7f16d0154a50 con 0x7f16b40039b0 2024-04-24T23:27:00.495 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:27:00.495 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5081,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/1190759626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1190759626},{"type":"v1","addr":"172.21.15.6:6834","nonce":1190759626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":5249,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6838/2416336040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2416336040},{"type":"v1","addr":"172.21.15.190:6838","nonce":2416336040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":42},{"gid":5503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/3928731059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":3928731059},{"type":"v1","addr":"172.21.15.6:6837","nonce":3928731059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:25:42.695130+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5725,"mds_1":5599},"failed":[],"damaged":[],"stopped":[2,3,4],"info":{"gid_5599":{"gid":5599,"name":"d","rank":1,"incarnation":43,"state":"up:active","state_seq":25,"addr":"172.21.15.190:6839/2643233757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":2643233757},{"type":"v1","addr":"172.21.15.190:6839","nonce":2643233757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5725":{"gid":5725,"name":"f","rank":0,"incarnation":49,"state":"up:replay","state_seq":1,"addr":"172.21.15.190:6837/1151527362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":1151527362},{"type":"v1","addr":"172.21.15.190:6837","nonce":1151527362}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5599,"qdb_cluster":[5599]},"id":1}]} 2024-04-24T23:27:00.496 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 49 2024-04-24T23:27:00.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:00.497+0000 7f16d8400640 1 -- 172.21.15.6:0/3707540456 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f16b006bf00 msgr2=0x7f16b006e3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:00.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:00.497+0000 7f16d8400640 1 --2- 172.21.15.6:0/3707540456 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f16b006bf00 0x7f16b006e3c0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f16d0156730 tx=0x7f16bc009040 comp rx=0 tx=0).stop 2024-04-24T23:27:00.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:00.497+0000 7f16d8400640 1 -- 172.21.15.6:0/3707540456 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f16b40039b0 msgr2=0x7f16d0150850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:00.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:00.497+0000 7f16d8400640 1 --2- 172.21.15.6:0/3707540456 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f16b40039b0 0x7f16d0150850 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f16c8030000 tx=0x7f16c8001970 comp rx=0 tx=0).stop 2024-04-24T23:27:00.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:00.497+0000 7f16d8400640 1 -- 172.21.15.6:0/3707540456 shutdown_connections 2024-04-24T23:27:00.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:00.497+0000 7f16d8400640 1 --2- 172.21.15.6:0/3707540456 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f16b006bf00 0x7f16b006e3c0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:00.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:00.497+0000 7f16d8400640 1 --2- 172.21.15.6:0/3707540456 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f16d00721c0 0x7f16d01ef580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:00.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:00.497+0000 7f16d8400640 1 --2- 172.21.15.6:0/3707540456 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f16d0155ab0 0x7f16d0148830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:00.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:00.497+0000 7f16d8400640 1 --2- 172.21.15.6:0/3707540456 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f16b40039b0 0x7f16d0150850 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:00.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:00.497+0000 7f16d8400640 1 -- 172.21.15.6:0/3707540456 >> 172.21.15.6:0/3707540456 conn(0x7f16d006b0e0 msgr2=0x7f16d00c47a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:27:00.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:00.497+0000 7f16d8400640 1 -- 172.21.15.6:0/3707540456 shutdown_connections 2024-04-24T23:27:00.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:00.497+0000 7f16d8400640 1 -- 172.21.15.6:0/3707540456 wait complete. 2024-04-24T23:27:00.514 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-24T23:27:02.516 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:27:03.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:03.460+0000 7fa6f3200640 1 Processor -- start 2024-04-24T23:27:03.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:03.461+0000 7fa6f3200640 1 -- start start 2024-04-24T23:27:03.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:03.463+0000 7fa6f3200640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fa6f40cb090 0x7fa6f40d0820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:03.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:03.463+0000 7fa6f3200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa6f40d0d60 0x7fa6f415e000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:03.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:03.463+0000 7fa6f3200640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa6f4155170 con 0x7fa6f4154d90 2024-04-24T23:27:03.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:03.463+0000 7fa6f3200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7fa6f40cc9a0 con 0x7fa6f40d0d60 2024-04-24T23:27:03.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:03.463+0000 7fa6f3200640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7fa6f415ad60 con 0x7fa6f40cb090 2024-04-24T23:27:03.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:03.464+0000 7fa6f1400640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fa6f40cb090 0x7fa6f40d0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:03.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:03.464+0000 7fa6f1400640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fa6f40cb090 0x7fa6f40d0820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:56058/0 (socket says 172.21.15.6:56058) 2024-04-24T23:27:03.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:03.464+0000 7fa6f1400640 1 -- 172.21.15.6:0/2667374732 learned_addr learned my addr 172.21.15.6:0/2667374732 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:27:03.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:03.464+0000 7fa6f2800640 1 --2- 172.21.15.6:0/2667374732 >> v2:172.21.15.6:3300/0 conn(0x7fa6f40d0d60 0x7fa6f415e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:03.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:03.465+0000 7fa6f2800640 1 -- 172.21.15.6:0/2667374732 >> v2:172.21.15.190:3300/0 conn(0x7fa6f40cb090 msgr2=0x7fa6f40d0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:27:03.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:03.465+0000 7fa6f2800640 1 --2- 172.21.15.6:0/2667374732 >> v2:172.21.15.190:3300/0 conn(0x7fa6f40cb090 0x7fa6f40d0820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:03.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:03.465+0000 7fa6f2800640 1 -- 172.21.15.6:0/2667374732 >> v1:172.21.15.190:6789/0 conn(0x7fa6f4154d90 legacy=0x7fa6f40cbe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:27:03.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:03.465+0000 7fa6f2800640 1 -- 172.21.15.6:0/2667374732 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6f415b040 con 0x7fa6f40d0d60 2024-04-24T23:27:03.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:03.467+0000 7fa6f2800640 1 --2- 172.21.15.6:0/2667374732 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa6f40d0d60 0x7fa6f415e000 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fa6f408dce0 tx=0x7fa6e402a200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a4b8ea5c3d897cc server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:27:03.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:03.468+0000 7fa6f0a00640 1 -- 172.21.15.6:0/2667374732 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6e4037070 con 0x7fa6f40d0d60 2024-04-24T23:27:03.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:03.468+0000 7fa6f0a00640 1 -- 172.21.15.6:0/2667374732 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa6e4032440 con 0x7fa6f40d0d60 2024-04-24T23:27:03.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:03.468+0000 7fa6f0a00640 1 -- 172.21.15.6:0/2667374732 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6e4032830 con 0x7fa6f40d0d60 2024-04-24T23:27:03.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:03.469+0000 7fa6f3200640 1 -- 172.21.15.6:0/2667374732 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa6f40d0d60 msgr2=0x7fa6f415e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:03.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:03.469+0000 7fa6f3200640 1 --2- 172.21.15.6:0/2667374732 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa6f40d0d60 0x7fa6f415e000 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fa6f408dce0 tx=0x7fa6e402a200 comp rx=0 tx=0).stop 2024-04-24T23:27:03.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:03.470+0000 7fa6f3200640 1 -- 172.21.15.6:0/2667374732 shutdown_connections 2024-04-24T23:27:03.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:03.470+0000 7fa6f3200640 1 --2- 172.21.15.6:0/2667374732 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa6f40d0d60 0x7fa6f415e000 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:03.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:03.470+0000 7fa6f3200640 1 --2- 172.21.15.6:0/2667374732 >> v2:172.21.15.190:3300/0 conn(0x7fa6f40cb090 0x7fa6f40d0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:03.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:03.470+0000 7fa6f3200640 1 -- 172.21.15.6:0/2667374732 >> 172.21.15.6:0/2667374732 conn(0x7fa6f406be10 msgr2=0x7fa6f406c1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:27:03.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:03.471+0000 7fa6f3200640 1 -- 172.21.15.6:0/2667374732 shutdown_connections 2024-04-24T23:27:03.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:03.471+0000 7fa6f3200640 1 -- 172.21.15.6:0/2667374732 wait complete. 2024-04-24T23:27:03.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:03.474+0000 7fa6f3200640 1 Processor -- start 2024-04-24T23:27:03.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:03.474+0000 7fa6f3200640 1 -- start start 2024-04-24T23:27:03.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:03.476+0000 7fa6f3200640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa6f40cb090 0x7fa6f41f7d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:03.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:03.476+0000 7fa6f3200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa6f40d0d60 0x7fa6f415d940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:03.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:03.477+0000 7fa6f1e00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa6f40cb090 0x7fa6f41f7d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:03.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:03.477+0000 7fa6f1400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa6f40d0d60 0x7fa6f415d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:03.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:03.477+0000 7fa6f1e00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa6f40cb090 0x7fa6f41f7d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.6:39362/0 (socket says 172.21.15.6:39362) 2024-04-24T23:27:03.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:03.477+0000 7fa6f1400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa6f40d0d60 0x7fa6f415d940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38968/0 (socket says 172.21.15.6:38968) 2024-04-24T23:27:03.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:03.477+0000 7fa6f3200640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa6f4154d90 0x7fa6f415de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:03.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:03.477+0000 7fa6f1e00640 1 -- 172.21.15.6:0/4098543757 learned_addr learned my addr 172.21.15.6:0/4098543757 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:27:03.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:03.478+0000 7fa6f3200640 1 -- 172.21.15.6:0/4098543757 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fa6f415aa30 con 0x7fa6f40d0d60 2024-04-24T23:27:03.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:03.478+0000 7fa6f2800640 1 --2- 172.21.15.6:0/4098543757 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa6f4154d90 0x7fa6f415de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:03.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:03.478+0000 7fa6f3200640 1 -- 172.21.15.6:0/4098543757 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fa6f415b600 con 0x7fa6f4154d90 2024-04-24T23:27:03.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:03.479+0000 7fa6f3200640 1 -- 172.21.15.6:0/4098543757 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fa6f40cc590 con 0x7fa6f40cb090 2024-04-24T23:27:03.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:03.480+0000 7fa6f1400640 1 -- 172.21.15.6:0/4098543757 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa6f40cb090 msgr2=0x7fa6f41f7d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:03.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:03.480+0000 7fa6f1400640 1 --2- 172.21.15.6:0/4098543757 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa6f40cb090 0x7fa6f41f7d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:03.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:03.480+0000 7fa6f1400640 1 -- 172.21.15.6:0/4098543757 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa6f4154d90 msgr2=0x7fa6f415de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:03.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:03.480+0000 7fa6f1400640 1 --2- 172.21.15.6:0/4098543757 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa6f4154d90 0x7fa6f415de80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:03.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:03.480+0000 7fa6f1400640 1 -- 172.21.15.6:0/4098543757 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6f41f86d0 con 0x7fa6f40d0d60 2024-04-24T23:27:03.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:03.480+0000 7fa6f2800640 1 --2- 172.21.15.6:0/4098543757 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa6f4154d90 0x7fa6f415de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:27:03.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:03.480+0000 7fa6f1e00640 1 --2- 172.21.15.6:0/4098543757 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa6f40cb090 0x7fa6f41f7d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:27:03.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:03.482+0000 7fa6f1400640 1 --2- 172.21.15.6:0/4098543757 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa6f40d0d60 0x7fa6f415d940 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fa6dc00e850 tx=0x7fa6dc00ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:27:03.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:03.482+0000 7fa6ea000640 1 -- 172.21.15.6:0/4098543757 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6dc017070 con 0x7fa6f40d0d60 2024-04-24T23:27:03.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:03.483+0000 7fa6f3200640 1 -- 172.21.15.6:0/4098543757 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6f40c7740 con 0x7fa6f40d0d60 2024-04-24T23:27:03.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:03.483+0000 7fa6f3200640 1 -- 172.21.15.6:0/4098543757 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6f40c7d00 con 0x7fa6f40d0d60 2024-04-24T23:27:03.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:03.483+0000 7fa6ea000640 1 -- 172.21.15.6:0/4098543757 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa6dc00c940 con 0x7fa6f40d0d60 2024-04-24T23:27:03.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:03.483+0000 7fa6ea000640 1 -- 172.21.15.6:0/4098543757 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6dc00ccf0 con 0x7fa6f40d0d60 2024-04-24T23:27:03.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:03.486+0000 7fa6f3200640 1 -- 172.21.15.6:0/4098543757 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa6cc005290 con 0x7fa6f40d0d60 2024-04-24T23:27:03.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:03.487+0000 7fa6ea000640 1 -- 172.21.15.6:0/4098543757 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fa6dc01a070 con 0x7fa6f40d0d60 2024-04-24T23:27:03.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:03.489+0000 7fa6ea000640 1 --2- 172.21.15.6:0/4098543757 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fa6d006beb0 0x7fa6d006e370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:03.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:03.489+0000 7fa6ea000640 1 -- 172.21.15.6:0/4098543757 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6934+0+0 (secure 0 0 0) 0x7fa6dc08b4f0 con 0x7fa6f40d0d60 2024-04-24T23:27:03.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:03.489+0000 7fa6f1e00640 1 --2- 172.21.15.6:0/4098543757 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fa6d006beb0 0x7fa6d006e370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:03.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:03.490+0000 7fa6f1e00640 1 --2- 172.21.15.6:0/4098543757 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fa6d006beb0 0x7fa6d006e370 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fa6ec00c4b0 tx=0x7fa6ec007560 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:27:03.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:03.495+0000 7fa6ea000640 1 -- 172.21.15.6:0/4098543757 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fa6dc059ab0 con 0x7fa6f40d0d60 2024-04-24T23:27:03.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:03.906+0000 7fa6f3200640 1 -- 172.21.15.6:0/4098543757 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa6cc005850 con 0x7fa6f40d0d60 2024-04-24T23:27:03.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:03.907+0000 7fa6ea000640 1 -- 172.21.15.6:0/4098543757 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) ==== 94+0+6035 (secure 0 0 0) 0x7fa6dc0593b0 con 0x7fa6f40d0d60 2024-04-24T23:27:03.908 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:27:03.908 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":50,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5081,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/1190759626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1190759626},{"type":"v1","addr":"172.21.15.6:6834","nonce":1190759626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":5249,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6838/2416336040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2416336040},{"type":"v1","addr":"172.21.15.190:6838","nonce":2416336040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":42},{"gid":5503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/3928731059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":3928731059},{"type":"v1","addr":"172.21.15.6:6837","nonce":3928731059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:27:01.572734+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5725,"mds_1":5599},"failed":[],"damaged":[],"stopped":[2,3,4],"info":{"gid_5599":{"gid":5599,"name":"d","rank":1,"incarnation":43,"state":"up:active","state_seq":25,"addr":"172.21.15.190:6839/2643233757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":2643233757},{"type":"v1","addr":"172.21.15.190:6839","nonce":2643233757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5725":{"gid":5725,"name":"f","rank":0,"incarnation":49,"state":"up:resolve","state_seq":30,"addr":"172.21.15.190:6837/1151527362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":1151527362},{"type":"v1","addr":"172.21.15.190:6837","nonce":1151527362}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5599,"qdb_cluster":[5599]},"id":1}]} 2024-04-24T23:27:03.909 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 50 2024-04-24T23:27:03.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:03.910+0000 7fa6f3200640 1 -- 172.21.15.6:0/4098543757 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fa6d006beb0 msgr2=0x7fa6d006e370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:03.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:03.910+0000 7fa6f3200640 1 --2- 172.21.15.6:0/4098543757 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fa6d006beb0 0x7fa6d006e370 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fa6ec00c4b0 tx=0x7fa6ec007560 comp rx=0 tx=0).stop 2024-04-24T23:27:03.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:03.910+0000 7fa6f3200640 1 -- 172.21.15.6:0/4098543757 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa6f40d0d60 msgr2=0x7fa6f415d940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:03.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:03.910+0000 7fa6f3200640 1 --2- 172.21.15.6:0/4098543757 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa6f40d0d60 0x7fa6f415d940 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fa6dc00e850 tx=0x7fa6dc00ed10 comp rx=0 tx=0).stop 2024-04-24T23:27:03.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:03.910+0000 7fa6f3200640 1 -- 172.21.15.6:0/4098543757 shutdown_connections 2024-04-24T23:27:03.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:03.910+0000 7fa6f3200640 1 --2- 172.21.15.6:0/4098543757 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fa6d006beb0 0x7fa6d006e370 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:03.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:03.910+0000 7fa6f3200640 1 --2- 172.21.15.6:0/4098543757 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa6f4154d90 0x7fa6f415de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:03.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:03.910+0000 7fa6f3200640 1 --2- 172.21.15.6:0/4098543757 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa6f40d0d60 0x7fa6f415d940 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:03.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:03.910+0000 7fa6f3200640 1 --2- 172.21.15.6:0/4098543757 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa6f40cb090 0x7fa6f41f7d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:03.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:03.910+0000 7fa6f3200640 1 -- 172.21.15.6:0/4098543757 >> 172.21.15.6:0/4098543757 conn(0x7fa6f406be10 msgr2=0x7fa6f406c1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:27:03.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:03.910+0000 7fa6f3200640 1 -- 172.21.15.6:0/4098543757 shutdown_connections 2024-04-24T23:27:03.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:03.911+0000 7fa6f3200640 1 -- 172.21.15.6:0/4098543757 wait complete. 2024-04-24T23:27:03.932 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 50, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-24T23:15:36.400716+0000", "damaged": [], "data_pools": [ 3, 4 ], "enabled": true, "epoch": 50, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_5599": { "addr": "172.21.15.190:6839/2643233757", "addrs": { "addrvec": [ { "addr": "172.21.15.190:6836", "nonce": 2643233757, "type": "v2" }, { "addr": "172.21.15.190:6839", "nonce": 2643233757, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5599, "incarnation": 43, "join_fscid": -1, "name": "d", "rank": 1, "state": "up:active", "state_seq": 25 }, "gid_5725": { "addr": "172.21.15.190:6837/1151527362", "addrs": { "addrvec": [ { "addr": "172.21.15.190:6834", "nonce": 1151527362, "type": "v2" }, { "addr": "172.21.15.190:6837", "nonce": 1151527362, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5725, "incarnation": 49, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:resolve", "state_seq": 30 } }, "last_failure": 0, "last_failure_osd_epoch": 30, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-04-24T23:27:01.572734+0000", "qdb_cluster": [ 5599 ], "qdb_leader": 5599, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 2, 3, 4 ], "tableserver": 0, "up": { "mds_0": 5725, "mds_1": 5599 } } } ], "standbys": [ { "addr": "172.21.15.6:6834/1190759626", "addrs": { "addrvec": [ { "addr": "172.21.15.6:6832", "nonce": 1190759626, "type": "v2" }, { "addr": "172.21.15.6:6834", "nonce": 1190759626, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 30, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5081, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.190:6838/2416336040", "addrs": { "addrvec": [ { "addr": "172.21.15.190:6835", "nonce": 2416336040, "type": "v2" }, { "addr": "172.21.15.190:6838", "nonce": 2416336040, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 42, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5249, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.6:6837/3928731059", "addrs": { "addrvec": [ { "addr": "172.21.15.6:6836", "nonce": 3928731059, "type": "v2" }, { "addr": "172.21.15.6:6837", "nonce": 3928731059, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 35, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5503, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-04-24T23:27:05.933 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:27:06.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:06.185+0000 7f3386200640 1 Processor -- start 2024-04-24T23:27:06.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:06.185+0000 7f3386200640 1 -- start start 2024-04-24T23:27:06.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:06.187+0000 7f3386200640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3380159110 0x7f3380072420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:06.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:06.187+0000 7f3386200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f33800c7dc0 0x7f33801623a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:06.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:06.187+0000 7f3386200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f33800c7230 con 0x7f33800c81c0 2024-04-24T23:27:06.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:06.187+0000 7f3386200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f33801594f0 con 0x7f33800c7dc0 2024-04-24T23:27:06.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:06.187+0000 7f3386200640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f33800c74d0 con 0x7f3380159110 2024-04-24T23:27:06.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:06.188+0000 7f337fe00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f33800c81c0 0x7f33801628e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:50954/0 (socket says 172.21.15.6:50954) 2024-04-24T23:27:06.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:06.188+0000 7f337fe00640 1 -- 172.21.15.6:0/4061951997 learned_addr learned my addr 172.21.15.6:0/4061951997 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:27:06.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:06.188+0000 7f337ea00640 1 --2- 172.21.15.6:0/4061951997 >> v2:172.21.15.6:3300/0 conn(0x7f33800c7dc0 0x7f33801623a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:06.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:06.188+0000 7f337e000640 1 -- 172.21.15.6:0/4061951997 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1012560700 0 0) 0x7f33800c7230 con 0x7f33800c81c0 2024-04-24T23:27:06.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:06.188+0000 7f337e000640 1 -- 172.21.15.6:0/4061951997 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3364003520 con 0x7f33800c81c0 2024-04-24T23:27:06.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:06.189+0000 7f337e000640 1 -- 172.21.15.6:0/4061951997 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1230276919 0 0) 0x7f3364003520 con 0x7f33800c81c0 2024-04-24T23:27:06.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:06.189+0000 7f337e000640 1 -- 172.21.15.6:0/4061951997 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3380159110 msgr2=0x7f3380072420 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-24T23:27:06.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:06.189+0000 7f337e000640 1 --2- 172.21.15.6:0/4061951997 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3380159110 0x7f3380072420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:06.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:06.189+0000 7f337e000640 1 -- 172.21.15.6:0/4061951997 >> v2:172.21.15.6:3300/0 conn(0x7f33800c7dc0 msgr2=0x7f33801623a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:27:06.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:06.189+0000 7f337e000640 1 --2- 172.21.15.6:0/4061951997 >> v2:172.21.15.6:3300/0 conn(0x7f33800c7dc0 0x7f33801623a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:06.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:06.189+0000 7f337e000640 1 -- 172.21.15.6:0/4061951997 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f338006e440 con 0x7f33800c81c0 2024-04-24T23:27:06.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:06.189+0000 7f337e000640 1 -- 172.21.15.6:0/4061951997 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3463268701 0 0) 0x7f33780019d0 con 0x7f33800c81c0 2024-04-24T23:27:06.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:06.190+0000 7f337e000640 1 -- 172.21.15.6:0/4061951997 >> v1:172.21.15.6:6789/0 conn(0x7f33800c81c0 legacy=0x7f33801628e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:06.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:06.190+0000 7f337e000640 1 --2- 172.21.15.6:0/4061951997 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3380159110 0x7f33640039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:06.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:06.190+0000 7f337e000640 1 --2- 172.21.15.6:0/4061951997 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3364003f10 0x7f3364004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:06.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:06.190+0000 7f337e000640 1 --2- 172.21.15.6:0/4061951997 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3364004d40 0x7f33640051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:06.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:06.190+0000 7f337e000640 1 -- 172.21.15.6:0/4061951997 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f33801594f0 con 0x7f3364004d40 2024-04-24T23:27:06.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:06.190+0000 7f337e000640 1 -- 172.21.15.6:0/4061951997 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f33800c74d0 con 0x7f3364003f10 2024-04-24T23:27:06.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:06.190+0000 7f337e000640 1 -- 172.21.15.6:0/4061951997 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f336400bc10 con 0x7f3380159110 2024-04-24T23:27:06.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:06.190+0000 7f337ea00640 1 --2- 172.21.15.6:0/4061951997 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3364003f10 0x7f3364004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:06.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:06.190+0000 7f337fe00640 1 --2- 172.21.15.6:0/4061951997 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3364004d40 0x7f33640051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:06.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:06.190+0000 7f337f400640 1 --2- 172.21.15.6:0/4061951997 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3380159110 0x7f33640039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:06.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:06.191+0000 7f337ea00640 1 -- 172.21.15.6:0/4061951997 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3380159110 msgr2=0x7f33640039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:27:06.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:06.191+0000 7f337ea00640 1 --2- 172.21.15.6:0/4061951997 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3380159110 0x7f33640039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:06.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:06.191+0000 7f337ea00640 1 -- 172.21.15.6:0/4061951997 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3364004d40 msgr2=0x7f33640051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:27:06.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:06.191+0000 7f337ea00640 1 --2- 172.21.15.6:0/4061951997 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3364004d40 0x7f33640051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:06.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:06.191+0000 7f337ea00640 1 -- 172.21.15.6:0/4061951997 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3364003520 con 0x7f3364003f10 2024-04-24T23:27:06.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:06.191+0000 7f337fe00640 1 --2- 172.21.15.6:0/4061951997 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3364004d40 0x7f33640051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:27:06.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:06.193+0000 7f337ea00640 1 --2- 172.21.15.6:0/4061951997 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3364003f10 0x7f3364004800 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f336c009a90 tx=0x7f3364005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c26cc3235b4ea909 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:27:06.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:06.193+0000 7f337e000640 1 -- 172.21.15.6:0/4061951997 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f336c036070 con 0x7f3364003f10 2024-04-24T23:27:06.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:06.193+0000 7f337e000640 1 -- 172.21.15.6:0/4061951997 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f336c031400 con 0x7f3364003f10 2024-04-24T23:27:06.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:06.193+0000 7f337e000640 1 -- 172.21.15.6:0/4061951997 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f336c009690 con 0x7f3364003f10 2024-04-24T23:27:06.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:06.194+0000 7f3386200640 1 -- 172.21.15.6:0/4061951997 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3364003f10 msgr2=0x7f3364004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:06.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:06.194+0000 7f3386200640 1 --2- 172.21.15.6:0/4061951997 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3364003f10 0x7f3364004800 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f336c009a90 tx=0x7f3364005830 comp rx=0 tx=0).stop 2024-04-24T23:27:06.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:06.194+0000 7f337fe00640 1 -- 172.21.15.6:0/4061951997 reap_dead start 2024-04-24T23:27:06.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:06.194+0000 7f3386200640 1 -- 172.21.15.6:0/4061951997 shutdown_connections 2024-04-24T23:27:06.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:06.194+0000 7f3386200640 1 -- 172.21.15.6:0/4061951997 >> 172.21.15.6:0/4061951997 conn(0x7f338006b0e0 msgr2=0x7f338006b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:27:06.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:06.198+0000 7f3386200640 1 -- 172.21.15.6:0/4061951997 shutdown_connections 2024-04-24T23:27:06.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:06.199+0000 7f3386200640 1 -- 172.21.15.6:0/4061951997 wait complete. 2024-04-24T23:27:06.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:06.201+0000 7f3386200640 1 Processor -- start 2024-04-24T23:27:06.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:06.201+0000 7f3386200640 1 -- start start 2024-04-24T23:27:06.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:06.202+0000 7f3386200640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3380152940 0x7f338015ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:06.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:06.202+0000 7f3386200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3380152d20 0x7f3380155560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:06.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:06.202+0000 7f3386200640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f338015ff70 0x7f3380155aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:06.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:06.202+0000 7f3386200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f338006ece0 con 0x7f3380152d20 2024-04-24T23:27:06.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:06.202+0000 7f3386200640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f338006ea00 con 0x7f338015ff70 2024-04-24T23:27:06.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:06.202+0000 7f3386200640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f338006f200 con 0x7f3380152940 2024-04-24T23:27:06.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:06.203+0000 7f337f400640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3380152940 0x7f338015ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:06.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:06.203+0000 7f337fe00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f338015ff70 0x7f3380155aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:06.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:06.203+0000 7f337fe00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f338015ff70 0x7f3380155aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:56440/0 (socket says 172.21.15.6:56440) 2024-04-24T23:27:06.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:06.203+0000 7f337fe00640 1 -- 172.21.15.6:0/310754602 learned_addr learned my addr 172.21.15.6:0/310754602 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:27:06.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:06.203+0000 7f337fe00640 1 -- 172.21.15.6:0/310754602 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3380152940 msgr2=0x7f338015ddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:06.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:06.203+0000 7f337fe00640 1 --2- 172.21.15.6:0/310754602 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3380152940 0x7f338015ddd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:06.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:06.203+0000 7f337fe00640 1 -- 172.21.15.6:0/310754602 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3380152d20 msgr2=0x7f3380155560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:06.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:06.203+0000 7f337fe00640 1 --2- 172.21.15.6:0/310754602 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3380152d20 0x7f3380155560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:06.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:06.203+0000 7f337fe00640 1 -- 172.21.15.6:0/310754602 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33801f4ff0 con 0x7f338015ff70 2024-04-24T23:27:06.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:06.203+0000 7f337f400640 1 --2- 172.21.15.6:0/310754602 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3380152940 0x7f338015ddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:27:06.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:06.204+0000 7f337fe00640 1 --2- 172.21.15.6:0/310754602 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f338015ff70 0x7f3380155aa0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f3378004cb0 tx=0x7f3378008430 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:27:06.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:06.204+0000 7f3377400640 1 -- 172.21.15.6:0/310754602 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f337801a070 con 0x7f338015ff70 2024-04-24T23:27:06.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:06.204+0000 7f3377400640 1 -- 172.21.15.6:0/310754602 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3378008a90 con 0x7f338015ff70 2024-04-24T23:27:06.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:06.204+0000 7f3377400640 1 -- 172.21.15.6:0/310754602 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3378005610 con 0x7f338015ff70 2024-04-24T23:27:06.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:06.204+0000 7f3386200640 1 -- 172.21.15.6:0/310754602 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33801f5290 con 0x7f338015ff70 2024-04-24T23:27:06.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:06.204+0000 7f3386200640 1 -- 172.21.15.6:0/310754602 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33801f5850 con 0x7f338015ff70 2024-04-24T23:27:06.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:06.206+0000 7f3386200640 1 -- 172.21.15.6:0/310754602 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f33800767b0 con 0x7f338015ff70 2024-04-24T23:27:06.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:06.208+0000 7f3377400640 1 -- 172.21.15.6:0/310754602 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f337801d070 con 0x7f338015ff70 2024-04-24T23:27:06.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:06.209+0000 7f3377400640 1 --2- 172.21.15.6:0/310754602 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f336006beb0 0x7f336006e370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:06.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:06.209+0000 7f3377400640 1 -- 172.21.15.6:0/310754602 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6934+0+0 (secure 0 0 0) 0x7f337808b4e0 con 0x7f338015ff70 2024-04-24T23:27:06.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:06.210+0000 7f337f400640 1 --2- 172.21.15.6:0/310754602 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f336006beb0 0x7f336006e370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:06.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:06.211+0000 7f337f400640 1 --2- 172.21.15.6:0/310754602 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f336006beb0 0x7f336006e370 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f3370009e80 tx=0x7f3370006ee0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:27:06.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:06.213+0000 7f3377400640 1 -- 172.21.15.6:0/310754602 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f33780593e0 con 0x7f338015ff70 2024-04-24T23:27:06.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:06.652+0000 7f3386200640 1 -- 172.21.15.6:0/310754602 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3380157220 con 0x7f338015ff70 2024-04-24T23:27:06.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:06.654+0000 7f3377400640 1 -- 172.21.15.6:0/310754602 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) ==== 94+0+6035 (secure 0 0 0) 0x7f337805d6e0 con 0x7f338015ff70 2024-04-24T23:27:06.658 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:27:06.658 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":50,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5081,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/1190759626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1190759626},{"type":"v1","addr":"172.21.15.6:6834","nonce":1190759626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":5249,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6838/2416336040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2416336040},{"type":"v1","addr":"172.21.15.190:6838","nonce":2416336040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":42},{"gid":5503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/3928731059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":3928731059},{"type":"v1","addr":"172.21.15.6:6837","nonce":3928731059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:27:01.572734+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5725,"mds_1":5599},"failed":[],"damaged":[],"stopped":[2,3,4],"info":{"gid_5599":{"gid":5599,"name":"d","rank":1,"incarnation":43,"state":"up:active","state_seq":25,"addr":"172.21.15.190:6839/2643233757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":2643233757},{"type":"v1","addr":"172.21.15.190:6839","nonce":2643233757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5725":{"gid":5725,"name":"f","rank":0,"incarnation":49,"state":"up:resolve","state_seq":30,"addr":"172.21.15.190:6837/1151527362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":1151527362},{"type":"v1","addr":"172.21.15.190:6837","nonce":1151527362}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5599,"qdb_cluster":[5599]},"id":1}]} 2024-04-24T23:27:06.658 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 50 2024-04-24T23:27:06.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:06.658+0000 7f3386200640 1 -- 172.21.15.6:0/310754602 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f336006beb0 msgr2=0x7f336006e370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:06.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:06.658+0000 7f3386200640 1 --2- 172.21.15.6:0/310754602 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f336006beb0 0x7f336006e370 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f3370009e80 tx=0x7f3370006ee0 comp rx=0 tx=0).stop 2024-04-24T23:27:06.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:06.658+0000 7f3386200640 1 -- 172.21.15.6:0/310754602 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f338015ff70 msgr2=0x7f3380155aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:06.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:06.658+0000 7f3386200640 1 --2- 172.21.15.6:0/310754602 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f338015ff70 0x7f3380155aa0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f3378004cb0 tx=0x7f3378008430 comp rx=0 tx=0).stop 2024-04-24T23:27:06.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:06.658+0000 7f3386200640 1 -- 172.21.15.6:0/310754602 shutdown_connections 2024-04-24T23:27:06.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:06.658+0000 7f3386200640 1 --2- 172.21.15.6:0/310754602 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f336006beb0 0x7f336006e370 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:06.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:06.658+0000 7f3386200640 1 --2- 172.21.15.6:0/310754602 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f338015ff70 0x7f3380155aa0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:06.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:06.658+0000 7f3386200640 1 --2- 172.21.15.6:0/310754602 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3380152d20 0x7f3380155560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:06.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:06.658+0000 7f3386200640 1 --2- 172.21.15.6:0/310754602 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3380152940 0x7f338015ddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:06.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:06.658+0000 7f3386200640 1 -- 172.21.15.6:0/310754602 >> 172.21.15.6:0/310754602 conn(0x7f338006b0e0 msgr2=0x7f33800c9260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:27:06.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:06.658+0000 7f3386200640 1 -- 172.21.15.6:0/310754602 shutdown_connections 2024-04-24T23:27:06.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:06.658+0000 7f3386200640 1 -- 172.21.15.6:0/310754602 wait complete. 2024-04-24T23:27:06.691 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-24T23:27:08.692 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:27:08.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:08.929+0000 7faeeba00640 1 Processor -- start 2024-04-24T23:27:08.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:08.929+0000 7faeeba00640 1 -- start start 2024-04-24T23:27:08.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:08.931+0000 7faeeba00640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7faee4152bf0 0x7faee40c5140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:08.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:08.931+0000 7faeeba00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faee40c8d20 0x7faee40c99c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:08.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:08.931+0000 7faeeba00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faee40ca8e0 con 0x7faee40c9f00 2024-04-24T23:27:08.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:08.931+0000 7faeeba00640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7faee40cab80 con 0x7faee4152bf0 2024-04-24T23:27:08.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:08.931+0000 7faeeba00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7faee40c3340 con 0x7faee40c8d20 2024-04-24T23:27:08.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:08.932+0000 7faee9000640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7faee4152bf0 0x7faee40c5140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:08.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:08.932+0000 7faee9000640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7faee4152bf0 0x7faee40c5140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:56456/0 (socket says 172.21.15.6:56456) 2024-04-24T23:27:08.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:08.932+0000 7faee9000640 1 -- 172.21.15.6:0/121231138 learned_addr learned my addr 172.21.15.6:0/121231138 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:27:08.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:08.932+0000 7faee3e00640 1 --2- 172.21.15.6:0/121231138 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faee40c8d20 0x7faee40c99c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:08.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:08.932+0000 7faee9000640 1 -- 172.21.15.6:0/121231138 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faee40c8d20 msgr2=0x7faee40c99c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:27:08.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:08.932+0000 7faee9000640 1 --2- 172.21.15.6:0/121231138 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faee40c8d20 0x7faee40c99c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:08.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:08.932+0000 7faee9000640 1 -- 172.21.15.6:0/121231138 >> v1:172.21.15.6:6789/0 conn(0x7faee40c9f00 legacy=0x7faee40caed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:27:08.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:08.933+0000 7faee9000640 1 -- 172.21.15.6:0/121231138 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faee40c3620 con 0x7faee4152bf0 2024-04-24T23:27:08.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:08.933+0000 7faee3e00640 1 --2- 172.21.15.6:0/121231138 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faee40c8d20 0x7faee40c99c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:27:08.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:08.934+0000 7faee9000640 1 --2- 172.21.15.6:0/121231138 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faee4152bf0 0x7faee40c5140 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7faee4071810 tx=0x7faed80328e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e6c6dcf4f890fcad server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:27:08.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:08.935+0000 7faee3400640 1 -- 172.21.15.6:0/121231138 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faed80385a0 con 0x7faee4152bf0 2024-04-24T23:27:08.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:08.935+0000 7faee3400640 1 -- 172.21.15.6:0/121231138 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faed803e070 con 0x7faee4152bf0 2024-04-24T23:27:08.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:08.935+0000 7faee3400640 1 -- 172.21.15.6:0/121231138 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faed8038960 con 0x7faee4152bf0 2024-04-24T23:27:08.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:08.936+0000 7faeeba00640 1 -- 172.21.15.6:0/121231138 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faee4152bf0 msgr2=0x7faee40c5140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:08.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:08.936+0000 7faeeba00640 1 --2- 172.21.15.6:0/121231138 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faee4152bf0 0x7faee40c5140 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7faee4071810 tx=0x7faed80328e0 comp rx=0 tx=0).stop 2024-04-24T23:27:08.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:08.936+0000 7faeeba00640 1 -- 172.21.15.6:0/121231138 shutdown_connections 2024-04-24T23:27:08.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:08.936+0000 7faeeba00640 1 --2- 172.21.15.6:0/121231138 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faee40c8d20 0x7faee40c99c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:08.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:08.936+0000 7faeeba00640 1 --2- 172.21.15.6:0/121231138 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faee4152bf0 0x7faee40c5140 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:08.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:08.936+0000 7faeeba00640 1 -- 172.21.15.6:0/121231138 >> 172.21.15.6:0/121231138 conn(0x7faee4070280 msgr2=0x7faee406f320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:27:08.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:08.939+0000 7faeeba00640 1 -- 172.21.15.6:0/121231138 shutdown_connections 2024-04-24T23:27:08.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:08.939+0000 7faeeba00640 1 -- 172.21.15.6:0/121231138 wait complete. 2024-04-24T23:27:08.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:08.941+0000 7faeeba00640 1 Processor -- start 2024-04-24T23:27:08.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:08.942+0000 7faeeba00640 1 -- start start 2024-04-24T23:27:08.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:08.943+0000 7faeeba00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faee40c8d20 0x7faee414be60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:08.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:08.943+0000 7faeeba00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faee40c9f00 0x7faee40c6f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:08.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:08.943+0000 7faeeba00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faee4152bf0 0x7faee40c7470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:08.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:08.944+0000 7faeeba00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7faee40c9280 con 0x7faee4152bf0 2024-04-24T23:27:08.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:08.944+0000 7faeeba00640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7faee40c9100 con 0x7faee40c9f00 2024-04-24T23:27:08.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:08.944+0000 7faeeba00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7faee4152fd0 con 0x7faee40c8d20 2024-04-24T23:27:08.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:08.944+0000 7faee9a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faee4152bf0 0x7faee40c7470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:08.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:08.944+0000 7faee9a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faee4152bf0 0x7faee40c7470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:41782/0 (socket says 172.21.15.6:41782) 2024-04-24T23:27:08.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:08.944+0000 7faee9a00640 1 -- 172.21.15.6:0/3339183754 learned_addr learned my addr 172.21.15.6:0/3339183754 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:27:08.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:08.945+0000 7faee9000640 1 --2- 172.21.15.6:0/3339183754 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faee40c8d20 0x7faee414be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:08.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:08.945+0000 7faee9a00640 1 -- 172.21.15.6:0/3339183754 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faee40c8d20 msgr2=0x7faee414be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:08.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:08.945+0000 7faee9a00640 1 --2- 172.21.15.6:0/3339183754 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faee40c8d20 0x7faee414be60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:08.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:08.945+0000 7faee9a00640 1 -- 172.21.15.6:0/3339183754 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faee40c9f00 msgr2=0x7faee40c6f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:08.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:08.945+0000 7faee9a00640 1 --2- 172.21.15.6:0/3339183754 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faee40c9f00 0x7faee40c6f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:08.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:08.945+0000 7faee9a00640 1 -- 172.21.15.6:0/3339183754 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faee4145e40 con 0x7faee4152bf0 2024-04-24T23:27:08.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:08.945+0000 7faee9000640 1 --2- 172.21.15.6:0/3339183754 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faee40c8d20 0x7faee414be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:27:08.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:08.945+0000 7faee9a00640 1 --2- 172.21.15.6:0/3339183754 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faee4152bf0 0x7faee40c7470 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7faedc00c7b0 tx=0x7faedc00cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:27:08.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:08.946+0000 7faee0c00640 1 -- 172.21.15.6:0/3339183754 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faedc004b60 con 0x7faee4152bf0 2024-04-24T23:27:08.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:08.946+0000 7faee0c00640 1 -- 172.21.15.6:0/3339183754 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faedc004d00 con 0x7faee4152bf0 2024-04-24T23:27:08.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:08.946+0000 7faee0c00640 1 -- 172.21.15.6:0/3339183754 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faedc00aa90 con 0x7faee4152bf0 2024-04-24T23:27:08.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:08.947+0000 7faeeba00640 1 -- 172.21.15.6:0/3339183754 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faee4146120 con 0x7faee4152bf0 2024-04-24T23:27:08.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:08.947+0000 7faeeba00640 1 -- 172.21.15.6:0/3339183754 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faee40c6730 con 0x7faee4152bf0 2024-04-24T23:27:08.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:08.948+0000 7faeeba00640 1 -- 172.21.15.6:0/3339183754 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faee40767b0 con 0x7faee4152bf0 2024-04-24T23:27:08.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:08.951+0000 7faee0c00640 1 -- 172.21.15.6:0/3339183754 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7faedc00ac30 con 0x7faee4152bf0 2024-04-24T23:27:08.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:08.952+0000 7faee0c00640 1 --2- 172.21.15.6:0/3339183754 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7faec406beb0 0x7faec406e370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:08.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:08.952+0000 7faee0c00640 1 -- 172.21.15.6:0/3339183754 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6934+0+0 (secure 0 0 0) 0x7faedc08b490 con 0x7faee4152bf0 2024-04-24T23:27:08.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:08.953+0000 7faee9000640 1 --2- 172.21.15.6:0/3339183754 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7faec406beb0 0x7faec406e370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:08.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:08.957+0000 7faee9000640 1 --2- 172.21.15.6:0/3339183754 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7faec406beb0 0x7faec406e370 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7faed803d000 tx=0x7faed8008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:27:08.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:08.962+0000 7faee0c00640 1 -- 172.21.15.6:0/3339183754 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7faedc020070 con 0x7faee4152bf0 2024-04-24T23:27:09.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:09.350+0000 7faeeba00640 1 -- 172.21.15.6:0/3339183754 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7faee40cba10 con 0x7faee4152bf0 2024-04-24T23:27:09.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:09.351+0000 7faee0c00640 1 -- 172.21.15.6:0/3339183754 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) ==== 94+0+6037 (secure 0 0 0) 0x7faedc059a50 con 0x7faee4152bf0 2024-04-24T23:27:09.352 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:27:09.353 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":51,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5081,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/1190759626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1190759626},{"type":"v1","addr":"172.21.15.6:6834","nonce":1190759626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":5249,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6838/2416336040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2416336040},{"type":"v1","addr":"172.21.15.190:6838","nonce":2416336040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":42},{"gid":5503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/3928731059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":3928731059},{"type":"v1","addr":"172.21.15.6:6837","nonce":3928731059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":51,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:27:07.167164+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5725,"mds_1":5599},"failed":[],"damaged":[],"stopped":[2,3,4],"info":{"gid_5599":{"gid":5599,"name":"d","rank":1,"incarnation":43,"state":"up:active","state_seq":25,"addr":"172.21.15.190:6839/2643233757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":2643233757},{"type":"v1","addr":"172.21.15.190:6839","nonce":2643233757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5725":{"gid":5725,"name":"f","rank":0,"incarnation":49,"state":"up:reconnect","state_seq":32,"addr":"172.21.15.190:6837/1151527362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":1151527362},{"type":"v1","addr":"172.21.15.190:6837","nonce":1151527362}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5599,"qdb_cluster":[5599]},"id":1}]} 2024-04-24T23:27:09.353 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 51 2024-04-24T23:27:09.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:09.354+0000 7faeeba00640 1 -- 172.21.15.6:0/3339183754 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7faec406beb0 msgr2=0x7faec406e370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:09.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:09.354+0000 7faeeba00640 1 --2- 172.21.15.6:0/3339183754 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7faec406beb0 0x7faec406e370 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7faed803d000 tx=0x7faed8008030 comp rx=0 tx=0).stop 2024-04-24T23:27:09.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:09.354+0000 7faeeba00640 1 -- 172.21.15.6:0/3339183754 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faee4152bf0 msgr2=0x7faee40c7470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:09.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:09.354+0000 7faeeba00640 1 --2- 172.21.15.6:0/3339183754 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faee4152bf0 0x7faee40c7470 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7faedc00c7b0 tx=0x7faedc00cc70 comp rx=0 tx=0).stop 2024-04-24T23:27:09.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:09.354+0000 7faeeba00640 1 -- 172.21.15.6:0/3339183754 shutdown_connections 2024-04-24T23:27:09.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:09.354+0000 7faeeba00640 1 --2- 172.21.15.6:0/3339183754 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7faec406beb0 0x7faec406e370 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:09.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:09.354+0000 7faeeba00640 1 --2- 172.21.15.6:0/3339183754 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faee4152bf0 0x7faee40c7470 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:09.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:09.354+0000 7faeeba00640 1 --2- 172.21.15.6:0/3339183754 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faee40c9f00 0x7faee40c6f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:09.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:09.354+0000 7faeeba00640 1 --2- 172.21.15.6:0/3339183754 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faee40c8d20 0x7faee414be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:09.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:09.354+0000 7faeeba00640 1 -- 172.21.15.6:0/3339183754 >> 172.21.15.6:0/3339183754 conn(0x7faee4070280 msgr2=0x7faee4070e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:27:09.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:09.355+0000 7faeeba00640 1 -- 172.21.15.6:0/3339183754 shutdown_connections 2024-04-24T23:27:09.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:09.355+0000 7faeeba00640 1 -- 172.21.15.6:0/3339183754 wait complete. 2024-04-24T23:27:09.377 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-24T23:27:11.378 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:27:11.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:11.606+0000 7fe880a00640 1 Processor -- start 2024-04-24T23:27:11.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:11.606+0000 7fe880a00640 1 -- start start 2024-04-24T23:27:11.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:11.607+0000 7fe880a00640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fe87c0cc870 0x7fe87c16b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:11.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:11.607+0000 7fe880a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fe87c164a90 0x7fe87c164ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:11.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:11.607+0000 7fe880a00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe87c072370 con 0x7fe87c0ccc50 2024-04-24T23:27:11.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:11.607+0000 7fe880a00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7fe87c16bb30 con 0x7fe87c164a90 2024-04-24T23:27:11.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:11.607+0000 7fe880a00640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7fe87c070e70 con 0x7fe87c0cc870 2024-04-24T23:27:11.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:11.608+0000 7fe87a800640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fe87c0cc870 0x7fe87c16b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:11.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:11.608+0000 7fe87a800640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fe87c0cc870 0x7fe87c16b5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:56480/0 (socket says 172.21.15.6:56480) 2024-04-24T23:27:11.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:11.608+0000 7fe87a800640 1 -- 172.21.15.6:0/660249764 learned_addr learned my addr 172.21.15.6:0/660249764 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:27:11.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:11.608+0000 7fe87b200640 1 --2- 172.21.15.6:0/660249764 >> v2:172.21.15.6:3300/0 conn(0x7fe87c164a90 0x7fe87c164ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:11.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:11.609+0000 7fe879400640 1 -- 172.21.15.6:0/660249764 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3034578669 0 0) 0x7fe87c072370 con 0x7fe87c0ccc50 2024-04-24T23:27:11.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:11.693+0000 7fe879400640 1 -- 172.21.15.6:0/660249764 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe860003500 con 0x7fe87c0ccc50 2024-04-24T23:27:11.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:11.693+0000 7fe87a800640 1 -- 172.21.15.6:0/660249764 >> v2:172.21.15.6:3300/0 conn(0x7fe87c164a90 msgr2=0x7fe87c164ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:27:11.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:11.693+0000 7fe87a800640 1 --2- 172.21.15.6:0/660249764 >> v2:172.21.15.6:3300/0 conn(0x7fe87c164a90 0x7fe87c164ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:11.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:11.693+0000 7fe87a800640 1 -- 172.21.15.6:0/660249764 >> v1:172.21.15.6:6789/0 conn(0x7fe87c0ccc50 legacy=0x7fe87c06ddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:11.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:11.693+0000 7fe87a800640 1 -- 172.21.15.6:0/660249764 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe87c072650 con 0x7fe87c0cc870 2024-04-24T23:27:11.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:11.695+0000 7fe87a800640 1 --2- 172.21.15.6:0/660249764 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe87c0cc870 0x7fe87c16b5f0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fe87c06ce50 tx=0x7fe86c032c20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=931d1a8a6b01f4a server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:27:11.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:11.695+0000 7fe879400640 1 -- 172.21.15.6:0/660249764 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe86c040070 con 0x7fe87c0cc870 2024-04-24T23:27:11.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:11.695+0000 7fe879400640 1 -- 172.21.15.6:0/660249764 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe86c033970 con 0x7fe87c0cc870 2024-04-24T23:27:11.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:11.695+0000 7fe879400640 1 -- 172.21.15.6:0/660249764 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe86c033d40 con 0x7fe87c0cc870 2024-04-24T23:27:11.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:11.696+0000 7fe880a00640 1 -- 172.21.15.6:0/660249764 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe87c0cc870 msgr2=0x7fe87c16b5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:11.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:11.696+0000 7fe880a00640 1 --2- 172.21.15.6:0/660249764 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe87c0cc870 0x7fe87c16b5f0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fe87c06ce50 tx=0x7fe86c032c20 comp rx=0 tx=0).stop 2024-04-24T23:27:11.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:11.697+0000 7fe880a00640 1 -- 172.21.15.6:0/660249764 shutdown_connections 2024-04-24T23:27:11.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:11.697+0000 7fe880a00640 1 --2- 172.21.15.6:0/660249764 >> v2:172.21.15.6:3300/0 conn(0x7fe87c164a90 0x7fe87c164ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:11.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:11.697+0000 7fe880a00640 1 --2- 172.21.15.6:0/660249764 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe87c0cc870 0x7fe87c16b5f0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:11.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:11.697+0000 7fe880a00640 1 -- 172.21.15.6:0/660249764 >> 172.21.15.6:0/660249764 conn(0x7fe87c06ed80 msgr2=0x7fe87c06f190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:27:11.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:11.697+0000 7fe880a00640 1 -- 172.21.15.6:0/660249764 shutdown_connections 2024-04-24T23:27:11.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:11.697+0000 7fe880a00640 1 -- 172.21.15.6:0/660249764 wait complete. 2024-04-24T23:27:11.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:11.699+0000 7fe880a00640 1 Processor -- start 2024-04-24T23:27:11.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:11.699+0000 7fe880a00640 1 -- start start 2024-04-24T23:27:11.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:11.703+0000 7fe880a00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe87c0cc870 0x7fe87c0bbf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:11.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:11.703+0000 7fe880a00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe87c0ccc50 0x7fe87c166180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:11.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:11.703+0000 7fe880a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe87c164a90 0x7fe87c0c4be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:11.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:11.703+0000 7fe880a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fe87c0713a0 con 0x7fe87c164a90 2024-04-24T23:27:11.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:11.703+0000 7fe880a00640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fe87c0c3ba0 con 0x7fe87c0ccc50 2024-04-24T23:27:11.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:11.703+0000 7fe880a00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fe87c167ac0 con 0x7fe87c0cc870 2024-04-24T23:27:11.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:11.703+0000 7fe879e00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe87c0ccc50 0x7fe87c166180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:11.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:11.703+0000 7fe879e00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe87c0ccc50 0x7fe87c166180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:56488/0 (socket says 172.21.15.6:56488) 2024-04-24T23:27:11.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:11.703+0000 7fe879e00640 1 -- 172.21.15.6:0/788507461 learned_addr learned my addr 172.21.15.6:0/788507461 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:27:11.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:11.703+0000 7fe87a800640 1 --2- 172.21.15.6:0/788507461 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe87c0cc870 0x7fe87c0bbf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:11.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:11.703+0000 7fe87b200640 1 --2- 172.21.15.6:0/788507461 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe87c164a90 0x7fe87c0c4be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:11.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:11.704+0000 7fe87b200640 1 -- 172.21.15.6:0/788507461 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe87c0cc870 msgr2=0x7fe87c0bbf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:11.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:11.704+0000 7fe87b200640 1 --2- 172.21.15.6:0/788507461 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe87c0cc870 0x7fe87c0bbf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:11.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:11.704+0000 7fe87b200640 1 -- 172.21.15.6:0/788507461 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe87c0ccc50 msgr2=0x7fe87c166180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:11.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:11.704+0000 7fe87b200640 1 --2- 172.21.15.6:0/788507461 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe87c0ccc50 0x7fe87c166180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:11.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:11.704+0000 7fe87b200640 1 -- 172.21.15.6:0/788507461 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe87c201090 con 0x7fe87c164a90 2024-04-24T23:27:11.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:11.704+0000 7fe87b200640 1 --2- 172.21.15.6:0/788507461 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe87c164a90 0x7fe87c0c4be0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fe87400fb30 tx=0x7fe874007e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:27:11.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:11.704+0000 7fe872a00640 1 -- 172.21.15.6:0/788507461 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe874018020 con 0x7fe87c164a90 2024-04-24T23:27:11.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:11.705+0000 7fe872a00640 1 -- 172.21.15.6:0/788507461 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe87400da00 con 0x7fe87c164a90 2024-04-24T23:27:11.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:11.705+0000 7fe872a00640 1 -- 172.21.15.6:0/788507461 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe87400ddf0 con 0x7fe87c164a90 2024-04-24T23:27:11.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:11.706+0000 7fe880a00640 1 -- 172.21.15.6:0/788507461 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe87c0c4120 con 0x7fe87c164a90 2024-04-24T23:27:11.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:11.706+0000 7fe880a00640 1 -- 172.21.15.6:0/788507461 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe87c0c43e0 con 0x7fe87c164a90 2024-04-24T23:27:11.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:11.707+0000 7fe880a00640 1 -- 172.21.15.6:0/788507461 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe87c075b30 con 0x7fe87c164a90 2024-04-24T23:27:11.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:11.710+0000 7fe872a00640 1 -- 172.21.15.6:0/788507461 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fe87401b020 con 0x7fe87c164a90 2024-04-24T23:27:11.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:11.711+0000 7fe872a00640 1 --2- 172.21.15.6:0/788507461 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fe85c06be80 0x7fe85c06e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:11.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:11.711+0000 7fe872a00640 1 -- 172.21.15.6:0/788507461 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6934+0+0 (secure 0 0 0) 0x7fe87408b5d0 con 0x7fe87c164a90 2024-04-24T23:27:11.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:11.711+0000 7fe87a800640 1 --2- 172.21.15.6:0/788507461 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fe85c06be80 0x7fe85c06e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:11.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:11.712+0000 7fe87a800640 1 --2- 172.21.15.6:0/788507461 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fe85c06be80 0x7fe85c06e340 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fe86c03d000 tx=0x7fe86c037000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:27:11.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:11.718+0000 7fe872a00640 1 -- 172.21.15.6:0/788507461 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fe874022030 con 0x7fe87c164a90 2024-04-24T23:27:12.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:12.065+0000 7fe880a00640 1 -- 172.21.15.6:0/788507461 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe87c0cbdd0 con 0x7fe87c164a90 2024-04-24T23:27:12.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:12.066+0000 7fe872a00640 1 -- 172.21.15.6:0/788507461 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) ==== 94+0+6037 (secure 0 0 0) 0x7fe874059b90 con 0x7fe87c164a90 2024-04-24T23:27:12.070 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:27:12.070 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":51,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5081,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/1190759626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1190759626},{"type":"v1","addr":"172.21.15.6:6834","nonce":1190759626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":5249,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6838/2416336040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2416336040},{"type":"v1","addr":"172.21.15.190:6838","nonce":2416336040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":42},{"gid":5503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/3928731059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":3928731059},{"type":"v1","addr":"172.21.15.6:6837","nonce":3928731059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":51,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:27:07.167164+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5725,"mds_1":5599},"failed":[],"damaged":[],"stopped":[2,3,4],"info":{"gid_5599":{"gid":5599,"name":"d","rank":1,"incarnation":43,"state":"up:active","state_seq":25,"addr":"172.21.15.190:6839/2643233757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":2643233757},{"type":"v1","addr":"172.21.15.190:6839","nonce":2643233757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5725":{"gid":5725,"name":"f","rank":0,"incarnation":49,"state":"up:reconnect","state_seq":32,"addr":"172.21.15.190:6837/1151527362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":1151527362},{"type":"v1","addr":"172.21.15.190:6837","nonce":1151527362}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5599,"qdb_cluster":[5599]},"id":1}]} 2024-04-24T23:27:12.072 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 51 2024-04-24T23:27:12.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:12.069+0000 7fe880a00640 1 -- 172.21.15.6:0/788507461 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fe85c06be80 msgr2=0x7fe85c06e340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:12.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:12.069+0000 7fe880a00640 1 --2- 172.21.15.6:0/788507461 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fe85c06be80 0x7fe85c06e340 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fe86c03d000 tx=0x7fe86c037000 comp rx=0 tx=0).stop 2024-04-24T23:27:12.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:12.069+0000 7fe880a00640 1 -- 172.21.15.6:0/788507461 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe87c164a90 msgr2=0x7fe87c0c4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:12.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:12.069+0000 7fe880a00640 1 --2- 172.21.15.6:0/788507461 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe87c164a90 0x7fe87c0c4be0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fe87400fb30 tx=0x7fe874007e20 comp rx=0 tx=0).stop 2024-04-24T23:27:12.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:12.069+0000 7fe880a00640 1 -- 172.21.15.6:0/788507461 shutdown_connections 2024-04-24T23:27:12.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:12.069+0000 7fe880a00640 1 --2- 172.21.15.6:0/788507461 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fe85c06be80 0x7fe85c06e340 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:12.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:12.069+0000 7fe880a00640 1 --2- 172.21.15.6:0/788507461 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe87c164a90 0x7fe87c0c4be0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:12.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:12.069+0000 7fe880a00640 1 --2- 172.21.15.6:0/788507461 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe87c0ccc50 0x7fe87c166180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:12.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:12.069+0000 7fe880a00640 1 --2- 172.21.15.6:0/788507461 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe87c0cc870 0x7fe87c0bbf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:12.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:12.069+0000 7fe880a00640 1 -- 172.21.15.6:0/788507461 >> 172.21.15.6:0/788507461 conn(0x7fe87c06ed80 msgr2=0x7fe87c074630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:27:12.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:12.069+0000 7fe880a00640 1 -- 172.21.15.6:0/788507461 shutdown_connections 2024-04-24T23:27:12.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:12.069+0000 7fe880a00640 1 -- 172.21.15.6:0/788507461 wait complete. 2024-04-24T23:27:12.089 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-24T23:27:14.091 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:27:14.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:14.300+0000 7fe60a400640 1 Processor -- start 2024-04-24T23:27:14.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:14.300+0000 7fe60a400640 1 -- start start 2024-04-24T23:27:14.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:14.302+0000 7fe60a400640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fe6040cc870 0x7fe60416b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:14.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:14.302+0000 7fe60a400640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe604072370 con 0x7fe6040ccc50 2024-04-24T23:27:14.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:14.302+0000 7fe60a400640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe604072650 con 0x7fe604164a90 2024-04-24T23:27:14.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:14.302+0000 7fe60a400640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7fe60416bb30 con 0x7fe6040cc870 2024-04-24T23:27:14.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:14.302+0000 7fe603e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fe6040ccc50 0x7fe60406ddd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:50974/0 (socket says 172.21.15.6:50974) 2024-04-24T23:27:14.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:14.302+0000 7fe603e00640 1 -- 172.21.15.6:0/1041400399 learned_addr learned my addr 172.21.15.6:0/1041400399 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:27:14.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:14.303+0000 7fe609000640 1 --2- 172.21.15.6:0/1041400399 >> v2:172.21.15.190:3300/0 conn(0x7fe6040cc870 0x7fe60416b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:14.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:14.303+0000 7fe603400640 1 -- 172.21.15.6:0/1041400399 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 823687987 0 0) 0x7fe604072370 con 0x7fe6040ccc50 2024-04-24T23:27:14.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:14.366+0000 7fe603400640 1 -- 172.21.15.6:0/1041400399 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe5ec003500 con 0x7fe6040ccc50 2024-04-24T23:27:14.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:14.366+0000 7fe603400640 1 -- 172.21.15.6:0/1041400399 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 510778193 0 0) 0x7fe604072650 con 0x7fe604164a90 2024-04-24T23:27:14.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:14.366+0000 7fe603400640 1 -- 172.21.15.6:0/1041400399 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe604072370 con 0x7fe604164a90 2024-04-24T23:27:14.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:14.366+0000 7fe603400640 1 -- 172.21.15.6:0/1041400399 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3597456073 0 0) 0x7fe604072370 con 0x7fe604164a90 2024-04-24T23:27:14.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:14.366+0000 7fe603400640 1 -- 172.21.15.6:0/1041400399 >> v2:172.21.15.190:3300/0 conn(0x7fe6040cc870 msgr2=0x7fe60416b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:27:14.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:14.366+0000 7fe603400640 1 --2- 172.21.15.6:0/1041400399 >> v2:172.21.15.190:3300/0 conn(0x7fe6040cc870 0x7fe60416b5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:14.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:14.366+0000 7fe603400640 1 -- 172.21.15.6:0/1041400399 >> v1:172.21.15.6:6789/0 conn(0x7fe6040ccc50 legacy=0x7fe60406ddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:14.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:14.366+0000 7fe603400640 1 -- 172.21.15.6:0/1041400399 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6040710c0 con 0x7fe604164a90 2024-04-24T23:27:14.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:14.366+0000 7fe603400640 1 -- 172.21.15.6:0/1041400399 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3463268701 0 0) 0x7fe5fc0019f0 con 0x7fe604164a90 2024-04-24T23:27:14.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:14.367+0000 7fe603400640 1 -- 172.21.15.6:0/1041400399 >> v1:172.21.15.190:6789/0 conn(0x7fe604164a90 legacy=0x7fe604167ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:14.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:14.367+0000 7fe603400640 1 --2- 172.21.15.6:0/1041400399 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe5ec003cd0 0x7fe5ec0040d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:14.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:14.367+0000 7fe603400640 1 --2- 172.21.15.6:0/1041400399 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe5ec004610 0x7fe5ec00d410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:14.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:14.367+0000 7fe603400640 1 --2- 172.21.15.6:0/1041400399 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe5ec00d950 0x7fe5ec00fd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:14.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:14.367+0000 7fe603400640 1 -- 172.21.15.6:0/1041400399 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fe60416bb30 con 0x7fe5ec004610 2024-04-24T23:27:14.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:14.367+0000 7fe603400640 1 -- 172.21.15.6:0/1041400399 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fe5ec010510 con 0x7fe5ec00d950 2024-04-24T23:27:14.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:14.367+0000 7fe603400640 1 -- 172.21.15.6:0/1041400399 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fe5ec010710 con 0x7fe5ec003cd0 2024-04-24T23:27:14.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:14.367+0000 7fe609000640 1 --2- 172.21.15.6:0/1041400399 >> v2:172.21.15.190:3300/0 conn(0x7fe6040cc870 0x7fe60416b5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:27:14.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:14.367+0000 7fe603e00640 1 --2- 172.21.15.6:0/1041400399 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe5ec004610 0x7fe5ec00d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:14.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:14.367+0000 7fe609a00640 1 --2- 172.21.15.6:0/1041400399 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe5ec00d950 0x7fe5ec00fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:14.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:14.368+0000 7fe609000640 1 --2- 172.21.15.6:0/1041400399 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe5ec003cd0 0x7fe5ec0040d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:14.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:14.368+0000 7fe603e00640 1 -- 172.21.15.6:0/1041400399 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe5ec003cd0 msgr2=0x7fe5ec0040d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:27:14.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:14.368+0000 7fe603e00640 1 --2- 172.21.15.6:0/1041400399 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe5ec003cd0 0x7fe5ec0040d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:14.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:14.368+0000 7fe603e00640 1 -- 172.21.15.6:0/1041400399 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe5ec00d950 msgr2=0x7fe5ec00fd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:27:14.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:14.368+0000 7fe603e00640 1 --2- 172.21.15.6:0/1041400399 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe5ec00d950 0x7fe5ec00fd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:14.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:14.368+0000 7fe603e00640 1 -- 172.21.15.6:0/1041400399 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe604072370 con 0x7fe5ec004610 2024-04-24T23:27:14.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:14.368+0000 7fe609a00640 1 --2- 172.21.15.6:0/1041400399 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe5ec00d950 0x7fe5ec00fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:27:14.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:14.368+0000 7fe609000640 1 --2- 172.21.15.6:0/1041400399 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe5ec003cd0 0x7fe5ec0040d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:27:14.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:14.368+0000 7fe609a00640 1 -- 172.21.15.6:0/1041400399 reap_dead start 2024-04-24T23:27:14.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:14.371+0000 7fe603e00640 1 --2- 172.21.15.6:0/1041400399 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe5ec004610 0x7fe5ec00d410 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fe6040cd060 tx=0x7fe5ec004d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e51d181d1dc38c88 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:27:14.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:14.372+0000 7fe603400640 1 -- 172.21.15.6:0/1041400399 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5f0001610 con 0x7fe5ec004610 2024-04-24T23:27:14.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:14.372+0000 7fe603400640 1 -- 172.21.15.6:0/1041400399 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe5f002e070 con 0x7fe5ec004610 2024-04-24T23:27:14.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:14.372+0000 7fe603400640 1 -- 172.21.15.6:0/1041400399 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5f00048a0 con 0x7fe5ec004610 2024-04-24T23:27:14.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:14.373+0000 7fe60a400640 1 -- 172.21.15.6:0/1041400399 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe5ec004610 msgr2=0x7fe5ec00d410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:14.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:14.373+0000 7fe60a400640 1 --2- 172.21.15.6:0/1041400399 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe5ec004610 0x7fe5ec00d410 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fe6040cd060 tx=0x7fe5ec004d50 comp rx=0 tx=0).stop 2024-04-24T23:27:14.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:14.373+0000 7fe60a400640 1 -- 172.21.15.6:0/1041400399 shutdown_connections 2024-04-24T23:27:14.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:14.373+0000 7fe60a400640 1 --2- 172.21.15.6:0/1041400399 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe5ec004610 0x7fe5ec00d410 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:14.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:14.373+0000 7fe60a400640 1 -- 172.21.15.6:0/1041400399 >> 172.21.15.6:0/1041400399 conn(0x7fe60406ed80 msgr2=0x7fe60406f190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:27:14.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:14.373+0000 7fe60a400640 1 -- 172.21.15.6:0/1041400399 shutdown_connections 2024-04-24T23:27:14.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:14.374+0000 7fe60a400640 1 -- 172.21.15.6:0/1041400399 wait complete. 2024-04-24T23:27:14.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:14.378+0000 7fe60a400640 1 Processor -- start 2024-04-24T23:27:14.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:14.379+0000 7fe60a400640 1 -- start start 2024-04-24T23:27:14.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:14.381+0000 7fe60a400640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe5ec004610 0x7fe6041603a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:14.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:14.381+0000 7fe60a400640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe6040c63f0 0x7fe60415cf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:14.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:14.382+0000 7fe60a400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe60415dfc0 0x7fe60415e510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:14.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:14.382+0000 7fe60a400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fe6040c3ba0 con 0x7fe60415dfc0 2024-04-24T23:27:14.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:14.382+0000 7fe60a400640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fe604070e70 con 0x7fe6040c63f0 2024-04-24T23:27:14.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:14.382+0000 7fe60a400640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fe60416b5f0 con 0x7fe5ec004610 2024-04-24T23:27:14.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:14.382+0000 7fe609000640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe5ec004610 0x7fe6041603a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:14.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:14.382+0000 7fe609000640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe5ec004610 0x7fe6041603a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.6:58282/0 (socket says 172.21.15.6:58282) 2024-04-24T23:27:14.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:14.382+0000 7fe609000640 1 -- 172.21.15.6:0/1473928259 learned_addr learned my addr 172.21.15.6:0/1473928259 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:27:14.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:14.382+0000 7fe603e00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe6040c63f0 0x7fe60415cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:14.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:14.382+0000 7fe609a00640 1 --2- 172.21.15.6:0/1473928259 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe60415dfc0 0x7fe60415e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:14.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:14.382+0000 7fe609000640 1 -- 172.21.15.6:0/1473928259 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe6040c63f0 msgr2=0x7fe60415cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:14.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:14.382+0000 7fe609000640 1 --2- 172.21.15.6:0/1473928259 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe6040c63f0 0x7fe60415cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:14.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:14.382+0000 7fe609000640 1 -- 172.21.15.6:0/1473928259 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe60415dfc0 msgr2=0x7fe60415e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:14.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:14.382+0000 7fe609000640 1 --2- 172.21.15.6:0/1473928259 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe60415dfc0 0x7fe60415e510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:14.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:14.382+0000 7fe609000640 1 -- 172.21.15.6:0/1473928259 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6040c98c0 con 0x7fe5ec004610 2024-04-24T23:27:14.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:14.383+0000 7fe609000640 1 --2- 172.21.15.6:0/1473928259 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe5ec004610 0x7fe6041603a0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fe5f8010900 tx=0x7fe5f8010d00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:27:14.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:14.383+0000 7fe600c00640 1 -- 172.21.15.6:0/1473928259 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5f800f040 con 0x7fe5ec004610 2024-04-24T23:27:14.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:14.383+0000 7fe600c00640 1 -- 172.21.15.6:0/1473928259 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe5f800eb60 con 0x7fe5ec004610 2024-04-24T23:27:14.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:14.383+0000 7fe600c00640 1 -- 172.21.15.6:0/1473928259 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5f801c400 con 0x7fe5ec004610 2024-04-24T23:27:14.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:14.383+0000 7fe60a400640 1 -- 172.21.15.6:0/1473928259 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe604160990 con 0x7fe5ec004610 2024-04-24T23:27:14.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:14.384+0000 7fe60a400640 1 -- 172.21.15.6:0/1473928259 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fe604200940 con 0x7fe5ec004610 2024-04-24T23:27:14.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:14.385+0000 7fe60a400640 1 -- 172.21.15.6:0/1473928259 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe604075b30 con 0x7fe5ec004610 2024-04-24T23:27:14.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:14.386+0000 7fe600c00640 1 -- 172.21.15.6:0/1473928259 <== mon.2 v2:172.21.15.190:3301/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fe5f801c5a0 con 0x7fe5ec004610 2024-04-24T23:27:14.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:14.387+0000 7fe600c00640 1 --2- 172.21.15.6:0/1473928259 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fe5e406be60 0x7fe5e406e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:14.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:14.388+0000 7fe600c00640 1 -- 172.21.15.6:0/1473928259 <== mon.2 v2:172.21.15.190:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6934+0+0 (secure 0 0 0) 0x7fe5f808b8c0 con 0x7fe5ec004610 2024-04-24T23:27:14.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:14.388+0000 7fe603e00640 1 --2- 172.21.15.6:0/1473928259 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fe5e406be60 0x7fe5e406e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:14.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:14.388+0000 7fe603e00640 1 --2- 172.21.15.6:0/1473928259 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fe5e406be60 0x7fe5e406e320 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fe5f0032000 tx=0x7fe5f0007000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:27:14.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:14.391+0000 7fe600c00640 1 -- 172.21.15.6:0/1473928259 <== mon.2 v2:172.21.15.190:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fe5f8018030 con 0x7fe5ec004610 2024-04-24T23:27:14.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:14.640+0000 7fe60a400640 1 -- 172.21.15.6:0/1473928259 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe604071ef0 con 0x7fe5ec004610 2024-04-24T23:27:14.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:14.641+0000 7fe600c00640 1 -- 172.21.15.6:0/1473928259 <== mon.2 v2:172.21.15.190:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 52 v52) ==== 94+0+6034 (secure 0 0 0) 0x7fe5f8059d00 con 0x7fe5ec004610 2024-04-24T23:27:14.643 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:27:14.643 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":52,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5081,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/1190759626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1190759626},{"type":"v1","addr":"172.21.15.6:6834","nonce":1190759626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":5249,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6838/2416336040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2416336040},{"type":"v1","addr":"172.21.15.190:6838","nonce":2416336040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":42},{"gid":5503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/3928731059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":3928731059},{"type":"v1","addr":"172.21.15.6:6837","nonce":3928731059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":52,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:27:12.194226+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5725,"mds_1":5599},"failed":[],"damaged":[],"stopped":[2,3,4],"info":{"gid_5599":{"gid":5599,"name":"d","rank":1,"incarnation":43,"state":"up:active","state_seq":25,"addr":"172.21.15.190:6839/2643233757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":2643233757},{"type":"v1","addr":"172.21.15.190:6839","nonce":2643233757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5725":{"gid":5725,"name":"f","rank":0,"incarnation":49,"state":"up:rejoin","state_seq":34,"addr":"172.21.15.190:6837/1151527362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":1151527362},{"type":"v1","addr":"172.21.15.190:6837","nonce":1151527362}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5599,"qdb_cluster":[5599]},"id":1}]} 2024-04-24T23:27:14.643 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 52 2024-04-24T23:27:14.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:14.644+0000 7fe60a400640 1 -- 172.21.15.6:0/1473928259 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fe5e406be60 msgr2=0x7fe5e406e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:14.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:14.644+0000 7fe60a400640 1 --2- 172.21.15.6:0/1473928259 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fe5e406be60 0x7fe5e406e320 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fe5f0032000 tx=0x7fe5f0007000 comp rx=0 tx=0).stop 2024-04-24T23:27:14.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:14.644+0000 7fe60a400640 1 -- 172.21.15.6:0/1473928259 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe5ec004610 msgr2=0x7fe6041603a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:14.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:14.644+0000 7fe60a400640 1 --2- 172.21.15.6:0/1473928259 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe5ec004610 0x7fe6041603a0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fe5f8010900 tx=0x7fe5f8010d00 comp rx=0 tx=0).stop 2024-04-24T23:27:14.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:14.644+0000 7fe60a400640 1 -- 172.21.15.6:0/1473928259 shutdown_connections 2024-04-24T23:27:14.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:14.644+0000 7fe60a400640 1 --2- 172.21.15.6:0/1473928259 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fe5e406be60 0x7fe5e406e320 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:14.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:14.644+0000 7fe60a400640 1 --2- 172.21.15.6:0/1473928259 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe60415dfc0 0x7fe60415e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:14.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:14.644+0000 7fe60a400640 1 --2- 172.21.15.6:0/1473928259 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe6040c63f0 0x7fe60415cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:14.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:14.644+0000 7fe60a400640 1 --2- 172.21.15.6:0/1473928259 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe5ec004610 0x7fe6041603a0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:14.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:14.644+0000 7fe60a400640 1 -- 172.21.15.6:0/1473928259 >> 172.21.15.6:0/1473928259 conn(0x7fe60406ed80 msgr2=0x7fe6040ca0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:27:14.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:14.644+0000 7fe60a400640 1 -- 172.21.15.6:0/1473928259 shutdown_connections 2024-04-24T23:27:14.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:14.644+0000 7fe60a400640 1 -- 172.21.15.6:0/1473928259 wait complete. 2024-04-24T23:27:14.665 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-24T23:27:16.666 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:27:16.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:16.864+0000 7fa7abe00640 1 Processor -- start 2024-04-24T23:27:16.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:16.864+0000 7fa7abe00640 1 -- start start 2024-04-24T23:27:16.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:16.866+0000 7fa7abe00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa7ac152990 0x7fa7ac071fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:16.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:16.866+0000 7fa7abe00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa7ac0c6a10 0x7fa7ac0c6e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:16.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:16.866+0000 7fa7abe00640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fa7ac158cf0 0x7fa7ac159160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:16.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:16.866+0000 7fa7abe00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7fa7ac1596a0 con 0x7fa7ac0c6a10 2024-04-24T23:27:16.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:16.866+0000 7fa7abe00640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7fa7ac070590 con 0x7fa7ac158cf0 2024-04-24T23:27:16.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:16.866+0000 7fa7abe00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fa7ac1587f0 con 0x7fa7ac152990 2024-04-24T23:27:16.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:16.866+0000 7fa7aa000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa7ac0c6a10 0x7fa7ac0c6e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:16.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:16.866+0000 7fa7ab400640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fa7ac158cf0 0x7fa7ac159160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:16.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:16.866+0000 7fa7aaa00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa7ac152990 0x7fa7ac071fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:16.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:16.866+0000 7fa7ab400640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fa7ac158cf0 0x7fa7ac159160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:37940/0 (socket says 172.21.15.6:37940) 2024-04-24T23:27:16.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:16.866+0000 7fa7aa000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa7ac0c6a10 0x7fa7ac0c6e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:59172/0 (socket says 172.21.15.6:59172) 2024-04-24T23:27:16.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:16.866+0000 7fa7aa000640 1 -- 172.21.15.6:0/4253212267 learned_addr learned my addr 172.21.15.6:0/4253212267 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:27:16.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:16.866+0000 7fa7aaa00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa7ac152990 0x7fa7ac071fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.6:57092/0 (socket says 172.21.15.6:57092) 2024-04-24T23:27:16.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:16.867+0000 7fa7aaa00640 1 -- 172.21.15.6:0/4253212267 >> v2:172.21.15.190:3300/0 conn(0x7fa7ac158cf0 msgr2=0x7fa7ac159160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:27:16.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:16.867+0000 7fa7aaa00640 1 --2- 172.21.15.6:0/4253212267 >> v2:172.21.15.190:3300/0 conn(0x7fa7ac158cf0 0x7fa7ac159160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:16.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:16.867+0000 7fa7aaa00640 1 -- 172.21.15.6:0/4253212267 >> v2:172.21.15.6:3300/0 conn(0x7fa7ac0c6a10 msgr2=0x7fa7ac0c6e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:27:16.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:16.867+0000 7fa7aaa00640 1 --2- 172.21.15.6:0/4253212267 >> v2:172.21.15.6:3300/0 conn(0x7fa7ac0c6a10 0x7fa7ac0c6e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:16.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:16.867+0000 7fa7aaa00640 1 -- 172.21.15.6:0/4253212267 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7ac0724e0 con 0x7fa7ac152990 2024-04-24T23:27:16.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:16.868+0000 7fa7aaa00640 1 --2- 172.21.15.6:0/4253212267 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa7ac152990 0x7fa7ac071fa0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fa7ac068150 tx=0x7fa79c02a1a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=87db144460b5404 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:27:16.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:16.868+0000 7fa7a9600640 1 -- 172.21.15.6:0/4253212267 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa79c037070 con 0x7fa7ac152990 2024-04-24T23:27:16.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:16.868+0000 7fa7a9600640 1 -- 172.21.15.6:0/4253212267 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa79c032440 con 0x7fa7ac152990 2024-04-24T23:27:16.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:16.868+0000 7fa7a9600640 1 -- 172.21.15.6:0/4253212267 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa79c032810 con 0x7fa7ac152990 2024-04-24T23:27:16.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:16.869+0000 7fa7abe00640 1 -- 172.21.15.6:0/4253212267 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa7ac152990 msgr2=0x7fa7ac071fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:16.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:16.869+0000 7fa7abe00640 1 --2- 172.21.15.6:0/4253212267 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa7ac152990 0x7fa7ac071fa0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fa7ac068150 tx=0x7fa79c02a1a0 comp rx=0 tx=0).stop 2024-04-24T23:27:16.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:16.869+0000 7fa7abe00640 1 -- 172.21.15.6:0/4253212267 shutdown_connections 2024-04-24T23:27:16.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:16.869+0000 7fa7abe00640 1 --2- 172.21.15.6:0/4253212267 >> v2:172.21.15.190:3300/0 conn(0x7fa7ac158cf0 0x7fa7ac159160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:16.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:16.869+0000 7fa7abe00640 1 --2- 172.21.15.6:0/4253212267 >> v2:172.21.15.6:3300/0 conn(0x7fa7ac0c6a10 0x7fa7ac0c6e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:16.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:16.869+0000 7fa7abe00640 1 --2- 172.21.15.6:0/4253212267 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa7ac152990 0x7fa7ac071fa0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:16.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:16.869+0000 7fa7abe00640 1 -- 172.21.15.6:0/4253212267 >> 172.21.15.6:0/4253212267 conn(0x7fa7ac06e590 msgr2=0x7fa7ac0cd300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:27:16.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:16.869+0000 7fa7abe00640 1 -- 172.21.15.6:0/4253212267 shutdown_connections 2024-04-24T23:27:16.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:16.869+0000 7fa7abe00640 1 -- 172.21.15.6:0/4253212267 wait complete. 2024-04-24T23:27:16.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:16.871+0000 7fa7abe00640 1 Processor -- start 2024-04-24T23:27:16.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:16.871+0000 7fa7abe00640 1 -- start start 2024-04-24T23:27:16.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:16.872+0000 7fa7abe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa7ac0c6a10 0x7fa7ac15c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:16.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:16.872+0000 7fa7abe00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa7ac152990 0x7fa7ac1eeaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:16.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:16.872+0000 7fa7abe00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa7ac158cf0 0x7fa7ac1e43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:16.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:16.872+0000 7fa7abe00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fa7ac072ee0 con 0x7fa7ac0c6a10 2024-04-24T23:27:16.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:16.872+0000 7fa7abe00640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fa7ac072c00 con 0x7fa7ac152990 2024-04-24T23:27:16.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:16.872+0000 7fa7abe00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fa7ac0731c0 con 0x7fa7ac158cf0 2024-04-24T23:27:16.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:16.872+0000 7fa7aaa00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa7ac0c6a10 0x7fa7ac15c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:16.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:16.872+0000 7fa7aaa00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa7ac0c6a10 0x7fa7ac15c890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:59176/0 (socket says 172.21.15.6:59176) 2024-04-24T23:27:16.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:16.872+0000 7fa7aaa00640 1 -- 172.21.15.6:0/2894768023 learned_addr learned my addr 172.21.15.6:0/2894768023 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:27:16.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:16.872+0000 7fa7aa000640 1 --2- 172.21.15.6:0/2894768023 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa7ac152990 0x7fa7ac1eeaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:16.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:16.872+0000 7fa7ab400640 1 --2- 172.21.15.6:0/2894768023 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa7ac158cf0 0x7fa7ac1e43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:16.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:16.872+0000 7fa7aaa00640 1 -- 172.21.15.6:0/2894768023 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa7ac158cf0 msgr2=0x7fa7ac1e43b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:16.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:16.872+0000 7fa7aaa00640 1 --2- 172.21.15.6:0/2894768023 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa7ac158cf0 0x7fa7ac1e43b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:16.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:16.872+0000 7fa7aaa00640 1 -- 172.21.15.6:0/2894768023 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa7ac152990 msgr2=0x7fa7ac1eeaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:16.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:16.872+0000 7fa7aaa00640 1 --2- 172.21.15.6:0/2894768023 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa7ac152990 0x7fa7ac1eeaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:16.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:16.873+0000 7fa7aaa00640 1 -- 172.21.15.6:0/2894768023 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7ac1599b0 con 0x7fa7ac0c6a10 2024-04-24T23:27:16.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:16.873+0000 7fa7ab400640 1 --2- 172.21.15.6:0/2894768023 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa7ac158cf0 0x7fa7ac1e43b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:27:16.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:16.873+0000 7fa7aaa00640 1 --2- 172.21.15.6:0/2894768023 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa7ac0c6a10 0x7fa7ac15c890 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fa79c009890 tx=0x7fa79c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:27:16.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:16.873+0000 7fa7a2a00640 1 -- 172.21.15.6:0/2894768023 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa79c041020 con 0x7fa7ac0c6a10 2024-04-24T23:27:16.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:16.873+0000 7fa7a2a00640 1 -- 172.21.15.6:0/2894768023 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa79c03d700 con 0x7fa7ac0c6a10 2024-04-24T23:27:16.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:16.873+0000 7fa7abe00640 1 -- 172.21.15.6:0/2894768023 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7ac15a230 con 0x7fa7ac0c6a10 2024-04-24T23:27:16.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:16.873+0000 7fa7a2a00640 1 -- 172.21.15.6:0/2894768023 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa79c037040 con 0x7fa7ac0c6a10 2024-04-24T23:27:16.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:16.873+0000 7fa7abe00640 1 -- 172.21.15.6:0/2894768023 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7ac159f50 con 0x7fa7ac0c6a10 2024-04-24T23:27:16.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:16.874+0000 7fa7abe00640 1 -- 172.21.15.6:0/2894768023 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa7ac0728b0 con 0x7fa7ac0c6a10 2024-04-24T23:27:16.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:16.875+0000 7fa7a2a00640 1 -- 172.21.15.6:0/2894768023 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fa79c044020 con 0x7fa7ac0c6a10 2024-04-24T23:27:16.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:16.877+0000 7fa7a2a00640 1 --2- 172.21.15.6:0/2894768023 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fa78c06be30 0x7fa78c06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:16.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:16.877+0000 7fa7a2a00640 1 -- 172.21.15.6:0/2894768023 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6934+0+0 (secure 0 0 0) 0x7fa79c0abb20 con 0x7fa7ac0c6a10 2024-04-24T23:27:16.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:16.877+0000 7fa7aa000640 1 --2- 172.21.15.6:0/2894768023 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fa78c06be30 0x7fa78c06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:16.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:16.878+0000 7fa7aa000640 1 --2- 172.21.15.6:0/2894768023 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fa78c06be30 0x7fa78c06e2f0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fa7ac1efbd0 tx=0x7fa798009240 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:27:16.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:16.881+0000 7fa7a2a00640 1 -- 172.21.15.6:0/2894768023 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fa7ac0728b0 con 0x7fa7ac0c6a10 2024-04-24T23:27:17.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:17.119+0000 7fa7abe00640 1 -- 172.21.15.6:0/2894768023 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa7ac075b30 con 0x7fa7ac0c6a10 2024-04-24T23:27:17.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:17.120+0000 7fa7a2a00640 1 -- 172.21.15.6:0/2894768023 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 52 v52) ==== 94+0+6034 (secure 0 0 0) 0x7fa79c04b030 con 0x7fa7ac0c6a10 2024-04-24T23:27:17.122 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:27:17.122 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":52,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5081,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/1190759626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1190759626},{"type":"v1","addr":"172.21.15.6:6834","nonce":1190759626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":5249,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6838/2416336040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2416336040},{"type":"v1","addr":"172.21.15.190:6838","nonce":2416336040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":42},{"gid":5503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/3928731059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":3928731059},{"type":"v1","addr":"172.21.15.6:6837","nonce":3928731059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":52,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:27:12.194226+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5725,"mds_1":5599},"failed":[],"damaged":[],"stopped":[2,3,4],"info":{"gid_5599":{"gid":5599,"name":"d","rank":1,"incarnation":43,"state":"up:active","state_seq":25,"addr":"172.21.15.190:6839/2643233757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":2643233757},{"type":"v1","addr":"172.21.15.190:6839","nonce":2643233757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5725":{"gid":5725,"name":"f","rank":0,"incarnation":49,"state":"up:rejoin","state_seq":34,"addr":"172.21.15.190:6837/1151527362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":1151527362},{"type":"v1","addr":"172.21.15.190:6837","nonce":1151527362}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5599,"qdb_cluster":[5599]},"id":1}]} 2024-04-24T23:27:17.122 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 52 2024-04-24T23:27:17.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:17.124+0000 7fa7abe00640 1 -- 172.21.15.6:0/2894768023 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fa78c06be30 msgr2=0x7fa78c06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:17.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:17.124+0000 7fa7abe00640 1 --2- 172.21.15.6:0/2894768023 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fa78c06be30 0x7fa78c06e2f0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fa7ac1efbd0 tx=0x7fa798009240 comp rx=0 tx=0).stop 2024-04-24T23:27:17.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:17.124+0000 7fa7abe00640 1 -- 172.21.15.6:0/2894768023 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa7ac0c6a10 msgr2=0x7fa7ac15c890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:17.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:17.124+0000 7fa7abe00640 1 --2- 172.21.15.6:0/2894768023 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa7ac0c6a10 0x7fa7ac15c890 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fa79c009890 tx=0x7fa79c0046c0 comp rx=0 tx=0).stop 2024-04-24T23:27:17.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:17.124+0000 7fa7abe00640 1 -- 172.21.15.6:0/2894768023 shutdown_connections 2024-04-24T23:27:17.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:17.124+0000 7fa7abe00640 1 --2- 172.21.15.6:0/2894768023 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fa78c06be30 0x7fa78c06e2f0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:17.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:17.124+0000 7fa7abe00640 1 --2- 172.21.15.6:0/2894768023 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa7ac158cf0 0x7fa7ac1e43b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:17.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:17.124+0000 7fa7abe00640 1 --2- 172.21.15.6:0/2894768023 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa7ac152990 0x7fa7ac1eeaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:17.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:17.124+0000 7fa7abe00640 1 --2- 172.21.15.6:0/2894768023 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa7ac0c6a10 0x7fa7ac15c890 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:17.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:17.124+0000 7fa7abe00640 1 -- 172.21.15.6:0/2894768023 >> 172.21.15.6:0/2894768023 conn(0x7fa7ac06e590 msgr2=0x7fa7ac0ca700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:27:17.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:17.124+0000 7fa7abe00640 1 -- 172.21.15.6:0/2894768023 shutdown_connections 2024-04-24T23:27:17.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:17.124+0000 7fa7abe00640 1 -- 172.21.15.6:0/2894768023 wait complete. 2024-04-24T23:27:17.152 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 52, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-24T23:15:36.400716+0000", "damaged": [], "data_pools": [ 3, 4 ], "enabled": true, "epoch": 52, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_5599": { "addr": "172.21.15.190:6839/2643233757", "addrs": { "addrvec": [ { "addr": "172.21.15.190:6836", "nonce": 2643233757, "type": "v2" }, { "addr": "172.21.15.190:6839", "nonce": 2643233757, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5599, "incarnation": 43, "join_fscid": -1, "name": "d", "rank": 1, "state": "up:active", "state_seq": 25 }, "gid_5725": { "addr": "172.21.15.190:6837/1151527362", "addrs": { "addrvec": [ { "addr": "172.21.15.190:6834", "nonce": 1151527362, "type": "v2" }, { "addr": "172.21.15.190:6837", "nonce": 1151527362, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5725, "incarnation": 49, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:rejoin", "state_seq": 34 } }, "last_failure": 0, "last_failure_osd_epoch": 30, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-04-24T23:27:12.194226+0000", "qdb_cluster": [ 5599 ], "qdb_leader": 5599, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 2, 3, 4 ], "tableserver": 0, "up": { "mds_0": 5725, "mds_1": 5599 } } } ], "standbys": [ { "addr": "172.21.15.6:6834/1190759626", "addrs": { "addrvec": [ { "addr": "172.21.15.6:6832", "nonce": 1190759626, "type": "v2" }, { "addr": "172.21.15.6:6834", "nonce": 1190759626, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 30, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5081, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.190:6838/2416336040", "addrs": { "addrvec": [ { "addr": "172.21.15.190:6835", "nonce": 2416336040, "type": "v2" }, { "addr": "172.21.15.190:6838", "nonce": 2416336040, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 42, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5249, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.6:6837/3928731059", "addrs": { "addrvec": [ { "addr": "172.21.15.6:6836", "nonce": 3928731059, "type": "v2" }, { "addr": "172.21.15.6:6837", "nonce": 3928731059, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 35, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5503, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-04-24T23:27:19.153 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:27:19.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:19.339+0000 7f1387200640 1 Processor -- start 2024-04-24T23:27:19.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:19.339+0000 7f1387200640 1 -- start start 2024-04-24T23:27:19.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:19.340+0000 7f1387200640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f138806ce30 0x7f13880cc600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:19.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:19.340+0000 7f1387200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f138806d390 con 0x7f1388152bd0 2024-04-24T23:27:19.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:19.340+0000 7f1387200640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f138806dbc0 con 0x7f13880c6be0 2024-04-24T23:27:19.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:19.340+0000 7f1387200640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f138806d970 con 0x7f138806ce30 2024-04-24T23:27:19.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:19.341+0000 7f1385e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f1388152bd0 0x7f1388158e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:35808/0 (socket says 172.21.15.6:35808) 2024-04-24T23:27:19.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:19.341+0000 7f1385e00640 1 -- 172.21.15.6:0/3975069638 learned_addr learned my addr 172.21.15.6:0/3975069638 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:27:19.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:19.341+0000 7f137fe00640 1 --2- 172.21.15.6:0/3975069638 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f138806ce30 0x7f13880cc600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:19.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:19.341+0000 7f1385400640 1 -- 172.21.15.6:0/3975069638 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4282097290 0 0) 0x7f138806dbc0 con 0x7f13880c6be0 2024-04-24T23:27:19.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:19.341+0000 7f1385400640 1 -- 172.21.15.6:0/3975069638 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f136c003520 con 0x7f13880c6be0 2024-04-24T23:27:19.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:19.341+0000 7f1385400640 1 -- 172.21.15.6:0/3975069638 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4273491851 0 0) 0x7f138806d390 con 0x7f1388152bd0 2024-04-24T23:27:19.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:19.341+0000 7f1385400640 1 -- 172.21.15.6:0/3975069638 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f138806dbc0 con 0x7f1388152bd0 2024-04-24T23:27:19.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:19.341+0000 7f137fe00640 1 -- 172.21.15.6:0/3975069638 >> v1:172.21.15.190:6789/0 conn(0x7f13880c6be0 legacy=0x7f1388163090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:19.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:19.341+0000 7f137fe00640 1 -- 172.21.15.6:0/3975069638 >> v1:172.21.15.6:6789/0 conn(0x7f1388152bd0 legacy=0x7f1388158e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:19.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:19.341+0000 7f137fe00640 1 -- 172.21.15.6:0/3975069638 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f138806dea0 con 0x7f138806ce30 2024-04-24T23:27:19.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:19.342+0000 7f137fe00640 1 --2- 172.21.15.6:0/3975069638 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f138806ce30 0x7f13880cc600 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f13700097d0 tx=0x7f137002a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d82ae2b76e6cddc4 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:27:19.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:19.343+0000 7f1385400640 1 -- 172.21.15.6:0/3975069638 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1370037070 con 0x7f138806ce30 2024-04-24T23:27:19.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:19.343+0000 7f1385400640 1 -- 172.21.15.6:0/3975069638 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1370032440 con 0x7f138806ce30 2024-04-24T23:27:19.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:19.343+0000 7f1385400640 1 -- 172.21.15.6:0/3975069638 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1370032810 con 0x7f138806ce30 2024-04-24T23:27:19.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:19.343+0000 7f1387200640 1 -- 172.21.15.6:0/3975069638 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f138806ce30 msgr2=0x7f13880cc600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:19.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:19.343+0000 7f1387200640 1 --2- 172.21.15.6:0/3975069638 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f138806ce30 0x7f13880cc600 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f13700097d0 tx=0x7f137002a180 comp rx=0 tx=0).stop 2024-04-24T23:27:19.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:19.343+0000 7f1387200640 1 -- 172.21.15.6:0/3975069638 shutdown_connections 2024-04-24T23:27:19.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:19.343+0000 7f1387200640 1 --2- 172.21.15.6:0/3975069638 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f138806ce30 0x7f13880cc600 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:19.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:19.343+0000 7f1387200640 1 -- 172.21.15.6:0/3975069638 >> 172.21.15.6:0/3975069638 conn(0x7f138806ed80 msgr2=0x7f138806f190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:27:19.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:19.343+0000 7f1387200640 1 -- 172.21.15.6:0/3975069638 shutdown_connections 2024-04-24T23:27:19.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:19.343+0000 7f1387200640 1 -- 172.21.15.6:0/3975069638 wait complete. 2024-04-24T23:27:19.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:19.345+0000 7f1387200640 1 Processor -- start 2024-04-24T23:27:19.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:19.345+0000 7f1387200640 1 -- start start 2024-04-24T23:27:19.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:19.346+0000 7f1387200640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f138806ce30 0x7f1388151dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:19.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:19.346+0000 7f1387200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f13880c6be0 0x7f1388156350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:19.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:19.346+0000 7f1387200640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1388152bd0 0x7f13881f0370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:19.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:19.346+0000 7f1387200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f1388072650 con 0x7f13880c6be0 2024-04-24T23:27:19.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:19.346+0000 7f1387200640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f1388071890 con 0x7f1388152bd0 2024-04-24T23:27:19.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:19.346+0000 7f1387200640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f13880727d0 con 0x7f138806ce30 2024-04-24T23:27:19.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:19.346+0000 7f137fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f13880c6be0 0x7f1388156350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:19.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:19.346+0000 7f137fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f13880c6be0 0x7f1388156350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:59188/0 (socket says 172.21.15.6:59188) 2024-04-24T23:27:19.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:19.346+0000 7f137fe00640 1 -- 172.21.15.6:0/2220446055 learned_addr learned my addr 172.21.15.6:0/2220446055 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:27:19.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:19.346+0000 7f1386800640 1 --2- 172.21.15.6:0/2220446055 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1388152bd0 0x7f13881f0370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:19.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:19.346+0000 7f1385e00640 1 --2- 172.21.15.6:0/2220446055 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f138806ce30 0x7f1388151dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:19.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:19.347+0000 7f137fe00640 1 -- 172.21.15.6:0/2220446055 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f138806ce30 msgr2=0x7f1388151dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:19.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:19.347+0000 7f137fe00640 1 --2- 172.21.15.6:0/2220446055 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f138806ce30 0x7f1388151dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:19.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:19.347+0000 7f137fe00640 1 -- 172.21.15.6:0/2220446055 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1388152bd0 msgr2=0x7f13881f0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:19.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:19.347+0000 7f137fe00640 1 --2- 172.21.15.6:0/2220446055 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1388152bd0 0x7f13881f0370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:19.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:19.347+0000 7f137fe00640 1 -- 172.21.15.6:0/2220446055 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13881f4540 con 0x7f13880c6be0 2024-04-24T23:27:19.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:19.347+0000 7f1385e00640 1 --2- 172.21.15.6:0/2220446055 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f138806ce30 0x7f1388151dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:27:19.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:19.347+0000 7f137fe00640 1 --2- 172.21.15.6:0/2220446055 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f13880c6be0 0x7f1388156350 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f13700098f0 tx=0x7f13700046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:27:19.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:19.347+0000 7f137e000640 1 -- 172.21.15.6:0/2220446055 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1370041020 con 0x7f13880c6be0 2024-04-24T23:27:19.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:19.347+0000 7f137e000640 1 -- 172.21.15.6:0/2220446055 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f137003d6e0 con 0x7f13880c6be0 2024-04-24T23:27:19.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:19.347+0000 7f137e000640 1 -- 172.21.15.6:0/2220446055 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1370037040 con 0x7f13880c6be0 2024-04-24T23:27:19.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:19.348+0000 7f1387200640 1 -- 172.21.15.6:0/2220446055 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13881556c0 con 0x7f13880c6be0 2024-04-24T23:27:19.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:19.348+0000 7f1387200640 1 -- 172.21.15.6:0/2220446055 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1388155c80 con 0x7f13880c6be0 2024-04-24T23:27:19.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:19.349+0000 7f1387200640 1 -- 172.21.15.6:0/2220446055 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1388075b30 con 0x7f13880c6be0 2024-04-24T23:27:19.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:19.350+0000 7f137e000640 1 -- 172.21.15.6:0/2220446055 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f1370044020 con 0x7f13880c6be0 2024-04-24T23:27:19.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:19.351+0000 7f137e000640 1 --2- 172.21.15.6:0/2220446055 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f136406be30 0x7f136406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:19.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:19.351+0000 7f137e000640 1 -- 172.21.15.6:0/2220446055 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6934+0+0 (secure 0 0 0) 0x7f13700aba60 con 0x7f13880c6be0 2024-04-24T23:27:19.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:19.351+0000 7f1385e00640 1 --2- 172.21.15.6:0/2220446055 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f136406be30 0x7f136406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:19.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:19.352+0000 7f1385e00640 1 --2- 172.21.15.6:0/2220446055 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f136406be30 0x7f136406e2f0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f1378007fd0 tx=0x7f1378009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:27:19.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:19.355+0000 7f137e000640 1 -- 172.21.15.6:0/2220446055 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f137004b030 con 0x7f13880c6be0 2024-04-24T23:27:19.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:19.627+0000 7f1387200640 1 -- 172.21.15.6:0/2220446055 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1388155e50 con 0x7f13880c6be0 2024-04-24T23:27:19.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:19.628+0000 7f137e000640 1 -- 172.21.15.6:0/2220446055 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) ==== 94+0+6039 (secure 0 0 0) 0x7f1370079e20 con 0x7f13880c6be0 2024-04-24T23:27:19.629 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:27:19.629 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":53,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5081,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/1190759626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1190759626},{"type":"v1","addr":"172.21.15.6:6834","nonce":1190759626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":5249,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6838/2416336040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2416336040},{"type":"v1","addr":"172.21.15.190:6838","nonce":2416336040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":42},{"gid":5503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/3928731059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":3928731059},{"type":"v1","addr":"172.21.15.6:6837","nonce":3928731059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":53,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:27:18.122830+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5725,"mds_1":5599},"failed":[],"damaged":[],"stopped":[2,3,4],"info":{"gid_5599":{"gid":5599,"name":"d","rank":1,"incarnation":43,"state":"up:active","state_seq":25,"addr":"172.21.15.190:6839/2643233757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":2643233757},{"type":"v1","addr":"172.21.15.190:6839","nonce":2643233757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5725":{"gid":5725,"name":"f","rank":0,"incarnation":49,"state":"up:active","state_seq":36,"addr":"172.21.15.190:6837/1151527362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":1151527362},{"type":"v1","addr":"172.21.15.190:6837","nonce":1151527362}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5725,"qdb_cluster":[5599,5725]},"id":1}]} 2024-04-24T23:27:19.629 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 53 2024-04-24T23:27:19.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:19.631+0000 7f1387200640 1 -- 172.21.15.6:0/2220446055 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f136406be30 msgr2=0x7f136406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:19.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:19.631+0000 7f1387200640 1 --2- 172.21.15.6:0/2220446055 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f136406be30 0x7f136406e2f0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f1378007fd0 tx=0x7f1378009040 comp rx=0 tx=0).stop 2024-04-24T23:27:19.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:19.631+0000 7f1387200640 1 -- 172.21.15.6:0/2220446055 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f13880c6be0 msgr2=0x7f1388156350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:19.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:19.631+0000 7f1387200640 1 --2- 172.21.15.6:0/2220446055 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f13880c6be0 0x7f1388156350 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f13700098f0 tx=0x7f13700046c0 comp rx=0 tx=0).stop 2024-04-24T23:27:19.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:19.631+0000 7f1387200640 1 -- 172.21.15.6:0/2220446055 shutdown_connections 2024-04-24T23:27:19.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:19.631+0000 7f1387200640 1 --2- 172.21.15.6:0/2220446055 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f136406be30 0x7f136406e2f0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:19.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:19.631+0000 7f1387200640 1 --2- 172.21.15.6:0/2220446055 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1388152bd0 0x7f13881f0370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:19.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:19.631+0000 7f1387200640 1 --2- 172.21.15.6:0/2220446055 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f13880c6be0 0x7f1388156350 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:19.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:19.631+0000 7f1387200640 1 --2- 172.21.15.6:0/2220446055 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f138806ce30 0x7f1388151dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:19.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:19.631+0000 7f1387200640 1 -- 172.21.15.6:0/2220446055 >> 172.21.15.6:0/2220446055 conn(0x7f138806ed80 msgr2=0x7f13880cc5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:27:19.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:19.631+0000 7f1387200640 1 -- 172.21.15.6:0/2220446055 shutdown_connections 2024-04-24T23:27:19.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:19.631+0000 7f1387200640 1 -- 172.21.15.6:0/2220446055 wait complete. 2024-04-24T23:27:19.652 INFO:tasks.mds_thrash.fs.[cephfs]:mds.f has gained rank=0, replacing gid=4306 2024-04-24T23:27:19.652 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 9 secs before reviving mds.c 2024-04-24T23:27:23.276 INFO:tasks.workunit.client.0.smithi006.stdout:Syncing()...113 sec 2024-04-24T23:27:23.276 INFO:tasks.workunit.client.0.smithi006.stdout:FFSB benchmark finished at: Wed Apr 24 23:27:23 2024 2024-04-24T23:27:23.276 INFO:tasks.workunit.client.0.smithi006.stdout: 2024-04-24T23:27:23.276 INFO:tasks.workunit.client.0.smithi006.stdout:Results: 2024-04-24T23:27:23.276 INFO:tasks.workunit.client.0.smithi006.stdout:Benchmark took 434.25 sec 2024-04-24T23:27:23.290 INFO:tasks.workunit.client.0.smithi006.stdout: 2024-04-24T23:27:23.290 INFO:tasks.workunit.client.0.smithi006.stdout:Total Results 2024-04-24T23:27:23.290 INFO:tasks.workunit.client.0.smithi006.stdout:=============== 2024-04-24T23:27:23.291 INFO:tasks.workunit.client.0.smithi006.stdout: Op Name Transactions Trans/sec % Trans % Op Weight Throughput 2024-04-24T23:27:23.291 INFO:tasks.workunit.client.0.smithi006.stdout: ======= ============ ========= ======= =========== ========== 2024-04-24T23:27:23.291 INFO:tasks.workunit.client.0.smithi006.stdout: write : 5864960 13505.85 100.000% 100.000% 52.8MB/sec 2024-04-24T23:27:23.291 INFO:tasks.workunit.client.0.smithi006.stdout:- 2024-04-24T23:27:23.291 INFO:tasks.workunit.client.0.smithi006.stdout:13505.85 Transactions per Second 2024-04-24T23:27:23.291 INFO:tasks.workunit.client.0.smithi006.stdout: 2024-04-24T23:27:23.291 INFO:tasks.workunit.client.0.smithi006.stdout:Throughput Results 2024-04-24T23:27:23.291 INFO:tasks.workunit.client.0.smithi006.stdout:=================== 2024-04-24T23:27:23.291 INFO:tasks.workunit.client.0.smithi006.stdout:Write Throughput: 52.8MB/sec 2024-04-24T23:27:23.385 INFO:tasks.workunit.client.0.smithi006.stdout: 2024-04-24T23:27:23.385 INFO:tasks.workunit.client.0.smithi006.stdout:System Call Latency statistics in millisecs 2024-04-24T23:27:23.385 INFO:tasks.workunit.client.0.smithi006.stdout:===== 2024-04-24T23:27:23.385 INFO:tasks.workunit.client.0.smithi006.stdout: Min Avg Max Total Calls 2024-04-24T23:27:23.386 INFO:tasks.workunit.client.0.smithi006.stdout: ======== ======== ======== ============ 2024-04-24T23:27:23.386 INFO:tasks.workunit.client.0.smithi006.stdout:[ open] 0.019000 0.410984 335.186005 4582 2024-04-24T23:27:23.386 INFO:tasks.workunit.client.0.smithi006.stdout: msec_range[0] 5000.000000 - 10000.000000 : 0 2024-04-24T23:27:23.386 INFO:tasks.workunit.client.0.smithi006.stdout: msec_range[1] 2000.000000 - 5000.000000 : 0 2024-04-24T23:27:23.386 INFO:tasks.workunit.client.0.smithi006.stdout: msec_range[2] 1000.000000 - 2000.000000 : 0 2024-04-24T23:27:23.386 INFO:tasks.workunit.client.0.smithi006.stdout: msec_range[3] 500.000000 - 1000.000000 : 0 2024-04-24T23:27:23.386 INFO:tasks.workunit.client.0.smithi006.stdout: msec_range[4] 200.000000 - 500.000000 : 1 2024-04-24T23:27:23.386 INFO:tasks.workunit.client.0.smithi006.stdout: msec_range[5] 100.000000 - 200.000000 : 0 2024-04-24T23:27:23.386 INFO:tasks.workunit.client.0.smithi006.stdout: msec_range[6] 50.000000 - 100.000000 : 2 2024-04-24T23:27:23.386 INFO:tasks.workunit.client.0.smithi006.stdout: msec_range[7] 20.000000 - 50.000000 : 11 2024-04-24T23:27:23.386 INFO:tasks.workunit.client.0.smithi006.stdout: msec_range[8] 10.000000 - 20.000000 : 30 2024-04-24T23:27:23.386 INFO:tasks.workunit.client.0.smithi006.stdout: msec_range[9] 5.000000 - 10.000000 : 27 2024-04-24T23:27:23.386 INFO:tasks.workunit.client.0.smithi006.stdout: msec_range[10] 2.000000 - 5.000000 : 54 2024-04-24T23:27:23.387 INFO:tasks.workunit.client.0.smithi006.stdout: msec_range[11] 1.000000 - 2.000000 : 38 2024-04-24T23:27:23.387 INFO:tasks.workunit.client.0.smithi006.stdout: msec_range[12] 0.500000 - 1.000000 : 30 2024-04-24T23:27:23.387 INFO:tasks.workunit.client.0.smithi006.stdout: msec_range[13] 0.200000 - 0.500000 : 55 2024-04-24T23:27:23.387 INFO:tasks.workunit.client.0.smithi006.stdout: msec_range[14] 0.100000 - 0.200000 : 176 2024-04-24T23:27:23.387 INFO:tasks.workunit.client.0.smithi006.stdout: msec_range[15] 0.050000 - 0.100000 : 333 2024-04-24T23:27:23.387 INFO:tasks.workunit.client.0.smithi006.stdout: msec_range[16] 0.020000 - 0.050000 : 3824 2024-04-24T23:27:23.387 INFO:tasks.workunit.client.0.smithi006.stdout: msec_range[17] 0.010000 - 0.020000 : 1 2024-04-24T23:27:23.387 INFO:tasks.workunit.client.0.smithi006.stdout: msec_range[18] 0.000000 - 0.010000 : 0 2024-04-24T23:27:23.387 INFO:tasks.workunit.client.0.smithi006.stdout: 2024-04-24T23:27:23.387 INFO:tasks.workunit.client.0.smithi006.stdout:[ write] 0.003000 1.672590 32611.199219 5864960 2024-04-24T23:27:23.387 INFO:tasks.workunit.client.0.smithi006.stdout: msec_range[0] 5000.000000 - 10000.000000 : 258 2024-04-24T23:27:23.387 INFO:tasks.workunit.client.0.smithi006.stdout: msec_range[1] 2000.000000 - 5000.000000 : 281 2024-04-24T23:27:23.387 INFO:tasks.workunit.client.0.smithi006.stdout: msec_range[2] 1000.000000 - 2000.000000 : 160 2024-04-24T23:27:23.387 INFO:tasks.workunit.client.0.smithi006.stdout: msec_range[3] 500.000000 - 1000.000000 : 205 2024-04-24T23:27:23.388 INFO:tasks.workunit.client.0.smithi006.stdout: msec_range[4] 200.000000 - 500.000000 : 4529 2024-04-24T23:27:23.388 INFO:tasks.workunit.client.0.smithi006.stdout: msec_range[5] 100.000000 - 200.000000 : 12563 2024-04-24T23:27:23.388 INFO:tasks.workunit.client.0.smithi006.stdout: msec_range[6] 50.000000 - 100.000000 : 3089 2024-04-24T23:27:23.388 INFO:tasks.workunit.client.0.smithi006.stdout: msec_range[7] 20.000000 - 50.000000 : 2692 2024-04-24T23:27:23.388 INFO:tasks.workunit.client.0.smithi006.stdout: msec_range[8] 10.000000 - 20.000000 : 1933 2024-04-24T23:27:23.388 INFO:tasks.workunit.client.0.smithi006.stdout: msec_range[9] 5.000000 - 10.000000 : 2136 2024-04-24T23:27:23.388 INFO:tasks.workunit.client.0.smithi006.stdout: msec_range[10] 2.000000 - 5.000000 : 4209 2024-04-24T23:27:23.388 INFO:tasks.workunit.client.0.smithi006.stdout: msec_range[11] 1.000000 - 2.000000 : 2806 2024-04-24T23:27:23.388 INFO:tasks.workunit.client.0.smithi006.stdout: msec_range[12] 0.500000 - 1.000000 : 2744 2024-04-24T23:27:23.388 INFO:tasks.workunit.client.0.smithi006.stdout: msec_range[13] 0.200000 - 0.500000 : 4676 2024-04-24T23:27:23.388 INFO:tasks.workunit.client.0.smithi006.stdout: msec_range[14] 0.100000 - 0.200000 : 7851 2024-04-24T23:27:23.388 INFO:tasks.workunit.client.0.smithi006.stdout: msec_range[15] 0.050000 - 0.100000 : 18471 2024-04-24T23:27:23.388 INFO:tasks.workunit.client.0.smithi006.stdout: msec_range[16] 0.020000 - 0.050000 : 173139 2024-04-24T23:27:23.388 INFO:tasks.workunit.client.0.smithi006.stdout: msec_range[17] 0.010000 - 0.020000 : 2779711 2024-04-24T23:27:23.389 INFO:tasks.workunit.client.0.smithi006.stdout: msec_range[18] 0.000000 - 0.010000 : 2843282 2024-04-24T23:27:23.389 INFO:tasks.workunit.client.0.smithi006.stdout: 2024-04-24T23:27:23.389 INFO:tasks.workunit.client.0.smithi006.stdout:[ lseek] 0.000000 0.002512 134.201004 5864960 2024-04-24T23:27:23.389 INFO:tasks.workunit.client.0.smithi006.stdout: msec_range[0] 5000.000000 - 10000.000000 : 0 2024-04-24T23:27:23.389 INFO:tasks.workunit.client.0.smithi006.stdout: msec_range[1] 2000.000000 - 5000.000000 : 0 2024-04-24T23:27:23.389 INFO:tasks.workunit.client.0.smithi006.stdout: msec_range[2] 1000.000000 - 2000.000000 : 0 2024-04-24T23:27:23.389 INFO:tasks.workunit.client.0.smithi006.stdout: msec_range[3] 500.000000 - 1000.000000 : 0 2024-04-24T23:27:23.389 INFO:tasks.workunit.client.0.smithi006.stdout: msec_range[4] 200.000000 - 500.000000 : 0 2024-04-24T23:27:23.389 INFO:tasks.workunit.client.0.smithi006.stdout: msec_range[5] 100.000000 - 200.000000 : 3 2024-04-24T23:27:23.390 INFO:tasks.workunit.client.0.smithi006.stdout: msec_range[6] 50.000000 - 100.000000 : 16 2024-04-24T23:27:23.390 INFO:tasks.workunit.client.0.smithi006.stdout: msec_range[7] 20.000000 - 50.000000 : 62 2024-04-24T23:27:23.390 INFO:tasks.workunit.client.0.smithi006.stdout: msec_range[8] 10.000000 - 20.000000 : 128 2024-04-24T23:27:23.390 INFO:tasks.workunit.client.0.smithi006.stdout: msec_range[9] 5.000000 - 10.000000 : 159 2024-04-24T23:27:23.390 INFO:tasks.workunit.client.0.smithi006.stdout: msec_range[10] 2.000000 - 5.000000 : 359 2024-04-24T23:27:23.390 INFO:tasks.workunit.client.0.smithi006.stdout: msec_range[11] 1.000000 - 2.000000 : 214 2024-04-24T23:27:23.390 INFO:tasks.workunit.client.0.smithi006.stdout: msec_range[12] 0.500000 - 1.000000 : 259 2024-04-24T23:27:23.390 INFO:tasks.workunit.client.0.smithi006.stdout: msec_range[13] 0.200000 - 0.500000 : 365 2024-04-24T23:27:23.390 INFO:tasks.workunit.client.0.smithi006.stdout: msec_range[14] 0.100000 - 0.200000 : 500 2024-04-24T23:27:23.390 INFO:tasks.workunit.client.0.smithi006.stdout: msec_range[15] 0.050000 - 0.100000 : 1032 2024-04-24T23:27:23.390 INFO:tasks.workunit.client.0.smithi006.stdout: msec_range[16] 0.020000 - 0.050000 : 3944 2024-04-24T23:27:23.390 INFO:tasks.workunit.client.0.smithi006.stdout: msec_range[17] 0.010000 - 0.020000 : 3475 2024-04-24T23:27:23.391 INFO:tasks.workunit.client.0.smithi006.stdout: msec_range[18] 0.000000 - 0.010000 : 5854444 2024-04-24T23:27:23.391 INFO:tasks.workunit.client.0.smithi006.stdout: 2024-04-24T23:27:23.391 INFO:tasks.workunit.client.0.smithi006.stdout:[ close] 0.004000 0.017994 15.702000 4582 2024-04-24T23:27:23.391 INFO:tasks.workunit.client.0.smithi006.stdout: msec_range[0] 5000.000000 - 10000.000000 : 0 2024-04-24T23:27:23.391 INFO:tasks.workunit.client.0.smithi006.stdout: msec_range[1] 2000.000000 - 5000.000000 : 0 2024-04-24T23:27:23.391 INFO:tasks.workunit.client.0.smithi006.stdout: msec_range[2] 1000.000000 - 2000.000000 : 0 2024-04-24T23:27:23.391 INFO:tasks.workunit.client.0.smithi006.stdout: msec_range[3] 500.000000 - 1000.000000 : 0 2024-04-24T23:27:23.391 INFO:tasks.workunit.client.0.smithi006.stdout: msec_range[4] 200.000000 - 500.000000 : 0 2024-04-24T23:27:23.391 INFO:tasks.workunit.client.0.smithi006.stdout: msec_range[5] 100.000000 - 200.000000 : 0 2024-04-24T23:27:23.391 INFO:tasks.workunit.client.0.smithi006.stdout: msec_range[6] 50.000000 - 100.000000 : 0 2024-04-24T23:27:23.391 INFO:tasks.workunit.client.0.smithi006.stdout: msec_range[7] 20.000000 - 50.000000 : 0 2024-04-24T23:27:23.391 INFO:tasks.workunit.client.0.smithi006.stdout: msec_range[8] 10.000000 - 20.000000 : 1 2024-04-24T23:27:23.392 INFO:tasks.workunit.client.0.smithi006.stdout: msec_range[9] 5.000000 - 10.000000 : 1 2024-04-24T23:27:23.392 INFO:tasks.workunit.client.0.smithi006.stdout: msec_range[10] 2.000000 - 5.000000 : 5 2024-04-24T23:27:23.392 INFO:tasks.workunit.client.0.smithi006.stdout: msec_range[11] 1.000000 - 2.000000 : 2 2024-04-24T23:27:23.392 INFO:tasks.workunit.client.0.smithi006.stdout: msec_range[12] 0.500000 - 1.000000 : 0 2024-04-24T23:27:23.392 INFO:tasks.workunit.client.0.smithi006.stdout: msec_range[13] 0.200000 - 0.500000 : 1 2024-04-24T23:27:23.392 INFO:tasks.workunit.client.0.smithi006.stdout: msec_range[14] 0.100000 - 0.200000 : 1 2024-04-24T23:27:23.392 INFO:tasks.workunit.client.0.smithi006.stdout: msec_range[15] 0.050000 - 0.100000 : 8 2024-04-24T23:27:23.392 INFO:tasks.workunit.client.0.smithi006.stdout: msec_range[16] 0.020000 - 0.050000 : 39 2024-04-24T23:27:23.392 INFO:tasks.workunit.client.0.smithi006.stdout: msec_range[17] 0.010000 - 0.020000 : 2281 2024-04-24T23:27:23.392 INFO:tasks.workunit.client.0.smithi006.stdout: msec_range[18] 0.000000 - 0.010000 : 2243 2024-04-24T23:27:23.392 INFO:tasks.workunit.client.0.smithi006.stdout: 2024-04-24T23:27:23.392 INFO:tasks.workunit.client.0.smithi006.stdout: 2024-04-24T23:27:23.392 INFO:tasks.workunit.client.0.smithi006.stdout: 2024-04-24T23:27:23.393 INFO:tasks.workunit.client.0.smithi006.stdout: 2024-04-24T23:27:23.393 INFO:tasks.workunit.client.0.smithi006.stdout:1.0% User Time 2024-04-24T23:27:23.393 INFO:tasks.workunit.client.0.smithi006.stdout:15.8% System Time 2024-04-24T23:27:23.393 INFO:tasks.workunit.client.0.smithi006.stdout:16.8% CPU Utilization 2024-04-24T23:27:23.418 INFO:tasks.workunit.client.0.smithi006.stderr:+ cd .. 2024-04-24T23:27:23.422 INFO:tasks.workunit.client.0.smithi006.stderr:+ rm -r tmp ffsb 2024-04-24T23:27:25.256 INFO:tasks.ceph.mds.f.smithi190.stderr:2024-04-24T23:27:25.255+0000 7f221cda0640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-24T23:27:25.256 INFO:tasks.ceph.mds.f.smithi190.stderr:2024-04-24T23:27:25.255+0000 7f221cda0640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-04-24T23:27:28.654 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.c 2024-04-24T23:27:28.654 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-24T23:27:28.654 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-24T23:27:28.666 INFO:tasks.ceph.mds.c:Started 2024-04-24T23:27:28.666 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:27:28.750 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-24T23:27:28.750 DEBUG:teuthology.orchestra.run.smithi006:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-24T23:27:28.916 INFO:tasks.workunit:Stopping ['suites/ffsb.sh'] on client.0... 2024-04-24T23:27:28.916 DEBUG:teuthology.orchestra.run.smithi006:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-04-24T23:27:28.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:28.984+0000 7f1389a00640 1 Processor -- start 2024-04-24T23:27:28.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:28.984+0000 7f1389a00640 1 -- start start 2024-04-24T23:27:28.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:28.986+0000 7f1389a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f13840ccc50 0x7f1384070e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:28.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:28.986+0000 7f1389a00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1384162ac0 0x7f1384162f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:28.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:28.987+0000 7f1389a00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13840715e0 con 0x7f13840cc870 2024-04-24T23:27:28.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:28.987+0000 7f1389a00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f13840713b0 con 0x7f13840ccc50 2024-04-24T23:27:28.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:28.987+0000 7f1389a00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f1384162350 con 0x7f1384162ac0 2024-04-24T23:27:28.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:28.988+0000 7f1389000640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1384162ac0 0x7f1384162f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:28.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:28.988+0000 7f1389000640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1384162ac0 0x7f1384162f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.6:48524/0 (socket says 172.21.15.6:48524) 2024-04-24T23:27:28.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:28.988+0000 7f1389000640 1 -- 172.21.15.6:0/4112423368 learned_addr learned my addr 172.21.15.6:0/4112423368 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:27:28.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:28.989+0000 7f1389000640 1 -- 172.21.15.6:0/4112423368 >> v2:172.21.15.6:3300/0 conn(0x7f13840ccc50 msgr2=0x7f1384070e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:27:28.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:28.989+0000 7f1389000640 1 --2- 172.21.15.6:0/4112423368 >> v2:172.21.15.6:3300/0 conn(0x7f13840ccc50 0x7f1384070e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:28.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:28.989+0000 7f1389000640 1 -- 172.21.15.6:0/4112423368 >> v1:172.21.15.6:6789/0 conn(0x7f13840cc870 legacy=0x7f138415f790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:28.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:28.990+0000 7f1389000640 1 -- 172.21.15.6:0/4112423368 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f138406dcb0 con 0x7f1384162ac0 2024-04-24T23:27:28.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:28.998+0000 7f1389000640 1 --2- 172.21.15.6:0/4112423368 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1384162ac0 0x7f1384162f00 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f137c009e50 tx=0x7f137c02a710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=79d275ad690f91db server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:27:29.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:28.999+0000 7f1382a00640 1 -- 172.21.15.6:0/4112423368 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f137c036070 con 0x7f1384162ac0 2024-04-24T23:27:29.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:29.000+0000 7f1382a00640 1 -- 172.21.15.6:0/4112423368 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f137c031600 con 0x7f1384162ac0 2024-04-24T23:27:29.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:29.000+0000 7f1382a00640 1 -- 172.21.15.6:0/4112423368 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f137c0319d0 con 0x7f1384162ac0 2024-04-24T23:27:29.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:29.002+0000 7f1389a00640 1 -- 172.21.15.6:0/4112423368 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1384162ac0 msgr2=0x7f1384162f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:29.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:29.003+0000 7f1389a00640 1 --2- 172.21.15.6:0/4112423368 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1384162ac0 0x7f1384162f00 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f137c009e50 tx=0x7f137c02a710 comp rx=0 tx=0).stop 2024-04-24T23:27:29.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:29.007+0000 7f1389a00640 1 -- 172.21.15.6:0/4112423368 shutdown_connections 2024-04-24T23:27:29.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:29.007+0000 7f1389a00640 1 --2- 172.21.15.6:0/4112423368 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1384162ac0 0x7f1384162f00 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:29.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:29.007+0000 7f1389a00640 1 --2- 172.21.15.6:0/4112423368 >> v2:172.21.15.6:3300/0 conn(0x7f13840ccc50 0x7f1384070e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:29.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:29.007+0000 7f1389a00640 1 -- 172.21.15.6:0/4112423368 >> 172.21.15.6:0/4112423368 conn(0x7f138406ed80 msgr2=0x7f138406f190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:27:29.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:29.007+0000 7f1389a00640 1 -- 172.21.15.6:0/4112423368 shutdown_connections 2024-04-24T23:27:29.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:29.007+0000 7f1389a00640 1 -- 172.21.15.6:0/4112423368 wait complete. 2024-04-24T23:27:29.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:29.014+0000 7f1389a00640 1 Processor -- start 2024-04-24T23:27:29.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:29.014+0000 7f1389a00640 1 -- start start 2024-04-24T23:27:29.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:29.016+0000 7f1389a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f13840cc870 0x7f1384156c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:29.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:29.017+0000 7f1389a00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f13840ccc50 0x7f1384157150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:29.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:29.017+0000 7f1389a00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1384162ac0 0x7f1384152200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:29.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:29.017+0000 7f1389a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f138406df50 con 0x7f13840cc870 2024-04-24T23:27:29.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:29.017+0000 7f1389a00640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f13840724e0 con 0x7f13840ccc50 2024-04-24T23:27:29.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:29.017+0000 7f1389a00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f138406e230 con 0x7f1384162ac0 2024-04-24T23:27:29.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:29.017+0000 7f1383400640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f13840ccc50 0x7f1384157150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:29.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:29.017+0000 7f1383400640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f13840ccc50 0x7f1384157150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:36006/0 (socket says 172.21.15.6:36006) 2024-04-24T23:27:29.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:29.017+0000 7f1383400640 1 -- 172.21.15.6:0/2008286172 learned_addr learned my addr 172.21.15.6:0/2008286172 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:27:29.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:29.017+0000 7f1389000640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1384162ac0 0x7f1384152200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:29.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:29.017+0000 7f1383e00640 1 --2- 172.21.15.6:0/2008286172 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f13840cc870 0x7f1384156c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:29.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:29.018+0000 7f1383e00640 1 -- 172.21.15.6:0/2008286172 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1384162ac0 msgr2=0x7f1384152200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:29.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:29.018+0000 7f1383e00640 1 --2- 172.21.15.6:0/2008286172 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1384162ac0 0x7f1384152200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:29.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:29.018+0000 7f1383e00640 1 -- 172.21.15.6:0/2008286172 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f13840ccc50 msgr2=0x7f1384157150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:29.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:29.018+0000 7f1383e00640 1 --2- 172.21.15.6:0/2008286172 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f13840ccc50 0x7f1384157150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:29.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:29.018+0000 7f1383e00640 1 -- 172.21.15.6:0/2008286172 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1384156430 con 0x7f13840cc870 2024-04-24T23:27:29.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:29.018+0000 7f1389000640 1 --2- 172.21.15.6:0/2008286172 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1384162ac0 0x7f1384152200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:27:29.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:29.019+0000 7f1383e00640 1 --2- 172.21.15.6:0/2008286172 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f13840cc870 0x7f1384156c10 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f137400c780 tx=0x7f137400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:27:29.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:29.020+0000 7f137be00640 1 -- 172.21.15.6:0/2008286172 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13740050d0 con 0x7f13840cc870 2024-04-24T23:27:29.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:29.020+0000 7f137be00640 1 -- 172.21.15.6:0/2008286172 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1374005270 con 0x7f13840cc870 2024-04-24T23:27:29.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:29.020+0000 7f137be00640 1 -- 172.21.15.6:0/2008286172 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f137400a640 con 0x7f13840cc870 2024-04-24T23:27:29.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:29.021+0000 7f1389a00640 1 -- 172.21.15.6:0/2008286172 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f138415db70 con 0x7f13840cc870 2024-04-24T23:27:29.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:29.021+0000 7f1389a00640 1 -- 172.21.15.6:0/2008286172 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13841566f0 con 0x7f13840cc870 2024-04-24T23:27:29.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:29.022+0000 7f1389a00640 1 -- 172.21.15.6:0/2008286172 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1384075b30 con 0x7f13840cc870 2024-04-24T23:27:29.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:29.025+0000 7f137be00640 1 -- 172.21.15.6:0/2008286172 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f137400a800 con 0x7f13840cc870 2024-04-24T23:27:29.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:29.031+0000 7f137be00640 1 --2- 172.21.15.6:0/2008286172 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f136406bf00 0x7f136406e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:29.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:29.031+0000 7f137be00640 1 -- 172.21.15.6:0/2008286172 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6934+0+0 (secure 0 0 0) 0x7f137408ac90 con 0x7f13840cc870 2024-04-24T23:27:29.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:29.033+0000 7f1383400640 1 --2- 172.21.15.6:0/2008286172 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f136406bf00 0x7f136406e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:29.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:29.039+0000 7f1383400640 1 --2- 172.21.15.6:0/2008286172 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f136406bf00 0x7f136406e3c0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f13841eaa40 tx=0x7f1370009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:27:29.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:29.047+0000 7f137be00640 1 -- 172.21.15.6:0/2008286172 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f1374058b90 con 0x7f13840cc870 2024-04-24T23:27:29.207 INFO:tasks.ceph.mds.c.smithi006.stdout:starting mds.c at 2024-04-24T23:27:29.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:29.339+0000 7f1389a00640 1 -- 172.21.15.6:0/2008286172 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1384072820 con 0x7f13840cc870 2024-04-24T23:27:29.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:29.340+0000 7f137be00640 1 -- 172.21.15.6:0/2008286172 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) ==== 94+0+6039 (secure 0 0 0) 0x7f137405ce90 con 0x7f13840cc870 2024-04-24T23:27:29.341 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:27:29.341 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":53,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5081,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/1190759626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1190759626},{"type":"v1","addr":"172.21.15.6:6834","nonce":1190759626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":5249,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6838/2416336040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2416336040},{"type":"v1","addr":"172.21.15.190:6838","nonce":2416336040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":42},{"gid":5503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/3928731059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":3928731059},{"type":"v1","addr":"172.21.15.6:6837","nonce":3928731059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":53,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:27:18.122830+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5725,"mds_1":5599},"failed":[],"damaged":[],"stopped":[2,3,4],"info":{"gid_5599":{"gid":5599,"name":"d","rank":1,"incarnation":43,"state":"up:active","state_seq":25,"addr":"172.21.15.190:6839/2643233757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":2643233757},{"type":"v1","addr":"172.21.15.190:6839","nonce":2643233757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5725":{"gid":5725,"name":"f","rank":0,"incarnation":49,"state":"up:active","state_seq":36,"addr":"172.21.15.190:6837/1151527362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":1151527362},{"type":"v1","addr":"172.21.15.190:6837","nonce":1151527362}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5725,"qdb_cluster":[5599,5725]},"id":1}]} 2024-04-24T23:27:29.341 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 53 2024-04-24T23:27:29.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:29.342+0000 7f1389a00640 1 -- 172.21.15.6:0/2008286172 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f136406bf00 msgr2=0x7f136406e3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:29.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:29.342+0000 7f1389a00640 1 --2- 172.21.15.6:0/2008286172 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f136406bf00 0x7f136406e3c0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f13841eaa40 tx=0x7f1370009190 comp rx=0 tx=0).stop 2024-04-24T23:27:29.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:29.342+0000 7f1389a00640 1 -- 172.21.15.6:0/2008286172 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f13840cc870 msgr2=0x7f1384156c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:29.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:29.342+0000 7f1389a00640 1 --2- 172.21.15.6:0/2008286172 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f13840cc870 0x7f1384156c10 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f137400c780 tx=0x7f137400cc40 comp rx=0 tx=0).stop 2024-04-24T23:27:29.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:29.342+0000 7f1389a00640 1 -- 172.21.15.6:0/2008286172 shutdown_connections 2024-04-24T23:27:29.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:29.342+0000 7f1389a00640 1 --2- 172.21.15.6:0/2008286172 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f136406bf00 0x7f136406e3c0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:29.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:29.342+0000 7f1389a00640 1 --2- 172.21.15.6:0/2008286172 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1384162ac0 0x7f1384152200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:29.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:29.342+0000 7f1389a00640 1 --2- 172.21.15.6:0/2008286172 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f13840ccc50 0x7f1384157150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:29.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:29.342+0000 7f1389a00640 1 --2- 172.21.15.6:0/2008286172 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f13840cc870 0x7f1384156c10 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:29.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:29.342+0000 7f1389a00640 1 -- 172.21.15.6:0/2008286172 >> 172.21.15.6:0/2008286172 conn(0x7f138406ed80 msgr2=0x7f13840c6e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:27:29.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:29.343+0000 7f1389a00640 1 -- 172.21.15.6:0/2008286172 shutdown_connections 2024-04-24T23:27:29.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:29.343+0000 7f1389a00640 1 -- 172.21.15.6:0/2008286172 wait complete. 2024-04-24T23:27:29.361 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is in active, standby or standby-replay 2024-04-24T23:27:31.362 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:27:31.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:31.543+0000 7fcb71a00640 1 Processor -- start 2024-04-24T23:27:31.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:31.544+0000 7fcb71a00640 1 -- start start 2024-04-24T23:27:31.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:31.546+0000 7fcb71a00640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fcb6c0c7080 0x7fcb6c06f020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:31.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:31.546+0000 7fcb71a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fcb6c071000 0x7fcb6c0c80e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:31.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:31.546+0000 7fcb71a00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcb6c072530 con 0x7fcb6c154b70 2024-04-24T23:27:31.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:31.546+0000 7fcb71a00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7fcb6c072a70 con 0x7fcb6c071000 2024-04-24T23:27:31.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:31.546+0000 7fcb71a00640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7fcb6c0727d0 con 0x7fcb6c0c7080 2024-04-24T23:27:31.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:31.546+0000 7fcb6a000640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fcb6c0c7080 0x7fcb6c06f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:31.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:31.546+0000 7fcb6b400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fcb6c071000 0x7fcb6c0c80e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:31.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:31.546+0000 7fcb6a000640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fcb6c0c7080 0x7fcb6c06f020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:36046/0 (socket says 172.21.15.6:36046) 2024-04-24T23:27:31.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:31.546+0000 7fcb6a000640 1 -- 172.21.15.6:0/2231215376 learned_addr learned my addr 172.21.15.6:0/2231215376 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:27:31.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:31.547+0000 7fcb6a000640 1 -- 172.21.15.6:0/2231215376 >> v2:172.21.15.6:3300/0 conn(0x7fcb6c071000 msgr2=0x7fcb6c0c80e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:27:31.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:31.547+0000 7fcb6a000640 1 --2- 172.21.15.6:0/2231215376 >> v2:172.21.15.6:3300/0 conn(0x7fcb6c071000 0x7fcb6c0c80e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:31.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:31.547+0000 7fcb6a000640 1 -- 172.21.15.6:0/2231215376 >> v1:172.21.15.6:6789/0 conn(0x7fcb6c154b70 legacy=0x7fcb6c15a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:31.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:31.547+0000 7fcb6a000640 1 -- 172.21.15.6:0/2231215376 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb6c06e540 con 0x7fcb6c0c7080 2024-04-24T23:27:31.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:31.548+0000 7fcb6a000640 1 --2- 172.21.15.6:0/2231215376 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcb6c0c7080 0x7fcb6c06f020 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fcb58009870 tx=0x7fcb5802a2a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=56ed0d3fee4ef443 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:27:31.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:31.548+0000 7fcb69600640 1 -- 172.21.15.6:0/2231215376 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb58037070 con 0x7fcb6c0c7080 2024-04-24T23:27:31.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:31.548+0000 7fcb69600640 1 -- 172.21.15.6:0/2231215376 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb58032440 con 0x7fcb6c0c7080 2024-04-24T23:27:31.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:31.548+0000 7fcb69600640 1 -- 172.21.15.6:0/2231215376 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb58032810 con 0x7fcb6c0c7080 2024-04-24T23:27:31.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:31.549+0000 7fcb71a00640 1 -- 172.21.15.6:0/2231215376 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcb6c0c7080 msgr2=0x7fcb6c06f020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:31.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:31.549+0000 7fcb71a00640 1 --2- 172.21.15.6:0/2231215376 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcb6c0c7080 0x7fcb6c06f020 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fcb58009870 tx=0x7fcb5802a2a0 comp rx=0 tx=0).stop 2024-04-24T23:27:31.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:31.549+0000 7fcb71a00640 1 -- 172.21.15.6:0/2231215376 shutdown_connections 2024-04-24T23:27:31.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:31.549+0000 7fcb71a00640 1 --2- 172.21.15.6:0/2231215376 >> v2:172.21.15.6:3300/0 conn(0x7fcb6c071000 0x7fcb6c0c80e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:31.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:31.549+0000 7fcb71a00640 1 --2- 172.21.15.6:0/2231215376 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcb6c0c7080 0x7fcb6c06f020 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:31.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:31.549+0000 7fcb71a00640 1 -- 172.21.15.6:0/2231215376 >> 172.21.15.6:0/2231215376 conn(0x7fcb6c06b0e0 msgr2=0x7fcb6c06b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:27:31.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:31.549+0000 7fcb71a00640 1 -- 172.21.15.6:0/2231215376 shutdown_connections 2024-04-24T23:27:31.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:31.549+0000 7fcb71a00640 1 -- 172.21.15.6:0/2231215376 wait complete. 2024-04-24T23:27:31.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:31.551+0000 7fcb71a00640 1 Processor -- start 2024-04-24T23:27:31.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:31.551+0000 7fcb71a00640 1 -- start start 2024-04-24T23:27:31.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:31.552+0000 7fcb71a00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcb6c071000 0x7fcb6c14b040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:31.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:31.552+0000 7fcb71a00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcb6c0c7080 0x7fcb6c14bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:31.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:31.552+0000 7fcb71a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcb6c154b70 0x7fcb6c14c340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:31.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:31.552+0000 7fcb71a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fcb6c154f50 con 0x7fcb6c154b70 2024-04-24T23:27:31.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:31.552+0000 7fcb71a00640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fcb6c06eb00 con 0x7fcb6c071000 2024-04-24T23:27:31.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:31.552+0000 7fcb71a00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fcb6c1550d0 con 0x7fcb6c0c7080 2024-04-24T23:27:31.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:31.552+0000 7fcb6aa00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcb6c071000 0x7fcb6c14b040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:31.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:31.552+0000 7fcb6a000640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcb6c0c7080 0x7fcb6c14bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:31.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:31.552+0000 7fcb6aa00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcb6c071000 0x7fcb6c14b040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:36054/0 (socket says 172.21.15.6:36054) 2024-04-24T23:27:31.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:31.552+0000 7fcb6aa00640 1 -- 172.21.15.6:0/1149677436 learned_addr learned my addr 172.21.15.6:0/1149677436 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:27:31.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:31.553+0000 7fcb6b400640 1 --2- 172.21.15.6:0/1149677436 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcb6c154b70 0x7fcb6c14c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:31.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:31.553+0000 7fcb6aa00640 1 -- 172.21.15.6:0/1149677436 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcb6c0c7080 msgr2=0x7fcb6c14bb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:31.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:31.553+0000 7fcb6aa00640 1 --2- 172.21.15.6:0/1149677436 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcb6c0c7080 0x7fcb6c14bb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:31.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:31.553+0000 7fcb6aa00640 1 -- 172.21.15.6:0/1149677436 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcb6c154b70 msgr2=0x7fcb6c14c340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:31.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:31.553+0000 7fcb6aa00640 1 --2- 172.21.15.6:0/1149677436 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcb6c154b70 0x7fcb6c14c340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:31.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:31.553+0000 7fcb6aa00640 1 -- 172.21.15.6:0/1149677436 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb6c1f0850 con 0x7fcb6c071000 2024-04-24T23:27:31.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:31.553+0000 7fcb6a000640 1 --2- 172.21.15.6:0/1149677436 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcb6c0c7080 0x7fcb6c14bb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:27:31.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:31.553+0000 7fcb6aa00640 1 --2- 172.21.15.6:0/1149677436 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcb6c071000 0x7fcb6c14b040 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fcb5c00c370 tx=0x7fcb5c00c830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:27:31.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:31.553+0000 7fcb62a00640 1 -- 172.21.15.6:0/1149677436 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb5c017070 con 0x7fcb6c071000 2024-04-24T23:27:31.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:31.553+0000 7fcb62a00640 1 -- 172.21.15.6:0/1149677436 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb5c0052d0 con 0x7fcb6c071000 2024-04-24T23:27:31.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:31.553+0000 7fcb62a00640 1 -- 172.21.15.6:0/1149677436 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb5c003850 con 0x7fcb6c071000 2024-04-24T23:27:31.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:31.553+0000 7fcb71a00640 1 -- 172.21.15.6:0/1149677436 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb6c1f0b30 con 0x7fcb6c071000 2024-04-24T23:27:31.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:31.554+0000 7fcb71a00640 1 -- 172.21.15.6:0/1149677436 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb6c1f10f0 con 0x7fcb6c071000 2024-04-24T23:27:31.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:31.555+0000 7fcb71a00640 1 -- 172.21.15.6:0/1149677436 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcb6c0767b0 con 0x7fcb6c071000 2024-04-24T23:27:31.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:31.556+0000 7fcb62a00640 1 -- 172.21.15.6:0/1149677436 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fcb5c01a070 con 0x7fcb6c071000 2024-04-24T23:27:31.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:31.557+0000 7fcb62a00640 1 --2- 172.21.15.6:0/1149677436 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fcb4c06be30 0x7fcb4c06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:31.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:31.557+0000 7fcb62a00640 1 -- 172.21.15.6:0/1149677436 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6934+0+0 (secure 0 0 0) 0x7fcb5c08a6d0 con 0x7fcb6c071000 2024-04-24T23:27:31.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:31.557+0000 7fcb6a000640 1 --2- 172.21.15.6:0/1149677436 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fcb4c06be30 0x7fcb4c06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:31.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:31.558+0000 7fcb6a000640 1 --2- 172.21.15.6:0/1149677436 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fcb4c06be30 0x7fcb4c06e2f0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fcb58034000 tx=0x7fcb5802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:27:31.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:31.561+0000 7fcb62a00640 1 -- 172.21.15.6:0/1149677436 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fcb5c058430 con 0x7fcb6c071000 2024-04-24T23:27:31.871 DEBUG:teuthology.parallel:result is None 2024-04-24T23:27:31.872 DEBUG:teuthology.orchestra.run.smithi006:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-04-24T23:27:31.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:31.930+0000 7fcb71a00640 1 -- 172.21.15.6:0/1149677436 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcb6c14d080 con 0x7fcb6c071000 2024-04-24T23:27:31.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:31.931+0000 7fcb62a00640 1 -- 172.21.15.6:0/1149677436 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) ==== 94+0+6861 (secure 0 0 0) 0x7fcb5c05c730 con 0x7fcb6c071000 2024-04-24T23:27:31.933 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:27:31.933 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":54,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5081,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/1190759626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1190759626},{"type":"v1","addr":"172.21.15.6:6834","nonce":1190759626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":5249,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6838/2416336040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2416336040},{"type":"v1","addr":"172.21.15.190:6838","nonce":2416336040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":42},{"gid":5503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/3928731059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":3928731059},{"type":"v1","addr":"172.21.15.6:6837","nonce":3928731059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":35},{"gid":5902,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/4245754886","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":4245754886},{"type":"v1","addr":"172.21.15.6:6835","nonce":4245754886}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":54}],"filesystems":[{"mdsmap":{"epoch":53,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:27:18.122830+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5725,"mds_1":5599},"failed":[],"damaged":[],"stopped":[2,3,4],"info":{"gid_5599":{"gid":5599,"name":"d","rank":1,"incarnation":43,"state":"up:active","state_seq":25,"addr":"172.21.15.190:6839/2643233757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":2643233757},{"type":"v1","addr":"172.21.15.190:6839","nonce":2643233757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5725":{"gid":5725,"name":"f","rank":0,"incarnation":49,"state":"up:active","state_seq":36,"addr":"172.21.15.190:6837/1151527362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":1151527362},{"type":"v1","addr":"172.21.15.190:6837","nonce":1151527362}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5725,"qdb_cluster":[5725,5599]},"id":1}]} 2024-04-24T23:27:31.933 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 54 2024-04-24T23:27:31.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:31.933+0000 7fcb71a00640 1 -- 172.21.15.6:0/1149677436 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fcb4c06be30 msgr2=0x7fcb4c06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:31.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:31.934+0000 7fcb71a00640 1 --2- 172.21.15.6:0/1149677436 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fcb4c06be30 0x7fcb4c06e2f0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fcb58034000 tx=0x7fcb5802e000 comp rx=0 tx=0).stop 2024-04-24T23:27:31.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:31.934+0000 7fcb71a00640 1 -- 172.21.15.6:0/1149677436 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcb6c071000 msgr2=0x7fcb6c14b040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:31.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:31.934+0000 7fcb71a00640 1 --2- 172.21.15.6:0/1149677436 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcb6c071000 0x7fcb6c14b040 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fcb5c00c370 tx=0x7fcb5c00c830 comp rx=0 tx=0).stop 2024-04-24T23:27:31.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:31.934+0000 7fcb71a00640 1 -- 172.21.15.6:0/1149677436 shutdown_connections 2024-04-24T23:27:31.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:31.934+0000 7fcb71a00640 1 --2- 172.21.15.6:0/1149677436 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fcb4c06be30 0x7fcb4c06e2f0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:31.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:31.934+0000 7fcb71a00640 1 --2- 172.21.15.6:0/1149677436 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcb6c0c7080 0x7fcb6c14bb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:31.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:31.934+0000 7fcb71a00640 1 --2- 172.21.15.6:0/1149677436 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcb6c154b70 0x7fcb6c14c340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:31.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:31.934+0000 7fcb71a00640 1 --2- 172.21.15.6:0/1149677436 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcb6c071000 0x7fcb6c14b040 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:31.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:31.934+0000 7fcb71a00640 1 -- 172.21.15.6:0/1149677436 >> 172.21.15.6:0/1149677436 conn(0x7fcb6c06b0e0 msgr2=0x7fcb6c06b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:27:31.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:31.934+0000 7fcb71a00640 1 -- 172.21.15.6:0/1149677436 shutdown_connections 2024-04-24T23:27:31.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:31.934+0000 7fcb71a00640 1 -- 172.21.15.6:0/1149677436 wait complete. 2024-04-24T23:27:31.953 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c reported in up:standby state 2024-04-24T23:27:31.953 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 110 secs before thrashing 2024-04-24T23:27:31.967 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-04-24T23:27:31.967 DEBUG:teuthology.run_tasks:Unwinding manager mds_thrash 2024-04-24T23:27:31.978 INFO:tasks.mds_thrash:joining mds_thrasher 2024-04-24T23:27:31.978 INFO:tasks.mds_thrash.fs.[cephfs]:stat['max_mds'] = 1 2024-04-24T23:27:31.978 INFO:tasks.mds_thrash.fs.[cephfs]:stat['deactivate'] = 0 2024-04-24T23:27:31.978 INFO:tasks.mds_thrash.fs.[cephfs]:stat['kill'] = 5 2024-04-24T23:27:31.979 INFO:tasks.mds_thrash:done joining 2024-04-24T23:27:31.979 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-04-24T23:27:32.024 INFO:tasks.kclient:Unmounting kernel clients... 2024-04-24T23:27:32.024 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-04-24T23:27:32.024 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-24T23:27:32.076 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-04-24T23:27:32.076 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-24T23:27:32.190 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T23:27:32.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:32.510+0000 7f6b0be00640 1 Processor -- start 2024-04-24T23:27:32.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:32.510+0000 7f6b0be00640 1 -- start start 2024-04-24T23:27:32.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:32.513+0000 7f6b0be00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f6b0c0c8d50 0x7f6b0c156a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:32.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:32.513+0000 7f6b0be00640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f6b0c06ce30 0x7f6b0c0c7770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:32.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:32.513+0000 7f6b0be00640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6b0c06d370 con 0x7f6b0c06e0b0 2024-04-24T23:27:32.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:32.513+0000 7f6b0be00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f6b0c06def0 con 0x7f6b0c0c8d50 2024-04-24T23:27:32.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:32.513+0000 7f6b0be00640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f6b0c06d650 con 0x7f6b0c06ce30 2024-04-24T23:27:32.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:32.514+0000 7f6b0aa00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f6b0c0c8d50 0x7f6b0c156a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:32.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:32.516+0000 7f6b0b400640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f6b0c06e0b0 0x7f6b0c1506b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.6:58208/0 (socket says 172.21.15.6:58208) 2024-04-24T23:27:32.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:32.516+0000 7f6b0b400640 1 -- 172.21.15.6:0/695641229 learned_addr learned my addr 172.21.15.6:0/695641229 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:27:32.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:32.516+0000 7f6b0a000640 1 --2- 172.21.15.6:0/695641229 >> v2:172.21.15.190:3300/0 conn(0x7f6b0c06ce30 0x7f6b0c0c7770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:32.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:32.516+0000 7f6b0aa00640 1 -- 172.21.15.6:0/695641229 >> v2:172.21.15.190:3300/0 conn(0x7f6b0c06ce30 msgr2=0x7f6b0c0c7770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:27:32.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:32.516+0000 7f6b0aa00640 1 --2- 172.21.15.6:0/695641229 >> v2:172.21.15.190:3300/0 conn(0x7f6b0c06ce30 0x7f6b0c0c7770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:32.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:32.517+0000 7f6b0aa00640 1 -- 172.21.15.6:0/695641229 >> v1:172.21.15.190:6789/0 conn(0x7f6b0c06e0b0 legacy=0x7f6b0c1506b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:32.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:32.517+0000 7f6b0aa00640 1 -- 172.21.15.6:0/695641229 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b0c06d930 con 0x7f6b0c0c8d50 2024-04-24T23:27:32.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:32.520+0000 7f6b0aa00640 1 --2- 172.21.15.6:0/695641229 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6b0c0c8d50 0x7f6b0c156a20 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f6b0c0c9820 tx=0x7f6b040332c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d288d0b0de5b554 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:27:32.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:32.521+0000 7f6b09600640 1 -- 172.21.15.6:0/695641229 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b04039070 con 0x7f6b0c0c8d50 2024-04-24T23:27:32.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:32.521+0000 7f6b09600640 1 -- 172.21.15.6:0/695641229 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6b0403d400 con 0x7f6b0c0c8d50 2024-04-24T23:27:32.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:32.521+0000 7f6b09600640 1 -- 172.21.15.6:0/695641229 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b0403d830 con 0x7f6b0c0c8d50 2024-04-24T23:27:32.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:32.522+0000 7f6b0be00640 1 -- 172.21.15.6:0/695641229 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6b0c0c8d50 msgr2=0x7f6b0c156a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:32.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:32.522+0000 7f6b0be00640 1 --2- 172.21.15.6:0/695641229 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6b0c0c8d50 0x7f6b0c156a20 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f6b0c0c9820 tx=0x7f6b040332c0 comp rx=0 tx=0).stop 2024-04-24T23:27:32.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:32.522+0000 7f6b0be00640 1 -- 172.21.15.6:0/695641229 shutdown_connections 2024-04-24T23:27:32.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:32.522+0000 7f6b0be00640 1 --2- 172.21.15.6:0/695641229 >> v2:172.21.15.190:3300/0 conn(0x7f6b0c06ce30 0x7f6b0c0c7770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:32.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:32.522+0000 7f6b0be00640 1 --2- 172.21.15.6:0/695641229 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6b0c0c8d50 0x7f6b0c156a20 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:32.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:32.522+0000 7f6b0be00640 1 -- 172.21.15.6:0/695641229 >> 172.21.15.6:0/695641229 conn(0x7f6b0c06ed80 msgr2=0x7f6b0c06f190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:27:32.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:32.522+0000 7f6b0be00640 1 -- 172.21.15.6:0/695641229 shutdown_connections 2024-04-24T23:27:32.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:32.522+0000 7f6b0be00640 1 -- 172.21.15.6:0/695641229 wait complete. 2024-04-24T23:27:32.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:32.526+0000 7f6b0be00640 1 Processor -- start 2024-04-24T23:27:32.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:32.526+0000 7f6b0be00640 1 -- start start 2024-04-24T23:27:32.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:32.528+0000 7f6b0be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6b0c06ce30 0x7f6b0c0c5730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:32.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:32.528+0000 7f6b0be00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6b0c06e0b0 0x7f6b0c14de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:32.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:32.528+0000 7f6b0be00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6b0c0c8d50 0x7f6b0c146a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:32.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:32.528+0000 7f6b0be00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f6b0c0cc1e0 con 0x7f6b0c06ce30 2024-04-24T23:27:32.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:32.528+0000 7f6b0be00640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f6b0c071020 con 0x7f6b0c06e0b0 2024-04-24T23:27:32.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:32.528+0000 7f6b0be00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f6b0c0cc360 con 0x7f6b0c0c8d50 2024-04-24T23:27:32.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:32.529+0000 7f6b0aa00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6b0c06ce30 0x7f6b0c0c5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:32.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:32.529+0000 7f6b0aa00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6b0c06ce30 0x7f6b0c0c5730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:48194/0 (socket says 172.21.15.6:48194) 2024-04-24T23:27:32.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:32.529+0000 7f6b0aa00640 1 -- 172.21.15.6:0/3659049992 learned_addr learned my addr 172.21.15.6:0/3659049992 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:27:32.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:32.529+0000 7f6b0a000640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6b0c06e0b0 0x7f6b0c14de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:32.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:32.529+0000 7f6b0b400640 1 --2- 172.21.15.6:0/3659049992 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6b0c0c8d50 0x7f6b0c146a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:32.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:32.529+0000 7f6b0a000640 1 -- 172.21.15.6:0/3659049992 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6b0c0c8d50 msgr2=0x7f6b0c146a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:32.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:32.529+0000 7f6b0a000640 1 --2- 172.21.15.6:0/3659049992 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6b0c0c8d50 0x7f6b0c146a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:32.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:32.529+0000 7f6b0a000640 1 -- 172.21.15.6:0/3659049992 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6b0c06ce30 msgr2=0x7f6b0c0c5730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:32.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:32.529+0000 7f6b0a000640 1 --2- 172.21.15.6:0/3659049992 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6b0c06ce30 0x7f6b0c0c5730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:32.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:32.529+0000 7f6b0a000640 1 -- 172.21.15.6:0/3659049992 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b0c157df0 con 0x7f6b0c06e0b0 2024-04-24T23:27:32.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:32.529+0000 7f6b0b400640 1 --2- 172.21.15.6:0/3659049992 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6b0c0c8d50 0x7f6b0c146a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:27:32.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:32.529+0000 7f6b0aa00640 1 --2- 172.21.15.6:0/3659049992 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6b0c06ce30 0x7f6b0c0c5730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:27:32.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:32.530+0000 7f6b0a000640 1 --2- 172.21.15.6:0/3659049992 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6b0c06e0b0 0x7f6b0c14de80 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f6af800d5c0 tx=0x7f6af800da80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:27:32.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:32.531+0000 7f6b02a00640 1 -- 172.21.15.6:0/3659049992 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6af80125f0 con 0x7f6b0c06e0b0 2024-04-24T23:27:32.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:32.531+0000 7f6b02a00640 1 -- 172.21.15.6:0/3659049992 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6af8019070 con 0x7f6b0c06e0b0 2024-04-24T23:27:32.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:32.531+0000 7f6b02a00640 1 -- 172.21.15.6:0/3659049992 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6af80129b0 con 0x7f6b0c06e0b0 2024-04-24T23:27:32.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:32.531+0000 7f6b0be00640 1 -- 172.21.15.6:0/3659049992 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b0c1580d0 con 0x7f6b0c06e0b0 2024-04-24T23:27:32.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:32.532+0000 7f6b0be00640 1 -- 172.21.15.6:0/3659049992 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b0c158690 con 0x7f6b0c06e0b0 2024-04-24T23:27:32.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:32.537+0000 7f6b0be00640 1 -- 172.21.15.6:0/3659049992 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b0c15a000 con 0x7f6b0c06e0b0 2024-04-24T23:27:32.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:32.538+0000 7f6b02a00640 1 -- 172.21.15.6:0/3659049992 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f6af8012b50 con 0x7f6b0c06e0b0 2024-04-24T23:27:32.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:32.539+0000 7f6b02a00640 1 --2- 172.21.15.6:0/3659049992 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f6aec06bf30 0x7f6aec06e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:32.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:32.539+0000 7f6b02a00640 1 -- 172.21.15.6:0/3659049992 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6934+0+0 (secure 0 0 0) 0x7f6af808b6d0 con 0x7f6b0c06e0b0 2024-04-24T23:27:32.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:32.543+0000 7f6b0aa00640 1 --2- 172.21.15.6:0/3659049992 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f6aec06bf30 0x7f6aec06e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:32.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:32.544+0000 7f6b0aa00640 1 --2- 172.21.15.6:0/3659049992 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f6aec06bf30 0x7f6aec06e3f0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f6b0403f040 tx=0x7f6b04039000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:27:32.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:32.548+0000 7f6b02a00640 1 -- 172.21.15.6:0/3659049992 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f6b0c15a000 con 0x7f6b0c06e0b0 2024-04-24T23:27:32.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:32.977+0000 7f6b0be00640 1 -- 172.21.15.6:0/3659049992 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6b0c075b30 con 0x7f6b0c06e0b0 2024-04-24T23:27:32.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:32.978+0000 7f6b02a00640 1 -- 172.21.15.6:0/3659049992 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v30) ==== 74+0+19063 (secure 0 0 0) 0x7f6af8059b90 con 0x7f6b0c06e0b0 2024-04-24T23:27:32.980 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:27:32.980 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":30,"fsid":"18bef68d-5e6d-4ec5-862b-b6f206e2588d","created":"2024-04-24T23:15:12.535072+0000","modified":"2024-04-24T23:25:42.694801+0000","last_up_change":"2024-04-24T23:15:23.773223+0000","last_in_change":"2024-04-24T23:15:14.905328+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T23:15:24.747441+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-24T23:15:34.051800+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-24T23:15:35.436870+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-24T23:15:39.034257+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"3b7a212d-d498-462e-90bf-ed493c803a09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":3022569437},{"type":"v1","addr":"172.21.15.6:6809","nonce":3022569437}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":3022569437},{"type":"v1","addr":"172.21.15.6:6811","nonce":3022569437}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":3022569437},{"type":"v1","addr":"172.21.15.6:6815","nonce":3022569437}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":3022569437},{"type":"v1","addr":"172.21.15.6:6813","nonce":3022569437}]},"public_addr":"172.21.15.6:6809/3022569437","cluster_addr":"172.21.15.6:6811/3022569437","heartbeat_back_addr":"172.21.15.6:6815/3022569437","heartbeat_front_addr":"172.21.15.6:6813/3022569437","state":["exists","up"]},{"osd":1,"uuid":"7c97f5a2-1735-4775-9f15-eb50099b19d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":3687836888},{"type":"v1","addr":"172.21.15.6:6817","nonce":3687836888}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":3687836888},{"type":"v1","addr":"172.21.15.6:6819","nonce":3687836888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":3687836888},{"type":"v1","addr":"172.21.15.6:6823","nonce":3687836888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":3687836888},{"type":"v1","addr":"172.21.15.6:6821","nonce":3687836888}]},"public_addr":"172.21.15.6:6817/3687836888","cluster_addr":"172.21.15.6:6819/3687836888","heartbeat_back_addr":"172.21.15.6:6823/3687836888","heartbeat_front_addr":"172.21.15.6:6821/3687836888","state":["exists","up"]},{"osd":2,"uuid":"d86bbc18-438c-4ec2-8018-ba7dd78f790b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":1020213724},{"type":"v1","addr":"172.21.15.6:6801","nonce":1020213724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":1020213724},{"type":"v1","addr":"172.21.15.6:6803","nonce":1020213724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":1020213724},{"type":"v1","addr":"172.21.15.6:6807","nonce":1020213724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":1020213724},{"type":"v1","addr":"172.21.15.6:6805","nonce":1020213724}]},"public_addr":"172.21.15.6:6801/1020213724","cluster_addr":"172.21.15.6:6803/1020213724","heartbeat_back_addr":"172.21.15.6:6807/1020213724","heartbeat_front_addr":"172.21.15.6:6805/1020213724","state":["exists","up"]},{"osd":3,"uuid":"3754ae16-4248-40dd-95f9-0f59c0eb5426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6824","nonce":3163883468},{"type":"v1","addr":"172.21.15.6:6825","nonce":3163883468}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":3163883468},{"type":"v1","addr":"172.21.15.6:6827","nonce":3163883468}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":3163883468},{"type":"v1","addr":"172.21.15.6:6831","nonce":3163883468}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":3163883468},{"type":"v1","addr":"172.21.15.6:6829","nonce":3163883468}]},"public_addr":"172.21.15.6:6825/3163883468","cluster_addr":"172.21.15.6:6827/3163883468","heartbeat_back_addr":"172.21.15.6:6831/3163883468","heartbeat_front_addr":"172.21.15.6:6829/3163883468","state":["exists","up"]},{"osd":4,"uuid":"8b2913a0-d006-479c-84ed-5b21e0e6c392","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":1989500934},{"type":"v1","addr":"172.21.15.190:6809","nonce":1989500934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":1989500934},{"type":"v1","addr":"172.21.15.190:6811","nonce":1989500934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":1989500934},{"type":"v1","addr":"172.21.15.190:6815","nonce":1989500934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":1989500934},{"type":"v1","addr":"172.21.15.190:6813","nonce":1989500934}]},"public_addr":"172.21.15.190:6809/1989500934","cluster_addr":"172.21.15.190:6811/1989500934","heartbeat_back_addr":"172.21.15.190:6815/1989500934","heartbeat_front_addr":"172.21.15.190:6813/1989500934","state":["exists","up"]},{"osd":5,"uuid":"499b79d0-0585-4dae-ab69-196a551793da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":3687548412},{"type":"v1","addr":"172.21.15.190:6801","nonce":3687548412}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":3687548412},{"type":"v1","addr":"172.21.15.190:6803","nonce":3687548412}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":3687548412},{"type":"v1","addr":"172.21.15.190:6807","nonce":3687548412}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":3687548412},{"type":"v1","addr":"172.21.15.190:6805","nonce":3687548412}]},"public_addr":"172.21.15.190:6801/3687548412","cluster_addr":"172.21.15.190:6803/3687548412","heartbeat_back_addr":"172.21.15.190:6807/3687548412","heartbeat_front_addr":"172.21.15.190:6805/3687548412","state":["exists","up"]},{"osd":6,"uuid":"6ba6df3b-4a6b-4a89-a890-0f79f387d4d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":870571564},{"type":"v1","addr":"172.21.15.190:6817","nonce":870571564}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":870571564},{"type":"v1","addr":"172.21.15.190:6819","nonce":870571564}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":870571564},{"type":"v1","addr":"172.21.15.190:6823","nonce":870571564}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":870571564},{"type":"v1","addr":"172.21.15.190:6821","nonce":870571564}]},"public_addr":"172.21.15.190:6817/870571564","cluster_addr":"172.21.15.190:6819/870571564","heartbeat_back_addr":"172.21.15.190:6823/870571564","heartbeat_front_addr":"172.21.15.190:6821/870571564","state":["exists","up"]},{"osd":7,"uuid":"e9668b1c-a000-4332-86e8-4efd73b02409","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":123639197},{"type":"v1","addr":"172.21.15.190:6825","nonce":123639197}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":123639197},{"type":"v1","addr":"172.21.15.190:6827","nonce":123639197}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":123639197},{"type":"v1","addr":"172.21.15.190:6831","nonce":123639197}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":123639197},{"type":"v1","addr":"172.21.15.190:6829","nonce":123639197}]},"public_addr":"172.21.15.190:6825/123639197","cluster_addr":"172.21.15.190:6827/123639197","heartbeat_back_addr":"172.21.15.190:6831/123639197","heartbeat_front_addr":"172.21.15.190:6829/123639197","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.321646+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.363873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.466963+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.531989+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:19.990707+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.043543+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.032492+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.115004+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.2f","mappings":[{"from":0,"to":7}]},{"pgid":"4.19","mappings":[{"from":7,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.6:6835/489092393":"2024-04-25T23:25:42.694416+0000","172.21.15.190:6837/580315269":"2024-04-25T23:24:32.678217+0000","172.21.15.6:6834/105267896":"2024-04-25T23:20:17.585424+0000","172.21.15.6:6833/489092393":"2024-04-25T23:25:42.694416+0000","172.21.15.6:6832/105267896":"2024-04-25T23:20:17.585424+0000","172.21.15.190:6834/580315269":"2024-04-25T23:24:32.678217+0000","172.21.15.6:6837/2590421094":"2024-04-25T23:18:27.556337+0000","172.21.15.6:6836/2590421094":"2024-04-25T23:18:27.556337+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T23:27:32.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:32.980+0000 7f6b0be00640 1 -- 172.21.15.6:0/3659049992 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f6aec06bf30 msgr2=0x7f6aec06e3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:32.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:32.980+0000 7f6b0be00640 1 --2- 172.21.15.6:0/3659049992 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f6aec06bf30 0x7f6aec06e3f0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f6b0403f040 tx=0x7f6b04039000 comp rx=0 tx=0).stop 2024-04-24T23:27:32.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:32.980+0000 7f6b0be00640 1 -- 172.21.15.6:0/3659049992 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6b0c06e0b0 msgr2=0x7f6b0c14de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:32.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:32.980+0000 7f6b0be00640 1 --2- 172.21.15.6:0/3659049992 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6b0c06e0b0 0x7f6b0c14de80 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f6af800d5c0 tx=0x7f6af800da80 comp rx=0 tx=0).stop 2024-04-24T23:27:32.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:32.980+0000 7f6b0be00640 1 -- 172.21.15.6:0/3659049992 shutdown_connections 2024-04-24T23:27:32.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:32.980+0000 7f6b0be00640 1 --2- 172.21.15.6:0/3659049992 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f6aec06bf30 0x7f6aec06e3f0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:32.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:32.980+0000 7f6b0be00640 1 --2- 172.21.15.6:0/3659049992 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6b0c0c8d50 0x7f6b0c146a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:32.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:32.980+0000 7f6b0be00640 1 --2- 172.21.15.6:0/3659049992 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6b0c06e0b0 0x7f6b0c14de80 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:32.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:32.980+0000 7f6b0be00640 1 --2- 172.21.15.6:0/3659049992 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6b0c06ce30 0x7f6b0c0c5730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:32.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:32.980+0000 7f6b0be00640 1 -- 172.21.15.6:0/3659049992 >> 172.21.15.6:0/3659049992 conn(0x7f6b0c06ed80 msgr2=0x7f6b0c0c7770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:27:32.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:32.981+0000 7f6b0be00640 1 -- 172.21.15.6:0/3659049992 shutdown_connections 2024-04-24T23:27:32.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:32.981+0000 7f6b0be00640 1 -- 172.21.15.6:0/3659049992 wait complete. 2024-04-24T23:27:32.998 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T23:15:24.747441+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-24T23:15:34.051800+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-24T23:15:35.436870+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-24T23:15:39.034257+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-24T23:27:32.999 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-24T23:27:33.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.286+0000 7f4f9ca00640 1 Processor -- start 2024-04-24T23:27:33.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.286+0000 7f4f9ca00640 1 -- start start 2024-04-24T23:27:33.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.288+0000 7f4f9ca00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4f9806eb80 0x7f4f980c9330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:33.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.289+0000 7f4f96800640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4f9806eb80 0x7f4f980c9330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:33.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.289+0000 7f4f95e00640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f4f9815ed90 0x7f4f98162a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.6:58212/0 (socket says 172.21.15.6:58212) 2024-04-24T23:27:33.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.290+0000 7f4f96800640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4f9806eb80 0x7f4f980c9330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.6:48590/0 (socket says 172.21.15.6:48590) 2024-04-24T23:27:33.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.290+0000 7f4f95e00640 1 -- 172.21.15.6:0/1005537922 learned_addr learned my addr 172.21.15.6:0/1005537922 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:27:33.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.290+0000 7f4f9ca00640 1 -- 172.21.15.6:0/1005537922 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f9815f240 con 0x7f4f9806ef60 2024-04-24T23:27:33.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.290+0000 7f4f9ca00640 1 -- 172.21.15.6:0/1005537922 --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f9815f520 con 0x7f4f9815ed90 2024-04-24T23:27:33.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.291+0000 7f4f9ca00640 1 -- 172.21.15.6:0/1005537922 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f4f980c8bd0 con 0x7f4f9806eb80 2024-04-24T23:27:33.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.291+0000 7f4f95400640 1 -- 172.21.15.6:0/1005537922 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3216229871 0 0) 0x7f4f9815f520 con 0x7f4f9815ed90 2024-04-24T23:27:33.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.291+0000 7f4f95400640 1 -- 172.21.15.6:0/1005537922 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4f7c003520 con 0x7f4f9815ed90 2024-04-24T23:27:33.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.292+0000 7f4f95400640 1 -- 172.21.15.6:0/1005537922 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 103534946 0 0) 0x7f4f9815f240 con 0x7f4f9806ef60 2024-04-24T23:27:33.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.292+0000 7f4f95400640 1 -- 172.21.15.6:0/1005537922 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4f9815f520 con 0x7f4f9806ef60 2024-04-24T23:27:33.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.292+0000 7f4f95400640 1 -- 172.21.15.6:0/1005537922 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4251618637 0 0) 0x7f4f7c003520 con 0x7f4f9815ed90 2024-04-24T23:27:33.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.292+0000 7f4f95400640 1 -- 172.21.15.6:0/1005537922 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4f9806eb80 msgr2=0x7f4f980c9330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:27:33.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.292+0000 7f4f95400640 1 --2- 172.21.15.6:0/1005537922 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4f9806eb80 0x7f4f980c9330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:33.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.292+0000 7f4f95400640 1 -- 172.21.15.6:0/1005537922 >> v1:172.21.15.6:6789/0 conn(0x7f4f9806ef60 legacy=0x7f4f98158b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:33.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.292+0000 7f4f95400640 1 -- 172.21.15.6:0/1005537922 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f9815f800 con 0x7f4f9815ed90 2024-04-24T23:27:33.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.292+0000 7f4f95400640 1 -- 172.21.15.6:0/1005537922 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3463268701 0 0) 0x7f4f80001670 con 0x7f4f9815ed90 2024-04-24T23:27:33.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.292+0000 7f4f95400640 1 -- 172.21.15.6:0/1005537922 >> v1:172.21.15.190:6789/0 conn(0x7f4f9815ed90 legacy=0x7f4f98162a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:33.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.292+0000 7f4f95400640 1 --2- 172.21.15.6:0/1005537922 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4f7c003b40 0x7f4f7c003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:33.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.292+0000 7f4f95400640 1 --2- 172.21.15.6:0/1005537922 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4f7c0079a0 0x7f4f7c009d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:33.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.292+0000 7f4f95400640 1 --2- 172.21.15.6:0/1005537922 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4f7c00a2d0 0x7f4f7c00c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:33.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.292+0000 7f4f95400640 1 -- 172.21.15.6:0/1005537922 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f4f980c8bd0 con 0x7f4f7c003b40 2024-04-24T23:27:33.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.292+0000 7f4f95400640 1 -- 172.21.15.6:0/1005537922 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f4f7c00ced0 con 0x7f4f7c0079a0 2024-04-24T23:27:33.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.292+0000 7f4f95400640 1 -- 172.21.15.6:0/1005537922 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f4f7c00d0d0 con 0x7f4f7c00a2d0 2024-04-24T23:27:33.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.293+0000 7f4f95e00640 1 --2- 172.21.15.6:0/1005537922 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4f7c0079a0 0x7f4f7c009d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:33.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.293+0000 7f4f97200640 1 --2- 172.21.15.6:0/1005537922 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4f7c00a2d0 0x7f4f7c00c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:33.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.293+0000 7f4f96800640 1 --2- 172.21.15.6:0/1005537922 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4f7c003b40 0x7f4f7c003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:33.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.293+0000 7f4f95e00640 1 -- 172.21.15.6:0/1005537922 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4f7c00a2d0 msgr2=0x7f4f7c00c6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:27:33.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.293+0000 7f4f95e00640 1 --2- 172.21.15.6:0/1005537922 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4f7c00a2d0 0x7f4f7c00c6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:33.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.293+0000 7f4f95e00640 1 -- 172.21.15.6:0/1005537922 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4f7c003b40 msgr2=0x7f4f7c003f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:27:33.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.293+0000 7f4f95e00640 1 --2- 172.21.15.6:0/1005537922 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4f7c003b40 0x7f4f7c003f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:33.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.293+0000 7f4f95e00640 1 -- 172.21.15.6:0/1005537922 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f9815f800 con 0x7f4f7c0079a0 2024-04-24T23:27:33.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.296+0000 7f4f95e00640 1 --2- 172.21.15.6:0/1005537922 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4f7c0079a0 0x7f4f7c009d90 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f4f8000c290 tx=0x7f4f7c005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9acd1fdf369cff70 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:27:33.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.297+0000 7f4f95400640 1 -- 172.21.15.6:0/1005537922 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f80007d10 con 0x7f4f7c0079a0 2024-04-24T23:27:33.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.297+0000 7f4f95400640 1 -- 172.21.15.6:0/1005537922 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f80005ad0 con 0x7f4f7c0079a0 2024-04-24T23:27:33.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.297+0000 7f4f95400640 1 -- 172.21.15.6:0/1005537922 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f80007070 con 0x7f4f7c0079a0 2024-04-24T23:27:33.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.298+0000 7f4f9ca00640 1 -- 172.21.15.6:0/1005537922 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4f7c0079a0 msgr2=0x7f4f7c009d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:33.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.298+0000 7f4f9ca00640 1 --2- 172.21.15.6:0/1005537922 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4f7c0079a0 0x7f4f7c009d90 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f4f8000c290 tx=0x7f4f7c005830 comp rx=0 tx=0).stop 2024-04-24T23:27:33.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.304+0000 7f4f97200640 1 -- 172.21.15.6:0/1005537922 reap_dead start 2024-04-24T23:27:33.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.305+0000 7f4f9ca00640 1 -- 172.21.15.6:0/1005537922 shutdown_connections 2024-04-24T23:27:33.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.305+0000 7f4f9ca00640 1 -- 172.21.15.6:0/1005537922 >> 172.21.15.6:0/1005537922 conn(0x7f4f9806cc40 msgr2=0x7f4f9806d050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:27:33.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.305+0000 7f4f9ca00640 1 -- 172.21.15.6:0/1005537922 shutdown_connections 2024-04-24T23:27:33.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.305+0000 7f4f9ca00640 1 -- 172.21.15.6:0/1005537922 wait complete. 2024-04-24T23:27:33.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.307+0000 7f4f9ca00640 1 Processor -- start 2024-04-24T23:27:33.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.308+0000 7f4f9ca00640 1 -- start start 2024-04-24T23:27:33.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.310+0000 7f4f9ca00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4f980c7140 0x7f4f981f9420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:33.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.310+0000 7f4f9ca00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4f980c7520 0x7f4f98157280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:33.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.310+0000 7f4f9ca00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4f98153b70 0x7f4f98154000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:33.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.310+0000 7f4f9ca00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f4f98163150 con 0x7f4f980c7140 2024-04-24T23:27:33.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.311+0000 7f4f9ca00640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f4f98162520 con 0x7f4f980c7520 2024-04-24T23:27:33.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.311+0000 7f4f95e00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4f980c7520 0x7f4f98157280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:33.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.311+0000 7f4f9ca00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f4f981632d0 con 0x7f4f98153b70 2024-04-24T23:27:33.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.311+0000 7f4f97200640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4f98153b70 0x7f4f98154000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:33.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.311+0000 7f4f95e00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4f980c7520 0x7f4f98157280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:36112/0 (socket says 172.21.15.6:36112) 2024-04-24T23:27:33.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.311+0000 7f4f95e00640 1 -- 172.21.15.6:0/2717250207 learned_addr learned my addr 172.21.15.6:0/2717250207 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:27:33.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.311+0000 7f4f96800640 1 --2- 172.21.15.6:0/2717250207 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4f980c7140 0x7f4f981f9420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:33.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.311+0000 7f4f95e00640 1 -- 172.21.15.6:0/2717250207 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4f98153b70 msgr2=0x7f4f98154000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:33.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.311+0000 7f4f95e00640 1 --2- 172.21.15.6:0/2717250207 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4f98153b70 0x7f4f98154000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:33.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.312+0000 7f4f95e00640 1 -- 172.21.15.6:0/2717250207 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4f980c7140 msgr2=0x7f4f981f9420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:33.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.312+0000 7f4f95e00640 1 --2- 172.21.15.6:0/2717250207 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4f980c7140 0x7f4f981f9420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:33.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.312+0000 7f4f95e00640 1 -- 172.21.15.6:0/2717250207 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f98154540 con 0x7f4f980c7520 2024-04-24T23:27:33.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.312+0000 7f4f96800640 1 --2- 172.21.15.6:0/2717250207 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4f980c7140 0x7f4f981f9420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:27:33.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.312+0000 7f4f95e00640 1 --2- 172.21.15.6:0/2717250207 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4f980c7520 0x7f4f98157280 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f4f80034040 tx=0x7f4f80001970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:27:33.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.314+0000 7f4f8ea00640 1 -- 172.21.15.6:0/2717250207 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f80041070 con 0x7f4f980c7520 2024-04-24T23:27:33.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.314+0000 7f4f8ea00640 1 -- 172.21.15.6:0/2717250207 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f80007070 con 0x7f4f980c7520 2024-04-24T23:27:33.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.314+0000 7f4f8ea00640 1 -- 172.21.15.6:0/2717250207 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f80007d10 con 0x7f4f980c7520 2024-04-24T23:27:33.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.314+0000 7f4f9ca00640 1 -- 172.21.15.6:0/2717250207 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f981f57d0 con 0x7f4f980c7520 2024-04-24T23:27:33.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.314+0000 7f4f9ca00640 1 -- 172.21.15.6:0/2717250207 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f981f5d90 con 0x7f4f980c7520 2024-04-24T23:27:33.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.317+0000 7f4f8ea00640 1 -- 172.21.15.6:0/2717250207 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f4f80044070 con 0x7f4f980c7520 2024-04-24T23:27:33.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.319+0000 7f4f9ca00640 1 -- 172.21.15.6:0/2717250207 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f98075b30 con 0x7f4f980c7520 2024-04-24T23:27:33.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.321+0000 7f4f8ea00640 1 --2- 172.21.15.6:0/2717250207 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f4f7c06d9b0 0x7f4f7c06fe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:33.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.322+0000 7f4f96800640 1 --2- 172.21.15.6:0/2717250207 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f4f7c06d9b0 0x7f4f7c06fe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:33.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.323+0000 7f4f8ea00640 1 -- 172.21.15.6:0/2717250207 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6934+0+0 (secure 0 0 0) 0x7f4f800ab490 con 0x7f4f980c7520 2024-04-24T23:27:33.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.334+0000 7f4f96800640 1 --2- 172.21.15.6:0/2717250207 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f4f7c06d9b0 0x7f4f7c06fe70 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f4f90009fd0 tx=0x7f4f9000b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:27:33.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.334+0000 7f4f8ea00640 1 -- 172.21.15.6:0/2717250207 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f4f8004b070 con 0x7f4f980c7520 2024-04-24T23:27:33.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.758+0000 7f4f9ca00640 1 -- 172.21.15.6:0/2717250207 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f4f980c9980 con 0x7f4f980c7520 2024-04-24T23:27:33.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.758+0000 7f4f8ea00640 1 -- 172.21.15.6:0/2717250207 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v30) ==== 93+0+10 (secure 0 0 0) 0x7f4f800799e0 con 0x7f4f980c7520 2024-04-24T23:27:33.760 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-04-24T23:27:33.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.761+0000 7f4f9ca00640 1 -- 172.21.15.6:0/2717250207 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f4f7c06d9b0 msgr2=0x7f4f7c06fe70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:33.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.761+0000 7f4f9ca00640 1 --2- 172.21.15.6:0/2717250207 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f4f7c06d9b0 0x7f4f7c06fe70 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f4f90009fd0 tx=0x7f4f9000b040 comp rx=0 tx=0).stop 2024-04-24T23:27:33.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.761+0000 7f4f9ca00640 1 -- 172.21.15.6:0/2717250207 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4f980c7520 msgr2=0x7f4f98157280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:33.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.761+0000 7f4f9ca00640 1 --2- 172.21.15.6:0/2717250207 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4f980c7520 0x7f4f98157280 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f4f80034040 tx=0x7f4f80001970 comp rx=0 tx=0).stop 2024-04-24T23:27:33.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.761+0000 7f4f9ca00640 1 -- 172.21.15.6:0/2717250207 shutdown_connections 2024-04-24T23:27:33.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.761+0000 7f4f9ca00640 1 --2- 172.21.15.6:0/2717250207 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f4f7c06d9b0 0x7f4f7c06fe70 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:33.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.761+0000 7f4f9ca00640 1 --2- 172.21.15.6:0/2717250207 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4f98153b70 0x7f4f98154000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:33.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.761+0000 7f4f9ca00640 1 --2- 172.21.15.6:0/2717250207 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4f980c7520 0x7f4f98157280 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:33.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.761+0000 7f4f9ca00640 1 --2- 172.21.15.6:0/2717250207 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4f980c7140 0x7f4f981f9420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:33.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.761+0000 7f4f9ca00640 1 -- 172.21.15.6:0/2717250207 >> 172.21.15.6:0/2717250207 conn(0x7f4f9806cc40 msgr2=0x7f4f98158940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:27:33.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.762+0000 7f4f9ca00640 1 -- 172.21.15.6:0/2717250207 shutdown_connections 2024-04-24T23:27:33.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.762+0000 7f4f9ca00640 1 -- 172.21.15.6:0/2717250207 wait complete. 2024-04-24T23:27:33.783 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-24T23:27:33.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.954+0000 7f0433800640 1 Processor -- start 2024-04-24T23:27:33.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.954+0000 7f0433800640 1 -- start start 2024-04-24T23:27:33.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.955+0000 7f0433800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f042c152bc0 0x7f042c159390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:33.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.955+0000 7f0433800640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f042c0c9360 0x7f042c15c610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:33.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.955+0000 7f0433800640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f042c0c6e70 con 0x7f042c0c8f50 2024-04-24T23:27:33.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.955+0000 7f0433800640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f042c0c73d0 con 0x7f042c152bc0 2024-04-24T23:27:33.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.955+0000 7f0433800640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f042c0c8ac0 con 0x7f042c0c9360 2024-04-24T23:27:33.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.955+0000 7f042be00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f042c0c8f50 0x7f042c15bf00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:38028/0 (socket says 172.21.15.6:38028) 2024-04-24T23:27:33.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.955+0000 7f042be00640 1 -- 172.21.15.6:0/1473639057 learned_addr learned my addr 172.21.15.6:0/1473639057 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:27:33.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.955+0000 7f0431800640 1 --2- 172.21.15.6:0/1473639057 >> v2:172.21.15.190:3300/0 conn(0x7f042c0c9360 0x7f042c15c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:33.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.955+0000 7f0430e00640 1 --2- 172.21.15.6:0/1473639057 >> v2:172.21.15.6:3300/0 conn(0x7f042c152bc0 0x7f042c159390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:33.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.956+0000 7f0431800640 1 -- 172.21.15.6:0/1473639057 >> v2:172.21.15.6:3300/0 conn(0x7f042c152bc0 msgr2=0x7f042c159390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:27:33.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.956+0000 7f0431800640 1 --2- 172.21.15.6:0/1473639057 >> v2:172.21.15.6:3300/0 conn(0x7f042c152bc0 0x7f042c159390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:33.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.956+0000 7f0431800640 1 -- 172.21.15.6:0/1473639057 >> v1:172.21.15.6:6789/0 conn(0x7f042c0c8f50 legacy=0x7f042c15bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:33.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.956+0000 7f0431800640 1 -- 172.21.15.6:0/1473639057 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f042c1598d0 con 0x7f042c0c9360 2024-04-24T23:27:33.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.956+0000 7f042b400640 1 -- 172.21.15.6:0/1473639057 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1027013265 0 0) 0x7f042c0c6e70 con 0x7f042c0c8f50 2024-04-24T23:27:33.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.957+0000 7f0431800640 1 --2- 172.21.15.6:0/1473639057 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f042c0c9360 0x7f042c15c610 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f042c0c7550 tx=0x7f042402a5d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dc2d9630790baddb server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:27:33.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.957+0000 7f042b400640 1 -- 172.21.15.6:0/1473639057 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0424031600 con 0x7f042c0c9360 2024-04-24T23:27:33.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.957+0000 7f042b400640 1 -- 172.21.15.6:0/1473639057 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f04240317a0 con 0x7f042c0c9360 2024-04-24T23:27:33.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.957+0000 7f042b400640 1 -- 172.21.15.6:0/1473639057 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0424031b70 con 0x7f042c0c9360 2024-04-24T23:27:33.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.958+0000 7f0433800640 1 -- 172.21.15.6:0/1473639057 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f042c0c9360 msgr2=0x7f042c15c610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:33.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.958+0000 7f0433800640 1 --2- 172.21.15.6:0/1473639057 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f042c0c9360 0x7f042c15c610 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f042c0c7550 tx=0x7f042402a5d0 comp rx=0 tx=0).stop 2024-04-24T23:27:33.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.958+0000 7f0433800640 1 -- 172.21.15.6:0/1473639057 shutdown_connections 2024-04-24T23:27:33.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.958+0000 7f0433800640 1 --2- 172.21.15.6:0/1473639057 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f042c0c9360 0x7f042c15c610 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:33.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.958+0000 7f0433800640 1 --2- 172.21.15.6:0/1473639057 >> v2:172.21.15.6:3300/0 conn(0x7f042c152bc0 0x7f042c159390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:33.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.958+0000 7f0433800640 1 -- 172.21.15.6:0/1473639057 >> 172.21.15.6:0/1473639057 conn(0x7f042c06b0e0 msgr2=0x7f042c06b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:27:33.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.958+0000 7f0433800640 1 -- 172.21.15.6:0/1473639057 shutdown_connections 2024-04-24T23:27:33.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.958+0000 7f0433800640 1 -- 172.21.15.6:0/1473639057 wait complete. 2024-04-24T23:27:33.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.960+0000 7f0433800640 1 Processor -- start 2024-04-24T23:27:33.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.960+0000 7f0433800640 1 -- start start 2024-04-24T23:27:33.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.961+0000 7f0433800640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f042c0c8f50 0x7f042c157360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:33.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.961+0000 7f0433800640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f042c0c9360 0x7f042c1578a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:33.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.961+0000 7f0433800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f042c152bc0 0x7f042c140f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:33.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.961+0000 7f0433800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f042c0c8470 con 0x7f042c152bc0 2024-04-24T23:27:33.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.961+0000 7f0433800640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f042c0c6700 con 0x7f042c0c9360 2024-04-24T23:27:33.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.961+0000 7f0433800640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f042c0c8750 con 0x7f042c0c8f50 2024-04-24T23:27:33.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.961+0000 7f0430e00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f042c0c8f50 0x7f042c157360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:33.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.961+0000 7f0430e00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f042c0c8f50 0x7f042c157360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.6:48626/0 (socket says 172.21.15.6:48626) 2024-04-24T23:27:33.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.961+0000 7f0430e00640 1 -- 172.21.15.6:0/1949092052 learned_addr learned my addr 172.21.15.6:0/1949092052 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:27:33.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.961+0000 7f042be00640 1 --2- 172.21.15.6:0/1949092052 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f042c0c9360 0x7f042c1578a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:33.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.961+0000 7f0431800640 1 --2- 172.21.15.6:0/1949092052 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f042c152bc0 0x7f042c140f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:33.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.961+0000 7f0430e00640 1 -- 172.21.15.6:0/1949092052 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f042c0c9360 msgr2=0x7f042c1578a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:33.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.961+0000 7f0430e00640 1 --2- 172.21.15.6:0/1949092052 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f042c0c9360 0x7f042c1578a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:33.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.961+0000 7f0430e00640 1 -- 172.21.15.6:0/1949092052 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f042c152bc0 msgr2=0x7f042c140f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:33.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.961+0000 7f0430e00640 1 --2- 172.21.15.6:0/1949092052 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f042c152bc0 0x7f042c140f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:33.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.961+0000 7f0430e00640 1 -- 172.21.15.6:0/1949092052 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f042c159270 con 0x7f042c0c8f50 2024-04-24T23:27:33.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.961+0000 7f0431800640 1 --2- 172.21.15.6:0/1949092052 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f042c152bc0 0x7f042c140f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:27:33.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.962+0000 7f0430e00640 1 --2- 172.21.15.6:0/1949092052 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f042c0c8f50 0x7f042c157360 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f042000e9b0 tx=0x7f042000ee70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:27:33.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.962+0000 7f0428c00640 1 -- 172.21.15.6:0/1949092052 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0420017070 con 0x7f042c0c8f50 2024-04-24T23:27:33.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.962+0000 7f0428c00640 1 -- 172.21.15.6:0/1949092052 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f042000c9a0 con 0x7f042c0c8f50 2024-04-24T23:27:33.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.962+0000 7f0428c00640 1 -- 172.21.15.6:0/1949092052 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f042000cd90 con 0x7f042c0c8f50 2024-04-24T23:27:33.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.962+0000 7f0433800640 1 -- 172.21.15.6:0/1949092052 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f042c141bd0 con 0x7f042c0c8f50 2024-04-24T23:27:33.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.962+0000 7f0433800640 1 -- 172.21.15.6:0/1949092052 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f042c072a50 con 0x7f042c0c8f50 2024-04-24T23:27:33.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.963+0000 7f0433800640 1 -- 172.21.15.6:0/1949092052 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f042c0767b0 con 0x7f042c0c8f50 2024-04-24T23:27:33.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.964+0000 7f0428c00640 1 -- 172.21.15.6:0/1949092052 <== mon.2 v2:172.21.15.190:3301/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f042001a070 con 0x7f042c0c8f50 2024-04-24T23:27:33.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.965+0000 7f0428c00640 1 --2- 172.21.15.6:0/1949092052 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f040c06be60 0x7f040c06e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:33.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.965+0000 7f0428c00640 1 -- 172.21.15.6:0/1949092052 <== mon.2 v2:172.21.15.190:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6934+0+0 (secure 0 0 0) 0x7f042008b600 con 0x7f042c0c8f50 2024-04-24T23:27:33.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.965+0000 7f042be00640 1 --2- 172.21.15.6:0/1949092052 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f040c06be60 0x7f040c06e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:33.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.965+0000 7f042be00640 1 --2- 172.21.15.6:0/1949092052 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f040c06be60 0x7f040c06e320 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f042c15cf50 tx=0x7f0418007de0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:27:33.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:33.969+0000 7f0428c00640 1 -- 172.21.15.6:0/1949092052 <== mon.2 v2:172.21.15.190:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f042c159270 con 0x7f042c0c8f50 2024-04-24T23:27:34.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:34.170+0000 7f0433800640 1 -- 172.21.15.6:0/1949092052 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f042c0c75e0 con 0x7f042c0c8f50 2024-04-24T23:27:34.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:34.170+0000 7f0428c00640 1 -- 172.21.15.6:0/1949092052 <== mon.2 v2:172.21.15.190:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v30) ==== 104+0+11 (secure 0 0 0) 0x7f0420059480 con 0x7f042c0c8f50 2024-04-24T23:27:34.172 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 32 2024-04-24T23:27:34.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:34.179+0000 7f0433800640 1 -- 172.21.15.6:0/1949092052 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f040c06be60 msgr2=0x7f040c06e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:34.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:34.179+0000 7f0433800640 1 --2- 172.21.15.6:0/1949092052 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f040c06be60 0x7f040c06e320 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f042c15cf50 tx=0x7f0418007de0 comp rx=0 tx=0).stop 2024-04-24T23:27:34.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:34.179+0000 7f0433800640 1 -- 172.21.15.6:0/1949092052 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f042c0c8f50 msgr2=0x7f042c157360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:34.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:34.179+0000 7f0433800640 1 --2- 172.21.15.6:0/1949092052 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f042c0c8f50 0x7f042c157360 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f042000e9b0 tx=0x7f042000ee70 comp rx=0 tx=0).stop 2024-04-24T23:27:34.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:34.180+0000 7f0433800640 1 -- 172.21.15.6:0/1949092052 shutdown_connections 2024-04-24T23:27:34.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:34.180+0000 7f0433800640 1 --2- 172.21.15.6:0/1949092052 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f040c06be60 0x7f040c06e320 secure :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f042c15cf50 tx=0x7f0418007de0 comp rx=0 tx=0).stop 2024-04-24T23:27:34.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:34.180+0000 7f0433800640 1 --2- 172.21.15.6:0/1949092052 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f042c152bc0 0x7f042c140f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:34.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:34.180+0000 7f0433800640 1 --2- 172.21.15.6:0/1949092052 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f042c0c9360 0x7f042c1578a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:34.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:34.180+0000 7f0433800640 1 --2- 172.21.15.6:0/1949092052 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f042c0c8f50 0x7f042c157360 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:34.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:34.180+0000 7f0433800640 1 -- 172.21.15.6:0/1949092052 >> 172.21.15.6:0/1949092052 conn(0x7f042c06b0e0 msgr2=0x7f042c06b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:27:34.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:34.183+0000 7f0433800640 1 -- 172.21.15.6:0/1949092052 shutdown_connections 2024-04-24T23:27:34.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:34.183+0000 7f0433800640 1 -- 172.21.15.6:0/1949092052 wait complete. 2024-04-24T23:27:34.229 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-24T23:27:34.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:34.423+0000 7f2aab400640 1 Processor -- start 2024-04-24T23:27:34.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:34.424+0000 7f2aab400640 1 -- start start 2024-04-24T23:27:34.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:34.425+0000 7f2aab400640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f2aa40c9880 0x7f2aa415b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:34.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:34.425+0000 7f2aab400640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2aa40cc710 con 0x7f2aa415ec80 2024-04-24T23:27:34.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:34.425+0000 7f2aab400640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2aa40cab00 con 0x7f2aa4154ef0 2024-04-24T23:27:34.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:34.425+0000 7f2aab400640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f2aa40cbca0 con 0x7f2aa40c9880 2024-04-24T23:27:34.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:34.425+0000 7f2aa9400640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f2aa40c9880 0x7f2aa415b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:34.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:34.425+0000 7f2aa3e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f2aa415ec80 0x7f2aa40c9170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:38032/0 (socket says 172.21.15.6:38032) 2024-04-24T23:27:34.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:34.425+0000 7f2aa3e00640 1 -- 172.21.15.6:0/3307120666 learned_addr learned my addr 172.21.15.6:0/3307120666 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:27:34.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:34.426+0000 7f2aa9400640 1 -- 172.21.15.6:0/3307120666 >> v1:172.21.15.190:6789/0 conn(0x7f2aa4154ef0 legacy=0x7f2aa415e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:34.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:34.426+0000 7f2aa9400640 1 -- 172.21.15.6:0/3307120666 >> v1:172.21.15.6:6789/0 conn(0x7f2aa415ec80 legacy=0x7f2aa40c9170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:34.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:34.426+0000 7f2aa9400640 1 -- 172.21.15.6:0/3307120666 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2aa40cade0 con 0x7f2aa40c9880 2024-04-24T23:27:34.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:34.426+0000 7f2aa3400640 1 -- 172.21.15.6:0/3307120666 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2125874101 0 0) 0x7f2aa40cc710 con 0x7f2aa415ec80 2024-04-24T23:27:34.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:34.427+0000 7f2aa9400640 1 --2- 172.21.15.6:0/3307120666 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2aa40c9880 0x7f2aa415b750 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f2a9c009d90 tx=0x7f2a9c02a5d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9ae592ac406d15cb server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:27:34.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:34.427+0000 7f2aa3400640 1 -- 172.21.15.6:0/3307120666 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a9c031600 con 0x7f2aa40c9880 2024-04-24T23:27:34.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:34.427+0000 7f2aa3400640 1 -- 172.21.15.6:0/3307120666 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a9c0317a0 con 0x7f2aa40c9880 2024-04-24T23:27:34.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:34.427+0000 7f2aa3400640 1 -- 172.21.15.6:0/3307120666 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a9c031b70 con 0x7f2aa40c9880 2024-04-24T23:27:34.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:34.427+0000 7f2aab400640 1 -- 172.21.15.6:0/3307120666 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2aa40c9880 msgr2=0x7f2aa415b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:34.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:34.427+0000 7f2aab400640 1 --2- 172.21.15.6:0/3307120666 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2aa40c9880 0x7f2aa415b750 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f2a9c009d90 tx=0x7f2a9c02a5d0 comp rx=0 tx=0).stop 2024-04-24T23:27:34.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:34.428+0000 7f2aab400640 1 -- 172.21.15.6:0/3307120666 shutdown_connections 2024-04-24T23:27:34.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:34.428+0000 7f2aab400640 1 --2- 172.21.15.6:0/3307120666 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2aa40c9880 0x7f2aa415b750 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:34.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:34.428+0000 7f2aab400640 1 -- 172.21.15.6:0/3307120666 >> 172.21.15.6:0/3307120666 conn(0x7f2aa40cc9a0 msgr2=0x7f2aa40ccd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:27:34.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:34.428+0000 7f2aab400640 1 -- 172.21.15.6:0/3307120666 shutdown_connections 2024-04-24T23:27:34.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:34.428+0000 7f2aab400640 1 -- 172.21.15.6:0/3307120666 wait complete. 2024-04-24T23:27:34.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:34.429+0000 7f2aab400640 1 Processor -- start 2024-04-24T23:27:34.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:34.430+0000 7f2aab400640 1 -- start start 2024-04-24T23:27:34.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:34.430+0000 7f2aab400640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2aa40c9880 0x7f2aa4072ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:34.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:34.431+0000 7f2aab400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2aa4154ef0 0x7f2aa4154280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:34.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:34.431+0000 7f2aab400640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2aa415ec80 0x7f2aa41535f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:34.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:34.431+0000 7f2aab400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f2aa40cbf00 con 0x7f2aa4154ef0 2024-04-24T23:27:34.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:34.431+0000 7f2aab400640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f2aa415bc90 con 0x7f2aa40c9880 2024-04-24T23:27:34.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:34.431+0000 7f2aab400640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f2aa40cc430 con 0x7f2aa415ec80 2024-04-24T23:27:34.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:34.431+0000 7f2aa8a00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2aa40c9880 0x7f2aa4072ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:34.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:34.431+0000 7f2aa3e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2aa4154ef0 0x7f2aa4154280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:34.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:34.431+0000 7f2aa9400640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2aa415ec80 0x7f2aa41535f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:34.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:34.431+0000 7f2aa3e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2aa4154ef0 0x7f2aa4154280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:48228/0 (socket says 172.21.15.6:48228) 2024-04-24T23:27:34.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:34.431+0000 7f2aa3e00640 1 -- 172.21.15.6:0/175117880 learned_addr learned my addr 172.21.15.6:0/175117880 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:27:34.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:34.431+0000 7f2aa3e00640 1 -- 172.21.15.6:0/175117880 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2aa415ec80 msgr2=0x7f2aa41535f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:34.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:34.431+0000 7f2aa3e00640 1 --2- 172.21.15.6:0/175117880 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2aa415ec80 0x7f2aa41535f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:34.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:34.431+0000 7f2aa3e00640 1 -- 172.21.15.6:0/175117880 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2aa40c9880 msgr2=0x7f2aa4072ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:34.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:34.431+0000 7f2aa3e00640 1 --2- 172.21.15.6:0/175117880 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2aa40c9880 0x7f2aa4072ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:34.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:34.431+0000 7f2aa3e00640 1 -- 172.21.15.6:0/175117880 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2aa41f43d0 con 0x7f2aa4154ef0 2024-04-24T23:27:34.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:34.431+0000 7f2aa9400640 1 --2- 172.21.15.6:0/175117880 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2aa415ec80 0x7f2aa41535f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:27:34.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:34.431+0000 7f2aa8a00640 1 --2- 172.21.15.6:0/175117880 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2aa40c9880 0x7f2aa4072ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:27:34.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:34.432+0000 7f2aa3e00640 1 --2- 172.21.15.6:0/175117880 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2aa4154ef0 0x7f2aa4154280 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f2a9000c460 tx=0x7f2a9000c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:27:34.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:34.432+0000 7f2aa0c00640 1 -- 172.21.15.6:0/175117880 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a90018070 con 0x7f2aa4154ef0 2024-04-24T23:27:34.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:34.432+0000 7f2aa0c00640 1 -- 172.21.15.6:0/175117880 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a90007500 con 0x7f2aa4154ef0 2024-04-24T23:27:34.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:34.432+0000 7f2aab400640 1 -- 172.21.15.6:0/175117880 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2aa415a980 con 0x7f2aa4154ef0 2024-04-24T23:27:34.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:34.432+0000 7f2aa0c00640 1 -- 172.21.15.6:0/175117880 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a90004a20 con 0x7f2aa4154ef0 2024-04-24T23:27:34.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:34.432+0000 7f2aab400640 1 -- 172.21.15.6:0/175117880 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2aa40c3570 con 0x7f2aa4154ef0 2024-04-24T23:27:34.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:34.433+0000 7f2aab400640 1 -- 172.21.15.6:0/175117880 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2aa40ca7a0 con 0x7f2aa4154ef0 2024-04-24T23:27:34.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:34.435+0000 7f2aa0c00640 1 -- 172.21.15.6:0/175117880 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f2a9001b070 con 0x7f2aa4154ef0 2024-04-24T23:27:34.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:34.436+0000 7f2aa0c00640 1 --2- 172.21.15.6:0/175117880 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f2a8406be30 0x7f2a8406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:34.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:34.436+0000 7f2aa0c00640 1 -- 172.21.15.6:0/175117880 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6934+0+0 (secure 0 0 0) 0x7f2a9008a660 con 0x7f2aa4154ef0 2024-04-24T23:27:34.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:34.437+0000 7f2aa8a00640 1 --2- 172.21.15.6:0/175117880 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f2a8406be30 0x7f2a8406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:34.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:34.437+0000 7f2aa8a00640 1 --2- 172.21.15.6:0/175117880 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f2a8406be30 0x7f2a8406e2f0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f2a98001b70 tx=0x7f2a98009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:27:34.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:34.441+0000 7f2aa0c00640 1 -- 172.21.15.6:0/175117880 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f2aa40ca7a0 con 0x7f2aa4154ef0 2024-04-24T23:27:34.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:34.635+0000 7f2aab400640 1 -- 172.21.15.6:0/175117880 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f2aa40767b0 con 0x7f2aa4154ef0 2024-04-24T23:27:34.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:34.650+0000 7f2aa0c00640 1 -- 172.21.15.6:0/175117880 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v30) ==== 100+0+11 (secure 0 0 0) 0x7f2a900583c0 con 0x7f2aa4154ef0 2024-04-24T23:27:34.652 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 64 2024-04-24T23:27:34.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:34.653+0000 7f2aab400640 1 -- 172.21.15.6:0/175117880 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f2a8406be30 msgr2=0x7f2a8406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:34.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:34.653+0000 7f2aab400640 1 --2- 172.21.15.6:0/175117880 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f2a8406be30 0x7f2a8406e2f0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f2a98001b70 tx=0x7f2a98009040 comp rx=0 tx=0).stop 2024-04-24T23:27:34.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:34.653+0000 7f2aab400640 1 -- 172.21.15.6:0/175117880 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2aa4154ef0 msgr2=0x7f2aa4154280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:34.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:34.653+0000 7f2aab400640 1 --2- 172.21.15.6:0/175117880 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2aa4154ef0 0x7f2aa4154280 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f2a9000c460 tx=0x7f2a9000c920 comp rx=0 tx=0).stop 2024-04-24T23:27:34.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:34.653+0000 7f2aab400640 1 -- 172.21.15.6:0/175117880 shutdown_connections 2024-04-24T23:27:34.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:34.653+0000 7f2aab400640 1 --2- 172.21.15.6:0/175117880 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f2a8406be30 0x7f2a8406e2f0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:34.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:34.653+0000 7f2aab400640 1 --2- 172.21.15.6:0/175117880 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2aa415ec80 0x7f2aa41535f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:34.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:34.653+0000 7f2aab400640 1 --2- 172.21.15.6:0/175117880 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2aa4154ef0 0x7f2aa4154280 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:34.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:34.653+0000 7f2aab400640 1 --2- 172.21.15.6:0/175117880 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2aa40c9880 0x7f2aa4072ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:34.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:34.653+0000 7f2aab400640 1 -- 172.21.15.6:0/175117880 >> 172.21.15.6:0/175117880 conn(0x7f2aa40cc9a0 msgr2=0x7f2aa406f130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:27:34.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:34.654+0000 7f2aab400640 1 -- 172.21.15.6:0/175117880 shutdown_connections 2024-04-24T23:27:34.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:34.654+0000 7f2aab400640 1 -- 172.21.15.6:0/175117880 wait complete. 2024-04-24T23:27:34.675 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-24T23:27:34.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:34.923+0000 7fbac2c00640 1 Processor -- start 2024-04-24T23:27:34.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:34.923+0000 7fbac2c00640 1 -- start start 2024-04-24T23:27:34.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:34.926+0000 7fbac2c00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbabc152bc0 0x7fbabc159390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:34.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:34.926+0000 7fbac2c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fbabc0c9330 0x7fbabc15c610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:34.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:34.926+0000 7fbac2c00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbabc0c6e70 con 0x7fbabc0c8f50 2024-04-24T23:27:34.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:34.926+0000 7fbac2c00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7fbabc0c73d0 con 0x7fbabc0c9330 2024-04-24T23:27:34.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:34.926+0000 7fbac2c00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fbabc0c8ac0 con 0x7fbabc152bc0 2024-04-24T23:27:34.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:34.926+0000 7fbabbe00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbabc152bc0 0x7fbabc159390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:34.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:34.926+0000 7fbac0c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fbabc0c9330 0x7fbabc15c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:34.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:34.926+0000 7fbab3e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fbabc0c8f50 0x7fbabc15bf00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:38050/0 (socket says 172.21.15.6:38050) 2024-04-24T23:27:34.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:34.926+0000 7fbab3e00640 1 -- 172.21.15.6:0/1201592033 learned_addr learned my addr 172.21.15.6:0/1201592033 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:27:34.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:34.927+0000 7fbabbe00640 1 -- 172.21.15.6:0/1201592033 >> v2:172.21.15.6:3300/0 conn(0x7fbabc0c9330 msgr2=0x7fbabc15c610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:27:34.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:34.927+0000 7fbabbe00640 1 --2- 172.21.15.6:0/1201592033 >> v2:172.21.15.6:3300/0 conn(0x7fbabc0c9330 0x7fbabc15c610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:34.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:34.927+0000 7fbabbe00640 1 -- 172.21.15.6:0/1201592033 >> v1:172.21.15.6:6789/0 conn(0x7fbabc0c8f50 legacy=0x7fbabc15bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:34.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:34.927+0000 7fbabbe00640 1 -- 172.21.15.6:0/1201592033 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbabc1598d0 con 0x7fbabc152bc0 2024-04-24T23:27:34.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:34.928+0000 7fbabbe00640 1 --2- 172.21.15.6:0/1201592033 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbabc152bc0 0x7fbabc159390 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fbaac0097a0 tx=0x7fbaac02a100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=566a6a516f2e1e3 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:27:34.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:34.928+0000 7fbabb400640 1 -- 172.21.15.6:0/1201592033 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbaac037070 con 0x7fbabc152bc0 2024-04-24T23:27:34.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:34.928+0000 7fbabb400640 1 -- 172.21.15.6:0/1201592033 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbaac02ae50 con 0x7fbabc152bc0 2024-04-24T23:27:34.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:34.928+0000 7fbabb400640 1 -- 172.21.15.6:0/1201592033 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbaac032660 con 0x7fbabc152bc0 2024-04-24T23:27:34.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:34.929+0000 7fbac2c00640 1 -- 172.21.15.6:0/1201592033 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbabc152bc0 msgr2=0x7fbabc159390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:34.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:34.929+0000 7fbac2c00640 1 --2- 172.21.15.6:0/1201592033 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbabc152bc0 0x7fbabc159390 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fbaac0097a0 tx=0x7fbaac02a100 comp rx=0 tx=0).stop 2024-04-24T23:27:34.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:34.929+0000 7fbac2c00640 1 -- 172.21.15.6:0/1201592033 shutdown_connections 2024-04-24T23:27:34.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:34.929+0000 7fbac2c00640 1 --2- 172.21.15.6:0/1201592033 >> v2:172.21.15.6:3300/0 conn(0x7fbabc0c9330 0x7fbabc15c610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:34.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:34.929+0000 7fbac2c00640 1 --2- 172.21.15.6:0/1201592033 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbabc152bc0 0x7fbabc159390 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:34.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:34.929+0000 7fbac2c00640 1 -- 172.21.15.6:0/1201592033 >> 172.21.15.6:0/1201592033 conn(0x7fbabc06b0e0 msgr2=0x7fbabc06b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:27:34.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:34.929+0000 7fbac2c00640 1 -- 172.21.15.6:0/1201592033 shutdown_connections 2024-04-24T23:27:34.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:34.929+0000 7fbac2c00640 1 -- 172.21.15.6:0/1201592033 wait complete. 2024-04-24T23:27:34.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:34.932+0000 7fbac2c00640 1 Processor -- start 2024-04-24T23:27:34.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:34.932+0000 7fbac2c00640 1 -- start start 2024-04-24T23:27:34.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:34.933+0000 7fbac2c00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbabc0c8f50 0x7fbabc13f210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:34.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:34.934+0000 7fbac2c00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbabc0c9330 0x7fbabc148710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:34.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:34.934+0000 7fbac2c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbabc152bc0 0x7fbabc141ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:34.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:34.934+0000 7fbac2c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fbabc0c8470 con 0x7fbabc152bc0 2024-04-24T23:27:34.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:34.934+0000 7fbac2c00640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fbabc0c6700 con 0x7fbabc0c9330 2024-04-24T23:27:34.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:34.934+0000 7fbac2c00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fbabc0c8750 con 0x7fbabc0c8f50 2024-04-24T23:27:34.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:34.934+0000 7fbabbe00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbabc0c8f50 0x7fbabc13f210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:34.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:34.934+0000 7fbabbe00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbabc0c8f50 0x7fbabc13f210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.6:52270/0 (socket says 172.21.15.6:52270) 2024-04-24T23:27:34.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:34.934+0000 7fbabbe00640 1 -- 172.21.15.6:0/1548934105 learned_addr learned my addr 172.21.15.6:0/1548934105 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:27:34.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:34.934+0000 7fbab3e00640 1 --2- 172.21.15.6:0/1548934105 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbabc0c9330 0x7fbabc148710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:34.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:34.934+0000 7fbac0c00640 1 --2- 172.21.15.6:0/1548934105 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbabc152bc0 0x7fbabc141ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:34.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:34.934+0000 7fbabbe00640 1 -- 172.21.15.6:0/1548934105 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbabc0c9330 msgr2=0x7fbabc148710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:34.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:34.934+0000 7fbabbe00640 1 --2- 172.21.15.6:0/1548934105 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbabc0c9330 0x7fbabc148710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:34.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:34.934+0000 7fbabbe00640 1 -- 172.21.15.6:0/1548934105 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbabc152bc0 msgr2=0x7fbabc141ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:34.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:34.934+0000 7fbabbe00640 1 --2- 172.21.15.6:0/1548934105 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbabc152bc0 0x7fbabc141ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:34.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:34.934+0000 7fbabbe00640 1 -- 172.21.15.6:0/1548934105 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbabc1ee760 con 0x7fbabc0c8f50 2024-04-24T23:27:34.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:34.935+0000 7fbabbe00640 1 --2- 172.21.15.6:0/1548934105 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbabc0c8f50 0x7fbabc13f210 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fbaac000c00 tx=0x7fbaac0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:27:34.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:34.935+0000 7fbab8c00640 1 -- 172.21.15.6:0/1548934105 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbaac041070 con 0x7fbabc0c8f50 2024-04-24T23:27:34.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:34.935+0000 7fbab8c00640 1 -- 172.21.15.6:0/1548934105 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbaac03c460 con 0x7fbabc0c8f50 2024-04-24T23:27:34.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:34.935+0000 7fbab8c00640 1 -- 172.21.15.6:0/1548934105 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbaac037040 con 0x7fbabc0c8f50 2024-04-24T23:27:34.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:34.935+0000 7fbac2c00640 1 -- 172.21.15.6:0/1548934105 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbabc1eea40 con 0x7fbabc0c8f50 2024-04-24T23:27:34.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:34.935+0000 7fbac2c00640 1 -- 172.21.15.6:0/1548934105 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fbabc1ef000 con 0x7fbabc0c8f50 2024-04-24T23:27:34.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:34.936+0000 7fbac2c00640 1 -- 172.21.15.6:0/1548934105 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbabc151ce0 con 0x7fbabc0c8f50 2024-04-24T23:27:34.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:34.937+0000 7fbab8c00640 1 -- 172.21.15.6:0/1548934105 <== mon.2 v2:172.21.15.190:3301/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fbaac044070 con 0x7fbabc0c8f50 2024-04-24T23:27:34.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:34.938+0000 7fbab8c00640 1 --2- 172.21.15.6:0/1548934105 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fba9806be30 0x7fba9806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:34.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:34.938+0000 7fbab8c00640 1 -- 172.21.15.6:0/1548934105 <== mon.2 v2:172.21.15.190:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6934+0+0 (secure 0 0 0) 0x7fbaac0abd10 con 0x7fbabc0c8f50 2024-04-24T23:27:34.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:34.938+0000 7fbab3e00640 1 --2- 172.21.15.6:0/1548934105 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fba9806be30 0x7fba9806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:34.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:34.939+0000 7fbab3e00640 1 --2- 172.21.15.6:0/1548934105 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fba9806be30 0x7fba9806e2f0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fbabc06bb30 tx=0x7fbaa4007e80 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:27:34.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:34.942+0000 7fbab8c00640 1 -- 172.21.15.6:0/1548934105 <== mon.2 v2:172.21.15.190:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fbabc151ce0 con 0x7fbabc0c8f50 2024-04-24T23:27:35.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:35.180+0000 7fbac2c00640 1 -- 172.21.15.6:0/1548934105 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fbabc0767b0 con 0x7fbabc0c8f50 2024-04-24T23:27:35.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:35.180+0000 7fbab8c00640 1 -- 172.21.15.6:0/1548934105 <== mon.2 v2:172.21.15.190:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v30) ==== 103+0+11 (secure 0 0 0) 0x7fbaac0b1020 con 0x7fbabc0c8f50 2024-04-24T23:27:35.182 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 32 2024-04-24T23:27:35.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:35.183+0000 7fbac2c00640 1 -- 172.21.15.6:0/1548934105 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fba9806be30 msgr2=0x7fba9806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:35.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:35.183+0000 7fbac2c00640 1 --2- 172.21.15.6:0/1548934105 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fba9806be30 0x7fba9806e2f0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fbabc06bb30 tx=0x7fbaa4007e80 comp rx=0 tx=0).stop 2024-04-24T23:27:35.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:35.183+0000 7fbac2c00640 1 -- 172.21.15.6:0/1548934105 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbabc0c8f50 msgr2=0x7fbabc13f210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:35.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:35.183+0000 7fbac2c00640 1 --2- 172.21.15.6:0/1548934105 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbabc0c8f50 0x7fbabc13f210 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fbaac000c00 tx=0x7fbaac0046c0 comp rx=0 tx=0).stop 2024-04-24T23:27:35.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:35.184+0000 7fbac2c00640 1 -- 172.21.15.6:0/1548934105 shutdown_connections 2024-04-24T23:27:35.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:35.184+0000 7fbac2c00640 1 --2- 172.21.15.6:0/1548934105 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fba9806be30 0x7fba9806e2f0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:35.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:35.184+0000 7fbac2c00640 1 --2- 172.21.15.6:0/1548934105 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbabc152bc0 0x7fbabc141ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:35.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:35.184+0000 7fbac2c00640 1 --2- 172.21.15.6:0/1548934105 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbabc0c9330 0x7fbabc148710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:35.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:35.184+0000 7fbac2c00640 1 --2- 172.21.15.6:0/1548934105 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbabc0c8f50 0x7fbabc13f210 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:35.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:35.184+0000 7fbac2c00640 1 -- 172.21.15.6:0/1548934105 >> 172.21.15.6:0/1548934105 conn(0x7fbabc06b0e0 msgr2=0x7fbabc14ce60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:27:35.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:35.184+0000 7fbac2c00640 1 -- 172.21.15.6:0/1548934105 shutdown_connections 2024-04-24T23:27:35.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:35.184+0000 7fbac2c00640 1 -- 172.21.15.6:0/1548934105 wait complete. 2024-04-24T23:27:35.203 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-24T23:27:35.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:35.520+0000 7f27e0e00640 1 Processor -- start 2024-04-24T23:27:35.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:35.520+0000 7f27e0e00640 1 -- start start 2024-04-24T23:27:35.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:35.521+0000 7f27e0e00640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f27dc154da0 0x7f27dc15ad00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:35.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:35.522+0000 7f27e0e00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f27dc0c5bb0 0x7f27dc0c6040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:35.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:35.522+0000 7f27e0e00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f27dc0c6580 con 0x7f27dc0cdb60 2024-04-24T23:27:35.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:35.522+0000 7f27e0e00640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f27dc15e2f0 con 0x7f27dc154da0 2024-04-24T23:27:35.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:35.522+0000 7f27e0e00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f27dc15e590 con 0x7f27dc0c5bb0 2024-04-24T23:27:35.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:35.523+0000 7f27daa00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f27dc0cdb60 0x7f27dc06bc40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:38058/0 (socket says 172.21.15.6:38058) 2024-04-24T23:27:35.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:35.523+0000 7f27daa00640 1 -- 172.21.15.6:0/1357477125 learned_addr learned my addr 172.21.15.6:0/1357477125 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:27:35.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:35.523+0000 7f27dbe00640 1 --2- 172.21.15.6:0/1357477125 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f27dc0c5bb0 0x7f27dc0c6040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:35.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:35.523+0000 7f27db400640 1 --2- 172.21.15.6:0/1357477125 >> v2:172.21.15.190:3300/0 conn(0x7f27dc154da0 0x7f27dc15ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:35.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:35.524+0000 7f27da000640 1 -- 172.21.15.6:0/1357477125 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2375790579 0 0) 0x7f27dc0c6580 con 0x7f27dc0cdb60 2024-04-24T23:27:35.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:35.524+0000 7f27da000640 1 -- 172.21.15.6:0/1357477125 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f27c0003520 con 0x7f27dc0cdb60 2024-04-24T23:27:35.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:35.524+0000 7f27db400640 1 -- 172.21.15.6:0/1357477125 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f27dc0c5bb0 msgr2=0x7f27dc0c6040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:27:35.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:35.524+0000 7f27db400640 1 --2- 172.21.15.6:0/1357477125 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f27dc0c5bb0 0x7f27dc0c6040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:35.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:35.524+0000 7f27db400640 1 -- 172.21.15.6:0/1357477125 >> v1:172.21.15.6:6789/0 conn(0x7f27dc0cdb60 legacy=0x7f27dc06bc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:35.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:35.524+0000 7f27db400640 1 -- 172.21.15.6:0/1357477125 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27dc155180 con 0x7f27dc154da0 2024-04-24T23:27:35.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:35.526+0000 7f27db400640 1 --2- 172.21.15.6:0/1357477125 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f27dc154da0 0x7f27dc15ad00 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f27cc0097a0 tx=0x7f27cc02a080 comp rx=0 tx=0).ready entity=mon.1 client_cookie=33a889cc2d3bf141 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:27:35.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:35.526+0000 7f27da000640 1 -- 172.21.15.6:0/1357477125 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27cc037070 con 0x7f27dc154da0 2024-04-24T23:27:35.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:35.526+0000 7f27da000640 1 -- 172.21.15.6:0/1357477125 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f27cc02add0 con 0x7f27dc154da0 2024-04-24T23:27:35.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:35.526+0000 7f27da000640 1 -- 172.21.15.6:0/1357477125 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27cc032660 con 0x7f27dc154da0 2024-04-24T23:27:35.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:35.527+0000 7f27e0e00640 1 -- 172.21.15.6:0/1357477125 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f27dc154da0 msgr2=0x7f27dc15ad00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:35.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:35.527+0000 7f27e0e00640 1 --2- 172.21.15.6:0/1357477125 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f27dc154da0 0x7f27dc15ad00 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f27cc0097a0 tx=0x7f27cc02a080 comp rx=0 tx=0).stop 2024-04-24T23:27:35.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:35.528+0000 7f27e0e00640 1 -- 172.21.15.6:0/1357477125 shutdown_connections 2024-04-24T23:27:35.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:35.528+0000 7f27e0e00640 1 --2- 172.21.15.6:0/1357477125 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f27dc0c5bb0 0x7f27dc0c6040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:35.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:35.528+0000 7f27e0e00640 1 --2- 172.21.15.6:0/1357477125 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f27dc154da0 0x7f27dc15ad00 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:35.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:35.528+0000 7f27e0e00640 1 -- 172.21.15.6:0/1357477125 >> 172.21.15.6:0/1357477125 conn(0x7f27dc06df00 msgr2=0x7f27dc072810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:27:35.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:35.530+0000 7f27e0e00640 1 -- 172.21.15.6:0/1357477125 shutdown_connections 2024-04-24T23:27:35.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:35.531+0000 7f27e0e00640 1 -- 172.21.15.6:0/1357477125 wait complete. 2024-04-24T23:27:35.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:35.534+0000 7f27e0e00640 1 Processor -- start 2024-04-24T23:27:35.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:35.534+0000 7f27e0e00640 1 -- start start 2024-04-24T23:27:35.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:35.535+0000 7f27e0e00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f27dc0c5bb0 0x7f27dc13fd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:35.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:35.535+0000 7f27e0e00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f27dc0cdb60 0x7f27dc140290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:35.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:35.535+0000 7f27e0e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f27dc154da0 0x7f27dc0cf4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:35.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:35.535+0000 7f27e0e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f27dc15ee10 con 0x7f27dc154da0 2024-04-24T23:27:35.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:35.536+0000 7f27e0e00640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f27dc15eb30 con 0x7f27dc0cdb60 2024-04-24T23:27:35.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:35.536+0000 7f27e0e00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f27dc15f0f0 con 0x7f27dc0c5bb0 2024-04-24T23:27:35.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:35.536+0000 7f27daa00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f27dc0cdb60 0x7f27dc140290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:35.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:35.536+0000 7f27db400640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f27dc0c5bb0 0x7f27dc13fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:35.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:35.537+0000 7f27daa00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f27dc0cdb60 0x7f27dc140290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:40680/0 (socket says 172.21.15.6:40680) 2024-04-24T23:27:35.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:35.537+0000 7f27daa00640 1 -- 172.21.15.6:0/1256585477 learned_addr learned my addr 172.21.15.6:0/1256585477 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:27:35.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:35.537+0000 7f27dbe00640 1 --2- 172.21.15.6:0/1256585477 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f27dc154da0 0x7f27dc0cf4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:35.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:35.537+0000 7f27daa00640 1 -- 172.21.15.6:0/1256585477 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f27dc0c5bb0 msgr2=0x7f27dc13fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:35.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:35.537+0000 7f27daa00640 1 --2- 172.21.15.6:0/1256585477 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f27dc0c5bb0 0x7f27dc13fd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:35.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:35.537+0000 7f27daa00640 1 -- 172.21.15.6:0/1256585477 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f27dc154da0 msgr2=0x7f27dc0cf4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:35.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:35.537+0000 7f27daa00640 1 --2- 172.21.15.6:0/1256585477 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f27dc154da0 0x7f27dc0cf4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:35.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:35.537+0000 7f27daa00640 1 -- 172.21.15.6:0/1256585477 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27dc13f1b0 con 0x7f27dc0cdb60 2024-04-24T23:27:35.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:35.537+0000 7f27dbe00640 1 --2- 172.21.15.6:0/1256585477 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f27dc154da0 0x7f27dc0cf4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:27:35.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:35.538+0000 7f27daa00640 1 --2- 172.21.15.6:0/1256585477 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f27dc0cdb60 0x7f27dc140290 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f27c800c560 tx=0x7f27c800ca20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:27:35.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:35.538+0000 7f27d3400640 1 -- 172.21.15.6:0/1256585477 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27c8017070 con 0x7f27dc0cdb60 2024-04-24T23:27:35.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:35.539+0000 7f27d3400640 1 -- 172.21.15.6:0/1256585477 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f27c8005ce0 con 0x7f27dc0cdb60 2024-04-24T23:27:35.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:35.539+0000 7f27d3400640 1 -- 172.21.15.6:0/1256585477 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27c80072b0 con 0x7f27dc0cdb60 2024-04-24T23:27:35.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:35.539+0000 7f27e0e00640 1 -- 172.21.15.6:0/1256585477 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27dc13f470 con 0x7f27dc0cdb60 2024-04-24T23:27:35.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:35.539+0000 7f27e0e00640 1 -- 172.21.15.6:0/1256585477 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27dc15de50 con 0x7f27dc0cdb60 2024-04-24T23:27:35.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:35.542+0000 7f27e0e00640 1 -- 172.21.15.6:0/1256585477 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f27dc14f770 con 0x7f27dc0cdb60 2024-04-24T23:27:35.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:35.545+0000 7f27d3400640 1 -- 172.21.15.6:0/1256585477 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f27c801a070 con 0x7f27dc0cdb60 2024-04-24T23:27:35.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:35.546+0000 7f27d3400640 1 --2- 172.21.15.6:0/1256585477 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f27bc06bf30 0x7f27bc06e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:35.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:35.546+0000 7f27d3400640 1 -- 172.21.15.6:0/1256585477 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6934+0+0 (secure 0 0 0) 0x7f27c808a6e0 con 0x7f27dc0cdb60 2024-04-24T23:27:35.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:35.549+0000 7f27db400640 1 --2- 172.21.15.6:0/1256585477 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f27bc06bf30 0x7f27bc06e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:35.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:35.550+0000 7f27db400640 1 --2- 172.21.15.6:0/1256585477 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f27bc06bf30 0x7f27bc06e3f0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f27cc034000 tx=0x7f27cc02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:27:35.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:35.553+0000 7f27d3400640 1 -- 172.21.15.6:0/1256585477 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f27dc14f770 con 0x7f27dc0cdb60 2024-04-24T23:27:35.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:35.933+0000 7f27e0e00640 1 -- 172.21.15.6:0/1256585477 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f27dc0cc940 con 0x7f27dc0cdb60 2024-04-24T23:27:35.935 INFO:teuthology.orchestra.run.smithi006.stdout:172.21.15.6:6835/489092393 2024-04-25T23:25:42.694416+0000 2024-04-24T23:27:35.935 INFO:teuthology.orchestra.run.smithi006.stdout:172.21.15.190:6837/580315269 2024-04-25T23:24:32.678217+0000 2024-04-24T23:27:35.936 INFO:teuthology.orchestra.run.smithi006.stdout:172.21.15.6:6834/105267896 2024-04-25T23:20:17.585424+0000 2024-04-24T23:27:35.936 INFO:teuthology.orchestra.run.smithi006.stdout:172.21.15.6:6833/489092393 2024-04-25T23:25:42.694416+0000 2024-04-24T23:27:35.936 INFO:teuthology.orchestra.run.smithi006.stdout:172.21.15.6:6832/105267896 2024-04-25T23:20:17.585424+0000 2024-04-24T23:27:35.936 INFO:teuthology.orchestra.run.smithi006.stdout:172.21.15.190:6834/580315269 2024-04-25T23:24:32.678217+0000 2024-04-24T23:27:35.936 INFO:teuthology.orchestra.run.smithi006.stdout:172.21.15.6:6837/2590421094 2024-04-25T23:18:27.556337+0000 2024-04-24T23:27:35.936 INFO:teuthology.orchestra.run.smithi006.stdout:172.21.15.6:6836/2590421094 2024-04-25T23:18:27.556337+0000 2024-04-24T23:27:35.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:35.934+0000 7f27d3400640 1 -- 172.21.15.6:0/1256585477 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v30) ==== 80+0+478 (secure 0 0 0) 0x7f27dc0cc940 con 0x7f27dc0cdb60 2024-04-24T23:27:35.936 INFO:teuthology.orchestra.run.smithi006.stderr:listed 8 entries 2024-04-24T23:27:35.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:35.937+0000 7f27e0e00640 1 -- 172.21.15.6:0/1256585477 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f27bc06bf30 msgr2=0x7f27bc06e3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:35.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:35.937+0000 7f27e0e00640 1 --2- 172.21.15.6:0/1256585477 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f27bc06bf30 0x7f27bc06e3f0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f27cc034000 tx=0x7f27cc02e000 comp rx=0 tx=0).stop 2024-04-24T23:27:35.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:35.937+0000 7f27e0e00640 1 -- 172.21.15.6:0/1256585477 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f27dc0cdb60 msgr2=0x7f27dc140290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:35.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:35.937+0000 7f27e0e00640 1 --2- 172.21.15.6:0/1256585477 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f27dc0cdb60 0x7f27dc140290 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f27c800c560 tx=0x7f27c800ca20 comp rx=0 tx=0).stop 2024-04-24T23:27:35.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:35.937+0000 7f27e0e00640 1 -- 172.21.15.6:0/1256585477 shutdown_connections 2024-04-24T23:27:35.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:35.937+0000 7f27e0e00640 1 --2- 172.21.15.6:0/1256585477 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f27bc06bf30 0x7f27bc06e3f0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:35.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:35.937+0000 7f27e0e00640 1 --2- 172.21.15.6:0/1256585477 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f27dc154da0 0x7f27dc0cf4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:35.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:35.937+0000 7f27e0e00640 1 --2- 172.21.15.6:0/1256585477 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f27dc0cdb60 0x7f27dc140290 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:35.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:35.937+0000 7f27e0e00640 1 --2- 172.21.15.6:0/1256585477 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f27dc0c5bb0 0x7f27dc13fd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:35.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:35.937+0000 7f27e0e00640 1 -- 172.21.15.6:0/1256585477 >> 172.21.15.6:0/1256585477 conn(0x7f27dc06df00 msgr2=0x7f27dc152c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:27:35.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:35.937+0000 7f27e0e00640 1 -- 172.21.15.6:0/1256585477 shutdown_connections 2024-04-24T23:27:35.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:35.937+0000 7f27e0e00640 1 -- 172.21.15.6:0/1256585477 wait complete. 2024-04-24T23:27:35.957 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-24T23:27:35.957 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:27:35.957 DEBUG:teuthology.orchestra.run.smithi006:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-04-24T23:27:36.167 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi006.front.sepia.ceph.com 2024-04-24T23:27:36.167 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:27:36.167 DEBUG:teuthology.orchestra.run.smithi006:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-24T23:27:36.465 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:27:36.465 DEBUG:teuthology.orchestra.run.smithi006:> ip netns list 2024-04-24T23:27:41.177 INFO:teuthology.orchestra.run.smithi006.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-24T23:27:41.178 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:27:41.178 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-24T23:27:41.264 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:27:41.264 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip link delete ceph-brx 2024-04-24T23:27:41.432 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-04-24T23:27:41.443 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-24T23:27:41.443 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T23:27:41.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:41.720+0000 7f0b1e200640 1 Processor -- start 2024-04-24T23:27:41.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:41.720+0000 7f0b1e200640 1 -- start start 2024-04-24T23:27:41.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:41.722+0000 7f0b1e200640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f0b1815aab0 0x7f0b180c6e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:41.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:41.722+0000 7f0b1e200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0b1806ec50 con 0x7f0b180c7350 2024-04-24T23:27:41.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:41.722+0000 7f0b1e200640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0b1806ef30 con 0x7f0b18150d50 2024-04-24T23:27:41.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:41.722+0000 7f0b1e200640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f0b1806f210 con 0x7f0b1815aab0 2024-04-24T23:27:41.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:41.722+0000 7f0b16a00640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f0b1815aab0 0x7f0b180c6e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:41.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:41.722+0000 7f0b16a00640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f0b1815aab0 0x7f0b180c6e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:40682/0 (socket says 172.21.15.6:40682) 2024-04-24T23:27:41.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:41.723+0000 7f0b16a00640 1 -- 172.21.15.6:0/224114521 learned_addr learned my addr 172.21.15.6:0/224114521 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:27:41.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:41.723+0000 7f0b16a00640 1 -- 172.21.15.6:0/224114521 >> v1:172.21.15.190:6789/0 conn(0x7f0b18150d50 legacy=0x7f0b1806e3a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-24T23:27:41.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:41.723+0000 7f0b16a00640 1 -- 172.21.15.6:0/224114521 >> v1:172.21.15.6:6789/0 conn(0x7f0b180c7350 legacy=0x7f0b180c8d30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-24T23:27:41.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:41.724+0000 7f0b16a00640 1 -- 172.21.15.6:0/224114521 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b180c8250 con 0x7f0b1815aab0 2024-04-24T23:27:41.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:41.727+0000 7f0b16a00640 1 --2- 172.21.15.6:0/224114521 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0b1815aab0 0x7f0b180c6e10 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f0b08009870 tx=0x7f0b0802a330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dea0fdf8bc8349d4 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:27:41.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:41.727+0000 7f0b16000640 1 -- 172.21.15.6:0/224114521 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b08037070 con 0x7f0b1815aab0 2024-04-24T23:27:41.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:41.728+0000 7f0b16000640 1 -- 172.21.15.6:0/224114521 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b08032640 con 0x7f0b1815aab0 2024-04-24T23:27:41.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:41.728+0000 7f0b16000640 1 -- 172.21.15.6:0/224114521 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b08032a10 con 0x7f0b1815aab0 2024-04-24T23:27:41.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:41.729+0000 7f0b1e200640 1 -- 172.21.15.6:0/224114521 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0b1815aab0 msgr2=0x7f0b180c6e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:41.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:41.729+0000 7f0b1e200640 1 --2- 172.21.15.6:0/224114521 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0b1815aab0 0x7f0b180c6e10 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f0b08009870 tx=0x7f0b0802a330 comp rx=0 tx=0).stop 2024-04-24T23:27:41.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:41.729+0000 7f0b1e200640 1 -- 172.21.15.6:0/224114521 shutdown_connections 2024-04-24T23:27:41.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:41.729+0000 7f0b1e200640 1 --2- 172.21.15.6:0/224114521 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0b1815aab0 0x7f0b180c6e10 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:41.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:41.729+0000 7f0b1e200640 1 -- 172.21.15.6:0/224114521 >> 172.21.15.6:0/224114521 conn(0x7f0b1806b0e0 msgr2=0x7f0b1806b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:27:41.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:41.729+0000 7f0b1e200640 1 -- 172.21.15.6:0/224114521 shutdown_connections 2024-04-24T23:27:41.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:41.729+0000 7f0b1e200640 1 -- 172.21.15.6:0/224114521 wait complete. 2024-04-24T23:27:41.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:41.732+0000 7f0b1e200640 1 Processor -- start 2024-04-24T23:27:41.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:41.732+0000 7f0b1e200640 1 -- start start 2024-04-24T23:27:41.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:41.734+0000 7f0b1e200640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0b180c7350 0x7f0b18152420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:41.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:41.735+0000 7f0b1e200640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0b18150d50 0x7f0b18153f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:41.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:41.735+0000 7f0b1e200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0b1815aab0 0x7f0b1813b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:41.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:41.735+0000 7f0b1e200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f0b1815af00 con 0x7f0b1815aab0 2024-04-24T23:27:41.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:41.735+0000 7f0b17e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0b1815aab0 0x7f0b1813b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:41.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:41.735+0000 7f0b16a00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0b18150d50 0x7f0b18153f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:41.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:41.736+0000 7f0b16a00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0b18150d50 0x7f0b18153f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:40688/0 (socket says 172.21.15.6:40688) 2024-04-24T23:27:41.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:41.736+0000 7f0b17e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0b1815aab0 0x7f0b1813b540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:33506/0 (socket says 172.21.15.6:33506) 2024-04-24T23:27:41.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:41.736+0000 7f0b16a00640 1 -- 172.21.15.6:0/676797960 learned_addr learned my addr 172.21.15.6:0/676797960 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:27:41.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:41.735+0000 7f0b17400640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0b180c7350 0x7f0b18152420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:41.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:41.736+0000 7f0b1e200640 1 -- 172.21.15.6:0/676797960 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f0b180c8810 con 0x7f0b18150d50 2024-04-24T23:27:41.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:41.736+0000 7f0b1e200640 1 -- 172.21.15.6:0/676797960 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f0b18151130 con 0x7f0b180c7350 2024-04-24T23:27:41.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:41.737+0000 7f0b16a00640 1 -- 172.21.15.6:0/676797960 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0b180c7350 msgr2=0x7f0b18152420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:41.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:41.737+0000 7f0b16a00640 1 --2- 172.21.15.6:0/676797960 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0b180c7350 0x7f0b18152420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:41.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:41.737+0000 7f0b16a00640 1 -- 172.21.15.6:0/676797960 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0b1815aab0 msgr2=0x7f0b1813b540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:41.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:41.737+0000 7f0b16a00640 1 --2- 172.21.15.6:0/676797960 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0b1815aab0 0x7f0b1813b540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:41.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:41.737+0000 7f0b16a00640 1 -- 172.21.15.6:0/676797960 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b181591d0 con 0x7f0b18150d50 2024-04-24T23:27:41.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:41.738+0000 7f0b16a00640 1 --2- 172.21.15.6:0/676797960 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0b18150d50 0x7f0b18153f30 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f0b08009870 tx=0x7f0b080046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:27:41.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:41.738+0000 7f0b0f400640 1 -- 172.21.15.6:0/676797960 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b08041070 con 0x7f0b18150d50 2024-04-24T23:27:41.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:41.738+0000 7f0b0f400640 1 -- 172.21.15.6:0/676797960 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b0803c870 con 0x7f0b18150d50 2024-04-24T23:27:41.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:41.738+0000 7f0b0f400640 1 -- 172.21.15.6:0/676797960 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b08037040 con 0x7f0b18150d50 2024-04-24T23:27:41.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:41.738+0000 7f0b1e200640 1 -- 172.21.15.6:0/676797960 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b18159490 con 0x7f0b18150d50 2024-04-24T23:27:41.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:41.740+0000 7f0b1e200640 1 -- 172.21.15.6:0/676797960 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b1813ad40 con 0x7f0b18150d50 2024-04-24T23:27:41.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:41.742+0000 7f0b1e200640 1 -- 172.21.15.6:0/676797960 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0b180767b0 con 0x7f0b18150d50 2024-04-24T23:27:41.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:41.742+0000 7f0b0f400640 1 -- 172.21.15.6:0/676797960 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f0b08044070 con 0x7f0b18150d50 2024-04-24T23:27:41.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:41.744+0000 7f0b0f400640 1 --2- 172.21.15.6:0/676797960 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f0af8070270 0x7f0af8072730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:41.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:41.744+0000 7f0b0f400640 1 -- 172.21.15.6:0/676797960 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6934+0+0 (secure 0 0 0) 0x7f0b080ab7a0 con 0x7f0b18150d50 2024-04-24T23:27:41.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:41.744+0000 7f0b17400640 1 --2- 172.21.15.6:0/676797960 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f0af8070270 0x7f0af8072730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:41.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:41.745+0000 7f0b17400640 1 --2- 172.21.15.6:0/676797960 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f0af8070270 0x7f0af8072730 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f0b04006fd0 tx=0x7f0b040090e0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:27:41.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:41.748+0000 7f0b0f400640 1 -- 172.21.15.6:0/676797960 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f0b08079630 con 0x7f0b18150d50 2024-04-24T23:27:41.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:41.948+0000 7f0b1e200640 1 -- 172.21.15.6:0/676797960 --> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f0b18073350 con 0x7f0af8070270 2024-04-24T23:27:41.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:41.958+0000 7f0b0f400640 1 -- 172.21.15.6:0/676797960 <== mgr.4105 v2:172.21.15.190:6832/806769663 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+334839 (secure 0 0 0) 0x7f0b18073350 con 0x7f0af8070270 2024-04-24T23:27:41.966 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:27:41.967 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-04-24T23:27:41.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:41.964+0000 7f0b1e200640 1 -- 172.21.15.6:0/676797960 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f0af8070270 msgr2=0x7f0af8072730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:41.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:41.964+0000 7f0b1e200640 1 --2- 172.21.15.6:0/676797960 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f0af8070270 0x7f0af8072730 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f0b04006fd0 tx=0x7f0b040090e0 comp rx=0 tx=0).stop 2024-04-24T23:27:41.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:41.964+0000 7f0b1e200640 1 -- 172.21.15.6:0/676797960 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0b18150d50 msgr2=0x7f0b18153f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:41.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:41.964+0000 7f0b1e200640 1 --2- 172.21.15.6:0/676797960 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0b18150d50 0x7f0b18153f30 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f0b08009870 tx=0x7f0b080046c0 comp rx=0 tx=0).stop 2024-04-24T23:27:41.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:41.964+0000 7f0b1e200640 1 -- 172.21.15.6:0/676797960 shutdown_connections 2024-04-24T23:27:41.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:41.964+0000 7f0b1e200640 1 --2- 172.21.15.6:0/676797960 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f0af8070270 0x7f0af8072730 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:41.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:41.964+0000 7f0b1e200640 1 --2- 172.21.15.6:0/676797960 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0b1815aab0 0x7f0b1813b540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:41.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:41.964+0000 7f0b1e200640 1 --2- 172.21.15.6:0/676797960 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0b18150d50 0x7f0b18153f30 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:41.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:41.964+0000 7f0b1e200640 1 --2- 172.21.15.6:0/676797960 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0b180c7350 0x7f0b18152420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:41.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:41.964+0000 7f0b1e200640 1 -- 172.21.15.6:0/676797960 >> 172.21.15.6:0/676797960 conn(0x7f0b1806b0e0 msgr2=0x7f0b18072080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:27:41.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:41.964+0000 7f0b1e200640 1 -- 172.21.15.6:0/676797960 shutdown_connections 2024-04-24T23:27:41.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:41.965+0000 7f0b1e200640 1 -- 172.21.15.6:0/676797960 wait complete. 2024-04-24T23:27:41.983 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":391,"stamp":"2024-04-24T23:27:40.871751+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":7192421103,"num_objects":2015,"num_object_clones":0,"num_object_copies":7802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2015,"num_whiteouts":0,"num_read":2721,"num_read_kb":85149,"num_write":2063116,"num_write_kb":29276251,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":74,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":144180,"ondisk_log_size":144180,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":21574960,"kb_used_data":14558320,"kb_used_omap":12,"kb_used_meta":7012467,"kb_avail":733399760,"statfs":{"total":773094113280,"available":751001354240,"internally_reserved":0,"allocated":14907719680,"data_stored":14905992343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":7180766770},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,2,5,6,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":70,"apply_latency_ms":70,"commit_latency_ns":70000000,"apply_latency_ns":70000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-5552649622,"num_objects":-1323,"num_object_clones":0,"num_object_copies":-5294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1323,"num_whiteouts":0,"num_read":2,"num_read_kb":27,"num_write":1384,"num_write_kb":627,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003108"},"pg_stats":[{"pgid":"2.1f","version":"28'36","reported_seq":80,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:26:05.796187+0000","last_change":"2024-04-24T23:15:35.964646+0000","last_active":"2024-04-24T23:26:05.796187+0000","last_peered":"2024-04-24T23:26:05.796187+0000","last_clean":"2024-04-24T23:26:05.796187+0000","last_became_active":"2024-04-24T23:15:35.964357+0000","last_became_peered":"2024-04-24T23:15:35.964357+0000","last_unstale":"2024-04-24T23:26:05.796187+0000","last_undegraded":"2024-04-24T23:26:05.796187+0000","last_fullsized":"2024-04-24T23:26:05.796187+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:01:17.912993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194976,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":7,"num_read_kb":4102,"num_write":33,"num_write_kb":4110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"27'3","reported_seq":40,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:25:42.719794+0000","last_change":"2024-04-24T23:15:36.989772+0000","last_active":"2024-04-24T23:25:42.719794+0000","last_peered":"2024-04-24T23:25:42.719794+0000","last_clean":"2024-04-24T23:25:42.719794+0000","last_became_active":"2024-04-24T23:15:36.989543+0000","last_became_peered":"2024-04-24T23:15:36.989543+0000","last_unstale":"2024-04-24T23:25:42.719794+0000","last_undegraded":"2024-04-24T23:25:42.719794+0000","last_fullsized":"2024-04-24T23:25:42.719794+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:37:16.737354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"30'4701","reported_seq":4795,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:37.613675+0000","last_change":"2024-04-24T23:16:18.300096+0000","last_active":"2024-04-24T23:27:37.613675+0000","last_peered":"2024-04-24T23:27:37.613675+0000","last_clean":"2024-04-24T23:27:37.613675+0000","last_became_active":"2024-04-24T23:16:18.299217+0000","last_became_peered":"2024-04-24T23:16:18.299217+0000","last_unstale":"2024-04-24T23:27:37.613675+0000","last_undegraded":"2024-04-24T23:27:37.613675+0000","last_fullsized":"2024-04-24T23:27:37.613675+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":4701,"log_dups_size":0,"ondisk_log_size":4701,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:08:19.426862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":255971335,"num_objects":71,"num_object_clones":0,"num_object_copies":284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71,"num_whiteouts":0,"num_read":70,"num_read_kb":0,"num_write":69516,"num_write_kb":969009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4,1],"acting":[0,6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"28'8","reported_seq":48,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:25:42.720012+0000","last_change":"2024-04-24T23:15:35.946083+0000","last_active":"2024-04-24T23:25:42.720012+0000","last_peered":"2024-04-24T23:25:42.720012+0000","last_clean":"2024-04-24T23:25:42.720012+0000","last_became_active":"2024-04-24T23:15:35.945227+0000","last_became_peered":"2024-04-24T23:15:35.945227+0000","last_unstale":"2024-04-24T23:25:42.720012+0000","last_undegraded":"2024-04-24T23:25:42.720012+0000","last_fullsized":"2024-04-24T23:25:42.720012+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:05:17.727725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":4096,"num_write":7,"num_write_kb":4105,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1f","version":"27'6","reported_seq":43,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:25:42.719955+0000","last_change":"2024-04-24T23:15:36.998245+0000","last_active":"2024-04-24T23:25:42.719955+0000","last_peered":"2024-04-24T23:25:42.719955+0000","last_clean":"2024-04-24T23:25:42.719955+0000","last_became_active":"2024-04-24T23:15:36.998021+0000","last_became_peered":"2024-04-24T23:15:36.998021+0000","last_unstale":"2024-04-24T23:25:42.719955+0000","last_undegraded":"2024-04-24T23:25:42.719955+0000","last_fullsized":"2024-04-24T23:25:42.719955+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:03:42.359117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"30'4538","reported_seq":4643,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:39.372938+0000","last_change":"2024-04-24T23:15:41.008313+0000","last_active":"2024-04-24T23:27:39.372938+0000","last_peered":"2024-04-24T23:27:39.372938+0000","last_clean":"2024-04-24T23:27:39.372938+0000","last_became_active":"2024-04-24T23:15:41.007908+0000","last_became_peered":"2024-04-24T23:15:41.007908+0000","last_unstale":"2024-04-24T23:27:39.372938+0000","last_undegraded":"2024-04-24T23:27:39.372938+0000","last_fullsized":"2024-04-24T23:27:39.372938+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":4538,"log_dups_size":0,"ondisk_log_size":4538,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:02:11.902575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":176186537,"num_objects":46,"num_object_clones":0,"num_object_copies":184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":75,"num_read_kb":0,"num_write":66573,"num_write_kb":930809,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4,6],"acting":[2,7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1d","version":"30'40","reported_seq":92,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:23.196288+0000","last_change":"2024-04-24T23:15:35.968312+0000","last_active":"2024-04-24T23:27:23.196288+0000","last_peered":"2024-04-24T23:27:23.196288+0000","last_clean":"2024-04-24T23:27:23.196288+0000","last_became_active":"2024-04-24T23:15:35.967873+0000","last_became_peered":"2024-04-24T23:15:35.967873+0000","last_unstale":"2024-04-24T23:27:23.196288+0000","last_undegraded":"2024-04-24T23:27:23.196288+0000","last_fullsized":"2024-04-24T23:27:23.196288+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:21:40.116977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":38,"num_read_kb":8226,"num_write":37,"num_write_kb":8205,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"27'7","reported_seq":43,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:25:42.821075+0000","last_change":"2024-04-24T23:15:36.997361+0000","last_active":"2024-04-24T23:25:42.821075+0000","last_peered":"2024-04-24T23:25:42.821075+0000","last_clean":"2024-04-24T23:25:42.821075+0000","last_became_active":"2024-04-24T23:15:36.997083+0000","last_became_peered":"2024-04-24T23:15:36.997083+0000","last_unstale":"2024-04-24T23:25:42.821075+0000","last_undegraded":"2024-04-24T23:25:42.821075+0000","last_fullsized":"2024-04-24T23:25:42.821075+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:47:59.334093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"30'4235","reported_seq":4335,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:39.416539+0000","last_change":"2024-04-24T23:15:41.025153+0000","last_active":"2024-04-24T23:27:39.416539+0000","last_peered":"2024-04-24T23:27:39.416539+0000","last_clean":"2024-04-24T23:27:39.416539+0000","last_became_active":"2024-04-24T23:15:41.024739+0000","last_became_peered":"2024-04-24T23:15:41.024739+0000","last_unstale":"2024-04-24T23:27:39.416539+0000","last_undegraded":"2024-04-24T23:27:39.416539+0000","last_fullsized":"2024-04-24T23:27:39.416539+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":4235,"log_dups_size":0,"ondisk_log_size":4235,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:31:00.160213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184565702,"num_objects":54,"num_object_clones":0,"num_object_copies":216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":70,"num_read_kb":0,"num_write":62343,"num_write_kb":877947,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3,6],"acting":[2,5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1c","version":"30'17","reported_seq":63,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:36.086972+0000","last_change":"2024-04-24T23:15:35.967338+0000","last_active":"2024-04-24T23:27:36.086972+0000","last_peered":"2024-04-24T23:27:36.086972+0000","last_clean":"2024-04-24T23:27:36.086972+0000","last_became_active":"2024-04-24T23:15:35.966951+0000","last_became_peered":"2024-04-24T23:15:35.966951+0000","last_unstale":"2024-04-24T23:27:36.086972+0000","last_undegraded":"2024-04-24T23:27:36.086972+0000","last_fullsized":"2024-04-24T23:27:36.086972+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:08:05.248330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":232445,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":21,"num_read_kb":20,"num_write":17,"num_write_kb":234,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1d","version":"30'7","reported_seq":43,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:26.779216+0000","last_change":"2024-04-24T23:15:36.998072+0000","last_active":"2024-04-24T23:27:26.779216+0000","last_peered":"2024-04-24T23:27:26.779216+0000","last_clean":"2024-04-24T23:27:26.779216+0000","last_became_active":"2024-04-24T23:15:36.997735+0000","last_became_peered":"2024-04-24T23:15:36.997735+0000","last_unstale":"2024-04-24T23:27:26.779216+0000","last_undegraded":"2024-04-24T23:27:26.779216+0000","last_fullsized":"2024-04-24T23:27:26.779216+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:22:06.869644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"30'3724","reported_seq":3813,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:39.072670+0000","last_change":"2024-04-24T23:15:41.008895+0000","last_active":"2024-04-24T23:27:39.072670+0000","last_peered":"2024-04-24T23:27:39.072670+0000","last_clean":"2024-04-24T23:27:39.072670+0000","last_became_active":"2024-04-24T23:15:41.006199+0000","last_became_peered":"2024-04-24T23:15:41.006199+0000","last_unstale":"2024-04-24T23:27:39.072670+0000","last_undegraded":"2024-04-24T23:27:39.072670+0000","last_fullsized":"2024-04-24T23:27:39.072670+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":3724,"log_dups_size":0,"ondisk_log_size":3724,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:13:34.583602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184573346,"num_objects":48,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":59,"num_read_kb":0,"num_write":54757,"num_write_kb":774507,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5,7],"acting":[2,6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"30'8","reported_seq":45,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:26:34.900000+0000","last_change":"2024-04-24T23:15:37.004074+0000","last_active":"2024-04-24T23:26:34.900000+0000","last_peered":"2024-04-24T23:26:34.900000+0000","last_clean":"2024-04-24T23:26:34.900000+0000","last_became_active":"2024-04-24T23:15:37.003544+0000","last_became_peered":"2024-04-24T23:15:37.003544+0000","last_unstale":"2024-04-24T23:26:34.900000+0000","last_undegraded":"2024-04-24T23:26:34.900000+0000","last_fullsized":"2024-04-24T23:26:34.900000+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:38:28.024702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"30'1458","reported_seq":1502,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:26.160448+0000","last_change":"2024-04-24T23:15:35.963000+0000","last_active":"2024-04-24T23:27:26.160448+0000","last_peered":"2024-04-24T23:27:26.160448+0000","last_clean":"2024-04-24T23:27:26.160448+0000","last_became_active":"2024-04-24T23:15:35.961960+0000","last_became_peered":"2024-04-24T23:15:35.961960+0000","last_unstale":"2024-04-24T23:27:26.160448+0000","last_undegraded":"2024-04-24T23:27:26.160448+0000","last_fullsized":"2024-04-24T23:27:26.160448+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":1458,"log_dups_size":0,"ondisk_log_size":1458,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:56:26.651193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":10,"num_read_kb":8200,"num_write":1452,"num_write_kb":5429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"30'4317","reported_seq":4413,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:36.092914+0000","last_change":"2024-04-24T23:15:41.022481+0000","last_active":"2024-04-24T23:27:36.092914+0000","last_peered":"2024-04-24T23:27:36.092914+0000","last_clean":"2024-04-24T23:27:36.092914+0000","last_became_active":"2024-04-24T23:15:41.020034+0000","last_became_peered":"2024-04-24T23:15:41.020034+0000","last_unstale":"2024-04-24T23:27:36.092914+0000","last_undegraded":"2024-04-24T23:27:36.092914+0000","last_fullsized":"2024-04-24T23:27:36.092914+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":4317,"log_dups_size":0,"ondisk_log_size":4317,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:27:25.337503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":222443060,"num_objects":60,"num_object_clones":0,"num_object_copies":240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":65,"num_read_kb":0,"num_write":63602,"num_write_kb":904417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,5],"acting":[3,1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"30'6","reported_seq":43,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:26:52.416421+0000","last_change":"2024-04-24T23:15:36.972020+0000","last_active":"2024-04-24T23:26:52.416421+0000","last_peered":"2024-04-24T23:26:52.416421+0000","last_clean":"2024-04-24T23:26:52.416421+0000","last_became_active":"2024-04-24T23:15:36.971583+0000","last_became_peered":"2024-04-24T23:15:36.971583+0000","last_unstale":"2024-04-24T23:26:52.416421+0000","last_undegraded":"2024-04-24T23:26:52.416421+0000","last_fullsized":"2024-04-24T23:26:52.416421+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:29:05.568608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"30'7","reported_seq":47,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:26:31.416252+0000","last_change":"2024-04-24T23:15:35.945916+0000","last_active":"2024-04-24T23:26:31.416252+0000","last_peered":"2024-04-24T23:26:31.416252+0000","last_clean":"2024-04-24T23:26:31.416252+0000","last_became_active":"2024-04-24T23:15:35.945610+0000","last_became_peered":"2024-04-24T23:15:35.945610+0000","last_unstale":"2024-04-24T23:26:31.416252+0000","last_undegraded":"2024-04-24T23:26:31.416252+0000","last_fullsized":"2024-04-24T23:26:31.416252+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:57:05.976074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"30'4388","reported_seq":4489,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:37.625508+0000","last_change":"2024-04-24T23:15:41.030165+0000","last_active":"2024-04-24T23:27:37.625508+0000","last_peered":"2024-04-24T23:27:37.625508+0000","last_clean":"2024-04-24T23:27:37.625508+0000","last_became_active":"2024-04-24T23:15:41.029685+0000","last_became_peered":"2024-04-24T23:15:41.029685+0000","last_unstale":"2024-04-24T23:27:37.625508+0000","last_undegraded":"2024-04-24T23:27:37.625508+0000","last_fullsized":"2024-04-24T23:27:37.625508+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":4388,"log_dups_size":0,"ondisk_log_size":4388,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:20:43.498190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":171984928,"num_objects":49,"num_object_clones":0,"num_object_copies":196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49,"num_whiteouts":0,"num_read":70,"num_read_kb":0,"num_write":64497,"num_write_kb":914755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6,1],"acting":[0,4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"30'8","reported_seq":45,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:26:30.926375+0000","last_change":"2024-04-24T23:15:36.965949+0000","last_active":"2024-04-24T23:26:30.926375+0000","last_peered":"2024-04-24T23:26:30.926375+0000","last_clean":"2024-04-24T23:26:30.926375+0000","last_became_active":"2024-04-24T23:15:36.965658+0000","last_became_peered":"2024-04-24T23:15:36.965658+0000","last_unstale":"2024-04-24T23:26:30.926375+0000","last_undegraded":"2024-04-24T23:26:30.926375+0000","last_fullsized":"2024-04-24T23:26:30.926375+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:38:28.024702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"28'8","reported_seq":54,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:25:42.710930+0000","last_change":"2024-04-24T23:15:35.972165+0000","last_active":"2024-04-24T23:25:42.710930+0000","last_peered":"2024-04-24T23:25:42.710930+0000","last_clean":"2024-04-24T23:25:42.710930+0000","last_became_active":"2024-04-24T23:15:35.971972+0000","last_became_peered":"2024-04-24T23:15:35.971972+0000","last_unstale":"2024-04-24T23:25:42.710930+0000","last_undegraded":"2024-04-24T23:25:42.710930+0000","last_fullsized":"2024-04-24T23:25:42.710930+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:11:12.088698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":20,"num_read_kb":19,"num_write":11,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"30'5152","reported_seq":5264,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:35.888667+0000","last_change":"2024-04-24T23:15:41.019799+0000","last_active":"2024-04-24T23:27:35.888667+0000","last_peered":"2024-04-24T23:27:35.888667+0000","last_clean":"2024-04-24T23:27:35.888667+0000","last_became_active":"2024-04-24T23:15:41.019339+0000","last_became_peered":"2024-04-24T23:15:41.019339+0000","last_unstale":"2024-04-24T23:27:35.888667+0000","last_undegraded":"2024-04-24T23:27:35.888667+0000","last_fullsized":"2024-04-24T23:27:35.888667+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":5152,"log_dups_size":0,"ondisk_log_size":5152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:59:34.601004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":251755637,"num_objects":65,"num_object_clones":0,"num_object_copies":260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":0,"num_read":81,"num_read_kb":0,"num_write":75840,"num_write_kb":1074791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2,3],"acting":[7,4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"27'4","reported_seq":41,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:25:42.747473+0000","last_change":"2024-04-24T23:15:36.991991+0000","last_active":"2024-04-24T23:25:42.747473+0000","last_peered":"2024-04-24T23:25:42.747473+0000","last_clean":"2024-04-24T23:25:42.747473+0000","last_became_active":"2024-04-24T23:15:36.989336+0000","last_became_peered":"2024-04-24T23:15:36.989336+0000","last_unstale":"2024-04-24T23:25:42.747473+0000","last_undegraded":"2024-04-24T23:25:42.747473+0000","last_fullsized":"2024-04-24T23:25:42.747473+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:25:56.038903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.8","version":"28'31","reported_seq":73,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:26:32.455095+0000","last_change":"2024-04-24T23:15:35.965443+0000","last_active":"2024-04-24T23:26:32.455095+0000","last_peered":"2024-04-24T23:26:32.455095+0000","last_clean":"2024-04-24T23:26:32.455095+0000","last_became_active":"2024-04-24T23:15:35.965043+0000","last_became_peered":"2024-04-24T23:15:35.965043+0000","last_unstale":"2024-04-24T23:26:32.455095+0000","last_undegraded":"2024-04-24T23:26:32.455095+0000","last_fullsized":"2024-04-24T23:26:32.455095+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:07:23.320042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":4102,"num_write":30,"num_write_kb":4111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"30'3783","reported_seq":3875,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:39.129224+0000","last_change":"2024-04-24T23:15:41.025131+0000","last_active":"2024-04-24T23:27:39.129224+0000","last_peered":"2024-04-24T23:27:39.129224+0000","last_clean":"2024-04-24T23:27:39.129224+0000","last_became_active":"2024-04-24T23:15:41.024731+0000","last_became_peered":"2024-04-24T23:15:41.024731+0000","last_unstale":"2024-04-24T23:27:39.129224+0000","last_undegraded":"2024-04-24T23:27:39.129224+0000","last_fullsized":"2024-04-24T23:27:39.129224+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":3783,"log_dups_size":0,"ondisk_log_size":3783,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:28:07.952854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180570789,"num_objects":53,"num_object_clones":0,"num_object_copies":212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":61,"num_read_kb":0,"num_write":55520,"num_write_kb":793811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,7],"acting":[2,1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"30'15","reported_seq":52,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:26:31.027261+0000","last_change":"2024-04-24T23:15:36.965850+0000","last_active":"2024-04-24T23:26:31.027261+0000","last_peered":"2024-04-24T23:26:31.027261+0000","last_clean":"2024-04-24T23:26:31.027261+0000","last_became_active":"2024-04-24T23:15:36.965547+0000","last_became_peered":"2024-04-24T23:15:36.965547+0000","last_unstale":"2024-04-24T23:26:31.027261+0000","last_undegraded":"2024-04-24T23:26:31.027261+0000","last_fullsized":"2024-04-24T23:26:31.027261+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:06:37.125573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"30'9","reported_seq":56,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:23.206469+0000","last_change":"2024-04-24T23:15:35.971040+0000","last_active":"2024-04-24T23:27:23.206469+0000","last_peered":"2024-04-24T23:27:23.206469+0000","last_clean":"2024-04-24T23:27:23.206469+0000","last_became_active":"2024-04-24T23:15:35.970184+0000","last_became_peered":"2024-04-24T23:15:35.970184+0000","last_unstale":"2024-04-24T23:27:23.206469+0000","last_undegraded":"2024-04-24T23:27:23.206469+0000","last_fullsized":"2024-04-24T23:27:23.206469+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:45:20.913759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":24,"num_write":11,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"30'4371","reported_seq":4476,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:38.781850+0000","last_change":"2024-04-24T23:15:41.025813+0000","last_active":"2024-04-24T23:27:38.781850+0000","last_peered":"2024-04-24T23:27:38.781850+0000","last_clean":"2024-04-24T23:27:38.781850+0000","last_became_active":"2024-04-24T23:15:41.025137+0000","last_became_peered":"2024-04-24T23:15:41.025137+0000","last_unstale":"2024-04-24T23:27:38.781850+0000","last_undegraded":"2024-04-24T23:27:38.781850+0000","last_fullsized":"2024-04-24T23:27:38.781850+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":4371,"log_dups_size":0,"ondisk_log_size":4371,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:35:17.784854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":205532220,"num_objects":52,"num_object_clones":0,"num_object_copies":208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":0,"num_read":74,"num_read_kb":0,"num_write":64361,"num_write_kb":917129,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,3,1],"acting":[2,6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"27'10","reported_seq":47,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:25:42.711648+0000","last_change":"2024-04-24T23:15:36.967476+0000","last_active":"2024-04-24T23:25:42.711648+0000","last_peered":"2024-04-24T23:25:42.711648+0000","last_clean":"2024-04-24T23:25:42.711648+0000","last_became_active":"2024-04-24T23:15:36.967098+0000","last_became_peered":"2024-04-24T23:15:36.967098+0000","last_unstale":"2024-04-24T23:25:42.711648+0000","last_undegraded":"2024-04-24T23:25:42.711648+0000","last_fullsized":"2024-04-24T23:25:42.711648+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:25:56.038903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"30'261","reported_seq":301,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:33.175189+0000","last_change":"2024-04-24T23:15:35.970630+0000","last_active":"2024-04-24T23:27:33.175189+0000","last_peered":"2024-04-24T23:27:33.175189+0000","last_clean":"2024-04-24T23:27:33.175189+0000","last_became_active":"2024-04-24T23:15:35.970430+0000","last_became_peered":"2024-04-24T23:15:35.970430+0000","last_unstale":"2024-04-24T23:27:33.175189+0000","last_undegraded":"2024-04-24T23:27:33.175189+0000","last_fullsized":"2024-04-24T23:27:33.175189+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":261,"log_dups_size":0,"ondisk_log_size":261,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:15:59.219084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":9,"num_write":559,"num_write_kb":230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"30'4712","reported_seq":4820,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:36.160259+0000","last_change":"2024-04-24T23:15:41.005022+0000","last_active":"2024-04-24T23:27:36.160259+0000","last_peered":"2024-04-24T23:27:36.160259+0000","last_clean":"2024-04-24T23:27:36.160259+0000","last_became_active":"2024-04-24T23:15:41.004648+0000","last_became_peered":"2024-04-24T23:15:41.004648+0000","last_unstale":"2024-04-24T23:27:36.160259+0000","last_undegraded":"2024-04-24T23:27:36.160259+0000","last_fullsized":"2024-04-24T23:27:36.160259+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":4712,"log_dups_size":0,"ondisk_log_size":4712,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:06:01.559760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":243285285,"num_objects":64,"num_object_clones":0,"num_object_copies":256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":77,"num_read_kb":0,"num_write":69520,"num_write_kb":1012990,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6,3],"acting":[4,1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.4","version":"30'4","reported_seq":41,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:26:06.045309+0000","last_change":"2024-04-24T23:15:36.967053+0000","last_active":"2024-04-24T23:26:06.045309+0000","last_peered":"2024-04-24T23:26:06.045309+0000","last_clean":"2024-04-24T23:26:06.045309+0000","last_became_active":"2024-04-24T23:15:36.966746+0000","last_became_peered":"2024-04-24T23:15:36.966746+0000","last_unstale":"2024-04-24T23:26:06.045309+0000","last_undegraded":"2024-04-24T23:26:06.045309+0000","last_fullsized":"2024-04-24T23:26:06.045309+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:16:08.779860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"26'4","reported_seq":46,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:25:42.781078+0000","last_change":"2024-04-24T23:15:35.965890+0000","last_active":"2024-04-24T23:25:42.781078+0000","last_peered":"2024-04-24T23:25:42.781078+0000","last_clean":"2024-04-24T23:25:42.781078+0000","last_became_active":"2024-04-24T23:15:35.965631+0000","last_became_peered":"2024-04-24T23:15:35.965631+0000","last_unstale":"2024-04-24T23:25:42.781078+0000","last_undegraded":"2024-04-24T23:25:42.781078+0000","last_fullsized":"2024-04-24T23:25:42.781078+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:17:35.650731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"30'3992","reported_seq":4083,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:35.280472+0000","last_change":"2024-04-24T23:15:41.011405+0000","last_active":"2024-04-24T23:27:35.280472+0000","last_peered":"2024-04-24T23:27:35.280472+0000","last_clean":"2024-04-24T23:27:35.280472+0000","last_became_active":"2024-04-24T23:15:41.010982+0000","last_became_peered":"2024-04-24T23:15:41.010982+0000","last_unstale":"2024-04-24T23:27:35.280472+0000","last_undegraded":"2024-04-24T23:27:35.280472+0000","last_fullsized":"2024-04-24T23:27:35.280472+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":3992,"log_dups_size":0,"ondisk_log_size":3992,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:08:00.914296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":209724275,"num_objects":54,"num_object_clones":0,"num_object_copies":216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":60,"num_read_kb":0,"num_write":58600,"num_write_kb":828032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2,7],"acting":[6,0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"30'10","reported_seq":47,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:27.778246+0000","last_change":"2024-04-24T23:15:36.975244+0000","last_active":"2024-04-24T23:27:27.778246+0000","last_peered":"2024-04-24T23:27:27.778246+0000","last_clean":"2024-04-24T23:27:27.778246+0000","last_became_active":"2024-04-24T23:15:36.974872+0000","last_became_peered":"2024-04-24T23:15:36.974872+0000","last_unstale":"2024-04-24T23:27:27.778246+0000","last_undegraded":"2024-04-24T23:27:27.778246+0000","last_fullsized":"2024-04-24T23:27:27.778246+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:45:57.018205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"27'2","reported_seq":41,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:25:42.718435+0000","last_change":"2024-04-24T23:15:35.945792+0000","last_active":"2024-04-24T23:25:42.718435+0000","last_peered":"2024-04-24T23:25:42.718435+0000","last_clean":"2024-04-24T23:25:42.718435+0000","last_became_active":"2024-04-24T23:15:35.945094+0000","last_became_peered":"2024-04-24T23:15:35.945094+0000","last_unstale":"2024-04-24T23:25:42.718435+0000","last_undegraded":"2024-04-24T23:25:42.718435+0000","last_fullsized":"2024-04-24T23:25:42.718435+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:49:12.719127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"30'4888","reported_seq":4996,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:36.715831+0000","last_change":"2024-04-24T23:15:41.022105+0000","last_active":"2024-04-24T23:27:36.715831+0000","last_peered":"2024-04-24T23:27:36.715831+0000","last_clean":"2024-04-24T23:27:36.715831+0000","last_became_active":"2024-04-24T23:15:41.021645+0000","last_became_peered":"2024-04-24T23:15:41.021645+0000","last_unstale":"2024-04-24T23:27:36.715831+0000","last_undegraded":"2024-04-24T23:27:36.715831+0000","last_fullsized":"2024-04-24T23:27:36.715831+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":4888,"log_dups_size":0,"ondisk_log_size":4888,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:03:52.974835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":297814443,"num_objects":74,"num_object_clones":0,"num_object_copies":296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74,"num_whiteouts":0,"num_read":77,"num_read_kb":0,"num_write":72362,"num_write_kb":1021329,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5,7],"acting":[6,3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.3","version":"30'8","reported_seq":45,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:31.704499+0000","last_change":"2024-04-24T23:15:36.991239+0000","last_active":"2024-04-24T23:27:31.704499+0000","last_peered":"2024-04-24T23:27:31.704499+0000","last_clean":"2024-04-24T23:27:31.704499+0000","last_became_active":"2024-04-24T23:15:36.990963+0000","last_became_peered":"2024-04-24T23:15:36.990963+0000","last_unstale":"2024-04-24T23:27:31.704499+0000","last_undegraded":"2024-04-24T23:27:31.704499+0000","last_fullsized":"2024-04-24T23:27:31.704499+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:53:28.101334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"29'42","reported_seq":91,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:26:59.824686+0000","last_change":"2024-04-24T23:15:35.957931+0000","last_active":"2024-04-24T23:26:59.824686+0000","last_peered":"2024-04-24T23:26:59.824686+0000","last_clean":"2024-04-24T23:26:59.824686+0000","last_became_active":"2024-04-24T23:15:35.957432+0000","last_became_peered":"2024-04-24T23:15:35.957432+0000","last_unstale":"2024-04-24T23:26:59.824686+0000","last_undegraded":"2024-04-24T23:26:59.824686+0000","last_fullsized":"2024-04-24T23:26:59.824686+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:11:38.773425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8389784,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":19,"num_read_kb":8206,"num_write":36,"num_write_kb":8211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.4","version":"30'4138","reported_seq":4234,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:35.795175+0000","last_change":"2024-04-24T23:15:41.007857+0000","last_active":"2024-04-24T23:27:35.795175+0000","last_peered":"2024-04-24T23:27:35.795175+0000","last_clean":"2024-04-24T23:27:35.795175+0000","last_became_active":"2024-04-24T23:15:41.007449+0000","last_became_peered":"2024-04-24T23:15:41.007449+0000","last_unstale":"2024-04-24T23:27:35.795175+0000","last_undegraded":"2024-04-24T23:27:35.795175+0000","last_fullsized":"2024-04-24T23:27:35.795175+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":4138,"log_dups_size":0,"ondisk_log_size":4138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:05:06.145440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":230763667,"num_objects":61,"num_object_clones":0,"num_object_copies":244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61,"num_whiteouts":0,"num_read":65,"num_read_kb":0,"num_write":60863,"num_write_kb":882911,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0,3],"acting":[6,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"27'11","reported_seq":48,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:25:42.717581+0000","last_change":"2024-04-24T23:15:36.970444+0000","last_active":"2024-04-24T23:25:42.717581+0000","last_peered":"2024-04-24T23:25:42.717581+0000","last_clean":"2024-04-24T23:25:42.717581+0000","last_became_active":"2024-04-24T23:15:36.970159+0000","last_became_peered":"2024-04-24T23:15:36.970159+0000","last_unstale":"2024-04-24T23:25:42.717581+0000","last_undegraded":"2024-04-24T23:25:42.717581+0000","last_fullsized":"2024-04-24T23:25:42.717581+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:37:02.956224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"30'43","reported_seq":115,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:33.304734+0000","last_change":"2024-04-24T23:15:35.945461+0000","last_active":"2024-04-24T23:27:33.304734+0000","last_peered":"2024-04-24T23:27:33.304734+0000","last_clean":"2024-04-24T23:27:33.304734+0000","last_became_active":"2024-04-24T23:15:35.945185+0000","last_became_peered":"2024-04-24T23:15:35.945185+0000","last_unstale":"2024-04-24T23:27:33.304734+0000","last_undegraded":"2024-04-24T23:27:33.304734+0000","last_fullsized":"2024-04-24T23:27:33.304734+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:37:34.175539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":94280,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":33,"num_read_kb":113,"num_write":40,"num_write_kb":118,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"30'4689","reported_seq":4794,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:35.983024+0000","last_change":"2024-04-24T23:15:41.014584+0000","last_active":"2024-04-24T23:27:35.983024+0000","last_peered":"2024-04-24T23:27:35.983024+0000","last_clean":"2024-04-24T23:27:35.983024+0000","last_became_active":"2024-04-24T23:15:41.014137+0000","last_became_peered":"2024-04-24T23:15:41.014137+0000","last_unstale":"2024-04-24T23:27:35.983024+0000","last_undegraded":"2024-04-24T23:27:35.983024+0000","last_fullsized":"2024-04-24T23:27:35.983024+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":4689,"log_dups_size":0,"ondisk_log_size":4689,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:23:50.387635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":272680432,"num_objects":72,"num_object_clones":0,"num_object_copies":288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72,"num_whiteouts":0,"num_read":74,"num_read_kb":0,"num_write":69042,"num_write_kb":969120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,7],"acting":[6,0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"30'5","reported_seq":42,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:26:06.044607+0000","last_change":"2024-04-24T23:15:36.981783+0000","last_active":"2024-04-24T23:26:06.044607+0000","last_peered":"2024-04-24T23:26:06.044607+0000","last_clean":"2024-04-24T23:26:06.044607+0000","last_became_active":"2024-04-24T23:15:36.981502+0000","last_became_peered":"2024-04-24T23:15:36.981502+0000","last_unstale":"2024-04-24T23:26:06.044607+0000","last_undegraded":"2024-04-24T23:26:06.044607+0000","last_fullsized":"2024-04-24T23:26:06.044607+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:56:08.483410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"29'34","reported_seq":75,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:25:52.069108+0000","last_change":"2024-04-24T23:15:35.960519+0000","last_active":"2024-04-24T23:25:52.069108+0000","last_peered":"2024-04-24T23:25:52.069108+0000","last_clean":"2024-04-24T23:25:52.069108+0000","last_became_active":"2024-04-24T23:15:35.960180+0000","last_became_peered":"2024-04-24T23:15:35.960180+0000","last_unstale":"2024-04-24T23:25:52.069108+0000","last_undegraded":"2024-04-24T23:25:52.069108+0000","last_fullsized":"2024-04-24T23:25:52.069108+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:50:00.199120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":8192,"num_write":28,"num_write_kb":8204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"30'3806","reported_seq":3898,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:36.034378+0000","last_change":"2024-04-24T23:15:41.012876+0000","last_active":"2024-04-24T23:27:36.034378+0000","last_peered":"2024-04-24T23:27:36.034378+0000","last_clean":"2024-04-24T23:27:36.034378+0000","last_became_active":"2024-04-24T23:15:41.012493+0000","last_became_peered":"2024-04-24T23:15:41.012493+0000","last_unstale":"2024-04-24T23:27:36.034378+0000","last_undegraded":"2024-04-24T23:27:36.034378+0000","last_fullsized":"2024-04-24T23:27:36.034378+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":3806,"log_dups_size":0,"ondisk_log_size":3806,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:21:28.287059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":218537650,"num_objects":58,"num_object_clones":0,"num_object_copies":232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58,"num_whiteouts":0,"num_read":61,"num_read_kb":307,"num_write":55909,"num_write_kb":780987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6,1],"acting":[5,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"27'4","reported_seq":41,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:25:42.712269+0000","last_change":"2024-04-24T23:15:36.981449+0000","last_active":"2024-04-24T23:25:42.712269+0000","last_peered":"2024-04-24T23:25:42.712269+0000","last_clean":"2024-04-24T23:25:42.712269+0000","last_became_active":"2024-04-24T23:15:36.980837+0000","last_became_peered":"2024-04-24T23:15:36.980837+0000","last_unstale":"2024-04-24T23:25:42.712269+0000","last_undegraded":"2024-04-24T23:25:42.712269+0000","last_fullsized":"2024-04-24T23:25:42.712269+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:15:13.513341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.3","version":"27'7","reported_seq":50,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:25:42.817265+0000","last_change":"2024-04-24T23:15:35.952947+0000","last_active":"2024-04-24T23:25:42.817265+0000","last_peered":"2024-04-24T23:25:42.817265+0000","last_clean":"2024-04-24T23:25:42.817265+0000","last_became_active":"2024-04-24T23:15:35.952655+0000","last_became_peered":"2024-04-24T23:15:35.952655+0000","last_unstale":"2024-04-24T23:25:42.817265+0000","last_undegraded":"2024-04-24T23:25:42.817265+0000","last_fullsized":"2024-04-24T23:25:42.817265+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:50:10.939396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":7,"num_read_kb":11,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":196,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:25:50.717692+0000","last_change":"2024-04-24T23:15:25.826673+0000","last_active":"2024-04-24T23:25:50.717692+0000","last_peered":"2024-04-24T23:25:50.717692+0000","last_clean":"2024-04-24T23:25:50.717692+0000","last_became_active":"2024-04-24T23:15:25.820030+0000","last_became_peered":"2024-04-24T23:15:25.820030+0000","last_unstale":"2024-04-24T23:25:50.717692+0000","last_undegraded":"2024-04-24T23:25:50.717692+0000","last_fullsized":"2024-04-24T23:25:50.717692+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:24.790928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:24.790928+0000","last_clean_scrub_stamp":"2024-04-24T23:15:24.790928+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:37:24.054332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"30'4230","reported_seq":4337,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:36.048269+0000","last_change":"2024-04-24T23:15:41.024712+0000","last_active":"2024-04-24T23:27:36.048269+0000","last_peered":"2024-04-24T23:27:36.048269+0000","last_clean":"2024-04-24T23:27:36.048269+0000","last_became_active":"2024-04-24T23:15:41.022697+0000","last_became_peered":"2024-04-24T23:15:41.022697+0000","last_unstale":"2024-04-24T23:27:36.048269+0000","last_undegraded":"2024-04-24T23:27:36.048269+0000","last_fullsized":"2024-04-24T23:27:36.048269+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":4230,"log_dups_size":0,"ondisk_log_size":4230,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:37:08.231105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":213919236,"num_objects":56,"num_object_clones":0,"num_object_copies":224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56,"num_whiteouts":0,"num_read":76,"num_read_kb":0,"num_write":62598,"num_write_kb":898105,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2,0],"acting":[3,1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.d","version":"30'8","reported_seq":45,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:28.621617+0000","last_change":"2024-04-24T23:15:37.000489+0000","last_active":"2024-04-24T23:27:28.621617+0000","last_peered":"2024-04-24T23:27:28.621617+0000","last_clean":"2024-04-24T23:27:28.621617+0000","last_became_active":"2024-04-24T23:15:37.000179+0000","last_became_peered":"2024-04-24T23:15:37.000179+0000","last_unstale":"2024-04-24T23:27:28.621617+0000","last_undegraded":"2024-04-24T23:27:28.621617+0000","last_fullsized":"2024-04-24T23:27:28.621617+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:06:37.125573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.c","version":"28'10","reported_seq":49,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:25:42.744989+0000","last_change":"2024-04-24T23:15:35.966020+0000","last_active":"2024-04-24T23:25:42.744989+0000","last_peered":"2024-04-24T23:25:42.744989+0000","last_clean":"2024-04-24T23:25:42.744989+0000","last_became_active":"2024-04-24T23:15:35.965758+0000","last_became_peered":"2024-04-24T23:15:35.965758+0000","last_unstale":"2024-04-24T23:25:42.744989+0000","last_undegraded":"2024-04-24T23:25:42.744989+0000","last_fullsized":"2024-04-24T23:25:42.744989+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:39:21.233461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2157,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"30'4343","reported_seq":4444,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:39.335432+0000","last_change":"2024-04-24T23:15:41.024489+0000","last_active":"2024-04-24T23:27:39.335432+0000","last_peered":"2024-04-24T23:27:39.335432+0000","last_clean":"2024-04-24T23:27:39.335432+0000","last_became_active":"2024-04-24T23:15:41.024041+0000","last_became_peered":"2024-04-24T23:15:41.024041+0000","last_unstale":"2024-04-24T23:27:39.335432+0000","last_undegraded":"2024-04-24T23:27:39.335432+0000","last_fullsized":"2024-04-24T23:27:39.335432+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":4343,"log_dups_size":0,"ondisk_log_size":4343,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:27:18.197003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":159390415,"num_objects":41,"num_object_clones":0,"num_object_copies":164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":70,"num_read_kb":0,"num_write":63845,"num_write_kb":912579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7,4],"acting":[2,3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"27'7","reported_seq":44,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:25:42.773524+0000","last_change":"2024-04-24T23:15:36.979887+0000","last_active":"2024-04-24T23:25:42.773524+0000","last_peered":"2024-04-24T23:25:42.773524+0000","last_clean":"2024-04-24T23:25:42.773524+0000","last_became_active":"2024-04-24T23:15:36.979611+0000","last_became_peered":"2024-04-24T23:15:36.979611+0000","last_unstale":"2024-04-24T23:25:42.773524+0000","last_undegraded":"2024-04-24T23:25:42.773524+0000","last_fullsized":"2024-04-24T23:25:42.773524+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:07:18.261077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"28'25","reported_seq":66,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:26:56.337078+0000","last_change":"2024-04-24T23:15:35.965192+0000","last_active":"2024-04-24T23:26:56.337078+0000","last_peered":"2024-04-24T23:26:56.337078+0000","last_clean":"2024-04-24T23:26:56.337078+0000","last_became_active":"2024-04-24T23:15:35.964873+0000","last_became_peered":"2024-04-24T23:15:35.964873+0000","last_unstale":"2024-04-24T23:26:56.337078+0000","last_undegraded":"2024-04-24T23:26:56.337078+0000","last_fullsized":"2024-04-24T23:26:56.337078+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:14:27.884944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194374,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":4097,"num_write":20,"num_write_kb":4108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"30'4075","reported_seq":4177,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:36.049480+0000","last_change":"2024-04-24T23:15:41.004002+0000","last_active":"2024-04-24T23:27:36.049480+0000","last_peered":"2024-04-24T23:27:36.049480+0000","last_clean":"2024-04-24T23:27:36.049480+0000","last_became_active":"2024-04-24T23:15:41.003335+0000","last_became_peered":"2024-04-24T23:15:41.003335+0000","last_unstale":"2024-04-24T23:27:36.049480+0000","last_undegraded":"2024-04-24T23:27:36.049480+0000","last_fullsized":"2024-04-24T23:27:36.049480+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":4075,"log_dups_size":0,"ondisk_log_size":4075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:49:36.206903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":209740292,"num_objects":59,"num_object_clones":0,"num_object_copies":236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59,"num_whiteouts":0,"num_read":71,"num_read_kb":0,"num_write":60093,"num_write_kb":843409,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1,3],"acting":[5,4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"27'8","reported_seq":45,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:25:42.818254+0000","last_change":"2024-04-24T23:15:37.002854+0000","last_active":"2024-04-24T23:25:42.818254+0000","last_peered":"2024-04-24T23:25:42.818254+0000","last_clean":"2024-04-24T23:25:42.818254+0000","last_became_active":"2024-04-24T23:15:37.002583+0000","last_became_peered":"2024-04-24T23:15:37.002583+0000","last_unstale":"2024-04-24T23:25:42.818254+0000","last_undegraded":"2024-04-24T23:25:42.818254+0000","last_fullsized":"2024-04-24T23:25:42.818254+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:27:04.251547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"30'166","reported_seq":213,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:28.158437+0000","last_change":"2024-04-24T23:15:35.948752+0000","last_active":"2024-04-24T23:27:28.158437+0000","last_peered":"2024-04-24T23:27:28.158437+0000","last_clean":"2024-04-24T23:27:28.158437+0000","last_became_active":"2024-04-24T23:15:35.948438+0000","last_became_peered":"2024-04-24T23:15:35.948438+0000","last_unstale":"2024-04-24T23:27:28.158437+0000","last_undegraded":"2024-04-24T23:27:28.158437+0000","last_fullsized":"2024-04-24T23:27:28.158437+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":166,"log_dups_size":0,"ondisk_log_size":166,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:45:56.006278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":21,"num_read_kb":2419,"num_write":164,"num_write_kb":4172,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.8","version":"30'4085","reported_seq":4184,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:36.007089+0000","last_change":"2024-04-24T23:15:41.017524+0000","last_active":"2024-04-24T23:27:36.007089+0000","last_peered":"2024-04-24T23:27:36.007089+0000","last_clean":"2024-04-24T23:27:36.007089+0000","last_became_active":"2024-04-24T23:15:41.017092+0000","last_became_peered":"2024-04-24T23:15:41.017092+0000","last_unstale":"2024-04-24T23:27:36.007089+0000","last_undegraded":"2024-04-24T23:27:36.007089+0000","last_fullsized":"2024-04-24T23:27:36.007089+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":4085,"log_dups_size":0,"ondisk_log_size":4085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:04:53.235038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":222316892,"num_objects":57,"num_object_clones":0,"num_object_copies":228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57,"num_whiteouts":0,"num_read":68,"num_read_kb":0,"num_write":60373,"num_write_kb":856855,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"27'4","reported_seq":41,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:25:42.735229+0000","last_change":"2024-04-24T23:15:36.990441+0000","last_active":"2024-04-24T23:25:42.735229+0000","last_peered":"2024-04-24T23:25:42.735229+0000","last_clean":"2024-04-24T23:25:42.735229+0000","last_became_active":"2024-04-24T23:15:36.989863+0000","last_became_peered":"2024-04-24T23:15:36.989863+0000","last_unstale":"2024-04-24T23:25:42.735229+0000","last_undegraded":"2024-04-24T23:25:42.735229+0000","last_fullsized":"2024-04-24T23:25:42.735229+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:50:07.975347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.f","version":"30'6","reported_seq":46,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:01.577469+0000","last_change":"2024-04-24T23:15:35.969317+0000","last_active":"2024-04-24T23:27:01.577469+0000","last_peered":"2024-04-24T23:27:01.577469+0000","last_clean":"2024-04-24T23:27:01.577469+0000","last_became_active":"2024-04-24T23:15:35.968994+0000","last_became_peered":"2024-04-24T23:15:35.968994+0000","last_unstale":"2024-04-24T23:27:01.577469+0000","last_undegraded":"2024-04-24T23:27:01.577469+0000","last_fullsized":"2024-04-24T23:27:01.577469+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:51:57.100772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"30'4118","reported_seq":4222,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:37.895480+0000","last_change":"2024-04-24T23:15:41.010103+0000","last_active":"2024-04-24T23:27:37.895480+0000","last_peered":"2024-04-24T23:27:37.895480+0000","last_clean":"2024-04-24T23:27:37.895480+0000","last_became_active":"2024-04-24T23:15:41.009328+0000","last_became_peered":"2024-04-24T23:15:41.009328+0000","last_unstale":"2024-04-24T23:27:37.895480+0000","last_undegraded":"2024-04-24T23:27:37.895480+0000","last_fullsized":"2024-04-24T23:27:37.895480+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":4118,"log_dups_size":0,"ondisk_log_size":4118,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:46:49.579501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":193017293,"num_objects":50,"num_object_clones":0,"num_object_copies":200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":73,"num_read_kb":0,"num_write":60557,"num_write_kb":866216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,2],"acting":[1,5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.11","version":"27'10","reported_seq":47,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:25:42.747653+0000","last_change":"2024-04-24T23:15:36.996765+0000","last_active":"2024-04-24T23:25:42.747653+0000","last_peered":"2024-04-24T23:25:42.747653+0000","last_clean":"2024-04-24T23:25:42.747653+0000","last_became_active":"2024-04-24T23:15:36.995801+0000","last_became_peered":"2024-04-24T23:15:36.995801+0000","last_unstale":"2024-04-24T23:25:42.747653+0000","last_undegraded":"2024-04-24T23:25:42.747653+0000","last_fullsized":"2024-04-24T23:25:42.747653+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:25:19.421667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"28'19","reported_seq":66,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:23.188825+0000","last_change":"2024-04-24T23:15:35.944023+0000","last_active":"2024-04-24T23:27:23.188825+0000","last_peered":"2024-04-24T23:27:23.188825+0000","last_clean":"2024-04-24T23:27:23.188825+0000","last_became_active":"2024-04-24T23:15:35.943710+0000","last_became_peered":"2024-04-24T23:15:35.943710+0000","last_unstale":"2024-04-24T23:27:23.188825+0000","last_undegraded":"2024-04-24T23:27:23.188825+0000","last_fullsized":"2024-04-24T23:27:23.188825+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:46:48.090958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"30'4617","reported_seq":4717,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:36.027519+0000","last_change":"2024-04-24T23:15:41.013027+0000","last_active":"2024-04-24T23:27:36.027519+0000","last_peered":"2024-04-24T23:27:36.027519+0000","last_clean":"2024-04-24T23:27:36.027519+0000","last_became_active":"2024-04-24T23:15:41.012587+0000","last_became_peered":"2024-04-24T23:15:41.012587+0000","last_unstale":"2024-04-24T23:27:36.027519+0000","last_undegraded":"2024-04-24T23:27:36.027519+0000","last_fullsized":"2024-04-24T23:27:36.027519+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":4617,"log_dups_size":0,"ondisk_log_size":4617,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:27:04.276246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":235134753,"num_objects":67,"num_object_clones":0,"num_object_copies":268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67,"num_whiteouts":0,"num_read":70,"num_read_kb":0,"num_write":67760,"num_write_kb":966739,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1,6],"acting":[5,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"30'7","reported_seq":44,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:17.987795+0000","last_change":"2024-04-24T23:15:36.979702+0000","last_active":"2024-04-24T23:27:17.987795+0000","last_peered":"2024-04-24T23:27:17.987795+0000","last_clean":"2024-04-24T23:27:17.987795+0000","last_became_active":"2024-04-24T23:15:36.979356+0000","last_became_peered":"2024-04-24T23:15:36.979356+0000","last_unstale":"2024-04-24T23:27:17.987795+0000","last_undegraded":"2024-04-24T23:27:17.987795+0000","last_fullsized":"2024-04-24T23:27:17.987795+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:14:24.309192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.11","version":"28'12","reported_seq":51,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:25:42.827287+0000","last_change":"2024-04-24T23:15:35.973026+0000","last_active":"2024-04-24T23:25:42.827287+0000","last_peered":"2024-04-24T23:25:42.827287+0000","last_clean":"2024-04-24T23:25:42.827287+0000","last_became_active":"2024-04-24T23:15:35.972655+0000","last_became_peered":"2024-04-24T23:15:35.972655+0000","last_unstale":"2024-04-24T23:25:42.827287+0000","last_undegraded":"2024-04-24T23:25:42.827287+0000","last_fullsized":"2024-04-24T23:25:42.827287+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:25:34.352908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1575,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"30'4460","reported_seq":4560,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:35.552454+0000","last_change":"2024-04-24T23:15:41.004221+0000","last_active":"2024-04-24T23:27:35.552454+0000","last_peered":"2024-04-24T23:27:35.552454+0000","last_clean":"2024-04-24T23:27:35.552454+0000","last_became_active":"2024-04-24T23:15:41.003790+0000","last_became_peered":"2024-04-24T23:15:41.003790+0000","last_unstale":"2024-04-24T23:27:35.552454+0000","last_undegraded":"2024-04-24T23:27:35.552454+0000","last_fullsized":"2024-04-24T23:27:35.552454+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":4460,"log_dups_size":0,"ondisk_log_size":4460,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:28:58.866024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":239134950,"num_objects":63,"num_object_clones":0,"num_object_copies":252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63,"num_whiteouts":0,"num_read":70,"num_read_kb":0,"num_write":65414,"num_write_kb":956853,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6,4],"acting":[5,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"27'10","reported_seq":47,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:25:42.816663+0000","last_change":"2024-04-24T23:15:36.997389+0000","last_active":"2024-04-24T23:25:42.816663+0000","last_peered":"2024-04-24T23:25:42.816663+0000","last_clean":"2024-04-24T23:25:42.816663+0000","last_became_active":"2024-04-24T23:15:36.995923+0000","last_became_peered":"2024-04-24T23:15:36.995923+0000","last_unstale":"2024-04-24T23:25:42.816663+0000","last_undegraded":"2024-04-24T23:25:42.816663+0000","last_fullsized":"2024-04-24T23:25:42.816663+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:55:47.134320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"29'7","reported_seq":50,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:25:43.845951+0000","last_change":"2024-04-24T23:15:35.957715+0000","last_active":"2024-04-24T23:25:43.845951+0000","last_peered":"2024-04-24T23:25:43.845951+0000","last_clean":"2024-04-24T23:25:43.845951+0000","last_became_active":"2024-04-24T23:15:35.957088+0000","last_became_peered":"2024-04-24T23:15:35.957088+0000","last_unstale":"2024-04-24T23:25:43.845951+0000","last_undegraded":"2024-04-24T23:25:43.845951+0000","last_fullsized":"2024-04-24T23:25:43.845951+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:30:34.483669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":7,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"30'4618","reported_seq":4720,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:35.891055+0000","last_change":"2024-04-24T23:15:41.008165+0000","last_active":"2024-04-24T23:27:35.891055+0000","last_peered":"2024-04-24T23:27:35.891055+0000","last_clean":"2024-04-24T23:27:35.891055+0000","last_became_active":"2024-04-24T23:15:41.007555+0000","last_became_peered":"2024-04-24T23:15:41.007555+0000","last_unstale":"2024-04-24T23:27:35.891055+0000","last_undegraded":"2024-04-24T23:27:35.891055+0000","last_fullsized":"2024-04-24T23:27:35.891055+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":4618,"log_dups_size":0,"ondisk_log_size":4618,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:18:08.195747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":226724555,"num_objects":60,"num_object_clones":0,"num_object_copies":240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":71,"num_read_kb":0,"num_write":67988,"num_write_kb":979211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0,7],"acting":[6,4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.12","version":"30'11","reported_seq":48,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:21.805159+0000","last_change":"2024-04-24T23:15:36.991382+0000","last_active":"2024-04-24T23:27:21.805159+0000","last_peered":"2024-04-24T23:27:21.805159+0000","last_clean":"2024-04-24T23:27:21.805159+0000","last_became_active":"2024-04-24T23:15:36.991115+0000","last_became_peered":"2024-04-24T23:15:36.991115+0000","last_unstale":"2024-04-24T23:27:21.805159+0000","last_undegraded":"2024-04-24T23:27:21.805159+0000","last_fullsized":"2024-04-24T23:27:21.805159+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:21:51.863947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"28'40","reported_seq":90,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:22.419341+0000","last_change":"2024-04-24T23:15:35.964245+0000","last_active":"2024-04-24T23:27:22.419341+0000","last_peered":"2024-04-24T23:27:22.419341+0000","last_clean":"2024-04-24T23:27:22.419341+0000","last_became_active":"2024-04-24T23:15:35.963907+0000","last_became_peered":"2024-04-24T23:15:35.963907+0000","last_unstale":"2024-04-24T23:27:22.419341+0000","last_undegraded":"2024-04-24T23:27:22.419341+0000","last_fullsized":"2024-04-24T23:27:22.419341+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:15:00.913899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388676,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":26,"num_read_kb":12314,"num_write":42,"num_write_kb":8216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"30'4224","reported_seq":4327,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:36.153694+0000","last_change":"2024-04-24T23:15:41.021981+0000","last_active":"2024-04-24T23:27:36.153694+0000","last_peered":"2024-04-24T23:27:36.153694+0000","last_clean":"2024-04-24T23:27:36.153694+0000","last_became_active":"2024-04-24T23:15:41.019193+0000","last_became_peered":"2024-04-24T23:15:41.019193+0000","last_unstale":"2024-04-24T23:27:36.153694+0000","last_undegraded":"2024-04-24T23:27:36.153694+0000","last_fullsized":"2024-04-24T23:27:36.153694+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":4224,"log_dups_size":0,"ondisk_log_size":4224,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:04:39.828502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184549376,"num_objects":44,"num_object_clones":0,"num_object_copies":176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":0,"num_read":73,"num_read_kb":0,"num_write":62280,"num_write_kb":864711,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5,6],"acting":[3,0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"30'10","reported_seq":47,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:37.674563+0000","last_change":"2024-04-24T23:15:36.986833+0000","last_active":"2024-04-24T23:27:37.674563+0000","last_peered":"2024-04-24T23:27:37.674563+0000","last_clean":"2024-04-24T23:27:37.674563+0000","last_became_active":"2024-04-24T23:15:36.986484+0000","last_became_peered":"2024-04-24T23:15:36.986484+0000","last_unstale":"2024-04-24T23:27:37.674563+0000","last_undegraded":"2024-04-24T23:27:37.674563+0000","last_fullsized":"2024-04-24T23:27:37.674563+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:12:32.245267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.14","version":"30'236","reported_seq":286,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:29.519952+0000","last_change":"2024-04-24T23:15:35.957124+0000","last_active":"2024-04-24T23:27:29.519952+0000","last_peered":"2024-04-24T23:27:29.519952+0000","last_clean":"2024-04-24T23:27:29.519952+0000","last_became_active":"2024-04-24T23:15:35.951923+0000","last_became_peered":"2024-04-24T23:15:35.951923+0000","last_unstale":"2024-04-24T23:27:29.519952+0000","last_undegraded":"2024-04-24T23:27:29.519952+0000","last_fullsized":"2024-04-24T23:27:29.519952+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":236,"log_dups_size":0,"ondisk_log_size":236,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:44:16.820640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194976,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":28,"num_read_kb":4122,"num_write":236,"num_write_kb":4244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"30'4326","reported_seq":4433,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:36.108011+0000","last_change":"2024-04-24T23:15:41.015494+0000","last_active":"2024-04-24T23:27:36.108011+0000","last_peered":"2024-04-24T23:27:36.108011+0000","last_clean":"2024-04-24T23:27:36.108011+0000","last_became_active":"2024-04-24T23:15:41.015179+0000","last_became_peered":"2024-04-24T23:15:41.015179+0000","last_unstale":"2024-04-24T23:27:36.108011+0000","last_undegraded":"2024-04-24T23:27:36.108011+0000","last_fullsized":"2024-04-24T23:27:36.108011+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":4326,"log_dups_size":0,"ondisk_log_size":4326,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:18:32.208506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":251741026,"num_objects":65,"num_object_clones":0,"num_object_copies":260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":0,"num_read":76,"num_read_kb":0,"num_write":63877,"num_write_kb":903699,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5,0],"acting":[4,3,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"30'6","reported_seq":43,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:28.524851+0000","last_change":"2024-04-24T23:15:36.999334+0000","last_active":"2024-04-24T23:27:28.524851+0000","last_peered":"2024-04-24T23:27:28.524851+0000","last_clean":"2024-04-24T23:27:28.524851+0000","last_became_active":"2024-04-24T23:15:36.995470+0000","last_became_peered":"2024-04-24T23:15:36.995470+0000","last_unstale":"2024-04-24T23:27:28.524851+0000","last_undegraded":"2024-04-24T23:27:28.524851+0000","last_fullsized":"2024-04-24T23:27:28.524851+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:19:57.973402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.15","version":"30'19","reported_seq":60,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:23.191235+0000","last_change":"2024-04-24T23:15:35.946041+0000","last_active":"2024-04-24T23:27:23.191235+0000","last_peered":"2024-04-24T23:27:23.191235+0000","last_clean":"2024-04-24T23:27:23.191235+0000","last_became_active":"2024-04-24T23:15:35.944805+0000","last_became_peered":"2024-04-24T23:15:35.944805+0000","last_unstale":"2024-04-24T23:27:23.191235+0000","last_undegraded":"2024-04-24T23:27:23.191235+0000","last_fullsized":"2024-04-24T23:27:23.191235+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:52:19.261306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":21,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"30'4379","reported_seq":4484,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:39.436460+0000","last_change":"2024-04-24T23:15:41.024392+0000","last_active":"2024-04-24T23:27:39.436460+0000","last_peered":"2024-04-24T23:27:39.436460+0000","last_clean":"2024-04-24T23:27:39.436460+0000","last_became_active":"2024-04-24T23:15:41.023951+0000","last_became_peered":"2024-04-24T23:15:41.023951+0000","last_unstale":"2024-04-24T23:27:39.436460+0000","last_undegraded":"2024-04-24T23:27:39.436460+0000","last_fullsized":"2024-04-24T23:27:39.436460+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":4379,"log_dups_size":0,"ondisk_log_size":4379,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:24:17.810117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":176268273,"num_objects":52,"num_object_clones":0,"num_object_copies":208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":0,"num_read":74,"num_read_kb":0,"num_write":64448,"num_write_kb":925608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1,7],"acting":[2,3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.17","version":"30'9","reported_seq":45,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:25:42.720085+0000","last_change":"2024-04-24T23:15:36.992243+0000","last_active":"2024-04-24T23:25:42.720085+0000","last_peered":"2024-04-24T23:25:42.720085+0000","last_clean":"2024-04-24T23:25:42.720085+0000","last_became_active":"2024-04-24T23:15:36.992002+0000","last_became_peered":"2024-04-24T23:15:36.992002+0000","last_unstale":"2024-04-24T23:25:42.720085+0000","last_undegraded":"2024-04-24T23:25:42.720085+0000","last_fullsized":"2024-04-24T23:25:42.720085+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:52:15.911009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"28'5","reported_seq":50,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:25:42.767820+0000","last_change":"2024-04-24T23:15:35.967408+0000","last_active":"2024-04-24T23:25:42.767820+0000","last_peered":"2024-04-24T23:25:42.767820+0000","last_clean":"2024-04-24T23:25:42.767820+0000","last_became_active":"2024-04-24T23:15:35.966646+0000","last_became_peered":"2024-04-24T23:15:35.966646+0000","last_unstale":"2024-04-24T23:25:42.767820+0000","last_undegraded":"2024-04-24T23:25:42.767820+0000","last_fullsized":"2024-04-24T23:25:42.767820+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:07:07.931773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":19,"num_read_kb":18,"num_write":7,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"30'4259","reported_seq":4358,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:35.920731+0000","last_change":"2024-04-24T23:15:41.021697+0000","last_active":"2024-04-24T23:27:35.920731+0000","last_peered":"2024-04-24T23:27:35.920731+0000","last_clean":"2024-04-24T23:27:35.920731+0000","last_became_active":"2024-04-24T23:15:41.021281+0000","last_became_peered":"2024-04-24T23:15:41.021281+0000","last_unstale":"2024-04-24T23:27:35.920731+0000","last_undegraded":"2024-04-24T23:27:35.920731+0000","last_fullsized":"2024-04-24T23:27:35.920731+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":4259,"log_dups_size":0,"ondisk_log_size":4259,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:59:37.306593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247532416,"num_objects":66,"num_object_clones":0,"num_object_copies":264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":68,"num_read_kb":0,"num_write":62836,"num_write_kb":881151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3,2],"acting":[7,1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"30'6","reported_seq":42,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:25:42.735229+0000","last_change":"2024-04-24T23:15:36.990988+0000","last_active":"2024-04-24T23:25:42.735229+0000","last_peered":"2024-04-24T23:25:42.735229+0000","last_clean":"2024-04-24T23:25:42.735229+0000","last_became_active":"2024-04-24T23:15:36.990711+0000","last_became_peered":"2024-04-24T23:15:36.990711+0000","last_unstale":"2024-04-24T23:25:42.735229+0000","last_undegraded":"2024-04-24T23:25:42.735229+0000","last_fullsized":"2024-04-24T23:25:42.735229+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:33:20.663776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.17","version":"30'14","reported_seq":55,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:26.156910+0000","last_change":"2024-04-24T23:15:35.957838+0000","last_active":"2024-04-24T23:27:26.156910+0000","last_peered":"2024-04-24T23:27:26.156910+0000","last_clean":"2024-04-24T23:27:26.156910+0000","last_became_active":"2024-04-24T23:15:35.956870+0000","last_became_peered":"2024-04-24T23:15:35.956870+0000","last_unstale":"2024-04-24T23:27:26.156910+0000","last_undegraded":"2024-04-24T23:27:26.156910+0000","last_fullsized":"2024-04-24T23:27:26.156910+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:56:03.889693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":13,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"30'4340","reported_seq":4435,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:36.086690+0000","last_change":"2024-04-24T23:15:41.006098+0000","last_active":"2024-04-24T23:27:36.086690+0000","last_peered":"2024-04-24T23:27:36.086690+0000","last_clean":"2024-04-24T23:27:36.086690+0000","last_became_active":"2024-04-24T23:15:41.005578+0000","last_became_peered":"2024-04-24T23:15:41.005578+0000","last_unstale":"2024-04-24T23:27:36.086690+0000","last_undegraded":"2024-04-24T23:27:36.086690+0000","last_fullsized":"2024-04-24T23:27:36.086690+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":4340,"log_dups_size":0,"ondisk_log_size":4340,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:21:12.802112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":260077201,"num_objects":68,"num_object_clones":0,"num_object_copies":272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68,"num_whiteouts":0,"num_read":64,"num_read_kb":0,"num_write":64049,"num_write_kb":908953,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3,7],"acting":[4,0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1e","version":"30'5102","reported_seq":5216,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:36.739307+0000","last_change":"2024-04-24T23:15:41.022122+0000","last_active":"2024-04-24T23:27:36.739307+0000","last_peered":"2024-04-24T23:27:36.739307+0000","last_clean":"2024-04-24T23:27:36.739307+0000","last_became_active":"2024-04-24T23:15:41.021698+0000","last_became_peered":"2024-04-24T23:15:41.021698+0000","last_unstale":"2024-04-24T23:27:36.739307+0000","last_undegraded":"2024-04-24T23:27:36.739307+0000","last_fullsized":"2024-04-24T23:27:36.739307+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":5102,"log_dups_size":0,"ondisk_log_size":5102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:13:56.730050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":314584202,"num_objects":78,"num_object_clones":0,"num_object_copies":312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":78,"num_whiteouts":0,"num_read":83,"num_read_kb":0,"num_write":75810,"num_write_kb":1069592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5,3],"acting":[4,1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"27'3","reported_seq":39,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:25:42.710767+0000","last_change":"2024-04-24T23:15:36.979816+0000","last_active":"2024-04-24T23:25:42.710767+0000","last_peered":"2024-04-24T23:25:42.710767+0000","last_clean":"2024-04-24T23:25:42.710767+0000","last_became_active":"2024-04-24T23:15:36.979553+0000","last_became_peered":"2024-04-24T23:15:36.979553+0000","last_unstale":"2024-04-24T23:25:42.710767+0000","last_undegraded":"2024-04-24T23:25:42.710767+0000","last_fullsized":"2024-04-24T23:25:42.710767+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:55:47.134320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.18","version":"29'33","reported_seq":73,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:25:53.591106+0000","last_change":"2024-04-24T23:15:35.964744+0000","last_active":"2024-04-24T23:25:53.591106+0000","last_peered":"2024-04-24T23:25:53.591106+0000","last_clean":"2024-04-24T23:25:53.591106+0000","last_became_active":"2024-04-24T23:15:35.964434+0000","last_became_peered":"2024-04-24T23:15:35.964434+0000","last_unstale":"2024-04-24T23:25:53.591106+0000","last_undegraded":"2024-04-24T23:25:53.591106+0000","last_fullsized":"2024-04-24T23:25:53.591106+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:26:32.587197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8389190,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":8192,"num_write":27,"num_write_kb":8217,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"30'4492","reported_seq":4592,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:35.983834+0000","last_change":"2024-04-24T23:15:41.022032+0000","last_active":"2024-04-24T23:27:35.983834+0000","last_peered":"2024-04-24T23:27:35.983834+0000","last_clean":"2024-04-24T23:27:35.983834+0000","last_became_active":"2024-04-24T23:15:41.021616+0000","last_became_peered":"2024-04-24T23:15:41.021616+0000","last_unstale":"2024-04-24T23:27:35.983834+0000","last_undegraded":"2024-04-24T23:27:35.983834+0000","last_fullsized":"2024-04-24T23:27:35.983834+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":4492,"log_dups_size":0,"ondisk_log_size":4492,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:08:59.242392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":239158476,"num_objects":68,"num_object_clones":0,"num_object_copies":272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68,"num_whiteouts":0,"num_read":69,"num_read_kb":0,"num_write":66209,"num_write_kb":941784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5,1],"acting":[3,2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"30'15","reported_seq":51,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:25:42.717761+0000","last_change":"2024-04-24T23:15:36.967198+0000","last_active":"2024-04-24T23:25:42.717761+0000","last_peered":"2024-04-24T23:25:42.717761+0000","last_clean":"2024-04-24T23:25:42.717761+0000","last_became_active":"2024-04-24T23:15:36.966910+0000","last_became_peered":"2024-04-24T23:15:36.966910+0000","last_unstale":"2024-04-24T23:25:42.717761+0000","last_undegraded":"2024-04-24T23:25:42.717761+0000","last_fullsized":"2024-04-24T23:25:42.717761+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:51:19.788437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"28'6","reported_seq":46,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:25:42.740651+0000","last_change":"2024-04-24T23:15:35.967169+0000","last_active":"2024-04-24T23:25:42.740651+0000","last_peered":"2024-04-24T23:25:42.740651+0000","last_clean":"2024-04-24T23:25:42.740651+0000","last_became_active":"2024-04-24T23:15:35.966939+0000","last_became_peered":"2024-04-24T23:15:35.966939+0000","last_unstale":"2024-04-24T23:25:42.740651+0000","last_undegraded":"2024-04-24T23:25:42.740651+0000","last_fullsized":"2024-04-24T23:25:42.740651+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:38:55.460569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":14,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"30'17","reported_seq":56,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:26:59.813995+0000","last_change":"2024-04-24T23:15:35.971236+0000","last_active":"2024-04-24T23:26:59.813995+0000","last_peered":"2024-04-24T23:26:59.813995+0000","last_clean":"2024-04-24T23:26:59.813995+0000","last_became_active":"2024-04-24T23:15:35.970908+0000","last_became_peered":"2024-04-24T23:15:35.970908+0000","last_unstale":"2024-04-24T23:26:59.813995+0000","last_undegraded":"2024-04-24T23:26:59.813995+0000","last_fullsized":"2024-04-24T23:26:59.813995+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:14:15.548024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":1,"num_read_kb":4096,"num_write":17,"num_write_kb":4106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"27'8","reported_seq":44,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:25:42.714394+0000","last_change":"2024-04-24T23:15:36.988156+0000","last_active":"2024-04-24T23:25:42.714394+0000","last_peered":"2024-04-24T23:25:42.714394+0000","last_clean":"2024-04-24T23:25:42.714394+0000","last_became_active":"2024-04-24T23:15:36.987896+0000","last_became_peered":"2024-04-24T23:15:36.987896+0000","last_unstale":"2024-04-24T23:25:42.714394+0000","last_undegraded":"2024-04-24T23:25:42.714394+0000","last_fullsized":"2024-04-24T23:25:42.714394+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:22:06.869644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"30'3603","reported_seq":3693,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:35.997627+0000","last_change":"2024-04-24T23:15:40.997559+0000","last_active":"2024-04-24T23:27:35.997627+0000","last_peered":"2024-04-24T23:27:35.997627+0000","last_clean":"2024-04-24T23:27:35.997627+0000","last_became_active":"2024-04-24T23:15:40.997102+0000","last_became_peered":"2024-04-24T23:15:40.997102+0000","last_unstale":"2024-04-24T23:27:35.997627+0000","last_undegraded":"2024-04-24T23:27:35.997627+0000","last_fullsized":"2024-04-24T23:27:35.997627+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":3603,"log_dups_size":0,"ondisk_log_size":3603,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:25:02.046761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":163579583,"num_objects":40,"num_object_clones":0,"num_object_copies":160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":60,"num_read_kb":0,"num_write":53429,"num_write_kb":749552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,7],"acting":[4,6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"30'1217","reported_seq":1259,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:25:51.681033+0000","last_change":"2024-04-24T23:15:35.944842+0000","last_active":"2024-04-24T23:25:51.681033+0000","last_peered":"2024-04-24T23:25:51.681033+0000","last_clean":"2024-04-24T23:25:51.681033+0000","last_became_active":"2024-04-24T23:15:35.944563+0000","last_became_peered":"2024-04-24T23:15:35.944563+0000","last_unstale":"2024-04-24T23:25:51.681033+0000","last_undegraded":"2024-04-24T23:25:51.681033+0000","last_fullsized":"2024-04-24T23:25:51.681033+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":1217,"log_dups_size":0,"ondisk_log_size":1217,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:40:58.933384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4195441,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":4,"num_read_kb":4098,"num_write":1212,"num_write_kb":5208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"27'8","reported_seq":44,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:25:42.807080+0000","last_change":"2024-04-24T23:15:36.981103+0000","last_active":"2024-04-24T23:25:42.807080+0000","last_peered":"2024-04-24T23:25:42.807080+0000","last_clean":"2024-04-24T23:25:42.807080+0000","last_became_active":"2024-04-24T23:15:36.979637+0000","last_became_peered":"2024-04-24T23:15:36.979637+0000","last_unstale":"2024-04-24T23:25:42.807080+0000","last_undegraded":"2024-04-24T23:25:42.807080+0000","last_fullsized":"2024-04-24T23:25:42.807080+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:09:39.750272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"30'5005","reported_seq":5112,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:36.669829+0000","last_change":"2024-04-24T23:15:41.017087+0000","last_active":"2024-04-24T23:27:36.669829+0000","last_peered":"2024-04-24T23:27:36.669829+0000","last_clean":"2024-04-24T23:27:36.669829+0000","last_became_active":"2024-04-24T23:15:41.016696+0000","last_became_peered":"2024-04-24T23:15:41.016696+0000","last_unstale":"2024-04-24T23:27:36.669829+0000","last_undegraded":"2024-04-24T23:27:36.669829+0000","last_fullsized":"2024-04-24T23:27:36.669829+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":5005,"log_dups_size":0,"ondisk_log_size":5005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:05:57.823447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268544512,"num_objects":71,"num_object_clones":0,"num_object_copies":284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71,"num_whiteouts":0,"num_read":77,"num_read_kb":0,"num_write":73985,"num_write_kb":1014885,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1,0],"acting":[7,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"30'9","reported_seq":46,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:31.253765+0000","last_change":"2024-04-24T23:15:36.970211+0000","last_active":"2024-04-24T23:27:31.253765+0000","last_peered":"2024-04-24T23:27:31.253765+0000","last_clean":"2024-04-24T23:27:31.253765+0000","last_became_active":"2024-04-24T23:15:36.969090+0000","last_became_peered":"2024-04-24T23:15:36.969090+0000","last_unstale":"2024-04-24T23:27:31.253765+0000","last_undegraded":"2024-04-24T23:27:31.253765+0000","last_fullsized":"2024-04-24T23:27:31.253765+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:16:00.231012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.21","version":"30'12","reported_seq":49,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:26:05.729132+0000","last_change":"2024-04-24T23:15:36.982528+0000","last_active":"2024-04-24T23:26:05.729132+0000","last_peered":"2024-04-24T23:26:05.729132+0000","last_clean":"2024-04-24T23:26:05.729132+0000","last_became_active":"2024-04-24T23:15:36.982299+0000","last_became_peered":"2024-04-24T23:15:36.982299+0000","last_unstale":"2024-04-24T23:26:05.729132+0000","last_undegraded":"2024-04-24T23:26:05.729132+0000","last_fullsized":"2024-04-24T23:26:05.729132+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:02:17.732324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.22","version":"30'10","reported_seq":47,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:28.417018+0000","last_change":"2024-04-24T23:15:36.989899+0000","last_active":"2024-04-24T23:27:28.417018+0000","last_peered":"2024-04-24T23:27:28.417018+0000","last_clean":"2024-04-24T23:27:28.417018+0000","last_became_active":"2024-04-24T23:15:36.989557+0000","last_became_peered":"2024-04-24T23:15:36.989557+0000","last_unstale":"2024-04-24T23:27:28.417018+0000","last_undegraded":"2024-04-24T23:27:28.417018+0000","last_fullsized":"2024-04-24T23:27:28.417018+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:15:13.513341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"30'9","reported_seq":46,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:26:05.334192+0000","last_change":"2024-04-24T23:15:36.969628+0000","last_active":"2024-04-24T23:26:05.334192+0000","last_peered":"2024-04-24T23:26:05.334192+0000","last_clean":"2024-04-24T23:26:05.334192+0000","last_became_active":"2024-04-24T23:15:36.969327+0000","last_became_peered":"2024-04-24T23:15:36.969327+0000","last_unstale":"2024-04-24T23:26:05.334192+0000","last_undegraded":"2024-04-24T23:26:05.334192+0000","last_fullsized":"2024-04-24T23:26:05.334192+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:08:47.021999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.24","version":"30'8","reported_seq":45,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:26:31.416932+0000","last_change":"2024-04-24T23:15:36.986343+0000","last_active":"2024-04-24T23:26:31.416932+0000","last_peered":"2024-04-24T23:26:31.416932+0000","last_clean":"2024-04-24T23:26:31.416932+0000","last_became_active":"2024-04-24T23:15:36.986026+0000","last_became_peered":"2024-04-24T23:15:36.986026+0000","last_unstale":"2024-04-24T23:26:31.416932+0000","last_undegraded":"2024-04-24T23:26:31.416932+0000","last_fullsized":"2024-04-24T23:26:31.416932+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:10:00.347826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"30'7","reported_seq":44,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:26:31.416961+0000","last_change":"2024-04-24T23:15:36.968538+0000","last_active":"2024-04-24T23:26:31.416961+0000","last_peered":"2024-04-24T23:26:31.416961+0000","last_clean":"2024-04-24T23:26:31.416961+0000","last_became_active":"2024-04-24T23:15:36.968263+0000","last_became_peered":"2024-04-24T23:15:36.968263+0000","last_unstale":"2024-04-24T23:26:31.416961+0000","last_undegraded":"2024-04-24T23:26:31.416961+0000","last_fullsized":"2024-04-24T23:26:31.416961+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:07:43.731738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.26","version":"30'5","reported_seq":42,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:26:33.160187+0000","last_change":"2024-04-24T23:15:36.991008+0000","last_active":"2024-04-24T23:26:33.160187+0000","last_peered":"2024-04-24T23:26:33.160187+0000","last_clean":"2024-04-24T23:26:33.160187+0000","last_became_active":"2024-04-24T23:15:36.990764+0000","last_became_peered":"2024-04-24T23:15:36.990764+0000","last_unstale":"2024-04-24T23:26:33.160187+0000","last_undegraded":"2024-04-24T23:26:33.160187+0000","last_fullsized":"2024-04-24T23:26:33.160187+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:50:21.865233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"30'10","reported_seq":47,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:26:31.284444+0000","last_change":"2024-04-24T23:15:36.993550+0000","last_active":"2024-04-24T23:26:31.284444+0000","last_peered":"2024-04-24T23:26:31.284444+0000","last_clean":"2024-04-24T23:26:31.284444+0000","last_became_active":"2024-04-24T23:15:36.993310+0000","last_became_peered":"2024-04-24T23:15:36.993310+0000","last_unstale":"2024-04-24T23:26:31.284444+0000","last_undegraded":"2024-04-24T23:26:31.284444+0000","last_fullsized":"2024-04-24T23:26:31.284444+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:21:20.617940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.28","version":"30'8","reported_seq":45,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:26:34.899945+0000","last_change":"2024-04-24T23:15:36.987810+0000","last_active":"2024-04-24T23:26:34.899945+0000","last_peered":"2024-04-24T23:26:34.899945+0000","last_clean":"2024-04-24T23:26:34.899945+0000","last_became_active":"2024-04-24T23:15:36.987540+0000","last_became_peered":"2024-04-24T23:15:36.987540+0000","last_unstale":"2024-04-24T23:26:34.899945+0000","last_undegraded":"2024-04-24T23:26:34.899945+0000","last_fullsized":"2024-04-24T23:26:34.899945+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:49:56.176021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.29","version":"30'13","reported_seq":50,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:31.113409+0000","last_change":"2024-04-24T23:15:36.981187+0000","last_active":"2024-04-24T23:27:31.113409+0000","last_peered":"2024-04-24T23:27:31.113409+0000","last_clean":"2024-04-24T23:27:31.113409+0000","last_became_active":"2024-04-24T23:15:36.980838+0000","last_became_peered":"2024-04-24T23:15:36.980838+0000","last_unstale":"2024-04-24T23:27:31.113409+0000","last_undegraded":"2024-04-24T23:27:31.113409+0000","last_fullsized":"2024-04-24T23:27:31.113409+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:57:41.178367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2a","version":"30'11","reported_seq":48,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:27.562545+0000","last_change":"2024-04-24T23:15:36.999238+0000","last_active":"2024-04-24T23:27:27.562545+0000","last_peered":"2024-04-24T23:27:27.562545+0000","last_clean":"2024-04-24T23:27:27.562545+0000","last_became_active":"2024-04-24T23:15:36.996245+0000","last_became_peered":"2024-04-24T23:15:36.996245+0000","last_unstale":"2024-04-24T23:27:27.562545+0000","last_undegraded":"2024-04-24T23:27:27.562545+0000","last_fullsized":"2024-04-24T23:27:27.562545+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:21:20.443640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"30'9","reported_seq":46,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:26:57.039900+0000","last_change":"2024-04-24T23:15:36.981219+0000","last_active":"2024-04-24T23:26:57.039900+0000","last_peered":"2024-04-24T23:26:57.039900+0000","last_clean":"2024-04-24T23:26:57.039900+0000","last_became_active":"2024-04-24T23:15:36.980911+0000","last_became_peered":"2024-04-24T23:15:36.980911+0000","last_unstale":"2024-04-24T23:26:57.039900+0000","last_undegraded":"2024-04-24T23:26:57.039900+0000","last_fullsized":"2024-04-24T23:26:57.039900+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:19:57.973402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2c","version":"30'7","reported_seq":44,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:02.756928+0000","last_change":"2024-04-24T23:15:36.991418+0000","last_active":"2024-04-24T23:27:02.756928+0000","last_peered":"2024-04-24T23:27:02.756928+0000","last_clean":"2024-04-24T23:27:02.756928+0000","last_became_active":"2024-04-24T23:15:36.989792+0000","last_became_peered":"2024-04-24T23:15:36.989792+0000","last_unstale":"2024-04-24T23:27:02.756928+0000","last_undegraded":"2024-04-24T23:27:02.756928+0000","last_fullsized":"2024-04-24T23:27:02.756928+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:02:53.601311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2d","version":"30'12","reported_seq":49,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:28.000118+0000","last_change":"2024-04-24T23:15:36.987810+0000","last_active":"2024-04-24T23:27:28.000118+0000","last_peered":"2024-04-24T23:27:28.000118+0000","last_clean":"2024-04-24T23:27:28.000118+0000","last_became_active":"2024-04-24T23:15:36.987494+0000","last_became_peered":"2024-04-24T23:15:36.987494+0000","last_unstale":"2024-04-24T23:27:28.000118+0000","last_undegraded":"2024-04-24T23:27:28.000118+0000","last_fullsized":"2024-04-24T23:27:28.000118+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:43:21.141172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2e","version":"30'10","reported_seq":47,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:17.418750+0000","last_change":"2024-04-24T23:15:36.986576+0000","last_active":"2024-04-24T23:27:17.418750+0000","last_peered":"2024-04-24T23:27:17.418750+0000","last_clean":"2024-04-24T23:27:17.418750+0000","last_became_active":"2024-04-24T23:15:36.986272+0000","last_became_peered":"2024-04-24T23:15:36.986272+0000","last_unstale":"2024-04-24T23:27:17.418750+0000","last_undegraded":"2024-04-24T23:27:17.418750+0000","last_fullsized":"2024-04-24T23:27:17.418750+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:55:08.894945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2f","version":"30'8","reported_seq":50,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:33.578178+0000","last_change":"2024-04-24T23:16:18.294151+0000","last_active":"2024-04-24T23:27:33.578178+0000","last_peered":"2024-04-24T23:27:33.578178+0000","last_clean":"2024-04-24T23:27:33.578178+0000","last_became_active":"2024-04-24T23:16:18.293579+0000","last_became_peered":"2024-04-24T23:16:18.293579+0000","last_unstale":"2024-04-24T23:27:33.578178+0000","last_undegraded":"2024-04-24T23:27:33.578178+0000","last_fullsized":"2024-04-24T23:27:33.578178+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:22:20.603555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.30","version":"30'9","reported_seq":46,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:26.286870+0000","last_change":"2024-04-24T23:15:36.996521+0000","last_active":"2024-04-24T23:27:26.286870+0000","last_peered":"2024-04-24T23:27:26.286870+0000","last_clean":"2024-04-24T23:27:26.286870+0000","last_became_active":"2024-04-24T23:15:36.995507+0000","last_became_peered":"2024-04-24T23:15:36.995507+0000","last_unstale":"2024-04-24T23:27:26.286870+0000","last_undegraded":"2024-04-24T23:27:26.286870+0000","last_fullsized":"2024-04-24T23:27:26.286870+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:58:50.153471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.31","version":"30'11","reported_seq":48,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:21.339459+0000","last_change":"2024-04-24T23:15:36.984994+0000","last_active":"2024-04-24T23:27:21.339459+0000","last_peered":"2024-04-24T23:27:21.339459+0000","last_clean":"2024-04-24T23:27:21.339459+0000","last_became_active":"2024-04-24T23:15:36.984741+0000","last_became_peered":"2024-04-24T23:15:36.984741+0000","last_unstale":"2024-04-24T23:27:21.339459+0000","last_undegraded":"2024-04-24T23:27:21.339459+0000","last_fullsized":"2024-04-24T23:27:21.339459+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:45:23.429836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.32","version":"30'10","reported_seq":47,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:21.805138+0000","last_change":"2024-04-24T23:15:36.966165+0000","last_active":"2024-04-24T23:27:21.805138+0000","last_peered":"2024-04-24T23:27:21.805138+0000","last_clean":"2024-04-24T23:27:21.805138+0000","last_became_active":"2024-04-24T23:15:36.965877+0000","last_became_peered":"2024-04-24T23:15:36.965877+0000","last_unstale":"2024-04-24T23:27:21.805138+0000","last_undegraded":"2024-04-24T23:27:21.805138+0000","last_fullsized":"2024-04-24T23:27:21.805138+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:30:12.390978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.33","version":"30'9","reported_seq":45,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:25:42.741941+0000","last_change":"2024-04-24T23:15:36.990667+0000","last_active":"2024-04-24T23:25:42.741941+0000","last_peered":"2024-04-24T23:25:42.741941+0000","last_clean":"2024-04-24T23:25:42.741941+0000","last_became_active":"2024-04-24T23:15:36.990383+0000","last_became_peered":"2024-04-24T23:15:36.990383+0000","last_unstale":"2024-04-24T23:25:42.741941+0000","last_undegraded":"2024-04-24T23:25:42.741941+0000","last_fullsized":"2024-04-24T23:25:42.741941+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:36:43.850046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"27'6","reported_seq":43,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:25:42.773242+0000","last_change":"2024-04-24T23:15:36.991686+0000","last_active":"2024-04-24T23:25:42.773242+0000","last_peered":"2024-04-24T23:25:42.773242+0000","last_clean":"2024-04-24T23:25:42.773242+0000","last_became_active":"2024-04-24T23:15:36.990155+0000","last_became_peered":"2024-04-24T23:15:36.990155+0000","last_unstale":"2024-04-24T23:25:42.773242+0000","last_undegraded":"2024-04-24T23:25:42.773242+0000","last_fullsized":"2024-04-24T23:25:42.773242+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:00:52.424019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.35","version":"30'12","reported_seq":48,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:25:42.767307+0000","last_change":"2024-04-24T23:15:36.992958+0000","last_active":"2024-04-24T23:25:42.767307+0000","last_peered":"2024-04-24T23:25:42.767307+0000","last_clean":"2024-04-24T23:25:42.767307+0000","last_became_active":"2024-04-24T23:15:36.992608+0000","last_became_peered":"2024-04-24T23:15:36.992608+0000","last_unstale":"2024-04-24T23:25:42.767307+0000","last_undegraded":"2024-04-24T23:25:42.767307+0000","last_fullsized":"2024-04-24T23:25:42.767307+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:15:30.416763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.36","version":"30'10","reported_seq":46,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:25:42.767610+0000","last_change":"2024-04-24T23:15:36.992943+0000","last_active":"2024-04-24T23:25:42.767610+0000","last_peered":"2024-04-24T23:25:42.767610+0000","last_clean":"2024-04-24T23:25:42.767610+0000","last_became_active":"2024-04-24T23:15:36.992567+0000","last_became_peered":"2024-04-24T23:15:36.992567+0000","last_unstale":"2024-04-24T23:25:42.767610+0000","last_undegraded":"2024-04-24T23:25:42.767610+0000","last_fullsized":"2024-04-24T23:25:42.767610+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:50:43.209232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.37","version":"30'12","reported_seq":48,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:27.436849+0000","last_change":"2024-04-24T23:15:36.999697+0000","last_active":"2024-04-24T23:27:27.436849+0000","last_peered":"2024-04-24T23:27:27.436849+0000","last_clean":"2024-04-24T23:27:27.436849+0000","last_became_active":"2024-04-24T23:15:36.999413+0000","last_became_peered":"2024-04-24T23:15:36.999413+0000","last_unstale":"2024-04-24T23:27:27.436849+0000","last_undegraded":"2024-04-24T23:27:27.436849+0000","last_fullsized":"2024-04-24T23:27:27.436849+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:25:42.856471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"27'3","reported_seq":39,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:25:42.717468+0000","last_change":"2024-04-24T23:15:36.986742+0000","last_active":"2024-04-24T23:25:42.717468+0000","last_peered":"2024-04-24T23:25:42.717468+0000","last_clean":"2024-04-24T23:25:42.717468+0000","last_became_active":"2024-04-24T23:15:36.986013+0000","last_became_peered":"2024-04-24T23:15:36.986013+0000","last_unstale":"2024-04-24T23:25:42.717468+0000","last_undegraded":"2024-04-24T23:25:42.717468+0000","last_fullsized":"2024-04-24T23:25:42.717468+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:16:15.203151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"30'4","reported_seq":40,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:25:42.827178+0000","last_change":"2024-04-24T23:15:36.984336+0000","last_active":"2024-04-24T23:25:42.827178+0000","last_peered":"2024-04-24T23:25:42.827178+0000","last_clean":"2024-04-24T23:25:42.827178+0000","last_became_active":"2024-04-24T23:15:36.983966+0000","last_became_peered":"2024-04-24T23:15:36.983966+0000","last_unstale":"2024-04-24T23:25:42.827178+0000","last_undegraded":"2024-04-24T23:25:42.827178+0000","last_fullsized":"2024-04-24T23:25:42.827178+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:54:16.319608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3a","version":"30'9","reported_seq":45,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:36.677504+0000","last_change":"2024-04-24T23:15:36.979079+0000","last_active":"2024-04-24T23:27:36.677504+0000","last_peered":"2024-04-24T23:27:36.677504+0000","last_clean":"2024-04-24T23:27:36.677504+0000","last_became_active":"2024-04-24T23:15:36.978706+0000","last_became_peered":"2024-04-24T23:15:36.978706+0000","last_unstale":"2024-04-24T23:27:36.677504+0000","last_undegraded":"2024-04-24T23:27:36.677504+0000","last_fullsized":"2024-04-24T23:27:36.677504+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:53:39.888150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"27'6","reported_seq":42,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:25:42.769902+0000","last_change":"2024-04-24T23:15:36.986008+0000","last_active":"2024-04-24T23:25:42.769902+0000","last_peered":"2024-04-24T23:25:42.769902+0000","last_clean":"2024-04-24T23:25:42.769902+0000","last_became_active":"2024-04-24T23:15:36.985552+0000","last_became_peered":"2024-04-24T23:15:36.985552+0000","last_unstale":"2024-04-24T23:25:42.769902+0000","last_undegraded":"2024-04-24T23:25:42.769902+0000","last_fullsized":"2024-04-24T23:25:42.769902+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:36:50.827503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"30'6","reported_seq":43,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:25:44.818461+0000","last_change":"2024-04-24T23:15:36.978673+0000","last_active":"2024-04-24T23:25:44.818461+0000","last_peered":"2024-04-24T23:25:44.818461+0000","last_clean":"2024-04-24T23:25:44.818461+0000","last_became_active":"2024-04-24T23:15:36.972744+0000","last_became_peered":"2024-04-24T23:15:36.972744+0000","last_unstale":"2024-04-24T23:25:44.818461+0000","last_undegraded":"2024-04-24T23:25:44.818461+0000","last_fullsized":"2024-04-24T23:25:44.818461+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:28:57.028825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3d","version":"30'7","reported_seq":44,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:26:05.334229+0000","last_change":"2024-04-24T23:15:36.970238+0000","last_active":"2024-04-24T23:26:05.334229+0000","last_peered":"2024-04-24T23:26:05.334229+0000","last_clean":"2024-04-24T23:26:05.334229+0000","last_became_active":"2024-04-24T23:15:36.969751+0000","last_became_peered":"2024-04-24T23:15:36.969751+0000","last_unstale":"2024-04-24T23:26:05.334229+0000","last_undegraded":"2024-04-24T23:26:05.334229+0000","last_fullsized":"2024-04-24T23:26:05.334229+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:19:56.456801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3e","version":"26'2","reported_seq":39,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:25:42.719286+0000","last_change":"2024-04-24T23:15:36.968825+0000","last_active":"2024-04-24T23:25:42.719286+0000","last_peered":"2024-04-24T23:25:42.719286+0000","last_clean":"2024-04-24T23:25:42.719286+0000","last_became_active":"2024-04-24T23:15:36.968543+0000","last_became_peered":"2024-04-24T23:15:36.968543+0000","last_unstale":"2024-04-24T23:25:42.719286+0000","last_undegraded":"2024-04-24T23:25:42.719286+0000","last_fullsized":"2024-04-24T23:25:42.719286+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:36:48.795062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3f","version":"27'4","reported_seq":41,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:25:42.818133+0000","last_change":"2024-04-24T23:15:36.992800+0000","last_active":"2024-04-24T23:25:42.818133+0000","last_peered":"2024-04-24T23:25:42.818133+0000","last_clean":"2024-04-24T23:25:42.818133+0000","last_became_active":"2024-04-24T23:15:36.992476+0000","last_became_peered":"2024-04-24T23:15:36.992476+0000","last_unstale":"2024-04-24T23:25:42.818133+0000","last_undegraded":"2024-04-24T23:25:42.818133+0000","last_fullsized":"2024-04-24T23:25:42.818133+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:01:21.103559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":7111802757,"num_objects":1886,"num_object_clones":0,"num_object_copies":7544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1886,"num_whiteouts":0,"num_read":2252,"num_read_kb":307,"num_write":2058856,"num_write_kb":29192446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14351450112,"data_stored":14351450112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139710,"ondisk_log_size":139710,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":514,"ondisk_log_size":514,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":80027978,"num_objects":127,"num_object_clones":0,"num_object_copies":254,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":127,"num_whiteouts":0,"num_read":353,"num_read_kb":84742,"num_write":4101,"num_write_kb":81389,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":74,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":160292864,"data_stored":160055956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3839,"ondisk_log_size":3839,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574998,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3104376,"kb_used_data":2063984,"kb_used_omap":1,"kb_used_meta":1040382,"kb_avail":91267464,"statfs":{"total":96636764160,"available":93457883136,"internally_reserved":0,"allocated":2113519616,"data_stored":2113298368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":1065351627},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574998,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2884404,"kb_used_data":1938668,"kb_used_omap":1,"kb_used_meta":945726,"kb_avail":91487436,"statfs":{"total":96636764160,"available":93683134464,"internally_reserved":0,"allocated":1985196032,"data_stored":1984994624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":968423878},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574998,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2748124,"kb_used_data":1955668,"kb_used_omap":1,"kb_used_meta":792446,"kb_avail":91623716,"statfs":{"total":96636764160,"available":93822685184,"internally_reserved":0,"allocated":2002604032,"data_stored":2002391666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":811465157},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574999,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2501912,"kb_used_data":1727440,"kb_used_omap":1,"kb_used_meta":774462,"kb_avail":91869928,"statfs":{"total":96636764160,"available":94074806272,"internally_reserved":0,"allocated":1768898560,"data_stored":1768662424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":793049541},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574994,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3261728,"kb_used_data":2229400,"kb_used_omap":1,"kb_used_meta":1032318,"kb_avail":91110112,"statfs":{"total":96636764160,"available":93296754688,"internally_reserved":0,"allocated":2282905600,"data_stored":2282679443,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":1057094087},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,5,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574994,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2047988,"kb_used_data":1323948,"kb_used_omap":1,"kb_used_meta":724030,"kb_avail":92323852,"statfs":{"total":96636764160,"available":94539624448,"internally_reserved":0,"allocated":1355722752,"data_stored":1355521710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":741407173},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574994,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3096668,"kb_used_data":2038100,"kb_used_omap":1,"kb_used_meta":1054462,"kb_avail":91275172,"statfs":{"total":96636764160,"available":93465776128,"internally_reserved":0,"allocated":2087014400,"data_stored":2086804775,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1079769541},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574997,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1929760,"kb_used_data":1281112,"kb_used_omap":1,"kb_used_meta":648638,"kb_avail":92442080,"statfs":{"total":96636764160,"available":94660689920,"internally_reserved":0,"allocated":1311858688,"data_stored":1311639333,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":664205766},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":25436160,"data_stored":25402203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":21225472,"data_stored":21204797,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12599296,"data_stored":12583652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":37793792,"data_stored":37753033,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12636160,"data_stored":12585422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":25284608,"data_stored":25261333,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12603392,"data_stored":12587382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12713984,"data_stored":12678134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1210597376,"data_stored":1210597376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2048090112,"data_stored":2048090112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1273589760,"data_stored":1273589760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2242686976,"data_stored":2242686976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1737060352,"data_stored":1737060352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1789751296,"data_stored":1789751296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1949196288,"data_stored":1949196288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2100477952,"data_stored":2100477952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T23:27:41.989 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T23:27:42.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:42.204+0000 7fe762c00640 1 Processor -- start 2024-04-24T23:27:42.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:42.204+0000 7fe762c00640 1 -- start start 2024-04-24T23:27:42.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:42.206+0000 7fe762c00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe75c0c8f50 0x7fe75c06e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:42.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:42.206+0000 7fe762c00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe75c0c6e70 con 0x7fe75c0c9330 2024-04-24T23:27:42.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:42.206+0000 7fe762c00640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe75c0c7110 con 0x7fe75c152bc0 2024-04-24T23:27:42.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:42.206+0000 7fe762c00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fe75c0c73d0 con 0x7fe75c0c8f50 2024-04-24T23:27:42.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:42.207+0000 7fe75be00640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7fe75c152bc0 0x7fe75c159390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.6:44624/0 (socket says 172.21.15.6:44624) 2024-04-24T23:27:42.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:42.207+0000 7fe75be00640 1 -- 172.21.15.6:0/193583051 learned_addr learned my addr 172.21.15.6:0/193583051 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:27:42.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:42.207+0000 7fe753e00640 1 --2- 172.21.15.6:0/193583051 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe75c0c8f50 0x7fe75c06e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:42.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:42.207+0000 7fe75b400640 1 -- 172.21.15.6:0/193583051 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3717640548 0 0) 0x7fe75c0c7110 con 0x7fe75c152bc0 2024-04-24T23:27:42.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:42.207+0000 7fe75b400640 1 -- 172.21.15.6:0/193583051 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe740003520 con 0x7fe75c152bc0 2024-04-24T23:27:42.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:42.207+0000 7fe753e00640 1 -- 172.21.15.6:0/193583051 >> v1:172.21.15.190:6789/0 conn(0x7fe75c152bc0 legacy=0x7fe75c159390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:42.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:42.207+0000 7fe753e00640 1 -- 172.21.15.6:0/193583051 >> v1:172.21.15.6:6789/0 conn(0x7fe75c0c9330 legacy=0x7fe75c06e8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:42.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:42.207+0000 7fe75b400640 1 -- 172.21.15.6:0/193583051 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3906126330 0 0) 0x7fe75c0c6e70 con 0x7fe75c0c9330 2024-04-24T23:27:42.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:42.207+0000 7fe753e00640 1 -- 172.21.15.6:0/193583051 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe75c0c6700 con 0x7fe75c0c8f50 2024-04-24T23:27:42.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:42.209+0000 7fe753e00640 1 --2- 172.21.15.6:0/193583051 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe75c0c8f50 0x7fe75c06e3a0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fe75c0c7550 tx=0x7fe74402a0d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1762db6958dac2a3 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:27:42.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:42.209+0000 7fe75b400640 1 -- 172.21.15.6:0/193583051 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe744037070 con 0x7fe75c0c8f50 2024-04-24T23:27:42.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:42.209+0000 7fe75b400640 1 -- 172.21.15.6:0/193583051 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe74402ae20 con 0x7fe75c0c8f50 2024-04-24T23:27:42.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:42.209+0000 7fe75b400640 1 -- 172.21.15.6:0/193583051 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe744032640 con 0x7fe75c0c8f50 2024-04-24T23:27:42.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:42.210+0000 7fe762c00640 1 -- 172.21.15.6:0/193583051 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe75c0c8f50 msgr2=0x7fe75c06e3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:42.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:42.210+0000 7fe762c00640 1 --2- 172.21.15.6:0/193583051 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe75c0c8f50 0x7fe75c06e3a0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fe75c0c7550 tx=0x7fe74402a0d0 comp rx=0 tx=0).stop 2024-04-24T23:27:42.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:42.210+0000 7fe762c00640 1 -- 172.21.15.6:0/193583051 shutdown_connections 2024-04-24T23:27:42.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:42.210+0000 7fe762c00640 1 --2- 172.21.15.6:0/193583051 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe75c0c8f50 0x7fe75c06e3a0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:42.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:42.210+0000 7fe762c00640 1 -- 172.21.15.6:0/193583051 >> 172.21.15.6:0/193583051 conn(0x7fe75c06b0e0 msgr2=0x7fe75c06b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:27:42.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:42.210+0000 7fe762c00640 1 -- 172.21.15.6:0/193583051 shutdown_connections 2024-04-24T23:27:42.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:42.210+0000 7fe762c00640 1 -- 172.21.15.6:0/193583051 wait complete. 2024-04-24T23:27:42.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:42.212+0000 7fe762c00640 1 Processor -- start 2024-04-24T23:27:42.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:42.212+0000 7fe762c00640 1 -- start start 2024-04-24T23:27:42.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:42.213+0000 7fe762c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe75c0c8f50 0x7fe75c145930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:42.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:42.213+0000 7fe762c00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe75c0c9330 0x7fe75c158e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:42.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:42.213+0000 7fe762c00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe75c152bc0 0x7fe75c159380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:42.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:42.213+0000 7fe762c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fe75c0c8750 con 0x7fe75c0c8f50 2024-04-24T23:27:42.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:42.213+0000 7fe762c00640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fe75c0c8ac0 con 0x7fe75c0c9330 2024-04-24T23:27:42.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:42.213+0000 7fe762c00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fe75c152fa0 con 0x7fe75c152bc0 2024-04-24T23:27:42.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:42.213+0000 7fe760c00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe75c152bc0 0x7fe75c159380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:42.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:42.213+0000 7fe753e00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe75c0c9330 0x7fe75c158e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:42.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:42.213+0000 7fe75be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe75c0c8f50 0x7fe75c145930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:42.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:42.213+0000 7fe753e00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe75c0c9330 0x7fe75c158e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:40704/0 (socket says 172.21.15.6:40704) 2024-04-24T23:27:42.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:42.213+0000 7fe760c00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe75c152bc0 0x7fe75c159380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.6:52304/0 (socket says 172.21.15.6:52304) 2024-04-24T23:27:42.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:42.213+0000 7fe753e00640 1 -- 172.21.15.6:0/142376727 learned_addr learned my addr 172.21.15.6:0/142376727 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:27:42.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:42.213+0000 7fe760c00640 1 -- 172.21.15.6:0/142376727 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe75c0c9330 msgr2=0x7fe75c158e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:42.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:42.213+0000 7fe760c00640 1 --2- 172.21.15.6:0/142376727 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe75c0c9330 0x7fe75c158e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:42.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:42.213+0000 7fe760c00640 1 -- 172.21.15.6:0/142376727 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe75c0c8f50 msgr2=0x7fe75c145930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:42.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:42.213+0000 7fe760c00640 1 --2- 172.21.15.6:0/142376727 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe75c0c8f50 0x7fe75c145930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:42.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:42.213+0000 7fe760c00640 1 -- 172.21.15.6:0/142376727 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe75c1447e0 con 0x7fe75c152bc0 2024-04-24T23:27:42.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:42.214+0000 7fe760c00640 1 --2- 172.21.15.6:0/142376727 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe75c152bc0 0x7fe75c159380 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fe75400c780 tx=0x7fe75400cc40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:27:42.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:42.214+0000 7fe758c00640 1 -- 172.21.15.6:0/142376727 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe754003b70 con 0x7fe75c152bc0 2024-04-24T23:27:42.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:42.214+0000 7fe758c00640 1 -- 172.21.15.6:0/142376727 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe754003d30 con 0x7fe75c152bc0 2024-04-24T23:27:42.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:42.214+0000 7fe758c00640 1 -- 172.21.15.6:0/142376727 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7540052b0 con 0x7fe75c152bc0 2024-04-24T23:27:42.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:42.214+0000 7fe762c00640 1 -- 172.21.15.6:0/142376727 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe75c1572c0 con 0x7fe75c152bc0 2024-04-24T23:27:42.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:42.214+0000 7fe762c00640 1 -- 172.21.15.6:0/142376727 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fe75c15be80 con 0x7fe75c152bc0 2024-04-24T23:27:42.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:42.215+0000 7fe762c00640 1 -- 172.21.15.6:0/142376727 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe75c1578b0 con 0x7fe75c152bc0 2024-04-24T23:27:42.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:42.216+0000 7fe758c00640 1 -- 172.21.15.6:0/142376727 <== mon.2 v2:172.21.15.190:3301/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fe754005450 con 0x7fe75c152bc0 2024-04-24T23:27:42.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:42.217+0000 7fe758c00640 1 --2- 172.21.15.6:0/142376727 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fe73806be30 0x7fe73806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:42.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:42.217+0000 7fe758c00640 1 -- 172.21.15.6:0/142376727 <== mon.2 v2:172.21.15.190:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6934+0+0 (secure 0 0 0) 0x7fe75408b0b0 con 0x7fe75c152bc0 2024-04-24T23:27:42.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:42.217+0000 7fe75be00640 1 --2- 172.21.15.6:0/142376727 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fe73806be30 0x7fe73806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:42.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:42.218+0000 7fe75be00640 1 --2- 172.21.15.6:0/142376727 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fe73806be30 0x7fe73806e2f0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fe74c0011e0 tx=0x7fe74c007d30 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:27:42.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:42.221+0000 7fe758c00640 1 -- 172.21.15.6:0/142376727 <== mon.2 v2:172.21.15.190:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fe754058ea0 con 0x7fe75c152bc0 2024-04-24T23:27:42.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:42.591+0000 7fe762c00640 1 -- 172.21.15.6:0/142376727 --> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fe75c144ac0 con 0x7fe73806be30 2024-04-24T23:27:42.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:42.605+0000 7fe758c00640 1 -- 172.21.15.6:0/142376727 <== mgr.4105 v2:172.21.15.190:6832/806769663 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+334839 (secure 0 0 0) 0x7fe75c144ac0 con 0x7fe73806be30 2024-04-24T23:27:42.610 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:27:42.612 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-04-24T23:27:42.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:42.614+0000 7fe762c00640 1 -- 172.21.15.6:0/142376727 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fe73806be30 msgr2=0x7fe73806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:42.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:42.614+0000 7fe762c00640 1 --2- 172.21.15.6:0/142376727 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fe73806be30 0x7fe73806e2f0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fe74c0011e0 tx=0x7fe74c007d30 comp rx=0 tx=0).stop 2024-04-24T23:27:42.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:42.614+0000 7fe762c00640 1 -- 172.21.15.6:0/142376727 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe75c152bc0 msgr2=0x7fe75c159380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:42.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:42.614+0000 7fe762c00640 1 --2- 172.21.15.6:0/142376727 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe75c152bc0 0x7fe75c159380 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fe75400c780 tx=0x7fe75400cc40 comp rx=0 tx=0).stop 2024-04-24T23:27:42.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:42.614+0000 7fe762c00640 1 -- 172.21.15.6:0/142376727 shutdown_connections 2024-04-24T23:27:42.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:42.614+0000 7fe762c00640 1 --2- 172.21.15.6:0/142376727 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fe73806be30 0x7fe73806e2f0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:42.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:42.614+0000 7fe762c00640 1 --2- 172.21.15.6:0/142376727 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe75c152bc0 0x7fe75c159380 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:42.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:42.614+0000 7fe762c00640 1 --2- 172.21.15.6:0/142376727 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe75c0c9330 0x7fe75c158e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:42.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:42.614+0000 7fe762c00640 1 --2- 172.21.15.6:0/142376727 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe75c0c8f50 0x7fe75c145930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:42.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:42.614+0000 7fe762c00640 1 -- 172.21.15.6:0/142376727 >> 172.21.15.6:0/142376727 conn(0x7fe75c06b0e0 msgr2=0x7fe75c06e130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:27:42.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:42.614+0000 7fe762c00640 1 -- 172.21.15.6:0/142376727 shutdown_connections 2024-04-24T23:27:42.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:42.615+0000 7fe762c00640 1 -- 172.21.15.6:0/142376727 wait complete. 2024-04-24T23:27:42.658 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":391,"stamp":"2024-04-24T23:27:40.871751+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":7192421103,"num_objects":2015,"num_object_clones":0,"num_object_copies":7802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2015,"num_whiteouts":0,"num_read":2721,"num_read_kb":85149,"num_write":2063116,"num_write_kb":29276251,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":74,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":144180,"ondisk_log_size":144180,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":21574960,"kb_used_data":14558320,"kb_used_omap":12,"kb_used_meta":7012467,"kb_avail":733399760,"statfs":{"total":773094113280,"available":751001354240,"internally_reserved":0,"allocated":14907719680,"data_stored":14905992343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":7180766770},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,2,5,6,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":70,"apply_latency_ms":70,"commit_latency_ns":70000000,"apply_latency_ns":70000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-5552649622,"num_objects":-1323,"num_object_clones":0,"num_object_copies":-5294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1323,"num_whiteouts":0,"num_read":2,"num_read_kb":27,"num_write":1384,"num_write_kb":627,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003108"},"pg_stats":[{"pgid":"2.1f","version":"28'36","reported_seq":80,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:26:05.796187+0000","last_change":"2024-04-24T23:15:35.964646+0000","last_active":"2024-04-24T23:26:05.796187+0000","last_peered":"2024-04-24T23:26:05.796187+0000","last_clean":"2024-04-24T23:26:05.796187+0000","last_became_active":"2024-04-24T23:15:35.964357+0000","last_became_peered":"2024-04-24T23:15:35.964357+0000","last_unstale":"2024-04-24T23:26:05.796187+0000","last_undegraded":"2024-04-24T23:26:05.796187+0000","last_fullsized":"2024-04-24T23:26:05.796187+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:01:17.912993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194976,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":7,"num_read_kb":4102,"num_write":33,"num_write_kb":4110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"27'3","reported_seq":40,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:25:42.719794+0000","last_change":"2024-04-24T23:15:36.989772+0000","last_active":"2024-04-24T23:25:42.719794+0000","last_peered":"2024-04-24T23:25:42.719794+0000","last_clean":"2024-04-24T23:25:42.719794+0000","last_became_active":"2024-04-24T23:15:36.989543+0000","last_became_peered":"2024-04-24T23:15:36.989543+0000","last_unstale":"2024-04-24T23:25:42.719794+0000","last_undegraded":"2024-04-24T23:25:42.719794+0000","last_fullsized":"2024-04-24T23:25:42.719794+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:37:16.737354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"30'4701","reported_seq":4795,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:37.613675+0000","last_change":"2024-04-24T23:16:18.300096+0000","last_active":"2024-04-24T23:27:37.613675+0000","last_peered":"2024-04-24T23:27:37.613675+0000","last_clean":"2024-04-24T23:27:37.613675+0000","last_became_active":"2024-04-24T23:16:18.299217+0000","last_became_peered":"2024-04-24T23:16:18.299217+0000","last_unstale":"2024-04-24T23:27:37.613675+0000","last_undegraded":"2024-04-24T23:27:37.613675+0000","last_fullsized":"2024-04-24T23:27:37.613675+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":4701,"log_dups_size":0,"ondisk_log_size":4701,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:08:19.426862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":255971335,"num_objects":71,"num_object_clones":0,"num_object_copies":284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71,"num_whiteouts":0,"num_read":70,"num_read_kb":0,"num_write":69516,"num_write_kb":969009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4,1],"acting":[0,6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"28'8","reported_seq":48,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:25:42.720012+0000","last_change":"2024-04-24T23:15:35.946083+0000","last_active":"2024-04-24T23:25:42.720012+0000","last_peered":"2024-04-24T23:25:42.720012+0000","last_clean":"2024-04-24T23:25:42.720012+0000","last_became_active":"2024-04-24T23:15:35.945227+0000","last_became_peered":"2024-04-24T23:15:35.945227+0000","last_unstale":"2024-04-24T23:25:42.720012+0000","last_undegraded":"2024-04-24T23:25:42.720012+0000","last_fullsized":"2024-04-24T23:25:42.720012+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:05:17.727725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":4096,"num_write":7,"num_write_kb":4105,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1f","version":"27'6","reported_seq":43,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:25:42.719955+0000","last_change":"2024-04-24T23:15:36.998245+0000","last_active":"2024-04-24T23:25:42.719955+0000","last_peered":"2024-04-24T23:25:42.719955+0000","last_clean":"2024-04-24T23:25:42.719955+0000","last_became_active":"2024-04-24T23:15:36.998021+0000","last_became_peered":"2024-04-24T23:15:36.998021+0000","last_unstale":"2024-04-24T23:25:42.719955+0000","last_undegraded":"2024-04-24T23:25:42.719955+0000","last_fullsized":"2024-04-24T23:25:42.719955+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:03:42.359117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"30'4538","reported_seq":4643,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:39.372938+0000","last_change":"2024-04-24T23:15:41.008313+0000","last_active":"2024-04-24T23:27:39.372938+0000","last_peered":"2024-04-24T23:27:39.372938+0000","last_clean":"2024-04-24T23:27:39.372938+0000","last_became_active":"2024-04-24T23:15:41.007908+0000","last_became_peered":"2024-04-24T23:15:41.007908+0000","last_unstale":"2024-04-24T23:27:39.372938+0000","last_undegraded":"2024-04-24T23:27:39.372938+0000","last_fullsized":"2024-04-24T23:27:39.372938+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":4538,"log_dups_size":0,"ondisk_log_size":4538,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:02:11.902575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":176186537,"num_objects":46,"num_object_clones":0,"num_object_copies":184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":75,"num_read_kb":0,"num_write":66573,"num_write_kb":930809,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4,6],"acting":[2,7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1d","version":"30'40","reported_seq":92,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:23.196288+0000","last_change":"2024-04-24T23:15:35.968312+0000","last_active":"2024-04-24T23:27:23.196288+0000","last_peered":"2024-04-24T23:27:23.196288+0000","last_clean":"2024-04-24T23:27:23.196288+0000","last_became_active":"2024-04-24T23:15:35.967873+0000","last_became_peered":"2024-04-24T23:15:35.967873+0000","last_unstale":"2024-04-24T23:27:23.196288+0000","last_undegraded":"2024-04-24T23:27:23.196288+0000","last_fullsized":"2024-04-24T23:27:23.196288+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:21:40.116977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":38,"num_read_kb":8226,"num_write":37,"num_write_kb":8205,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"27'7","reported_seq":43,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:25:42.821075+0000","last_change":"2024-04-24T23:15:36.997361+0000","last_active":"2024-04-24T23:25:42.821075+0000","last_peered":"2024-04-24T23:25:42.821075+0000","last_clean":"2024-04-24T23:25:42.821075+0000","last_became_active":"2024-04-24T23:15:36.997083+0000","last_became_peered":"2024-04-24T23:15:36.997083+0000","last_unstale":"2024-04-24T23:25:42.821075+0000","last_undegraded":"2024-04-24T23:25:42.821075+0000","last_fullsized":"2024-04-24T23:25:42.821075+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:47:59.334093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"30'4235","reported_seq":4335,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:39.416539+0000","last_change":"2024-04-24T23:15:41.025153+0000","last_active":"2024-04-24T23:27:39.416539+0000","last_peered":"2024-04-24T23:27:39.416539+0000","last_clean":"2024-04-24T23:27:39.416539+0000","last_became_active":"2024-04-24T23:15:41.024739+0000","last_became_peered":"2024-04-24T23:15:41.024739+0000","last_unstale":"2024-04-24T23:27:39.416539+0000","last_undegraded":"2024-04-24T23:27:39.416539+0000","last_fullsized":"2024-04-24T23:27:39.416539+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":4235,"log_dups_size":0,"ondisk_log_size":4235,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:31:00.160213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184565702,"num_objects":54,"num_object_clones":0,"num_object_copies":216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":70,"num_read_kb":0,"num_write":62343,"num_write_kb":877947,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3,6],"acting":[2,5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1c","version":"30'17","reported_seq":63,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:36.086972+0000","last_change":"2024-04-24T23:15:35.967338+0000","last_active":"2024-04-24T23:27:36.086972+0000","last_peered":"2024-04-24T23:27:36.086972+0000","last_clean":"2024-04-24T23:27:36.086972+0000","last_became_active":"2024-04-24T23:15:35.966951+0000","last_became_peered":"2024-04-24T23:15:35.966951+0000","last_unstale":"2024-04-24T23:27:36.086972+0000","last_undegraded":"2024-04-24T23:27:36.086972+0000","last_fullsized":"2024-04-24T23:27:36.086972+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:08:05.248330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":232445,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":21,"num_read_kb":20,"num_write":17,"num_write_kb":234,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1d","version":"30'7","reported_seq":43,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:26.779216+0000","last_change":"2024-04-24T23:15:36.998072+0000","last_active":"2024-04-24T23:27:26.779216+0000","last_peered":"2024-04-24T23:27:26.779216+0000","last_clean":"2024-04-24T23:27:26.779216+0000","last_became_active":"2024-04-24T23:15:36.997735+0000","last_became_peered":"2024-04-24T23:15:36.997735+0000","last_unstale":"2024-04-24T23:27:26.779216+0000","last_undegraded":"2024-04-24T23:27:26.779216+0000","last_fullsized":"2024-04-24T23:27:26.779216+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:22:06.869644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"30'3724","reported_seq":3813,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:39.072670+0000","last_change":"2024-04-24T23:15:41.008895+0000","last_active":"2024-04-24T23:27:39.072670+0000","last_peered":"2024-04-24T23:27:39.072670+0000","last_clean":"2024-04-24T23:27:39.072670+0000","last_became_active":"2024-04-24T23:15:41.006199+0000","last_became_peered":"2024-04-24T23:15:41.006199+0000","last_unstale":"2024-04-24T23:27:39.072670+0000","last_undegraded":"2024-04-24T23:27:39.072670+0000","last_fullsized":"2024-04-24T23:27:39.072670+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":3724,"log_dups_size":0,"ondisk_log_size":3724,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:13:34.583602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184573346,"num_objects":48,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":59,"num_read_kb":0,"num_write":54757,"num_write_kb":774507,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5,7],"acting":[2,6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"30'8","reported_seq":45,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:26:34.900000+0000","last_change":"2024-04-24T23:15:37.004074+0000","last_active":"2024-04-24T23:26:34.900000+0000","last_peered":"2024-04-24T23:26:34.900000+0000","last_clean":"2024-04-24T23:26:34.900000+0000","last_became_active":"2024-04-24T23:15:37.003544+0000","last_became_peered":"2024-04-24T23:15:37.003544+0000","last_unstale":"2024-04-24T23:26:34.900000+0000","last_undegraded":"2024-04-24T23:26:34.900000+0000","last_fullsized":"2024-04-24T23:26:34.900000+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:38:28.024702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"30'1458","reported_seq":1502,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:26.160448+0000","last_change":"2024-04-24T23:15:35.963000+0000","last_active":"2024-04-24T23:27:26.160448+0000","last_peered":"2024-04-24T23:27:26.160448+0000","last_clean":"2024-04-24T23:27:26.160448+0000","last_became_active":"2024-04-24T23:15:35.961960+0000","last_became_peered":"2024-04-24T23:15:35.961960+0000","last_unstale":"2024-04-24T23:27:26.160448+0000","last_undegraded":"2024-04-24T23:27:26.160448+0000","last_fullsized":"2024-04-24T23:27:26.160448+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":1458,"log_dups_size":0,"ondisk_log_size":1458,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:56:26.651193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":10,"num_read_kb":8200,"num_write":1452,"num_write_kb":5429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"30'4317","reported_seq":4413,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:36.092914+0000","last_change":"2024-04-24T23:15:41.022481+0000","last_active":"2024-04-24T23:27:36.092914+0000","last_peered":"2024-04-24T23:27:36.092914+0000","last_clean":"2024-04-24T23:27:36.092914+0000","last_became_active":"2024-04-24T23:15:41.020034+0000","last_became_peered":"2024-04-24T23:15:41.020034+0000","last_unstale":"2024-04-24T23:27:36.092914+0000","last_undegraded":"2024-04-24T23:27:36.092914+0000","last_fullsized":"2024-04-24T23:27:36.092914+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":4317,"log_dups_size":0,"ondisk_log_size":4317,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:27:25.337503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":222443060,"num_objects":60,"num_object_clones":0,"num_object_copies":240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":65,"num_read_kb":0,"num_write":63602,"num_write_kb":904417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,5],"acting":[3,1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"30'6","reported_seq":43,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:26:52.416421+0000","last_change":"2024-04-24T23:15:36.972020+0000","last_active":"2024-04-24T23:26:52.416421+0000","last_peered":"2024-04-24T23:26:52.416421+0000","last_clean":"2024-04-24T23:26:52.416421+0000","last_became_active":"2024-04-24T23:15:36.971583+0000","last_became_peered":"2024-04-24T23:15:36.971583+0000","last_unstale":"2024-04-24T23:26:52.416421+0000","last_undegraded":"2024-04-24T23:26:52.416421+0000","last_fullsized":"2024-04-24T23:26:52.416421+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:29:05.568608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"30'7","reported_seq":47,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:26:31.416252+0000","last_change":"2024-04-24T23:15:35.945916+0000","last_active":"2024-04-24T23:26:31.416252+0000","last_peered":"2024-04-24T23:26:31.416252+0000","last_clean":"2024-04-24T23:26:31.416252+0000","last_became_active":"2024-04-24T23:15:35.945610+0000","last_became_peered":"2024-04-24T23:15:35.945610+0000","last_unstale":"2024-04-24T23:26:31.416252+0000","last_undegraded":"2024-04-24T23:26:31.416252+0000","last_fullsized":"2024-04-24T23:26:31.416252+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:57:05.976074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"30'4388","reported_seq":4489,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:37.625508+0000","last_change":"2024-04-24T23:15:41.030165+0000","last_active":"2024-04-24T23:27:37.625508+0000","last_peered":"2024-04-24T23:27:37.625508+0000","last_clean":"2024-04-24T23:27:37.625508+0000","last_became_active":"2024-04-24T23:15:41.029685+0000","last_became_peered":"2024-04-24T23:15:41.029685+0000","last_unstale":"2024-04-24T23:27:37.625508+0000","last_undegraded":"2024-04-24T23:27:37.625508+0000","last_fullsized":"2024-04-24T23:27:37.625508+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":4388,"log_dups_size":0,"ondisk_log_size":4388,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:20:43.498190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":171984928,"num_objects":49,"num_object_clones":0,"num_object_copies":196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49,"num_whiteouts":0,"num_read":70,"num_read_kb":0,"num_write":64497,"num_write_kb":914755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6,1],"acting":[0,4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"30'8","reported_seq":45,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:26:30.926375+0000","last_change":"2024-04-24T23:15:36.965949+0000","last_active":"2024-04-24T23:26:30.926375+0000","last_peered":"2024-04-24T23:26:30.926375+0000","last_clean":"2024-04-24T23:26:30.926375+0000","last_became_active":"2024-04-24T23:15:36.965658+0000","last_became_peered":"2024-04-24T23:15:36.965658+0000","last_unstale":"2024-04-24T23:26:30.926375+0000","last_undegraded":"2024-04-24T23:26:30.926375+0000","last_fullsized":"2024-04-24T23:26:30.926375+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:38:28.024702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"28'8","reported_seq":54,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:25:42.710930+0000","last_change":"2024-04-24T23:15:35.972165+0000","last_active":"2024-04-24T23:25:42.710930+0000","last_peered":"2024-04-24T23:25:42.710930+0000","last_clean":"2024-04-24T23:25:42.710930+0000","last_became_active":"2024-04-24T23:15:35.971972+0000","last_became_peered":"2024-04-24T23:15:35.971972+0000","last_unstale":"2024-04-24T23:25:42.710930+0000","last_undegraded":"2024-04-24T23:25:42.710930+0000","last_fullsized":"2024-04-24T23:25:42.710930+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:11:12.088698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":20,"num_read_kb":19,"num_write":11,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"30'5152","reported_seq":5264,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:35.888667+0000","last_change":"2024-04-24T23:15:41.019799+0000","last_active":"2024-04-24T23:27:35.888667+0000","last_peered":"2024-04-24T23:27:35.888667+0000","last_clean":"2024-04-24T23:27:35.888667+0000","last_became_active":"2024-04-24T23:15:41.019339+0000","last_became_peered":"2024-04-24T23:15:41.019339+0000","last_unstale":"2024-04-24T23:27:35.888667+0000","last_undegraded":"2024-04-24T23:27:35.888667+0000","last_fullsized":"2024-04-24T23:27:35.888667+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":5152,"log_dups_size":0,"ondisk_log_size":5152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:59:34.601004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":251755637,"num_objects":65,"num_object_clones":0,"num_object_copies":260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":0,"num_read":81,"num_read_kb":0,"num_write":75840,"num_write_kb":1074791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2,3],"acting":[7,4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"27'4","reported_seq":41,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:25:42.747473+0000","last_change":"2024-04-24T23:15:36.991991+0000","last_active":"2024-04-24T23:25:42.747473+0000","last_peered":"2024-04-24T23:25:42.747473+0000","last_clean":"2024-04-24T23:25:42.747473+0000","last_became_active":"2024-04-24T23:15:36.989336+0000","last_became_peered":"2024-04-24T23:15:36.989336+0000","last_unstale":"2024-04-24T23:25:42.747473+0000","last_undegraded":"2024-04-24T23:25:42.747473+0000","last_fullsized":"2024-04-24T23:25:42.747473+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:25:56.038903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.8","version":"28'31","reported_seq":73,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:26:32.455095+0000","last_change":"2024-04-24T23:15:35.965443+0000","last_active":"2024-04-24T23:26:32.455095+0000","last_peered":"2024-04-24T23:26:32.455095+0000","last_clean":"2024-04-24T23:26:32.455095+0000","last_became_active":"2024-04-24T23:15:35.965043+0000","last_became_peered":"2024-04-24T23:15:35.965043+0000","last_unstale":"2024-04-24T23:26:32.455095+0000","last_undegraded":"2024-04-24T23:26:32.455095+0000","last_fullsized":"2024-04-24T23:26:32.455095+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:07:23.320042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":4102,"num_write":30,"num_write_kb":4111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"30'3783","reported_seq":3875,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:39.129224+0000","last_change":"2024-04-24T23:15:41.025131+0000","last_active":"2024-04-24T23:27:39.129224+0000","last_peered":"2024-04-24T23:27:39.129224+0000","last_clean":"2024-04-24T23:27:39.129224+0000","last_became_active":"2024-04-24T23:15:41.024731+0000","last_became_peered":"2024-04-24T23:15:41.024731+0000","last_unstale":"2024-04-24T23:27:39.129224+0000","last_undegraded":"2024-04-24T23:27:39.129224+0000","last_fullsized":"2024-04-24T23:27:39.129224+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":3783,"log_dups_size":0,"ondisk_log_size":3783,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:28:07.952854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180570789,"num_objects":53,"num_object_clones":0,"num_object_copies":212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":61,"num_read_kb":0,"num_write":55520,"num_write_kb":793811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,7],"acting":[2,1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"30'15","reported_seq":52,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:26:31.027261+0000","last_change":"2024-04-24T23:15:36.965850+0000","last_active":"2024-04-24T23:26:31.027261+0000","last_peered":"2024-04-24T23:26:31.027261+0000","last_clean":"2024-04-24T23:26:31.027261+0000","last_became_active":"2024-04-24T23:15:36.965547+0000","last_became_peered":"2024-04-24T23:15:36.965547+0000","last_unstale":"2024-04-24T23:26:31.027261+0000","last_undegraded":"2024-04-24T23:26:31.027261+0000","last_fullsized":"2024-04-24T23:26:31.027261+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:06:37.125573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"30'9","reported_seq":56,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:23.206469+0000","last_change":"2024-04-24T23:15:35.971040+0000","last_active":"2024-04-24T23:27:23.206469+0000","last_peered":"2024-04-24T23:27:23.206469+0000","last_clean":"2024-04-24T23:27:23.206469+0000","last_became_active":"2024-04-24T23:15:35.970184+0000","last_became_peered":"2024-04-24T23:15:35.970184+0000","last_unstale":"2024-04-24T23:27:23.206469+0000","last_undegraded":"2024-04-24T23:27:23.206469+0000","last_fullsized":"2024-04-24T23:27:23.206469+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:45:20.913759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":24,"num_write":11,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"30'4371","reported_seq":4476,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:38.781850+0000","last_change":"2024-04-24T23:15:41.025813+0000","last_active":"2024-04-24T23:27:38.781850+0000","last_peered":"2024-04-24T23:27:38.781850+0000","last_clean":"2024-04-24T23:27:38.781850+0000","last_became_active":"2024-04-24T23:15:41.025137+0000","last_became_peered":"2024-04-24T23:15:41.025137+0000","last_unstale":"2024-04-24T23:27:38.781850+0000","last_undegraded":"2024-04-24T23:27:38.781850+0000","last_fullsized":"2024-04-24T23:27:38.781850+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":4371,"log_dups_size":0,"ondisk_log_size":4371,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:35:17.784854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":205532220,"num_objects":52,"num_object_clones":0,"num_object_copies":208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":0,"num_read":74,"num_read_kb":0,"num_write":64361,"num_write_kb":917129,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,3,1],"acting":[2,6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"27'10","reported_seq":47,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:25:42.711648+0000","last_change":"2024-04-24T23:15:36.967476+0000","last_active":"2024-04-24T23:25:42.711648+0000","last_peered":"2024-04-24T23:25:42.711648+0000","last_clean":"2024-04-24T23:25:42.711648+0000","last_became_active":"2024-04-24T23:15:36.967098+0000","last_became_peered":"2024-04-24T23:15:36.967098+0000","last_unstale":"2024-04-24T23:25:42.711648+0000","last_undegraded":"2024-04-24T23:25:42.711648+0000","last_fullsized":"2024-04-24T23:25:42.711648+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:25:56.038903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"30'261","reported_seq":301,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:33.175189+0000","last_change":"2024-04-24T23:15:35.970630+0000","last_active":"2024-04-24T23:27:33.175189+0000","last_peered":"2024-04-24T23:27:33.175189+0000","last_clean":"2024-04-24T23:27:33.175189+0000","last_became_active":"2024-04-24T23:15:35.970430+0000","last_became_peered":"2024-04-24T23:15:35.970430+0000","last_unstale":"2024-04-24T23:27:33.175189+0000","last_undegraded":"2024-04-24T23:27:33.175189+0000","last_fullsized":"2024-04-24T23:27:33.175189+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":261,"log_dups_size":0,"ondisk_log_size":261,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:15:59.219084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":9,"num_write":559,"num_write_kb":230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"30'4712","reported_seq":4820,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:36.160259+0000","last_change":"2024-04-24T23:15:41.005022+0000","last_active":"2024-04-24T23:27:36.160259+0000","last_peered":"2024-04-24T23:27:36.160259+0000","last_clean":"2024-04-24T23:27:36.160259+0000","last_became_active":"2024-04-24T23:15:41.004648+0000","last_became_peered":"2024-04-24T23:15:41.004648+0000","last_unstale":"2024-04-24T23:27:36.160259+0000","last_undegraded":"2024-04-24T23:27:36.160259+0000","last_fullsized":"2024-04-24T23:27:36.160259+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":4712,"log_dups_size":0,"ondisk_log_size":4712,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:06:01.559760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":243285285,"num_objects":64,"num_object_clones":0,"num_object_copies":256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":77,"num_read_kb":0,"num_write":69520,"num_write_kb":1012990,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6,3],"acting":[4,1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.4","version":"30'4","reported_seq":41,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:26:06.045309+0000","last_change":"2024-04-24T23:15:36.967053+0000","last_active":"2024-04-24T23:26:06.045309+0000","last_peered":"2024-04-24T23:26:06.045309+0000","last_clean":"2024-04-24T23:26:06.045309+0000","last_became_active":"2024-04-24T23:15:36.966746+0000","last_became_peered":"2024-04-24T23:15:36.966746+0000","last_unstale":"2024-04-24T23:26:06.045309+0000","last_undegraded":"2024-04-24T23:26:06.045309+0000","last_fullsized":"2024-04-24T23:26:06.045309+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:16:08.779860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"26'4","reported_seq":46,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:25:42.781078+0000","last_change":"2024-04-24T23:15:35.965890+0000","last_active":"2024-04-24T23:25:42.781078+0000","last_peered":"2024-04-24T23:25:42.781078+0000","last_clean":"2024-04-24T23:25:42.781078+0000","last_became_active":"2024-04-24T23:15:35.965631+0000","last_became_peered":"2024-04-24T23:15:35.965631+0000","last_unstale":"2024-04-24T23:25:42.781078+0000","last_undegraded":"2024-04-24T23:25:42.781078+0000","last_fullsized":"2024-04-24T23:25:42.781078+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:17:35.650731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"30'3992","reported_seq":4083,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:35.280472+0000","last_change":"2024-04-24T23:15:41.011405+0000","last_active":"2024-04-24T23:27:35.280472+0000","last_peered":"2024-04-24T23:27:35.280472+0000","last_clean":"2024-04-24T23:27:35.280472+0000","last_became_active":"2024-04-24T23:15:41.010982+0000","last_became_peered":"2024-04-24T23:15:41.010982+0000","last_unstale":"2024-04-24T23:27:35.280472+0000","last_undegraded":"2024-04-24T23:27:35.280472+0000","last_fullsized":"2024-04-24T23:27:35.280472+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":3992,"log_dups_size":0,"ondisk_log_size":3992,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:08:00.914296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":209724275,"num_objects":54,"num_object_clones":0,"num_object_copies":216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":60,"num_read_kb":0,"num_write":58600,"num_write_kb":828032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2,7],"acting":[6,0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"30'10","reported_seq":47,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:27.778246+0000","last_change":"2024-04-24T23:15:36.975244+0000","last_active":"2024-04-24T23:27:27.778246+0000","last_peered":"2024-04-24T23:27:27.778246+0000","last_clean":"2024-04-24T23:27:27.778246+0000","last_became_active":"2024-04-24T23:15:36.974872+0000","last_became_peered":"2024-04-24T23:15:36.974872+0000","last_unstale":"2024-04-24T23:27:27.778246+0000","last_undegraded":"2024-04-24T23:27:27.778246+0000","last_fullsized":"2024-04-24T23:27:27.778246+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:45:57.018205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"27'2","reported_seq":41,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:25:42.718435+0000","last_change":"2024-04-24T23:15:35.945792+0000","last_active":"2024-04-24T23:25:42.718435+0000","last_peered":"2024-04-24T23:25:42.718435+0000","last_clean":"2024-04-24T23:25:42.718435+0000","last_became_active":"2024-04-24T23:15:35.945094+0000","last_became_peered":"2024-04-24T23:15:35.945094+0000","last_unstale":"2024-04-24T23:25:42.718435+0000","last_undegraded":"2024-04-24T23:25:42.718435+0000","last_fullsized":"2024-04-24T23:25:42.718435+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:49:12.719127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"30'4888","reported_seq":4996,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:36.715831+0000","last_change":"2024-04-24T23:15:41.022105+0000","last_active":"2024-04-24T23:27:36.715831+0000","last_peered":"2024-04-24T23:27:36.715831+0000","last_clean":"2024-04-24T23:27:36.715831+0000","last_became_active":"2024-04-24T23:15:41.021645+0000","last_became_peered":"2024-04-24T23:15:41.021645+0000","last_unstale":"2024-04-24T23:27:36.715831+0000","last_undegraded":"2024-04-24T23:27:36.715831+0000","last_fullsized":"2024-04-24T23:27:36.715831+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":4888,"log_dups_size":0,"ondisk_log_size":4888,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:03:52.974835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":297814443,"num_objects":74,"num_object_clones":0,"num_object_copies":296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74,"num_whiteouts":0,"num_read":77,"num_read_kb":0,"num_write":72362,"num_write_kb":1021329,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5,7],"acting":[6,3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.3","version":"30'8","reported_seq":45,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:31.704499+0000","last_change":"2024-04-24T23:15:36.991239+0000","last_active":"2024-04-24T23:27:31.704499+0000","last_peered":"2024-04-24T23:27:31.704499+0000","last_clean":"2024-04-24T23:27:31.704499+0000","last_became_active":"2024-04-24T23:15:36.990963+0000","last_became_peered":"2024-04-24T23:15:36.990963+0000","last_unstale":"2024-04-24T23:27:31.704499+0000","last_undegraded":"2024-04-24T23:27:31.704499+0000","last_fullsized":"2024-04-24T23:27:31.704499+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:53:28.101334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"29'42","reported_seq":91,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:26:59.824686+0000","last_change":"2024-04-24T23:15:35.957931+0000","last_active":"2024-04-24T23:26:59.824686+0000","last_peered":"2024-04-24T23:26:59.824686+0000","last_clean":"2024-04-24T23:26:59.824686+0000","last_became_active":"2024-04-24T23:15:35.957432+0000","last_became_peered":"2024-04-24T23:15:35.957432+0000","last_unstale":"2024-04-24T23:26:59.824686+0000","last_undegraded":"2024-04-24T23:26:59.824686+0000","last_fullsized":"2024-04-24T23:26:59.824686+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:11:38.773425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8389784,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":19,"num_read_kb":8206,"num_write":36,"num_write_kb":8211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.4","version":"30'4138","reported_seq":4234,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:35.795175+0000","last_change":"2024-04-24T23:15:41.007857+0000","last_active":"2024-04-24T23:27:35.795175+0000","last_peered":"2024-04-24T23:27:35.795175+0000","last_clean":"2024-04-24T23:27:35.795175+0000","last_became_active":"2024-04-24T23:15:41.007449+0000","last_became_peered":"2024-04-24T23:15:41.007449+0000","last_unstale":"2024-04-24T23:27:35.795175+0000","last_undegraded":"2024-04-24T23:27:35.795175+0000","last_fullsized":"2024-04-24T23:27:35.795175+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":4138,"log_dups_size":0,"ondisk_log_size":4138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:05:06.145440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":230763667,"num_objects":61,"num_object_clones":0,"num_object_copies":244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61,"num_whiteouts":0,"num_read":65,"num_read_kb":0,"num_write":60863,"num_write_kb":882911,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0,3],"acting":[6,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"27'11","reported_seq":48,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:25:42.717581+0000","last_change":"2024-04-24T23:15:36.970444+0000","last_active":"2024-04-24T23:25:42.717581+0000","last_peered":"2024-04-24T23:25:42.717581+0000","last_clean":"2024-04-24T23:25:42.717581+0000","last_became_active":"2024-04-24T23:15:36.970159+0000","last_became_peered":"2024-04-24T23:15:36.970159+0000","last_unstale":"2024-04-24T23:25:42.717581+0000","last_undegraded":"2024-04-24T23:25:42.717581+0000","last_fullsized":"2024-04-24T23:25:42.717581+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:37:02.956224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"30'43","reported_seq":115,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:33.304734+0000","last_change":"2024-04-24T23:15:35.945461+0000","last_active":"2024-04-24T23:27:33.304734+0000","last_peered":"2024-04-24T23:27:33.304734+0000","last_clean":"2024-04-24T23:27:33.304734+0000","last_became_active":"2024-04-24T23:15:35.945185+0000","last_became_peered":"2024-04-24T23:15:35.945185+0000","last_unstale":"2024-04-24T23:27:33.304734+0000","last_undegraded":"2024-04-24T23:27:33.304734+0000","last_fullsized":"2024-04-24T23:27:33.304734+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:37:34.175539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":94280,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":33,"num_read_kb":113,"num_write":40,"num_write_kb":118,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"30'4689","reported_seq":4794,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:35.983024+0000","last_change":"2024-04-24T23:15:41.014584+0000","last_active":"2024-04-24T23:27:35.983024+0000","last_peered":"2024-04-24T23:27:35.983024+0000","last_clean":"2024-04-24T23:27:35.983024+0000","last_became_active":"2024-04-24T23:15:41.014137+0000","last_became_peered":"2024-04-24T23:15:41.014137+0000","last_unstale":"2024-04-24T23:27:35.983024+0000","last_undegraded":"2024-04-24T23:27:35.983024+0000","last_fullsized":"2024-04-24T23:27:35.983024+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":4689,"log_dups_size":0,"ondisk_log_size":4689,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:23:50.387635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":272680432,"num_objects":72,"num_object_clones":0,"num_object_copies":288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72,"num_whiteouts":0,"num_read":74,"num_read_kb":0,"num_write":69042,"num_write_kb":969120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,7],"acting":[6,0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"30'5","reported_seq":42,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:26:06.044607+0000","last_change":"2024-04-24T23:15:36.981783+0000","last_active":"2024-04-24T23:26:06.044607+0000","last_peered":"2024-04-24T23:26:06.044607+0000","last_clean":"2024-04-24T23:26:06.044607+0000","last_became_active":"2024-04-24T23:15:36.981502+0000","last_became_peered":"2024-04-24T23:15:36.981502+0000","last_unstale":"2024-04-24T23:26:06.044607+0000","last_undegraded":"2024-04-24T23:26:06.044607+0000","last_fullsized":"2024-04-24T23:26:06.044607+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:56:08.483410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"29'34","reported_seq":75,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:25:52.069108+0000","last_change":"2024-04-24T23:15:35.960519+0000","last_active":"2024-04-24T23:25:52.069108+0000","last_peered":"2024-04-24T23:25:52.069108+0000","last_clean":"2024-04-24T23:25:52.069108+0000","last_became_active":"2024-04-24T23:15:35.960180+0000","last_became_peered":"2024-04-24T23:15:35.960180+0000","last_unstale":"2024-04-24T23:25:52.069108+0000","last_undegraded":"2024-04-24T23:25:52.069108+0000","last_fullsized":"2024-04-24T23:25:52.069108+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:50:00.199120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":8192,"num_write":28,"num_write_kb":8204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"30'3806","reported_seq":3898,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:36.034378+0000","last_change":"2024-04-24T23:15:41.012876+0000","last_active":"2024-04-24T23:27:36.034378+0000","last_peered":"2024-04-24T23:27:36.034378+0000","last_clean":"2024-04-24T23:27:36.034378+0000","last_became_active":"2024-04-24T23:15:41.012493+0000","last_became_peered":"2024-04-24T23:15:41.012493+0000","last_unstale":"2024-04-24T23:27:36.034378+0000","last_undegraded":"2024-04-24T23:27:36.034378+0000","last_fullsized":"2024-04-24T23:27:36.034378+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":3806,"log_dups_size":0,"ondisk_log_size":3806,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:21:28.287059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":218537650,"num_objects":58,"num_object_clones":0,"num_object_copies":232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58,"num_whiteouts":0,"num_read":61,"num_read_kb":307,"num_write":55909,"num_write_kb":780987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6,1],"acting":[5,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"27'4","reported_seq":41,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:25:42.712269+0000","last_change":"2024-04-24T23:15:36.981449+0000","last_active":"2024-04-24T23:25:42.712269+0000","last_peered":"2024-04-24T23:25:42.712269+0000","last_clean":"2024-04-24T23:25:42.712269+0000","last_became_active":"2024-04-24T23:15:36.980837+0000","last_became_peered":"2024-04-24T23:15:36.980837+0000","last_unstale":"2024-04-24T23:25:42.712269+0000","last_undegraded":"2024-04-24T23:25:42.712269+0000","last_fullsized":"2024-04-24T23:25:42.712269+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:15:13.513341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.3","version":"27'7","reported_seq":50,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:25:42.817265+0000","last_change":"2024-04-24T23:15:35.952947+0000","last_active":"2024-04-24T23:25:42.817265+0000","last_peered":"2024-04-24T23:25:42.817265+0000","last_clean":"2024-04-24T23:25:42.817265+0000","last_became_active":"2024-04-24T23:15:35.952655+0000","last_became_peered":"2024-04-24T23:15:35.952655+0000","last_unstale":"2024-04-24T23:25:42.817265+0000","last_undegraded":"2024-04-24T23:25:42.817265+0000","last_fullsized":"2024-04-24T23:25:42.817265+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:50:10.939396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":7,"num_read_kb":11,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":196,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:25:50.717692+0000","last_change":"2024-04-24T23:15:25.826673+0000","last_active":"2024-04-24T23:25:50.717692+0000","last_peered":"2024-04-24T23:25:50.717692+0000","last_clean":"2024-04-24T23:25:50.717692+0000","last_became_active":"2024-04-24T23:15:25.820030+0000","last_became_peered":"2024-04-24T23:15:25.820030+0000","last_unstale":"2024-04-24T23:25:50.717692+0000","last_undegraded":"2024-04-24T23:25:50.717692+0000","last_fullsized":"2024-04-24T23:25:50.717692+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:24.790928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:24.790928+0000","last_clean_scrub_stamp":"2024-04-24T23:15:24.790928+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:37:24.054332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"30'4230","reported_seq":4337,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:36.048269+0000","last_change":"2024-04-24T23:15:41.024712+0000","last_active":"2024-04-24T23:27:36.048269+0000","last_peered":"2024-04-24T23:27:36.048269+0000","last_clean":"2024-04-24T23:27:36.048269+0000","last_became_active":"2024-04-24T23:15:41.022697+0000","last_became_peered":"2024-04-24T23:15:41.022697+0000","last_unstale":"2024-04-24T23:27:36.048269+0000","last_undegraded":"2024-04-24T23:27:36.048269+0000","last_fullsized":"2024-04-24T23:27:36.048269+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":4230,"log_dups_size":0,"ondisk_log_size":4230,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:37:08.231105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":213919236,"num_objects":56,"num_object_clones":0,"num_object_copies":224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56,"num_whiteouts":0,"num_read":76,"num_read_kb":0,"num_write":62598,"num_write_kb":898105,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2,0],"acting":[3,1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.d","version":"30'8","reported_seq":45,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:28.621617+0000","last_change":"2024-04-24T23:15:37.000489+0000","last_active":"2024-04-24T23:27:28.621617+0000","last_peered":"2024-04-24T23:27:28.621617+0000","last_clean":"2024-04-24T23:27:28.621617+0000","last_became_active":"2024-04-24T23:15:37.000179+0000","last_became_peered":"2024-04-24T23:15:37.000179+0000","last_unstale":"2024-04-24T23:27:28.621617+0000","last_undegraded":"2024-04-24T23:27:28.621617+0000","last_fullsized":"2024-04-24T23:27:28.621617+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:06:37.125573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.c","version":"28'10","reported_seq":49,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:25:42.744989+0000","last_change":"2024-04-24T23:15:35.966020+0000","last_active":"2024-04-24T23:25:42.744989+0000","last_peered":"2024-04-24T23:25:42.744989+0000","last_clean":"2024-04-24T23:25:42.744989+0000","last_became_active":"2024-04-24T23:15:35.965758+0000","last_became_peered":"2024-04-24T23:15:35.965758+0000","last_unstale":"2024-04-24T23:25:42.744989+0000","last_undegraded":"2024-04-24T23:25:42.744989+0000","last_fullsized":"2024-04-24T23:25:42.744989+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:39:21.233461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2157,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"30'4343","reported_seq":4444,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:39.335432+0000","last_change":"2024-04-24T23:15:41.024489+0000","last_active":"2024-04-24T23:27:39.335432+0000","last_peered":"2024-04-24T23:27:39.335432+0000","last_clean":"2024-04-24T23:27:39.335432+0000","last_became_active":"2024-04-24T23:15:41.024041+0000","last_became_peered":"2024-04-24T23:15:41.024041+0000","last_unstale":"2024-04-24T23:27:39.335432+0000","last_undegraded":"2024-04-24T23:27:39.335432+0000","last_fullsized":"2024-04-24T23:27:39.335432+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":4343,"log_dups_size":0,"ondisk_log_size":4343,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:27:18.197003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":159390415,"num_objects":41,"num_object_clones":0,"num_object_copies":164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":70,"num_read_kb":0,"num_write":63845,"num_write_kb":912579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7,4],"acting":[2,3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"27'7","reported_seq":44,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:25:42.773524+0000","last_change":"2024-04-24T23:15:36.979887+0000","last_active":"2024-04-24T23:25:42.773524+0000","last_peered":"2024-04-24T23:25:42.773524+0000","last_clean":"2024-04-24T23:25:42.773524+0000","last_became_active":"2024-04-24T23:15:36.979611+0000","last_became_peered":"2024-04-24T23:15:36.979611+0000","last_unstale":"2024-04-24T23:25:42.773524+0000","last_undegraded":"2024-04-24T23:25:42.773524+0000","last_fullsized":"2024-04-24T23:25:42.773524+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:07:18.261077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"28'25","reported_seq":66,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:26:56.337078+0000","last_change":"2024-04-24T23:15:35.965192+0000","last_active":"2024-04-24T23:26:56.337078+0000","last_peered":"2024-04-24T23:26:56.337078+0000","last_clean":"2024-04-24T23:26:56.337078+0000","last_became_active":"2024-04-24T23:15:35.964873+0000","last_became_peered":"2024-04-24T23:15:35.964873+0000","last_unstale":"2024-04-24T23:26:56.337078+0000","last_undegraded":"2024-04-24T23:26:56.337078+0000","last_fullsized":"2024-04-24T23:26:56.337078+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:14:27.884944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194374,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":4097,"num_write":20,"num_write_kb":4108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"30'4075","reported_seq":4177,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:36.049480+0000","last_change":"2024-04-24T23:15:41.004002+0000","last_active":"2024-04-24T23:27:36.049480+0000","last_peered":"2024-04-24T23:27:36.049480+0000","last_clean":"2024-04-24T23:27:36.049480+0000","last_became_active":"2024-04-24T23:15:41.003335+0000","last_became_peered":"2024-04-24T23:15:41.003335+0000","last_unstale":"2024-04-24T23:27:36.049480+0000","last_undegraded":"2024-04-24T23:27:36.049480+0000","last_fullsized":"2024-04-24T23:27:36.049480+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":4075,"log_dups_size":0,"ondisk_log_size":4075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:49:36.206903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":209740292,"num_objects":59,"num_object_clones":0,"num_object_copies":236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59,"num_whiteouts":0,"num_read":71,"num_read_kb":0,"num_write":60093,"num_write_kb":843409,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1,3],"acting":[5,4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"27'8","reported_seq":45,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:25:42.818254+0000","last_change":"2024-04-24T23:15:37.002854+0000","last_active":"2024-04-24T23:25:42.818254+0000","last_peered":"2024-04-24T23:25:42.818254+0000","last_clean":"2024-04-24T23:25:42.818254+0000","last_became_active":"2024-04-24T23:15:37.002583+0000","last_became_peered":"2024-04-24T23:15:37.002583+0000","last_unstale":"2024-04-24T23:25:42.818254+0000","last_undegraded":"2024-04-24T23:25:42.818254+0000","last_fullsized":"2024-04-24T23:25:42.818254+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:27:04.251547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"30'166","reported_seq":213,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:28.158437+0000","last_change":"2024-04-24T23:15:35.948752+0000","last_active":"2024-04-24T23:27:28.158437+0000","last_peered":"2024-04-24T23:27:28.158437+0000","last_clean":"2024-04-24T23:27:28.158437+0000","last_became_active":"2024-04-24T23:15:35.948438+0000","last_became_peered":"2024-04-24T23:15:35.948438+0000","last_unstale":"2024-04-24T23:27:28.158437+0000","last_undegraded":"2024-04-24T23:27:28.158437+0000","last_fullsized":"2024-04-24T23:27:28.158437+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":166,"log_dups_size":0,"ondisk_log_size":166,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:45:56.006278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":21,"num_read_kb":2419,"num_write":164,"num_write_kb":4172,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.8","version":"30'4085","reported_seq":4184,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:36.007089+0000","last_change":"2024-04-24T23:15:41.017524+0000","last_active":"2024-04-24T23:27:36.007089+0000","last_peered":"2024-04-24T23:27:36.007089+0000","last_clean":"2024-04-24T23:27:36.007089+0000","last_became_active":"2024-04-24T23:15:41.017092+0000","last_became_peered":"2024-04-24T23:15:41.017092+0000","last_unstale":"2024-04-24T23:27:36.007089+0000","last_undegraded":"2024-04-24T23:27:36.007089+0000","last_fullsized":"2024-04-24T23:27:36.007089+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":4085,"log_dups_size":0,"ondisk_log_size":4085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:04:53.235038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":222316892,"num_objects":57,"num_object_clones":0,"num_object_copies":228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57,"num_whiteouts":0,"num_read":68,"num_read_kb":0,"num_write":60373,"num_write_kb":856855,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"27'4","reported_seq":41,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:25:42.735229+0000","last_change":"2024-04-24T23:15:36.990441+0000","last_active":"2024-04-24T23:25:42.735229+0000","last_peered":"2024-04-24T23:25:42.735229+0000","last_clean":"2024-04-24T23:25:42.735229+0000","last_became_active":"2024-04-24T23:15:36.989863+0000","last_became_peered":"2024-04-24T23:15:36.989863+0000","last_unstale":"2024-04-24T23:25:42.735229+0000","last_undegraded":"2024-04-24T23:25:42.735229+0000","last_fullsized":"2024-04-24T23:25:42.735229+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:50:07.975347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.f","version":"30'6","reported_seq":46,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:01.577469+0000","last_change":"2024-04-24T23:15:35.969317+0000","last_active":"2024-04-24T23:27:01.577469+0000","last_peered":"2024-04-24T23:27:01.577469+0000","last_clean":"2024-04-24T23:27:01.577469+0000","last_became_active":"2024-04-24T23:15:35.968994+0000","last_became_peered":"2024-04-24T23:15:35.968994+0000","last_unstale":"2024-04-24T23:27:01.577469+0000","last_undegraded":"2024-04-24T23:27:01.577469+0000","last_fullsized":"2024-04-24T23:27:01.577469+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:51:57.100772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"30'4118","reported_seq":4222,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:37.895480+0000","last_change":"2024-04-24T23:15:41.010103+0000","last_active":"2024-04-24T23:27:37.895480+0000","last_peered":"2024-04-24T23:27:37.895480+0000","last_clean":"2024-04-24T23:27:37.895480+0000","last_became_active":"2024-04-24T23:15:41.009328+0000","last_became_peered":"2024-04-24T23:15:41.009328+0000","last_unstale":"2024-04-24T23:27:37.895480+0000","last_undegraded":"2024-04-24T23:27:37.895480+0000","last_fullsized":"2024-04-24T23:27:37.895480+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":4118,"log_dups_size":0,"ondisk_log_size":4118,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:46:49.579501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":193017293,"num_objects":50,"num_object_clones":0,"num_object_copies":200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":73,"num_read_kb":0,"num_write":60557,"num_write_kb":866216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,2],"acting":[1,5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.11","version":"27'10","reported_seq":47,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:25:42.747653+0000","last_change":"2024-04-24T23:15:36.996765+0000","last_active":"2024-04-24T23:25:42.747653+0000","last_peered":"2024-04-24T23:25:42.747653+0000","last_clean":"2024-04-24T23:25:42.747653+0000","last_became_active":"2024-04-24T23:15:36.995801+0000","last_became_peered":"2024-04-24T23:15:36.995801+0000","last_unstale":"2024-04-24T23:25:42.747653+0000","last_undegraded":"2024-04-24T23:25:42.747653+0000","last_fullsized":"2024-04-24T23:25:42.747653+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:25:19.421667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"28'19","reported_seq":66,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:23.188825+0000","last_change":"2024-04-24T23:15:35.944023+0000","last_active":"2024-04-24T23:27:23.188825+0000","last_peered":"2024-04-24T23:27:23.188825+0000","last_clean":"2024-04-24T23:27:23.188825+0000","last_became_active":"2024-04-24T23:15:35.943710+0000","last_became_peered":"2024-04-24T23:15:35.943710+0000","last_unstale":"2024-04-24T23:27:23.188825+0000","last_undegraded":"2024-04-24T23:27:23.188825+0000","last_fullsized":"2024-04-24T23:27:23.188825+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:46:48.090958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"30'4617","reported_seq":4717,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:36.027519+0000","last_change":"2024-04-24T23:15:41.013027+0000","last_active":"2024-04-24T23:27:36.027519+0000","last_peered":"2024-04-24T23:27:36.027519+0000","last_clean":"2024-04-24T23:27:36.027519+0000","last_became_active":"2024-04-24T23:15:41.012587+0000","last_became_peered":"2024-04-24T23:15:41.012587+0000","last_unstale":"2024-04-24T23:27:36.027519+0000","last_undegraded":"2024-04-24T23:27:36.027519+0000","last_fullsized":"2024-04-24T23:27:36.027519+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":4617,"log_dups_size":0,"ondisk_log_size":4617,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:27:04.276246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":235134753,"num_objects":67,"num_object_clones":0,"num_object_copies":268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67,"num_whiteouts":0,"num_read":70,"num_read_kb":0,"num_write":67760,"num_write_kb":966739,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1,6],"acting":[5,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"30'7","reported_seq":44,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:17.987795+0000","last_change":"2024-04-24T23:15:36.979702+0000","last_active":"2024-04-24T23:27:17.987795+0000","last_peered":"2024-04-24T23:27:17.987795+0000","last_clean":"2024-04-24T23:27:17.987795+0000","last_became_active":"2024-04-24T23:15:36.979356+0000","last_became_peered":"2024-04-24T23:15:36.979356+0000","last_unstale":"2024-04-24T23:27:17.987795+0000","last_undegraded":"2024-04-24T23:27:17.987795+0000","last_fullsized":"2024-04-24T23:27:17.987795+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:14:24.309192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.11","version":"28'12","reported_seq":51,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:25:42.827287+0000","last_change":"2024-04-24T23:15:35.973026+0000","last_active":"2024-04-24T23:25:42.827287+0000","last_peered":"2024-04-24T23:25:42.827287+0000","last_clean":"2024-04-24T23:25:42.827287+0000","last_became_active":"2024-04-24T23:15:35.972655+0000","last_became_peered":"2024-04-24T23:15:35.972655+0000","last_unstale":"2024-04-24T23:25:42.827287+0000","last_undegraded":"2024-04-24T23:25:42.827287+0000","last_fullsized":"2024-04-24T23:25:42.827287+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:25:34.352908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1575,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"30'4460","reported_seq":4560,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:35.552454+0000","last_change":"2024-04-24T23:15:41.004221+0000","last_active":"2024-04-24T23:27:35.552454+0000","last_peered":"2024-04-24T23:27:35.552454+0000","last_clean":"2024-04-24T23:27:35.552454+0000","last_became_active":"2024-04-24T23:15:41.003790+0000","last_became_peered":"2024-04-24T23:15:41.003790+0000","last_unstale":"2024-04-24T23:27:35.552454+0000","last_undegraded":"2024-04-24T23:27:35.552454+0000","last_fullsized":"2024-04-24T23:27:35.552454+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":4460,"log_dups_size":0,"ondisk_log_size":4460,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:28:58.866024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":239134950,"num_objects":63,"num_object_clones":0,"num_object_copies":252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63,"num_whiteouts":0,"num_read":70,"num_read_kb":0,"num_write":65414,"num_write_kb":956853,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6,4],"acting":[5,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"27'10","reported_seq":47,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:25:42.816663+0000","last_change":"2024-04-24T23:15:36.997389+0000","last_active":"2024-04-24T23:25:42.816663+0000","last_peered":"2024-04-24T23:25:42.816663+0000","last_clean":"2024-04-24T23:25:42.816663+0000","last_became_active":"2024-04-24T23:15:36.995923+0000","last_became_peered":"2024-04-24T23:15:36.995923+0000","last_unstale":"2024-04-24T23:25:42.816663+0000","last_undegraded":"2024-04-24T23:25:42.816663+0000","last_fullsized":"2024-04-24T23:25:42.816663+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:55:47.134320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"29'7","reported_seq":50,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:25:43.845951+0000","last_change":"2024-04-24T23:15:35.957715+0000","last_active":"2024-04-24T23:25:43.845951+0000","last_peered":"2024-04-24T23:25:43.845951+0000","last_clean":"2024-04-24T23:25:43.845951+0000","last_became_active":"2024-04-24T23:15:35.957088+0000","last_became_peered":"2024-04-24T23:15:35.957088+0000","last_unstale":"2024-04-24T23:25:43.845951+0000","last_undegraded":"2024-04-24T23:25:43.845951+0000","last_fullsized":"2024-04-24T23:25:43.845951+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:30:34.483669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":7,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"30'4618","reported_seq":4720,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:35.891055+0000","last_change":"2024-04-24T23:15:41.008165+0000","last_active":"2024-04-24T23:27:35.891055+0000","last_peered":"2024-04-24T23:27:35.891055+0000","last_clean":"2024-04-24T23:27:35.891055+0000","last_became_active":"2024-04-24T23:15:41.007555+0000","last_became_peered":"2024-04-24T23:15:41.007555+0000","last_unstale":"2024-04-24T23:27:35.891055+0000","last_undegraded":"2024-04-24T23:27:35.891055+0000","last_fullsized":"2024-04-24T23:27:35.891055+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":4618,"log_dups_size":0,"ondisk_log_size":4618,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:18:08.195747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":226724555,"num_objects":60,"num_object_clones":0,"num_object_copies":240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":71,"num_read_kb":0,"num_write":67988,"num_write_kb":979211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0,7],"acting":[6,4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.12","version":"30'11","reported_seq":48,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:21.805159+0000","last_change":"2024-04-24T23:15:36.991382+0000","last_active":"2024-04-24T23:27:21.805159+0000","last_peered":"2024-04-24T23:27:21.805159+0000","last_clean":"2024-04-24T23:27:21.805159+0000","last_became_active":"2024-04-24T23:15:36.991115+0000","last_became_peered":"2024-04-24T23:15:36.991115+0000","last_unstale":"2024-04-24T23:27:21.805159+0000","last_undegraded":"2024-04-24T23:27:21.805159+0000","last_fullsized":"2024-04-24T23:27:21.805159+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:21:51.863947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"28'40","reported_seq":90,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:22.419341+0000","last_change":"2024-04-24T23:15:35.964245+0000","last_active":"2024-04-24T23:27:22.419341+0000","last_peered":"2024-04-24T23:27:22.419341+0000","last_clean":"2024-04-24T23:27:22.419341+0000","last_became_active":"2024-04-24T23:15:35.963907+0000","last_became_peered":"2024-04-24T23:15:35.963907+0000","last_unstale":"2024-04-24T23:27:22.419341+0000","last_undegraded":"2024-04-24T23:27:22.419341+0000","last_fullsized":"2024-04-24T23:27:22.419341+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:15:00.913899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388676,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":26,"num_read_kb":12314,"num_write":42,"num_write_kb":8216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"30'4224","reported_seq":4327,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:36.153694+0000","last_change":"2024-04-24T23:15:41.021981+0000","last_active":"2024-04-24T23:27:36.153694+0000","last_peered":"2024-04-24T23:27:36.153694+0000","last_clean":"2024-04-24T23:27:36.153694+0000","last_became_active":"2024-04-24T23:15:41.019193+0000","last_became_peered":"2024-04-24T23:15:41.019193+0000","last_unstale":"2024-04-24T23:27:36.153694+0000","last_undegraded":"2024-04-24T23:27:36.153694+0000","last_fullsized":"2024-04-24T23:27:36.153694+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":4224,"log_dups_size":0,"ondisk_log_size":4224,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:04:39.828502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184549376,"num_objects":44,"num_object_clones":0,"num_object_copies":176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":0,"num_read":73,"num_read_kb":0,"num_write":62280,"num_write_kb":864711,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5,6],"acting":[3,0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"30'10","reported_seq":47,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:37.674563+0000","last_change":"2024-04-24T23:15:36.986833+0000","last_active":"2024-04-24T23:27:37.674563+0000","last_peered":"2024-04-24T23:27:37.674563+0000","last_clean":"2024-04-24T23:27:37.674563+0000","last_became_active":"2024-04-24T23:15:36.986484+0000","last_became_peered":"2024-04-24T23:15:36.986484+0000","last_unstale":"2024-04-24T23:27:37.674563+0000","last_undegraded":"2024-04-24T23:27:37.674563+0000","last_fullsized":"2024-04-24T23:27:37.674563+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:12:32.245267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.14","version":"30'236","reported_seq":286,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:29.519952+0000","last_change":"2024-04-24T23:15:35.957124+0000","last_active":"2024-04-24T23:27:29.519952+0000","last_peered":"2024-04-24T23:27:29.519952+0000","last_clean":"2024-04-24T23:27:29.519952+0000","last_became_active":"2024-04-24T23:15:35.951923+0000","last_became_peered":"2024-04-24T23:15:35.951923+0000","last_unstale":"2024-04-24T23:27:29.519952+0000","last_undegraded":"2024-04-24T23:27:29.519952+0000","last_fullsized":"2024-04-24T23:27:29.519952+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":236,"log_dups_size":0,"ondisk_log_size":236,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:44:16.820640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194976,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":28,"num_read_kb":4122,"num_write":236,"num_write_kb":4244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"30'4326","reported_seq":4433,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:36.108011+0000","last_change":"2024-04-24T23:15:41.015494+0000","last_active":"2024-04-24T23:27:36.108011+0000","last_peered":"2024-04-24T23:27:36.108011+0000","last_clean":"2024-04-24T23:27:36.108011+0000","last_became_active":"2024-04-24T23:15:41.015179+0000","last_became_peered":"2024-04-24T23:15:41.015179+0000","last_unstale":"2024-04-24T23:27:36.108011+0000","last_undegraded":"2024-04-24T23:27:36.108011+0000","last_fullsized":"2024-04-24T23:27:36.108011+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":4326,"log_dups_size":0,"ondisk_log_size":4326,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:18:32.208506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":251741026,"num_objects":65,"num_object_clones":0,"num_object_copies":260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":0,"num_read":76,"num_read_kb":0,"num_write":63877,"num_write_kb":903699,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5,0],"acting":[4,3,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"30'6","reported_seq":43,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:28.524851+0000","last_change":"2024-04-24T23:15:36.999334+0000","last_active":"2024-04-24T23:27:28.524851+0000","last_peered":"2024-04-24T23:27:28.524851+0000","last_clean":"2024-04-24T23:27:28.524851+0000","last_became_active":"2024-04-24T23:15:36.995470+0000","last_became_peered":"2024-04-24T23:15:36.995470+0000","last_unstale":"2024-04-24T23:27:28.524851+0000","last_undegraded":"2024-04-24T23:27:28.524851+0000","last_fullsized":"2024-04-24T23:27:28.524851+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:19:57.973402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.15","version":"30'19","reported_seq":60,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:23.191235+0000","last_change":"2024-04-24T23:15:35.946041+0000","last_active":"2024-04-24T23:27:23.191235+0000","last_peered":"2024-04-24T23:27:23.191235+0000","last_clean":"2024-04-24T23:27:23.191235+0000","last_became_active":"2024-04-24T23:15:35.944805+0000","last_became_peered":"2024-04-24T23:15:35.944805+0000","last_unstale":"2024-04-24T23:27:23.191235+0000","last_undegraded":"2024-04-24T23:27:23.191235+0000","last_fullsized":"2024-04-24T23:27:23.191235+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:52:19.261306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":21,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"30'4379","reported_seq":4484,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:39.436460+0000","last_change":"2024-04-24T23:15:41.024392+0000","last_active":"2024-04-24T23:27:39.436460+0000","last_peered":"2024-04-24T23:27:39.436460+0000","last_clean":"2024-04-24T23:27:39.436460+0000","last_became_active":"2024-04-24T23:15:41.023951+0000","last_became_peered":"2024-04-24T23:15:41.023951+0000","last_unstale":"2024-04-24T23:27:39.436460+0000","last_undegraded":"2024-04-24T23:27:39.436460+0000","last_fullsized":"2024-04-24T23:27:39.436460+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":4379,"log_dups_size":0,"ondisk_log_size":4379,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:24:17.810117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":176268273,"num_objects":52,"num_object_clones":0,"num_object_copies":208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":0,"num_read":74,"num_read_kb":0,"num_write":64448,"num_write_kb":925608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1,7],"acting":[2,3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.17","version":"30'9","reported_seq":45,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:25:42.720085+0000","last_change":"2024-04-24T23:15:36.992243+0000","last_active":"2024-04-24T23:25:42.720085+0000","last_peered":"2024-04-24T23:25:42.720085+0000","last_clean":"2024-04-24T23:25:42.720085+0000","last_became_active":"2024-04-24T23:15:36.992002+0000","last_became_peered":"2024-04-24T23:15:36.992002+0000","last_unstale":"2024-04-24T23:25:42.720085+0000","last_undegraded":"2024-04-24T23:25:42.720085+0000","last_fullsized":"2024-04-24T23:25:42.720085+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:52:15.911009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"28'5","reported_seq":50,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:25:42.767820+0000","last_change":"2024-04-24T23:15:35.967408+0000","last_active":"2024-04-24T23:25:42.767820+0000","last_peered":"2024-04-24T23:25:42.767820+0000","last_clean":"2024-04-24T23:25:42.767820+0000","last_became_active":"2024-04-24T23:15:35.966646+0000","last_became_peered":"2024-04-24T23:15:35.966646+0000","last_unstale":"2024-04-24T23:25:42.767820+0000","last_undegraded":"2024-04-24T23:25:42.767820+0000","last_fullsized":"2024-04-24T23:25:42.767820+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:07:07.931773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":19,"num_read_kb":18,"num_write":7,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"30'4259","reported_seq":4358,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:35.920731+0000","last_change":"2024-04-24T23:15:41.021697+0000","last_active":"2024-04-24T23:27:35.920731+0000","last_peered":"2024-04-24T23:27:35.920731+0000","last_clean":"2024-04-24T23:27:35.920731+0000","last_became_active":"2024-04-24T23:15:41.021281+0000","last_became_peered":"2024-04-24T23:15:41.021281+0000","last_unstale":"2024-04-24T23:27:35.920731+0000","last_undegraded":"2024-04-24T23:27:35.920731+0000","last_fullsized":"2024-04-24T23:27:35.920731+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":4259,"log_dups_size":0,"ondisk_log_size":4259,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:59:37.306593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247532416,"num_objects":66,"num_object_clones":0,"num_object_copies":264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":68,"num_read_kb":0,"num_write":62836,"num_write_kb":881151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3,2],"acting":[7,1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"30'6","reported_seq":42,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:25:42.735229+0000","last_change":"2024-04-24T23:15:36.990988+0000","last_active":"2024-04-24T23:25:42.735229+0000","last_peered":"2024-04-24T23:25:42.735229+0000","last_clean":"2024-04-24T23:25:42.735229+0000","last_became_active":"2024-04-24T23:15:36.990711+0000","last_became_peered":"2024-04-24T23:15:36.990711+0000","last_unstale":"2024-04-24T23:25:42.735229+0000","last_undegraded":"2024-04-24T23:25:42.735229+0000","last_fullsized":"2024-04-24T23:25:42.735229+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:33:20.663776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.17","version":"30'14","reported_seq":55,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:26.156910+0000","last_change":"2024-04-24T23:15:35.957838+0000","last_active":"2024-04-24T23:27:26.156910+0000","last_peered":"2024-04-24T23:27:26.156910+0000","last_clean":"2024-04-24T23:27:26.156910+0000","last_became_active":"2024-04-24T23:15:35.956870+0000","last_became_peered":"2024-04-24T23:15:35.956870+0000","last_unstale":"2024-04-24T23:27:26.156910+0000","last_undegraded":"2024-04-24T23:27:26.156910+0000","last_fullsized":"2024-04-24T23:27:26.156910+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:56:03.889693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":13,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"30'4340","reported_seq":4435,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:36.086690+0000","last_change":"2024-04-24T23:15:41.006098+0000","last_active":"2024-04-24T23:27:36.086690+0000","last_peered":"2024-04-24T23:27:36.086690+0000","last_clean":"2024-04-24T23:27:36.086690+0000","last_became_active":"2024-04-24T23:15:41.005578+0000","last_became_peered":"2024-04-24T23:15:41.005578+0000","last_unstale":"2024-04-24T23:27:36.086690+0000","last_undegraded":"2024-04-24T23:27:36.086690+0000","last_fullsized":"2024-04-24T23:27:36.086690+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":4340,"log_dups_size":0,"ondisk_log_size":4340,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:21:12.802112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":260077201,"num_objects":68,"num_object_clones":0,"num_object_copies":272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68,"num_whiteouts":0,"num_read":64,"num_read_kb":0,"num_write":64049,"num_write_kb":908953,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3,7],"acting":[4,0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1e","version":"30'5102","reported_seq":5216,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:36.739307+0000","last_change":"2024-04-24T23:15:41.022122+0000","last_active":"2024-04-24T23:27:36.739307+0000","last_peered":"2024-04-24T23:27:36.739307+0000","last_clean":"2024-04-24T23:27:36.739307+0000","last_became_active":"2024-04-24T23:15:41.021698+0000","last_became_peered":"2024-04-24T23:15:41.021698+0000","last_unstale":"2024-04-24T23:27:36.739307+0000","last_undegraded":"2024-04-24T23:27:36.739307+0000","last_fullsized":"2024-04-24T23:27:36.739307+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":5102,"log_dups_size":0,"ondisk_log_size":5102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:13:56.730050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":314584202,"num_objects":78,"num_object_clones":0,"num_object_copies":312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":78,"num_whiteouts":0,"num_read":83,"num_read_kb":0,"num_write":75810,"num_write_kb":1069592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5,3],"acting":[4,1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"27'3","reported_seq":39,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:25:42.710767+0000","last_change":"2024-04-24T23:15:36.979816+0000","last_active":"2024-04-24T23:25:42.710767+0000","last_peered":"2024-04-24T23:25:42.710767+0000","last_clean":"2024-04-24T23:25:42.710767+0000","last_became_active":"2024-04-24T23:15:36.979553+0000","last_became_peered":"2024-04-24T23:15:36.979553+0000","last_unstale":"2024-04-24T23:25:42.710767+0000","last_undegraded":"2024-04-24T23:25:42.710767+0000","last_fullsized":"2024-04-24T23:25:42.710767+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:55:47.134320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.18","version":"29'33","reported_seq":73,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:25:53.591106+0000","last_change":"2024-04-24T23:15:35.964744+0000","last_active":"2024-04-24T23:25:53.591106+0000","last_peered":"2024-04-24T23:25:53.591106+0000","last_clean":"2024-04-24T23:25:53.591106+0000","last_became_active":"2024-04-24T23:15:35.964434+0000","last_became_peered":"2024-04-24T23:15:35.964434+0000","last_unstale":"2024-04-24T23:25:53.591106+0000","last_undegraded":"2024-04-24T23:25:53.591106+0000","last_fullsized":"2024-04-24T23:25:53.591106+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:26:32.587197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8389190,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":8192,"num_write":27,"num_write_kb":8217,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"30'4492","reported_seq":4592,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:35.983834+0000","last_change":"2024-04-24T23:15:41.022032+0000","last_active":"2024-04-24T23:27:35.983834+0000","last_peered":"2024-04-24T23:27:35.983834+0000","last_clean":"2024-04-24T23:27:35.983834+0000","last_became_active":"2024-04-24T23:15:41.021616+0000","last_became_peered":"2024-04-24T23:15:41.021616+0000","last_unstale":"2024-04-24T23:27:35.983834+0000","last_undegraded":"2024-04-24T23:27:35.983834+0000","last_fullsized":"2024-04-24T23:27:35.983834+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":4492,"log_dups_size":0,"ondisk_log_size":4492,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:08:59.242392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":239158476,"num_objects":68,"num_object_clones":0,"num_object_copies":272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68,"num_whiteouts":0,"num_read":69,"num_read_kb":0,"num_write":66209,"num_write_kb":941784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5,1],"acting":[3,2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"30'15","reported_seq":51,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:25:42.717761+0000","last_change":"2024-04-24T23:15:36.967198+0000","last_active":"2024-04-24T23:25:42.717761+0000","last_peered":"2024-04-24T23:25:42.717761+0000","last_clean":"2024-04-24T23:25:42.717761+0000","last_became_active":"2024-04-24T23:15:36.966910+0000","last_became_peered":"2024-04-24T23:15:36.966910+0000","last_unstale":"2024-04-24T23:25:42.717761+0000","last_undegraded":"2024-04-24T23:25:42.717761+0000","last_fullsized":"2024-04-24T23:25:42.717761+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:51:19.788437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"28'6","reported_seq":46,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:25:42.740651+0000","last_change":"2024-04-24T23:15:35.967169+0000","last_active":"2024-04-24T23:25:42.740651+0000","last_peered":"2024-04-24T23:25:42.740651+0000","last_clean":"2024-04-24T23:25:42.740651+0000","last_became_active":"2024-04-24T23:15:35.966939+0000","last_became_peered":"2024-04-24T23:15:35.966939+0000","last_unstale":"2024-04-24T23:25:42.740651+0000","last_undegraded":"2024-04-24T23:25:42.740651+0000","last_fullsized":"2024-04-24T23:25:42.740651+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:38:55.460569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":14,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"30'17","reported_seq":56,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:26:59.813995+0000","last_change":"2024-04-24T23:15:35.971236+0000","last_active":"2024-04-24T23:26:59.813995+0000","last_peered":"2024-04-24T23:26:59.813995+0000","last_clean":"2024-04-24T23:26:59.813995+0000","last_became_active":"2024-04-24T23:15:35.970908+0000","last_became_peered":"2024-04-24T23:15:35.970908+0000","last_unstale":"2024-04-24T23:26:59.813995+0000","last_undegraded":"2024-04-24T23:26:59.813995+0000","last_fullsized":"2024-04-24T23:26:59.813995+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:14:15.548024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":1,"num_read_kb":4096,"num_write":17,"num_write_kb":4106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"27'8","reported_seq":44,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:25:42.714394+0000","last_change":"2024-04-24T23:15:36.988156+0000","last_active":"2024-04-24T23:25:42.714394+0000","last_peered":"2024-04-24T23:25:42.714394+0000","last_clean":"2024-04-24T23:25:42.714394+0000","last_became_active":"2024-04-24T23:15:36.987896+0000","last_became_peered":"2024-04-24T23:15:36.987896+0000","last_unstale":"2024-04-24T23:25:42.714394+0000","last_undegraded":"2024-04-24T23:25:42.714394+0000","last_fullsized":"2024-04-24T23:25:42.714394+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:22:06.869644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"30'3603","reported_seq":3693,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:35.997627+0000","last_change":"2024-04-24T23:15:40.997559+0000","last_active":"2024-04-24T23:27:35.997627+0000","last_peered":"2024-04-24T23:27:35.997627+0000","last_clean":"2024-04-24T23:27:35.997627+0000","last_became_active":"2024-04-24T23:15:40.997102+0000","last_became_peered":"2024-04-24T23:15:40.997102+0000","last_unstale":"2024-04-24T23:27:35.997627+0000","last_undegraded":"2024-04-24T23:27:35.997627+0000","last_fullsized":"2024-04-24T23:27:35.997627+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":3603,"log_dups_size":0,"ondisk_log_size":3603,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:25:02.046761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":163579583,"num_objects":40,"num_object_clones":0,"num_object_copies":160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":60,"num_read_kb":0,"num_write":53429,"num_write_kb":749552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,7],"acting":[4,6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"30'1217","reported_seq":1259,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:25:51.681033+0000","last_change":"2024-04-24T23:15:35.944842+0000","last_active":"2024-04-24T23:25:51.681033+0000","last_peered":"2024-04-24T23:25:51.681033+0000","last_clean":"2024-04-24T23:25:51.681033+0000","last_became_active":"2024-04-24T23:15:35.944563+0000","last_became_peered":"2024-04-24T23:15:35.944563+0000","last_unstale":"2024-04-24T23:25:51.681033+0000","last_undegraded":"2024-04-24T23:25:51.681033+0000","last_fullsized":"2024-04-24T23:25:51.681033+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":1217,"log_dups_size":0,"ondisk_log_size":1217,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:40:58.933384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4195441,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":4,"num_read_kb":4098,"num_write":1212,"num_write_kb":5208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"27'8","reported_seq":44,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:25:42.807080+0000","last_change":"2024-04-24T23:15:36.981103+0000","last_active":"2024-04-24T23:25:42.807080+0000","last_peered":"2024-04-24T23:25:42.807080+0000","last_clean":"2024-04-24T23:25:42.807080+0000","last_became_active":"2024-04-24T23:15:36.979637+0000","last_became_peered":"2024-04-24T23:15:36.979637+0000","last_unstale":"2024-04-24T23:25:42.807080+0000","last_undegraded":"2024-04-24T23:25:42.807080+0000","last_fullsized":"2024-04-24T23:25:42.807080+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:09:39.750272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"30'5005","reported_seq":5112,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:36.669829+0000","last_change":"2024-04-24T23:15:41.017087+0000","last_active":"2024-04-24T23:27:36.669829+0000","last_peered":"2024-04-24T23:27:36.669829+0000","last_clean":"2024-04-24T23:27:36.669829+0000","last_became_active":"2024-04-24T23:15:41.016696+0000","last_became_peered":"2024-04-24T23:15:41.016696+0000","last_unstale":"2024-04-24T23:27:36.669829+0000","last_undegraded":"2024-04-24T23:27:36.669829+0000","last_fullsized":"2024-04-24T23:27:36.669829+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":5005,"log_dups_size":0,"ondisk_log_size":5005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:05:57.823447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268544512,"num_objects":71,"num_object_clones":0,"num_object_copies":284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71,"num_whiteouts":0,"num_read":77,"num_read_kb":0,"num_write":73985,"num_write_kb":1014885,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1,0],"acting":[7,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"30'9","reported_seq":46,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:31.253765+0000","last_change":"2024-04-24T23:15:36.970211+0000","last_active":"2024-04-24T23:27:31.253765+0000","last_peered":"2024-04-24T23:27:31.253765+0000","last_clean":"2024-04-24T23:27:31.253765+0000","last_became_active":"2024-04-24T23:15:36.969090+0000","last_became_peered":"2024-04-24T23:15:36.969090+0000","last_unstale":"2024-04-24T23:27:31.253765+0000","last_undegraded":"2024-04-24T23:27:31.253765+0000","last_fullsized":"2024-04-24T23:27:31.253765+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:16:00.231012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.21","version":"30'12","reported_seq":49,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:26:05.729132+0000","last_change":"2024-04-24T23:15:36.982528+0000","last_active":"2024-04-24T23:26:05.729132+0000","last_peered":"2024-04-24T23:26:05.729132+0000","last_clean":"2024-04-24T23:26:05.729132+0000","last_became_active":"2024-04-24T23:15:36.982299+0000","last_became_peered":"2024-04-24T23:15:36.982299+0000","last_unstale":"2024-04-24T23:26:05.729132+0000","last_undegraded":"2024-04-24T23:26:05.729132+0000","last_fullsized":"2024-04-24T23:26:05.729132+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:02:17.732324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.22","version":"30'10","reported_seq":47,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:28.417018+0000","last_change":"2024-04-24T23:15:36.989899+0000","last_active":"2024-04-24T23:27:28.417018+0000","last_peered":"2024-04-24T23:27:28.417018+0000","last_clean":"2024-04-24T23:27:28.417018+0000","last_became_active":"2024-04-24T23:15:36.989557+0000","last_became_peered":"2024-04-24T23:15:36.989557+0000","last_unstale":"2024-04-24T23:27:28.417018+0000","last_undegraded":"2024-04-24T23:27:28.417018+0000","last_fullsized":"2024-04-24T23:27:28.417018+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:15:13.513341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"30'9","reported_seq":46,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:26:05.334192+0000","last_change":"2024-04-24T23:15:36.969628+0000","last_active":"2024-04-24T23:26:05.334192+0000","last_peered":"2024-04-24T23:26:05.334192+0000","last_clean":"2024-04-24T23:26:05.334192+0000","last_became_active":"2024-04-24T23:15:36.969327+0000","last_became_peered":"2024-04-24T23:15:36.969327+0000","last_unstale":"2024-04-24T23:26:05.334192+0000","last_undegraded":"2024-04-24T23:26:05.334192+0000","last_fullsized":"2024-04-24T23:26:05.334192+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:08:47.021999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.24","version":"30'8","reported_seq":45,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:26:31.416932+0000","last_change":"2024-04-24T23:15:36.986343+0000","last_active":"2024-04-24T23:26:31.416932+0000","last_peered":"2024-04-24T23:26:31.416932+0000","last_clean":"2024-04-24T23:26:31.416932+0000","last_became_active":"2024-04-24T23:15:36.986026+0000","last_became_peered":"2024-04-24T23:15:36.986026+0000","last_unstale":"2024-04-24T23:26:31.416932+0000","last_undegraded":"2024-04-24T23:26:31.416932+0000","last_fullsized":"2024-04-24T23:26:31.416932+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:10:00.347826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"30'7","reported_seq":44,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:26:31.416961+0000","last_change":"2024-04-24T23:15:36.968538+0000","last_active":"2024-04-24T23:26:31.416961+0000","last_peered":"2024-04-24T23:26:31.416961+0000","last_clean":"2024-04-24T23:26:31.416961+0000","last_became_active":"2024-04-24T23:15:36.968263+0000","last_became_peered":"2024-04-24T23:15:36.968263+0000","last_unstale":"2024-04-24T23:26:31.416961+0000","last_undegraded":"2024-04-24T23:26:31.416961+0000","last_fullsized":"2024-04-24T23:26:31.416961+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:07:43.731738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.26","version":"30'5","reported_seq":42,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:26:33.160187+0000","last_change":"2024-04-24T23:15:36.991008+0000","last_active":"2024-04-24T23:26:33.160187+0000","last_peered":"2024-04-24T23:26:33.160187+0000","last_clean":"2024-04-24T23:26:33.160187+0000","last_became_active":"2024-04-24T23:15:36.990764+0000","last_became_peered":"2024-04-24T23:15:36.990764+0000","last_unstale":"2024-04-24T23:26:33.160187+0000","last_undegraded":"2024-04-24T23:26:33.160187+0000","last_fullsized":"2024-04-24T23:26:33.160187+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:50:21.865233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"30'10","reported_seq":47,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:26:31.284444+0000","last_change":"2024-04-24T23:15:36.993550+0000","last_active":"2024-04-24T23:26:31.284444+0000","last_peered":"2024-04-24T23:26:31.284444+0000","last_clean":"2024-04-24T23:26:31.284444+0000","last_became_active":"2024-04-24T23:15:36.993310+0000","last_became_peered":"2024-04-24T23:15:36.993310+0000","last_unstale":"2024-04-24T23:26:31.284444+0000","last_undegraded":"2024-04-24T23:26:31.284444+0000","last_fullsized":"2024-04-24T23:26:31.284444+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:21:20.617940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.28","version":"30'8","reported_seq":45,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:26:34.899945+0000","last_change":"2024-04-24T23:15:36.987810+0000","last_active":"2024-04-24T23:26:34.899945+0000","last_peered":"2024-04-24T23:26:34.899945+0000","last_clean":"2024-04-24T23:26:34.899945+0000","last_became_active":"2024-04-24T23:15:36.987540+0000","last_became_peered":"2024-04-24T23:15:36.987540+0000","last_unstale":"2024-04-24T23:26:34.899945+0000","last_undegraded":"2024-04-24T23:26:34.899945+0000","last_fullsized":"2024-04-24T23:26:34.899945+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:49:56.176021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.29","version":"30'13","reported_seq":50,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:31.113409+0000","last_change":"2024-04-24T23:15:36.981187+0000","last_active":"2024-04-24T23:27:31.113409+0000","last_peered":"2024-04-24T23:27:31.113409+0000","last_clean":"2024-04-24T23:27:31.113409+0000","last_became_active":"2024-04-24T23:15:36.980838+0000","last_became_peered":"2024-04-24T23:15:36.980838+0000","last_unstale":"2024-04-24T23:27:31.113409+0000","last_undegraded":"2024-04-24T23:27:31.113409+0000","last_fullsized":"2024-04-24T23:27:31.113409+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:57:41.178367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2a","version":"30'11","reported_seq":48,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:27.562545+0000","last_change":"2024-04-24T23:15:36.999238+0000","last_active":"2024-04-24T23:27:27.562545+0000","last_peered":"2024-04-24T23:27:27.562545+0000","last_clean":"2024-04-24T23:27:27.562545+0000","last_became_active":"2024-04-24T23:15:36.996245+0000","last_became_peered":"2024-04-24T23:15:36.996245+0000","last_unstale":"2024-04-24T23:27:27.562545+0000","last_undegraded":"2024-04-24T23:27:27.562545+0000","last_fullsized":"2024-04-24T23:27:27.562545+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:21:20.443640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"30'9","reported_seq":46,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:26:57.039900+0000","last_change":"2024-04-24T23:15:36.981219+0000","last_active":"2024-04-24T23:26:57.039900+0000","last_peered":"2024-04-24T23:26:57.039900+0000","last_clean":"2024-04-24T23:26:57.039900+0000","last_became_active":"2024-04-24T23:15:36.980911+0000","last_became_peered":"2024-04-24T23:15:36.980911+0000","last_unstale":"2024-04-24T23:26:57.039900+0000","last_undegraded":"2024-04-24T23:26:57.039900+0000","last_fullsized":"2024-04-24T23:26:57.039900+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:19:57.973402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2c","version":"30'7","reported_seq":44,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:02.756928+0000","last_change":"2024-04-24T23:15:36.991418+0000","last_active":"2024-04-24T23:27:02.756928+0000","last_peered":"2024-04-24T23:27:02.756928+0000","last_clean":"2024-04-24T23:27:02.756928+0000","last_became_active":"2024-04-24T23:15:36.989792+0000","last_became_peered":"2024-04-24T23:15:36.989792+0000","last_unstale":"2024-04-24T23:27:02.756928+0000","last_undegraded":"2024-04-24T23:27:02.756928+0000","last_fullsized":"2024-04-24T23:27:02.756928+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:02:53.601311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2d","version":"30'12","reported_seq":49,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:28.000118+0000","last_change":"2024-04-24T23:15:36.987810+0000","last_active":"2024-04-24T23:27:28.000118+0000","last_peered":"2024-04-24T23:27:28.000118+0000","last_clean":"2024-04-24T23:27:28.000118+0000","last_became_active":"2024-04-24T23:15:36.987494+0000","last_became_peered":"2024-04-24T23:15:36.987494+0000","last_unstale":"2024-04-24T23:27:28.000118+0000","last_undegraded":"2024-04-24T23:27:28.000118+0000","last_fullsized":"2024-04-24T23:27:28.000118+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:43:21.141172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2e","version":"30'10","reported_seq":47,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:17.418750+0000","last_change":"2024-04-24T23:15:36.986576+0000","last_active":"2024-04-24T23:27:17.418750+0000","last_peered":"2024-04-24T23:27:17.418750+0000","last_clean":"2024-04-24T23:27:17.418750+0000","last_became_active":"2024-04-24T23:15:36.986272+0000","last_became_peered":"2024-04-24T23:15:36.986272+0000","last_unstale":"2024-04-24T23:27:17.418750+0000","last_undegraded":"2024-04-24T23:27:17.418750+0000","last_fullsized":"2024-04-24T23:27:17.418750+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:55:08.894945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2f","version":"30'8","reported_seq":50,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:33.578178+0000","last_change":"2024-04-24T23:16:18.294151+0000","last_active":"2024-04-24T23:27:33.578178+0000","last_peered":"2024-04-24T23:27:33.578178+0000","last_clean":"2024-04-24T23:27:33.578178+0000","last_became_active":"2024-04-24T23:16:18.293579+0000","last_became_peered":"2024-04-24T23:16:18.293579+0000","last_unstale":"2024-04-24T23:27:33.578178+0000","last_undegraded":"2024-04-24T23:27:33.578178+0000","last_fullsized":"2024-04-24T23:27:33.578178+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:22:20.603555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.30","version":"30'9","reported_seq":46,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:26.286870+0000","last_change":"2024-04-24T23:15:36.996521+0000","last_active":"2024-04-24T23:27:26.286870+0000","last_peered":"2024-04-24T23:27:26.286870+0000","last_clean":"2024-04-24T23:27:26.286870+0000","last_became_active":"2024-04-24T23:15:36.995507+0000","last_became_peered":"2024-04-24T23:15:36.995507+0000","last_unstale":"2024-04-24T23:27:26.286870+0000","last_undegraded":"2024-04-24T23:27:26.286870+0000","last_fullsized":"2024-04-24T23:27:26.286870+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:58:50.153471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.31","version":"30'11","reported_seq":48,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:21.339459+0000","last_change":"2024-04-24T23:15:36.984994+0000","last_active":"2024-04-24T23:27:21.339459+0000","last_peered":"2024-04-24T23:27:21.339459+0000","last_clean":"2024-04-24T23:27:21.339459+0000","last_became_active":"2024-04-24T23:15:36.984741+0000","last_became_peered":"2024-04-24T23:15:36.984741+0000","last_unstale":"2024-04-24T23:27:21.339459+0000","last_undegraded":"2024-04-24T23:27:21.339459+0000","last_fullsized":"2024-04-24T23:27:21.339459+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:45:23.429836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.32","version":"30'10","reported_seq":47,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:21.805138+0000","last_change":"2024-04-24T23:15:36.966165+0000","last_active":"2024-04-24T23:27:21.805138+0000","last_peered":"2024-04-24T23:27:21.805138+0000","last_clean":"2024-04-24T23:27:21.805138+0000","last_became_active":"2024-04-24T23:15:36.965877+0000","last_became_peered":"2024-04-24T23:15:36.965877+0000","last_unstale":"2024-04-24T23:27:21.805138+0000","last_undegraded":"2024-04-24T23:27:21.805138+0000","last_fullsized":"2024-04-24T23:27:21.805138+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:30:12.390978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.33","version":"30'9","reported_seq":45,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:25:42.741941+0000","last_change":"2024-04-24T23:15:36.990667+0000","last_active":"2024-04-24T23:25:42.741941+0000","last_peered":"2024-04-24T23:25:42.741941+0000","last_clean":"2024-04-24T23:25:42.741941+0000","last_became_active":"2024-04-24T23:15:36.990383+0000","last_became_peered":"2024-04-24T23:15:36.990383+0000","last_unstale":"2024-04-24T23:25:42.741941+0000","last_undegraded":"2024-04-24T23:25:42.741941+0000","last_fullsized":"2024-04-24T23:25:42.741941+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:36:43.850046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"27'6","reported_seq":43,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:25:42.773242+0000","last_change":"2024-04-24T23:15:36.991686+0000","last_active":"2024-04-24T23:25:42.773242+0000","last_peered":"2024-04-24T23:25:42.773242+0000","last_clean":"2024-04-24T23:25:42.773242+0000","last_became_active":"2024-04-24T23:15:36.990155+0000","last_became_peered":"2024-04-24T23:15:36.990155+0000","last_unstale":"2024-04-24T23:25:42.773242+0000","last_undegraded":"2024-04-24T23:25:42.773242+0000","last_fullsized":"2024-04-24T23:25:42.773242+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:00:52.424019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.35","version":"30'12","reported_seq":48,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:25:42.767307+0000","last_change":"2024-04-24T23:15:36.992958+0000","last_active":"2024-04-24T23:25:42.767307+0000","last_peered":"2024-04-24T23:25:42.767307+0000","last_clean":"2024-04-24T23:25:42.767307+0000","last_became_active":"2024-04-24T23:15:36.992608+0000","last_became_peered":"2024-04-24T23:15:36.992608+0000","last_unstale":"2024-04-24T23:25:42.767307+0000","last_undegraded":"2024-04-24T23:25:42.767307+0000","last_fullsized":"2024-04-24T23:25:42.767307+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:15:30.416763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.36","version":"30'10","reported_seq":46,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:25:42.767610+0000","last_change":"2024-04-24T23:15:36.992943+0000","last_active":"2024-04-24T23:25:42.767610+0000","last_peered":"2024-04-24T23:25:42.767610+0000","last_clean":"2024-04-24T23:25:42.767610+0000","last_became_active":"2024-04-24T23:15:36.992567+0000","last_became_peered":"2024-04-24T23:15:36.992567+0000","last_unstale":"2024-04-24T23:25:42.767610+0000","last_undegraded":"2024-04-24T23:25:42.767610+0000","last_fullsized":"2024-04-24T23:25:42.767610+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:50:43.209232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.37","version":"30'12","reported_seq":48,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:27.436849+0000","last_change":"2024-04-24T23:15:36.999697+0000","last_active":"2024-04-24T23:27:27.436849+0000","last_peered":"2024-04-24T23:27:27.436849+0000","last_clean":"2024-04-24T23:27:27.436849+0000","last_became_active":"2024-04-24T23:15:36.999413+0000","last_became_peered":"2024-04-24T23:15:36.999413+0000","last_unstale":"2024-04-24T23:27:27.436849+0000","last_undegraded":"2024-04-24T23:27:27.436849+0000","last_fullsized":"2024-04-24T23:27:27.436849+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:25:42.856471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"27'3","reported_seq":39,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:25:42.717468+0000","last_change":"2024-04-24T23:15:36.986742+0000","last_active":"2024-04-24T23:25:42.717468+0000","last_peered":"2024-04-24T23:25:42.717468+0000","last_clean":"2024-04-24T23:25:42.717468+0000","last_became_active":"2024-04-24T23:15:36.986013+0000","last_became_peered":"2024-04-24T23:15:36.986013+0000","last_unstale":"2024-04-24T23:25:42.717468+0000","last_undegraded":"2024-04-24T23:25:42.717468+0000","last_fullsized":"2024-04-24T23:25:42.717468+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:16:15.203151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"30'4","reported_seq":40,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:25:42.827178+0000","last_change":"2024-04-24T23:15:36.984336+0000","last_active":"2024-04-24T23:25:42.827178+0000","last_peered":"2024-04-24T23:25:42.827178+0000","last_clean":"2024-04-24T23:25:42.827178+0000","last_became_active":"2024-04-24T23:15:36.983966+0000","last_became_peered":"2024-04-24T23:15:36.983966+0000","last_unstale":"2024-04-24T23:25:42.827178+0000","last_undegraded":"2024-04-24T23:25:42.827178+0000","last_fullsized":"2024-04-24T23:25:42.827178+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:54:16.319608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3a","version":"30'9","reported_seq":45,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:36.677504+0000","last_change":"2024-04-24T23:15:36.979079+0000","last_active":"2024-04-24T23:27:36.677504+0000","last_peered":"2024-04-24T23:27:36.677504+0000","last_clean":"2024-04-24T23:27:36.677504+0000","last_became_active":"2024-04-24T23:15:36.978706+0000","last_became_peered":"2024-04-24T23:15:36.978706+0000","last_unstale":"2024-04-24T23:27:36.677504+0000","last_undegraded":"2024-04-24T23:27:36.677504+0000","last_fullsized":"2024-04-24T23:27:36.677504+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:53:39.888150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"27'6","reported_seq":42,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:25:42.769902+0000","last_change":"2024-04-24T23:15:36.986008+0000","last_active":"2024-04-24T23:25:42.769902+0000","last_peered":"2024-04-24T23:25:42.769902+0000","last_clean":"2024-04-24T23:25:42.769902+0000","last_became_active":"2024-04-24T23:15:36.985552+0000","last_became_peered":"2024-04-24T23:15:36.985552+0000","last_unstale":"2024-04-24T23:25:42.769902+0000","last_undegraded":"2024-04-24T23:25:42.769902+0000","last_fullsized":"2024-04-24T23:25:42.769902+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:36:50.827503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"30'6","reported_seq":43,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:25:44.818461+0000","last_change":"2024-04-24T23:15:36.978673+0000","last_active":"2024-04-24T23:25:44.818461+0000","last_peered":"2024-04-24T23:25:44.818461+0000","last_clean":"2024-04-24T23:25:44.818461+0000","last_became_active":"2024-04-24T23:15:36.972744+0000","last_became_peered":"2024-04-24T23:15:36.972744+0000","last_unstale":"2024-04-24T23:25:44.818461+0000","last_undegraded":"2024-04-24T23:25:44.818461+0000","last_fullsized":"2024-04-24T23:25:44.818461+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:28:57.028825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3d","version":"30'7","reported_seq":44,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:26:05.334229+0000","last_change":"2024-04-24T23:15:36.970238+0000","last_active":"2024-04-24T23:26:05.334229+0000","last_peered":"2024-04-24T23:26:05.334229+0000","last_clean":"2024-04-24T23:26:05.334229+0000","last_became_active":"2024-04-24T23:15:36.969751+0000","last_became_peered":"2024-04-24T23:15:36.969751+0000","last_unstale":"2024-04-24T23:26:05.334229+0000","last_undegraded":"2024-04-24T23:26:05.334229+0000","last_fullsized":"2024-04-24T23:26:05.334229+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:19:56.456801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3e","version":"26'2","reported_seq":39,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:25:42.719286+0000","last_change":"2024-04-24T23:15:36.968825+0000","last_active":"2024-04-24T23:25:42.719286+0000","last_peered":"2024-04-24T23:25:42.719286+0000","last_clean":"2024-04-24T23:25:42.719286+0000","last_became_active":"2024-04-24T23:15:36.968543+0000","last_became_peered":"2024-04-24T23:15:36.968543+0000","last_unstale":"2024-04-24T23:25:42.719286+0000","last_undegraded":"2024-04-24T23:25:42.719286+0000","last_fullsized":"2024-04-24T23:25:42.719286+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:36:48.795062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3f","version":"27'4","reported_seq":41,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:25:42.818133+0000","last_change":"2024-04-24T23:15:36.992800+0000","last_active":"2024-04-24T23:25:42.818133+0000","last_peered":"2024-04-24T23:25:42.818133+0000","last_clean":"2024-04-24T23:25:42.818133+0000","last_became_active":"2024-04-24T23:15:36.992476+0000","last_became_peered":"2024-04-24T23:15:36.992476+0000","last_unstale":"2024-04-24T23:25:42.818133+0000","last_undegraded":"2024-04-24T23:25:42.818133+0000","last_fullsized":"2024-04-24T23:25:42.818133+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:01:21.103559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":7111802757,"num_objects":1886,"num_object_clones":0,"num_object_copies":7544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1886,"num_whiteouts":0,"num_read":2252,"num_read_kb":307,"num_write":2058856,"num_write_kb":29192446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14351450112,"data_stored":14351450112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139710,"ondisk_log_size":139710,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":514,"ondisk_log_size":514,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":80027978,"num_objects":127,"num_object_clones":0,"num_object_copies":254,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":127,"num_whiteouts":0,"num_read":353,"num_read_kb":84742,"num_write":4101,"num_write_kb":81389,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":74,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":160292864,"data_stored":160055956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3839,"ondisk_log_size":3839,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574998,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3104376,"kb_used_data":2063984,"kb_used_omap":1,"kb_used_meta":1040382,"kb_avail":91267464,"statfs":{"total":96636764160,"available":93457883136,"internally_reserved":0,"allocated":2113519616,"data_stored":2113298368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":1065351627},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574998,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2884404,"kb_used_data":1938668,"kb_used_omap":1,"kb_used_meta":945726,"kb_avail":91487436,"statfs":{"total":96636764160,"available":93683134464,"internally_reserved":0,"allocated":1985196032,"data_stored":1984994624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":968423878},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574998,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2748124,"kb_used_data":1955668,"kb_used_omap":1,"kb_used_meta":792446,"kb_avail":91623716,"statfs":{"total":96636764160,"available":93822685184,"internally_reserved":0,"allocated":2002604032,"data_stored":2002391666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":811465157},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574999,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2501912,"kb_used_data":1727440,"kb_used_omap":1,"kb_used_meta":774462,"kb_avail":91869928,"statfs":{"total":96636764160,"available":94074806272,"internally_reserved":0,"allocated":1768898560,"data_stored":1768662424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":793049541},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574994,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3261728,"kb_used_data":2229400,"kb_used_omap":1,"kb_used_meta":1032318,"kb_avail":91110112,"statfs":{"total":96636764160,"available":93296754688,"internally_reserved":0,"allocated":2282905600,"data_stored":2282679443,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":1057094087},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,5,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574994,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2047988,"kb_used_data":1323948,"kb_used_omap":1,"kb_used_meta":724030,"kb_avail":92323852,"statfs":{"total":96636764160,"available":94539624448,"internally_reserved":0,"allocated":1355722752,"data_stored":1355521710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":741407173},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574994,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3096668,"kb_used_data":2038100,"kb_used_omap":1,"kb_used_meta":1054462,"kb_avail":91275172,"statfs":{"total":96636764160,"available":93465776128,"internally_reserved":0,"allocated":2087014400,"data_stored":2086804775,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1079769541},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574997,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1929760,"kb_used_data":1281112,"kb_used_omap":1,"kb_used_meta":648638,"kb_avail":92442080,"statfs":{"total":96636764160,"available":94660689920,"internally_reserved":0,"allocated":1311858688,"data_stored":1311639333,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":664205766},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":25436160,"data_stored":25402203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":21225472,"data_stored":21204797,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12599296,"data_stored":12583652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":37793792,"data_stored":37753033,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12636160,"data_stored":12585422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":25284608,"data_stored":25261333,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12603392,"data_stored":12587382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12713984,"data_stored":12678134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1210597376,"data_stored":1210597376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2048090112,"data_stored":2048090112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1273589760,"data_stored":1273589760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2242686976,"data_stored":2242686976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1737060352,"data_stored":1737060352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1789751296,"data_stored":1789751296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1949196288,"data_stored":1949196288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2100477952,"data_stored":2100477952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T23:27:42.663 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-24T23:27:42.664 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T23:27:42.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:42.988+0000 7f48a8000640 1 Processor -- start 2024-04-24T23:27:42.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:42.988+0000 7f48a8000640 1 -- start start 2024-04-24T23:27:42.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:42.991+0000 7f48a8000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f48a00c81c0 0x7f48a0162ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:42.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:42.991+0000 7f48a8000640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f48a00c7230 con 0x7f48a0159110 2024-04-24T23:27:42.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:42.991+0000 7f48a8000640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f48a00c74d0 con 0x7f48a00c7dc0 2024-04-24T23:27:42.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:42.991+0000 7f48a8000640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f48a01594f0 con 0x7f48a00c81c0 2024-04-24T23:27:42.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:42.991+0000 7f48a4c00640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f48a00c7dc0 0x7f48a01623a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.6:44638/0 (socket says 172.21.15.6:44638) 2024-04-24T23:27:42.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:42.991+0000 7f48a4c00640 1 -- 172.21.15.6:0/166580601 learned_addr learned my addr 172.21.15.6:0/166580601 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:27:42.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:42.992+0000 7f48a6000640 1 --2- 172.21.15.6:0/166580601 >> v2:172.21.15.6:3300/0 conn(0x7f48a00c81c0 0x7f48a0162ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:42.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:42.992+0000 7f489fe00640 1 -- 172.21.15.6:0/166580601 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 609506551 0 0) 0x7f48a00c74d0 con 0x7f48a00c7dc0 2024-04-24T23:27:42.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:42.992+0000 7f489fe00640 1 -- 172.21.15.6:0/166580601 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4888003520 con 0x7f48a00c7dc0 2024-04-24T23:27:42.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:42.992+0000 7f48a6000640 1 -- 172.21.15.6:0/166580601 >> v1:172.21.15.190:6789/0 conn(0x7f48a00c7dc0 legacy=0x7f48a01623a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:42.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:42.992+0000 7f48a6000640 1 -- 172.21.15.6:0/166580601 >> v1:172.21.15.6:6789/0 conn(0x7f48a0159110 legacy=0x7f48a0072420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:42.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:42.992+0000 7f48a6000640 1 -- 172.21.15.6:0/166580601 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48a006e440 con 0x7f48a00c81c0 2024-04-24T23:27:42.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:42.994+0000 7f48a6000640 1 --2- 172.21.15.6:0/166580601 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f48a00c81c0 0x7f48a0162ab0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f48980097d0 tx=0x7f489802a300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88ac395e232606e8 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:27:42.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:42.994+0000 7f489fe00640 1 -- 172.21.15.6:0/166580601 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4898037070 con 0x7f48a00c81c0 2024-04-24T23:27:42.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:42.994+0000 7f489fe00640 1 -- 172.21.15.6:0/166580601 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4898032640 con 0x7f48a00c81c0 2024-04-24T23:27:42.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:42.994+0000 7f489fe00640 1 -- 172.21.15.6:0/166580601 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4898032a10 con 0x7f48a00c81c0 2024-04-24T23:27:42.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:42.995+0000 7f48a8000640 1 -- 172.21.15.6:0/166580601 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f48a00c81c0 msgr2=0x7f48a0162ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:42.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:42.996+0000 7f48a8000640 1 --2- 172.21.15.6:0/166580601 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f48a00c81c0 0x7f48a0162ab0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f48980097d0 tx=0x7f489802a300 comp rx=0 tx=0).stop 2024-04-24T23:27:42.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:42.997+0000 7f48a8000640 1 -- 172.21.15.6:0/166580601 shutdown_connections 2024-04-24T23:27:42.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:42.997+0000 7f48a8000640 1 --2- 172.21.15.6:0/166580601 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f48a00c81c0 0x7f48a0162ab0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:42.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:42.997+0000 7f48a8000640 1 -- 172.21.15.6:0/166580601 >> 172.21.15.6:0/166580601 conn(0x7f48a006b0e0 msgr2=0x7f48a006b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:27:42.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:42.997+0000 7f48a8000640 1 -- 172.21.15.6:0/166580601 shutdown_connections 2024-04-24T23:27:42.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:42.997+0000 7f48a8000640 1 -- 172.21.15.6:0/166580601 wait complete. 2024-04-24T23:27:43.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:43.000+0000 7f48a8000640 1 Processor -- start 2024-04-24T23:27:43.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:43.000+0000 7f48a8000640 1 -- start start 2024-04-24T23:27:43.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:43.002+0000 7f48a8000640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f48a00c7dc0 0x7f48a00ca360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:43.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:43.002+0000 7f48a8000640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f48a00c81c0 0x7f48a01596a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:43.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:43.002+0000 7f48a8000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f48a0159110 0x7f48a0159be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:43.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:43.004+0000 7f48a4c00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f48a00c81c0 0x7f48a01596a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:43.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:43.004+0000 7f48a8000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f48a006ece0 con 0x7f48a0159110 2024-04-24T23:27:43.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:43.004+0000 7f48a8000640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f48a006ea00 con 0x7f48a00c81c0 2024-04-24T23:27:43.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:43.004+0000 7f48a8000640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f48a006f200 con 0x7f48a00c7dc0 2024-04-24T23:27:43.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:43.004+0000 7f48a5600640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f48a00c7dc0 0x7f48a00ca360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:43.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:43.004+0000 7f48a6000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f48a0159110 0x7f48a0159be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:43.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:43.004+0000 7f48a6000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f48a0159110 0x7f48a0159be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:33536/0 (socket says 172.21.15.6:33536) 2024-04-24T23:27:43.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:43.004+0000 7f48a6000640 1 -- 172.21.15.6:0/311214909 learned_addr learned my addr 172.21.15.6:0/311214909 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:27:43.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:43.005+0000 7f48a6000640 1 -- 172.21.15.6:0/311214909 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f48a00c7dc0 msgr2=0x7f48a00ca360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:43.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:43.005+0000 7f48a6000640 1 --2- 172.21.15.6:0/311214909 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f48a00c7dc0 0x7f48a00ca360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:43.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:43.005+0000 7f48a6000640 1 -- 172.21.15.6:0/311214909 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f48a00c81c0 msgr2=0x7f48a01596a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:43.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:43.005+0000 7f48a6000640 1 --2- 172.21.15.6:0/311214909 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f48a00c81c0 0x7f48a01596a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:43.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:43.005+0000 7f48a6000640 1 -- 172.21.15.6:0/311214909 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48a00c5ff0 con 0x7f48a0159110 2024-04-24T23:27:43.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:43.006+0000 7f48a5600640 1 --2- 172.21.15.6:0/311214909 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f48a00c7dc0 0x7f48a00ca360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:27:43.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:43.006+0000 7f48a6000640 1 --2- 172.21.15.6:0/311214909 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f48a0159110 0x7f48a0159be0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f48980097d0 tx=0x7f48980046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:27:43.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:43.007+0000 7f489d600640 1 -- 172.21.15.6:0/311214909 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4898040020 con 0x7f48a0159110 2024-04-24T23:27:43.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:43.007+0000 7f489d600640 1 -- 172.21.15.6:0/311214909 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4898038040 con 0x7f48a0159110 2024-04-24T23:27:43.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:43.007+0000 7f489d600640 1 -- 172.21.15.6:0/311214909 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4898037070 con 0x7f48a0159110 2024-04-24T23:27:43.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:43.007+0000 7f48a8000640 1 -- 172.21.15.6:0/311214909 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48a00c6290 con 0x7f48a0159110 2024-04-24T23:27:43.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:43.009+0000 7f48a8000640 1 -- 172.21.15.6:0/311214909 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48a00c4cb0 con 0x7f48a0159110 2024-04-24T23:27:43.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:43.011+0000 7f48a8000640 1 -- 172.21.15.6:0/311214909 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f48a00c74d0 con 0x7f48a0159110 2024-04-24T23:27:43.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:43.017+0000 7f489d600640 1 -- 172.21.15.6:0/311214909 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f4898043020 con 0x7f48a0159110 2024-04-24T23:27:43.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:43.018+0000 7f489d600640 1 --2- 172.21.15.6:0/311214909 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f488006bf30 0x7f488006e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:43.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:43.018+0000 7f489d600640 1 -- 172.21.15.6:0/311214909 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6934+0+0 (secure 0 0 0) 0x7f48980ab1c0 con 0x7f48a0159110 2024-04-24T23:27:43.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:43.020+0000 7f48a5600640 1 --2- 172.21.15.6:0/311214909 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f488006bf30 0x7f488006e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:43.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:43.022+0000 7f489d600640 1 -- 172.21.15.6:0/311214909 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f48a00c74d0 con 0x7f48a0159110 2024-04-24T23:27:43.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:43.026+0000 7f48a5600640 1 --2- 172.21.15.6:0/311214909 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f488006bf30 0x7f488006e3f0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f48940012e0 tx=0x7f4894009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:27:43.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:43.447+0000 7f48a8000640 1 -- 172.21.15.6:0/311214909 --> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f48a00767b0 con 0x7f488006bf30 2024-04-24T23:27:43.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:43.464+0000 7f489d600640 1 -- 172.21.15.6:0/311214909 <== mgr.4105 v2:172.21.15.190:6832/806769663 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+334810 (secure 0 0 0) 0x7f48a00767b0 con 0x7f488006bf30 2024-04-24T23:27:43.470 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:27:43.472 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-04-24T23:27:43.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:43.472+0000 7f48a8000640 1 -- 172.21.15.6:0/311214909 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f488006bf30 msgr2=0x7f488006e3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:43.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:43.472+0000 7f48a8000640 1 --2- 172.21.15.6:0/311214909 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f488006bf30 0x7f488006e3f0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f48940012e0 tx=0x7f4894009040 comp rx=0 tx=0).stop 2024-04-24T23:27:43.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:43.472+0000 7f48a8000640 1 -- 172.21.15.6:0/311214909 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f48a0159110 msgr2=0x7f48a0159be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:43.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:43.472+0000 7f48a8000640 1 --2- 172.21.15.6:0/311214909 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f48a0159110 0x7f48a0159be0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f48980097d0 tx=0x7f48980046c0 comp rx=0 tx=0).stop 2024-04-24T23:27:43.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:43.474+0000 7f48a8000640 1 -- 172.21.15.6:0/311214909 shutdown_connections 2024-04-24T23:27:43.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:43.474+0000 7f48a8000640 1 --2- 172.21.15.6:0/311214909 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f488006bf30 0x7f488006e3f0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:43.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:43.474+0000 7f48a8000640 1 --2- 172.21.15.6:0/311214909 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f48a0159110 0x7f48a0159be0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:43.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:43.474+0000 7f48a8000640 1 --2- 172.21.15.6:0/311214909 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f48a00c81c0 0x7f48a01596a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:43.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:43.474+0000 7f48a8000640 1 --2- 172.21.15.6:0/311214909 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f48a00c7dc0 0x7f48a00ca360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:43.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:43.474+0000 7f48a8000640 1 -- 172.21.15.6:0/311214909 >> 172.21.15.6:0/311214909 conn(0x7f48a006b0e0 msgr2=0x7f48a0155470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:27:43.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:43.474+0000 7f48a8000640 1 -- 172.21.15.6:0/311214909 shutdown_connections 2024-04-24T23:27:43.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:43.476+0000 7f48a8000640 1 -- 172.21.15.6:0/311214909 wait complete. 2024-04-24T23:27:43.511 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":392,"stamp":"2024-04-24T23:27:42.872194+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4780696303,"num_objects":1440,"num_object_clones":0,"num_object_copies":5502,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1440,"num_whiteouts":0,"num_read":2721,"num_read_kb":85149,"num_write":2063694,"num_write_kb":29276252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":74,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":144778,"ondisk_log_size":144778,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":17253680,"kb_used_data":10171504,"kb_used_omap":12,"kb_used_meta":7071859,"kb_avail":737721040,"statfs":{"total":773094113280,"available":755426344960,"internally_reserved":0,"allocated":10415620096,"data_stored":10413892900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":7241584178},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,2,3,7,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":69,"apply_latency_ms":69,"commit_latency_ns":69000000,"apply_latency_ns":69000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-7393949078,"num_objects":-1762,"num_object_clones":0,"num_object_copies":-7050,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1762,"num_whiteouts":0,"num_read":2,"num_read_kb":27,"num_write":1826,"num_write_kb":628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003079"},"pg_stats":[{"pgid":"2.1f","version":"28'36","reported_seq":80,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:26:05.796187+0000","last_change":"2024-04-24T23:15:35.964646+0000","last_active":"2024-04-24T23:26:05.796187+0000","last_peered":"2024-04-24T23:26:05.796187+0000","last_clean":"2024-04-24T23:26:05.796187+0000","last_became_active":"2024-04-24T23:15:35.964357+0000","last_became_peered":"2024-04-24T23:15:35.964357+0000","last_unstale":"2024-04-24T23:26:05.796187+0000","last_undegraded":"2024-04-24T23:26:05.796187+0000","last_fullsized":"2024-04-24T23:26:05.796187+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:01:17.912993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194976,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":7,"num_read_kb":4102,"num_write":33,"num_write_kb":4110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"27'3","reported_seq":41,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:25:42.719837+0000","last_change":"2024-04-24T23:15:36.989772+0000","last_active":"2024-04-24T23:25:42.719837+0000","last_peered":"2024-04-24T23:25:42.719837+0000","last_clean":"2024-04-24T23:25:42.719837+0000","last_became_active":"2024-04-24T23:15:36.989543+0000","last_became_peered":"2024-04-24T23:15:36.989543+0000","last_unstale":"2024-04-24T23:25:42.719837+0000","last_undegraded":"2024-04-24T23:25:42.719837+0000","last_fullsized":"2024-04-24T23:25:42.719837+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:37:16.737354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"30'4729","reported_seq":4823,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:42.417319+0000","last_change":"2024-04-24T23:16:18.300096+0000","last_active":"2024-04-24T23:27:42.417319+0000","last_peered":"2024-04-24T23:27:42.417319+0000","last_clean":"2024-04-24T23:27:42.417319+0000","last_became_active":"2024-04-24T23:16:18.299217+0000","last_became_peered":"2024-04-24T23:16:18.299217+0000","last_unstale":"2024-04-24T23:27:42.417319+0000","last_undegraded":"2024-04-24T23:27:42.417319+0000","last_fullsized":"2024-04-24T23:27:42.417319+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":4729,"log_dups_size":0,"ondisk_log_size":4729,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:08:19.426862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":138530823,"num_objects":43,"num_object_clones":0,"num_object_copies":172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":70,"num_read_kb":0,"num_write":69544,"num_write_kb":969009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4,1],"acting":[0,6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"28'8","reported_seq":49,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:25:51.674845+0000","last_change":"2024-04-24T23:15:35.946083+0000","last_active":"2024-04-24T23:25:51.674845+0000","last_peered":"2024-04-24T23:25:51.674845+0000","last_clean":"2024-04-24T23:25:51.674845+0000","last_became_active":"2024-04-24T23:15:35.945227+0000","last_became_peered":"2024-04-24T23:15:35.945227+0000","last_unstale":"2024-04-24T23:25:51.674845+0000","last_undegraded":"2024-04-24T23:25:51.674845+0000","last_fullsized":"2024-04-24T23:25:51.674845+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:05:17.727725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":4096,"num_write":7,"num_write_kb":4105,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1f","version":"30'7","reported_seq":44,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:26:05.334280+0000","last_change":"2024-04-24T23:15:36.998245+0000","last_active":"2024-04-24T23:26:05.334280+0000","last_peered":"2024-04-24T23:26:05.334280+0000","last_clean":"2024-04-24T23:26:05.334280+0000","last_became_active":"2024-04-24T23:15:36.998021+0000","last_became_peered":"2024-04-24T23:15:36.998021+0000","last_unstale":"2024-04-24T23:26:05.334280+0000","last_undegraded":"2024-04-24T23:26:05.334280+0000","last_fullsized":"2024-04-24T23:26:05.334280+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:03:42.359117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"30'4538","reported_seq":4643,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:39.372938+0000","last_change":"2024-04-24T23:15:41.008313+0000","last_active":"2024-04-24T23:27:39.372938+0000","last_peered":"2024-04-24T23:27:39.372938+0000","last_clean":"2024-04-24T23:27:39.372938+0000","last_became_active":"2024-04-24T23:15:41.007908+0000","last_became_peered":"2024-04-24T23:15:41.007908+0000","last_unstale":"2024-04-24T23:27:39.372938+0000","last_undegraded":"2024-04-24T23:27:39.372938+0000","last_fullsized":"2024-04-24T23:27:39.372938+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":4538,"log_dups_size":0,"ondisk_log_size":4538,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:02:11.902575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":176186537,"num_objects":46,"num_object_clones":0,"num_object_copies":184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":75,"num_read_kb":0,"num_write":66573,"num_write_kb":930809,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4,6],"acting":[2,7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1d","version":"30'40","reported_seq":92,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:23.196288+0000","last_change":"2024-04-24T23:15:35.968312+0000","last_active":"2024-04-24T23:27:23.196288+0000","last_peered":"2024-04-24T23:27:23.196288+0000","last_clean":"2024-04-24T23:27:23.196288+0000","last_became_active":"2024-04-24T23:15:35.967873+0000","last_became_peered":"2024-04-24T23:15:35.967873+0000","last_unstale":"2024-04-24T23:27:23.196288+0000","last_undegraded":"2024-04-24T23:27:23.196288+0000","last_fullsized":"2024-04-24T23:27:23.196288+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:21:40.116977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":38,"num_read_kb":8226,"num_write":37,"num_write_kb":8205,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"27'7","reported_seq":43,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:25:42.821075+0000","last_change":"2024-04-24T23:15:36.997361+0000","last_active":"2024-04-24T23:25:42.821075+0000","last_peered":"2024-04-24T23:25:42.821075+0000","last_clean":"2024-04-24T23:25:42.821075+0000","last_became_active":"2024-04-24T23:15:36.997083+0000","last_became_peered":"2024-04-24T23:15:36.997083+0000","last_unstale":"2024-04-24T23:25:42.821075+0000","last_undegraded":"2024-04-24T23:25:42.821075+0000","last_fullsized":"2024-04-24T23:25:42.821075+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:47:59.334093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"30'4235","reported_seq":4335,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:39.416539+0000","last_change":"2024-04-24T23:15:41.025153+0000","last_active":"2024-04-24T23:27:39.416539+0000","last_peered":"2024-04-24T23:27:39.416539+0000","last_clean":"2024-04-24T23:27:39.416539+0000","last_became_active":"2024-04-24T23:15:41.024739+0000","last_became_peered":"2024-04-24T23:15:41.024739+0000","last_unstale":"2024-04-24T23:27:39.416539+0000","last_undegraded":"2024-04-24T23:27:39.416539+0000","last_fullsized":"2024-04-24T23:27:39.416539+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":4235,"log_dups_size":0,"ondisk_log_size":4235,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:31:00.160213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184565702,"num_objects":54,"num_object_clones":0,"num_object_copies":216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":70,"num_read_kb":0,"num_write":62343,"num_write_kb":877947,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3,6],"acting":[2,5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1c","version":"30'17","reported_seq":63,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:36.086972+0000","last_change":"2024-04-24T23:15:35.967338+0000","last_active":"2024-04-24T23:27:36.086972+0000","last_peered":"2024-04-24T23:27:36.086972+0000","last_clean":"2024-04-24T23:27:36.086972+0000","last_became_active":"2024-04-24T23:15:35.966951+0000","last_became_peered":"2024-04-24T23:15:35.966951+0000","last_unstale":"2024-04-24T23:27:36.086972+0000","last_undegraded":"2024-04-24T23:27:36.086972+0000","last_fullsized":"2024-04-24T23:27:36.086972+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:08:05.248330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":232445,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":21,"num_read_kb":20,"num_write":17,"num_write_kb":234,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1d","version":"30'8","reported_seq":44,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:27.171671+0000","last_change":"2024-04-24T23:15:36.998072+0000","last_active":"2024-04-24T23:27:27.171671+0000","last_peered":"2024-04-24T23:27:27.171671+0000","last_clean":"2024-04-24T23:27:27.171671+0000","last_became_active":"2024-04-24T23:15:36.997735+0000","last_became_peered":"2024-04-24T23:15:36.997735+0000","last_unstale":"2024-04-24T23:27:27.171671+0000","last_undegraded":"2024-04-24T23:27:27.171671+0000","last_fullsized":"2024-04-24T23:27:27.171671+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:22:06.869644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"30'3724","reported_seq":3813,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:39.072670+0000","last_change":"2024-04-24T23:15:41.008895+0000","last_active":"2024-04-24T23:27:39.072670+0000","last_peered":"2024-04-24T23:27:39.072670+0000","last_clean":"2024-04-24T23:27:39.072670+0000","last_became_active":"2024-04-24T23:15:41.006199+0000","last_became_peered":"2024-04-24T23:15:41.006199+0000","last_unstale":"2024-04-24T23:27:39.072670+0000","last_undegraded":"2024-04-24T23:27:39.072670+0000","last_fullsized":"2024-04-24T23:27:39.072670+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":3724,"log_dups_size":0,"ondisk_log_size":3724,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:13:34.583602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184573346,"num_objects":48,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":59,"num_read_kb":0,"num_write":54757,"num_write_kb":774507,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5,7],"acting":[2,6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"30'8","reported_seq":45,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:26:34.900000+0000","last_change":"2024-04-24T23:15:37.004074+0000","last_active":"2024-04-24T23:26:34.900000+0000","last_peered":"2024-04-24T23:26:34.900000+0000","last_clean":"2024-04-24T23:26:34.900000+0000","last_became_active":"2024-04-24T23:15:37.003544+0000","last_became_peered":"2024-04-24T23:15:37.003544+0000","last_unstale":"2024-04-24T23:26:34.900000+0000","last_undegraded":"2024-04-24T23:26:34.900000+0000","last_fullsized":"2024-04-24T23:26:34.900000+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:38:28.024702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"30'1458","reported_seq":1502,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:26.160448+0000","last_change":"2024-04-24T23:15:35.963000+0000","last_active":"2024-04-24T23:27:26.160448+0000","last_peered":"2024-04-24T23:27:26.160448+0000","last_clean":"2024-04-24T23:27:26.160448+0000","last_became_active":"2024-04-24T23:15:35.961960+0000","last_became_peered":"2024-04-24T23:15:35.961960+0000","last_unstale":"2024-04-24T23:27:26.160448+0000","last_undegraded":"2024-04-24T23:27:26.160448+0000","last_fullsized":"2024-04-24T23:27:26.160448+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":1458,"log_dups_size":0,"ondisk_log_size":1458,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:56:26.651193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":10,"num_read_kb":8200,"num_write":1452,"num_write_kb":5429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"30'4346","reported_seq":4442,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:41.735928+0000","last_change":"2024-04-24T23:15:41.022481+0000","last_active":"2024-04-24T23:27:41.735928+0000","last_peered":"2024-04-24T23:27:41.735928+0000","last_clean":"2024-04-24T23:27:41.735928+0000","last_became_active":"2024-04-24T23:15:41.020034+0000","last_became_peered":"2024-04-24T23:15:41.020034+0000","last_unstale":"2024-04-24T23:27:41.735928+0000","last_undegraded":"2024-04-24T23:27:41.735928+0000","last_fullsized":"2024-04-24T23:27:41.735928+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":4346,"log_dups_size":0,"ondisk_log_size":4346,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:27:25.337503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100808244,"num_objects":31,"num_object_clones":0,"num_object_copies":124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":65,"num_read_kb":0,"num_write":63631,"num_write_kb":904417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,5],"acting":[3,1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"30'6","reported_seq":43,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:26:52.416421+0000","last_change":"2024-04-24T23:15:36.972020+0000","last_active":"2024-04-24T23:26:52.416421+0000","last_peered":"2024-04-24T23:26:52.416421+0000","last_clean":"2024-04-24T23:26:52.416421+0000","last_became_active":"2024-04-24T23:15:36.971583+0000","last_became_peered":"2024-04-24T23:15:36.971583+0000","last_unstale":"2024-04-24T23:26:52.416421+0000","last_undegraded":"2024-04-24T23:26:52.416421+0000","last_fullsized":"2024-04-24T23:26:52.416421+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:29:05.568608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"30'7","reported_seq":47,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:26:31.416252+0000","last_change":"2024-04-24T23:15:35.945916+0000","last_active":"2024-04-24T23:26:31.416252+0000","last_peered":"2024-04-24T23:26:31.416252+0000","last_clean":"2024-04-24T23:26:31.416252+0000","last_became_active":"2024-04-24T23:15:35.945610+0000","last_became_peered":"2024-04-24T23:15:35.945610+0000","last_unstale":"2024-04-24T23:26:31.416252+0000","last_undegraded":"2024-04-24T23:26:31.416252+0000","last_fullsized":"2024-04-24T23:26:31.416252+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:57:05.976074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"30'4410","reported_seq":4511,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:42.710892+0000","last_change":"2024-04-24T23:15:41.030165+0000","last_active":"2024-04-24T23:27:42.710892+0000","last_peered":"2024-04-24T23:27:42.710892+0000","last_clean":"2024-04-24T23:27:42.710892+0000","last_became_active":"2024-04-24T23:15:41.029685+0000","last_became_peered":"2024-04-24T23:15:41.029685+0000","last_unstale":"2024-04-24T23:27:42.710892+0000","last_undegraded":"2024-04-24T23:27:42.710892+0000","last_fullsized":"2024-04-24T23:27:42.710892+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":4410,"log_dups_size":0,"ondisk_log_size":4410,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:20:43.498190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79710240,"num_objects":27,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":70,"num_read_kb":0,"num_write":64519,"num_write_kb":914755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6,1],"acting":[0,4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"30'8","reported_seq":45,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:26:30.926375+0000","last_change":"2024-04-24T23:15:36.965949+0000","last_active":"2024-04-24T23:26:30.926375+0000","last_peered":"2024-04-24T23:26:30.926375+0000","last_clean":"2024-04-24T23:26:30.926375+0000","last_became_active":"2024-04-24T23:15:36.965658+0000","last_became_peered":"2024-04-24T23:15:36.965658+0000","last_unstale":"2024-04-24T23:26:30.926375+0000","last_undegraded":"2024-04-24T23:26:30.926375+0000","last_fullsized":"2024-04-24T23:26:30.926375+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:38:28.024702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"28'8","reported_seq":54,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:25:42.710930+0000","last_change":"2024-04-24T23:15:35.972165+0000","last_active":"2024-04-24T23:25:42.710930+0000","last_peered":"2024-04-24T23:25:42.710930+0000","last_clean":"2024-04-24T23:25:42.710930+0000","last_became_active":"2024-04-24T23:15:35.971972+0000","last_became_peered":"2024-04-24T23:15:35.971972+0000","last_unstale":"2024-04-24T23:25:42.710930+0000","last_undegraded":"2024-04-24T23:25:42.710930+0000","last_fullsized":"2024-04-24T23:25:42.710930+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:11:12.088698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":20,"num_read_kb":19,"num_write":11,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"30'5175","reported_seq":5287,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:42.313803+0000","last_change":"2024-04-24T23:15:41.019799+0000","last_active":"2024-04-24T23:27:42.313803+0000","last_peered":"2024-04-24T23:27:42.313803+0000","last_clean":"2024-04-24T23:27:42.313803+0000","last_became_active":"2024-04-24T23:15:41.019339+0000","last_became_peered":"2024-04-24T23:15:41.019339+0000","last_unstale":"2024-04-24T23:27:42.313803+0000","last_undegraded":"2024-04-24T23:27:42.313803+0000","last_fullsized":"2024-04-24T23:27:42.313803+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":5175,"log_dups_size":0,"ondisk_log_size":5175,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:59:34.601004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":155286645,"num_objects":42,"num_object_clones":0,"num_object_copies":168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":81,"num_read_kb":0,"num_write":75863,"num_write_kb":1074791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2,3],"acting":[7,4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"30'6","reported_seq":43,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:41.442462+0000","last_change":"2024-04-24T23:15:36.991991+0000","last_active":"2024-04-24T23:27:41.442462+0000","last_peered":"2024-04-24T23:27:41.442462+0000","last_clean":"2024-04-24T23:27:41.442462+0000","last_became_active":"2024-04-24T23:15:36.989336+0000","last_became_peered":"2024-04-24T23:15:36.989336+0000","last_unstale":"2024-04-24T23:27:41.442462+0000","last_undegraded":"2024-04-24T23:27:41.442462+0000","last_fullsized":"2024-04-24T23:27:41.442462+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:25:56.038903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.8","version":"28'31","reported_seq":73,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:26:32.455095+0000","last_change":"2024-04-24T23:15:35.965443+0000","last_active":"2024-04-24T23:26:32.455095+0000","last_peered":"2024-04-24T23:26:32.455095+0000","last_clean":"2024-04-24T23:26:32.455095+0000","last_became_active":"2024-04-24T23:15:35.965043+0000","last_became_peered":"2024-04-24T23:15:35.965043+0000","last_unstale":"2024-04-24T23:26:32.455095+0000","last_undegraded":"2024-04-24T23:26:32.455095+0000","last_fullsized":"2024-04-24T23:26:32.455095+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:07:23.320042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":4102,"num_write":30,"num_write_kb":4111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"30'3783","reported_seq":3875,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:39.129224+0000","last_change":"2024-04-24T23:15:41.025131+0000","last_active":"2024-04-24T23:27:39.129224+0000","last_peered":"2024-04-24T23:27:39.129224+0000","last_clean":"2024-04-24T23:27:39.129224+0000","last_became_active":"2024-04-24T23:15:41.024731+0000","last_became_peered":"2024-04-24T23:15:41.024731+0000","last_unstale":"2024-04-24T23:27:39.129224+0000","last_undegraded":"2024-04-24T23:27:39.129224+0000","last_fullsized":"2024-04-24T23:27:39.129224+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":3783,"log_dups_size":0,"ondisk_log_size":3783,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:28:07.952854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180570789,"num_objects":53,"num_object_clones":0,"num_object_copies":212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":61,"num_read_kb":0,"num_write":55520,"num_write_kb":793811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,7],"acting":[2,1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"30'15","reported_seq":52,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:26:31.027261+0000","last_change":"2024-04-24T23:15:36.965850+0000","last_active":"2024-04-24T23:26:31.027261+0000","last_peered":"2024-04-24T23:26:31.027261+0000","last_clean":"2024-04-24T23:26:31.027261+0000","last_became_active":"2024-04-24T23:15:36.965547+0000","last_became_peered":"2024-04-24T23:15:36.965547+0000","last_unstale":"2024-04-24T23:26:31.027261+0000","last_undegraded":"2024-04-24T23:26:31.027261+0000","last_fullsized":"2024-04-24T23:26:31.027261+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:06:37.125573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"30'9","reported_seq":56,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:23.206469+0000","last_change":"2024-04-24T23:15:35.971040+0000","last_active":"2024-04-24T23:27:23.206469+0000","last_peered":"2024-04-24T23:27:23.206469+0000","last_clean":"2024-04-24T23:27:23.206469+0000","last_became_active":"2024-04-24T23:15:35.970184+0000","last_became_peered":"2024-04-24T23:15:35.970184+0000","last_unstale":"2024-04-24T23:27:23.206469+0000","last_undegraded":"2024-04-24T23:27:23.206469+0000","last_fullsized":"2024-04-24T23:27:23.206469+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:45:20.913759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":24,"num_write":11,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"30'4371","reported_seq":4476,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:38.781850+0000","last_change":"2024-04-24T23:15:41.025813+0000","last_active":"2024-04-24T23:27:38.781850+0000","last_peered":"2024-04-24T23:27:38.781850+0000","last_clean":"2024-04-24T23:27:38.781850+0000","last_became_active":"2024-04-24T23:15:41.025137+0000","last_became_peered":"2024-04-24T23:15:41.025137+0000","last_unstale":"2024-04-24T23:27:38.781850+0000","last_undegraded":"2024-04-24T23:27:38.781850+0000","last_fullsized":"2024-04-24T23:27:38.781850+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":4371,"log_dups_size":0,"ondisk_log_size":4371,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:35:17.784854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":205532220,"num_objects":52,"num_object_clones":0,"num_object_copies":208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":0,"num_read":74,"num_read_kb":0,"num_write":64361,"num_write_kb":917129,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,3,1],"acting":[2,6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"27'10","reported_seq":47,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:25:42.711648+0000","last_change":"2024-04-24T23:15:36.967476+0000","last_active":"2024-04-24T23:25:42.711648+0000","last_peered":"2024-04-24T23:25:42.711648+0000","last_clean":"2024-04-24T23:25:42.711648+0000","last_became_active":"2024-04-24T23:15:36.967098+0000","last_became_peered":"2024-04-24T23:15:36.967098+0000","last_unstale":"2024-04-24T23:25:42.711648+0000","last_undegraded":"2024-04-24T23:25:42.711648+0000","last_fullsized":"2024-04-24T23:25:42.711648+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:25:56.038903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"30'262","reported_seq":302,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:34.158914+0000","last_change":"2024-04-24T23:15:35.970630+0000","last_active":"2024-04-24T23:27:34.158914+0000","last_peered":"2024-04-24T23:27:34.158914+0000","last_clean":"2024-04-24T23:27:34.158914+0000","last_became_active":"2024-04-24T23:15:35.970430+0000","last_became_peered":"2024-04-24T23:15:35.970430+0000","last_unstale":"2024-04-24T23:27:34.158914+0000","last_undegraded":"2024-04-24T23:27:34.158914+0000","last_fullsized":"2024-04-24T23:27:34.158914+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":262,"log_dups_size":0,"ondisk_log_size":262,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:15:59.219084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":9,"num_write":561,"num_write_kb":230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"30'4741","reported_seq":4849,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:41.646388+0000","last_change":"2024-04-24T23:15:41.005022+0000","last_active":"2024-04-24T23:27:41.646388+0000","last_peered":"2024-04-24T23:27:41.646388+0000","last_clean":"2024-04-24T23:27:41.646388+0000","last_became_active":"2024-04-24T23:15:41.004648+0000","last_became_peered":"2024-04-24T23:15:41.004648+0000","last_unstale":"2024-04-24T23:27:41.646388+0000","last_undegraded":"2024-04-24T23:27:41.646388+0000","last_fullsized":"2024-04-24T23:27:41.646388+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":4741,"log_dups_size":0,"ondisk_log_size":4741,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:06:01.559760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121650469,"num_objects":35,"num_object_clones":0,"num_object_copies":140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":77,"num_read_kb":0,"num_write":69549,"num_write_kb":1012990,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6,3],"acting":[4,1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.4","version":"30'4","reported_seq":41,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:26:06.045309+0000","last_change":"2024-04-24T23:15:36.967053+0000","last_active":"2024-04-24T23:26:06.045309+0000","last_peered":"2024-04-24T23:26:06.045309+0000","last_clean":"2024-04-24T23:26:06.045309+0000","last_became_active":"2024-04-24T23:15:36.966746+0000","last_became_peered":"2024-04-24T23:15:36.966746+0000","last_unstale":"2024-04-24T23:26:06.045309+0000","last_undegraded":"2024-04-24T23:26:06.045309+0000","last_fullsized":"2024-04-24T23:26:06.045309+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:16:08.779860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"26'4","reported_seq":46,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:25:42.781078+0000","last_change":"2024-04-24T23:15:35.965890+0000","last_active":"2024-04-24T23:25:42.781078+0000","last_peered":"2024-04-24T23:25:42.781078+0000","last_clean":"2024-04-24T23:25:42.781078+0000","last_became_active":"2024-04-24T23:15:35.965631+0000","last_became_peered":"2024-04-24T23:15:35.965631+0000","last_unstale":"2024-04-24T23:25:42.781078+0000","last_undegraded":"2024-04-24T23:25:42.781078+0000","last_fullsized":"2024-04-24T23:25:42.781078+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:17:35.650731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"30'4014","reported_seq":4105,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:41.484376+0000","last_change":"2024-04-24T23:15:41.011405+0000","last_active":"2024-04-24T23:27:41.484376+0000","last_peered":"2024-04-24T23:27:41.484376+0000","last_clean":"2024-04-24T23:27:41.484376+0000","last_became_active":"2024-04-24T23:15:41.010982+0000","last_became_peered":"2024-04-24T23:15:41.010982+0000","last_unstale":"2024-04-24T23:27:41.484376+0000","last_undegraded":"2024-04-24T23:27:41.484376+0000","last_fullsized":"2024-04-24T23:27:41.484376+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":4014,"log_dups_size":0,"ondisk_log_size":4014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:08:00.914296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117449587,"num_objects":32,"num_object_clones":0,"num_object_copies":128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":60,"num_read_kb":0,"num_write":58622,"num_write_kb":828032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2,7],"acting":[6,0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"30'10","reported_seq":47,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:27.778246+0000","last_change":"2024-04-24T23:15:36.975244+0000","last_active":"2024-04-24T23:27:27.778246+0000","last_peered":"2024-04-24T23:27:27.778246+0000","last_clean":"2024-04-24T23:27:27.778246+0000","last_became_active":"2024-04-24T23:15:36.974872+0000","last_became_peered":"2024-04-24T23:15:36.974872+0000","last_unstale":"2024-04-24T23:27:27.778246+0000","last_undegraded":"2024-04-24T23:27:27.778246+0000","last_fullsized":"2024-04-24T23:27:27.778246+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:45:57.018205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"27'2","reported_seq":41,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:25:42.718435+0000","last_change":"2024-04-24T23:15:35.945792+0000","last_active":"2024-04-24T23:25:42.718435+0000","last_peered":"2024-04-24T23:25:42.718435+0000","last_clean":"2024-04-24T23:25:42.718435+0000","last_became_active":"2024-04-24T23:15:35.945094+0000","last_became_peered":"2024-04-24T23:15:35.945094+0000","last_unstale":"2024-04-24T23:25:42.718435+0000","last_undegraded":"2024-04-24T23:25:42.718435+0000","last_fullsized":"2024-04-24T23:25:42.718435+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:49:12.719127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"30'4911","reported_seq":5019,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:41.447723+0000","last_change":"2024-04-24T23:15:41.022105+0000","last_active":"2024-04-24T23:27:41.447723+0000","last_peered":"2024-04-24T23:27:41.447723+0000","last_clean":"2024-04-24T23:27:41.447723+0000","last_became_active":"2024-04-24T23:15:41.021645+0000","last_became_peered":"2024-04-24T23:15:41.021645+0000","last_unstale":"2024-04-24T23:27:41.447723+0000","last_undegraded":"2024-04-24T23:27:41.447723+0000","last_fullsized":"2024-04-24T23:27:41.447723+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":4911,"log_dups_size":0,"ondisk_log_size":4911,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:03:52.974835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":201345451,"num_objects":51,"num_object_clones":0,"num_object_copies":204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":77,"num_read_kb":0,"num_write":72385,"num_write_kb":1021329,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5,7],"acting":[6,3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.3","version":"30'8","reported_seq":45,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:31.704499+0000","last_change":"2024-04-24T23:15:36.991239+0000","last_active":"2024-04-24T23:27:31.704499+0000","last_peered":"2024-04-24T23:27:31.704499+0000","last_clean":"2024-04-24T23:27:31.704499+0000","last_became_active":"2024-04-24T23:15:36.990963+0000","last_became_peered":"2024-04-24T23:15:36.990963+0000","last_unstale":"2024-04-24T23:27:31.704499+0000","last_undegraded":"2024-04-24T23:27:31.704499+0000","last_fullsized":"2024-04-24T23:27:31.704499+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:53:28.101334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"29'42","reported_seq":91,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:26:59.824686+0000","last_change":"2024-04-24T23:15:35.957931+0000","last_active":"2024-04-24T23:26:59.824686+0000","last_peered":"2024-04-24T23:26:59.824686+0000","last_clean":"2024-04-24T23:26:59.824686+0000","last_became_active":"2024-04-24T23:15:35.957432+0000","last_became_peered":"2024-04-24T23:15:35.957432+0000","last_unstale":"2024-04-24T23:26:59.824686+0000","last_undegraded":"2024-04-24T23:26:59.824686+0000","last_fullsized":"2024-04-24T23:26:59.824686+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:11:38.773425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8389784,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":19,"num_read_kb":8206,"num_write":36,"num_write_kb":8211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.4","version":"30'4162","reported_seq":4258,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:41.330900+0000","last_change":"2024-04-24T23:15:41.007857+0000","last_active":"2024-04-24T23:27:41.330900+0000","last_peered":"2024-04-24T23:27:41.330900+0000","last_clean":"2024-04-24T23:27:41.330900+0000","last_became_active":"2024-04-24T23:15:41.007449+0000","last_became_peered":"2024-04-24T23:15:41.007449+0000","last_unstale":"2024-04-24T23:27:41.330900+0000","last_undegraded":"2024-04-24T23:27:41.330900+0000","last_fullsized":"2024-04-24T23:27:41.330900+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":4162,"log_dups_size":0,"ondisk_log_size":4162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:05:06.145440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":130100371,"num_objects":37,"num_object_clones":0,"num_object_copies":148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":65,"num_read_kb":0,"num_write":60887,"num_write_kb":882911,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0,3],"acting":[6,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"27'11","reported_seq":49,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:25:42.717621+0000","last_change":"2024-04-24T23:15:36.970444+0000","last_active":"2024-04-24T23:25:42.717621+0000","last_peered":"2024-04-24T23:25:42.717621+0000","last_clean":"2024-04-24T23:25:42.717621+0000","last_became_active":"2024-04-24T23:15:36.970159+0000","last_became_peered":"2024-04-24T23:15:36.970159+0000","last_unstale":"2024-04-24T23:25:42.717621+0000","last_undegraded":"2024-04-24T23:25:42.717621+0000","last_fullsized":"2024-04-24T23:25:42.717621+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:37:02.956224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"30'43","reported_seq":115,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:33.304734+0000","last_change":"2024-04-24T23:15:35.945461+0000","last_active":"2024-04-24T23:27:33.304734+0000","last_peered":"2024-04-24T23:27:33.304734+0000","last_clean":"2024-04-24T23:27:33.304734+0000","last_became_active":"2024-04-24T23:15:35.945185+0000","last_became_peered":"2024-04-24T23:15:35.945185+0000","last_unstale":"2024-04-24T23:27:33.304734+0000","last_undegraded":"2024-04-24T23:27:33.304734+0000","last_fullsized":"2024-04-24T23:27:33.304734+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:37:34.175539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":94280,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":33,"num_read_kb":113,"num_write":40,"num_write_kb":118,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"30'4717","reported_seq":4822,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:41.571617+0000","last_change":"2024-04-24T23:15:41.014584+0000","last_active":"2024-04-24T23:27:41.571617+0000","last_peered":"2024-04-24T23:27:41.571617+0000","last_clean":"2024-04-24T23:27:41.571617+0000","last_became_active":"2024-04-24T23:15:41.014137+0000","last_became_peered":"2024-04-24T23:15:41.014137+0000","last_unstale":"2024-04-24T23:27:41.571617+0000","last_undegraded":"2024-04-24T23:27:41.571617+0000","last_fullsized":"2024-04-24T23:27:41.571617+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":4717,"log_dups_size":0,"ondisk_log_size":4717,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:23:50.387635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":155239920,"num_objects":44,"num_object_clones":0,"num_object_copies":176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":0,"num_read":74,"num_read_kb":0,"num_write":69070,"num_write_kb":969120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,7],"acting":[6,0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"30'5","reported_seq":42,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:26:06.044607+0000","last_change":"2024-04-24T23:15:36.981783+0000","last_active":"2024-04-24T23:26:06.044607+0000","last_peered":"2024-04-24T23:26:06.044607+0000","last_clean":"2024-04-24T23:26:06.044607+0000","last_became_active":"2024-04-24T23:15:36.981502+0000","last_became_peered":"2024-04-24T23:15:36.981502+0000","last_unstale":"2024-04-24T23:26:06.044607+0000","last_undegraded":"2024-04-24T23:26:06.044607+0000","last_fullsized":"2024-04-24T23:26:06.044607+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:56:08.483410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"29'34","reported_seq":75,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:25:52.069108+0000","last_change":"2024-04-24T23:15:35.960519+0000","last_active":"2024-04-24T23:25:52.069108+0000","last_peered":"2024-04-24T23:25:52.069108+0000","last_clean":"2024-04-24T23:25:52.069108+0000","last_became_active":"2024-04-24T23:15:35.960180+0000","last_became_peered":"2024-04-24T23:15:35.960180+0000","last_unstale":"2024-04-24T23:25:52.069108+0000","last_undegraded":"2024-04-24T23:25:52.069108+0000","last_fullsized":"2024-04-24T23:25:52.069108+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:50:00.199120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":8192,"num_write":28,"num_write_kb":8204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"30'3823","reported_seq":3915,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:40.972674+0000","last_change":"2024-04-24T23:15:41.012876+0000","last_active":"2024-04-24T23:27:40.972674+0000","last_peered":"2024-04-24T23:27:40.972674+0000","last_clean":"2024-04-24T23:27:40.972674+0000","last_became_active":"2024-04-24T23:15:41.012493+0000","last_became_peered":"2024-04-24T23:15:41.012493+0000","last_unstale":"2024-04-24T23:27:40.972674+0000","last_undegraded":"2024-04-24T23:27:40.972674+0000","last_fullsized":"2024-04-24T23:27:40.972674+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":3823,"log_dups_size":0,"ondisk_log_size":3823,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:21:28.287059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":147234482,"num_objects":41,"num_object_clones":0,"num_object_copies":164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":61,"num_read_kb":307,"num_write":55926,"num_write_kb":780987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6,1],"acting":[5,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"30'5","reported_seq":42,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:26:05.796138+0000","last_change":"2024-04-24T23:15:36.981449+0000","last_active":"2024-04-24T23:26:05.796138+0000","last_peered":"2024-04-24T23:26:05.796138+0000","last_clean":"2024-04-24T23:26:05.796138+0000","last_became_active":"2024-04-24T23:15:36.980837+0000","last_became_peered":"2024-04-24T23:15:36.980837+0000","last_unstale":"2024-04-24T23:26:05.796138+0000","last_undegraded":"2024-04-24T23:26:05.796138+0000","last_fullsized":"2024-04-24T23:26:05.796138+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:15:13.513341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.3","version":"27'7","reported_seq":50,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:25:42.817265+0000","last_change":"2024-04-24T23:15:35.952947+0000","last_active":"2024-04-24T23:25:42.817265+0000","last_peered":"2024-04-24T23:25:42.817265+0000","last_clean":"2024-04-24T23:25:42.817265+0000","last_became_active":"2024-04-24T23:15:35.952655+0000","last_became_peered":"2024-04-24T23:15:35.952655+0000","last_unstale":"2024-04-24T23:25:42.817265+0000","last_undegraded":"2024-04-24T23:25:42.817265+0000","last_fullsized":"2024-04-24T23:25:42.817265+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:50:10.939396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":7,"num_read_kb":11,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":196,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:25:50.717692+0000","last_change":"2024-04-24T23:15:25.826673+0000","last_active":"2024-04-24T23:25:50.717692+0000","last_peered":"2024-04-24T23:25:50.717692+0000","last_clean":"2024-04-24T23:25:50.717692+0000","last_became_active":"2024-04-24T23:15:25.820030+0000","last_became_peered":"2024-04-24T23:15:25.820030+0000","last_unstale":"2024-04-24T23:25:50.717692+0000","last_undegraded":"2024-04-24T23:25:50.717692+0000","last_fullsized":"2024-04-24T23:25:50.717692+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:24.790928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:24.790928+0000","last_clean_scrub_stamp":"2024-04-24T23:15:24.790928+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:37:24.054332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"30'4249","reported_seq":4356,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:41.676626+0000","last_change":"2024-04-24T23:15:41.024712+0000","last_active":"2024-04-24T23:27:41.676626+0000","last_peered":"2024-04-24T23:27:41.676626+0000","last_clean":"2024-04-24T23:27:41.676626+0000","last_became_active":"2024-04-24T23:15:41.022697+0000","last_became_peered":"2024-04-24T23:15:41.022697+0000","last_unstale":"2024-04-24T23:27:41.676626+0000","last_undegraded":"2024-04-24T23:27:41.676626+0000","last_fullsized":"2024-04-24T23:27:41.676626+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":4249,"log_dups_size":0,"ondisk_log_size":4249,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:37:08.231105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":134227460,"num_objects":37,"num_object_clones":0,"num_object_copies":148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":76,"num_read_kb":0,"num_write":62617,"num_write_kb":898105,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2,0],"acting":[3,1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.d","version":"30'8","reported_seq":45,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:28.621617+0000","last_change":"2024-04-24T23:15:37.000489+0000","last_active":"2024-04-24T23:27:28.621617+0000","last_peered":"2024-04-24T23:27:28.621617+0000","last_clean":"2024-04-24T23:27:28.621617+0000","last_became_active":"2024-04-24T23:15:37.000179+0000","last_became_peered":"2024-04-24T23:15:37.000179+0000","last_unstale":"2024-04-24T23:27:28.621617+0000","last_undegraded":"2024-04-24T23:27:28.621617+0000","last_fullsized":"2024-04-24T23:27:28.621617+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:06:37.125573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.c","version":"28'10","reported_seq":49,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:25:42.744989+0000","last_change":"2024-04-24T23:15:35.966020+0000","last_active":"2024-04-24T23:25:42.744989+0000","last_peered":"2024-04-24T23:25:42.744989+0000","last_clean":"2024-04-24T23:25:42.744989+0000","last_became_active":"2024-04-24T23:15:35.965758+0000","last_became_peered":"2024-04-24T23:15:35.965758+0000","last_unstale":"2024-04-24T23:25:42.744989+0000","last_undegraded":"2024-04-24T23:25:42.744989+0000","last_fullsized":"2024-04-24T23:25:42.744989+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:39:21.233461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2157,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"30'4343","reported_seq":4444,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:39.335432+0000","last_change":"2024-04-24T23:15:41.024489+0000","last_active":"2024-04-24T23:27:39.335432+0000","last_peered":"2024-04-24T23:27:39.335432+0000","last_clean":"2024-04-24T23:27:39.335432+0000","last_became_active":"2024-04-24T23:15:41.024041+0000","last_became_peered":"2024-04-24T23:15:41.024041+0000","last_unstale":"2024-04-24T23:27:39.335432+0000","last_undegraded":"2024-04-24T23:27:39.335432+0000","last_fullsized":"2024-04-24T23:27:39.335432+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":4343,"log_dups_size":0,"ondisk_log_size":4343,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:27:18.197003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":159390415,"num_objects":41,"num_object_clones":0,"num_object_copies":164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":70,"num_read_kb":0,"num_write":63845,"num_write_kb":912579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7,4],"acting":[2,3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"27'7","reported_seq":44,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:25:42.773524+0000","last_change":"2024-04-24T23:15:36.979887+0000","last_active":"2024-04-24T23:25:42.773524+0000","last_peered":"2024-04-24T23:25:42.773524+0000","last_clean":"2024-04-24T23:25:42.773524+0000","last_became_active":"2024-04-24T23:15:36.979611+0000","last_became_peered":"2024-04-24T23:15:36.979611+0000","last_unstale":"2024-04-24T23:25:42.773524+0000","last_undegraded":"2024-04-24T23:25:42.773524+0000","last_fullsized":"2024-04-24T23:25:42.773524+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:07:18.261077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"28'25","reported_seq":66,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:26:56.337078+0000","last_change":"2024-04-24T23:15:35.965192+0000","last_active":"2024-04-24T23:26:56.337078+0000","last_peered":"2024-04-24T23:26:56.337078+0000","last_clean":"2024-04-24T23:26:56.337078+0000","last_became_active":"2024-04-24T23:15:35.964873+0000","last_became_peered":"2024-04-24T23:15:35.964873+0000","last_unstale":"2024-04-24T23:26:56.337078+0000","last_undegraded":"2024-04-24T23:26:56.337078+0000","last_fullsized":"2024-04-24T23:26:56.337078+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:14:27.884944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194374,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":4097,"num_write":20,"num_write_kb":4108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"30'4098","reported_seq":4200,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:40.854680+0000","last_change":"2024-04-24T23:15:41.004002+0000","last_active":"2024-04-24T23:27:40.854680+0000","last_peered":"2024-04-24T23:27:40.854680+0000","last_clean":"2024-04-24T23:27:40.854680+0000","last_became_active":"2024-04-24T23:15:41.003335+0000","last_became_peered":"2024-04-24T23:15:41.003335+0000","last_unstale":"2024-04-24T23:27:40.854680+0000","last_undegraded":"2024-04-24T23:27:40.854680+0000","last_fullsized":"2024-04-24T23:27:40.854680+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":4098,"log_dups_size":0,"ondisk_log_size":4098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:49:36.206903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113271300,"num_objects":36,"num_object_clones":0,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":71,"num_read_kb":0,"num_write":60116,"num_write_kb":843409,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1,3],"acting":[5,4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"27'8","reported_seq":45,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:25:42.818254+0000","last_change":"2024-04-24T23:15:37.002854+0000","last_active":"2024-04-24T23:25:42.818254+0000","last_peered":"2024-04-24T23:25:42.818254+0000","last_clean":"2024-04-24T23:25:42.818254+0000","last_became_active":"2024-04-24T23:15:37.002583+0000","last_became_peered":"2024-04-24T23:15:37.002583+0000","last_unstale":"2024-04-24T23:25:42.818254+0000","last_undegraded":"2024-04-24T23:25:42.818254+0000","last_fullsized":"2024-04-24T23:25:42.818254+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:27:04.251547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"30'166","reported_seq":213,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:28.158437+0000","last_change":"2024-04-24T23:15:35.948752+0000","last_active":"2024-04-24T23:27:28.158437+0000","last_peered":"2024-04-24T23:27:28.158437+0000","last_clean":"2024-04-24T23:27:28.158437+0000","last_became_active":"2024-04-24T23:15:35.948438+0000","last_became_peered":"2024-04-24T23:15:35.948438+0000","last_unstale":"2024-04-24T23:27:28.158437+0000","last_undegraded":"2024-04-24T23:27:28.158437+0000","last_fullsized":"2024-04-24T23:27:28.158437+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":166,"log_dups_size":0,"ondisk_log_size":166,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:45:56.006278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":21,"num_read_kb":2419,"num_write":164,"num_write_kb":4172,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.8","version":"30'4111","reported_seq":4210,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:42.308514+0000","last_change":"2024-04-24T23:15:41.017524+0000","last_active":"2024-04-24T23:27:42.308514+0000","last_peered":"2024-04-24T23:27:42.308514+0000","last_clean":"2024-04-24T23:27:42.308514+0000","last_became_active":"2024-04-24T23:15:41.017092+0000","last_became_peered":"2024-04-24T23:15:41.017092+0000","last_unstale":"2024-04-24T23:27:42.308514+0000","last_undegraded":"2024-04-24T23:27:42.308514+0000","last_fullsized":"2024-04-24T23:27:42.308514+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":4111,"log_dups_size":0,"ondisk_log_size":4111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:04:53.235038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113264988,"num_objects":31,"num_object_clones":0,"num_object_copies":124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":68,"num_read_kb":0,"num_write":60399,"num_write_kb":856855,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"27'4","reported_seq":41,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:25:42.735229+0000","last_change":"2024-04-24T23:15:36.990441+0000","last_active":"2024-04-24T23:25:42.735229+0000","last_peered":"2024-04-24T23:25:42.735229+0000","last_clean":"2024-04-24T23:25:42.735229+0000","last_became_active":"2024-04-24T23:15:36.989863+0000","last_became_peered":"2024-04-24T23:15:36.989863+0000","last_unstale":"2024-04-24T23:25:42.735229+0000","last_undegraded":"2024-04-24T23:25:42.735229+0000","last_fullsized":"2024-04-24T23:25:42.735229+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:50:07.975347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.f","version":"30'6","reported_seq":46,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:01.577469+0000","last_change":"2024-04-24T23:15:35.969317+0000","last_active":"2024-04-24T23:27:01.577469+0000","last_peered":"2024-04-24T23:27:01.577469+0000","last_clean":"2024-04-24T23:27:01.577469+0000","last_became_active":"2024-04-24T23:15:35.968994+0000","last_became_peered":"2024-04-24T23:15:35.968994+0000","last_unstale":"2024-04-24T23:27:01.577469+0000","last_undegraded":"2024-04-24T23:27:01.577469+0000","last_fullsized":"2024-04-24T23:27:01.577469+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:51:57.100772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"30'4118","reported_seq":4222,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:37.895480+0000","last_change":"2024-04-24T23:15:41.010103+0000","last_active":"2024-04-24T23:27:37.895480+0000","last_peered":"2024-04-24T23:27:37.895480+0000","last_clean":"2024-04-24T23:27:37.895480+0000","last_became_active":"2024-04-24T23:15:41.009328+0000","last_became_peered":"2024-04-24T23:15:41.009328+0000","last_unstale":"2024-04-24T23:27:37.895480+0000","last_undegraded":"2024-04-24T23:27:37.895480+0000","last_fullsized":"2024-04-24T23:27:37.895480+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":4118,"log_dups_size":0,"ondisk_log_size":4118,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:46:49.579501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":193017293,"num_objects":50,"num_object_clones":0,"num_object_copies":200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":73,"num_read_kb":0,"num_write":60557,"num_write_kb":866216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,2],"acting":[1,5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.11","version":"27'10","reported_seq":47,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:25:42.747653+0000","last_change":"2024-04-24T23:15:36.996765+0000","last_active":"2024-04-24T23:25:42.747653+0000","last_peered":"2024-04-24T23:25:42.747653+0000","last_clean":"2024-04-24T23:25:42.747653+0000","last_became_active":"2024-04-24T23:15:36.995801+0000","last_became_peered":"2024-04-24T23:15:36.995801+0000","last_unstale":"2024-04-24T23:25:42.747653+0000","last_undegraded":"2024-04-24T23:25:42.747653+0000","last_fullsized":"2024-04-24T23:25:42.747653+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:25:19.421667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"28'19","reported_seq":66,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:23.188825+0000","last_change":"2024-04-24T23:15:35.944023+0000","last_active":"2024-04-24T23:27:23.188825+0000","last_peered":"2024-04-24T23:27:23.188825+0000","last_clean":"2024-04-24T23:27:23.188825+0000","last_became_active":"2024-04-24T23:15:35.943710+0000","last_became_peered":"2024-04-24T23:15:35.943710+0000","last_unstale":"2024-04-24T23:27:23.188825+0000","last_undegraded":"2024-04-24T23:27:23.188825+0000","last_fullsized":"2024-04-24T23:27:23.188825+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:46:48.090958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"30'4647","reported_seq":4747,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:41.252050+0000","last_change":"2024-04-24T23:15:41.013027+0000","last_active":"2024-04-24T23:27:41.252050+0000","last_peered":"2024-04-24T23:27:41.252050+0000","last_clean":"2024-04-24T23:27:41.252050+0000","last_became_active":"2024-04-24T23:15:41.012587+0000","last_became_peered":"2024-04-24T23:15:41.012587+0000","last_unstale":"2024-04-24T23:27:41.252050+0000","last_undegraded":"2024-04-24T23:27:41.252050+0000","last_fullsized":"2024-04-24T23:27:41.252050+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":4647,"log_dups_size":0,"ondisk_log_size":4647,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:27:04.276246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109305633,"num_objects":37,"num_object_clones":0,"num_object_copies":148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":70,"num_read_kb":0,"num_write":67790,"num_write_kb":966739,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1,6],"acting":[5,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"30'7","reported_seq":44,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:17.987795+0000","last_change":"2024-04-24T23:15:36.979702+0000","last_active":"2024-04-24T23:27:17.987795+0000","last_peered":"2024-04-24T23:27:17.987795+0000","last_clean":"2024-04-24T23:27:17.987795+0000","last_became_active":"2024-04-24T23:15:36.979356+0000","last_became_peered":"2024-04-24T23:15:36.979356+0000","last_unstale":"2024-04-24T23:27:17.987795+0000","last_undegraded":"2024-04-24T23:27:17.987795+0000","last_fullsized":"2024-04-24T23:27:17.987795+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:14:24.309192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.11","version":"28'12","reported_seq":51,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:25:42.827287+0000","last_change":"2024-04-24T23:15:35.973026+0000","last_active":"2024-04-24T23:25:42.827287+0000","last_peered":"2024-04-24T23:25:42.827287+0000","last_clean":"2024-04-24T23:25:42.827287+0000","last_became_active":"2024-04-24T23:15:35.972655+0000","last_became_peered":"2024-04-24T23:15:35.972655+0000","last_unstale":"2024-04-24T23:25:42.827287+0000","last_undegraded":"2024-04-24T23:25:42.827287+0000","last_fullsized":"2024-04-24T23:25:42.827287+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:25:34.352908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1575,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"30'4482","reported_seq":4582,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:40.902076+0000","last_change":"2024-04-24T23:15:41.004221+0000","last_active":"2024-04-24T23:27:40.902076+0000","last_peered":"2024-04-24T23:27:40.902076+0000","last_clean":"2024-04-24T23:27:40.902076+0000","last_became_active":"2024-04-24T23:15:41.003790+0000","last_became_peered":"2024-04-24T23:15:41.003790+0000","last_unstale":"2024-04-24T23:27:40.902076+0000","last_undegraded":"2024-04-24T23:27:40.902076+0000","last_fullsized":"2024-04-24T23:27:40.902076+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:28:58.866024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146860262,"num_objects":41,"num_object_clones":0,"num_object_copies":164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":70,"num_read_kb":0,"num_write":65436,"num_write_kb":956853,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6,4],"acting":[5,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"27'10","reported_seq":47,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:25:42.816663+0000","last_change":"2024-04-24T23:15:36.997389+0000","last_active":"2024-04-24T23:25:42.816663+0000","last_peered":"2024-04-24T23:25:42.816663+0000","last_clean":"2024-04-24T23:25:42.816663+0000","last_became_active":"2024-04-24T23:15:36.995923+0000","last_became_peered":"2024-04-24T23:15:36.995923+0000","last_unstale":"2024-04-24T23:25:42.816663+0000","last_undegraded":"2024-04-24T23:25:42.816663+0000","last_fullsized":"2024-04-24T23:25:42.816663+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:55:47.134320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"29'7","reported_seq":50,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:25:43.845951+0000","last_change":"2024-04-24T23:15:35.957715+0000","last_active":"2024-04-24T23:25:43.845951+0000","last_peered":"2024-04-24T23:25:43.845951+0000","last_clean":"2024-04-24T23:25:43.845951+0000","last_became_active":"2024-04-24T23:15:35.957088+0000","last_became_peered":"2024-04-24T23:15:35.957088+0000","last_unstale":"2024-04-24T23:25:43.845951+0000","last_undegraded":"2024-04-24T23:25:43.845951+0000","last_fullsized":"2024-04-24T23:25:43.845951+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:30:34.483669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":7,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"30'4647","reported_seq":4749,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:41.710954+0000","last_change":"2024-04-24T23:15:41.008165+0000","last_active":"2024-04-24T23:27:41.710954+0000","last_peered":"2024-04-24T23:27:41.710954+0000","last_clean":"2024-04-24T23:27:41.710954+0000","last_became_active":"2024-04-24T23:15:41.007555+0000","last_became_peered":"2024-04-24T23:15:41.007555+0000","last_unstale":"2024-04-24T23:27:41.710954+0000","last_undegraded":"2024-04-24T23:27:41.710954+0000","last_fullsized":"2024-04-24T23:27:41.710954+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":4647,"log_dups_size":0,"ondisk_log_size":4647,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:18:08.195747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":105089739,"num_objects":31,"num_object_clones":0,"num_object_copies":124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":71,"num_read_kb":0,"num_write":68017,"num_write_kb":979211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0,7],"acting":[6,4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.12","version":"30'12","reported_seq":49,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:32.279103+0000","last_change":"2024-04-24T23:15:36.991382+0000","last_active":"2024-04-24T23:27:32.279103+0000","last_peered":"2024-04-24T23:27:32.279103+0000","last_clean":"2024-04-24T23:27:32.279103+0000","last_became_active":"2024-04-24T23:15:36.991115+0000","last_became_peered":"2024-04-24T23:15:36.991115+0000","last_unstale":"2024-04-24T23:27:32.279103+0000","last_undegraded":"2024-04-24T23:27:32.279103+0000","last_fullsized":"2024-04-24T23:27:32.279103+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:21:51.863947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"28'40","reported_seq":90,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:22.419341+0000","last_change":"2024-04-24T23:15:35.964245+0000","last_active":"2024-04-24T23:27:22.419341+0000","last_peered":"2024-04-24T23:27:22.419341+0000","last_clean":"2024-04-24T23:27:22.419341+0000","last_became_active":"2024-04-24T23:15:35.963907+0000","last_became_peered":"2024-04-24T23:15:35.963907+0000","last_unstale":"2024-04-24T23:27:22.419341+0000","last_undegraded":"2024-04-24T23:27:22.419341+0000","last_fullsized":"2024-04-24T23:27:22.419341+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:15:00.913899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388676,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":26,"num_read_kb":12314,"num_write":42,"num_write_kb":8216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"30'4239","reported_seq":4343,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:42.429376+0000","last_change":"2024-04-24T23:15:41.021981+0000","last_active":"2024-04-24T23:27:42.429376+0000","last_peered":"2024-04-24T23:27:42.429376+0000","last_clean":"2024-04-24T23:27:42.429376+0000","last_became_active":"2024-04-24T23:15:41.019193+0000","last_became_peered":"2024-04-24T23:15:41.019193+0000","last_unstale":"2024-04-24T23:27:42.429376+0000","last_undegraded":"2024-04-24T23:27:42.429376+0000","last_fullsized":"2024-04-24T23:27:42.429376+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":4239,"log_dups_size":0,"ondisk_log_size":4239,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:04:39.828502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121634816,"num_objects":29,"num_object_clones":0,"num_object_copies":116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":0,"num_read":73,"num_read_kb":0,"num_write":62295,"num_write_kb":864711,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5,6],"acting":[3,0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"30'10","reported_seq":47,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:37.674563+0000","last_change":"2024-04-24T23:15:36.986833+0000","last_active":"2024-04-24T23:27:37.674563+0000","last_peered":"2024-04-24T23:27:37.674563+0000","last_clean":"2024-04-24T23:27:37.674563+0000","last_became_active":"2024-04-24T23:15:36.986484+0000","last_became_peered":"2024-04-24T23:15:36.986484+0000","last_unstale":"2024-04-24T23:27:37.674563+0000","last_undegraded":"2024-04-24T23:27:37.674563+0000","last_fullsized":"2024-04-24T23:27:37.674563+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:12:32.245267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.14","version":"30'236","reported_seq":286,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:29.519952+0000","last_change":"2024-04-24T23:15:35.957124+0000","last_active":"2024-04-24T23:27:29.519952+0000","last_peered":"2024-04-24T23:27:29.519952+0000","last_clean":"2024-04-24T23:27:29.519952+0000","last_became_active":"2024-04-24T23:15:35.951923+0000","last_became_peered":"2024-04-24T23:15:35.951923+0000","last_unstale":"2024-04-24T23:27:29.519952+0000","last_undegraded":"2024-04-24T23:27:29.519952+0000","last_fullsized":"2024-04-24T23:27:29.519952+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":236,"log_dups_size":0,"ondisk_log_size":236,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:44:16.820640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194976,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":28,"num_read_kb":4122,"num_write":236,"num_write_kb":4244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"30'4347","reported_seq":4454,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:41.594148+0000","last_change":"2024-04-24T23:15:41.015494+0000","last_active":"2024-04-24T23:27:41.594148+0000","last_peered":"2024-04-24T23:27:41.594148+0000","last_clean":"2024-04-24T23:27:41.594148+0000","last_became_active":"2024-04-24T23:15:41.015179+0000","last_became_peered":"2024-04-24T23:15:41.015179+0000","last_unstale":"2024-04-24T23:27:41.594148+0000","last_undegraded":"2024-04-24T23:27:41.594148+0000","last_fullsized":"2024-04-24T23:27:41.594148+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":4347,"log_dups_size":0,"ondisk_log_size":4347,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:18:32.208506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":163660642,"num_objects":44,"num_object_clones":0,"num_object_copies":176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":0,"num_read":76,"num_read_kb":0,"num_write":63898,"num_write_kb":903699,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5,0],"acting":[4,3,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"30'7","reported_seq":44,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:33.442182+0000","last_change":"2024-04-24T23:15:36.999334+0000","last_active":"2024-04-24T23:27:33.442182+0000","last_peered":"2024-04-24T23:27:33.442182+0000","last_clean":"2024-04-24T23:27:33.442182+0000","last_became_active":"2024-04-24T23:15:36.995470+0000","last_became_peered":"2024-04-24T23:15:36.995470+0000","last_unstale":"2024-04-24T23:27:33.442182+0000","last_undegraded":"2024-04-24T23:27:33.442182+0000","last_fullsized":"2024-04-24T23:27:33.442182+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:19:57.973402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.15","version":"30'20","reported_seq":62,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:41.283118+0000","last_change":"2024-04-24T23:15:35.946041+0000","last_active":"2024-04-24T23:27:41.283118+0000","last_peered":"2024-04-24T23:27:41.283118+0000","last_clean":"2024-04-24T23:27:41.283118+0000","last_became_active":"2024-04-24T23:15:35.944805+0000","last_became_peered":"2024-04-24T23:15:35.944805+0000","last_unstale":"2024-04-24T23:27:41.283118+0000","last_undegraded":"2024-04-24T23:27:41.283118+0000","last_fullsized":"2024-04-24T23:27:41.283118+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:52:19.261306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":22,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"30'4379","reported_seq":4484,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:39.436460+0000","last_change":"2024-04-24T23:15:41.024392+0000","last_active":"2024-04-24T23:27:39.436460+0000","last_peered":"2024-04-24T23:27:39.436460+0000","last_clean":"2024-04-24T23:27:39.436460+0000","last_became_active":"2024-04-24T23:15:41.023951+0000","last_became_peered":"2024-04-24T23:15:41.023951+0000","last_unstale":"2024-04-24T23:27:39.436460+0000","last_undegraded":"2024-04-24T23:27:39.436460+0000","last_fullsized":"2024-04-24T23:27:39.436460+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":4379,"log_dups_size":0,"ondisk_log_size":4379,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:24:17.810117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":176268273,"num_objects":52,"num_object_clones":0,"num_object_copies":208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":0,"num_read":74,"num_read_kb":0,"num_write":64448,"num_write_kb":925608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1,7],"acting":[2,3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.17","version":"30'11","reported_seq":47,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:37.981684+0000","last_change":"2024-04-24T23:15:36.992243+0000","last_active":"2024-04-24T23:27:37.981684+0000","last_peered":"2024-04-24T23:27:37.981684+0000","last_clean":"2024-04-24T23:27:37.981684+0000","last_became_active":"2024-04-24T23:15:36.992002+0000","last_became_peered":"2024-04-24T23:15:36.992002+0000","last_unstale":"2024-04-24T23:27:37.981684+0000","last_undegraded":"2024-04-24T23:27:37.981684+0000","last_fullsized":"2024-04-24T23:27:37.981684+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:52:15.911009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"28'5","reported_seq":50,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:25:42.767820+0000","last_change":"2024-04-24T23:15:35.967408+0000","last_active":"2024-04-24T23:25:42.767820+0000","last_peered":"2024-04-24T23:25:42.767820+0000","last_clean":"2024-04-24T23:25:42.767820+0000","last_became_active":"2024-04-24T23:15:35.966646+0000","last_became_peered":"2024-04-24T23:15:35.966646+0000","last_unstale":"2024-04-24T23:25:42.767820+0000","last_undegraded":"2024-04-24T23:25:42.767820+0000","last_fullsized":"2024-04-24T23:25:42.767820+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:07:07.931773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":19,"num_read_kb":18,"num_write":7,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"30'4287","reported_seq":4386,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:41.572104+0000","last_change":"2024-04-24T23:15:41.021697+0000","last_active":"2024-04-24T23:27:41.572104+0000","last_peered":"2024-04-24T23:27:41.572104+0000","last_clean":"2024-04-24T23:27:41.572104+0000","last_became_active":"2024-04-24T23:15:41.021281+0000","last_became_peered":"2024-04-24T23:15:41.021281+0000","last_unstale":"2024-04-24T23:27:41.572104+0000","last_undegraded":"2024-04-24T23:27:41.572104+0000","last_fullsized":"2024-04-24T23:27:41.572104+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":4287,"log_dups_size":0,"ondisk_log_size":4287,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:59:37.306593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":130091904,"num_objects":38,"num_object_clones":0,"num_object_copies":152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":68,"num_read_kb":0,"num_write":62864,"num_write_kb":881151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3,2],"acting":[7,1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"30'6","reported_seq":42,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:25:42.735229+0000","last_change":"2024-04-24T23:15:36.990988+0000","last_active":"2024-04-24T23:25:42.735229+0000","last_peered":"2024-04-24T23:25:42.735229+0000","last_clean":"2024-04-24T23:25:42.735229+0000","last_became_active":"2024-04-24T23:15:36.990711+0000","last_became_peered":"2024-04-24T23:15:36.990711+0000","last_unstale":"2024-04-24T23:25:42.735229+0000","last_undegraded":"2024-04-24T23:25:42.735229+0000","last_fullsized":"2024-04-24T23:25:42.735229+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:33:20.663776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.17","version":"30'14","reported_seq":55,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:26.156910+0000","last_change":"2024-04-24T23:15:35.957838+0000","last_active":"2024-04-24T23:27:26.156910+0000","last_peered":"2024-04-24T23:27:26.156910+0000","last_clean":"2024-04-24T23:27:26.156910+0000","last_became_active":"2024-04-24T23:15:35.956870+0000","last_became_peered":"2024-04-24T23:15:35.956870+0000","last_unstale":"2024-04-24T23:27:26.156910+0000","last_undegraded":"2024-04-24T23:27:26.156910+0000","last_fullsized":"2024-04-24T23:27:26.156910+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:56:03.889693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":13,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"30'4364","reported_seq":4459,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:41.392627+0000","last_change":"2024-04-24T23:15:41.006098+0000","last_active":"2024-04-24T23:27:41.392627+0000","last_peered":"2024-04-24T23:27:41.392627+0000","last_clean":"2024-04-24T23:27:41.392627+0000","last_became_active":"2024-04-24T23:15:41.005578+0000","last_became_peered":"2024-04-24T23:15:41.005578+0000","last_unstale":"2024-04-24T23:27:41.392627+0000","last_undegraded":"2024-04-24T23:27:41.392627+0000","last_fullsized":"2024-04-24T23:27:41.392627+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":4364,"log_dups_size":0,"ondisk_log_size":4364,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:21:12.802112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":159413905,"num_objects":44,"num_object_clones":0,"num_object_copies":176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":0,"num_read":64,"num_read_kb":0,"num_write":64073,"num_write_kb":908953,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3,7],"acting":[4,0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1e","version":"30'5129","reported_seq":5243,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:41.710458+0000","last_change":"2024-04-24T23:15:41.022122+0000","last_active":"2024-04-24T23:27:41.710458+0000","last_peered":"2024-04-24T23:27:41.710458+0000","last_clean":"2024-04-24T23:27:41.710458+0000","last_became_active":"2024-04-24T23:15:41.021698+0000","last_became_peered":"2024-04-24T23:15:41.021698+0000","last_unstale":"2024-04-24T23:27:41.710458+0000","last_undegraded":"2024-04-24T23:27:41.710458+0000","last_fullsized":"2024-04-24T23:27:41.710458+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":5129,"log_dups_size":0,"ondisk_log_size":5129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:13:56.730050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":201337994,"num_objects":51,"num_object_clones":0,"num_object_copies":204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":83,"num_read_kb":0,"num_write":75837,"num_write_kb":1069592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5,3],"acting":[4,1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"27'3","reported_seq":39,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:25:42.710767+0000","last_change":"2024-04-24T23:15:36.979816+0000","last_active":"2024-04-24T23:25:42.710767+0000","last_peered":"2024-04-24T23:25:42.710767+0000","last_clean":"2024-04-24T23:25:42.710767+0000","last_became_active":"2024-04-24T23:15:36.979553+0000","last_became_peered":"2024-04-24T23:15:36.979553+0000","last_unstale":"2024-04-24T23:25:42.710767+0000","last_undegraded":"2024-04-24T23:25:42.710767+0000","last_fullsized":"2024-04-24T23:25:42.710767+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:55:47.134320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.18","version":"29'33","reported_seq":73,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:25:53.591106+0000","last_change":"2024-04-24T23:15:35.964744+0000","last_active":"2024-04-24T23:25:53.591106+0000","last_peered":"2024-04-24T23:25:53.591106+0000","last_clean":"2024-04-24T23:25:53.591106+0000","last_became_active":"2024-04-24T23:15:35.964434+0000","last_became_peered":"2024-04-24T23:15:35.964434+0000","last_unstale":"2024-04-24T23:25:53.591106+0000","last_undegraded":"2024-04-24T23:25:53.591106+0000","last_fullsized":"2024-04-24T23:25:53.591106+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:26:32.587197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8389190,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":8192,"num_write":27,"num_write_kb":8217,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"30'4510","reported_seq":4610,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:40.494772+0000","last_change":"2024-04-24T23:15:41.022032+0000","last_active":"2024-04-24T23:27:40.494772+0000","last_peered":"2024-04-24T23:27:40.494772+0000","last_clean":"2024-04-24T23:27:40.494772+0000","last_became_active":"2024-04-24T23:15:41.021616+0000","last_became_peered":"2024-04-24T23:15:41.021616+0000","last_unstale":"2024-04-24T23:27:40.494772+0000","last_undegraded":"2024-04-24T23:27:40.494772+0000","last_fullsized":"2024-04-24T23:27:40.494772+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":4510,"log_dups_size":0,"ondisk_log_size":4510,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:08:59.242392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":163661004,"num_objects":50,"num_object_clones":0,"num_object_copies":200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":69,"num_read_kb":0,"num_write":66227,"num_write_kb":941784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5,1],"acting":[3,2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"30'16","reported_seq":52,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:33.301198+0000","last_change":"2024-04-24T23:15:36.967198+0000","last_active":"2024-04-24T23:27:33.301198+0000","last_peered":"2024-04-24T23:27:33.301198+0000","last_clean":"2024-04-24T23:27:33.301198+0000","last_became_active":"2024-04-24T23:15:36.966910+0000","last_became_peered":"2024-04-24T23:15:36.966910+0000","last_unstale":"2024-04-24T23:27:33.301198+0000","last_undegraded":"2024-04-24T23:27:33.301198+0000","last_fullsized":"2024-04-24T23:27:33.301198+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:51:19.788437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"28'6","reported_seq":46,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:25:42.740651+0000","last_change":"2024-04-24T23:15:35.967169+0000","last_active":"2024-04-24T23:25:42.740651+0000","last_peered":"2024-04-24T23:25:42.740651+0000","last_clean":"2024-04-24T23:25:42.740651+0000","last_became_active":"2024-04-24T23:15:35.966939+0000","last_became_peered":"2024-04-24T23:15:35.966939+0000","last_unstale":"2024-04-24T23:25:42.740651+0000","last_undegraded":"2024-04-24T23:25:42.740651+0000","last_fullsized":"2024-04-24T23:25:42.740651+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:38:55.460569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":14,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"30'17","reported_seq":56,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:26:59.813995+0000","last_change":"2024-04-24T23:15:35.971236+0000","last_active":"2024-04-24T23:26:59.813995+0000","last_peered":"2024-04-24T23:26:59.813995+0000","last_clean":"2024-04-24T23:26:59.813995+0000","last_became_active":"2024-04-24T23:15:35.970908+0000","last_became_peered":"2024-04-24T23:15:35.970908+0000","last_unstale":"2024-04-24T23:26:59.813995+0000","last_undegraded":"2024-04-24T23:26:59.813995+0000","last_fullsized":"2024-04-24T23:26:59.813995+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:14:15.548024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":1,"num_read_kb":4096,"num_write":17,"num_write_kb":4106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"27'8","reported_seq":44,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:25:42.714394+0000","last_change":"2024-04-24T23:15:36.988156+0000","last_active":"2024-04-24T23:25:42.714394+0000","last_peered":"2024-04-24T23:25:42.714394+0000","last_clean":"2024-04-24T23:25:42.714394+0000","last_became_active":"2024-04-24T23:15:36.987896+0000","last_became_peered":"2024-04-24T23:15:36.987896+0000","last_unstale":"2024-04-24T23:25:42.714394+0000","last_undegraded":"2024-04-24T23:25:42.714394+0000","last_fullsized":"2024-04-24T23:25:42.714394+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:22:06.869644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"30'3620","reported_seq":3710,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:41.499345+0000","last_change":"2024-04-24T23:15:40.997559+0000","last_active":"2024-04-24T23:27:41.499345+0000","last_peered":"2024-04-24T23:27:41.499345+0000","last_clean":"2024-04-24T23:27:41.499345+0000","last_became_active":"2024-04-24T23:15:40.997102+0000","last_became_peered":"2024-04-24T23:15:40.997102+0000","last_unstale":"2024-04-24T23:27:41.499345+0000","last_undegraded":"2024-04-24T23:27:41.499345+0000","last_fullsized":"2024-04-24T23:27:41.499345+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":3620,"log_dups_size":0,"ondisk_log_size":3620,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:25:02.046761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92276415,"num_objects":23,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":60,"num_read_kb":0,"num_write":53446,"num_write_kb":749552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,7],"acting":[4,6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"30'1217","reported_seq":1259,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:25:51.681033+0000","last_change":"2024-04-24T23:15:35.944842+0000","last_active":"2024-04-24T23:25:51.681033+0000","last_peered":"2024-04-24T23:25:51.681033+0000","last_clean":"2024-04-24T23:25:51.681033+0000","last_became_active":"2024-04-24T23:15:35.944563+0000","last_became_peered":"2024-04-24T23:15:35.944563+0000","last_unstale":"2024-04-24T23:25:51.681033+0000","last_undegraded":"2024-04-24T23:25:51.681033+0000","last_fullsized":"2024-04-24T23:25:51.681033+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":1217,"log_dups_size":0,"ondisk_log_size":1217,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:40:58.933384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4195441,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":4,"num_read_kb":4098,"num_write":1212,"num_write_kb":5208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"27'8","reported_seq":44,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:25:42.807080+0000","last_change":"2024-04-24T23:15:36.981103+0000","last_active":"2024-04-24T23:25:42.807080+0000","last_peered":"2024-04-24T23:25:42.807080+0000","last_clean":"2024-04-24T23:25:42.807080+0000","last_became_active":"2024-04-24T23:15:36.979637+0000","last_became_peered":"2024-04-24T23:15:36.979637+0000","last_unstale":"2024-04-24T23:25:42.807080+0000","last_undegraded":"2024-04-24T23:25:42.807080+0000","last_fullsized":"2024-04-24T23:25:42.807080+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:09:39.750272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"30'5036","reported_seq":5143,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:42.344043+0000","last_change":"2024-04-24T23:15:41.017087+0000","last_active":"2024-04-24T23:27:42.344043+0000","last_peered":"2024-04-24T23:27:42.344043+0000","last_clean":"2024-04-24T23:27:42.344043+0000","last_became_active":"2024-04-24T23:15:41.016696+0000","last_became_peered":"2024-04-24T23:15:41.016696+0000","last_unstale":"2024-04-24T23:27:42.344043+0000","last_undegraded":"2024-04-24T23:27:42.344043+0000","last_fullsized":"2024-04-24T23:27:42.344043+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":5036,"log_dups_size":0,"ondisk_log_size":5036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:05:57.823447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":138521088,"num_objects":40,"num_object_clones":0,"num_object_copies":160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":77,"num_read_kb":0,"num_write":74016,"num_write_kb":1014885,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1,0],"acting":[7,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"30'9","reported_seq":46,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:31.253765+0000","last_change":"2024-04-24T23:15:36.970211+0000","last_active":"2024-04-24T23:27:31.253765+0000","last_peered":"2024-04-24T23:27:31.253765+0000","last_clean":"2024-04-24T23:27:31.253765+0000","last_became_active":"2024-04-24T23:15:36.969090+0000","last_became_peered":"2024-04-24T23:15:36.969090+0000","last_unstale":"2024-04-24T23:27:31.253765+0000","last_undegraded":"2024-04-24T23:27:31.253765+0000","last_fullsized":"2024-04-24T23:27:31.253765+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:16:00.231012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.21","version":"30'12","reported_seq":49,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:26:05.729132+0000","last_change":"2024-04-24T23:15:36.982528+0000","last_active":"2024-04-24T23:26:05.729132+0000","last_peered":"2024-04-24T23:26:05.729132+0000","last_clean":"2024-04-24T23:26:05.729132+0000","last_became_active":"2024-04-24T23:15:36.982299+0000","last_became_peered":"2024-04-24T23:15:36.982299+0000","last_unstale":"2024-04-24T23:26:05.729132+0000","last_undegraded":"2024-04-24T23:26:05.729132+0000","last_fullsized":"2024-04-24T23:26:05.729132+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:02:17.732324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.22","version":"30'11","reported_seq":48,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:29.623420+0000","last_change":"2024-04-24T23:15:36.989899+0000","last_active":"2024-04-24T23:27:29.623420+0000","last_peered":"2024-04-24T23:27:29.623420+0000","last_clean":"2024-04-24T23:27:29.623420+0000","last_became_active":"2024-04-24T23:15:36.989557+0000","last_became_peered":"2024-04-24T23:15:36.989557+0000","last_unstale":"2024-04-24T23:27:29.623420+0000","last_undegraded":"2024-04-24T23:27:29.623420+0000","last_fullsized":"2024-04-24T23:27:29.623420+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:15:13.513341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"30'9","reported_seq":46,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:26:05.334192+0000","last_change":"2024-04-24T23:15:36.969628+0000","last_active":"2024-04-24T23:26:05.334192+0000","last_peered":"2024-04-24T23:26:05.334192+0000","last_clean":"2024-04-24T23:26:05.334192+0000","last_became_active":"2024-04-24T23:15:36.969327+0000","last_became_peered":"2024-04-24T23:15:36.969327+0000","last_unstale":"2024-04-24T23:26:05.334192+0000","last_undegraded":"2024-04-24T23:26:05.334192+0000","last_fullsized":"2024-04-24T23:26:05.334192+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:08:47.021999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.24","version":"30'8","reported_seq":45,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:26:31.416932+0000","last_change":"2024-04-24T23:15:36.986343+0000","last_active":"2024-04-24T23:26:31.416932+0000","last_peered":"2024-04-24T23:26:31.416932+0000","last_clean":"2024-04-24T23:26:31.416932+0000","last_became_active":"2024-04-24T23:15:36.986026+0000","last_became_peered":"2024-04-24T23:15:36.986026+0000","last_unstale":"2024-04-24T23:26:31.416932+0000","last_undegraded":"2024-04-24T23:26:31.416932+0000","last_fullsized":"2024-04-24T23:26:31.416932+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:10:00.347826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"30'8","reported_seq":45,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:28.112837+0000","last_change":"2024-04-24T23:15:36.968538+0000","last_active":"2024-04-24T23:27:28.112837+0000","last_peered":"2024-04-24T23:27:28.112837+0000","last_clean":"2024-04-24T23:27:28.112837+0000","last_became_active":"2024-04-24T23:15:36.968263+0000","last_became_peered":"2024-04-24T23:15:36.968263+0000","last_unstale":"2024-04-24T23:27:28.112837+0000","last_undegraded":"2024-04-24T23:27:28.112837+0000","last_fullsized":"2024-04-24T23:27:28.112837+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:07:43.731738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.26","version":"30'5","reported_seq":42,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:26:33.160187+0000","last_change":"2024-04-24T23:15:36.991008+0000","last_active":"2024-04-24T23:26:33.160187+0000","last_peered":"2024-04-24T23:26:33.160187+0000","last_clean":"2024-04-24T23:26:33.160187+0000","last_became_active":"2024-04-24T23:15:36.990764+0000","last_became_peered":"2024-04-24T23:15:36.990764+0000","last_unstale":"2024-04-24T23:26:33.160187+0000","last_undegraded":"2024-04-24T23:26:33.160187+0000","last_fullsized":"2024-04-24T23:26:33.160187+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:50:21.865233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"30'10","reported_seq":47,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:26:31.284444+0000","last_change":"2024-04-24T23:15:36.993550+0000","last_active":"2024-04-24T23:26:31.284444+0000","last_peered":"2024-04-24T23:26:31.284444+0000","last_clean":"2024-04-24T23:26:31.284444+0000","last_became_active":"2024-04-24T23:15:36.993310+0000","last_became_peered":"2024-04-24T23:15:36.993310+0000","last_unstale":"2024-04-24T23:26:31.284444+0000","last_undegraded":"2024-04-24T23:26:31.284444+0000","last_fullsized":"2024-04-24T23:26:31.284444+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:21:20.617940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.28","version":"30'9","reported_seq":46,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:30.082036+0000","last_change":"2024-04-24T23:15:36.987810+0000","last_active":"2024-04-24T23:27:30.082036+0000","last_peered":"2024-04-24T23:27:30.082036+0000","last_clean":"2024-04-24T23:27:30.082036+0000","last_became_active":"2024-04-24T23:15:36.987540+0000","last_became_peered":"2024-04-24T23:15:36.987540+0000","last_unstale":"2024-04-24T23:27:30.082036+0000","last_undegraded":"2024-04-24T23:27:30.082036+0000","last_fullsized":"2024-04-24T23:27:30.082036+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:49:56.176021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.29","version":"30'13","reported_seq":50,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:31.113409+0000","last_change":"2024-04-24T23:15:36.981187+0000","last_active":"2024-04-24T23:27:31.113409+0000","last_peered":"2024-04-24T23:27:31.113409+0000","last_clean":"2024-04-24T23:27:31.113409+0000","last_became_active":"2024-04-24T23:15:36.980838+0000","last_became_peered":"2024-04-24T23:15:36.980838+0000","last_unstale":"2024-04-24T23:27:31.113409+0000","last_undegraded":"2024-04-24T23:27:31.113409+0000","last_fullsized":"2024-04-24T23:27:31.113409+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:57:41.178367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2a","version":"30'11","reported_seq":48,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:27.562545+0000","last_change":"2024-04-24T23:15:36.999238+0000","last_active":"2024-04-24T23:27:27.562545+0000","last_peered":"2024-04-24T23:27:27.562545+0000","last_clean":"2024-04-24T23:27:27.562545+0000","last_became_active":"2024-04-24T23:15:36.996245+0000","last_became_peered":"2024-04-24T23:15:36.996245+0000","last_unstale":"2024-04-24T23:27:27.562545+0000","last_undegraded":"2024-04-24T23:27:27.562545+0000","last_fullsized":"2024-04-24T23:27:27.562545+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:21:20.443640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"30'10","reported_seq":47,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:30.193701+0000","last_change":"2024-04-24T23:15:36.981219+0000","last_active":"2024-04-24T23:27:30.193701+0000","last_peered":"2024-04-24T23:27:30.193701+0000","last_clean":"2024-04-24T23:27:30.193701+0000","last_became_active":"2024-04-24T23:15:36.980911+0000","last_became_peered":"2024-04-24T23:15:36.980911+0000","last_unstale":"2024-04-24T23:27:30.193701+0000","last_undegraded":"2024-04-24T23:27:30.193701+0000","last_fullsized":"2024-04-24T23:27:30.193701+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:19:57.973402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2c","version":"30'7","reported_seq":44,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:02.756928+0000","last_change":"2024-04-24T23:15:36.991418+0000","last_active":"2024-04-24T23:27:02.756928+0000","last_peered":"2024-04-24T23:27:02.756928+0000","last_clean":"2024-04-24T23:27:02.756928+0000","last_became_active":"2024-04-24T23:15:36.989792+0000","last_became_peered":"2024-04-24T23:15:36.989792+0000","last_unstale":"2024-04-24T23:27:02.756928+0000","last_undegraded":"2024-04-24T23:27:02.756928+0000","last_fullsized":"2024-04-24T23:27:02.756928+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:02:53.601311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2d","version":"30'12","reported_seq":49,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:28.000118+0000","last_change":"2024-04-24T23:15:36.987810+0000","last_active":"2024-04-24T23:27:28.000118+0000","last_peered":"2024-04-24T23:27:28.000118+0000","last_clean":"2024-04-24T23:27:28.000118+0000","last_became_active":"2024-04-24T23:15:36.987494+0000","last_became_peered":"2024-04-24T23:15:36.987494+0000","last_unstale":"2024-04-24T23:27:28.000118+0000","last_undegraded":"2024-04-24T23:27:28.000118+0000","last_fullsized":"2024-04-24T23:27:28.000118+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:43:21.141172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2e","version":"30'10","reported_seq":47,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:17.418750+0000","last_change":"2024-04-24T23:15:36.986576+0000","last_active":"2024-04-24T23:27:17.418750+0000","last_peered":"2024-04-24T23:27:17.418750+0000","last_clean":"2024-04-24T23:27:17.418750+0000","last_became_active":"2024-04-24T23:15:36.986272+0000","last_became_peered":"2024-04-24T23:15:36.986272+0000","last_unstale":"2024-04-24T23:27:17.418750+0000","last_undegraded":"2024-04-24T23:27:17.418750+0000","last_fullsized":"2024-04-24T23:27:17.418750+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:55:08.894945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2f","version":"30'9","reported_seq":51,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:37.471843+0000","last_change":"2024-04-24T23:16:18.294151+0000","last_active":"2024-04-24T23:27:37.471843+0000","last_peered":"2024-04-24T23:27:37.471843+0000","last_clean":"2024-04-24T23:27:37.471843+0000","last_became_active":"2024-04-24T23:16:18.293579+0000","last_became_peered":"2024-04-24T23:16:18.293579+0000","last_unstale":"2024-04-24T23:27:37.471843+0000","last_undegraded":"2024-04-24T23:27:37.471843+0000","last_fullsized":"2024-04-24T23:27:37.471843+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:22:20.603555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.30","version":"30'10","reported_seq":47,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:26.414660+0000","last_change":"2024-04-24T23:15:36.996521+0000","last_active":"2024-04-24T23:27:26.414660+0000","last_peered":"2024-04-24T23:27:26.414660+0000","last_clean":"2024-04-24T23:27:26.414660+0000","last_became_active":"2024-04-24T23:15:36.995507+0000","last_became_peered":"2024-04-24T23:15:36.995507+0000","last_unstale":"2024-04-24T23:27:26.414660+0000","last_undegraded":"2024-04-24T23:27:26.414660+0000","last_fullsized":"2024-04-24T23:27:26.414660+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:58:50.153471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.31","version":"30'11","reported_seq":48,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:21.339459+0000","last_change":"2024-04-24T23:15:36.984994+0000","last_active":"2024-04-24T23:27:21.339459+0000","last_peered":"2024-04-24T23:27:21.339459+0000","last_clean":"2024-04-24T23:27:21.339459+0000","last_became_active":"2024-04-24T23:15:36.984741+0000","last_became_peered":"2024-04-24T23:15:36.984741+0000","last_unstale":"2024-04-24T23:27:21.339459+0000","last_undegraded":"2024-04-24T23:27:21.339459+0000","last_fullsized":"2024-04-24T23:27:21.339459+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:45:23.429836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.32","version":"30'11","reported_seq":48,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:26.511382+0000","last_change":"2024-04-24T23:15:36.966165+0000","last_active":"2024-04-24T23:27:26.511382+0000","last_peered":"2024-04-24T23:27:26.511382+0000","last_clean":"2024-04-24T23:27:26.511382+0000","last_became_active":"2024-04-24T23:15:36.965877+0000","last_became_peered":"2024-04-24T23:15:36.965877+0000","last_unstale":"2024-04-24T23:27:26.511382+0000","last_undegraded":"2024-04-24T23:27:26.511382+0000","last_fullsized":"2024-04-24T23:27:26.511382+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:30:12.390978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.33","version":"30'9","reported_seq":45,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:25:42.741941+0000","last_change":"2024-04-24T23:15:36.990667+0000","last_active":"2024-04-24T23:25:42.741941+0000","last_peered":"2024-04-24T23:25:42.741941+0000","last_clean":"2024-04-24T23:25:42.741941+0000","last_became_active":"2024-04-24T23:15:36.990383+0000","last_became_peered":"2024-04-24T23:15:36.990383+0000","last_unstale":"2024-04-24T23:25:42.741941+0000","last_undegraded":"2024-04-24T23:25:42.741941+0000","last_fullsized":"2024-04-24T23:25:42.741941+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:36:43.850046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"30'7","reported_seq":44,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:37.844796+0000","last_change":"2024-04-24T23:15:36.991686+0000","last_active":"2024-04-24T23:27:37.844796+0000","last_peered":"2024-04-24T23:27:37.844796+0000","last_clean":"2024-04-24T23:27:37.844796+0000","last_became_active":"2024-04-24T23:15:36.990155+0000","last_became_peered":"2024-04-24T23:15:36.990155+0000","last_unstale":"2024-04-24T23:27:37.844796+0000","last_undegraded":"2024-04-24T23:27:37.844796+0000","last_fullsized":"2024-04-24T23:27:37.844796+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:00:52.424019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.35","version":"30'12","reported_seq":48,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:25:42.767307+0000","last_change":"2024-04-24T23:15:36.992958+0000","last_active":"2024-04-24T23:25:42.767307+0000","last_peered":"2024-04-24T23:25:42.767307+0000","last_clean":"2024-04-24T23:25:42.767307+0000","last_became_active":"2024-04-24T23:15:36.992608+0000","last_became_peered":"2024-04-24T23:15:36.992608+0000","last_unstale":"2024-04-24T23:25:42.767307+0000","last_undegraded":"2024-04-24T23:25:42.767307+0000","last_fullsized":"2024-04-24T23:25:42.767307+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:15:30.416763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.36","version":"30'10","reported_seq":46,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:25:42.767610+0000","last_change":"2024-04-24T23:15:36.992943+0000","last_active":"2024-04-24T23:25:42.767610+0000","last_peered":"2024-04-24T23:25:42.767610+0000","last_clean":"2024-04-24T23:25:42.767610+0000","last_became_active":"2024-04-24T23:15:36.992567+0000","last_became_peered":"2024-04-24T23:15:36.992567+0000","last_unstale":"2024-04-24T23:25:42.767610+0000","last_undegraded":"2024-04-24T23:25:42.767610+0000","last_fullsized":"2024-04-24T23:25:42.767610+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:50:43.209232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.37","version":"30'12","reported_seq":48,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:27.436849+0000","last_change":"2024-04-24T23:15:36.999697+0000","last_active":"2024-04-24T23:27:27.436849+0000","last_peered":"2024-04-24T23:27:27.436849+0000","last_clean":"2024-04-24T23:27:27.436849+0000","last_became_active":"2024-04-24T23:15:36.999413+0000","last_became_peered":"2024-04-24T23:15:36.999413+0000","last_unstale":"2024-04-24T23:27:27.436849+0000","last_undegraded":"2024-04-24T23:27:27.436849+0000","last_fullsized":"2024-04-24T23:27:27.436849+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:25:42.856471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"27'3","reported_seq":39,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:25:42.717468+0000","last_change":"2024-04-24T23:15:36.986742+0000","last_active":"2024-04-24T23:25:42.717468+0000","last_peered":"2024-04-24T23:25:42.717468+0000","last_clean":"2024-04-24T23:25:42.717468+0000","last_became_active":"2024-04-24T23:15:36.986013+0000","last_became_peered":"2024-04-24T23:15:36.986013+0000","last_unstale":"2024-04-24T23:25:42.717468+0000","last_undegraded":"2024-04-24T23:25:42.717468+0000","last_fullsized":"2024-04-24T23:25:42.717468+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:16:15.203151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"30'5","reported_seq":41,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:26.849159+0000","last_change":"2024-04-24T23:15:36.984336+0000","last_active":"2024-04-24T23:27:26.849159+0000","last_peered":"2024-04-24T23:27:26.849159+0000","last_clean":"2024-04-24T23:27:26.849159+0000","last_became_active":"2024-04-24T23:15:36.983966+0000","last_became_peered":"2024-04-24T23:15:36.983966+0000","last_unstale":"2024-04-24T23:27:26.849159+0000","last_undegraded":"2024-04-24T23:27:26.849159+0000","last_fullsized":"2024-04-24T23:27:26.849159+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:54:16.319608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3a","version":"30'9","reported_seq":45,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:36.677504+0000","last_change":"2024-04-24T23:15:36.979079+0000","last_active":"2024-04-24T23:27:36.677504+0000","last_peered":"2024-04-24T23:27:36.677504+0000","last_clean":"2024-04-24T23:27:36.677504+0000","last_became_active":"2024-04-24T23:15:36.978706+0000","last_became_peered":"2024-04-24T23:15:36.978706+0000","last_unstale":"2024-04-24T23:27:36.677504+0000","last_undegraded":"2024-04-24T23:27:36.677504+0000","last_fullsized":"2024-04-24T23:27:36.677504+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:53:39.888150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"30'7","reported_seq":43,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:25:46.408265+0000","last_change":"2024-04-24T23:15:36.986008+0000","last_active":"2024-04-24T23:25:46.408265+0000","last_peered":"2024-04-24T23:25:46.408265+0000","last_clean":"2024-04-24T23:25:46.408265+0000","last_became_active":"2024-04-24T23:15:36.985552+0000","last_became_peered":"2024-04-24T23:15:36.985552+0000","last_unstale":"2024-04-24T23:25:46.408265+0000","last_undegraded":"2024-04-24T23:25:46.408265+0000","last_fullsized":"2024-04-24T23:25:46.408265+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:36:50.827503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"30'6","reported_seq":43,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:25:44.818461+0000","last_change":"2024-04-24T23:15:36.978673+0000","last_active":"2024-04-24T23:25:44.818461+0000","last_peered":"2024-04-24T23:25:44.818461+0000","last_clean":"2024-04-24T23:25:44.818461+0000","last_became_active":"2024-04-24T23:15:36.972744+0000","last_became_peered":"2024-04-24T23:15:36.972744+0000","last_unstale":"2024-04-24T23:25:44.818461+0000","last_undegraded":"2024-04-24T23:25:44.818461+0000","last_fullsized":"2024-04-24T23:25:44.818461+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:28:57.028825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3d","version":"30'8","reported_seq":45,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:27.328493+0000","last_change":"2024-04-24T23:15:36.970238+0000","last_active":"2024-04-24T23:27:27.328493+0000","last_peered":"2024-04-24T23:27:27.328493+0000","last_clean":"2024-04-24T23:27:27.328493+0000","last_became_active":"2024-04-24T23:15:36.969751+0000","last_became_peered":"2024-04-24T23:15:36.969751+0000","last_unstale":"2024-04-24T23:27:27.328493+0000","last_undegraded":"2024-04-24T23:27:27.328493+0000","last_fullsized":"2024-04-24T23:27:27.328493+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:19:56.456801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3e","version":"26'2","reported_seq":39,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:25:42.719286+0000","last_change":"2024-04-24T23:15:36.968825+0000","last_active":"2024-04-24T23:25:42.719286+0000","last_peered":"2024-04-24T23:25:42.719286+0000","last_clean":"2024-04-24T23:25:42.719286+0000","last_became_active":"2024-04-24T23:15:36.968543+0000","last_became_peered":"2024-04-24T23:15:36.968543+0000","last_unstale":"2024-04-24T23:25:42.719286+0000","last_undegraded":"2024-04-24T23:25:42.719286+0000","last_fullsized":"2024-04-24T23:25:42.719286+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:36:48.795062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3f","version":"27'4","reported_seq":41,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:25:42.818133+0000","last_change":"2024-04-24T23:15:36.992800+0000","last_active":"2024-04-24T23:25:42.818133+0000","last_peered":"2024-04-24T23:25:42.818133+0000","last_clean":"2024-04-24T23:25:42.818133+0000","last_became_active":"2024-04-24T23:15:36.992476+0000","last_became_peered":"2024-04-24T23:15:36.992476+0000","last_unstale":"2024-04-24T23:25:42.818133+0000","last_undegraded":"2024-04-24T23:25:42.818133+0000","last_fullsized":"2024-04-24T23:25:42.818133+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:01:21.103559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":4700077957,"num_objects":1311,"num_object_clones":0,"num_object_copies":5244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1311,"num_whiteouts":0,"num_read":2252,"num_read_kb":307,"num_write":2059431,"num_write_kb":29192446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":9611886592,"data_stored":9611886592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":140285,"ondisk_log_size":140285,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":535,"ondisk_log_size":535,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":80027978,"num_objects":127,"num_object_clones":0,"num_object_copies":254,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":127,"num_whiteouts":0,"num_read":353,"num_read_kb":84742,"num_write":4104,"num_write_kb":81390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":74,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":160292864,"data_stored":160055956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3841,"ondisk_log_size":3841,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574999,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2180728,"kb_used_data":1138288,"kb_used_omap":1,"kb_used_meta":1040382,"kb_avail":92191112,"statfs":{"total":96636764160,"available":94403698688,"internally_reserved":0,"allocated":1165606912,"data_stored":1165385664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":1065351627},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,5,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574999,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2143028,"kb_used_data":1193196,"kb_used_omap":1,"kb_used_meta":945726,"kb_avail":92228812,"statfs":{"total":96636764160,"available":94442303488,"internally_reserved":0,"allocated":1221832704,"data_stored":1221631296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":968423878},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574999,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2098588,"kb_used_data":1283924,"kb_used_omap":1,"kb_used_meta":814654,"kb_avail":92273252,"statfs":{"total":96636764160,"available":94487810048,"internally_reserved":0,"allocated":1314738176,"data_stored":1314525951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":834206149},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834575000,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1887128,"kb_used_data":1094608,"kb_used_omap":1,"kb_used_meta":792510,"kb_avail":92484712,"statfs":{"total":96636764160,"available":94704345088,"internally_reserved":0,"allocated":1120878592,"data_stored":1120642456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":811530693},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574995,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2374944,"kb_used_data":1342616,"kb_used_omap":1,"kb_used_meta":1032318,"kb_avail":91996896,"statfs":{"total":96636764160,"available":94204821504,"internally_reserved":0,"allocated":1374838784,"data_stored":1374612627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":1057094087},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574994,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2047988,"kb_used_data":1323948,"kb_used_omap":1,"kb_used_meta":724030,"kb_avail":92323852,"statfs":{"total":96636764160,"available":94539624448,"internally_reserved":0,"allocated":1355722752,"data_stored":1355521710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":741407173},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574994,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3096668,"kb_used_data":2038100,"kb_used_omap":1,"kb_used_meta":1054462,"kb_avail":91275172,"statfs":{"total":96636764160,"available":93465776128,"internally_reserved":0,"allocated":2087014400,"data_stored":2086804775,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1079769541},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574998,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1424608,"kb_used_data":756824,"kb_used_omap":1,"kb_used_meta":667774,"kb_avail":92947232,"statfs":{"total":96636764160,"available":95177965568,"internally_reserved":0,"allocated":774987776,"data_stored":774768421,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":683801030},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":25436160,"data_stored":25402203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":21225472,"data_stored":21204797,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12599296,"data_stored":12583652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":37793792,"data_stored":37753033,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12636160,"data_stored":12585422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":25284608,"data_stored":25261333,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12603392,"data_stored":12587382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12713984,"data_stored":12678134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":648560640,"data_stored":648560640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2048090112,"data_stored":2048090112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1273589760,"data_stored":1273589760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1263316992,"data_stored":1263316992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":986279936,"data_stored":986279936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1141731328,"data_stored":1141731328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1118724096,"data_stored":1118724096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1131593728,"data_stored":1131593728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T23:27:43.516 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T23:27:43.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:43.884+0000 7f217f800640 1 Processor -- start 2024-04-24T23:27:43.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:43.884+0000 7f217f800640 1 -- start start 2024-04-24T23:27:43.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:43.886+0000 7f217f800640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f21780c7dc0 0x7f21781623a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:43.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:43.886+0000 7f217f800640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f21780c7230 con 0x7f2178159110 2024-04-24T23:27:43.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:43.886+0000 7f217f800640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f21780c74d0 con 0x7f21780c81c0 2024-04-24T23:27:43.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:43.886+0000 7f217f800640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f21781594f0 con 0x7f21780c7dc0 2024-04-24T23:27:43.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:43.887+0000 7f217ce00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f2178159110 0x7f2178072420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:38096/0 (socket says 172.21.15.6:38096) 2024-04-24T23:27:43.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:43.887+0000 7f217ce00640 1 -- 172.21.15.6:0/3767873820 learned_addr learned my addr 172.21.15.6:0/3767873820 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:27:43.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:43.888+0000 7f2177400640 1 -- 172.21.15.6:0/3767873820 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 847262011 0 0) 0x7f21780c7230 con 0x7f2178159110 2024-04-24T23:27:43.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:43.888+0000 7f2177400640 1 -- 172.21.15.6:0/3767873820 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2160003520 con 0x7f2178159110 2024-04-24T23:27:43.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:43.888+0000 7f2177400640 1 -- 172.21.15.6:0/3767873820 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3804053590 0 0) 0x7f2160003520 con 0x7f2178159110 2024-04-24T23:27:43.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:43.888+0000 7f2177400640 1 -- 172.21.15.6:0/3767873820 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f21780c7dc0 msgr2=0x7f21781623a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-24T23:27:43.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:43.888+0000 7f2177400640 1 --2- 172.21.15.6:0/3767873820 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f21780c7dc0 0x7f21781623a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:43.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:43.888+0000 7f2177400640 1 -- 172.21.15.6:0/3767873820 >> v1:172.21.15.190:6789/0 conn(0x7f21780c81c0 legacy=0x7f21781628e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-24T23:27:43.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:43.888+0000 7f2177400640 1 -- 172.21.15.6:0/3767873820 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f217806e440 con 0x7f2178159110 2024-04-24T23:27:43.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:43.889+0000 7f2177400640 1 -- 172.21.15.6:0/3767873820 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3463268701 0 0) 0x7f216c001690 con 0x7f2178159110 2024-04-24T23:27:43.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:43.889+0000 7f2177400640 1 -- 172.21.15.6:0/3767873820 >> v1:172.21.15.6:6789/0 conn(0x7f2178159110 legacy=0x7f2178072420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:43.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:43.889+0000 7f2177400640 1 --2- 172.21.15.6:0/3767873820 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f21600039e0 0x7f2160003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:43.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:43.889+0000 7f2177400640 1 --2- 172.21.15.6:0/3767873820 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2160004800 0x7f2160004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:43.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:43.889+0000 7f2177400640 1 --2- 172.21.15.6:0/3767873820 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2160005210 0x7f216000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:43.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:43.889+0000 7f2177400640 1 -- 172.21.15.6:0/3767873820 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f21781594f0 con 0x7f2160005210 2024-04-24T23:27:43.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:43.889+0000 7f2177400640 1 -- 172.21.15.6:0/3767873820 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f216000c0d0 con 0x7f21600039e0 2024-04-24T23:27:43.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:43.889+0000 7f2177400640 1 -- 172.21.15.6:0/3767873820 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f216000c2d0 con 0x7f2160004800 2024-04-24T23:27:43.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:43.890+0000 7f2177e00640 1 --2- 172.21.15.6:0/3767873820 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2160004800 0x7f2160004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:43.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:43.890+0000 7f217d800640 1 --2- 172.21.15.6:0/3767873820 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2160005210 0x7f216000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:43.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:43.891+0000 7f2177e00640 1 -- 172.21.15.6:0/3767873820 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f21600039e0 msgr2=0x7f2160003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:27:43.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:43.891+0000 7f217ce00640 1 --2- 172.21.15.6:0/3767873820 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f21600039e0 0x7f2160003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:43.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:43.891+0000 7f2177e00640 1 --2- 172.21.15.6:0/3767873820 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f21600039e0 0x7f2160003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:43.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:43.891+0000 7f2177e00640 1 -- 172.21.15.6:0/3767873820 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2160005210 msgr2=0x7f216000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:27:43.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:43.891+0000 7f2177e00640 1 --2- 172.21.15.6:0/3767873820 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2160005210 0x7f216000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:43.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:43.891+0000 7f2177e00640 1 -- 172.21.15.6:0/3767873820 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2160003520 con 0x7f2160004800 2024-04-24T23:27:43.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:43.895+0000 7f2177e00640 1 --2- 172.21.15.6:0/3767873820 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2160004800 0x7f2160004c80 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f2178070fb0 tx=0x7f2160005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=37e57144914987ef server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:27:43.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:43.895+0000 7f2177400640 1 -- 172.21.15.6:0/3767873820 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2164037070 con 0x7f2160004800 2024-04-24T23:27:43.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:43.895+0000 7f2177400640 1 -- 172.21.15.6:0/3767873820 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f216402a950 con 0x7f2160004800 2024-04-24T23:27:43.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:43.896+0000 7f217f800640 1 -- 172.21.15.6:0/3767873820 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2160004800 msgr2=0x7f2160004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:43.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:43.896+0000 7f217f800640 1 --2- 172.21.15.6:0/3767873820 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2160004800 0x7f2160004c80 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f2178070fb0 tx=0x7f2160005830 comp rx=0 tx=0).stop 2024-04-24T23:27:43.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:43.897+0000 7f217d800640 1 -- 172.21.15.6:0/3767873820 reap_dead start 2024-04-24T23:27:43.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:43.898+0000 7f217ce00640 1 --2- 172.21.15.6:0/3767873820 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f21600039e0 0x7f2160003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:27:43.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:43.899+0000 7f217f800640 1 -- 172.21.15.6:0/3767873820 shutdown_connections 2024-04-24T23:27:43.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:43.899+0000 7f217f800640 1 -- 172.21.15.6:0/3767873820 >> 172.21.15.6:0/3767873820 conn(0x7f217806b0e0 msgr2=0x7f217806b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:27:43.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:43.899+0000 7f217f800640 1 -- 172.21.15.6:0/3767873820 shutdown_connections 2024-04-24T23:27:43.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:43.899+0000 7f217f800640 1 -- 172.21.15.6:0/3767873820 wait complete. 2024-04-24T23:27:43.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:43.903+0000 7f217f800640 1 Processor -- start 2024-04-24T23:27:43.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:43.909+0000 7f217f800640 1 -- start start 2024-04-24T23:27:43.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:43.913+0000 7f217f800640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2160004800 0x7f21781ec910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:43.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:43.914+0000 7f217f800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f217815f9c0 0x7f21781ece50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:43.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:43.914+0000 7f217f800640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f217815def0 0x7f217815e370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:43.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:43.914+0000 7f217f800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f217806ece0 con 0x7f217815f9c0 2024-04-24T23:27:43.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:43.914+0000 7f217f800640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f217806ea00 con 0x7f2160004800 2024-04-24T23:27:43.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:43.914+0000 7f217f800640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f217806f200 con 0x7f217815def0 2024-04-24T23:27:43.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:43.914+0000 7f217ce00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2160004800 0x7f21781ec910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:43.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:43.915+0000 7f217ce00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2160004800 0x7f21781ec910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:40722/0 (socket says 172.21.15.6:40722) 2024-04-24T23:27:43.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:43.915+0000 7f217ce00640 1 -- 172.21.15.6:0/3712804555 learned_addr learned my addr 172.21.15.6:0/3712804555 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:27:43.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:43.916+0000 7f217d800640 1 --2- 172.21.15.6:0/3712804555 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f217815def0 0x7f217815e370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:43.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:43.917+0000 7f217d800640 1 -- 172.21.15.6:0/3712804555 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2160004800 msgr2=0x7f21781ec910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:43.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:43.917+0000 7f217d800640 1 --2- 172.21.15.6:0/3712804555 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2160004800 0x7f21781ec910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:43.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:43.917+0000 7f217d800640 1 -- 172.21.15.6:0/3712804555 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f217815f9c0 msgr2=0x7f21781ece50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-24T23:27:43.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:43.917+0000 7f217d800640 1 --2- 172.21.15.6:0/3712804555 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f217815f9c0 0x7f21781ece50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:43.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:43.917+0000 7f217d800640 1 -- 172.21.15.6:0/3712804555 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f217815ce80 con 0x7f217815def0 2024-04-24T23:27:43.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:43.918+0000 7f217d800640 1 --2- 172.21.15.6:0/3712804555 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f217815def0 0x7f217815e370 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f21780cd360 tx=0x7f217000b700 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:27:43.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:43.918+0000 7f2174c00640 1 -- 172.21.15.6:0/3712804555 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2170014070 con 0x7f217815def0 2024-04-24T23:27:43.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:43.918+0000 7f217f800640 1 -- 172.21.15.6:0/3712804555 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f217815ff70 con 0x7f217815def0 2024-04-24T23:27:43.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:43.918+0000 7f217f800640 1 -- 172.21.15.6:0/3712804555 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f217815d4c0 con 0x7f217815def0 2024-04-24T23:27:43.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:43.918+0000 7f2174c00640 1 -- 172.21.15.6:0/3712804555 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f217000e620 con 0x7f217815def0 2024-04-24T23:27:43.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:43.918+0000 7f2174c00640 1 -- 172.21.15.6:0/3712804555 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2170008040 con 0x7f217815def0 2024-04-24T23:27:43.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:43.921+0000 7f217f800640 1 -- 172.21.15.6:0/3712804555 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2154005290 con 0x7f217815def0 2024-04-24T23:27:43.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:43.929+0000 7f2174c00640 1 -- 172.21.15.6:0/3712804555 <== mon.2 v2:172.21.15.190:3301/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f217000e9f0 con 0x7f217815def0 2024-04-24T23:27:43.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:43.935+0000 7f2174c00640 1 --2- 172.21.15.6:0/3712804555 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f215806c030 0x7f215806e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:43.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:43.936+0000 7f2174c00640 1 -- 172.21.15.6:0/3712804555 <== mon.2 v2:172.21.15.190:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6934+0+0 (secure 0 0 0) 0x7f217000e7d0 con 0x7f217815def0 2024-04-24T23:27:43.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:43.937+0000 7f217ce00640 1 --2- 172.21.15.6:0/3712804555 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f215806c030 0x7f215806e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:43.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:43.945+0000 7f217ce00640 1 --2- 172.21.15.6:0/3712804555 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f215806c030 0x7f215806e4f0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f216c001ab0 tx=0x7f216c001e30 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:27:43.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:43.945+0000 7f2174c00640 1 -- 172.21.15.6:0/3712804555 <== mon.2 v2:172.21.15.190:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f2170059770 con 0x7f217815def0 2024-04-24T23:27:44.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:44.243+0000 7f217f800640 1 -- 172.21.15.6:0/3712804555 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2154005850 con 0x7f217815def0 2024-04-24T23:27:44.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:44.244+0000 7f2174c00640 1 -- 172.21.15.6:0/3712804555 <== mon.2 v2:172.21.15.190:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v30) ==== 74+0+19063 (secure 0 0 0) 0x7f2170059070 con 0x7f217815def0 2024-04-24T23:27:44.246 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:27:44.246 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":30,"fsid":"18bef68d-5e6d-4ec5-862b-b6f206e2588d","created":"2024-04-24T23:15:12.535072+0000","modified":"2024-04-24T23:25:42.694801+0000","last_up_change":"2024-04-24T23:15:23.773223+0000","last_in_change":"2024-04-24T23:15:14.905328+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T23:15:24.747441+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-24T23:15:34.051800+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-24T23:15:35.436870+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-24T23:15:39.034257+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"3b7a212d-d498-462e-90bf-ed493c803a09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":3022569437},{"type":"v1","addr":"172.21.15.6:6809","nonce":3022569437}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":3022569437},{"type":"v1","addr":"172.21.15.6:6811","nonce":3022569437}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":3022569437},{"type":"v1","addr":"172.21.15.6:6815","nonce":3022569437}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":3022569437},{"type":"v1","addr":"172.21.15.6:6813","nonce":3022569437}]},"public_addr":"172.21.15.6:6809/3022569437","cluster_addr":"172.21.15.6:6811/3022569437","heartbeat_back_addr":"172.21.15.6:6815/3022569437","heartbeat_front_addr":"172.21.15.6:6813/3022569437","state":["exists","up"]},{"osd":1,"uuid":"7c97f5a2-1735-4775-9f15-eb50099b19d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":3687836888},{"type":"v1","addr":"172.21.15.6:6817","nonce":3687836888}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":3687836888},{"type":"v1","addr":"172.21.15.6:6819","nonce":3687836888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":3687836888},{"type":"v1","addr":"172.21.15.6:6823","nonce":3687836888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":3687836888},{"type":"v1","addr":"172.21.15.6:6821","nonce":3687836888}]},"public_addr":"172.21.15.6:6817/3687836888","cluster_addr":"172.21.15.6:6819/3687836888","heartbeat_back_addr":"172.21.15.6:6823/3687836888","heartbeat_front_addr":"172.21.15.6:6821/3687836888","state":["exists","up"]},{"osd":2,"uuid":"d86bbc18-438c-4ec2-8018-ba7dd78f790b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":1020213724},{"type":"v1","addr":"172.21.15.6:6801","nonce":1020213724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":1020213724},{"type":"v1","addr":"172.21.15.6:6803","nonce":1020213724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":1020213724},{"type":"v1","addr":"172.21.15.6:6807","nonce":1020213724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":1020213724},{"type":"v1","addr":"172.21.15.6:6805","nonce":1020213724}]},"public_addr":"172.21.15.6:6801/1020213724","cluster_addr":"172.21.15.6:6803/1020213724","heartbeat_back_addr":"172.21.15.6:6807/1020213724","heartbeat_front_addr":"172.21.15.6:6805/1020213724","state":["exists","up"]},{"osd":3,"uuid":"3754ae16-4248-40dd-95f9-0f59c0eb5426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6824","nonce":3163883468},{"type":"v1","addr":"172.21.15.6:6825","nonce":3163883468}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":3163883468},{"type":"v1","addr":"172.21.15.6:6827","nonce":3163883468}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":3163883468},{"type":"v1","addr":"172.21.15.6:6831","nonce":3163883468}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":3163883468},{"type":"v1","addr":"172.21.15.6:6829","nonce":3163883468}]},"public_addr":"172.21.15.6:6825/3163883468","cluster_addr":"172.21.15.6:6827/3163883468","heartbeat_back_addr":"172.21.15.6:6831/3163883468","heartbeat_front_addr":"172.21.15.6:6829/3163883468","state":["exists","up"]},{"osd":4,"uuid":"8b2913a0-d006-479c-84ed-5b21e0e6c392","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":1989500934},{"type":"v1","addr":"172.21.15.190:6809","nonce":1989500934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":1989500934},{"type":"v1","addr":"172.21.15.190:6811","nonce":1989500934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":1989500934},{"type":"v1","addr":"172.21.15.190:6815","nonce":1989500934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":1989500934},{"type":"v1","addr":"172.21.15.190:6813","nonce":1989500934}]},"public_addr":"172.21.15.190:6809/1989500934","cluster_addr":"172.21.15.190:6811/1989500934","heartbeat_back_addr":"172.21.15.190:6815/1989500934","heartbeat_front_addr":"172.21.15.190:6813/1989500934","state":["exists","up"]},{"osd":5,"uuid":"499b79d0-0585-4dae-ab69-196a551793da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":3687548412},{"type":"v1","addr":"172.21.15.190:6801","nonce":3687548412}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":3687548412},{"type":"v1","addr":"172.21.15.190:6803","nonce":3687548412}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":3687548412},{"type":"v1","addr":"172.21.15.190:6807","nonce":3687548412}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":3687548412},{"type":"v1","addr":"172.21.15.190:6805","nonce":3687548412}]},"public_addr":"172.21.15.190:6801/3687548412","cluster_addr":"172.21.15.190:6803/3687548412","heartbeat_back_addr":"172.21.15.190:6807/3687548412","heartbeat_front_addr":"172.21.15.190:6805/3687548412","state":["exists","up"]},{"osd":6,"uuid":"6ba6df3b-4a6b-4a89-a890-0f79f387d4d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":870571564},{"type":"v1","addr":"172.21.15.190:6817","nonce":870571564}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":870571564},{"type":"v1","addr":"172.21.15.190:6819","nonce":870571564}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":870571564},{"type":"v1","addr":"172.21.15.190:6823","nonce":870571564}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":870571564},{"type":"v1","addr":"172.21.15.190:6821","nonce":870571564}]},"public_addr":"172.21.15.190:6817/870571564","cluster_addr":"172.21.15.190:6819/870571564","heartbeat_back_addr":"172.21.15.190:6823/870571564","heartbeat_front_addr":"172.21.15.190:6821/870571564","state":["exists","up"]},{"osd":7,"uuid":"e9668b1c-a000-4332-86e8-4efd73b02409","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":123639197},{"type":"v1","addr":"172.21.15.190:6825","nonce":123639197}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":123639197},{"type":"v1","addr":"172.21.15.190:6827","nonce":123639197}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":123639197},{"type":"v1","addr":"172.21.15.190:6831","nonce":123639197}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":123639197},{"type":"v1","addr":"172.21.15.190:6829","nonce":123639197}]},"public_addr":"172.21.15.190:6825/123639197","cluster_addr":"172.21.15.190:6827/123639197","heartbeat_back_addr":"172.21.15.190:6831/123639197","heartbeat_front_addr":"172.21.15.190:6829/123639197","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.321646+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.363873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.466963+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.531989+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:19.990707+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.043543+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.032492+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.115004+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.2f","mappings":[{"from":0,"to":7}]},{"pgid":"4.19","mappings":[{"from":7,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.6:6835/489092393":"2024-04-25T23:25:42.694416+0000","172.21.15.190:6837/580315269":"2024-04-25T23:24:32.678217+0000","172.21.15.6:6834/105267896":"2024-04-25T23:20:17.585424+0000","172.21.15.6:6833/489092393":"2024-04-25T23:25:42.694416+0000","172.21.15.6:6832/105267896":"2024-04-25T23:20:17.585424+0000","172.21.15.190:6834/580315269":"2024-04-25T23:24:32.678217+0000","172.21.15.6:6837/2590421094":"2024-04-25T23:18:27.556337+0000","172.21.15.6:6836/2590421094":"2024-04-25T23:18:27.556337+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T23:27:44.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:44.248+0000 7f217f800640 1 -- 172.21.15.6:0/3712804555 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f215806c030 msgr2=0x7f215806e4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:44.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:44.248+0000 7f217f800640 1 --2- 172.21.15.6:0/3712804555 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f215806c030 0x7f215806e4f0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f216c001ab0 tx=0x7f216c001e30 comp rx=0 tx=0).stop 2024-04-24T23:27:44.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:44.248+0000 7f217f800640 1 -- 172.21.15.6:0/3712804555 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f217815def0 msgr2=0x7f217815e370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:44.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:44.248+0000 7f217f800640 1 --2- 172.21.15.6:0/3712804555 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f217815def0 0x7f217815e370 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f21780cd360 tx=0x7f217000b700 comp rx=0 tx=0).stop 2024-04-24T23:27:44.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:44.248+0000 7f217f800640 1 -- 172.21.15.6:0/3712804555 shutdown_connections 2024-04-24T23:27:44.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:44.248+0000 7f217f800640 1 --2- 172.21.15.6:0/3712804555 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f215806c030 0x7f215806e4f0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:44.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:44.248+0000 7f217f800640 1 --2- 172.21.15.6:0/3712804555 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f217815def0 0x7f217815e370 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:44.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:44.248+0000 7f217f800640 1 --2- 172.21.15.6:0/3712804555 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f217815f9c0 0x7f21781ece50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:44.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:44.248+0000 7f217f800640 1 --2- 172.21.15.6:0/3712804555 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2160004800 0x7f21781ec910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:44.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:44.248+0000 7f217f800640 1 -- 172.21.15.6:0/3712804555 >> 172.21.15.6:0/3712804555 conn(0x7f217806b0e0 msgr2=0x7f2178155940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:27:44.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:44.248+0000 7f217f800640 1 -- 172.21.15.6:0/3712804555 shutdown_connections 2024-04-24T23:27:44.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:44.248+0000 7f217f800640 1 -- 172.21.15.6:0/3712804555 wait complete. 2024-04-24T23:27:45.275 INFO:tasks.ceph:Scrubbing osd.0 2024-04-24T23:27:45.275 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-04-24T23:27:45.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:45.618+0000 7fa0bd800640 1 Processor -- start 2024-04-24T23:27:45.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:45.618+0000 7fa0bd800640 1 -- start start 2024-04-24T23:27:45.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:45.619+0000 7fa0bd800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa0b8151de0 0x7fa0b80c36e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:45.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:45.619+0000 7fa0bd800640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fa0b8157940 0x7fa0b8071840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:45.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:45.619+0000 7fa0bd800640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa0b8071280 con 0x7fa0b8071ed0 2024-04-24T23:27:45.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:45.619+0000 7fa0bd800640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7fa0b8157db0 con 0x7fa0b8151de0 2024-04-24T23:27:45.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:45.619+0000 7fa0bd800640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7fa0b8071560 con 0x7fa0b8157940 2024-04-24T23:27:45.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:45.620+0000 7fa0b6000640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fa0b8071ed0 0x7fa0b80c51b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:34846/0 (socket says 172.21.15.6:34846) 2024-04-24T23:27:45.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:45.620+0000 7fa0b6000640 1 -- 172.21.15.6:0/2527823191 learned_addr learned my addr 172.21.15.6:0/2527823191 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:27:45.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:45.621+0000 7fa0b5600640 1 -- 172.21.15.6:0/2527823191 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3068905089 0 0) 0x7fa0b8071280 con 0x7fa0b8071ed0 2024-04-24T23:27:45.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:45.621+0000 7fa0b5600640 1 -- 172.21.15.6:0/2527823191 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa09c003520 con 0x7fa0b8071ed0 2024-04-24T23:27:45.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:45.621+0000 7fa0b5600640 1 -- 172.21.15.6:0/2527823191 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 479418853 0 0) 0x7fa09c003520 con 0x7fa0b8071ed0 2024-04-24T23:27:45.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:45.621+0000 7fa0b5600640 1 -- 172.21.15.6:0/2527823191 >> v2:172.21.15.190:3300/0 conn(0x7fa0b8157940 msgr2=0x7fa0b8071840 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-24T23:27:45.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:45.621+0000 7fa0b5600640 1 --2- 172.21.15.6:0/2527823191 >> v2:172.21.15.190:3300/0 conn(0x7fa0b8157940 0x7fa0b8071840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:45.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:45.621+0000 7fa0b5600640 1 -- 172.21.15.6:0/2527823191 >> v2:172.21.15.6:3300/0 conn(0x7fa0b8151de0 msgr2=0x7fa0b80c36e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:27:45.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:45.621+0000 7fa0b5600640 1 --2- 172.21.15.6:0/2527823191 >> v2:172.21.15.6:3300/0 conn(0x7fa0b8151de0 0x7fa0b80c36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:45.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:45.621+0000 7fa0b5600640 1 -- 172.21.15.6:0/2527823191 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0b815af30 con 0x7fa0b8071ed0 2024-04-24T23:27:45.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:45.621+0000 7fa0b5600640 1 -- 172.21.15.6:0/2527823191 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3463268701 0 0) 0x7fa0b00019f0 con 0x7fa0b8071ed0 2024-04-24T23:27:45.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:45.621+0000 7fa0b5600640 1 -- 172.21.15.6:0/2527823191 >> v1:172.21.15.6:6789/0 conn(0x7fa0b8071ed0 legacy=0x7fa0b80c51b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:45.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:45.621+0000 7fa0b5600640 1 --2- 172.21.15.6:0/2527823191 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa09c003a00 0x7fa09c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:45.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:45.621+0000 7fa0b5600640 1 --2- 172.21.15.6:0/2527823191 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa09c004800 0x7fa09c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:45.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:45.621+0000 7fa0b5600640 1 --2- 172.21.15.6:0/2527823191 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa09c0053c0 0x7fa09c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:45.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:45.621+0000 7fa0b5600640 1 -- 172.21.15.6:0/2527823191 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fa0b8157db0 con 0x7fa09c0053c0 2024-04-24T23:27:45.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:45.621+0000 7fa0b5600640 1 -- 172.21.15.6:0/2527823191 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fa0b8071560 con 0x7fa09c004800 2024-04-24T23:27:45.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:45.621+0000 7fa0b5600640 1 -- 172.21.15.6:0/2527823191 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fa09c00c2b0 con 0x7fa09c003a00 2024-04-24T23:27:45.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:45.622+0000 7fa0b6a00640 1 --2- 172.21.15.6:0/2527823191 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa09c003a00 0x7fa09c003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:45.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:45.622+0000 7fa0b6000640 1 --2- 172.21.15.6:0/2527823191 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa09c004800 0x7fa09c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:45.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:45.624+0000 7fa0b6a00640 1 -- 172.21.15.6:0/2527823191 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa09c004800 msgr2=0x7fa09c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:27:45.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:45.624+0000 7fa0b6a00640 1 --2- 172.21.15.6:0/2527823191 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa09c004800 0x7fa09c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:45.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:45.624+0000 7fa0b6a00640 1 -- 172.21.15.6:0/2527823191 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa09c0053c0 msgr2=0x7fa09c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:27:45.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:45.624+0000 7fa0b6a00640 1 --2- 172.21.15.6:0/2527823191 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa09c0053c0 0x7fa09c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:45.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:45.624+0000 7fa0b6a00640 1 -- 172.21.15.6:0/2527823191 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa09c003520 con 0x7fa09c003a00 2024-04-24T23:27:45.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:45.624+0000 7fa0b7400640 1 -- 172.21.15.6:0/2527823191 reap_dead start 2024-04-24T23:27:45.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:45.625+0000 7fa0b6a00640 1 --2- 172.21.15.6:0/2527823191 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa09c003a00 0x7fa09c003e20 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fa0a4009860 tx=0x7fa09c005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=42043926be7882f1 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:27:45.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:45.625+0000 7fa0b5600640 1 -- 172.21.15.6:0/2527823191 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0a4037070 con 0x7fa09c003a00 2024-04-24T23:27:45.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:45.625+0000 7fa0b5600640 1 -- 172.21.15.6:0/2527823191 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa0a402aa80 con 0x7fa09c003a00 2024-04-24T23:27:45.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:45.625+0000 7fa0b5600640 1 -- 172.21.15.6:0/2527823191 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0a4009460 con 0x7fa09c003a00 2024-04-24T23:27:45.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:45.626+0000 7fa0bd800640 1 -- 172.21.15.6:0/2527823191 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa09c003a00 msgr2=0x7fa09c003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:45.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:45.626+0000 7fa0bd800640 1 --2- 172.21.15.6:0/2527823191 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa09c003a00 0x7fa09c003e20 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fa0a4009860 tx=0x7fa09c005830 comp rx=0 tx=0).stop 2024-04-24T23:27:45.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:45.627+0000 7fa0bd800640 1 -- 172.21.15.6:0/2527823191 shutdown_connections 2024-04-24T23:27:45.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:45.627+0000 7fa0bd800640 1 --2- 172.21.15.6:0/2527823191 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa09c003a00 0x7fa09c003e20 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:45.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:45.627+0000 7fa0bd800640 1 -- 172.21.15.6:0/2527823191 >> 172.21.15.6:0/2527823191 conn(0x7fa0b806b0e0 msgr2=0x7fa0b806b4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:27:45.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:45.628+0000 7fa0bd800640 1 -- 172.21.15.6:0/2527823191 shutdown_connections 2024-04-24T23:27:45.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:45.628+0000 7fa0bd800640 1 -- 172.21.15.6:0/2527823191 wait complete. 2024-04-24T23:27:45.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:45.631+0000 7fa0bd800640 1 Processor -- start 2024-04-24T23:27:45.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:45.632+0000 7fa0bd800640 1 -- start start 2024-04-24T23:27:45.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:45.633+0000 7fa0bd800640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa09c003a00 0x7fa0b80c6fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:45.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:45.634+0000 7fa0bd800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa0b8136210 0x7fa0b8133d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:45.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:45.634+0000 7fa0bd800640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa0b8132fb0 0x7fa0b815a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:45.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:45.634+0000 7fa0bd800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fa0b80c36e0 con 0x7fa0b8136210 2024-04-24T23:27:45.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:45.634+0000 7fa0bd800640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fa0b815b4f0 con 0x7fa0b8132fb0 2024-04-24T23:27:45.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:45.634+0000 7fa0bd800640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fa0b80c3860 con 0x7fa09c003a00 2024-04-24T23:27:45.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:45.634+0000 7fa0b7400640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa0b8132fb0 0x7fa0b815a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:45.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:45.635+0000 7fa0b7400640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa0b8132fb0 0x7fa0b815a280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:53202/0 (socket says 172.21.15.6:53202) 2024-04-24T23:27:45.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:45.635+0000 7fa0b7400640 1 -- 172.21.15.6:0/378862021 learned_addr learned my addr 172.21.15.6:0/378862021 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:27:45.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:45.635+0000 7fa0b6a00640 1 --2- 172.21.15.6:0/378862021 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa09c003a00 0x7fa0b80c6fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:45.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:45.635+0000 7fa0b7400640 1 -- 172.21.15.6:0/378862021 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa09c003a00 msgr2=0x7fa0b80c6fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:45.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:45.635+0000 7fa0b7400640 1 --2- 172.21.15.6:0/378862021 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa09c003a00 0x7fa0b80c6fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:45.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:45.635+0000 7fa0b7400640 1 -- 172.21.15.6:0/378862021 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa0b8136210 msgr2=0x7fa0b8133d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:45.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:45.635+0000 7fa0b7400640 1 --2- 172.21.15.6:0/378862021 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa0b8136210 0x7fa0b8133d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:45.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:45.635+0000 7fa0b7400640 1 -- 172.21.15.6:0/378862021 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0b813a8c0 con 0x7fa0b8132fb0 2024-04-24T23:27:45.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:45.635+0000 7fa0b6a00640 1 --2- 172.21.15.6:0/378862021 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa09c003a00 0x7fa0b80c6fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:27:45.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:45.636+0000 7fa0b7400640 1 --2- 172.21.15.6:0/378862021 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa0b8132fb0 0x7fa0b815a280 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fa0b8157d20 tx=0x7fa0a800a7a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:27:45.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:45.637+0000 7fa0aea00640 1 -- 172.21.15.6:0/378862021 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0a8018070 con 0x7fa0b8132fb0 2024-04-24T23:27:45.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:45.637+0000 7fa0bd800640 1 -- 172.21.15.6:0/378862021 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0b81574d0 con 0x7fa0b8132fb0 2024-04-24T23:27:45.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:45.637+0000 7fa0bd800640 1 -- 172.21.15.6:0/378862021 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0b8159780 con 0x7fa0b8132fb0 2024-04-24T23:27:45.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:45.638+0000 7fa0aea00640 1 -- 172.21.15.6:0/378862021 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa0a8005d20 con 0x7fa0b8132fb0 2024-04-24T23:27:45.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:45.638+0000 7fa0aea00640 1 -- 172.21.15.6:0/378862021 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0a8007270 con 0x7fa0b8132fb0 2024-04-24T23:27:45.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:45.643+0000 7fa0aea00640 1 -- 172.21.15.6:0/378862021 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fa0a8007410 con 0x7fa0b8132fb0 2024-04-24T23:27:45.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:45.643+0000 7fa0bd800640 1 -- 172.21.15.6:0/378862021 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fa090001030 con 0x7fa0b8132fb0 2024-04-24T23:27:45.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:45.645+0000 7fa0aea00640 1 --2- 172.21.15.6:0/378862021 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fa09806be60 0x7fa09806e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:45.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:45.645+0000 7fa0aea00640 1 -- 172.21.15.6:0/378862021 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6934+0+0 (secure 0 0 0) 0x7fa0a808a090 con 0x7fa0b8132fb0 2024-04-24T23:27:45.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:45.645+0000 7fa0aea00640 1 --2- 172.21.15.6:0/378862021 >> [v2:172.21.15.6:6808/3022569437,v1:172.21.15.6:6809/3022569437] conn(0x7fa098075980 0x7fa098077e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:45.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:45.646+0000 7fa0aea00640 1 -- 172.21.15.6:0/378862021 --> [v2:172.21.15.6:6808/3022569437,v1:172.21.15.6:6809/3022569437] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa0980784d0 con 0x7fa098075980 2024-04-24T23:27:45.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:45.647+0000 7fa0aea00640 1 -- 172.21.15.6:0/378862021 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fa0a8091050 con 0x7fa0b8132fb0 2024-04-24T23:27:45.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:45.646+0000 7fa0b6a00640 1 --2- 172.21.15.6:0/378862021 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fa09806be60 0x7fa09806e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:45.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:45.653+0000 7fa0b6a00640 1 --2- 172.21.15.6:0/378862021 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fa09806be60 0x7fa09806e320 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fa0a4034000 tx=0x7fa0a402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:27:45.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:45.655+0000 7fa0b6000640 1 --2- 172.21.15.6:0/378862021 >> [v2:172.21.15.6:6808/3022569437,v1:172.21.15.6:6809/3022569437] conn(0x7fa098075980 0x7fa098077e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:45.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:45.658+0000 7fa0b6000640 1 --2- 172.21.15.6:0/378862021 >> [v2:172.21.15.6:6808/3022569437,v1:172.21.15.6:6809/3022569437] conn(0x7fa098075980 0x7fa098077e00 crc :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:27:45.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:45.699+0000 7fa0aea00640 1 -- 172.21.15.6:0/378862021 <== osd.0 v2:172.21.15.6:6808/3022569437 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa0980784d0 con 0x7fa098075980 2024-04-24T23:27:45.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:45.735+0000 7fa0bd800640 1 -- 172.21.15.6:0/378862021 --> [v2:172.21.15.6:6808/3022569437,v1:172.21.15.6:6809/3022569437] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fa090002e30 con 0x7fa098075980 2024-04-24T23:27:45.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:45.756+0000 7fa0aea00640 1 -- 172.21.15.6:0/378862021 <== osd.0 v2:172.21.15.6:6808/3022569437 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fa090002e30 con 0x7fa098075980 2024-04-24T23:27:45.757 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-04-24T23:27:45.757 INFO:teuthology.orchestra.run.smithi006.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-24T23:27:45.757 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-04-24T23:27:45.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:45.756+0000 7fa0bd800640 1 -- 172.21.15.6:0/378862021 >> [v2:172.21.15.6:6808/3022569437,v1:172.21.15.6:6809/3022569437] conn(0x7fa098075980 msgr2=0x7fa098077e00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:45.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:45.756+0000 7fa0bd800640 1 --2- 172.21.15.6:0/378862021 >> [v2:172.21.15.6:6808/3022569437,v1:172.21.15.6:6809/3022569437] conn(0x7fa098075980 0x7fa098077e00 crc :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:45.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:45.757+0000 7fa0bd800640 1 -- 172.21.15.6:0/378862021 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fa09806be60 msgr2=0x7fa09806e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:45.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:45.757+0000 7fa0bd800640 1 --2- 172.21.15.6:0/378862021 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fa09806be60 0x7fa09806e320 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fa0a4034000 tx=0x7fa0a402e000 comp rx=0 tx=0).stop 2024-04-24T23:27:45.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:45.757+0000 7fa0bd800640 1 -- 172.21.15.6:0/378862021 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa0b8132fb0 msgr2=0x7fa0b815a280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:45.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:45.757+0000 7fa0bd800640 1 --2- 172.21.15.6:0/378862021 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa0b8132fb0 0x7fa0b815a280 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fa0b8157d20 tx=0x7fa0a800a7a0 comp rx=0 tx=0).stop 2024-04-24T23:27:45.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:45.758+0000 7fa0b7400640 1 -- 172.21.15.6:0/378862021 reap_dead start 2024-04-24T23:27:45.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:45.758+0000 7fa0bd800640 1 -- 172.21.15.6:0/378862021 shutdown_connections 2024-04-24T23:27:45.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:45.758+0000 7fa0bd800640 1 -- 172.21.15.6:0/378862021 >> 172.21.15.6:0/378862021 conn(0x7fa0b806b0e0 msgr2=0x7fa0b806b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:27:45.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:45.761+0000 7fa0bd800640 1 -- 172.21.15.6:0/378862021 shutdown_connections 2024-04-24T23:27:45.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:45.761+0000 7fa0bd800640 1 -- 172.21.15.6:0/378862021 wait complete. 2024-04-24T23:27:45.806 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-04-24T23:27:46.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.090+0000 7fc604a00640 1 Processor -- start 2024-04-24T23:27:46.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.090+0000 7fc604a00640 1 -- start start 2024-04-24T23:27:46.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.092+0000 7fc604a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc6000c8f30 0x7fc60015f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:46.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.092+0000 7fc604a00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc600163720 con 0x7fc600071390 2024-04-24T23:27:46.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.092+0000 7fc604a00640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc60006c320 con 0x7fc60015cd50 2024-04-24T23:27:46.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.092+0000 7fc604a00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7fc6001631e0 con 0x7fc6000c8f30 2024-04-24T23:27:46.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.093+0000 7fc5fe800640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7fc60015cd50 0x7fc600169b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.6:38396/0 (socket says 172.21.15.6:38396) 2024-04-24T23:27:46.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.093+0000 7fc5fe800640 1 -- 172.21.15.6:0/2296721373 learned_addr learned my addr 172.21.15.6:0/2296721373 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:27:46.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.093+0000 7fc5fd400640 1 --2- 172.21.15.6:0/2296721373 >> v2:172.21.15.6:3300/0 conn(0x7fc6000c8f30 0x7fc60015f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:46.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.095+0000 7fc5fca00640 1 -- 172.21.15.6:0/2296721373 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1475366883 0 0) 0x7fc60006c320 con 0x7fc60015cd50 2024-04-24T23:27:46.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.095+0000 7fc5fca00640 1 -- 172.21.15.6:0/2296721373 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc5e4003520 con 0x7fc60015cd50 2024-04-24T23:27:46.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.095+0000 7fc5fca00640 1 -- 172.21.15.6:0/2296721373 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1136865714 0 0) 0x7fc600163720 con 0x7fc600071390 2024-04-24T23:27:46.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.095+0000 7fc5fca00640 1 -- 172.21.15.6:0/2296721373 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc60006c320 con 0x7fc600071390 2024-04-24T23:27:46.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.096+0000 7fc5fca00640 1 -- 172.21.15.6:0/2296721373 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2915441852 0 0) 0x7fc60006c320 con 0x7fc600071390 2024-04-24T23:27:46.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.096+0000 7fc5fca00640 1 -- 172.21.15.6:0/2296721373 >> v2:172.21.15.6:3300/0 conn(0x7fc6000c8f30 msgr2=0x7fc60015f2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:27:46.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.096+0000 7fc5fca00640 1 --2- 172.21.15.6:0/2296721373 >> v2:172.21.15.6:3300/0 conn(0x7fc6000c8f30 0x7fc60015f2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:46.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.096+0000 7fc5fca00640 1 -- 172.21.15.6:0/2296721373 >> v1:172.21.15.190:6789/0 conn(0x7fc60015cd50 legacy=0x7fc600169b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:46.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.097+0000 7fc5fca00640 1 -- 172.21.15.6:0/2296721373 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6000c5450 con 0x7fc600071390 2024-04-24T23:27:46.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.097+0000 7fc5fca00640 1 -- 172.21.15.6:0/2296721373 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3463268701 0 0) 0x7fc5f0001520 con 0x7fc600071390 2024-04-24T23:27:46.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.097+0000 7fc5fca00640 1 -- 172.21.15.6:0/2296721373 >> v1:172.21.15.6:6789/0 conn(0x7fc600071390 legacy=0x7fc60015c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:46.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.097+0000 7fc5fca00640 1 --2- 172.21.15.6:0/2296721373 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc5e4004c40 0x7fc5e4005040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:46.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.097+0000 7fc5fca00640 1 --2- 172.21.15.6:0/2296721373 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc5e40037a0 0x7fc5e4003c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:46.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.097+0000 7fc5fca00640 1 --2- 172.21.15.6:0/2296721373 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc5e4004160 0x7fc5e400ba40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:46.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.097+0000 7fc5fca00640 1 -- 172.21.15.6:0/2296721373 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fc6001631e0 con 0x7fc5e4004c40 2024-04-24T23:27:46.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.097+0000 7fc5fca00640 1 -- 172.21.15.6:0/2296721373 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fc5e400c170 con 0x7fc5e40037a0 2024-04-24T23:27:46.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.097+0000 7fc5fca00640 1 -- 172.21.15.6:0/2296721373 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fc5e400c370 con 0x7fc5e4004160 2024-04-24T23:27:46.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.098+0000 7fc5fd400640 1 --2- 172.21.15.6:0/2296721373 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc5e40037a0 0x7fc5e4003c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:46.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.098+0000 7fc5fde00640 1 --2- 172.21.15.6:0/2296721373 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc5e4004c40 0x7fc5e4005040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:46.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.099+0000 7fc5fe800640 1 --2- 172.21.15.6:0/2296721373 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc5e4004160 0x7fc5e400ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:46.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.099+0000 7fc5fd400640 1 -- 172.21.15.6:0/2296721373 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc5e4004160 msgr2=0x7fc5e400ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:27:46.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.099+0000 7fc5fd400640 1 --2- 172.21.15.6:0/2296721373 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc5e4004160 0x7fc5e400ba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:46.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.100+0000 7fc5fd400640 1 -- 172.21.15.6:0/2296721373 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc5e4004c40 msgr2=0x7fc5e4005040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:27:46.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.100+0000 7fc5fd400640 1 --2- 172.21.15.6:0/2296721373 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc5e4004c40 0x7fc5e4005040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:46.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.100+0000 7fc5fd400640 1 -- 172.21.15.6:0/2296721373 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc60006c320 con 0x7fc5e40037a0 2024-04-24T23:27:46.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.100+0000 7fc5fe800640 1 -- 172.21.15.6:0/2296721373 reap_dead start 2024-04-24T23:27:46.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.101+0000 7fc5fd400640 1 --2- 172.21.15.6:0/2296721373 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc5e40037a0 0x7fc5e4003c20 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fc5f8009f90 tx=0x7fc5e4005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cdbfe78a2cf7ef6d server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:27:46.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.101+0000 7fc5fca00640 1 -- 172.21.15.6:0/2296721373 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5f8009c10 con 0x7fc5e40037a0 2024-04-24T23:27:46.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.102+0000 7fc5fca00640 1 -- 172.21.15.6:0/2296721373 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5f8036070 con 0x7fc5e40037a0 2024-04-24T23:27:46.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.102+0000 7fc5fca00640 1 -- 172.21.15.6:0/2296721373 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5f8031630 con 0x7fc5e40037a0 2024-04-24T23:27:46.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.102+0000 7fc604a00640 1 -- 172.21.15.6:0/2296721373 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc5e40037a0 msgr2=0x7fc5e4003c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:46.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.102+0000 7fc604a00640 1 --2- 172.21.15.6:0/2296721373 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc5e40037a0 0x7fc5e4003c20 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fc5f8009f90 tx=0x7fc5e4005830 comp rx=0 tx=0).stop 2024-04-24T23:27:46.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.103+0000 7fc604a00640 1 -- 172.21.15.6:0/2296721373 shutdown_connections 2024-04-24T23:27:46.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.103+0000 7fc604a00640 1 --2- 172.21.15.6:0/2296721373 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc5e40037a0 0x7fc5e4003c20 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:46.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.103+0000 7fc604a00640 1 -- 172.21.15.6:0/2296721373 >> 172.21.15.6:0/2296721373 conn(0x7fc60006fdc0 msgr2=0x7fc6000701d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:27:46.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.103+0000 7fc604a00640 1 -- 172.21.15.6:0/2296721373 shutdown_connections 2024-04-24T23:27:46.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.103+0000 7fc604a00640 1 -- 172.21.15.6:0/2296721373 wait complete. 2024-04-24T23:27:46.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.106+0000 7fc604a00640 1 Processor -- start 2024-04-24T23:27:46.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.111+0000 7fc604a00640 1 -- start start 2024-04-24T23:27:46.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.112+0000 7fc604a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc5e40037a0 0x7fc6001ebda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:46.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.112+0000 7fc604a00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc6001f5030 0x7fc600157d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:46.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.112+0000 7fc604a00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc6000cb2b0 0x7fc600155700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:46.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.112+0000 7fc604a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fc6000cb860 con 0x7fc5e40037a0 2024-04-24T23:27:46.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.112+0000 7fc604a00640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fc60006c5c0 con 0x7fc6001f5030 2024-04-24T23:27:46.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.112+0000 7fc604a00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fc6000cbd90 con 0x7fc6000cb2b0 2024-04-24T23:27:46.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.113+0000 7fc5fde00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc5e40037a0 0x7fc6001ebda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:46.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.113+0000 7fc5fde00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc5e40037a0 0x7fc6001ebda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57936/0 (socket says 172.21.15.6:57936) 2024-04-24T23:27:46.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.114+0000 7fc5fde00640 1 -- 172.21.15.6:0/1655856190 learned_addr learned my addr 172.21.15.6:0/1655856190 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:27:46.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.115+0000 7fc5fe800640 1 --2- 172.21.15.6:0/1655856190 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc6000cb2b0 0x7fc600155700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:46.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.115+0000 7fc5fd400640 1 --2- 172.21.15.6:0/1655856190 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc6001f5030 0x7fc600157d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:46.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.116+0000 7fc5fe800640 1 -- 172.21.15.6:0/1655856190 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc6001f5030 msgr2=0x7fc600157d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:46.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.116+0000 7fc5fe800640 1 --2- 172.21.15.6:0/1655856190 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc6001f5030 0x7fc600157d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:46.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.116+0000 7fc5fe800640 1 -- 172.21.15.6:0/1655856190 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc5e40037a0 msgr2=0x7fc6001ebda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:46.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.116+0000 7fc5fe800640 1 --2- 172.21.15.6:0/1655856190 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc5e40037a0 0x7fc6001ebda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:46.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.116+0000 7fc5fe800640 1 -- 172.21.15.6:0/1655856190 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6001eb8f0 con 0x7fc6000cb2b0 2024-04-24T23:27:46.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.116+0000 7fc5fe800640 1 --2- 172.21.15.6:0/1655856190 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc6000cb2b0 0x7fc600155700 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fc60006dcb0 tx=0x7fc5ec0017e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:27:46.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.116+0000 7fc5f6000640 1 -- 172.21.15.6:0/1655856190 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5ec019070 con 0x7fc6000cb2b0 2024-04-24T23:27:46.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.116+0000 7fc5f6000640 1 -- 172.21.15.6:0/1655856190 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5ec0114f0 con 0x7fc6000cb2b0 2024-04-24T23:27:46.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.116+0000 7fc5f6000640 1 -- 172.21.15.6:0/1655856190 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5ec0118e0 con 0x7fc6000cb2b0 2024-04-24T23:27:46.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.117+0000 7fc604a00640 1 -- 172.21.15.6:0/1655856190 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc600160cb0 con 0x7fc6000cb2b0 2024-04-24T23:27:46.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.117+0000 7fc604a00640 1 -- 172.21.15.6:0/1655856190 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc600160f50 con 0x7fc6000cb2b0 2024-04-24T23:27:46.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.119+0000 7fc604a00640 1 -- 172.21.15.6:0/1655856190 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc60004f620 con 0x7fc6000cb2b0 2024-04-24T23:27:46.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.120+0000 7fc5f6000640 1 -- 172.21.15.6:0/1655856190 <== mon.2 v2:172.21.15.190:3301/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fc5ec011a80 con 0x7fc6000cb2b0 2024-04-24T23:27:46.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.121+0000 7fc5f6000640 1 --2- 172.21.15.6:0/1655856190 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fc5e006be60 0x7fc5e006e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:46.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.121+0000 7fc5f6000640 1 -- 172.21.15.6:0/1655856190 <== mon.2 v2:172.21.15.190:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6934+0+0 (secure 0 0 0) 0x7fc5ec08a5d0 con 0x7fc6000cb2b0 2024-04-24T23:27:46.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.122+0000 7fc5fde00640 1 --2- 172.21.15.6:0/1655856190 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fc5e006be60 0x7fc5e006e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:46.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.122+0000 7fc5fde00640 1 --2- 172.21.15.6:0/1655856190 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fc5e006be60 0x7fc5e006e320 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fc5f0000c00 tx=0x7fc5f0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:27:46.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.125+0000 7fc5f6000640 1 -- 172.21.15.6:0/1655856190 <== mon.2 v2:172.21.15.190:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fc60004f620 con 0x7fc6000cb2b0 2024-04-24T23:27:46.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.320+0000 7fc604a00640 1 -- 172.21.15.6:0/1655856190 --> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) -- 0x7fc6000c4920 con 0x7fc5e006be60 2024-04-24T23:27:46.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.320+0000 7fc5f6000640 1 -- 172.21.15.6:0/1655856190 <== mgr.4105 v2:172.21.15.190:6832/806769663 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fc6000c4920 con 0x7fc5e006be60 2024-04-24T23:27:46.322 INFO:teuthology.orchestra.run.smithi006.stderr:instructed osd(s) 0 to deep-scrub 2024-04-24T23:27:46.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.323+0000 7fc604a00640 1 -- 172.21.15.6:0/1655856190 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fc5e006be60 msgr2=0x7fc5e006e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:46.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.323+0000 7fc604a00640 1 --2- 172.21.15.6:0/1655856190 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fc5e006be60 0x7fc5e006e320 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fc5f0000c00 tx=0x7fc5f0009040 comp rx=0 tx=0).stop 2024-04-24T23:27:46.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.323+0000 7fc604a00640 1 -- 172.21.15.6:0/1655856190 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc6000cb2b0 msgr2=0x7fc600155700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:46.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.323+0000 7fc604a00640 1 --2- 172.21.15.6:0/1655856190 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc6000cb2b0 0x7fc600155700 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fc60006dcb0 tx=0x7fc5ec0017e0 comp rx=0 tx=0).stop 2024-04-24T23:27:46.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.323+0000 7fc604a00640 1 -- 172.21.15.6:0/1655856190 shutdown_connections 2024-04-24T23:27:46.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.323+0000 7fc604a00640 1 --2- 172.21.15.6:0/1655856190 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fc5e006be60 0x7fc5e006e320 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:46.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.323+0000 7fc604a00640 1 --2- 172.21.15.6:0/1655856190 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc6000cb2b0 0x7fc600155700 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:46.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.323+0000 7fc604a00640 1 --2- 172.21.15.6:0/1655856190 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc6001f5030 0x7fc600157d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:46.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.323+0000 7fc604a00640 1 --2- 172.21.15.6:0/1655856190 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc5e40037a0 0x7fc6001ebda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:46.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.323+0000 7fc604a00640 1 -- 172.21.15.6:0/1655856190 >> 172.21.15.6:0/1655856190 conn(0x7fc60006fdc0 msgr2=0x7fc60015c4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:27:46.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.323+0000 7fc604a00640 1 -- 172.21.15.6:0/1655856190 shutdown_connections 2024-04-24T23:27:46.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.323+0000 7fc604a00640 1 -- 172.21.15.6:0/1655856190 wait complete. 2024-04-24T23:27:46.342 INFO:tasks.ceph:Scrubbing osd.1 2024-04-24T23:27:46.342 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-04-24T23:27:46.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.516+0000 7f8b7be00640 1 Processor -- start 2024-04-24T23:27:46.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.516+0000 7f8b7be00640 1 -- start start 2024-04-24T23:27:46.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.518+0000 7f8b7be00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8b7c0cabe0 0x7f8b7c15b700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:46.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.518+0000 7f8b7be00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f8b7c0c5950 0x7f8b7c0c5dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:46.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.518+0000 7f8b7be00640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8b7c0c6300 con 0x7f8b7c151930 2024-04-24T23:27:46.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.518+0000 7f8b7be00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f8b7c0b5710 con 0x7f8b7c0c5950 2024-04-24T23:27:46.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.518+0000 7f8b7be00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f8b7c06b960 con 0x7f8b7c0cabe0 2024-04-24T23:27:46.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.518+0000 7f8b7aa00640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f8b7c151930 0x7f8b7c0cb6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.6:38400/0 (socket says 172.21.15.6:38400) 2024-04-24T23:27:46.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.518+0000 7f8b7aa00640 1 -- 172.21.15.6:0/2549358620 learned_addr learned my addr 172.21.15.6:0/2549358620 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:27:46.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.518+0000 7f8b7b400640 1 --2- 172.21.15.6:0/2549358620 >> v2:172.21.15.6:3300/0 conn(0x7f8b7c0c5950 0x7f8b7c0c5dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:46.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.518+0000 7f8b73e00640 1 --2- 172.21.15.6:0/2549358620 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8b7c0cabe0 0x7f8b7c15b700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:46.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.519+0000 7f8b7a000640 1 -- 172.21.15.6:0/2549358620 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2235082858 0 0) 0x7f8b7c0c6300 con 0x7f8b7c151930 2024-04-24T23:27:46.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.519+0000 7f8b7a000640 1 -- 172.21.15.6:0/2549358620 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8b60003520 con 0x7f8b7c151930 2024-04-24T23:27:46.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.519+0000 7f8b73e00640 1 -- 172.21.15.6:0/2549358620 >> v2:172.21.15.6:3300/0 conn(0x7f8b7c0c5950 msgr2=0x7f8b7c0c5dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:27:46.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.519+0000 7f8b73e00640 1 --2- 172.21.15.6:0/2549358620 >> v2:172.21.15.6:3300/0 conn(0x7f8b7c0c5950 0x7f8b7c0c5dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:46.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.519+0000 7f8b73e00640 1 -- 172.21.15.6:0/2549358620 >> v1:172.21.15.190:6789/0 conn(0x7f8b7c151930 legacy=0x7f8b7c0cb6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:46.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.519+0000 7f8b73e00640 1 -- 172.21.15.6:0/2549358620 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b7c06bc40 con 0x7f8b7c0cabe0 2024-04-24T23:27:46.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.520+0000 7f8b73e00640 1 --2- 172.21.15.6:0/2549358620 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8b7c0cabe0 0x7f8b7c15b700 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f8b640097a0 tx=0x7f8b6402a210 comp rx=0 tx=0).ready entity=mon.2 client_cookie=222931dbf46c6249 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:27:46.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.520+0000 7f8b7a000640 1 -- 172.21.15.6:0/2549358620 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b64037070 con 0x7f8b7c0cabe0 2024-04-24T23:27:46.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.520+0000 7f8b7a000640 1 -- 172.21.15.6:0/2549358620 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8b64032440 con 0x7f8b7c0cabe0 2024-04-24T23:27:46.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.520+0000 7f8b7a000640 1 -- 172.21.15.6:0/2549358620 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b64032810 con 0x7f8b7c0cabe0 2024-04-24T23:27:46.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.521+0000 7f8b7be00640 1 -- 172.21.15.6:0/2549358620 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8b7c0cabe0 msgr2=0x7f8b7c15b700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:46.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.521+0000 7f8b7be00640 1 --2- 172.21.15.6:0/2549358620 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8b7c0cabe0 0x7f8b7c15b700 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f8b640097a0 tx=0x7f8b6402a210 comp rx=0 tx=0).stop 2024-04-24T23:27:46.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.521+0000 7f8b7be00640 1 -- 172.21.15.6:0/2549358620 shutdown_connections 2024-04-24T23:27:46.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.521+0000 7f8b7be00640 1 --2- 172.21.15.6:0/2549358620 >> v2:172.21.15.6:3300/0 conn(0x7f8b7c0c5950 0x7f8b7c0c5dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:46.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.521+0000 7f8b7be00640 1 --2- 172.21.15.6:0/2549358620 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8b7c0cabe0 0x7f8b7c15b700 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:46.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.521+0000 7f8b7be00640 1 -- 172.21.15.6:0/2549358620 >> 172.21.15.6:0/2549358620 conn(0x7f8b7c06ded0 msgr2=0x7f8b7c066ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:27:46.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.521+0000 7f8b7be00640 1 -- 172.21.15.6:0/2549358620 shutdown_connections 2024-04-24T23:27:46.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.521+0000 7f8b7be00640 1 -- 172.21.15.6:0/2549358620 wait complete. 2024-04-24T23:27:46.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.523+0000 7f8b7be00640 1 Processor -- start 2024-04-24T23:27:46.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.523+0000 7f8b7be00640 1 -- start start 2024-04-24T23:27:46.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.525+0000 7f8b7be00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8b7c0c5950 0x7f8b7c0cb960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:46.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.525+0000 7f8b7be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8b7c0cabe0 0x7f8b7c0d01c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:46.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.525+0000 7f8b7be00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8b7c151930 0x7f8b7c1f1370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:46.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.525+0000 7f8b7be00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f8b7c06c4e0 con 0x7f8b7c0cabe0 2024-04-24T23:27:46.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.525+0000 7f8b7be00640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f8b7c06c200 con 0x7f8b7c0c5950 2024-04-24T23:27:46.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.525+0000 7f8b7be00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f8b7c06c7c0 con 0x7f8b7c151930 2024-04-24T23:27:46.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.525+0000 7f8b73e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8b7c0cabe0 0x7f8b7c0d01c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:46.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.525+0000 7f8b7b400640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8b7c151930 0x7f8b7c1f1370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:46.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.525+0000 7f8b7aa00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8b7c0c5950 0x7f8b7c0cb960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:46.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.525+0000 7f8b73e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8b7c0cabe0 0x7f8b7c0d01c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57956/0 (socket says 172.21.15.6:57956) 2024-04-24T23:27:46.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.525+0000 7f8b73e00640 1 -- 172.21.15.6:0/1884010543 learned_addr learned my addr 172.21.15.6:0/1884010543 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:27:46.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.525+0000 7f8b73e00640 1 -- 172.21.15.6:0/1884010543 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8b7c151930 msgr2=0x7f8b7c1f1370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:46.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.525+0000 7f8b73e00640 1 --2- 172.21.15.6:0/1884010543 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8b7c151930 0x7f8b7c1f1370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:46.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.525+0000 7f8b73e00640 1 -- 172.21.15.6:0/1884010543 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8b7c0c5950 msgr2=0x7f8b7c0cb960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:46.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.525+0000 7f8b73e00640 1 --2- 172.21.15.6:0/1884010543 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8b7c0c5950 0x7f8b7c0cb960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:46.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.525+0000 7f8b73e00640 1 -- 172.21.15.6:0/1884010543 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b7c1f3510 con 0x7f8b7c0cabe0 2024-04-24T23:27:46.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.525+0000 7f8b7aa00640 1 --2- 172.21.15.6:0/1884010543 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8b7c0c5950 0x7f8b7c0cb960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:27:46.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.525+0000 7f8b7b400640 1 --2- 172.21.15.6:0/1884010543 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8b7c151930 0x7f8b7c1f1370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:27:46.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.526+0000 7f8b73e00640 1 --2- 172.21.15.6:0/1884010543 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8b7c0cabe0 0x7f8b7c0d01c0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f8b64035040 tx=0x7f8b640046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:27:46.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.526+0000 7f8b72a00640 1 -- 172.21.15.6:0/1884010543 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b64041030 con 0x7f8b7c0cabe0 2024-04-24T23:27:46.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.526+0000 7f8b72a00640 1 -- 172.21.15.6:0/1884010543 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8b6403d4e0 con 0x7f8b7c0cabe0 2024-04-24T23:27:46.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.526+0000 7f8b72a00640 1 -- 172.21.15.6:0/1884010543 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b64037040 con 0x7f8b7c0cabe0 2024-04-24T23:27:46.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.526+0000 7f8b7be00640 1 -- 172.21.15.6:0/1884010543 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b7c1f37b0 con 0x7f8b7c0cabe0 2024-04-24T23:27:46.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.526+0000 7f8b7be00640 1 -- 172.21.15.6:0/1884010543 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b7c1f3d70 con 0x7f8b7c0cabe0 2024-04-24T23:27:46.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.527+0000 7f8b7be00640 1 -- 172.21.15.6:0/1884010543 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f8b7c14feb0 con 0x7f8b7c0cabe0 2024-04-24T23:27:46.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.528+0000 7f8b72a00640 1 -- 172.21.15.6:0/1884010543 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f8b64044070 con 0x7f8b7c0cabe0 2024-04-24T23:27:46.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.529+0000 7f8b72a00640 1 --2- 172.21.15.6:0/1884010543 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f8b5806be30 0x7f8b5806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:46.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.529+0000 7f8b72a00640 1 -- 172.21.15.6:0/1884010543 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6934+0+0 (secure 0 0 0) 0x7f8b640ab8a0 con 0x7f8b7c0cabe0 2024-04-24T23:27:46.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.529+0000 7f8b72a00640 1 --2- 172.21.15.6:0/1884010543 >> [v2:172.21.15.6:6816/3687836888,v1:172.21.15.6:6817/3687836888] conn(0x7f8b580758e0 0x7f8b58077d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:46.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.529+0000 7f8b72a00640 1 -- 172.21.15.6:0/1884010543 --> [v2:172.21.15.6:6816/3687836888,v1:172.21.15.6:6817/3687836888] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8b64032440 con 0x7f8b580758e0 2024-04-24T23:27:46.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.529+0000 7f8b72a00640 1 -- 172.21.15.6:0/1884010543 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f8b640b02a0 con 0x7f8b7c0cabe0 2024-04-24T23:27:46.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.530+0000 7f8b7aa00640 1 --2- 172.21.15.6:0/1884010543 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f8b5806be30 0x7f8b5806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:46.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.530+0000 7f8b7b400640 1 --2- 172.21.15.6:0/1884010543 >> [v2:172.21.15.6:6816/3687836888,v1:172.21.15.6:6817/3687836888] conn(0x7f8b580758e0 0x7f8b58077d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:46.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.530+0000 7f8b7aa00640 1 --2- 172.21.15.6:0/1884010543 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f8b5806be30 0x7f8b5806e2f0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f8b6c0011e0 tx=0x7f8b6c007d30 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:27:46.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.530+0000 7f8b7b400640 1 --2- 172.21.15.6:0/1884010543 >> [v2:172.21.15.6:6816/3687836888,v1:172.21.15.6:6817/3687836888] conn(0x7f8b580758e0 0x7f8b58077d60 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:27:46.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.531+0000 7f8b72a00640 1 -- 172.21.15.6:0/1884010543 <== osd.1 v2:172.21.15.6:6816/3687836888 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8b64032440 con 0x7f8b580758e0 2024-04-24T23:27:46.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.559+0000 7f8b7be00640 1 -- 172.21.15.6:0/1884010543 --> [v2:172.21.15.6:6816/3687836888,v1:172.21.15.6:6817/3687836888] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f8b7c075b30 con 0x7f8b580758e0 2024-04-24T23:27:46.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.559+0000 7f8b72a00640 1 -- 172.21.15.6:0/1884010543 <== osd.1 v2:172.21.15.6:6816/3687836888 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f8b7c075b30 con 0x7f8b580758e0 2024-04-24T23:27:46.561 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-04-24T23:27:46.561 INFO:teuthology.orchestra.run.smithi006.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-24T23:27:46.561 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-04-24T23:27:46.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.560+0000 7f8b7be00640 1 -- 172.21.15.6:0/1884010543 >> [v2:172.21.15.6:6816/3687836888,v1:172.21.15.6:6817/3687836888] conn(0x7f8b580758e0 msgr2=0x7f8b58077d60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:46.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.560+0000 7f8b7be00640 1 --2- 172.21.15.6:0/1884010543 >> [v2:172.21.15.6:6816/3687836888,v1:172.21.15.6:6817/3687836888] conn(0x7f8b580758e0 0x7f8b58077d60 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:46.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.560+0000 7f8b7be00640 1 -- 172.21.15.6:0/1884010543 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f8b5806be30 msgr2=0x7f8b5806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:46.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.560+0000 7f8b7be00640 1 --2- 172.21.15.6:0/1884010543 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f8b5806be30 0x7f8b5806e2f0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f8b6c0011e0 tx=0x7f8b6c007d30 comp rx=0 tx=0).stop 2024-04-24T23:27:46.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.560+0000 7f8b7be00640 1 -- 172.21.15.6:0/1884010543 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8b7c0cabe0 msgr2=0x7f8b7c0d01c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:46.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.560+0000 7f8b7be00640 1 --2- 172.21.15.6:0/1884010543 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8b7c0cabe0 0x7f8b7c0d01c0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f8b64035040 tx=0x7f8b640046c0 comp rx=0 tx=0).stop 2024-04-24T23:27:46.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.560+0000 7f8b7b400640 1 -- 172.21.15.6:0/1884010543 reap_dead start 2024-04-24T23:27:46.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.560+0000 7f8b7be00640 1 -- 172.21.15.6:0/1884010543 shutdown_connections 2024-04-24T23:27:46.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.560+0000 7f8b7be00640 1 -- 172.21.15.6:0/1884010543 >> 172.21.15.6:0/1884010543 conn(0x7f8b7c06ded0 msgr2=0x7f8b7c066ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:27:46.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.561+0000 7f8b7be00640 1 -- 172.21.15.6:0/1884010543 shutdown_connections 2024-04-24T23:27:46.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.561+0000 7f8b7be00640 1 -- 172.21.15.6:0/1884010543 wait complete. 2024-04-24T23:27:46.582 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-04-24T23:27:46.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.772+0000 7f568d800640 1 Processor -- start 2024-04-24T23:27:46.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.772+0000 7f568d800640 1 -- start start 2024-04-24T23:27:46.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.774+0000 7f568d800640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f5688154d80 0x7f56880c5310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:46.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.774+0000 7f568d800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f56880d03c0 0x7f56880d07f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:46.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.774+0000 7f568d800640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f56880c5850 con 0x7f56880cdaa0 2024-04-24T23:27:46.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.774+0000 7f568d800640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f56880cd610 con 0x7f56880d03c0 2024-04-24T23:27:46.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.774+0000 7f568d800640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f56880cc720 con 0x7f5688154d80 2024-04-24T23:27:46.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.775+0000 7f5687400640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f56880cdaa0 0x7f56880cb260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.6:38402/0 (socket says 172.21.15.6:38402) 2024-04-24T23:27:46.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.775+0000 7f5687400640 1 -- 172.21.15.6:0/1309623429 learned_addr learned my addr 172.21.15.6:0/1309623429 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:27:46.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.775+0000 7f5687e00640 1 --2- 172.21.15.6:0/1309623429 >> v2:172.21.15.190:3300/0 conn(0x7f5688154d80 0x7f56880c5310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:46.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.775+0000 7f568ce00640 1 --2- 172.21.15.6:0/1309623429 >> v2:172.21.15.6:3300/0 conn(0x7f56880d03c0 0x7f56880d07f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:46.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.775+0000 7f5686a00640 1 -- 172.21.15.6:0/1309623429 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2332727480 0 0) 0x7f56880c5850 con 0x7f56880cdaa0 2024-04-24T23:27:46.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.775+0000 7f5686a00640 1 -- 172.21.15.6:0/1309623429 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5668003520 con 0x7f56880cdaa0 2024-04-24T23:27:46.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.775+0000 7f5687e00640 1 -- 172.21.15.6:0/1309623429 >> v2:172.21.15.6:3300/0 conn(0x7f56880d03c0 msgr2=0x7f56880d07f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:27:46.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.775+0000 7f5687e00640 1 --2- 172.21.15.6:0/1309623429 >> v2:172.21.15.6:3300/0 conn(0x7f56880d03c0 0x7f56880d07f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:46.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.775+0000 7f5687e00640 1 -- 172.21.15.6:0/1309623429 >> v1:172.21.15.190:6789/0 conn(0x7f56880cdaa0 legacy=0x7f56880cb260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:46.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.775+0000 7f5687e00640 1 -- 172.21.15.6:0/1309623429 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56880cca00 con 0x7f5688154d80 2024-04-24T23:27:46.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.775+0000 7f568ce00640 1 --2- 172.21.15.6:0/1309623429 >> v2:172.21.15.6:3300/0 conn(0x7f56880d03c0 0x7f56880d07f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:27:46.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.776+0000 7f5687e00640 1 --2- 172.21.15.6:0/1309623429 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5688154d80 0x7f56880c5310 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f56880c74f0 tx=0x7f567802a1a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d0371c9ee3970907 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:27:46.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.777+0000 7f5686a00640 1 -- 172.21.15.6:0/1309623429 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5678037070 con 0x7f5688154d80 2024-04-24T23:27:46.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.777+0000 7f5686a00640 1 -- 172.21.15.6:0/1309623429 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5678032440 con 0x7f5688154d80 2024-04-24T23:27:46.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.777+0000 7f5686a00640 1 -- 172.21.15.6:0/1309623429 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5678032810 con 0x7f5688154d80 2024-04-24T23:27:46.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.777+0000 7f568d800640 1 -- 172.21.15.6:0/1309623429 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5688154d80 msgr2=0x7f56880c5310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:46.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.777+0000 7f568d800640 1 --2- 172.21.15.6:0/1309623429 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5688154d80 0x7f56880c5310 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f56880c74f0 tx=0x7f567802a1a0 comp rx=0 tx=0).stop 2024-04-24T23:27:46.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.777+0000 7f568d800640 1 -- 172.21.15.6:0/1309623429 shutdown_connections 2024-04-24T23:27:46.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.777+0000 7f568d800640 1 --2- 172.21.15.6:0/1309623429 >> v2:172.21.15.6:3300/0 conn(0x7f56880d03c0 0x7f56880d07f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:46.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.777+0000 7f568d800640 1 --2- 172.21.15.6:0/1309623429 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5688154d80 0x7f56880c5310 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:46.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.777+0000 7f568d800640 1 -- 172.21.15.6:0/1309623429 >> 172.21.15.6:0/1309623429 conn(0x7f568806c990 msgr2=0x7f568806cda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:27:46.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.777+0000 7f568d800640 1 -- 172.21.15.6:0/1309623429 shutdown_connections 2024-04-24T23:27:46.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.777+0000 7f568d800640 1 -- 172.21.15.6:0/1309623429 wait complete. 2024-04-24T23:27:46.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.779+0000 7f568d800640 1 Processor -- start 2024-04-24T23:27:46.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.780+0000 7f568d800640 1 -- start start 2024-04-24T23:27:46.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.781+0000 7f568d800640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f56880cdaa0 0x7f56880cf510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:46.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.781+0000 7f568d800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f56880d03c0 0x7f5688146da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:46.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.781+0000 7f568d800640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5688154d80 0x7f56881472e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:46.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.781+0000 7f568d800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f56880cd2a0 con 0x7f56880d03c0 2024-04-24T23:27:46.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.781+0000 7f568d800640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f56880ccfc0 con 0x7f5688154d80 2024-04-24T23:27:46.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.781+0000 7f568d800640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f568815bb00 con 0x7f56880cdaa0 2024-04-24T23:27:46.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.781+0000 7f5687400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f56880d03c0 0x7f5688146da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:46.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.781+0000 7f568ce00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5688154d80 0x7f56881472e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:46.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.781+0000 7f5687e00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f56880cdaa0 0x7f56880cf510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:46.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.781+0000 7f5687400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f56880d03c0 0x7f5688146da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57984/0 (socket says 172.21.15.6:57984) 2024-04-24T23:27:46.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.781+0000 7f5687400640 1 -- 172.21.15.6:0/3199916613 learned_addr learned my addr 172.21.15.6:0/3199916613 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:27:46.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.782+0000 7f568ce00640 1 -- 172.21.15.6:0/3199916613 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f56880cdaa0 msgr2=0x7f56880cf510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:46.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.782+0000 7f568ce00640 1 --2- 172.21.15.6:0/3199916613 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f56880cdaa0 0x7f56880cf510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:46.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.782+0000 7f568ce00640 1 -- 172.21.15.6:0/3199916613 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f56880d03c0 msgr2=0x7f5688146da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:46.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.782+0000 7f568ce00640 1 --2- 172.21.15.6:0/3199916613 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f56880d03c0 0x7f5688146da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:46.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.782+0000 7f568ce00640 1 -- 172.21.15.6:0/3199916613 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56881f0b60 con 0x7f5688154d80 2024-04-24T23:27:46.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.782+0000 7f5687e00640 1 --2- 172.21.15.6:0/3199916613 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f56880cdaa0 0x7f56880cf510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:27:46.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.782+0000 7f568ce00640 1 --2- 172.21.15.6:0/3199916613 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5688154d80 0x7f56881472e0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f568000e560 tx=0x7f568000ea20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:27:46.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.782+0000 7f567fe00640 1 -- 172.21.15.6:0/3199916613 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f568000f040 con 0x7f5688154d80 2024-04-24T23:27:46.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.782+0000 7f567fe00640 1 -- 172.21.15.6:0/3199916613 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5680004230 con 0x7f5688154d80 2024-04-24T23:27:46.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.782+0000 7f567fe00640 1 -- 172.21.15.6:0/3199916613 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5680003610 con 0x7f5688154d80 2024-04-24T23:27:46.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.782+0000 7f568d800640 1 -- 172.21.15.6:0/3199916613 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56881f0e40 con 0x7f5688154d80 2024-04-24T23:27:46.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.782+0000 7f568d800640 1 -- 172.21.15.6:0/3199916613 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56881f1400 con 0x7f5688154d80 2024-04-24T23:27:46.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.784+0000 7f568d800640 1 -- 172.21.15.6:0/3199916613 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5688075ba0 con 0x7f5688154d80 2024-04-24T23:27:46.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.784+0000 7f567fe00640 1 -- 172.21.15.6:0/3199916613 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f56800037b0 con 0x7f5688154d80 2024-04-24T23:27:46.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.785+0000 7f567fe00640 1 --2- 172.21.15.6:0/3199916613 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f566406be60 0x7f566406e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:46.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.785+0000 7f567fe00640 1 -- 172.21.15.6:0/3199916613 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6934+0+0 (secure 0 0 0) 0x7f568008af30 con 0x7f5688154d80 2024-04-24T23:27:46.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.786+0000 7f5687e00640 1 --2- 172.21.15.6:0/3199916613 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f566406be60 0x7f566406e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:46.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.786+0000 7f5687e00640 1 --2- 172.21.15.6:0/3199916613 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f566406be60 0x7f566406e320 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f5678034000 tx=0x7f567802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:27:46.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.790+0000 7f567fe00640 1 -- 172.21.15.6:0/3199916613 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f5680017020 con 0x7f5688154d80 2024-04-24T23:27:46.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.993+0000 7f568d800640 1 -- 172.21.15.6:0/3199916613 --> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) -- 0x7f56880ca8a0 con 0x7f566406be60 2024-04-24T23:27:46.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.994+0000 7f567fe00640 1 -- 172.21.15.6:0/3199916613 <== mgr.4105 v2:172.21.15.190:6832/806769663 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f56880ca8a0 con 0x7f566406be60 2024-04-24T23:27:46.995 INFO:teuthology.orchestra.run.smithi006.stderr:instructed osd(s) 1 to deep-scrub 2024-04-24T23:27:46.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.996+0000 7f568d800640 1 -- 172.21.15.6:0/3199916613 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f566406be60 msgr2=0x7f566406e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:46.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.996+0000 7f568d800640 1 --2- 172.21.15.6:0/3199916613 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f566406be60 0x7f566406e320 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f5678034000 tx=0x7f567802e000 comp rx=0 tx=0).stop 2024-04-24T23:27:46.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.996+0000 7f568d800640 1 -- 172.21.15.6:0/3199916613 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5688154d80 msgr2=0x7f56881472e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:46.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.996+0000 7f568d800640 1 --2- 172.21.15.6:0/3199916613 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5688154d80 0x7f56881472e0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f568000e560 tx=0x7f568000ea20 comp rx=0 tx=0).stop 2024-04-24T23:27:46.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.997+0000 7f568d800640 1 -- 172.21.15.6:0/3199916613 shutdown_connections 2024-04-24T23:27:46.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.997+0000 7f568d800640 1 --2- 172.21.15.6:0/3199916613 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f566406be60 0x7f566406e320 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:46.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.997+0000 7f568d800640 1 --2- 172.21.15.6:0/3199916613 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5688154d80 0x7f56881472e0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:46.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.997+0000 7f568d800640 1 --2- 172.21.15.6:0/3199916613 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f56880d03c0 0x7f5688146da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:46.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.997+0000 7f568d800640 1 --2- 172.21.15.6:0/3199916613 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f56880cdaa0 0x7f56880cf510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:46.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.997+0000 7f568d800640 1 -- 172.21.15.6:0/3199916613 >> 172.21.15.6:0/3199916613 conn(0x7f568806c990 msgr2=0x7f56880cb1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:27:46.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.997+0000 7f568d800640 1 -- 172.21.15.6:0/3199916613 shutdown_connections 2024-04-24T23:27:46.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:46.997+0000 7f568d800640 1 -- 172.21.15.6:0/3199916613 wait complete. 2024-04-24T23:27:47.017 INFO:tasks.ceph:Scrubbing osd.2 2024-04-24T23:27:47.017 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-04-24T23:27:47.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.307+0000 7faf5ee00640 1 Processor -- start 2024-04-24T23:27:47.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.308+0000 7faf5ee00640 1 -- start start 2024-04-24T23:27:47.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.310+0000 7faf5ee00640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7faf581592c0 0x7faf58071070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:47.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.310+0000 7faf5ee00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faf580c5e90 0x7faf581626e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:47.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.310+0000 7faf5ee00640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faf580c9220 con 0x7faf580c8c70 2024-04-24T23:27:47.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.310+0000 7faf5ee00640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7faf580719e0 con 0x7faf581592c0 2024-04-24T23:27:47.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.310+0000 7faf5ee00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7faf580c94e0 con 0x7faf580c5e90 2024-04-24T23:27:47.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.311+0000 7faf57400640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faf580c5e90 0x7faf581626e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:47.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.311+0000 7faf57e00640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7faf581592c0 0x7faf58071070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:47.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.311+0000 7faf57e00640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7faf581592c0 0x7faf58071070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:53256/0 (socket says 172.21.15.6:53256) 2024-04-24T23:27:47.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.311+0000 7faf57e00640 1 -- 172.21.15.6:0/675091830 learned_addr learned my addr 172.21.15.6:0/675091830 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:27:47.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.312+0000 7faf57e00640 1 -- 172.21.15.6:0/675091830 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faf580c5e90 msgr2=0x7faf581626e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:27:47.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.312+0000 7faf57e00640 1 --2- 172.21.15.6:0/675091830 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faf580c5e90 0x7faf581626e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:47.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.312+0000 7faf57e00640 1 -- 172.21.15.6:0/675091830 >> v1:172.21.15.190:6789/0 conn(0x7faf580c8c70 legacy=0x7faf580c6be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:47.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.312+0000 7faf57e00640 1 -- 172.21.15.6:0/675091830 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf58071ca0 con 0x7faf581592c0 2024-04-24T23:27:47.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.314+0000 7faf57e00640 1 --2- 172.21.15.6:0/675091830 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faf581592c0 0x7faf58071070 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7faf48009870 tx=0x7faf4802a1d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2506c0dea8a64518 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:27:47.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.314+0000 7faf56a00640 1 -- 172.21.15.6:0/675091830 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf48037070 con 0x7faf581592c0 2024-04-24T23:27:47.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.314+0000 7faf56a00640 1 -- 172.21.15.6:0/675091830 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faf48032440 con 0x7faf581592c0 2024-04-24T23:27:47.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.314+0000 7faf56a00640 1 -- 172.21.15.6:0/675091830 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf48032810 con 0x7faf581592c0 2024-04-24T23:27:47.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.319+0000 7faf5ee00640 1 -- 172.21.15.6:0/675091830 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faf581592c0 msgr2=0x7faf58071070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:47.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.319+0000 7faf5ee00640 1 --2- 172.21.15.6:0/675091830 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faf581592c0 0x7faf58071070 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7faf48009870 tx=0x7faf4802a1d0 comp rx=0 tx=0).stop 2024-04-24T23:27:47.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.320+0000 7faf5ee00640 1 -- 172.21.15.6:0/675091830 shutdown_connections 2024-04-24T23:27:47.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.320+0000 7faf5ee00640 1 --2- 172.21.15.6:0/675091830 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faf580c5e90 0x7faf581626e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:47.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.321+0000 7faf5ee00640 1 --2- 172.21.15.6:0/675091830 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faf581592c0 0x7faf58071070 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:47.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.321+0000 7faf5ee00640 1 -- 172.21.15.6:0/675091830 >> 172.21.15.6:0/675091830 conn(0x7faf5806b0e0 msgr2=0x7faf5806b4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:27:47.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.321+0000 7faf5ee00640 1 -- 172.21.15.6:0/675091830 shutdown_connections 2024-04-24T23:27:47.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.321+0000 7faf5ee00640 1 -- 172.21.15.6:0/675091830 wait complete. 2024-04-24T23:27:47.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.324+0000 7faf5ee00640 1 Processor -- start 2024-04-24T23:27:47.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.324+0000 7faf5ee00640 1 -- start start 2024-04-24T23:27:47.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.325+0000 7faf5ee00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faf580c5e90 0x7faf58143250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:47.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.325+0000 7faf5ee00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faf580c8c70 0x7faf58149eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:47.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.325+0000 7faf5ee00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faf581592c0 0x7faf5814a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:47.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.326+0000 7faf5ee00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7faf58072780 con 0x7faf581592c0 2024-04-24T23:27:47.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.326+0000 7faf5ee00640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7faf58162c20 con 0x7faf580c5e90 2024-04-24T23:27:47.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.326+0000 7faf5ee00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7faf58142a90 con 0x7faf580c8c70 2024-04-24T23:27:47.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.326+0000 7faf5ce00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faf581592c0 0x7faf5814a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:47.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.326+0000 7faf57e00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faf580c5e90 0x7faf58143250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:47.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.326+0000 7faf57e00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faf580c5e90 0x7faf58143250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:53272/0 (socket says 172.21.15.6:53272) 2024-04-24T23:27:47.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.326+0000 7faf57e00640 1 -- 172.21.15.6:0/2992004117 learned_addr learned my addr 172.21.15.6:0/2992004117 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:27:47.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.326+0000 7faf57400640 1 --2- 172.21.15.6:0/2992004117 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faf580c8c70 0x7faf58149eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:47.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.327+0000 7faf5ce00640 1 -- 172.21.15.6:0/2992004117 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faf580c8c70 msgr2=0x7faf58149eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:47.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.327+0000 7faf5ce00640 1 --2- 172.21.15.6:0/2992004117 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faf580c8c70 0x7faf58149eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:47.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.327+0000 7faf5ce00640 1 -- 172.21.15.6:0/2992004117 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faf580c5e90 msgr2=0x7faf58143250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:47.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.327+0000 7faf5ce00640 1 --2- 172.21.15.6:0/2992004117 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faf580c5e90 0x7faf58143250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:47.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.327+0000 7faf5ce00640 1 -- 172.21.15.6:0/2992004117 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf58162940 con 0x7faf581592c0 2024-04-24T23:27:47.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.328+0000 7faf5ce00640 1 --2- 172.21.15.6:0/2992004117 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faf581592c0 0x7faf5814a3f0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7faf5000c3c0 tx=0x7faf5000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:27:47.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.329+0000 7faf4fe00640 1 -- 172.21.15.6:0/2992004117 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf500052d0 con 0x7faf581592c0 2024-04-24T23:27:47.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.329+0000 7faf4fe00640 1 -- 172.21.15.6:0/2992004117 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faf50005470 con 0x7faf581592c0 2024-04-24T23:27:47.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.329+0000 7faf4fe00640 1 -- 172.21.15.6:0/2992004117 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf50007260 con 0x7faf581592c0 2024-04-24T23:27:47.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.329+0000 7faf5ee00640 1 -- 172.21.15.6:0/2992004117 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf580ca640 con 0x7faf581592c0 2024-04-24T23:27:47.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.330+0000 7faf5ee00640 1 -- 172.21.15.6:0/2992004117 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf58142d30 con 0x7faf581592c0 2024-04-24T23:27:47.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.331+0000 7faf5ee00640 1 -- 172.21.15.6:0/2992004117 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7faf58075c10 con 0x7faf581592c0 2024-04-24T23:27:47.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.340+0000 7faf4fe00640 1 -- 172.21.15.6:0/2992004117 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7faf50007400 con 0x7faf581592c0 2024-04-24T23:27:47.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.341+0000 7faf4fe00640 1 --2- 172.21.15.6:0/2992004117 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7faf3406bf00 0x7faf3406e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:47.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.341+0000 7faf4fe00640 1 -- 172.21.15.6:0/2992004117 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6934+0+0 (secure 0 0 0) 0x7faf5008a450 con 0x7faf581592c0 2024-04-24T23:27:47.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.341+0000 7faf4fe00640 1 --2- 172.21.15.6:0/2992004117 >> [v2:172.21.15.6:6800/1020213724,v1:172.21.15.6:6801/1020213724] conn(0x7faf34075a30 0x7faf34077eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:47.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.342+0000 7faf4fe00640 1 -- 172.21.15.6:0/2992004117 --> [v2:172.21.15.6:6800/1020213724,v1:172.21.15.6:6801/1020213724] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7faf5000ee50 con 0x7faf34075a30 2024-04-24T23:27:47.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.342+0000 7faf4fe00640 1 -- 172.21.15.6:0/2992004117 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7faf5008f060 con 0x7faf581592c0 2024-04-24T23:27:47.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.347+0000 7faf57e00640 1 --2- 172.21.15.6:0/2992004117 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7faf3406bf00 0x7faf3406e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:47.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.348+0000 7faf57400640 1 --2- 172.21.15.6:0/2992004117 >> [v2:172.21.15.6:6800/1020213724,v1:172.21.15.6:6801/1020213724] conn(0x7faf34075a30 0x7faf34077eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:47.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.351+0000 7faf57e00640 1 --2- 172.21.15.6:0/2992004117 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7faf3406bf00 0x7faf3406e3c0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7faf480090b0 tx=0x7faf4802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:27:47.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.353+0000 7faf57400640 1 --2- 172.21.15.6:0/2992004117 >> [v2:172.21.15.6:6800/1020213724,v1:172.21.15.6:6801/1020213724] conn(0x7faf34075a30 0x7faf34077eb0 crc :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:27:47.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.356+0000 7faf4fe00640 1 -- 172.21.15.6:0/2992004117 <== osd.2 v2:172.21.15.6:6800/1020213724 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7faf5000ee50 con 0x7faf34075a30 2024-04-24T23:27:47.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.401+0000 7faf5ee00640 1 -- 172.21.15.6:0/2992004117 --> [v2:172.21.15.6:6800/1020213724,v1:172.21.15.6:6801/1020213724] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7faf58142f30 con 0x7faf34075a30 2024-04-24T23:27:47.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.416+0000 7faf4fe00640 1 -- 172.21.15.6:0/2992004117 <== osd.2 v2:172.21.15.6:6800/1020213724 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7faf58142f30 con 0x7faf34075a30 2024-04-24T23:27:47.421 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-04-24T23:27:47.421 INFO:teuthology.orchestra.run.smithi006.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-24T23:27:47.421 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-04-24T23:27:47.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.422+0000 7faf5ee00640 1 -- 172.21.15.6:0/2992004117 >> [v2:172.21.15.6:6800/1020213724,v1:172.21.15.6:6801/1020213724] conn(0x7faf34075a30 msgr2=0x7faf34077eb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:47.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.422+0000 7faf5ee00640 1 --2- 172.21.15.6:0/2992004117 >> [v2:172.21.15.6:6800/1020213724,v1:172.21.15.6:6801/1020213724] conn(0x7faf34075a30 0x7faf34077eb0 crc :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:47.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.426+0000 7faf5ee00640 1 -- 172.21.15.6:0/2992004117 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7faf3406bf00 msgr2=0x7faf3406e3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:47.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.426+0000 7faf5ee00640 1 --2- 172.21.15.6:0/2992004117 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7faf3406bf00 0x7faf3406e3c0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7faf480090b0 tx=0x7faf4802e000 comp rx=0 tx=0).stop 2024-04-24T23:27:47.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.426+0000 7faf5ee00640 1 -- 172.21.15.6:0/2992004117 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faf581592c0 msgr2=0x7faf5814a3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:47.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.426+0000 7faf5ee00640 1 --2- 172.21.15.6:0/2992004117 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faf581592c0 0x7faf5814a3f0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7faf5000c3c0 tx=0x7faf5000c880 comp rx=0 tx=0).stop 2024-04-24T23:27:47.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.426+0000 7faf5ce00640 1 -- 172.21.15.6:0/2992004117 reap_dead start 2024-04-24T23:27:47.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.426+0000 7faf5ee00640 1 -- 172.21.15.6:0/2992004117 shutdown_connections 2024-04-24T23:27:47.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.426+0000 7faf5ee00640 1 -- 172.21.15.6:0/2992004117 >> 172.21.15.6:0/2992004117 conn(0x7faf5806b0e0 msgr2=0x7faf5814b610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:27:47.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.426+0000 7faf5ee00640 1 -- 172.21.15.6:0/2992004117 shutdown_connections 2024-04-24T23:27:47.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.426+0000 7faf5ee00640 1 -- 172.21.15.6:0/2992004117 wait complete. 2024-04-24T23:27:47.453 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-04-24T23:27:47.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.742+0000 7f8cd1000640 1 Processor -- start 2024-04-24T23:27:47.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.742+0000 7f8cd1000640 1 -- start start 2024-04-24T23:27:47.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.743+0000 7f8cd1000640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8ccc152d40 0x7f8ccc1592b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:47.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.743+0000 7f8cd1000640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f8ccc06d6d0 0x7f8ccc0c4060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:47.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.743+0000 7f8cd1000640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8ccc153120 con 0x7f8ccc0c45a0 2024-04-24T23:27:47.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.743+0000 7f8cd1000640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f8ccc0c8aa0 con 0x7f8ccc06d6d0 2024-04-24T23:27:47.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.743+0000 7f8cd1000640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f8ccc06d230 con 0x7f8ccc152d40 2024-04-24T23:27:47.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.744+0000 7f8cc3e00640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f8ccc06d6d0 0x7f8ccc0c4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:47.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.744+0000 7f8cc9e00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8ccc152d40 0x7f8ccc1592b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:47.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.744+0000 7f8cca800640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f8ccc0c45a0 0x7f8ccc15bf90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.6:38434/0 (socket says 172.21.15.6:38434) 2024-04-24T23:27:47.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.744+0000 7f8cca800640 1 -- 172.21.15.6:0/2514239047 learned_addr learned my addr 172.21.15.6:0/2514239047 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:27:47.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.744+0000 7f8cc9e00640 1 -- 172.21.15.6:0/2514239047 >> v2:172.21.15.190:3300/0 conn(0x7f8ccc06d6d0 msgr2=0x7f8ccc0c4060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:27:47.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.744+0000 7f8cc9e00640 1 --2- 172.21.15.6:0/2514239047 >> v2:172.21.15.190:3300/0 conn(0x7f8ccc06d6d0 0x7f8ccc0c4060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:47.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.744+0000 7f8cc9e00640 1 -- 172.21.15.6:0/2514239047 >> v1:172.21.15.190:6789/0 conn(0x7f8ccc0c45a0 legacy=0x7f8ccc15bf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:47.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.744+0000 7f8cc9e00640 1 -- 172.21.15.6:0/2514239047 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ccc0c63d0 con 0x7f8ccc152d40 2024-04-24T23:27:47.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.744+0000 7f8cc9400640 1 -- 172.21.15.6:0/2514239047 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 731867744 0 0) 0x7f8ccc153120 con 0x7f8ccc0c45a0 2024-04-24T23:27:47.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.744+0000 7f8cc3e00640 1 --2- 172.21.15.6:0/2514239047 >> v2:172.21.15.190:3300/0 conn(0x7f8ccc06d6d0 0x7f8ccc0c4060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:27:47.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.745+0000 7f8cc9e00640 1 --2- 172.21.15.6:0/2514239047 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8ccc152d40 0x7f8ccc1592b0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f8cbc0097a0 tx=0x7f8cbc02a210 comp rx=0 tx=0).ready entity=mon.2 client_cookie=85e5e8c5da94ae7a server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:27:47.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.746+0000 7f8cc9400640 1 -- 172.21.15.6:0/2514239047 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cbc037070 con 0x7f8ccc152d40 2024-04-24T23:27:47.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.746+0000 7f8cc9400640 1 -- 172.21.15.6:0/2514239047 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8cbc032440 con 0x7f8ccc152d40 2024-04-24T23:27:47.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.746+0000 7f8cc9400640 1 -- 172.21.15.6:0/2514239047 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cbc032810 con 0x7f8ccc152d40 2024-04-24T23:27:47.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.746+0000 7f8cd1000640 1 -- 172.21.15.6:0/2514239047 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8ccc152d40 msgr2=0x7f8ccc1592b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:47.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.746+0000 7f8cd1000640 1 --2- 172.21.15.6:0/2514239047 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8ccc152d40 0x7f8ccc1592b0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f8cbc0097a0 tx=0x7f8cbc02a210 comp rx=0 tx=0).stop 2024-04-24T23:27:47.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.746+0000 7f8cd1000640 1 -- 172.21.15.6:0/2514239047 shutdown_connections 2024-04-24T23:27:47.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.746+0000 7f8cd1000640 1 --2- 172.21.15.6:0/2514239047 >> v2:172.21.15.190:3300/0 conn(0x7f8ccc06d6d0 0x7f8ccc0c4060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:47.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.746+0000 7f8cd1000640 1 --2- 172.21.15.6:0/2514239047 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8ccc152d40 0x7f8ccc1592b0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:47.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.746+0000 7f8cd1000640 1 -- 172.21.15.6:0/2514239047 >> 172.21.15.6:0/2514239047 conn(0x7f8ccc0c8c50 msgr2=0x7f8ccc0c9060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:27:47.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.746+0000 7f8cd1000640 1 -- 172.21.15.6:0/2514239047 shutdown_connections 2024-04-24T23:27:47.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.746+0000 7f8cd1000640 1 -- 172.21.15.6:0/2514239047 wait complete. 2024-04-24T23:27:47.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.748+0000 7f8cd1000640 1 Processor -- start 2024-04-24T23:27:47.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.748+0000 7f8cd1000640 1 -- start start 2024-04-24T23:27:47.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.749+0000 7f8cd1000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8ccc06d6d0 0x7f8ccc0be850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:47.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.749+0000 7f8cd1000640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8ccc0c45a0 0x7f8ccc0c7490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:47.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.749+0000 7f8cd1000640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8ccc152d40 0x7f8ccc0c0db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:47.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.749+0000 7f8cd1000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f8ccc06f8d0 con 0x7f8ccc06d6d0 2024-04-24T23:27:47.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.749+0000 7f8cd1000640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f8ccc158ab0 con 0x7f8ccc0c45a0 2024-04-24T23:27:47.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.749+0000 7f8cd1000640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f8ccc0c6a50 con 0x7f8ccc152d40 2024-04-24T23:27:47.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.749+0000 7f8cc9e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8ccc06d6d0 0x7f8ccc0be850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:47.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.749+0000 7f8cc9e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8ccc06d6d0 0x7f8ccc0be850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:58016/0 (socket says 172.21.15.6:58016) 2024-04-24T23:27:47.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.749+0000 7f8cc9e00640 1 -- 172.21.15.6:0/1396157200 learned_addr learned my addr 172.21.15.6:0/1396157200 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:27:47.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.749+0000 7f8cc3e00640 1 --2- 172.21.15.6:0/1396157200 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8ccc0c45a0 0x7f8ccc0c7490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:47.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.749+0000 7f8cca800640 1 --2- 172.21.15.6:0/1396157200 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8ccc152d40 0x7f8ccc0c0db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:47.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.750+0000 7f8cc9e00640 1 -- 172.21.15.6:0/1396157200 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8ccc152d40 msgr2=0x7f8ccc0c0db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:47.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.750+0000 7f8cc9e00640 1 --2- 172.21.15.6:0/1396157200 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8ccc152d40 0x7f8ccc0c0db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:47.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.750+0000 7f8cc9e00640 1 -- 172.21.15.6:0/1396157200 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8ccc0c45a0 msgr2=0x7f8ccc0c7490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:47.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.750+0000 7f8cc9e00640 1 --2- 172.21.15.6:0/1396157200 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8ccc0c45a0 0x7f8ccc0c7490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:47.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.750+0000 7f8cc9e00640 1 -- 172.21.15.6:0/1396157200 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ccc0c0590 con 0x7f8ccc06d6d0 2024-04-24T23:27:47.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.750+0000 7f8cc9e00640 1 --2- 172.21.15.6:0/1396157200 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8ccc06d6d0 0x7f8ccc0be850 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f8cbc0098c0 tx=0x7f8cbc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:27:47.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.750+0000 7f8cc2000640 1 -- 172.21.15.6:0/1396157200 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cbc041070 con 0x7f8ccc06d6d0 2024-04-24T23:27:47.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.750+0000 7f8cc2000640 1 -- 172.21.15.6:0/1396157200 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8cbc03c870 con 0x7f8ccc06d6d0 2024-04-24T23:27:47.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.750+0000 7f8cc2000640 1 -- 172.21.15.6:0/1396157200 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cbc037040 con 0x7f8ccc06d6d0 2024-04-24T23:27:47.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.751+0000 7f8cd1000640 1 -- 172.21.15.6:0/1396157200 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ccc0be290 con 0x7f8ccc06d6d0 2024-04-24T23:27:47.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.751+0000 7f8cd1000640 1 -- 172.21.15.6:0/1396157200 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ccc155130 con 0x7f8ccc06d6d0 2024-04-24T23:27:47.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.752+0000 7f8cd1000640 1 -- 172.21.15.6:0/1396157200 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ccc0767b0 con 0x7f8ccc06d6d0 2024-04-24T23:27:47.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.753+0000 7f8cc2000640 1 -- 172.21.15.6:0/1396157200 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f8cbc044070 con 0x7f8ccc06d6d0 2024-04-24T23:27:47.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.754+0000 7f8cc2000640 1 --2- 172.21.15.6:0/1396157200 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f8ca806be60 0x7f8ca806e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:47.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.754+0000 7f8cc2000640 1 -- 172.21.15.6:0/1396157200 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6934+0+0 (secure 0 0 0) 0x7f8cbc0abeb0 con 0x7f8ccc06d6d0 2024-04-24T23:27:47.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.754+0000 7f8cc3e00640 1 --2- 172.21.15.6:0/1396157200 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f8ca806be60 0x7f8ca806e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:47.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.755+0000 7f8cc3e00640 1 --2- 172.21.15.6:0/1396157200 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f8ca806be60 0x7f8ca806e320 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f8cb4009560 tx=0x7f8cb4009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:27:47.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.759+0000 7f8cc2000640 1 -- 172.21.15.6:0/1396157200 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f8cbc079d30 con 0x7f8ccc06d6d0 2024-04-24T23:27:47.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.979+0000 7f8cd1000640 1 -- 172.21.15.6:0/1396157200 --> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) -- 0x7f8ccc0c4380 con 0x7f8ca806be60 2024-04-24T23:27:47.981 INFO:teuthology.orchestra.run.smithi006.stderr:instructed osd(s) 2 to deep-scrub 2024-04-24T23:27:47.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.980+0000 7f8cc2000640 1 -- 172.21.15.6:0/1396157200 <== mgr.4105 v2:172.21.15.190:6832/806769663 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f8ccc0c4380 con 0x7f8ca806be60 2024-04-24T23:27:47.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.982+0000 7f8cd1000640 1 -- 172.21.15.6:0/1396157200 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f8ca806be60 msgr2=0x7f8ca806e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:47.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.982+0000 7f8cd1000640 1 --2- 172.21.15.6:0/1396157200 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f8ca806be60 0x7f8ca806e320 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f8cb4009560 tx=0x7f8cb4009190 comp rx=0 tx=0).stop 2024-04-24T23:27:47.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.982+0000 7f8cd1000640 1 -- 172.21.15.6:0/1396157200 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8ccc06d6d0 msgr2=0x7f8ccc0be850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:47.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.982+0000 7f8cd1000640 1 --2- 172.21.15.6:0/1396157200 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8ccc06d6d0 0x7f8ccc0be850 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f8cbc0098c0 tx=0x7f8cbc0046c0 comp rx=0 tx=0).stop 2024-04-24T23:27:47.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.982+0000 7f8cd1000640 1 -- 172.21.15.6:0/1396157200 shutdown_connections 2024-04-24T23:27:47.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.982+0000 7f8cd1000640 1 --2- 172.21.15.6:0/1396157200 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f8ca806be60 0x7f8ca806e320 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:47.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.982+0000 7f8cd1000640 1 --2- 172.21.15.6:0/1396157200 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8ccc152d40 0x7f8ccc0c0db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:47.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.982+0000 7f8cd1000640 1 --2- 172.21.15.6:0/1396157200 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8ccc0c45a0 0x7f8ccc0c7490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:47.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.982+0000 7f8cd1000640 1 --2- 172.21.15.6:0/1396157200 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8ccc06d6d0 0x7f8ccc0be850 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:47.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.982+0000 7f8cd1000640 1 -- 172.21.15.6:0/1396157200 >> 172.21.15.6:0/1396157200 conn(0x7f8ccc0c8c50 msgr2=0x7f8ccc151f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:27:47.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.983+0000 7f8cd1000640 1 -- 172.21.15.6:0/1396157200 shutdown_connections 2024-04-24T23:27:47.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:47.983+0000 7f8cd1000640 1 -- 172.21.15.6:0/1396157200 wait complete. 2024-04-24T23:27:48.002 INFO:tasks.ceph:Scrubbing osd.3 2024-04-24T23:27:48.002 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-04-24T23:27:48.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:48.206+0000 7f611ae00640 1 Processor -- start 2024-04-24T23:27:48.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:48.206+0000 7f611ae00640 1 -- start start 2024-04-24T23:27:48.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:48.207+0000 7f611ae00640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f61140c9c00 0x7f61140ce150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:48.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:48.208+0000 7f611ae00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f611406f560 con 0x7f61140ca350 2024-04-24T23:27:48.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:48.208+0000 7f611ae00640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f611415ef30 con 0x7f6114154c60 2024-04-24T23:27:48.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:48.208+0000 7f611ae00640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f611415ea10 con 0x7f61140c9c00 2024-04-24T23:27:48.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:48.208+0000 7f6119000640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f61140c9c00 0x7f61140ce150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:48.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:48.208+0000 7f611a400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f61140ca350 0x7f61141653c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:34874/0 (socket says 172.21.15.6:34874) 2024-04-24T23:27:48.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:48.208+0000 7f611a400640 1 -- 172.21.15.6:0/740356399 learned_addr learned my addr 172.21.15.6:0/740356399 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:27:48.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:48.208+0000 7f6119000640 1 -- 172.21.15.6:0/740356399 >> v1:172.21.15.190:6789/0 conn(0x7f6114154c60 legacy=0x7f611415b350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:48.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:48.208+0000 7f6119000640 1 -- 172.21.15.6:0/740356399 >> v1:172.21.15.6:6789/0 conn(0x7f61140ca350 legacy=0x7f61141653c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:48.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:48.208+0000 7f6119000640 1 -- 172.21.15.6:0/740356399 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f611406f800 con 0x7f61140c9c00 2024-04-24T23:27:48.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:48.209+0000 7f6119000640 1 --2- 172.21.15.6:0/740356399 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f61140c9c00 0x7f61140ce150 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f61000097a0 tx=0x7f610002a1a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fce2c730a57f7cb3 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:27:48.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:48.210+0000 7f6113e00640 1 -- 172.21.15.6:0/740356399 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6100037070 con 0x7f61140c9c00 2024-04-24T23:27:48.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:48.210+0000 7f6113e00640 1 -- 172.21.15.6:0/740356399 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6100032440 con 0x7f61140c9c00 2024-04-24T23:27:48.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:48.210+0000 7f6113e00640 1 -- 172.21.15.6:0/740356399 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6100032810 con 0x7f61140c9c00 2024-04-24T23:27:48.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:48.210+0000 7f611ae00640 1 -- 172.21.15.6:0/740356399 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f61140c9c00 msgr2=0x7f61140ce150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:48.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:48.210+0000 7f611ae00640 1 --2- 172.21.15.6:0/740356399 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f61140c9c00 0x7f61140ce150 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f61000097a0 tx=0x7f610002a1a0 comp rx=0 tx=0).stop 2024-04-24T23:27:48.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:48.210+0000 7f611ae00640 1 -- 172.21.15.6:0/740356399 shutdown_connections 2024-04-24T23:27:48.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:48.210+0000 7f611ae00640 1 --2- 172.21.15.6:0/740356399 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f61140c9c00 0x7f61140ce150 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:48.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:48.210+0000 7f611ae00640 1 -- 172.21.15.6:0/740356399 >> 172.21.15.6:0/740356399 conn(0x7f6114066ea0 msgr2=0x7f6114067280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:27:48.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:48.211+0000 7f611ae00640 1 -- 172.21.15.6:0/740356399 shutdown_connections 2024-04-24T23:27:48.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:48.211+0000 7f611ae00640 1 -- 172.21.15.6:0/740356399 wait complete. 2024-04-24T23:27:48.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:48.213+0000 7f611ae00640 1 Processor -- start 2024-04-24T23:27:48.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:48.213+0000 7f611ae00640 1 -- start start 2024-04-24T23:27:48.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:48.214+0000 7f611ae00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f61140c9c00 0x7f611414d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:48.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:48.214+0000 7f611ae00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f61140ca350 0x7f611414dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:48.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:48.214+0000 7f611ae00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6114154c60 0x7f611415bd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:48.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:48.214+0000 7f611ae00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f611415ab50 con 0x7f6114154c60 2024-04-24T23:27:48.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:48.214+0000 7f611ae00640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f611415ba60 con 0x7f61140ca350 2024-04-24T23:27:48.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:48.214+0000 7f611ae00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f6114155040 con 0x7f61140c9c00 2024-04-24T23:27:48.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:48.214+0000 7f6119000640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f61140ca350 0x7f611414dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:48.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:48.214+0000 7f6119000640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f61140ca350 0x7f611414dcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:53314/0 (socket says 172.21.15.6:53314) 2024-04-24T23:27:48.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:48.214+0000 7f6119000640 1 -- 172.21.15.6:0/1891693571 learned_addr learned my addr 172.21.15.6:0/1891693571 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:27:48.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:48.215+0000 7f611a400640 1 --2- 172.21.15.6:0/1891693571 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6114154c60 0x7f611415bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:48.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:48.215+0000 7f6119a00640 1 --2- 172.21.15.6:0/1891693571 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f61140c9c00 0x7f611414d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:48.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:48.215+0000 7f6119000640 1 -- 172.21.15.6:0/1891693571 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f61140c9c00 msgr2=0x7f611414d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:48.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:48.215+0000 7f6119000640 1 --2- 172.21.15.6:0/1891693571 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f61140c9c00 0x7f611414d7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:48.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:48.215+0000 7f6119000640 1 -- 172.21.15.6:0/1891693571 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6114154c60 msgr2=0x7f611415bd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:48.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:48.215+0000 7f6119000640 1 --2- 172.21.15.6:0/1891693571 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6114154c60 0x7f611415bd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:48.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:48.215+0000 7f6119000640 1 -- 172.21.15.6:0/1891693571 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61141f4ae0 con 0x7f61140ca350 2024-04-24T23:27:48.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:48.215+0000 7f6119a00640 1 --2- 172.21.15.6:0/1891693571 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f61140c9c00 0x7f611414d7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:27:48.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:48.215+0000 7f611a400640 1 --2- 172.21.15.6:0/1891693571 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6114154c60 0x7f611415bd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:27:48.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:48.215+0000 7f6119000640 1 --2- 172.21.15.6:0/1891693571 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f61140ca350 0x7f611414dcf0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f6100035040 tx=0x7f61000046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:27:48.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:48.215+0000 7f6111600640 1 -- 172.21.15.6:0/1891693571 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6100037040 con 0x7f61140ca350 2024-04-24T23:27:48.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:48.215+0000 7f6111600640 1 -- 172.21.15.6:0/1891693571 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6100030770 con 0x7f61140ca350 2024-04-24T23:27:48.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:48.215+0000 7f6111600640 1 -- 172.21.15.6:0/1891693571 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6100030b40 con 0x7f61140ca350 2024-04-24T23:27:48.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:48.215+0000 7f611ae00640 1 -- 172.21.15.6:0/1891693571 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61141f0810 con 0x7f61140ca350 2024-04-24T23:27:48.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:48.216+0000 7f611ae00640 1 -- 172.21.15.6:0/1891693571 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61141f0ab0 con 0x7f61140ca350 2024-04-24T23:27:48.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:48.216+0000 7f611ae00640 1 -- 172.21.15.6:0/1891693571 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f6114068780 con 0x7f61140ca350 2024-04-24T23:27:48.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:48.218+0000 7f6111600640 1 -- 172.21.15.6:0/1891693571 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f6100044020 con 0x7f61140ca350 2024-04-24T23:27:48.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:48.219+0000 7f6111600640 1 --2- 172.21.15.6:0/1891693571 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f60f406be30 0x7f60f406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:48.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:48.219+0000 7f6111600640 1 -- 172.21.15.6:0/1891693571 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6934+0+0 (secure 0 0 0) 0x7f61000aba70 con 0x7f61140ca350 2024-04-24T23:27:48.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:48.220+0000 7f6111600640 1 --2- 172.21.15.6:0/1891693571 >> [v2:172.21.15.6:6824/3163883468,v1:172.21.15.6:6825/3163883468] conn(0x7f60f40758e0 0x7f60f4077d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:48.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:48.220+0000 7f6111600640 1 -- 172.21.15.6:0/1891693571 --> [v2:172.21.15.6:6824/3163883468,v1:172.21.15.6:6825/3163883468] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6100031ec0 con 0x7f60f40758e0 2024-04-24T23:27:48.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:48.220+0000 7f6111600640 1 -- 172.21.15.6:0/1891693571 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f61000b02a0 con 0x7f61140ca350 2024-04-24T23:27:48.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:48.220+0000 7f6119a00640 1 --2- 172.21.15.6:0/1891693571 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f60f406be30 0x7f60f406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:48.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:48.220+0000 7f611a400640 1 --2- 172.21.15.6:0/1891693571 >> [v2:172.21.15.6:6824/3163883468,v1:172.21.15.6:6825/3163883468] conn(0x7f60f40758e0 0x7f60f4077d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:48.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:48.220+0000 7f6119a00640 1 --2- 172.21.15.6:0/1891693571 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f60f406be30 0x7f60f406e2f0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f6108007fd0 tx=0x7f6108009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:27:48.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:48.221+0000 7f611a400640 1 --2- 172.21.15.6:0/1891693571 >> [v2:172.21.15.6:6824/3163883468,v1:172.21.15.6:6825/3163883468] conn(0x7f60f40758e0 0x7f60f4077d60 crc :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:27:48.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:48.222+0000 7f6111600640 1 -- 172.21.15.6:0/1891693571 <== osd.3 v2:172.21.15.6:6824/3163883468 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6100031ec0 con 0x7f60f40758e0 2024-04-24T23:27:48.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:48.236+0000 7f611ae00640 1 -- 172.21.15.6:0/1891693571 --> [v2:172.21.15.6:6824/3163883468,v1:172.21.15.6:6825/3163883468] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f6114075b30 con 0x7f60f40758e0 2024-04-24T23:27:48.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:48.237+0000 7f6111600640 1 -- 172.21.15.6:0/1891693571 <== osd.3 v2:172.21.15.6:6824/3163883468 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f6114075b30 con 0x7f60f40758e0 2024-04-24T23:27:48.238 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-04-24T23:27:48.238 INFO:teuthology.orchestra.run.smithi006.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-24T23:27:48.238 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-04-24T23:27:48.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:48.237+0000 7f611ae00640 1 -- 172.21.15.6:0/1891693571 >> [v2:172.21.15.6:6824/3163883468,v1:172.21.15.6:6825/3163883468] conn(0x7f60f40758e0 msgr2=0x7f60f4077d60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:48.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:48.237+0000 7f611ae00640 1 --2- 172.21.15.6:0/1891693571 >> [v2:172.21.15.6:6824/3163883468,v1:172.21.15.6:6825/3163883468] conn(0x7f60f40758e0 0x7f60f4077d60 crc :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:48.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:48.237+0000 7f611ae00640 1 -- 172.21.15.6:0/1891693571 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f60f406be30 msgr2=0x7f60f406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:48.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:48.237+0000 7f611ae00640 1 --2- 172.21.15.6:0/1891693571 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f60f406be30 0x7f60f406e2f0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f6108007fd0 tx=0x7f6108009040 comp rx=0 tx=0).stop 2024-04-24T23:27:48.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:48.237+0000 7f611ae00640 1 -- 172.21.15.6:0/1891693571 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f61140ca350 msgr2=0x7f611414dcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:48.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:48.237+0000 7f611ae00640 1 --2- 172.21.15.6:0/1891693571 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f61140ca350 0x7f611414dcf0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f6100035040 tx=0x7f61000046c0 comp rx=0 tx=0).stop 2024-04-24T23:27:48.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:48.237+0000 7f611a400640 1 -- 172.21.15.6:0/1891693571 reap_dead start 2024-04-24T23:27:48.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:48.237+0000 7f611ae00640 1 -- 172.21.15.6:0/1891693571 shutdown_connections 2024-04-24T23:27:48.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:48.237+0000 7f611ae00640 1 -- 172.21.15.6:0/1891693571 >> 172.21.15.6:0/1891693571 conn(0x7f6114066ea0 msgr2=0x7f61140c4700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:27:48.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:48.237+0000 7f611ae00640 1 -- 172.21.15.6:0/1891693571 shutdown_connections 2024-04-24T23:27:48.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:48.238+0000 7f611ae00640 1 -- 172.21.15.6:0/1891693571 wait complete. 2024-04-24T23:27:48.260 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-04-24T23:27:48.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:48.591+0000 7fd597e00640 1 Processor -- start 2024-04-24T23:27:48.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:48.591+0000 7fd597e00640 1 -- start start 2024-04-24T23:27:48.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:48.592+0000 7fd597e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd5900deda0 0x7fd590037160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:48.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:48.593+0000 7fd597e00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd5900df1e0 con 0x7fd5900376a0 2024-04-24T23:27:48.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:48.593+0000 7fd597e00640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd5900df4a0 con 0x7fd5900d9090 2024-04-24T23:27:48.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:48.593+0000 7fd597e00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7fd590037ae0 con 0x7fd5900deda0 2024-04-24T23:27:48.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:48.593+0000 7fd596000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd5900deda0 0x7fd590037160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:48.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:48.593+0000 7fd596000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd5900deda0 0x7fd590037160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:58034/0 (socket says 172.21.15.6:58034) 2024-04-24T23:27:48.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:48.593+0000 7fd596000640 1 -- 172.21.15.6:0/2773099024 learned_addr learned my addr 172.21.15.6:0/2773099024 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:27:48.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:48.594+0000 7fd595600640 1 -- 172.21.15.6:0/2773099024 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 100854024 0 0) 0x7fd5900df4a0 con 0x7fd5900d9090 2024-04-24T23:27:48.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:48.594+0000 7fd595600640 1 -- 172.21.15.6:0/2773099024 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd57c003520 con 0x7fd5900d9090 2024-04-24T23:27:48.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:48.594+0000 7fd596000640 1 -- 172.21.15.6:0/2773099024 >> v1:172.21.15.190:6789/0 conn(0x7fd5900d9090 legacy=0x7fd590050180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:48.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:48.594+0000 7fd596000640 1 -- 172.21.15.6:0/2773099024 >> v1:172.21.15.6:6789/0 conn(0x7fd5900376a0 legacy=0x7fd590038520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:48.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:48.594+0000 7fd596000640 1 -- 172.21.15.6:0/2773099024 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5900df780 con 0x7fd5900deda0 2024-04-24T23:27:48.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:48.595+0000 7fd596000640 1 --2- 172.21.15.6:0/2773099024 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd5900deda0 0x7fd590037160 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fd584009870 tx=0x7fd58402a300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=812a0b87da0ffbb3 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:27:48.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:48.595+0000 7fd595600640 1 -- 172.21.15.6:0/2773099024 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd584037070 con 0x7fd5900deda0 2024-04-24T23:27:48.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:48.595+0000 7fd595600640 1 -- 172.21.15.6:0/2773099024 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd58402ae50 con 0x7fd5900deda0 2024-04-24T23:27:48.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:48.595+0000 7fd595600640 1 -- 172.21.15.6:0/2773099024 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd584032860 con 0x7fd5900deda0 2024-04-24T23:27:48.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:48.596+0000 7fd597e00640 1 -- 172.21.15.6:0/2773099024 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd5900deda0 msgr2=0x7fd590037160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:48.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:48.596+0000 7fd597e00640 1 --2- 172.21.15.6:0/2773099024 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd5900deda0 0x7fd590037160 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fd584009870 tx=0x7fd58402a300 comp rx=0 tx=0).stop 2024-04-24T23:27:48.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:48.596+0000 7fd597e00640 1 -- 172.21.15.6:0/2773099024 shutdown_connections 2024-04-24T23:27:48.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:48.596+0000 7fd597e00640 1 --2- 172.21.15.6:0/2773099024 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd5900deda0 0x7fd590037160 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:48.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:48.596+0000 7fd597e00640 1 -- 172.21.15.6:0/2773099024 >> 172.21.15.6:0/2773099024 conn(0x7fd590054610 msgr2=0x7fd590054a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:27:48.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:48.596+0000 7fd597e00640 1 -- 172.21.15.6:0/2773099024 shutdown_connections 2024-04-24T23:27:48.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:48.596+0000 7fd597e00640 1 -- 172.21.15.6:0/2773099024 wait complete. 2024-04-24T23:27:48.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:48.599+0000 7fd597e00640 1 Processor -- start 2024-04-24T23:27:48.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:48.599+0000 7fd597e00640 1 -- start start 2024-04-24T23:27:48.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:48.600+0000 7fd597e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd5900376a0 0x7fd5900ccad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:48.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:48.600+0000 7fd597e00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd5900d9090 0x7fd5900cd930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:48.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:48.600+0000 7fd597e00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd5900deda0 0x7fd5900d6cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:48.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:48.600+0000 7fd597e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fd590037d20 con 0x7fd5900376a0 2024-04-24T23:27:48.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:48.600+0000 7fd597e00640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fd590038090 con 0x7fd5900d9090 2024-04-24T23:27:48.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:48.600+0000 7fd597e00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fd5900d9470 con 0x7fd5900deda0 2024-04-24T23:27:48.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:48.600+0000 7fd596a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd5900376a0 0x7fd5900ccad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:48.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:48.600+0000 7fd596000640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd5900d9090 0x7fd5900cd930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:48.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:48.600+0000 7fd597400640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd5900deda0 0x7fd5900d6cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:48.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:48.600+0000 7fd596a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd5900376a0 0x7fd5900ccad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:58050/0 (socket says 172.21.15.6:58050) 2024-04-24T23:27:48.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:48.600+0000 7fd596a00640 1 -- 172.21.15.6:0/422181628 learned_addr learned my addr 172.21.15.6:0/422181628 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:27:48.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:48.601+0000 7fd597400640 1 -- 172.21.15.6:0/422181628 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd5900d9090 msgr2=0x7fd5900cd930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:48.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:48.601+0000 7fd597400640 1 --2- 172.21.15.6:0/422181628 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd5900d9090 0x7fd5900cd930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:48.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:48.601+0000 7fd597400640 1 -- 172.21.15.6:0/422181628 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd5900376a0 msgr2=0x7fd5900ccad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:48.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:48.601+0000 7fd597400640 1 --2- 172.21.15.6:0/422181628 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd5900376a0 0x7fd5900ccad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:48.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:48.601+0000 7fd597400640 1 -- 172.21.15.6:0/422181628 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5900d8720 con 0x7fd5900deda0 2024-04-24T23:27:48.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:48.601+0000 7fd596000640 1 --2- 172.21.15.6:0/422181628 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd5900d9090 0x7fd5900cd930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:27:48.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:48.601+0000 7fd596a00640 1 --2- 172.21.15.6:0/422181628 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd5900376a0 0x7fd5900ccad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:27:48.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:48.601+0000 7fd597400640 1 --2- 172.21.15.6:0/422181628 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd5900deda0 0x7fd5900d6cd0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fd598064ed0 tx=0x7fd598076b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:27:48.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:48.601+0000 7fd58f400640 1 -- 172.21.15.6:0/422181628 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd598068b40 con 0x7fd5900deda0 2024-04-24T23:27:48.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:48.601+0000 7fd58f400640 1 -- 172.21.15.6:0/422181628 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd598090030 con 0x7fd5900deda0 2024-04-24T23:27:48.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:48.601+0000 7fd58f400640 1 -- 172.21.15.6:0/422181628 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd598068f60 con 0x7fd5900deda0 2024-04-24T23:27:48.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:48.601+0000 7fd597e00640 1 -- 172.21.15.6:0/422181628 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd590175ed0 con 0x7fd5900deda0 2024-04-24T23:27:48.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:48.601+0000 7fd597e00640 1 -- 172.21.15.6:0/422181628 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5900ce1f0 con 0x7fd5900deda0 2024-04-24T23:27:48.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:48.602+0000 7fd597e00640 1 -- 172.21.15.6:0/422181628 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd5900052c0 con 0x7fd5900deda0 2024-04-24T23:27:48.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:48.604+0000 7fd58f400640 1 -- 172.21.15.6:0/422181628 <== mon.2 v2:172.21.15.190:3301/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fd598061b80 con 0x7fd5900deda0 2024-04-24T23:27:48.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:48.604+0000 7fd58f400640 1 --2- 172.21.15.6:0/422181628 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fd57806be30 0x7fd57806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:48.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:48.605+0000 7fd58f400640 1 -- 172.21.15.6:0/422181628 <== mon.2 v2:172.21.15.190:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6934+0+0 (secure 0 0 0) 0x7fd598107600 con 0x7fd5900deda0 2024-04-24T23:27:48.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:48.605+0000 7fd596a00640 1 --2- 172.21.15.6:0/422181628 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fd57806be30 0x7fd57806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:48.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:48.606+0000 7fd596a00640 1 --2- 172.21.15.6:0/422181628 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fd57806be30 0x7fd57806e2f0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fd5880053c0 tx=0x7fd588007d30 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:27:48.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:48.610+0000 7fd58f400640 1 -- 172.21.15.6:0/422181628 <== mon.2 v2:172.21.15.190:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fd5980d5360 con 0x7fd5900deda0 2024-04-24T23:27:48.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:48.834+0000 7fd597e00640 1 -- 172.21.15.6:0/422181628 --> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) -- 0x7fd590007a30 con 0x7fd57806be30 2024-04-24T23:27:48.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:48.835+0000 7fd58f400640 1 -- 172.21.15.6:0/422181628 <== mgr.4105 v2:172.21.15.190:6832/806769663 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fd590007a30 con 0x7fd57806be30 2024-04-24T23:27:48.836 INFO:teuthology.orchestra.run.smithi006.stderr:instructed osd(s) 3 to deep-scrub 2024-04-24T23:27:48.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:48.838+0000 7fd597e00640 1 -- 172.21.15.6:0/422181628 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fd57806be30 msgr2=0x7fd57806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:48.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:48.838+0000 7fd597e00640 1 --2- 172.21.15.6:0/422181628 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fd57806be30 0x7fd57806e2f0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fd5880053c0 tx=0x7fd588007d30 comp rx=0 tx=0).stop 2024-04-24T23:27:48.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:48.838+0000 7fd597e00640 1 -- 172.21.15.6:0/422181628 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd5900deda0 msgr2=0x7fd5900d6cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:48.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:48.838+0000 7fd597e00640 1 --2- 172.21.15.6:0/422181628 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd5900deda0 0x7fd5900d6cd0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fd598064ed0 tx=0x7fd598076b80 comp rx=0 tx=0).stop 2024-04-24T23:27:48.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:48.839+0000 7fd597e00640 1 -- 172.21.15.6:0/422181628 shutdown_connections 2024-04-24T23:27:48.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:48.839+0000 7fd597e00640 1 --2- 172.21.15.6:0/422181628 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fd57806be30 0x7fd57806e2f0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:48.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:48.839+0000 7fd597e00640 1 --2- 172.21.15.6:0/422181628 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd5900deda0 0x7fd5900d6cd0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:48.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:48.839+0000 7fd597e00640 1 --2- 172.21.15.6:0/422181628 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd5900d9090 0x7fd5900cd930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:48.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:48.839+0000 7fd597e00640 1 --2- 172.21.15.6:0/422181628 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd5900376a0 0x7fd5900ccad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:48.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:48.839+0000 7fd597e00640 1 -- 172.21.15.6:0/422181628 >> 172.21.15.6:0/422181628 conn(0x7fd590054610 msgr2=0x7fd5900d4320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:27:48.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:48.839+0000 7fd597e00640 1 -- 172.21.15.6:0/422181628 shutdown_connections 2024-04-24T23:27:48.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:48.839+0000 7fd597e00640 1 -- 172.21.15.6:0/422181628 wait complete. 2024-04-24T23:27:48.860 INFO:tasks.ceph:Scrubbing osd.4 2024-04-24T23:27:48.860 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-04-24T23:27:49.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.065+0000 7f277bc00640 1 Processor -- start 2024-04-24T23:27:49.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.065+0000 7f277bc00640 1 -- start start 2024-04-24T23:27:49.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.066+0000 7f277bc00640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f2774151de0 0x7f27740c36e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:49.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.066+0000 7f277bc00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f2774157940 0x7f2774071840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:49.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.066+0000 7f277bc00640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2774071280 con 0x7f2774071ed0 2024-04-24T23:27:49.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.066+0000 7f277bc00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f2774157db0 con 0x7f2774157940 2024-04-24T23:27:49.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.066+0000 7f277bc00640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f2774071560 con 0x7f2774151de0 2024-04-24T23:27:49.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.067+0000 7f2779200640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f2774151de0 0x7f27740c36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:49.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.067+0000 7f2779200640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f2774151de0 0x7f27740c36e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:53330/0 (socket says 172.21.15.6:53330) 2024-04-24T23:27:49.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.067+0000 7f2779200640 1 -- 172.21.15.6:0/3456431083 learned_addr learned my addr 172.21.15.6:0/3456431083 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:27:49.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.067+0000 7f2773e00640 1 --1- 172.21.15.6:0/3456431083 >> v1:172.21.15.190:6789/0 conn(0x7f2774071ed0 0x7f27740c51b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.6:38464/0 (socket says 172.21.15.6:38464) 2024-04-24T23:27:49.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.067+0000 7f2779c00640 1 --2- 172.21.15.6:0/3456431083 >> v2:172.21.15.6:3300/0 conn(0x7f2774157940 0x7f2774071840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:49.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.067+0000 7f2779200640 1 -- 172.21.15.6:0/3456431083 >> v2:172.21.15.6:3300/0 conn(0x7f2774157940 msgr2=0x7f2774071840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:27:49.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.067+0000 7f2779200640 1 --2- 172.21.15.6:0/3456431083 >> v2:172.21.15.6:3300/0 conn(0x7f2774157940 0x7f2774071840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:49.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.067+0000 7f2779200640 1 -- 172.21.15.6:0/3456431083 >> v1:172.21.15.190:6789/0 conn(0x7f2774071ed0 legacy=0x7f27740c51b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:49.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.067+0000 7f2779200640 1 -- 172.21.15.6:0/3456431083 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f277415af30 con 0x7f2774151de0 2024-04-24T23:27:49.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.068+0000 7f2779200640 1 --2- 172.21.15.6:0/3456431083 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2774151de0 0x7f27740c36e0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f277406fbc0 tx=0x7f276402c830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=724eadbfcda936ff server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:27:49.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.069+0000 7f2773400640 1 -- 172.21.15.6:0/3456431083 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2764038070 con 0x7f2774151de0 2024-04-24T23:27:49.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.069+0000 7f2773400640 1 -- 172.21.15.6:0/3456431083 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2764031710 con 0x7f2774151de0 2024-04-24T23:27:49.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.069+0000 7f2773400640 1 -- 172.21.15.6:0/3456431083 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2764031ae0 con 0x7f2774151de0 2024-04-24T23:27:49.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.069+0000 7f277bc00640 1 -- 172.21.15.6:0/3456431083 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2774151de0 msgr2=0x7f27740c36e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:49.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.069+0000 7f277bc00640 1 --2- 172.21.15.6:0/3456431083 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2774151de0 0x7f27740c36e0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f277406fbc0 tx=0x7f276402c830 comp rx=0 tx=0).stop 2024-04-24T23:27:49.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.069+0000 7f277bc00640 1 -- 172.21.15.6:0/3456431083 shutdown_connections 2024-04-24T23:27:49.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.069+0000 7f277bc00640 1 --2- 172.21.15.6:0/3456431083 >> v2:172.21.15.6:3300/0 conn(0x7f2774157940 0x7f2774071840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:49.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.069+0000 7f277bc00640 1 --2- 172.21.15.6:0/3456431083 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2774151de0 0x7f27740c36e0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:49.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.069+0000 7f277bc00640 1 -- 172.21.15.6:0/3456431083 >> 172.21.15.6:0/3456431083 conn(0x7f277406b0e0 msgr2=0x7f277406b4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:27:49.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.069+0000 7f277bc00640 1 -- 172.21.15.6:0/3456431083 shutdown_connections 2024-04-24T23:27:49.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.070+0000 7f277bc00640 1 -- 172.21.15.6:0/3456431083 wait complete. 2024-04-24T23:27:49.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.071+0000 7f277bc00640 1 Processor -- start 2024-04-24T23:27:49.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.071+0000 7f277bc00640 1 -- start start 2024-04-24T23:27:49.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.072+0000 7f277bc00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2774071ed0 0x7f277414e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:49.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.072+0000 7f277bc00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2774151de0 0x7f27740ca0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:49.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.072+0000 7f277bc00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2774157940 0x7f27740ca5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:49.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.072+0000 7f277bc00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f27740c58c0 con 0x7f2774071ed0 2024-04-24T23:27:49.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.072+0000 7f277bc00640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f277415b4f0 con 0x7f2774157940 2024-04-24T23:27:49.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.072+0000 7f277bc00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f27740c5a40 con 0x7f2774151de0 2024-04-24T23:27:49.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.072+0000 7f2773e00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2774151de0 0x7f27740ca0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:49.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.072+0000 7f2779200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2774071ed0 0x7f277414e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:49.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.072+0000 7f2773e00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2774151de0 0x7f27740ca0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.6:43668/0 (socket says 172.21.15.6:43668) 2024-04-24T23:27:49.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.072+0000 7f2773e00640 1 -- 172.21.15.6:0/54669526 learned_addr learned my addr 172.21.15.6:0/54669526 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:27:49.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.073+0000 7f2779c00640 1 --2- 172.21.15.6:0/54669526 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2774157940 0x7f27740ca5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:49.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.073+0000 7f2779200640 1 -- 172.21.15.6:0/54669526 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2774151de0 msgr2=0x7f27740ca0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:49.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.073+0000 7f2779200640 1 --2- 172.21.15.6:0/54669526 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2774151de0 0x7f27740ca0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:49.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.073+0000 7f2779200640 1 -- 172.21.15.6:0/54669526 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2774157940 msgr2=0x7f27740ca5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:49.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.073+0000 7f2779200640 1 --2- 172.21.15.6:0/54669526 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2774157940 0x7f27740ca5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:49.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.073+0000 7f2779200640 1 -- 172.21.15.6:0/54669526 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f277414acf0 con 0x7f2774071ed0 2024-04-24T23:27:49.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.073+0000 7f2779200640 1 --2- 172.21.15.6:0/54669526 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2774071ed0 0x7f277414e3b0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f2764033000 tx=0x7f27640046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:27:49.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.074+0000 7f2770c00640 1 -- 172.21.15.6:0/54669526 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2764038040 con 0x7f2774071ed0 2024-04-24T23:27:49.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.074+0000 7f2770c00640 1 -- 172.21.15.6:0/54669526 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2764004230 con 0x7f2774071ed0 2024-04-24T23:27:49.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.074+0000 7f2770c00640 1 -- 172.21.15.6:0/54669526 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f276403d670 con 0x7f2774071ed0 2024-04-24T23:27:49.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.074+0000 7f277bc00640 1 -- 172.21.15.6:0/54669526 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f277414afd0 con 0x7f2774071ed0 2024-04-24T23:27:49.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.074+0000 7f277bc00640 1 -- 172.21.15.6:0/54669526 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f277414c9f0 con 0x7f2774071ed0 2024-04-24T23:27:49.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.075+0000 7f277bc00640 1 -- 172.21.15.6:0/54669526 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f2774075c10 con 0x7f2774071ed0 2024-04-24T23:27:49.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.076+0000 7f2770c00640 1 -- 172.21.15.6:0/54669526 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f276403b070 con 0x7f2774071ed0 2024-04-24T23:27:49.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.077+0000 7f2770c00640 1 --2- 172.21.15.6:0/54669526 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f275006be30 0x7f275006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:49.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.077+0000 7f2770c00640 1 -- 172.21.15.6:0/54669526 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6934+0+0 (secure 0 0 0) 0x7f27640adfe0 con 0x7f2774071ed0 2024-04-24T23:27:49.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.077+0000 7f2770c00640 1 --2- 172.21.15.6:0/54669526 >> [v2:172.21.15.190:6808/1989500934,v1:172.21.15.190:6809/1989500934] conn(0x7f27500758d0 0x7f2750077d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:49.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.077+0000 7f2770c00640 1 -- 172.21.15.6:0/54669526 --> [v2:172.21.15.190:6808/1989500934,v1:172.21.15.190:6809/1989500934] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2764044ec0 con 0x7f27500758d0 2024-04-24T23:27:49.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.077+0000 7f2770c00640 1 -- 172.21.15.6:0/54669526 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f27640b05c0 con 0x7f2774071ed0 2024-04-24T23:27:49.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.077+0000 7f2773e00640 1 --2- 172.21.15.6:0/54669526 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f275006be30 0x7f275006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:49.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.077+0000 7f2779c00640 1 --2- 172.21.15.6:0/54669526 >> [v2:172.21.15.190:6808/1989500934,v1:172.21.15.190:6809/1989500934] conn(0x7f27500758d0 0x7f2750077d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:49.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.078+0000 7f2773e00640 1 --2- 172.21.15.6:0/54669526 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f275006be30 0x7f275006e2f0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f27741517f0 tx=0x7f275c007ef0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:27:49.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.078+0000 7f2779c00640 1 --2- 172.21.15.6:0/54669526 >> [v2:172.21.15.190:6808/1989500934,v1:172.21.15.190:6809/1989500934] conn(0x7f27500758d0 0x7f2750077d50 crc :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:27:49.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.079+0000 7f2770c00640 1 -- 172.21.15.6:0/54669526 <== osd.4 v2:172.21.15.190:6808/1989500934 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f2764044ec0 con 0x7f27500758d0 2024-04-24T23:27:49.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.093+0000 7f277bc00640 1 -- 172.21.15.6:0/54669526 --> [v2:172.21.15.190:6808/1989500934,v1:172.21.15.190:6809/1989500934] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f277414dcf0 con 0x7f27500758d0 2024-04-24T23:27:49.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.093+0000 7f2770c00640 1 -- 172.21.15.6:0/54669526 <== osd.4 v2:172.21.15.190:6808/1989500934 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f277414dcf0 con 0x7f27500758d0 2024-04-24T23:27:49.094 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-04-24T23:27:49.094 INFO:teuthology.orchestra.run.smithi006.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-24T23:27:49.095 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-04-24T23:27:49.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.093+0000 7f277bc00640 1 -- 172.21.15.6:0/54669526 >> [v2:172.21.15.190:6808/1989500934,v1:172.21.15.190:6809/1989500934] conn(0x7f27500758d0 msgr2=0x7f2750077d50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:49.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.094+0000 7f277bc00640 1 --2- 172.21.15.6:0/54669526 >> [v2:172.21.15.190:6808/1989500934,v1:172.21.15.190:6809/1989500934] conn(0x7f27500758d0 0x7f2750077d50 crc :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:49.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.094+0000 7f277bc00640 1 -- 172.21.15.6:0/54669526 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f275006be30 msgr2=0x7f275006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:49.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.094+0000 7f277bc00640 1 --2- 172.21.15.6:0/54669526 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f275006be30 0x7f275006e2f0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f27741517f0 tx=0x7f275c007ef0 comp rx=0 tx=0).stop 2024-04-24T23:27:49.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.094+0000 7f277bc00640 1 -- 172.21.15.6:0/54669526 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2774071ed0 msgr2=0x7f277414e3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:49.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.094+0000 7f277bc00640 1 --2- 172.21.15.6:0/54669526 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2774071ed0 0x7f277414e3b0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f2764033000 tx=0x7f27640046c0 comp rx=0 tx=0).stop 2024-04-24T23:27:49.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.094+0000 7f2779c00640 1 -- 172.21.15.6:0/54669526 reap_dead start 2024-04-24T23:27:49.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.094+0000 7f277bc00640 1 -- 172.21.15.6:0/54669526 shutdown_connections 2024-04-24T23:27:49.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.094+0000 7f277bc00640 1 -- 172.21.15.6:0/54669526 >> 172.21.15.6:0/54669526 conn(0x7f277406b0e0 msgr2=0x7f277406ea80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:27:49.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.094+0000 7f277bc00640 1 -- 172.21.15.6:0/54669526 shutdown_connections 2024-04-24T23:27:49.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.094+0000 7f277bc00640 1 -- 172.21.15.6:0/54669526 wait complete. 2024-04-24T23:27:49.112 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-04-24T23:27:49.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.367+0000 7fbfd3200640 1 Processor -- start 2024-04-24T23:27:49.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.368+0000 7fbfd3200640 1 -- start start 2024-04-24T23:27:49.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.370+0000 7fbfd3200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fbfc40dac90 0x7fbfc404b910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:49.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.370+0000 7fbfcbe00640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7fbfc404b530 0x7fbfc4005f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.6:38468/0 (socket says 172.21.15.6:38468) 2024-04-24T23:27:49.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.370+0000 7fbfd3200640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fbfc4034a70 0x7fbfc4034ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:49.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.370+0000 7fbfcbe00640 1 -- 172.21.15.6:0/2240375436 learned_addr learned my addr 172.21.15.6:0/2240375436 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:27:49.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.371+0000 7fbfcb400640 1 --2- 172.21.15.6:0/2240375436 >> v2:172.21.15.6:3300/0 conn(0x7fbfc40dac90 0x7fbfc404b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:49.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.371+0000 7fbfd3200640 1 -- 172.21.15.6:0/2240375436 --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbfc4007790 con 0x7fbfc404b530 2024-04-24T23:27:49.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.371+0000 7fbfd3200640 1 -- 172.21.15.6:0/2240375436 --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7fbfc40353e0 con 0x7fbfc40dac90 2024-04-24T23:27:49.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.371+0000 7fbfd3200640 1 -- 172.21.15.6:0/2240375436 --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7fbfc4006930 con 0x7fbfc4034a70 2024-04-24T23:27:49.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.371+0000 7fbfcb400640 1 -- 172.21.15.6:0/2240375436 >> v2:172.21.15.190:3300/0 conn(0x7fbfc4034a70 msgr2=0x7fbfc4034ea0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-24T23:27:49.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.372+0000 7fbfcb400640 1 --2- 172.21.15.6:0/2240375436 >> v2:172.21.15.190:3300/0 conn(0x7fbfc4034a70 0x7fbfc4034ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:49.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.372+0000 7fbfcb400640 1 -- 172.21.15.6:0/2240375436 >> v1:172.21.15.190:6789/0 conn(0x7fbfc404b530 legacy=0x7fbfc4005f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:49.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.372+0000 7fbfcb400640 1 -- 172.21.15.6:0/2240375436 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfc4006c10 con 0x7fbfc40dac90 2024-04-24T23:27:49.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.376+0000 7fbfcb400640 1 --2- 172.21.15.6:0/2240375436 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbfc40dac90 0x7fbfc404b910 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fbfc404a7a0 tx=0x7fbfb802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58be927bac0d2d79 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:27:49.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.376+0000 7fbfcaa00640 1 -- 172.21.15.6:0/2240375436 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfb8037070 con 0x7fbfc40dac90 2024-04-24T23:27:49.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.377+0000 7fbfcaa00640 1 -- 172.21.15.6:0/2240375436 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbfb8032440 con 0x7fbfc40dac90 2024-04-24T23:27:49.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.377+0000 7fbfcaa00640 1 -- 172.21.15.6:0/2240375436 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfb8032810 con 0x7fbfc40dac90 2024-04-24T23:27:49.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.378+0000 7fbfd3200640 1 -- 172.21.15.6:0/2240375436 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbfc40dac90 msgr2=0x7fbfc404b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:49.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.378+0000 7fbfd3200640 1 --2- 172.21.15.6:0/2240375436 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbfc40dac90 0x7fbfc404b910 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fbfc404a7a0 tx=0x7fbfb802a270 comp rx=0 tx=0).stop 2024-04-24T23:27:49.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.378+0000 7fbfd3200640 1 -- 172.21.15.6:0/2240375436 shutdown_connections 2024-04-24T23:27:49.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.378+0000 7fbfd3200640 1 --2- 172.21.15.6:0/2240375436 >> v2:172.21.15.190:3300/0 conn(0x7fbfc4034a70 0x7fbfc4034ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:49.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.378+0000 7fbfd3200640 1 --2- 172.21.15.6:0/2240375436 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbfc40dac90 0x7fbfc404b910 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:49.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.378+0000 7fbfd3200640 1 -- 172.21.15.6:0/2240375436 >> 172.21.15.6:0/2240375436 conn(0x7fbfc40501b0 msgr2=0x7fbfc40505c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:27:49.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.379+0000 7fbfd3200640 1 -- 172.21.15.6:0/2240375436 shutdown_connections 2024-04-24T23:27:49.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.379+0000 7fbfd3200640 1 -- 172.21.15.6:0/2240375436 wait complete. 2024-04-24T23:27:49.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.381+0000 7fbfd3200640 1 Processor -- start 2024-04-24T23:27:49.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.382+0000 7fbfd3200640 1 -- start start 2024-04-24T23:27:49.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.383+0000 7fbfd3200640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbfc4034a70 0x7fbfc40c2f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:49.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.383+0000 7fbfd3200640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbfc404b530 0x7fbfc4179ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:49.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.383+0000 7fbfd3200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbfc404e2c0 0x7fbfc417a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:49.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.384+0000 7fbfd1200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbfc404e2c0 0x7fbfc417a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:49.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.385+0000 7fbfcbe00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbfc4034a70 0x7fbfc40c2f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:49.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.385+0000 7fbfd1200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbfc404e2c0 0x7fbfc417a0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:58080/0 (socket says 172.21.15.6:58080) 2024-04-24T23:27:49.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.385+0000 7fbfcbe00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbfc4034a70 0x7fbfc40c2f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:53364/0 (socket says 172.21.15.6:53364) 2024-04-24T23:27:49.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.385+0000 7fbfd1200640 1 -- 172.21.15.6:0/3485336138 learned_addr learned my addr 172.21.15.6:0/3485336138 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:27:49.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.385+0000 7fbfd3200640 1 -- 172.21.15.6:0/3485336138 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fbfc40074b0 con 0x7fbfc404e2c0 2024-04-24T23:27:49.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.385+0000 7fbfd3200640 1 -- 172.21.15.6:0/3485336138 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fbfc40071d0 con 0x7fbfc4034a70 2024-04-24T23:27:49.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.385+0000 7fbfd3200640 1 -- 172.21.15.6:0/3485336138 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fbfc4006620 con 0x7fbfc404b530 2024-04-24T23:27:49.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.386+0000 7fbfcb400640 1 --2- 172.21.15.6:0/3485336138 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbfc404b530 0x7fbfc4179ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:49.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.387+0000 7fbfd1200640 1 -- 172.21.15.6:0/3485336138 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbfc404b530 msgr2=0x7fbfc4179ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:49.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.387+0000 7fbfd1200640 1 --2- 172.21.15.6:0/3485336138 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbfc404b530 0x7fbfc4179ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:49.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.387+0000 7fbfd1200640 1 -- 172.21.15.6:0/3485336138 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbfc4034a70 msgr2=0x7fbfc40c2f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:49.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.387+0000 7fbfd1200640 1 --2- 172.21.15.6:0/3485336138 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbfc4034a70 0x7fbfc40c2f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:49.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.387+0000 7fbfd1200640 1 -- 172.21.15.6:0/3485336138 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfc40e5040 con 0x7fbfc404e2c0 2024-04-24T23:27:49.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.387+0000 7fbfcbe00640 1 --2- 172.21.15.6:0/3485336138 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbfc4034a70 0x7fbfc40c2f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:27:49.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.388+0000 7fbfcb400640 1 --2- 172.21.15.6:0/3485336138 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbfc404b530 0x7fbfc4179ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:27:49.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.388+0000 7fbfd1200640 1 --2- 172.21.15.6:0/3485336138 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbfc404e2c0 0x7fbfc417a0e0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fbfcc06a6e0 tx=0x7fbfcc0737c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:27:49.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.389+0000 7fbfbfe00640 1 -- 172.21.15.6:0/3485336138 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfcc076b20 con 0x7fbfc404e2c0 2024-04-24T23:27:49.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.389+0000 7fbfd3200640 1 -- 172.21.15.6:0/3485336138 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbfc404fbd0 con 0x7fbfc404e2c0 2024-04-24T23:27:49.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.389+0000 7fbfd3200640 1 -- 172.21.15.6:0/3485336138 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbfc40e4d60 con 0x7fbfc404e2c0 2024-04-24T23:27:49.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.389+0000 7fbfbfe00640 1 -- 172.21.15.6:0/3485336138 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbfcc090030 con 0x7fbfc404e2c0 2024-04-24T23:27:49.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.389+0000 7fbfbfe00640 1 -- 172.21.15.6:0/3485336138 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfcc076b20 con 0x7fbfc404e2c0 2024-04-24T23:27:49.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.391+0000 7fbfbd600640 1 -- 172.21.15.6:0/3485336138 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbfa8005290 con 0x7fbfc404e2c0 2024-04-24T23:27:49.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.393+0000 7fbfbfe00640 1 -- 172.21.15.6:0/3485336138 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fbfcc068b40 con 0x7fbfc404e2c0 2024-04-24T23:27:49.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.398+0000 7fbfbfe00640 1 --2- 172.21.15.6:0/3485336138 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fbfac06c050 0x7fbfac06e510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:49.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.398+0000 7fbfbfe00640 1 -- 172.21.15.6:0/3485336138 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6934+0+0 (secure 0 0 0) 0x7fbfcc107840 con 0x7fbfc404e2c0 2024-04-24T23:27:49.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.404+0000 7fbfcbe00640 1 --2- 172.21.15.6:0/3485336138 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fbfac06c050 0x7fbfac06e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:49.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.410+0000 7fbfbfe00640 1 -- 172.21.15.6:0/3485336138 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fbfcc0d5c90 con 0x7fbfc404e2c0 2024-04-24T23:27:49.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.410+0000 7fbfcbe00640 1 --2- 172.21.15.6:0/3485336138 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fbfac06c050 0x7fbfac06e510 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fbfc0000f60 tx=0x7fbfc0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:27:49.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.843+0000 7fbfbd600640 1 -- 172.21.15.6:0/3485336138 --> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) -- 0x7fbfa8000bc0 con 0x7fbfac06c050 2024-04-24T23:27:49.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.850+0000 7fbfbfe00640 1 -- 172.21.15.6:0/3485336138 <== mgr.4105 v2:172.21.15.190:6832/806769663 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fbfa8000bc0 con 0x7fbfac06c050 2024-04-24T23:27:49.851 INFO:teuthology.orchestra.run.smithi006.stderr:instructed osd(s) 4 to deep-scrub 2024-04-24T23:27:49.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.853+0000 7fbfbd600640 1 -- 172.21.15.6:0/3485336138 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fbfac06c050 msgr2=0x7fbfac06e510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:49.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.853+0000 7fbfbd600640 1 --2- 172.21.15.6:0/3485336138 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fbfac06c050 0x7fbfac06e510 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fbfc0000f60 tx=0x7fbfc0009040 comp rx=0 tx=0).stop 2024-04-24T23:27:49.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.853+0000 7fbfbd600640 1 -- 172.21.15.6:0/3485336138 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbfc404e2c0 msgr2=0x7fbfc417a0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:49.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.853+0000 7fbfbd600640 1 --2- 172.21.15.6:0/3485336138 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbfc404e2c0 0x7fbfc417a0e0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fbfcc06a6e0 tx=0x7fbfcc0737c0 comp rx=0 tx=0).stop 2024-04-24T23:27:49.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.854+0000 7fbfbd600640 1 -- 172.21.15.6:0/3485336138 shutdown_connections 2024-04-24T23:27:49.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.854+0000 7fbfbd600640 1 --2- 172.21.15.6:0/3485336138 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fbfac06c050 0x7fbfac06e510 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:49.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.854+0000 7fbfbd600640 1 --2- 172.21.15.6:0/3485336138 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbfc404b530 0x7fbfc4179ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:49.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.854+0000 7fbfbd600640 1 --2- 172.21.15.6:0/3485336138 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbfc404e2c0 0x7fbfc417a0e0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:49.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.854+0000 7fbfbd600640 1 --2- 172.21.15.6:0/3485336138 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbfc4034a70 0x7fbfc40c2f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:49.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.854+0000 7fbfbd600640 1 -- 172.21.15.6:0/3485336138 >> 172.21.15.6:0/3485336138 conn(0x7fbfc40501b0 msgr2=0x7fbfc40e6dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:27:49.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.855+0000 7fbfbd600640 1 -- 172.21.15.6:0/3485336138 shutdown_connections 2024-04-24T23:27:49.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:49.855+0000 7fbfbd600640 1 -- 172.21.15.6:0/3485336138 wait complete. 2024-04-24T23:27:49.894 INFO:tasks.ceph:Scrubbing osd.5 2024-04-24T23:27:49.895 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-04-24T23:27:50.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.102+0000 7f095da00640 1 Processor -- start 2024-04-24T23:27:50.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.103+0000 7f095da00640 1 -- start start 2024-04-24T23:27:50.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.104+0000 7f095da00640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f095815bff0 0x7f09580cb6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:50.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.104+0000 7f095da00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f095806c9a0 0x7f095806bd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:50.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.104+0000 7f095da00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f095815b350 con 0x7f09580ca2d0 2024-04-24T23:27:50.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.104+0000 7f095da00640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f09580cac90 con 0x7f095815bff0 2024-04-24T23:27:50.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.104+0000 7f095da00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f09580cbc20 con 0x7f095806c9a0 2024-04-24T23:27:50.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.104+0000 7f0957e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f09580ca2d0 0x7f095815ab80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:34890/0 (socket says 172.21.15.6:34890) 2024-04-24T23:27:50.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.104+0000 7f0957e00640 1 -- 172.21.15.6:0/4070371058 learned_addr learned my addr 172.21.15.6:0/4070371058 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:27:50.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.104+0000 7f095d000640 1 --2- 172.21.15.6:0/4070371058 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f095806c9a0 0x7f095806bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:50.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.104+0000 7f0957400640 1 --2- 172.21.15.6:0/4070371058 >> v2:172.21.15.190:3300/0 conn(0x7f095815bff0 0x7f09580cb6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:50.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.105+0000 7f095d000640 1 -- 172.21.15.6:0/4070371058 >> v2:172.21.15.190:3300/0 conn(0x7f095815bff0 msgr2=0x7f09580cb6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:27:50.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.105+0000 7f0956a00640 1 -- 172.21.15.6:0/4070371058 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1891834233 0 0) 0x7f095815b350 con 0x7f09580ca2d0 2024-04-24T23:27:50.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.105+0000 7f095d000640 1 --2- 172.21.15.6:0/4070371058 >> v2:172.21.15.190:3300/0 conn(0x7f095815bff0 0x7f09580cb6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:50.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.105+0000 7f095d000640 1 -- 172.21.15.6:0/4070371058 >> v1:172.21.15.6:6789/0 conn(0x7f09580ca2d0 legacy=0x7f095815ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:50.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.105+0000 7f095d000640 1 -- 172.21.15.6:0/4070371058 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f095815b630 con 0x7f095806c9a0 2024-04-24T23:27:50.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.105+0000 7f0957400640 1 --2- 172.21.15.6:0/4070371058 >> v2:172.21.15.190:3300/0 conn(0x7f095815bff0 0x7f09580cb6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:27:50.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.106+0000 7f095d000640 1 --2- 172.21.15.6:0/4070371058 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f095806c9a0 0x7f095806bd80 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f0950009d90 tx=0x7f095002a660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=75819e54cb7f0459 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:27:50.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.107+0000 7f0956a00640 1 -- 172.21.15.6:0/4070371058 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0950007740 con 0x7f095806c9a0 2024-04-24T23:27:50.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.107+0000 7f0956a00640 1 -- 172.21.15.6:0/4070371058 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f09500078e0 con 0x7f095806c9a0 2024-04-24T23:27:50.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.107+0000 7f0956a00640 1 -- 172.21.15.6:0/4070371058 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0950007cb0 con 0x7f095806c9a0 2024-04-24T23:27:50.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.107+0000 7f095da00640 1 -- 172.21.15.6:0/4070371058 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f095806c9a0 msgr2=0x7f095806bd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:50.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.107+0000 7f095da00640 1 --2- 172.21.15.6:0/4070371058 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f095806c9a0 0x7f095806bd80 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f0950009d90 tx=0x7f095002a660 comp rx=0 tx=0).stop 2024-04-24T23:27:50.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.107+0000 7f095da00640 1 -- 172.21.15.6:0/4070371058 shutdown_connections 2024-04-24T23:27:50.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.107+0000 7f095da00640 1 --2- 172.21.15.6:0/4070371058 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f095806c9a0 0x7f095806bd80 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:50.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.107+0000 7f095da00640 1 --2- 172.21.15.6:0/4070371058 >> v2:172.21.15.190:3300/0 conn(0x7f095815bff0 0x7f09580cb6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:50.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.107+0000 7f095da00640 1 -- 172.21.15.6:0/4070371058 >> 172.21.15.6:0/4070371058 conn(0x7f09580674d0 msgr2=0x7f09580d0310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:27:50.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.107+0000 7f095da00640 1 -- 172.21.15.6:0/4070371058 shutdown_connections 2024-04-24T23:27:50.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.107+0000 7f095da00640 1 -- 172.21.15.6:0/4070371058 wait complete. 2024-04-24T23:27:50.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.109+0000 7f095da00640 1 Processor -- start 2024-04-24T23:27:50.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.109+0000 7f095da00640 1 -- start start 2024-04-24T23:27:50.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.110+0000 7f095da00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f095806c9a0 0x7f095815a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:50.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.110+0000 7f095da00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f09580ca2d0 0x7f09580c5430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:50.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.110+0000 7f095da00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f095815bff0 0x7f09580c3f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:50.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.110+0000 7f095da00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f09580cb1c0 con 0x7f095806c9a0 2024-04-24T23:27:50.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.110+0000 7f095da00640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f09580caee0 con 0x7f09580ca2d0 2024-04-24T23:27:50.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.110+0000 7f095da00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f09580ca6b0 con 0x7f095815bff0 2024-04-24T23:27:50.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.110+0000 7f0957e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f095806c9a0 0x7f095815a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:50.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.110+0000 7f0957e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f095806c9a0 0x7f095815a0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:58084/0 (socket says 172.21.15.6:58084) 2024-04-24T23:27:50.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.110+0000 7f0957e00640 1 -- 172.21.15.6:0/1478726545 learned_addr learned my addr 172.21.15.6:0/1478726545 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:27:50.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.110+0000 7f0957400640 1 --2- 172.21.15.6:0/1478726545 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f09580ca2d0 0x7f09580c5430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:50.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.111+0000 7f095d000640 1 --2- 172.21.15.6:0/1478726545 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f095815bff0 0x7f09580c3f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:50.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.111+0000 7f0957400640 1 -- 172.21.15.6:0/1478726545 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f095815bff0 msgr2=0x7f09580c3f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:50.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.111+0000 7f0957400640 1 --2- 172.21.15.6:0/1478726545 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f095815bff0 0x7f09580c3f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:50.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.111+0000 7f0957400640 1 -- 172.21.15.6:0/1478726545 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f095806c9a0 msgr2=0x7f095815a0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:50.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.111+0000 7f0957400640 1 --2- 172.21.15.6:0/1478726545 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f095806c9a0 0x7f095815a0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:50.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.111+0000 7f0957400640 1 -- 172.21.15.6:0/1478726545 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09580c64a0 con 0x7f09580ca2d0 2024-04-24T23:27:50.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.111+0000 7f0957400640 1 --2- 172.21.15.6:0/1478726545 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f09580ca2d0 0x7f09580c5430 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f094400e960 tx=0x7f094400ee20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:27:50.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.111+0000 7f094fe00640 1 -- 172.21.15.6:0/1478726545 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0944017070 con 0x7f09580ca2d0 2024-04-24T23:27:50.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.112+0000 7f094fe00640 1 -- 172.21.15.6:0/1478726545 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f094400ca00 con 0x7f09580ca2d0 2024-04-24T23:27:50.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.112+0000 7f094fe00640 1 -- 172.21.15.6:0/1478726545 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f094400cdf0 con 0x7f09580ca2d0 2024-04-24T23:27:50.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.112+0000 7f095da00640 1 -- 172.21.15.6:0/1478726545 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09580c6780 con 0x7f09580ca2d0 2024-04-24T23:27:50.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.112+0000 7f095da00640 1 -- 172.21.15.6:0/1478726545 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09580c6d40 con 0x7f09580ca2d0 2024-04-24T23:27:50.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.113+0000 7f095da00640 1 -- 172.21.15.6:0/1478726545 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f0958075b30 con 0x7f09580ca2d0 2024-04-24T23:27:50.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.114+0000 7f094fe00640 1 -- 172.21.15.6:0/1478726545 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f094401a070 con 0x7f09580ca2d0 2024-04-24T23:27:50.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.115+0000 7f094fe00640 1 --2- 172.21.15.6:0/1478726545 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f093806be30 0x7f093806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:50.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.115+0000 7f094fe00640 1 -- 172.21.15.6:0/1478726545 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6934+0+0 (secure 0 0 0) 0x7f094408b4c0 con 0x7f09580ca2d0 2024-04-24T23:27:50.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.116+0000 7f0957e00640 1 --2- 172.21.15.6:0/1478726545 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f093806be30 0x7f093806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:50.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.116+0000 7f094fe00640 1 --2- 172.21.15.6:0/1478726545 >> [v2:172.21.15.190:6800/3687548412,v1:172.21.15.190:6801/3687548412] conn(0x7f09380758e0 0x7f0938077d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:50.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.116+0000 7f094fe00640 1 -- 172.21.15.6:0/1478726545 --> [v2:172.21.15.190:6800/3687548412,v1:172.21.15.190:6801/3687548412] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0938078430 con 0x7f09380758e0 2024-04-24T23:27:50.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.116+0000 7f094fe00640 1 -- 172.21.15.6:0/1478726545 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f0944092050 con 0x7f09580ca2d0 2024-04-24T23:27:50.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.116+0000 7f095d000640 1 --2- 172.21.15.6:0/1478726545 >> [v2:172.21.15.190:6800/3687548412,v1:172.21.15.190:6801/3687548412] conn(0x7f09380758e0 0x7f0938077d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:50.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.116+0000 7f0957e00640 1 --2- 172.21.15.6:0/1478726545 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f093806be30 0x7f093806e2f0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f0948000f60 tx=0x7f0948007e10 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:27:50.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.117+0000 7f095d000640 1 --2- 172.21.15.6:0/1478726545 >> [v2:172.21.15.190:6800/3687548412,v1:172.21.15.190:6801/3687548412] conn(0x7f09380758e0 0x7f0938077d60 crc :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:27:50.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.117+0000 7f094fe00640 1 -- 172.21.15.6:0/1478726545 <== osd.5 v2:172.21.15.190:6800/3687548412 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f0938078430 con 0x7f09380758e0 2024-04-24T23:27:50.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.131+0000 7f095da00640 1 -- 172.21.15.6:0/1478726545 --> [v2:172.21.15.190:6800/3687548412,v1:172.21.15.190:6801/3687548412] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f09580cbde0 con 0x7f09380758e0 2024-04-24T23:27:50.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.132+0000 7f094fe00640 1 -- 172.21.15.6:0/1478726545 <== osd.5 v2:172.21.15.190:6800/3687548412 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f09580cbde0 con 0x7f09380758e0 2024-04-24T23:27:50.133 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-04-24T23:27:50.133 INFO:teuthology.orchestra.run.smithi006.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-24T23:27:50.134 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-04-24T23:27:50.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.132+0000 7f095da00640 1 -- 172.21.15.6:0/1478726545 >> [v2:172.21.15.190:6800/3687548412,v1:172.21.15.190:6801/3687548412] conn(0x7f09380758e0 msgr2=0x7f0938077d60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:50.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.132+0000 7f095da00640 1 --2- 172.21.15.6:0/1478726545 >> [v2:172.21.15.190:6800/3687548412,v1:172.21.15.190:6801/3687548412] conn(0x7f09380758e0 0x7f0938077d60 crc :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:50.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.132+0000 7f095da00640 1 -- 172.21.15.6:0/1478726545 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f093806be30 msgr2=0x7f093806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:50.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.132+0000 7f095da00640 1 --2- 172.21.15.6:0/1478726545 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f093806be30 0x7f093806e2f0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f0948000f60 tx=0x7f0948007e10 comp rx=0 tx=0).stop 2024-04-24T23:27:50.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.132+0000 7f095da00640 1 -- 172.21.15.6:0/1478726545 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f09580ca2d0 msgr2=0x7f09580c5430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:50.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.132+0000 7f095da00640 1 --2- 172.21.15.6:0/1478726545 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f09580ca2d0 0x7f09580c5430 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f094400e960 tx=0x7f094400ee20 comp rx=0 tx=0).stop 2024-04-24T23:27:50.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.133+0000 7f095d000640 1 -- 172.21.15.6:0/1478726545 reap_dead start 2024-04-24T23:27:50.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.133+0000 7f095da00640 1 -- 172.21.15.6:0/1478726545 shutdown_connections 2024-04-24T23:27:50.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.133+0000 7f095da00640 1 -- 172.21.15.6:0/1478726545 >> 172.21.15.6:0/1478726545 conn(0x7f09580674d0 msgr2=0x7f09580c9150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:27:50.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.133+0000 7f095da00640 1 -- 172.21.15.6:0/1478726545 shutdown_connections 2024-04-24T23:27:50.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.133+0000 7f095da00640 1 -- 172.21.15.6:0/1478726545 wait complete. 2024-04-24T23:27:50.151 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-04-24T23:27:50.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.327+0000 7f50e2600640 1 Processor -- start 2024-04-24T23:27:50.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.327+0000 7f50e2600640 1 -- start start 2024-04-24T23:27:50.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.328+0000 7f50e2600640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f50dc06d6d0 0x7f50dc0c4060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:50.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.328+0000 7f50e2600640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f50dc0c45a0 0x7f50dc163320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:50.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.328+0000 7f50e2600640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f50dc153120 con 0x7f50dc152d40 2024-04-24T23:27:50.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.328+0000 7f50e2600640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f50dc1599c0 con 0x7f50dc06d6d0 2024-04-24T23:27:50.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.328+0000 7f50e2600640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f50dc0c8aa0 con 0x7f50dc0c45a0 2024-04-24T23:27:50.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.328+0000 7f50d3e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f50dc06d6d0 0x7f50dc0c4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:50.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.328+0000 7f50db400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f50dc152d40 0x7f50dc1592b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:34892/0 (socket says 172.21.15.6:34892) 2024-04-24T23:27:50.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.328+0000 7f50db400640 1 -- 172.21.15.6:0/2883236121 learned_addr learned my addr 172.21.15.6:0/2883236121 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:27:50.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.328+0000 7f50dbe00640 1 --2- 172.21.15.6:0/2883236121 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f50dc0c45a0 0x7f50dc163320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:50.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.329+0000 7f50dbe00640 1 -- 172.21.15.6:0/2883236121 >> v2:172.21.15.6:3300/0 conn(0x7f50dc06d6d0 msgr2=0x7f50dc0c4060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:27:50.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.329+0000 7f50dbe00640 1 --2- 172.21.15.6:0/2883236121 >> v2:172.21.15.6:3300/0 conn(0x7f50dc06d6d0 0x7f50dc0c4060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:50.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.329+0000 7f50dbe00640 1 -- 172.21.15.6:0/2883236121 >> v1:172.21.15.6:6789/0 conn(0x7f50dc152d40 legacy=0x7f50dc1592b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:50.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.329+0000 7f50dbe00640 1 -- 172.21.15.6:0/2883236121 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50dc06d230 con 0x7f50dc0c45a0 2024-04-24T23:27:50.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.330+0000 7f50dbe00640 1 --2- 172.21.15.6:0/2883236121 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f50dc0c45a0 0x7f50dc163320 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f50d4009d90 tx=0x7f50d402a6f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=95b546580967c28b server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:27:50.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.330+0000 7f50daa00640 1 -- 172.21.15.6:0/2883236121 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50d4007740 con 0x7f50dc0c45a0 2024-04-24T23:27:50.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.330+0000 7f50daa00640 1 -- 172.21.15.6:0/2883236121 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f50d40078e0 con 0x7f50dc0c45a0 2024-04-24T23:27:50.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.330+0000 7f50daa00640 1 -- 172.21.15.6:0/2883236121 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50d4007cb0 con 0x7f50dc0c45a0 2024-04-24T23:27:50.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.331+0000 7f50e2600640 1 -- 172.21.15.6:0/2883236121 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f50dc0c45a0 msgr2=0x7f50dc163320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:50.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.331+0000 7f50e2600640 1 --2- 172.21.15.6:0/2883236121 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f50dc0c45a0 0x7f50dc163320 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f50d4009d90 tx=0x7f50d402a6f0 comp rx=0 tx=0).stop 2024-04-24T23:27:50.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.331+0000 7f50e2600640 1 -- 172.21.15.6:0/2883236121 shutdown_connections 2024-04-24T23:27:50.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.331+0000 7f50e2600640 1 --2- 172.21.15.6:0/2883236121 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f50dc0c45a0 0x7f50dc163320 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:50.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.331+0000 7f50e2600640 1 --2- 172.21.15.6:0/2883236121 >> v2:172.21.15.6:3300/0 conn(0x7f50dc06d6d0 0x7f50dc0c4060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:50.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.331+0000 7f50e2600640 1 -- 172.21.15.6:0/2883236121 >> 172.21.15.6:0/2883236121 conn(0x7f50dc0c8c50 msgr2=0x7f50dc0c9060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:27:50.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.331+0000 7f50e2600640 1 -- 172.21.15.6:0/2883236121 shutdown_connections 2024-04-24T23:27:50.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.331+0000 7f50e2600640 1 -- 172.21.15.6:0/2883236121 wait complete. 2024-04-24T23:27:50.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.334+0000 7f50e2600640 1 Processor -- start 2024-04-24T23:27:50.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.334+0000 7f50e2600640 1 -- start start 2024-04-24T23:27:50.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.335+0000 7f50e2600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f50dc06d6d0 0x7f50dc145470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:50.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.335+0000 7f50e2600640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f50dc0c45a0 0x7f50dc1430c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:50.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.335+0000 7f50e2600640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f50dc152d40 0x7f50dc143600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:50.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.335+0000 7f50e2600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f50dc158ab0 con 0x7f50dc06d6d0 2024-04-24T23:27:50.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.335+0000 7f50e2600640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f50dc06cf50 con 0x7f50dc0c45a0 2024-04-24T23:27:50.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.335+0000 7f50e2600640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f50dc06f8d0 con 0x7f50dc152d40 2024-04-24T23:27:50.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.336+0000 7f50db400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f50dc06d6d0 0x7f50dc145470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:50.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.336+0000 7f50db400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f50dc06d6d0 0x7f50dc145470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:58112/0 (socket says 172.21.15.6:58112) 2024-04-24T23:27:50.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.336+0000 7f50d3e00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f50dc0c45a0 0x7f50dc1430c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:50.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.336+0000 7f50db400640 1 -- 172.21.15.6:0/3996261478 learned_addr learned my addr 172.21.15.6:0/3996261478 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:27:50.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.336+0000 7f50dbe00640 1 --2- 172.21.15.6:0/3996261478 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f50dc152d40 0x7f50dc143600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:50.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.336+0000 7f50db400640 1 -- 172.21.15.6:0/3996261478 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f50dc152d40 msgr2=0x7f50dc143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:50.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.336+0000 7f50db400640 1 --2- 172.21.15.6:0/3996261478 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f50dc152d40 0x7f50dc143600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:50.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.336+0000 7f50db400640 1 -- 172.21.15.6:0/3996261478 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f50dc0c45a0 msgr2=0x7f50dc1430c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:50.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.336+0000 7f50db400640 1 --2- 172.21.15.6:0/3996261478 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f50dc0c45a0 0x7f50dc1430c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:50.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.336+0000 7f50db400640 1 -- 172.21.15.6:0/3996261478 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50dc1f1dd0 con 0x7f50dc06d6d0 2024-04-24T23:27:50.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.336+0000 7f50d3e00640 1 --2- 172.21.15.6:0/3996261478 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f50dc0c45a0 0x7f50dc1430c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:27:50.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.337+0000 7f50db400640 1 --2- 172.21.15.6:0/3996261478 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f50dc06d6d0 0x7f50dc145470 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f50cc00c3f0 tx=0x7f50cc00c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:27:50.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.337+0000 7f50d3400640 1 -- 172.21.15.6:0/3996261478 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50cc017070 con 0x7f50dc06d6d0 2024-04-24T23:27:50.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.337+0000 7f50d3400640 1 -- 172.21.15.6:0/3996261478 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f50cc005ce0 con 0x7f50dc06d6d0 2024-04-24T23:27:50.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.337+0000 7f50e2600640 1 -- 172.21.15.6:0/3996261478 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50dc1f20b0 con 0x7f50dc06d6d0 2024-04-24T23:27:50.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.337+0000 7f50d3400640 1 -- 172.21.15.6:0/3996261478 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50cc0072f0 con 0x7f50dc06d6d0 2024-04-24T23:27:50.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.337+0000 7f50e2600640 1 -- 172.21.15.6:0/3996261478 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50dc1f2670 con 0x7f50dc06d6d0 2024-04-24T23:27:50.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.338+0000 7f50e2600640 1 -- 172.21.15.6:0/3996261478 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f50dc0767b0 con 0x7f50dc06d6d0 2024-04-24T23:27:50.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.340+0000 7f50d3400640 1 -- 172.21.15.6:0/3996261478 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f50cc01a070 con 0x7f50dc06d6d0 2024-04-24T23:27:50.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.342+0000 7f50d3400640 1 --2- 172.21.15.6:0/3996261478 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f50b806be30 0x7f50b806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:50.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.342+0000 7f50d3400640 1 -- 172.21.15.6:0/3996261478 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6934+0+0 (secure 0 0 0) 0x7f50cc08a680 con 0x7f50dc06d6d0 2024-04-24T23:27:50.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.342+0000 7f50d3e00640 1 --2- 172.21.15.6:0/3996261478 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f50b806be30 0x7f50b806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:50.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.343+0000 7f50d3e00640 1 --2- 172.21.15.6:0/3996261478 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f50b806be30 0x7f50b806e2f0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f50dc0cbdd0 tx=0x7f50c4012040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:27:50.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.346+0000 7f50d3400640 1 -- 172.21.15.6:0/3996261478 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f50cc058470 con 0x7f50dc06d6d0 2024-04-24T23:27:50.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.601+0000 7f50e2600640 1 -- 172.21.15.6:0/3996261478 --> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) -- 0x7f50dc1462c0 con 0x7f50b806be30 2024-04-24T23:27:50.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.605+0000 7f50d3400640 1 -- 172.21.15.6:0/3996261478 <== mgr.4105 v2:172.21.15.190:6832/806769663 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f50dc1462c0 con 0x7f50b806be30 2024-04-24T23:27:50.607 INFO:teuthology.orchestra.run.smithi006.stderr:instructed osd(s) 5 to deep-scrub 2024-04-24T23:27:50.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.610+0000 7f50e2600640 1 -- 172.21.15.6:0/3996261478 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f50b806be30 msgr2=0x7f50b806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:50.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.610+0000 7f50e2600640 1 --2- 172.21.15.6:0/3996261478 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f50b806be30 0x7f50b806e2f0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f50dc0cbdd0 tx=0x7f50c4012040 comp rx=0 tx=0).stop 2024-04-24T23:27:50.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.610+0000 7f50e2600640 1 -- 172.21.15.6:0/3996261478 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f50dc06d6d0 msgr2=0x7f50dc145470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:50.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.610+0000 7f50e2600640 1 --2- 172.21.15.6:0/3996261478 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f50dc06d6d0 0x7f50dc145470 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f50cc00c3f0 tx=0x7f50cc00c8b0 comp rx=0 tx=0).stop 2024-04-24T23:27:50.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.610+0000 7f50e2600640 1 -- 172.21.15.6:0/3996261478 shutdown_connections 2024-04-24T23:27:50.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.610+0000 7f50e2600640 1 --2- 172.21.15.6:0/3996261478 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f50b806be30 0x7f50b806e2f0 secure :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f50dc0cbdd0 tx=0x7f50c4012040 comp rx=0 tx=0).stop 2024-04-24T23:27:50.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.610+0000 7f50e2600640 1 --2- 172.21.15.6:0/3996261478 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f50dc152d40 0x7f50dc143600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:50.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.610+0000 7f50e2600640 1 --2- 172.21.15.6:0/3996261478 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f50dc0c45a0 0x7f50dc1430c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:50.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.610+0000 7f50e2600640 1 --2- 172.21.15.6:0/3996261478 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f50dc06d6d0 0x7f50dc145470 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:50.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.610+0000 7f50e2600640 1 -- 172.21.15.6:0/3996261478 >> 172.21.15.6:0/3996261478 conn(0x7f50dc0c8c50 msgr2=0x7f50dc156fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:27:50.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.614+0000 7f50e2600640 1 -- 172.21.15.6:0/3996261478 shutdown_connections 2024-04-24T23:27:50.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.614+0000 7f50e2600640 1 -- 172.21.15.6:0/3996261478 wait complete. 2024-04-24T23:27:50.643 INFO:tasks.ceph:Scrubbing osd.6 2024-04-24T23:27:50.643 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-04-24T23:27:50.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.955+0000 7fee81e00640 1 Processor -- start 2024-04-24T23:27:50.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.956+0000 7fee81e00640 1 -- start start 2024-04-24T23:27:50.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.958+0000 7fee81e00640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fee7c159050 0x7fee7c070e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:50.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.959+0000 7fee81e00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fee7c15f260 con 0x7fee7c159430 2024-04-24T23:27:50.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.959+0000 7fee81e00640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fee7c15f520 con 0x7fee7c06f150 2024-04-24T23:27:50.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.959+0000 7fee81e00640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7fee7c071390 con 0x7fee7c159050 2024-04-24T23:27:50.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.959+0000 7fee80a00640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fee7c159050 0x7fee7c070e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:50.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.959+0000 7fee80a00640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fee7c159050 0x7fee7c070e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:53394/0 (socket says 172.21.15.6:53394) 2024-04-24T23:27:50.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.959+0000 7fee80a00640 1 -- 172.21.15.6:0/4051911849 learned_addr learned my addr 172.21.15.6:0/4051911849 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:27:50.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.960+0000 7fee80a00640 1 -- 172.21.15.6:0/4051911849 >> v1:172.21.15.190:6789/0 conn(0x7fee7c06f150 legacy=0x7fee7c0c91a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:50.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.960+0000 7fee80a00640 1 -- 172.21.15.6:0/4051911849 >> v1:172.21.15.6:6789/0 conn(0x7fee7c159430 legacy=0x7fee7c0cc130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:50.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.960+0000 7fee80a00640 1 -- 172.21.15.6:0/4051911849 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee7c15f800 con 0x7fee7c159050 2024-04-24T23:27:50.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.962+0000 7fee80a00640 1 --2- 172.21.15.6:0/4051911849 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fee7c159050 0x7fee7c070e50 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fee70009870 tx=0x7fee7002a2a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=54f7d97be3f0af99 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:27:50.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.963+0000 7fee7b400640 1 -- 172.21.15.6:0/4051911849 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee70037070 con 0x7fee7c159050 2024-04-24T23:27:50.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.963+0000 7fee7b400640 1 -- 172.21.15.6:0/4051911849 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee70032440 con 0x7fee7c159050 2024-04-24T23:27:50.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.963+0000 7fee7b400640 1 -- 172.21.15.6:0/4051911849 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee70032810 con 0x7fee7c159050 2024-04-24T23:27:50.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.964+0000 7fee81e00640 1 -- 172.21.15.6:0/4051911849 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fee7c159050 msgr2=0x7fee7c070e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:50.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.964+0000 7fee81e00640 1 --2- 172.21.15.6:0/4051911849 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fee7c159050 0x7fee7c070e50 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fee70009870 tx=0x7fee7002a2a0 comp rx=0 tx=0).stop 2024-04-24T23:27:50.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.965+0000 7fee81e00640 1 -- 172.21.15.6:0/4051911849 shutdown_connections 2024-04-24T23:27:50.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.965+0000 7fee81e00640 1 --2- 172.21.15.6:0/4051911849 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fee7c159050 0x7fee7c070e50 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:50.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.965+0000 7fee81e00640 1 -- 172.21.15.6:0/4051911849 >> 172.21.15.6:0/4051911849 conn(0x7fee7c06cfe0 msgr2=0x7fee7c06ea40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:27:50.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.965+0000 7fee81e00640 1 -- 172.21.15.6:0/4051911849 shutdown_connections 2024-04-24T23:27:50.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.965+0000 7fee81e00640 1 -- 172.21.15.6:0/4051911849 wait complete. 2024-04-24T23:27:50.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.967+0000 7fee81e00640 1 Processor -- start 2024-04-24T23:27:50.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.967+0000 7fee81e00640 1 -- start start 2024-04-24T23:27:50.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.968+0000 7fee81e00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fee7c06f150 0x7fee7c0caf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:50.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.968+0000 7fee81e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fee7c159050 0x7fee7c154530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:50.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.968+0000 7fee81e00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fee7c159430 0x7fee7c151fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:50.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.968+0000 7fee81e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fee7c15ef50 con 0x7fee7c159050 2024-04-24T23:27:50.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.968+0000 7fee81e00640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fee7c0c8c80 con 0x7fee7c159430 2024-04-24T23:27:50.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.968+0000 7fee81e00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fee7c0c6cc0 con 0x7fee7c06f150 2024-04-24T23:27:50.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.969+0000 7fee7be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fee7c159050 0x7fee7c154530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:50.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.969+0000 7fee80a00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fee7c06f150 0x7fee7c0caf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:50.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.969+0000 7fee80a00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fee7c06f150 0x7fee7c0caf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.6:43712/0 (socket says 172.21.15.6:43712) 2024-04-24T23:27:50.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.969+0000 7fee80a00640 1 -- 172.21.15.6:0/3941573078 learned_addr learned my addr 172.21.15.6:0/3941573078 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:27:50.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.969+0000 7fee81400640 1 --2- 172.21.15.6:0/3941573078 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fee7c159430 0x7fee7c151fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:50.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.970+0000 7fee7be00640 1 -- 172.21.15.6:0/3941573078 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fee7c06f150 msgr2=0x7fee7c0caf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:50.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.970+0000 7fee7be00640 1 --2- 172.21.15.6:0/3941573078 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fee7c06f150 0x7fee7c0caf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:50.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.970+0000 7fee7be00640 1 -- 172.21.15.6:0/3941573078 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fee7c159430 msgr2=0x7fee7c151fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:50.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.970+0000 7fee7be00640 1 --2- 172.21.15.6:0/3941573078 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fee7c159430 0x7fee7c151fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:50.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.970+0000 7fee7be00640 1 -- 172.21.15.6:0/3941573078 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee7c153490 con 0x7fee7c159050 2024-04-24T23:27:50.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.971+0000 7fee7be00640 1 --2- 172.21.15.6:0/3941573078 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fee7c159050 0x7fee7c154530 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fee6800c750 tx=0x7fee6800cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:27:50.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.973+0000 7fee78c00640 1 -- 172.21.15.6:0/3941573078 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee68017070 con 0x7fee7c159050 2024-04-24T23:27:50.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.973+0000 7fee78c00640 1 -- 172.21.15.6:0/3941573078 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee68005270 con 0x7fee7c159050 2024-04-24T23:27:50.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.973+0000 7fee78c00640 1 -- 172.21.15.6:0/3941573078 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee6800ce60 con 0x7fee7c159050 2024-04-24T23:27:50.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.974+0000 7fee81e00640 1 -- 172.21.15.6:0/3941573078 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee7c153770 con 0x7fee7c159050 2024-04-24T23:27:50.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.974+0000 7fee81e00640 1 -- 172.21.15.6:0/3941573078 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee7c153d30 con 0x7fee7c159050 2024-04-24T23:27:50.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.976+0000 7fee6e000640 1 -- 172.21.15.6:0/3941573078 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fee50001030 con 0x7fee7c159050 2024-04-24T23:27:50.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.978+0000 7fee78c00640 1 -- 172.21.15.6:0/3941573078 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fee6801a070 con 0x7fee7c159050 2024-04-24T23:27:50.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.980+0000 7fee78c00640 1 --2- 172.21.15.6:0/3941573078 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fee5c06beb0 0x7fee5c06e370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:50.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.980+0000 7fee78c00640 1 -- 172.21.15.6:0/3941573078 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6934+0+0 (secure 0 0 0) 0x7fee6808a760 con 0x7fee7c159050 2024-04-24T23:27:50.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.980+0000 7fee78c00640 1 --2- 172.21.15.6:0/3941573078 >> [v2:172.21.15.190:6816/870571564,v1:172.21.15.190:6817/870571564] conn(0x7fee5c0759d0 0x7fee5c077e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:50.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.980+0000 7fee78c00640 1 -- 172.21.15.6:0/3941573078 --> [v2:172.21.15.190:6816/870571564,v1:172.21.15.190:6817/870571564] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fee5c078520 con 0x7fee5c0759d0 2024-04-24T23:27:50.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.980+0000 7fee78c00640 1 -- 172.21.15.6:0/3941573078 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fee68091050 con 0x7fee7c159050 2024-04-24T23:27:50.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.981+0000 7fee80a00640 1 --2- 172.21.15.6:0/3941573078 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fee5c06beb0 0x7fee5c06e370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:50.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.982+0000 7fee81400640 1 --2- 172.21.15.6:0/3941573078 >> [v2:172.21.15.190:6816/870571564,v1:172.21.15.190:6817/870571564] conn(0x7fee5c0759d0 0x7fee5c077e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:50.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.985+0000 7fee80a00640 1 --2- 172.21.15.6:0/3941573078 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fee5c06beb0 0x7fee5c06e370 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fee70000c00 tx=0x7fee7002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:27:50.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.985+0000 7fee81400640 1 --2- 172.21.15.6:0/3941573078 >> [v2:172.21.15.190:6816/870571564,v1:172.21.15.190:6817/870571564] conn(0x7fee5c0759d0 0x7fee5c077e50 crc :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:27:50.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:50.986+0000 7fee78c00640 1 -- 172.21.15.6:0/3941573078 <== osd.6 v2:172.21.15.190:6816/870571564 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fee5c078520 con 0x7fee5c0759d0 2024-04-24T23:27:51.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.022+0000 7fee6e000640 1 -- 172.21.15.6:0/3941573078 --> [v2:172.21.15.190:6816/870571564,v1:172.21.15.190:6817/870571564] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fee50002e30 con 0x7fee5c0759d0 2024-04-24T23:27:51.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.023+0000 7fee78c00640 1 -- 172.21.15.6:0/3941573078 <== osd.6 v2:172.21.15.190:6816/870571564 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fee50002e30 con 0x7fee5c0759d0 2024-04-24T23:27:51.024 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-04-24T23:27:51.024 INFO:teuthology.orchestra.run.smithi006.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-24T23:27:51.025 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-04-24T23:27:51.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.024+0000 7fee6e000640 1 -- 172.21.15.6:0/3941573078 >> [v2:172.21.15.190:6816/870571564,v1:172.21.15.190:6817/870571564] conn(0x7fee5c0759d0 msgr2=0x7fee5c077e50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:51.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.024+0000 7fee6e000640 1 --2- 172.21.15.6:0/3941573078 >> [v2:172.21.15.190:6816/870571564,v1:172.21.15.190:6817/870571564] conn(0x7fee5c0759d0 0x7fee5c077e50 crc :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:51.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.024+0000 7fee6e000640 1 -- 172.21.15.6:0/3941573078 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fee5c06beb0 msgr2=0x7fee5c06e370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:51.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.024+0000 7fee6e000640 1 --2- 172.21.15.6:0/3941573078 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fee5c06beb0 0x7fee5c06e370 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fee70000c00 tx=0x7fee7002e000 comp rx=0 tx=0).stop 2024-04-24T23:27:51.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.024+0000 7fee6e000640 1 -- 172.21.15.6:0/3941573078 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fee7c159050 msgr2=0x7fee7c154530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:51.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.024+0000 7fee6e000640 1 --2- 172.21.15.6:0/3941573078 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fee7c159050 0x7fee7c154530 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fee6800c750 tx=0x7fee6800cc10 comp rx=0 tx=0).stop 2024-04-24T23:27:51.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.025+0000 7fee81400640 1 -- 172.21.15.6:0/3941573078 reap_dead start 2024-04-24T23:27:51.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.025+0000 7fee6e000640 1 -- 172.21.15.6:0/3941573078 shutdown_connections 2024-04-24T23:27:51.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.025+0000 7fee6e000640 1 -- 172.21.15.6:0/3941573078 >> 172.21.15.6:0/3941573078 conn(0x7fee7c06cfe0 msgr2=0x7fee7c06b970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:27:51.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.025+0000 7fee6e000640 1 -- 172.21.15.6:0/3941573078 shutdown_connections 2024-04-24T23:27:51.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.025+0000 7fee6e000640 1 -- 172.21.15.6:0/3941573078 wait complete. 2024-04-24T23:27:51.054 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-04-24T23:27:51.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.388+0000 7efe86000640 1 Processor -- start 2024-04-24T23:27:51.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.389+0000 7efe86000640 1 -- start start 2024-04-24T23:27:51.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.391+0000 7efe86000640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efe80154d80 0x7efe800c5310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:51.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.391+0000 7efe86000640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7efe800cdaa0 0x7efe800cb260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:51.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.391+0000 7efe86000640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efe800c5850 con 0x7efe800cb7a0 2024-04-24T23:27:51.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.391+0000 7efe86000640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7efe800cd610 con 0x7efe800cdaa0 2024-04-24T23:27:51.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.391+0000 7efe86000640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7efe800cc720 con 0x7efe80154d80 2024-04-24T23:27:51.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.391+0000 7efe84c00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efe80154d80 0x7efe800c5310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:51.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.391+0000 7efe84c00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efe80154d80 0x7efe800c5310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.6:43726/0 (socket says 172.21.15.6:43726) 2024-04-24T23:27:51.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.392+0000 7efe84c00640 1 -- 172.21.15.6:0/3338689873 learned_addr learned my addr 172.21.15.6:0/3338689873 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:27:51.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.391+0000 7efe7fe00640 1 --2- 172.21.15.6:0/3338689873 >> v2:172.21.15.190:3300/0 conn(0x7efe800cdaa0 0x7efe800cb260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:51.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.393+0000 7efe7f400640 1 -- 172.21.15.6:0/3338689873 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3064544435 0 0) 0x7efe800c5850 con 0x7efe800cb7a0 2024-04-24T23:27:51.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.394+0000 7efe84c00640 1 -- 172.21.15.6:0/3338689873 >> v2:172.21.15.190:3300/0 conn(0x7efe800cdaa0 msgr2=0x7efe800cb260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:27:51.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.394+0000 7efe84c00640 1 --2- 172.21.15.6:0/3338689873 >> v2:172.21.15.190:3300/0 conn(0x7efe800cdaa0 0x7efe800cb260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:51.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.394+0000 7efe84c00640 1 -- 172.21.15.6:0/3338689873 >> v1:172.21.15.190:6789/0 conn(0x7efe800cb7a0 legacy=0x7efe800d03c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:51.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.394+0000 7efe84c00640 1 -- 172.21.15.6:0/3338689873 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe800cca00 con 0x7efe80154d80 2024-04-24T23:27:51.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.396+0000 7efe84c00640 1 --2- 172.21.15.6:0/3338689873 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efe80154d80 0x7efe800c5310 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7efe78009e50 tx=0x7efe7802a750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=503141fee303e991 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:27:51.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.397+0000 7efe7f400640 1 -- 172.21.15.6:0/3338689873 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe78036070 con 0x7efe80154d80 2024-04-24T23:27:51.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.397+0000 7efe7f400640 1 -- 172.21.15.6:0/3338689873 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efe78031600 con 0x7efe80154d80 2024-04-24T23:27:51.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.397+0000 7efe7f400640 1 -- 172.21.15.6:0/3338689873 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe78031a10 con 0x7efe80154d80 2024-04-24T23:27:51.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.398+0000 7efe86000640 1 -- 172.21.15.6:0/3338689873 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efe80154d80 msgr2=0x7efe800c5310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:51.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.398+0000 7efe86000640 1 --2- 172.21.15.6:0/3338689873 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efe80154d80 0x7efe800c5310 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7efe78009e50 tx=0x7efe7802a750 comp rx=0 tx=0).stop 2024-04-24T23:27:51.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.399+0000 7efe86000640 1 -- 172.21.15.6:0/3338689873 shutdown_connections 2024-04-24T23:27:51.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.399+0000 7efe86000640 1 --2- 172.21.15.6:0/3338689873 >> v2:172.21.15.190:3300/0 conn(0x7efe800cdaa0 0x7efe800cb260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:51.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.399+0000 7efe86000640 1 --2- 172.21.15.6:0/3338689873 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efe80154d80 0x7efe800c5310 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:51.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.399+0000 7efe86000640 1 -- 172.21.15.6:0/3338689873 >> 172.21.15.6:0/3338689873 conn(0x7efe8006c990 msgr2=0x7efe8006cda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:27:51.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.408+0000 7efe86000640 1 -- 172.21.15.6:0/3338689873 shutdown_connections 2024-04-24T23:27:51.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.409+0000 7efe86000640 1 -- 172.21.15.6:0/3338689873 wait complete. 2024-04-24T23:27:51.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.411+0000 7efe86000640 1 Processor -- start 2024-04-24T23:27:51.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.423+0000 7efe86000640 1 -- start start 2024-04-24T23:27:51.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.439+0000 7efe86000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efe800cb7a0 0x7efe800cfe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:51.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.440+0000 7efe86000640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efe80154d80 0x7efe800ce250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:51.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.444+0000 7efe86000640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efe800ce790 0x7efe801f0580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:51.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.444+0000 7efe7fe00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efe80154d80 0x7efe800ce250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:51.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.444+0000 7efe7fe00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efe80154d80 0x7efe800ce250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.6:43742/0 (socket says 172.21.15.6:43742) 2024-04-24T23:27:51.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.444+0000 7efe84c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efe800cb7a0 0x7efe800cfe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:51.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.444+0000 7efe84c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efe800cb7a0 0x7efe800cfe30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:58130/0 (socket says 172.21.15.6:58130) 2024-04-24T23:27:51.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.447+0000 7efe86000640 1 -- 172.21.15.6:0/1481592583 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7efe800cd2a0 con 0x7efe800cb7a0 2024-04-24T23:27:51.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.447+0000 7efe86000640 1 -- 172.21.15.6:0/1481592583 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7efe800ccfc0 con 0x7efe800ce790 2024-04-24T23:27:51.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.447+0000 7efe86000640 1 -- 172.21.15.6:0/1481592583 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7efe8015bb00 con 0x7efe80154d80 2024-04-24T23:27:51.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.445+0000 7efe7fe00640 1 -- 172.21.15.6:0/1481592583 learned_addr learned my addr 172.21.15.6:0/1481592583 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:27:51.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.445+0000 7efe85600640 1 --2- 172.21.15.6:0/1481592583 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efe800ce790 0x7efe801f0580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:51.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.449+0000 7efe85600640 1 -- 172.21.15.6:0/1481592583 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efe80154d80 msgr2=0x7efe800ce250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:51.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.449+0000 7efe85600640 1 --2- 172.21.15.6:0/1481592583 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efe80154d80 0x7efe800ce250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:51.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.449+0000 7efe85600640 1 -- 172.21.15.6:0/1481592583 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efe800cb7a0 msgr2=0x7efe800cfe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:51.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.449+0000 7efe85600640 1 --2- 172.21.15.6:0/1481592583 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efe800cb7a0 0x7efe800cfe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:51.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.449+0000 7efe85600640 1 -- 172.21.15.6:0/1481592583 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe801f3580 con 0x7efe800ce790 2024-04-24T23:27:51.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.449+0000 7efe84c00640 1 --2- 172.21.15.6:0/1481592583 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efe800cb7a0 0x7efe800cfe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:27:51.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.450+0000 7efe7fe00640 1 --2- 172.21.15.6:0/1481592583 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efe80154d80 0x7efe800ce250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello state changed while learned_addr, mark_down or replacing must be happened just now 2024-04-24T23:27:51.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.451+0000 7efe85600640 1 --2- 172.21.15.6:0/1481592583 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efe800ce790 0x7efe801f0580 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7efe7000c410 tx=0x7efe7000c8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:27:51.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.452+0000 7efe7cc00640 1 -- 172.21.15.6:0/1481592583 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe70017070 con 0x7efe800ce790 2024-04-24T23:27:51.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.452+0000 7efe7cc00640 1 -- 172.21.15.6:0/1481592583 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efe700035f0 con 0x7efe800ce790 2024-04-24T23:27:51.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.452+0000 7efe7cc00640 1 -- 172.21.15.6:0/1481592583 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe700039e0 con 0x7efe800ce790 2024-04-24T23:27:51.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.455+0000 7efe86000640 1 -- 172.21.15.6:0/1481592583 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe8015bf00 con 0x7efe800ce790 2024-04-24T23:27:51.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.455+0000 7efe86000640 1 -- 172.21.15.6:0/1481592583 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efe8015e950 con 0x7efe800ce790 2024-04-24T23:27:51.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.458+0000 7efe86000640 1 -- 172.21.15.6:0/1481592583 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efe80075ba0 con 0x7efe800ce790 2024-04-24T23:27:51.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.460+0000 7efe7cc00640 1 -- 172.21.15.6:0/1481592583 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7efe7001a070 con 0x7efe800ce790 2024-04-24T23:27:51.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.461+0000 7efe7cc00640 1 --2- 172.21.15.6:0/1481592583 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7efe6006bf00 0x7efe6006e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:51.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.461+0000 7efe7cc00640 1 -- 172.21.15.6:0/1481592583 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6934+0+0 (secure 0 0 0) 0x7efe7008a660 con 0x7efe800ce790 2024-04-24T23:27:51.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.463+0000 7efe84c00640 1 --2- 172.21.15.6:0/1481592583 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7efe6006bf00 0x7efe6006e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:51.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.465+0000 7efe84c00640 1 --2- 172.21.15.6:0/1481592583 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7efe6006bf00 0x7efe6006e3c0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7efe78033000 tx=0x7efe78009000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:27:51.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.466+0000 7efe7cc00640 1 -- 172.21.15.6:0/1481592583 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7efe70058bb0 con 0x7efe800ce790 2024-04-24T23:27:51.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.660+0000 7efe86000640 1 -- 172.21.15.6:0/1481592583 --> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) -- 0x7efe800cbdc0 con 0x7efe6006bf00 2024-04-24T23:27:51.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.661+0000 7efe7cc00640 1 -- 172.21.15.6:0/1481592583 <== mgr.4105 v2:172.21.15.190:6832/806769663 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7efe800cbdc0 con 0x7efe6006bf00 2024-04-24T23:27:51.662 INFO:teuthology.orchestra.run.smithi006.stderr:instructed osd(s) 6 to deep-scrub 2024-04-24T23:27:51.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.663+0000 7efe86000640 1 -- 172.21.15.6:0/1481592583 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7efe6006bf00 msgr2=0x7efe6006e3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:51.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.663+0000 7efe86000640 1 --2- 172.21.15.6:0/1481592583 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7efe6006bf00 0x7efe6006e3c0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7efe78033000 tx=0x7efe78009000 comp rx=0 tx=0).stop 2024-04-24T23:27:51.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.663+0000 7efe86000640 1 -- 172.21.15.6:0/1481592583 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efe800ce790 msgr2=0x7efe801f0580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:51.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.663+0000 7efe86000640 1 --2- 172.21.15.6:0/1481592583 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efe800ce790 0x7efe801f0580 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7efe7000c410 tx=0x7efe7000c8d0 comp rx=0 tx=0).stop 2024-04-24T23:27:51.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.664+0000 7efe86000640 1 -- 172.21.15.6:0/1481592583 shutdown_connections 2024-04-24T23:27:51.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.664+0000 7efe86000640 1 --2- 172.21.15.6:0/1481592583 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7efe6006bf00 0x7efe6006e3c0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:51.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.664+0000 7efe86000640 1 --2- 172.21.15.6:0/1481592583 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efe80154d80 0x7efe800ce250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:51.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.664+0000 7efe86000640 1 --2- 172.21.15.6:0/1481592583 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efe800ce790 0x7efe801f0580 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:51.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.664+0000 7efe86000640 1 --2- 172.21.15.6:0/1481592583 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efe800cb7a0 0x7efe800cfe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:51.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.664+0000 7efe86000640 1 -- 172.21.15.6:0/1481592583 >> 172.21.15.6:0/1481592583 conn(0x7efe8006c990 msgr2=0x7efe800c68f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:27:51.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.664+0000 7efe86000640 1 -- 172.21.15.6:0/1481592583 shutdown_connections 2024-04-24T23:27:51.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.664+0000 7efe86000640 1 -- 172.21.15.6:0/1481592583 wait complete. 2024-04-24T23:27:51.685 INFO:tasks.ceph:Scrubbing osd.7 2024-04-24T23:27:51.685 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-04-24T23:27:51.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.879+0000 7f531f200640 1 Processor -- start 2024-04-24T23:27:51.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.879+0000 7f531f200640 1 -- start start 2024-04-24T23:27:51.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.880+0000 7f531f200640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5320159430 0x7f5320070e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:51.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.881+0000 7f531f200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f532015f260 con 0x7f532006f150 2024-04-24T23:27:51.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.881+0000 7f531f200640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f532015f520 con 0x7f5320159050 2024-04-24T23:27:51.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.881+0000 7f531f200640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f5320071390 con 0x7f5320159430 2024-04-24T23:27:51.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.881+0000 7f531e800640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f532006f150 0x7f53200c91a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:34914/0 (socket says 172.21.15.6:34914) 2024-04-24T23:27:51.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.881+0000 7f531e800640 1 -- 172.21.15.6:0/2234887919 learned_addr learned my addr 172.21.15.6:0/2234887919 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:27:51.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.881+0000 7f5317e00640 1 --2- 172.21.15.6:0/2234887919 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5320159430 0x7f5320070e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:51.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.881+0000 7f531d400640 1 -- 172.21.15.6:0/2234887919 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 593117803 0 0) 0x7f532015f520 con 0x7f5320159050 2024-04-24T23:27:51.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.881+0000 7f531d400640 1 -- 172.21.15.6:0/2234887919 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5304003520 con 0x7f5320159050 2024-04-24T23:27:51.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.881+0000 7f5317e00640 1 -- 172.21.15.6:0/2234887919 >> v1:172.21.15.190:6789/0 conn(0x7f5320159050 legacy=0x7f53200cc130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:51.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.881+0000 7f5317e00640 1 -- 172.21.15.6:0/2234887919 >> v1:172.21.15.6:6789/0 conn(0x7f532006f150 legacy=0x7f53200c91a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:51.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.881+0000 7f531d400640 1 -- 172.21.15.6:0/2234887919 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3121557232 0 0) 0x7f532015f260 con 0x7f532006f150 2024-04-24T23:27:51.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.881+0000 7f5317e00640 1 -- 172.21.15.6:0/2234887919 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f532015f800 con 0x7f5320159430 2024-04-24T23:27:51.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.883+0000 7f5317e00640 1 --2- 172.21.15.6:0/2234887919 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5320159430 0x7f5320070e50 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f53080097d0 tx=0x7f530802a290 comp rx=0 tx=0).ready entity=mon.2 client_cookie=75f87e2742646f2c server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:27:51.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.883+0000 7f531d400640 1 -- 172.21.15.6:0/2234887919 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5308037070 con 0x7f5320159430 2024-04-24T23:27:51.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.883+0000 7f531d400640 1 -- 172.21.15.6:0/2234887919 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5308032440 con 0x7f5320159430 2024-04-24T23:27:51.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.883+0000 7f531d400640 1 -- 172.21.15.6:0/2234887919 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53080327f0 con 0x7f5320159430 2024-04-24T23:27:51.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.883+0000 7f531f200640 1 -- 172.21.15.6:0/2234887919 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5320159430 msgr2=0x7f5320070e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:51.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.883+0000 7f531f200640 1 --2- 172.21.15.6:0/2234887919 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5320159430 0x7f5320070e50 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f53080097d0 tx=0x7f530802a290 comp rx=0 tx=0).stop 2024-04-24T23:27:51.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.883+0000 7f531f200640 1 -- 172.21.15.6:0/2234887919 shutdown_connections 2024-04-24T23:27:51.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.883+0000 7f531f200640 1 --2- 172.21.15.6:0/2234887919 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5320159430 0x7f5320070e50 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:51.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.883+0000 7f531f200640 1 -- 172.21.15.6:0/2234887919 >> 172.21.15.6:0/2234887919 conn(0x7f532006cfe0 msgr2=0x7f532006ea40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:27:51.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.883+0000 7f531f200640 1 -- 172.21.15.6:0/2234887919 shutdown_connections 2024-04-24T23:27:51.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.884+0000 7f531f200640 1 -- 172.21.15.6:0/2234887919 wait complete. 2024-04-24T23:27:51.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.885+0000 7f531f200640 1 Processor -- start 2024-04-24T23:27:51.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.885+0000 7f531f200640 1 -- start start 2024-04-24T23:27:51.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.886+0000 7f531f200640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f532006f150 0x7f53201532f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:51.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.886+0000 7f531f200640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5320159050 0x7f5320161660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:51.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.886+0000 7f531f200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5320159430 0x7f5320162c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:51.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.886+0000 7f531f200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f532015ef50 con 0x7f5320159430 2024-04-24T23:27:51.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.886+0000 7f531f200640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f53200c8c80 con 0x7f532006f150 2024-04-24T23:27:51.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.886+0000 7f531f200640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f53200c6cc0 con 0x7f5320159050 2024-04-24T23:27:51.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.886+0000 7f5317e00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5320159050 0x7f5320161660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:51.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.886+0000 7f531e800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5320159430 0x7f5320162c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:51.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.886+0000 7f531de00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f532006f150 0x7f53201532f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:51.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.887+0000 7f5317e00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5320159050 0x7f5320161660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.6:43760/0 (socket says 172.21.15.6:43760) 2024-04-24T23:27:51.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.887+0000 7f5317e00640 1 -- 172.21.15.6:0/1705367827 learned_addr learned my addr 172.21.15.6:0/1705367827 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:27:51.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.887+0000 7f5317e00640 1 -- 172.21.15.6:0/1705367827 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f532006f150 msgr2=0x7f53201532f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:51.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.887+0000 7f5317e00640 1 --2- 172.21.15.6:0/1705367827 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f532006f150 0x7f53201532f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:51.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.887+0000 7f5317e00640 1 -- 172.21.15.6:0/1705367827 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5320159430 msgr2=0x7f5320162c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:51.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.887+0000 7f5317e00640 1 --2- 172.21.15.6:0/1705367827 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5320159430 0x7f5320162c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:51.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.887+0000 7f5317e00640 1 -- 172.21.15.6:0/1705367827 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53201f5010 con 0x7f5320159050 2024-04-24T23:27:51.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.887+0000 7f5317e00640 1 --2- 172.21.15.6:0/1705367827 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5320159050 0x7f5320161660 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f5308035040 tx=0x7f53080046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:27:51.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.888+0000 7f5316000640 1 -- 172.21.15.6:0/1705367827 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5308037050 con 0x7f5320159050 2024-04-24T23:27:51.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.888+0000 7f5316000640 1 -- 172.21.15.6:0/1705367827 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5308038040 con 0x7f5320159050 2024-04-24T23:27:51.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.888+0000 7f5316000640 1 -- 172.21.15.6:0/1705367827 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5308004470 con 0x7f5320159050 2024-04-24T23:27:51.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.888+0000 7f531f200640 1 -- 172.21.15.6:0/1705367827 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53201f52f0 con 0x7f5320159050 2024-04-24T23:27:51.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.888+0000 7f531f200640 1 -- 172.21.15.6:0/1705367827 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f53201f58b0 con 0x7f5320159050 2024-04-24T23:27:51.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.889+0000 7f531f200640 1 -- 172.21.15.6:0/1705367827 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f5320075b30 con 0x7f5320159050 2024-04-24T23:27:51.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.890+0000 7f5316000640 1 -- 172.21.15.6:0/1705367827 <== mon.2 v2:172.21.15.190:3301/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f530803c6d0 con 0x7f5320159050 2024-04-24T23:27:51.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.891+0000 7f5316000640 1 --2- 172.21.15.6:0/1705367827 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f52fc06be30 0x7f52fc06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:51.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.891+0000 7f5316000640 1 -- 172.21.15.6:0/1705367827 <== mon.2 v2:172.21.15.190:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6934+0+0 (secure 0 0 0) 0x7f53080abfe0 con 0x7f5320159050 2024-04-24T23:27:51.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.891+0000 7f531de00640 1 --2- 172.21.15.6:0/1705367827 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f52fc06be30 0x7f52fc06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:51.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.891+0000 7f5316000640 1 --2- 172.21.15.6:0/1705367827 >> [v2:172.21.15.190:6824/123639197,v1:172.21.15.190:6825/123639197] conn(0x7f52fc0758d0 0x7f52fc077d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:51.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.891+0000 7f5316000640 1 -- 172.21.15.6:0/1705367827 --> [v2:172.21.15.190:6824/123639197,v1:172.21.15.190:6825/123639197] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5308001d30 con 0x7f52fc0758d0 2024-04-24T23:27:51.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.891+0000 7f5316000640 1 -- 172.21.15.6:0/1705367827 <== mon.2 v2:172.21.15.190:3301/0 6 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f53080ac390 con 0x7f5320159050 2024-04-24T23:27:51.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.891+0000 7f531de00640 1 --2- 172.21.15.6:0/1705367827 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f52fc06be30 0x7f52fc06e2f0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f53100011e0 tx=0x7f5310009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:27:51.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.891+0000 7f531e800640 1 --2- 172.21.15.6:0/1705367827 >> [v2:172.21.15.190:6824/123639197,v1:172.21.15.190:6825/123639197] conn(0x7f52fc0758d0 0x7f52fc077d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:51.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.892+0000 7f531e800640 1 --2- 172.21.15.6:0/1705367827 >> [v2:172.21.15.190:6824/123639197,v1:172.21.15.190:6825/123639197] conn(0x7f52fc0758d0 0x7f52fc077d50 crc :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:27:51.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.893+0000 7f5316000640 1 -- 172.21.15.6:0/1705367827 <== osd.7 v2:172.21.15.190:6824/123639197 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f5308001d30 con 0x7f52fc0758d0 2024-04-24T23:27:51.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.909+0000 7f531f200640 1 -- 172.21.15.6:0/1705367827 --> [v2:172.21.15.190:6824/123639197,v1:172.21.15.190:6825/123639197] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f532015dd90 con 0x7f52fc0758d0 2024-04-24T23:27:51.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.910+0000 7f5316000640 1 -- 172.21.15.6:0/1705367827 <== osd.7 v2:172.21.15.190:6824/123639197 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f532015dd90 con 0x7f52fc0758d0 2024-04-24T23:27:51.911 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-04-24T23:27:51.911 INFO:teuthology.orchestra.run.smithi006.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-24T23:27:51.911 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-04-24T23:27:51.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.910+0000 7f531f200640 1 -- 172.21.15.6:0/1705367827 >> [v2:172.21.15.190:6824/123639197,v1:172.21.15.190:6825/123639197] conn(0x7f52fc0758d0 msgr2=0x7f52fc077d50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:51.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.910+0000 7f531f200640 1 --2- 172.21.15.6:0/1705367827 >> [v2:172.21.15.190:6824/123639197,v1:172.21.15.190:6825/123639197] conn(0x7f52fc0758d0 0x7f52fc077d50 crc :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:51.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.910+0000 7f531f200640 1 -- 172.21.15.6:0/1705367827 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f52fc06be30 msgr2=0x7f52fc06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:51.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.910+0000 7f531f200640 1 --2- 172.21.15.6:0/1705367827 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f52fc06be30 0x7f52fc06e2f0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f53100011e0 tx=0x7f5310009040 comp rx=0 tx=0).stop 2024-04-24T23:27:51.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.910+0000 7f531f200640 1 -- 172.21.15.6:0/1705367827 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5320159050 msgr2=0x7f5320161660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:51.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.910+0000 7f531f200640 1 --2- 172.21.15.6:0/1705367827 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5320159050 0x7f5320161660 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f5308035040 tx=0x7f53080046c0 comp rx=0 tx=0).stop 2024-04-24T23:27:51.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.910+0000 7f531e800640 1 -- 172.21.15.6:0/1705367827 reap_dead start 2024-04-24T23:27:51.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.910+0000 7f531f200640 1 -- 172.21.15.6:0/1705367827 shutdown_connections 2024-04-24T23:27:51.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.910+0000 7f531f200640 1 -- 172.21.15.6:0/1705367827 >> 172.21.15.6:0/1705367827 conn(0x7f532006cfe0 msgr2=0x7f532006ea40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:27:51.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.911+0000 7f531f200640 1 -- 172.21.15.6:0/1705367827 shutdown_connections 2024-04-24T23:27:51.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:51.911+0000 7f531f200640 1 -- 172.21.15.6:0/1705367827 wait complete. 2024-04-24T23:27:51.929 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-04-24T23:27:52.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:52.135+0000 7f189f200640 1 Processor -- start 2024-04-24T23:27:52.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:52.135+0000 7f189f200640 1 -- start start 2024-04-24T23:27:52.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:52.137+0000 7f189f200640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f18a00cc850 0x7f18a015b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:52.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:52.137+0000 7f189f200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f18a00d03f0 0x7f18a00d0820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:52.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:52.137+0000 7f189f200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f18a00c5640 con 0x7f18a0154d20 2024-04-24T23:27:52.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:52.137+0000 7f189f200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f18a00cba60 con 0x7f18a00d03f0 2024-04-24T23:27:52.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:52.137+0000 7f189f200640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f18a015bb10 con 0x7f18a00cc850 2024-04-24T23:27:52.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:52.137+0000 7f189e800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f18a00d03f0 0x7f18a00d0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:52.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:52.137+0000 7f189e800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f18a00d03f0 0x7f18a00d0820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:58158/0 (socket says 172.21.15.6:58158) 2024-04-24T23:27:52.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:52.137+0000 7f189e800640 1 -- 172.21.15.6:0/503145669 learned_addr learned my addr 172.21.15.6:0/503145669 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:27:52.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:52.137+0000 7f1895400640 1 --2- 172.21.15.6:0/503145669 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f18a00cc850 0x7f18a015b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:52.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:52.138+0000 7f189e800640 1 -- 172.21.15.6:0/503145669 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f18a00cc850 msgr2=0x7f18a015b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:27:52.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:52.138+0000 7f189e800640 1 --2- 172.21.15.6:0/503145669 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f18a00cc850 0x7f18a015b5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:52.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:52.138+0000 7f189e800640 1 -- 172.21.15.6:0/503145669 >> v1:172.21.15.6:6789/0 conn(0x7f18a0154d20 legacy=0x7f18a015e7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:52.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:52.138+0000 7f189e800640 1 -- 172.21.15.6:0/503145669 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18a00cb4a0 con 0x7f18a00d03f0 2024-04-24T23:27:52.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:52.139+0000 7f189e800640 1 --2- 172.21.15.6:0/503145669 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f18a00d03f0 0x7f18a00d0820 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f1898009870 tx=0x7f189802a300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28376c85fc8e9d49 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:27:52.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:52.139+0000 7f189d400640 1 -- 172.21.15.6:0/503145669 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1898037070 con 0x7f18a00d03f0 2024-04-24T23:27:52.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:52.139+0000 7f189d400640 1 -- 172.21.15.6:0/503145669 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f189802ae50 con 0x7f18a00d03f0 2024-04-24T23:27:52.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:52.139+0000 7f189d400640 1 -- 172.21.15.6:0/503145669 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1898032860 con 0x7f18a00d03f0 2024-04-24T23:27:52.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:52.139+0000 7f189f200640 1 -- 172.21.15.6:0/503145669 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f18a00d03f0 msgr2=0x7f18a00d0820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:52.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:52.139+0000 7f189f200640 1 --2- 172.21.15.6:0/503145669 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f18a00d03f0 0x7f18a00d0820 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f1898009870 tx=0x7f189802a300 comp rx=0 tx=0).stop 2024-04-24T23:27:52.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:52.140+0000 7f189f200640 1 -- 172.21.15.6:0/503145669 shutdown_connections 2024-04-24T23:27:52.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:52.140+0000 7f189f200640 1 --2- 172.21.15.6:0/503145669 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f18a00d03f0 0x7f18a00d0820 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:52.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:52.140+0000 7f189f200640 1 --2- 172.21.15.6:0/503145669 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f18a00cc850 0x7f18a015b5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:52.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:52.140+0000 7f189f200640 1 -- 172.21.15.6:0/503145669 >> 172.21.15.6:0/503145669 conn(0x7f18a006d040 msgr2=0x7f18a0066ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:27:52.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:52.140+0000 7f189f200640 1 -- 172.21.15.6:0/503145669 shutdown_connections 2024-04-24T23:27:52.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:52.140+0000 7f189f200640 1 -- 172.21.15.6:0/503145669 wait complete. 2024-04-24T23:27:52.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:52.141+0000 7f189f200640 1 Processor -- start 2024-04-24T23:27:52.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:52.141+0000 7f189f200640 1 -- start start 2024-04-24T23:27:52.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:52.142+0000 7f189f200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f18a00cc850 0x7f18a015da30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:52.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:52.142+0000 7f189f200640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f18a00d03f0 0x7f18a00b5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:52.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:52.142+0000 7f189f200640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f18a0154d20 0x7f18a00c8a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:52.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:52.142+0000 7f189f200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f18a015e2c0 con 0x7f18a00cc850 2024-04-24T23:27:52.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:52.142+0000 7f189f200640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f18a00c53f0 con 0x7f18a0154d20 2024-04-24T23:27:52.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:52.142+0000 7f189f200640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f18a00cbc80 con 0x7f18a00d03f0 2024-04-24T23:27:52.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:52.143+0000 7f189de00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f18a00cc850 0x7f18a015da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:52.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:52.143+0000 7f189de00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f18a00cc850 0x7f18a015da30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:58162/0 (socket says 172.21.15.6:58162) 2024-04-24T23:27:52.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:52.143+0000 7f189de00640 1 -- 172.21.15.6:0/350155006 learned_addr learned my addr 172.21.15.6:0/350155006 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:27:52.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:52.143+0000 7f1895400640 1 --2- 172.21.15.6:0/350155006 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f18a00d03f0 0x7f18a00b5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:52.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:52.143+0000 7f189e800640 1 --2- 172.21.15.6:0/350155006 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f18a0154d20 0x7f18a00c8a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:52.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:52.143+0000 7f189de00640 1 -- 172.21.15.6:0/350155006 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f18a00d03f0 msgr2=0x7f18a00b5870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:52.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:52.143+0000 7f189de00640 1 --2- 172.21.15.6:0/350155006 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f18a00d03f0 0x7f18a00b5870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:52.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:52.143+0000 7f189de00640 1 -- 172.21.15.6:0/350155006 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f18a0154d20 msgr2=0x7f18a00c8a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:52.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:52.143+0000 7f189de00640 1 --2- 172.21.15.6:0/350155006 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f18a0154d20 0x7f18a00c8a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:52.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:52.143+0000 7f189de00640 1 -- 172.21.15.6:0/350155006 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18a015a920 con 0x7f18a00cc850 2024-04-24T23:27:52.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:52.143+0000 7f189de00640 1 --2- 172.21.15.6:0/350155006 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f18a00cc850 0x7f18a015da30 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f189000c730 tx=0x7f189000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:27:52.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:52.144+0000 7f1896a00640 1 -- 172.21.15.6:0/350155006 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1890016020 con 0x7f18a00cc850 2024-04-24T23:27:52.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:52.144+0000 7f1896a00640 1 -- 172.21.15.6:0/350155006 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1890006d50 con 0x7f18a00cc850 2024-04-24T23:27:52.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:52.144+0000 7f1896a00640 1 -- 172.21.15.6:0/350155006 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1890003bb0 con 0x7f18a00cc850 2024-04-24T23:27:52.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:52.144+0000 7f189f200640 1 -- 172.21.15.6:0/350155006 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18a00c8250 con 0x7f18a00cc850 2024-04-24T23:27:52.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:52.144+0000 7f189f200640 1 -- 172.21.15.6:0/350155006 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18a01f16e0 con 0x7f18a00cc850 2024-04-24T23:27:52.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:52.145+0000 7f189f200640 1 -- 172.21.15.6:0/350155006 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f18a0075ba0 con 0x7f18a00cc850 2024-04-24T23:27:52.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:52.146+0000 7f1896a00640 1 -- 172.21.15.6:0/350155006 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f189000a850 con 0x7f18a00cc850 2024-04-24T23:27:52.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:52.147+0000 7f1896a00640 1 --2- 172.21.15.6:0/350155006 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f187c06be30 0x7f187c06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:52.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:52.147+0000 7f1896a00640 1 -- 172.21.15.6:0/350155006 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6934+0+0 (secure 0 0 0) 0x7f189008af00 con 0x7f18a00cc850 2024-04-24T23:27:52.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:52.147+0000 7f1895400640 1 --2- 172.21.15.6:0/350155006 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f187c06be30 0x7f187c06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:52.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:52.148+0000 7f1895400640 1 --2- 172.21.15.6:0/350155006 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f187c06be30 0x7f187c06e2f0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f18a00b6770 tx=0x7f1888008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:27:52.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:52.152+0000 7f1896a00640 1 -- 172.21.15.6:0/350155006 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f1890020020 con 0x7f18a00cc850 2024-04-24T23:27:52.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:52.352+0000 7f189f200640 1 -- 172.21.15.6:0/350155006 --> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) -- 0x7f18a00d1170 con 0x7f187c06be30 2024-04-24T23:27:52.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:52.353+0000 7f1896a00640 1 -- 172.21.15.6:0/350155006 <== mgr.4105 v2:172.21.15.190:6832/806769663 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f18a00d1170 con 0x7f187c06be30 2024-04-24T23:27:52.355 INFO:teuthology.orchestra.run.smithi006.stderr:instructed osd(s) 7 to deep-scrub 2024-04-24T23:27:52.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:52.356+0000 7f189f200640 1 -- 172.21.15.6:0/350155006 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f187c06be30 msgr2=0x7f187c06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:52.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:52.356+0000 7f189f200640 1 --2- 172.21.15.6:0/350155006 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f187c06be30 0x7f187c06e2f0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f18a00b6770 tx=0x7f1888008040 comp rx=0 tx=0).stop 2024-04-24T23:27:52.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:52.356+0000 7f189f200640 1 -- 172.21.15.6:0/350155006 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f18a00cc850 msgr2=0x7f18a015da30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:52.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:52.356+0000 7f189f200640 1 --2- 172.21.15.6:0/350155006 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f18a00cc850 0x7f18a015da30 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f189000c730 tx=0x7f189000cbf0 comp rx=0 tx=0).stop 2024-04-24T23:27:52.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:52.356+0000 7f189f200640 1 -- 172.21.15.6:0/350155006 shutdown_connections 2024-04-24T23:27:52.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:52.356+0000 7f189f200640 1 --2- 172.21.15.6:0/350155006 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f187c06be30 0x7f187c06e2f0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:52.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:52.356+0000 7f189f200640 1 --2- 172.21.15.6:0/350155006 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f18a00d03f0 0x7f18a00b5870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:52.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:52.356+0000 7f189f200640 1 --2- 172.21.15.6:0/350155006 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f18a0154d20 0x7f18a00c8a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:52.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:52.356+0000 7f189f200640 1 --2- 172.21.15.6:0/350155006 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f18a00cc850 0x7f18a015da30 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:52.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:52.356+0000 7f189f200640 1 -- 172.21.15.6:0/350155006 >> 172.21.15.6:0/350155006 conn(0x7f18a006d040 msgr2=0x7f18a00c5920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:27:52.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:52.356+0000 7f189f200640 1 -- 172.21.15.6:0/350155006 shutdown_connections 2024-04-24T23:27:52.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:52.356+0000 7f189f200640 1 -- 172.21.15.6:0/350155006 wait complete. 2024-04-24T23:27:52.378 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T23:27:52.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:52.546+0000 7f60b6a00640 1 Processor -- start 2024-04-24T23:27:52.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:52.546+0000 7f60b6a00640 1 -- start start 2024-04-24T23:27:52.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:52.548+0000 7f60b6a00640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f60b006d9e0 0x7f60b006ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:52.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:52.548+0000 7f60b6a00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f60b015b630 con 0x7f60b0154f50 2024-04-24T23:27:52.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:52.548+0000 7f60b6a00640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f60b015b910 con 0x7f60b00c75b0 2024-04-24T23:27:52.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:52.548+0000 7f60b6a00640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f60b00674f0 con 0x7f60b006d9e0 2024-04-24T23:27:52.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:52.549+0000 7f60afe00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f60b0154f50 0x7f60b00c7e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:34938/0 (socket says 172.21.15.6:34938) 2024-04-24T23:27:52.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:52.549+0000 7f60afe00640 1 -- 172.21.15.6:0/3671088558 learned_addr learned my addr 172.21.15.6:0/3671088558 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:27:52.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:52.549+0000 7f60b4a00640 1 --2- 172.21.15.6:0/3671088558 >> v2:172.21.15.190:3300/0 conn(0x7f60b006d9e0 0x7f60b006ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:52.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:52.549+0000 7f60aea00640 1 -- 172.21.15.6:0/3671088558 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4133531539 0 0) 0x7f60b015b910 con 0x7f60b00c75b0 2024-04-24T23:27:52.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:52.549+0000 7f60aea00640 1 -- 172.21.15.6:0/3671088558 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6094003520 con 0x7f60b00c75b0 2024-04-24T23:27:52.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:52.549+0000 7f60aea00640 1 -- 172.21.15.6:0/3671088558 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2319954919 0 0) 0x7f60b015b630 con 0x7f60b0154f50 2024-04-24T23:27:52.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:52.549+0000 7f60b4a00640 1 -- 172.21.15.6:0/3671088558 >> v1:172.21.15.190:6789/0 conn(0x7f60b00c75b0 legacy=0x7f60b015ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:52.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:52.549+0000 7f60b4a00640 1 -- 172.21.15.6:0/3671088558 >> v1:172.21.15.6:6789/0 conn(0x7f60b0154f50 legacy=0x7f60b00c7e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:52.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:52.549+0000 7f60b4a00640 1 -- 172.21.15.6:0/3671088558 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60b006f210 con 0x7f60b006d9e0 2024-04-24T23:27:52.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:52.550+0000 7f60b4a00640 1 --2- 172.21.15.6:0/3671088558 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f60b006d9e0 0x7f60b006ddf0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f60a8009d90 tx=0x7f60a802a780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ae1930bc373b0a1d server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:27:52.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:52.550+0000 7f60aea00640 1 -- 172.21.15.6:0/3671088558 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60a8037070 con 0x7f60b006d9e0 2024-04-24T23:27:52.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:52.550+0000 7f60aea00640 1 -- 172.21.15.6:0/3671088558 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f60a8007740 con 0x7f60b006d9e0 2024-04-24T23:27:52.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:52.550+0000 7f60aea00640 1 -- 172.21.15.6:0/3671088558 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60a8007b10 con 0x7f60b006d9e0 2024-04-24T23:27:52.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:52.551+0000 7f60b6a00640 1 -- 172.21.15.6:0/3671088558 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f60b006d9e0 msgr2=0x7f60b006ddf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:52.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:52.551+0000 7f60b6a00640 1 --2- 172.21.15.6:0/3671088558 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f60b006d9e0 0x7f60b006ddf0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f60a8009d90 tx=0x7f60a802a780 comp rx=0 tx=0).stop 2024-04-24T23:27:52.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:52.551+0000 7f60b6a00640 1 -- 172.21.15.6:0/3671088558 shutdown_connections 2024-04-24T23:27:52.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:52.551+0000 7f60b6a00640 1 --2- 172.21.15.6:0/3671088558 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f60b006d9e0 0x7f60b006ddf0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:52.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:52.551+0000 7f60b6a00640 1 -- 172.21.15.6:0/3671088558 >> 172.21.15.6:0/3671088558 conn(0x7f60b006b0e0 msgr2=0x7f60b006b4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:27:52.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:52.551+0000 7f60b6a00640 1 -- 172.21.15.6:0/3671088558 shutdown_connections 2024-04-24T23:27:52.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:52.551+0000 7f60b6a00640 1 -- 172.21.15.6:0/3671088558 wait complete. 2024-04-24T23:27:52.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:52.553+0000 7f60b6a00640 1 Processor -- start 2024-04-24T23:27:52.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:52.553+0000 7f60b6a00640 1 -- start start 2024-04-24T23:27:52.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:52.554+0000 7f60b6a00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f60b006d9e0 0x7f60b00c85c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:52.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:52.554+0000 7f60b6a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f60b00c75b0 0x7f60b014a980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:52.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:52.554+0000 7f60b6a00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f60b0154f50 0x7f60b0159210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:52.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:52.554+0000 7f60b6a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f60b006f860 con 0x7f60b00c75b0 2024-04-24T23:27:52.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:52.554+0000 7f60b6a00640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f60b006efc0 con 0x7f60b0154f50 2024-04-24T23:27:52.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:52.554+0000 7f60b6a00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f60b0155330 con 0x7f60b006d9e0 2024-04-24T23:27:52.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:52.554+0000 7f60af400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f60b00c75b0 0x7f60b014a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:52.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:52.554+0000 7f60af400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f60b00c75b0 0x7f60b014a980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:58178/0 (socket says 172.21.15.6:58178) 2024-04-24T23:27:52.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:52.554+0000 7f60af400640 1 -- 172.21.15.6:0/3498782050 learned_addr learned my addr 172.21.15.6:0/3498782050 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:27:52.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:52.554+0000 7f60afe00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f60b006d9e0 0x7f60b00c85c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:52.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:52.554+0000 7f60b4a00640 1 --2- 172.21.15.6:0/3498782050 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f60b0154f50 0x7f60b0159210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:52.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:52.554+0000 7f60af400640 1 -- 172.21.15.6:0/3498782050 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f60b006d9e0 msgr2=0x7f60b00c85c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:52.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:52.554+0000 7f60af400640 1 --2- 172.21.15.6:0/3498782050 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f60b006d9e0 0x7f60b00c85c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:52.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:52.554+0000 7f60af400640 1 -- 172.21.15.6:0/3498782050 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f60b0154f50 msgr2=0x7f60b0159210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:52.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:52.554+0000 7f60af400640 1 --2- 172.21.15.6:0/3498782050 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f60b0154f50 0x7f60b0159210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:52.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:52.554+0000 7f60af400640 1 -- 172.21.15.6:0/3498782050 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60b00cc970 con 0x7f60b00c75b0 2024-04-24T23:27:52.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:52.555+0000 7f60af400640 1 --2- 172.21.15.6:0/3498782050 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f60b00c75b0 0x7f60b014a980 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f609c00c500 tx=0x7f609c00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:27:52.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:52.555+0000 7f60a7e00640 1 -- 172.21.15.6:0/3498782050 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f609c0052d0 con 0x7f60b00c75b0 2024-04-24T23:27:52.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:52.555+0000 7f60a7e00640 1 -- 172.21.15.6:0/3498782050 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f609c005490 con 0x7f60b00c75b0 2024-04-24T23:27:52.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:52.555+0000 7f60b6a00640 1 -- 172.21.15.6:0/3498782050 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60b00c80c0 con 0x7f60b00c75b0 2024-04-24T23:27:52.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:52.555+0000 7f60a7e00640 1 -- 172.21.15.6:0/3498782050 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f609c012870 con 0x7f60b00c75b0 2024-04-24T23:27:52.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:52.555+0000 7f60b6a00640 1 -- 172.21.15.6:0/3498782050 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60b00cc3c0 con 0x7f60b00c75b0 2024-04-24T23:27:52.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:52.556+0000 7f60b6a00640 1 -- 172.21.15.6:0/3498782050 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f60b0068430 con 0x7f60b00c75b0 2024-04-24T23:27:52.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:52.557+0000 7f60a7e00640 1 -- 172.21.15.6:0/3498782050 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f609c012a10 con 0x7f60b00c75b0 2024-04-24T23:27:52.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:52.558+0000 7f60a7e00640 1 --2- 172.21.15.6:0/3498782050 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f609006be30 0x7f609006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:27:52.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:52.558+0000 7f60a7e00640 1 -- 172.21.15.6:0/3498782050 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6934+0+0 (secure 0 0 0) 0x7f609c08af60 con 0x7f60b00c75b0 2024-04-24T23:27:52.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:52.558+0000 7f60afe00640 1 --2- 172.21.15.6:0/3498782050 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f609006be30 0x7f609006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:27:52.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:52.559+0000 7f60afe00640 1 --2- 172.21.15.6:0/3498782050 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f609006be30 0x7f609006e2f0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f60a0007fd0 tx=0x7f60a0007de0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:27:52.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:52.562+0000 7f60a7e00640 1 -- 172.21.15.6:0/3498782050 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f609c058d60 con 0x7f60b00c75b0 2024-04-24T23:27:52.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:52.782+0000 7f60b6a00640 1 -- 172.21.15.6:0/3498782050 --> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f60b00c23f0 con 0x7f609006be30 2024-04-24T23:27:52.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:52.793+0000 7f60a7e00640 1 -- 172.21.15.6:0/3498782050 <== mgr.4105 v2:172.21.15.190:6832/806769663 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+333359 (secure 0 0 0) 0x7f60b00c23f0 con 0x7f609006be30 2024-04-24T23:27:52.796 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:27:52.801 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-04-24T23:27:52.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:52.801+0000 7f60b6a00640 1 -- 172.21.15.6:0/3498782050 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f609006be30 msgr2=0x7f609006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:52.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:52.801+0000 7f60b6a00640 1 --2- 172.21.15.6:0/3498782050 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f609006be30 0x7f609006e2f0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f60a0007fd0 tx=0x7f60a0007de0 comp rx=0 tx=0).stop 2024-04-24T23:27:52.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:52.801+0000 7f60b6a00640 1 -- 172.21.15.6:0/3498782050 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f60b00c75b0 msgr2=0x7f60b014a980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:27:52.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:52.801+0000 7f60b6a00640 1 --2- 172.21.15.6:0/3498782050 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f60b00c75b0 0x7f60b014a980 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f609c00c500 tx=0x7f609c00c9c0 comp rx=0 tx=0).stop 2024-04-24T23:27:52.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:52.802+0000 7f60b6a00640 1 -- 172.21.15.6:0/3498782050 shutdown_connections 2024-04-24T23:27:52.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:52.802+0000 7f60b6a00640 1 --2- 172.21.15.6:0/3498782050 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f609006be30 0x7f609006e2f0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:52.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:52.802+0000 7f60b6a00640 1 --2- 172.21.15.6:0/3498782050 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f60b0154f50 0x7f60b0159210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:52.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:52.802+0000 7f60b6a00640 1 --2- 172.21.15.6:0/3498782050 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f60b00c75b0 0x7f60b014a980 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:52.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:52.802+0000 7f60b6a00640 1 --2- 172.21.15.6:0/3498782050 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f60b006d9e0 0x7f60b00c85c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:27:52.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:52.802+0000 7f60b6a00640 1 -- 172.21.15.6:0/3498782050 >> 172.21.15.6:0/3498782050 conn(0x7f60b006b0e0 msgr2=0x7f60b014d4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:27:52.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:52.802+0000 7f60b6a00640 1 -- 172.21.15.6:0/3498782050 shutdown_connections 2024-04-24T23:27:52.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:27:52.802+0000 7f60b6a00640 1 -- 172.21.15.6:0/3498782050 wait complete. 2024-04-24T23:27:52.821 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":396,"stamp":"2024-04-24T23:27:50.874535+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1458807535,"num_objects":648,"num_object_clones":0,"num_object_copies":2334,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":648,"num_whiteouts":0,"num_read":2721,"num_read_kb":85149,"num_write":2064486,"num_write_kb":29276252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":74,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":145610,"ondisk_log_size":145610,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":10474160,"kb_used_data":3357808,"kb_used_omap":12,"kb_used_meta":7116275,"kb_avail":744500560,"statfs":{"total":773094113280,"available":762368573440,"internally_reserved":0,"allocated":3438395392,"data_stored":3436668196,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":7287066162},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,1,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":71,"apply_latency_ms":71,"commit_latency_ns":71000000,"apply_latency_ns":71000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-6270484480,"num_objects":-1495,"num_object_clones":0,"num_object_copies":-5980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1495,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1498,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003380"},"pg_stats":[{"pgid":"2.1f","version":"28'36","reported_seq":80,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:26:05.796187+0000","last_change":"2024-04-24T23:15:35.964646+0000","last_active":"2024-04-24T23:26:05.796187+0000","last_peered":"2024-04-24T23:26:05.796187+0000","last_clean":"2024-04-24T23:26:05.796187+0000","last_became_active":"2024-04-24T23:15:35.964357+0000","last_became_peered":"2024-04-24T23:15:35.964357+0000","last_unstale":"2024-04-24T23:26:05.796187+0000","last_undegraded":"2024-04-24T23:26:05.796187+0000","last_fullsized":"2024-04-24T23:26:05.796187+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:01:17.912993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194976,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":7,"num_read_kb":4102,"num_write":33,"num_write_kb":4110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"27'3","reported_seq":41,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:25:42.719837+0000","last_change":"2024-04-24T23:15:36.989772+0000","last_active":"2024-04-24T23:25:42.719837+0000","last_peered":"2024-04-24T23:25:42.719837+0000","last_clean":"2024-04-24T23:25:42.719837+0000","last_became_active":"2024-04-24T23:15:36.989543+0000","last_became_peered":"2024-04-24T23:15:36.989543+0000","last_unstale":"2024-04-24T23:25:42.719837+0000","last_undegraded":"2024-04-24T23:25:42.719837+0000","last_fullsized":"2024-04-24T23:25:42.719837+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:37:16.737354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"30'4754","reported_seq":4849,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:47.439903+0000","last_change":"2024-04-24T23:16:18.300096+0000","last_active":"2024-04-24T23:27:47.439903+0000","last_peered":"2024-04-24T23:27:47.439903+0000","last_clean":"2024-04-24T23:27:47.439903+0000","last_became_active":"2024-04-24T23:16:18.299217+0000","last_became_peered":"2024-04-24T23:16:18.299217+0000","last_unstale":"2024-04-24T23:27:47.439903+0000","last_undegraded":"2024-04-24T23:27:47.439903+0000","last_fullsized":"2024-04-24T23:27:47.439903+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":4754,"log_dups_size":0,"ondisk_log_size":4754,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33673223,"num_objects":18,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":70,"num_read_kb":0,"num_write":69569,"num_write_kb":969009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4,1],"acting":[0,6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"28'8","reported_seq":49,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:25:51.674845+0000","last_change":"2024-04-24T23:15:35.946083+0000","last_active":"2024-04-24T23:25:51.674845+0000","last_peered":"2024-04-24T23:25:51.674845+0000","last_clean":"2024-04-24T23:25:51.674845+0000","last_became_active":"2024-04-24T23:15:35.945227+0000","last_became_peered":"2024-04-24T23:15:35.945227+0000","last_unstale":"2024-04-24T23:25:51.674845+0000","last_undegraded":"2024-04-24T23:25:51.674845+0000","last_fullsized":"2024-04-24T23:25:51.674845+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:05:17.727725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":4096,"num_write":7,"num_write_kb":4105,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1f","version":"30'7","reported_seq":44,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:26:05.334280+0000","last_change":"2024-04-24T23:15:36.998245+0000","last_active":"2024-04-24T23:26:05.334280+0000","last_peered":"2024-04-24T23:26:05.334280+0000","last_clean":"2024-04-24T23:26:05.334280+0000","last_became_active":"2024-04-24T23:15:36.998021+0000","last_became_peered":"2024-04-24T23:15:36.998021+0000","last_unstale":"2024-04-24T23:26:05.334280+0000","last_undegraded":"2024-04-24T23:26:05.334280+0000","last_fullsized":"2024-04-24T23:26:05.334280+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:03:42.359117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"30'4577","reported_seq":4683,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:48.402287+0000","last_change":"2024-04-24T23:15:41.008313+0000","last_active":"2024-04-24T23:27:48.402287+0000","last_peered":"2024-04-24T23:27:48.402287+0000","last_clean":"2024-04-24T23:27:48.402287+0000","last_became_active":"2024-04-24T23:15:41.007908+0000","last_became_peered":"2024-04-24T23:15:41.007908+0000","last_unstale":"2024-04-24T23:27:48.402287+0000","last_undegraded":"2024-04-24T23:27:48.402287+0000","last_fullsized":"2024-04-24T23:27:48.402287+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":4577,"log_dups_size":0,"ondisk_log_size":4577,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12608681,"num_objects":7,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":75,"num_read_kb":0,"num_write":66612,"num_write_kb":930809,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4,6],"acting":[2,7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1d","version":"30'40","reported_seq":93,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:26.159820+0000","last_change":"2024-04-24T23:15:35.968312+0000","last_active":"2024-04-24T23:27:26.159820+0000","last_peered":"2024-04-24T23:27:26.159820+0000","last_clean":"2024-04-24T23:27:26.159820+0000","last_became_active":"2024-04-24T23:15:35.967873+0000","last_became_peered":"2024-04-24T23:15:35.967873+0000","last_unstale":"2024-04-24T23:27:26.159820+0000","last_undegraded":"2024-04-24T23:27:26.159820+0000","last_fullsized":"2024-04-24T23:27:26.159820+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":38,"num_read_kb":8226,"num_write":37,"num_write_kb":8205,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"27'7","reported_seq":44,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:25:42.821122+0000","last_change":"2024-04-24T23:15:36.997361+0000","last_active":"2024-04-24T23:25:42.821122+0000","last_peered":"2024-04-24T23:25:42.821122+0000","last_clean":"2024-04-24T23:25:42.821122+0000","last_became_active":"2024-04-24T23:15:36.997083+0000","last_became_peered":"2024-04-24T23:15:36.997083+0000","last_unstale":"2024-04-24T23:25:42.821122+0000","last_undegraded":"2024-04-24T23:25:42.821122+0000","last_fullsized":"2024-04-24T23:25:42.821122+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"30'4275","reported_seq":4376,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:48.608251+0000","last_change":"2024-04-24T23:15:41.025153+0000","last_active":"2024-04-24T23:27:48.608251+0000","last_peered":"2024-04-24T23:27:48.608251+0000","last_clean":"2024-04-24T23:27:48.608251+0000","last_became_active":"2024-04-24T23:15:41.024739+0000","last_became_peered":"2024-04-24T23:15:41.024739+0000","last_unstale":"2024-04-24T23:27:48.608251+0000","last_undegraded":"2024-04-24T23:27:48.608251+0000","last_fullsized":"2024-04-24T23:27:48.608251+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":4275,"log_dups_size":0,"ondisk_log_size":4275,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16793542,"num_objects":14,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":70,"num_read_kb":0,"num_write":62383,"num_write_kb":877947,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3,6],"acting":[2,5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1c","version":"30'17","reported_seq":64,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:36.148650+0000","last_change":"2024-04-24T23:15:35.967338+0000","last_active":"2024-04-24T23:27:36.148650+0000","last_peered":"2024-04-24T23:27:36.148650+0000","last_clean":"2024-04-24T23:27:36.148650+0000","last_became_active":"2024-04-24T23:15:35.966951+0000","last_became_peered":"2024-04-24T23:15:35.966951+0000","last_unstale":"2024-04-24T23:27:36.148650+0000","last_undegraded":"2024-04-24T23:27:36.148650+0000","last_fullsized":"2024-04-24T23:27:36.148650+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":232445,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":21,"num_read_kb":20,"num_write":17,"num_write_kb":234,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1d","version":"30'9","reported_seq":45,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:38.777791+0000","last_change":"2024-04-24T23:15:36.998072+0000","last_active":"2024-04-24T23:27:38.777791+0000","last_peered":"2024-04-24T23:27:38.777791+0000","last_clean":"2024-04-24T23:27:38.777791+0000","last_became_active":"2024-04-24T23:15:36.997735+0000","last_became_peered":"2024-04-24T23:15:36.997735+0000","last_unstale":"2024-04-24T23:27:38.777791+0000","last_undegraded":"2024-04-24T23:27:38.777791+0000","last_fullsized":"2024-04-24T23:27:38.777791+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:22:06.869644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"30'3764","reported_seq":3854,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:48.541881+0000","last_change":"2024-04-24T23:15:41.008895+0000","last_active":"2024-04-24T23:27:48.541881+0000","last_peered":"2024-04-24T23:27:48.541881+0000","last_clean":"2024-04-24T23:27:48.541881+0000","last_became_active":"2024-04-24T23:15:41.006199+0000","last_became_peered":"2024-04-24T23:15:41.006199+0000","last_unstale":"2024-04-24T23:27:48.541881+0000","last_undegraded":"2024-04-24T23:27:48.541881+0000","last_fullsized":"2024-04-24T23:27:48.541881+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":3764,"log_dups_size":0,"ondisk_log_size":3764,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16801186,"num_objects":8,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":59,"num_read_kb":0,"num_write":54797,"num_write_kb":774507,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5,7],"acting":[2,6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"30'8","reported_seq":45,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:26:34.900000+0000","last_change":"2024-04-24T23:15:37.004074+0000","last_active":"2024-04-24T23:26:34.900000+0000","last_peered":"2024-04-24T23:26:34.900000+0000","last_clean":"2024-04-24T23:26:34.900000+0000","last_became_active":"2024-04-24T23:15:37.003544+0000","last_became_peered":"2024-04-24T23:15:37.003544+0000","last_unstale":"2024-04-24T23:26:34.900000+0000","last_undegraded":"2024-04-24T23:26:34.900000+0000","last_fullsized":"2024-04-24T23:26:34.900000+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:38:28.024702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"30'1458","reported_seq":1502,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:26.160448+0000","last_change":"2024-04-24T23:15:35.963000+0000","last_active":"2024-04-24T23:27:26.160448+0000","last_peered":"2024-04-24T23:27:26.160448+0000","last_clean":"2024-04-24T23:27:26.160448+0000","last_became_active":"2024-04-24T23:15:35.961960+0000","last_became_peered":"2024-04-24T23:15:35.961960+0000","last_unstale":"2024-04-24T23:27:26.160448+0000","last_undegraded":"2024-04-24T23:27:26.160448+0000","last_fullsized":"2024-04-24T23:27:26.160448+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":1458,"log_dups_size":0,"ondisk_log_size":1458,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:56:26.651193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":10,"num_read_kb":8200,"num_write":1452,"num_write_kb":5429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"30'4361","reported_seq":4457,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:47.369579+0000","last_change":"2024-04-24T23:15:41.022481+0000","last_active":"2024-04-24T23:27:47.369579+0000","last_peered":"2024-04-24T23:27:47.369579+0000","last_clean":"2024-04-24T23:27:47.369579+0000","last_became_active":"2024-04-24T23:15:41.020034+0000","last_became_peered":"2024-04-24T23:15:41.020034+0000","last_unstale":"2024-04-24T23:27:47.369579+0000","last_undegraded":"2024-04-24T23:27:47.369579+0000","last_fullsized":"2024-04-24T23:27:47.369579+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":4361,"log_dups_size":0,"ondisk_log_size":4361,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:27:25.337503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37893684,"num_objects":16,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":65,"num_read_kb":0,"num_write":63646,"num_write_kb":904417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,5],"acting":[3,1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"30'6","reported_seq":43,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:26:52.416421+0000","last_change":"2024-04-24T23:15:36.972020+0000","last_active":"2024-04-24T23:26:52.416421+0000","last_peered":"2024-04-24T23:26:52.416421+0000","last_clean":"2024-04-24T23:26:52.416421+0000","last_became_active":"2024-04-24T23:15:36.971583+0000","last_became_peered":"2024-04-24T23:15:36.971583+0000","last_unstale":"2024-04-24T23:26:52.416421+0000","last_undegraded":"2024-04-24T23:26:52.416421+0000","last_fullsized":"2024-04-24T23:26:52.416421+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:29:05.568608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"30'7","reported_seq":47,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:26:31.416252+0000","last_change":"2024-04-24T23:15:35.945916+0000","last_active":"2024-04-24T23:26:31.416252+0000","last_peered":"2024-04-24T23:26:31.416252+0000","last_clean":"2024-04-24T23:26:31.416252+0000","last_became_active":"2024-04-24T23:15:35.945610+0000","last_became_peered":"2024-04-24T23:15:35.945610+0000","last_unstale":"2024-04-24T23:26:31.416252+0000","last_undegraded":"2024-04-24T23:26:31.416252+0000","last_fullsized":"2024-04-24T23:26:31.416252+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:57:05.976074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"30'4425","reported_seq":4527,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:47.147420+0000","last_change":"2024-04-24T23:15:41.030165+0000","last_active":"2024-04-24T23:27:47.147420+0000","last_peered":"2024-04-24T23:27:47.147420+0000","last_clean":"2024-04-24T23:27:47.147420+0000","last_became_active":"2024-04-24T23:15:41.029685+0000","last_became_peered":"2024-04-24T23:15:41.029685+0000","last_unstale":"2024-04-24T23:27:47.147420+0000","last_undegraded":"2024-04-24T23:27:47.147420+0000","last_fullsized":"2024-04-24T23:27:47.147420+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":4425,"log_dups_size":0,"ondisk_log_size":4425,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16795680,"num_objects":12,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":70,"num_read_kb":0,"num_write":64534,"num_write_kb":914755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6,1],"acting":[0,4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"30'8","reported_seq":45,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:26:30.926375+0000","last_change":"2024-04-24T23:15:36.965949+0000","last_active":"2024-04-24T23:26:30.926375+0000","last_peered":"2024-04-24T23:26:30.926375+0000","last_clean":"2024-04-24T23:26:30.926375+0000","last_became_active":"2024-04-24T23:15:36.965658+0000","last_became_peered":"2024-04-24T23:15:36.965658+0000","last_unstale":"2024-04-24T23:26:30.926375+0000","last_undegraded":"2024-04-24T23:26:30.926375+0000","last_fullsized":"2024-04-24T23:26:30.926375+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:38:28.024702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"28'8","reported_seq":54,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:25:42.710930+0000","last_change":"2024-04-24T23:15:35.972165+0000","last_active":"2024-04-24T23:25:42.710930+0000","last_peered":"2024-04-24T23:25:42.710930+0000","last_clean":"2024-04-24T23:25:42.710930+0000","last_became_active":"2024-04-24T23:15:35.971972+0000","last_became_peered":"2024-04-24T23:15:35.971972+0000","last_unstale":"2024-04-24T23:25:42.710930+0000","last_undegraded":"2024-04-24T23:25:42.710930+0000","last_fullsized":"2024-04-24T23:25:42.710930+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:11:12.088698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":20,"num_read_kb":19,"num_write":11,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"30'5198","reported_seq":5310,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:47.234133+0000","last_change":"2024-04-24T23:15:41.019799+0000","last_active":"2024-04-24T23:27:47.234133+0000","last_peered":"2024-04-24T23:27:47.234133+0000","last_clean":"2024-04-24T23:27:47.234133+0000","last_became_active":"2024-04-24T23:15:41.019339+0000","last_became_peered":"2024-04-24T23:15:41.019339+0000","last_unstale":"2024-04-24T23:27:47.234133+0000","last_undegraded":"2024-04-24T23:27:47.234133+0000","last_fullsized":"2024-04-24T23:27:47.234133+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":5198,"log_dups_size":0,"ondisk_log_size":5198,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:59:34.601004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58817653,"num_objects":19,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":81,"num_read_kb":0,"num_write":75886,"num_write_kb":1074791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2,3],"acting":[7,4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"30'7","reported_seq":44,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:42.364262+0000","last_change":"2024-04-24T23:15:36.991991+0000","last_active":"2024-04-24T23:27:42.364262+0000","last_peered":"2024-04-24T23:27:42.364262+0000","last_clean":"2024-04-24T23:27:42.364262+0000","last_became_active":"2024-04-24T23:15:36.989336+0000","last_became_peered":"2024-04-24T23:15:36.989336+0000","last_unstale":"2024-04-24T23:27:42.364262+0000","last_undegraded":"2024-04-24T23:27:42.364262+0000","last_fullsized":"2024-04-24T23:27:42.364262+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:25:56.038903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.8","version":"28'31","reported_seq":74,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:26:59.835062+0000","last_change":"2024-04-24T23:15:35.965443+0000","last_active":"2024-04-24T23:26:59.835062+0000","last_peered":"2024-04-24T23:26:59.835062+0000","last_clean":"2024-04-24T23:26:59.835062+0000","last_became_active":"2024-04-24T23:15:35.965043+0000","last_became_peered":"2024-04-24T23:15:35.965043+0000","last_unstale":"2024-04-24T23:26:59.835062+0000","last_undegraded":"2024-04-24T23:26:59.835062+0000","last_fullsized":"2024-04-24T23:26:59.835062+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":4102,"num_write":30,"num_write_kb":4111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"30'3822","reported_seq":3915,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:49.262475+0000","last_change":"2024-04-24T23:15:41.025131+0000","last_active":"2024-04-24T23:27:49.262475+0000","last_peered":"2024-04-24T23:27:49.262475+0000","last_clean":"2024-04-24T23:27:49.262475+0000","last_became_active":"2024-04-24T23:15:41.024731+0000","last_became_peered":"2024-04-24T23:15:41.024731+0000","last_unstale":"2024-04-24T23:27:49.262475+0000","last_undegraded":"2024-04-24T23:27:49.262475+0000","last_fullsized":"2024-04-24T23:27:49.262475+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":3822,"log_dups_size":0,"ondisk_log_size":3822,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16992933,"num_objects":14,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":61,"num_read_kb":0,"num_write":55559,"num_write_kb":793811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,7],"acting":[2,1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"30'15","reported_seq":52,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:26:31.027261+0000","last_change":"2024-04-24T23:15:36.965850+0000","last_active":"2024-04-24T23:26:31.027261+0000","last_peered":"2024-04-24T23:26:31.027261+0000","last_clean":"2024-04-24T23:26:31.027261+0000","last_became_active":"2024-04-24T23:15:36.965547+0000","last_became_peered":"2024-04-24T23:15:36.965547+0000","last_unstale":"2024-04-24T23:26:31.027261+0000","last_undegraded":"2024-04-24T23:26:31.027261+0000","last_fullsized":"2024-04-24T23:26:31.027261+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:06:37.125573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"30'9","reported_seq":56,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:23.206469+0000","last_change":"2024-04-24T23:15:35.971040+0000","last_active":"2024-04-24T23:27:23.206469+0000","last_peered":"2024-04-24T23:27:23.206469+0000","last_clean":"2024-04-24T23:27:23.206469+0000","last_became_active":"2024-04-24T23:15:35.970184+0000","last_became_peered":"2024-04-24T23:15:35.970184+0000","last_unstale":"2024-04-24T23:27:23.206469+0000","last_undegraded":"2024-04-24T23:27:23.206469+0000","last_fullsized":"2024-04-24T23:27:23.206469+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:45:20.913759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":24,"num_write":11,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"30'4415","reported_seq":4521,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:49.254955+0000","last_change":"2024-04-24T23:15:41.025813+0000","last_active":"2024-04-24T23:27:49.254955+0000","last_peered":"2024-04-24T23:27:49.254955+0000","last_clean":"2024-04-24T23:27:49.254955+0000","last_became_active":"2024-04-24T23:15:41.025137+0000","last_became_peered":"2024-04-24T23:15:41.025137+0000","last_unstale":"2024-04-24T23:27:49.254955+0000","last_undegraded":"2024-04-24T23:27:49.254955+0000","last_fullsized":"2024-04-24T23:27:49.254955+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":4415,"log_dups_size":0,"ondisk_log_size":4415,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20982844,"num_objects":8,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":74,"num_read_kb":0,"num_write":64405,"num_write_kb":917129,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,3,1],"acting":[2,6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"30'11","reported_seq":48,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:26:31.027294+0000","last_change":"2024-04-24T23:15:36.967476+0000","last_active":"2024-04-24T23:26:31.027294+0000","last_peered":"2024-04-24T23:26:31.027294+0000","last_clean":"2024-04-24T23:26:31.027294+0000","last_became_active":"2024-04-24T23:15:36.967098+0000","last_became_peered":"2024-04-24T23:15:36.967098+0000","last_unstale":"2024-04-24T23:26:31.027294+0000","last_undegraded":"2024-04-24T23:26:31.027294+0000","last_fullsized":"2024-04-24T23:26:31.027294+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:25:56.038903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"30'262","reported_seq":302,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:34.158914+0000","last_change":"2024-04-24T23:15:35.970630+0000","last_active":"2024-04-24T23:27:34.158914+0000","last_peered":"2024-04-24T23:27:34.158914+0000","last_clean":"2024-04-24T23:27:34.158914+0000","last_became_active":"2024-04-24T23:15:35.970430+0000","last_became_peered":"2024-04-24T23:15:35.970430+0000","last_unstale":"2024-04-24T23:27:34.158914+0000","last_undegraded":"2024-04-24T23:27:34.158914+0000","last_fullsized":"2024-04-24T23:27:34.158914+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":262,"log_dups_size":0,"ondisk_log_size":262,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:15:59.219084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":9,"num_write":561,"num_write_kb":230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"30'4762","reported_seq":4870,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:45.768315+0000","last_change":"2024-04-24T23:15:41.005022+0000","last_active":"2024-04-24T23:27:45.768315+0000","last_peered":"2024-04-24T23:27:45.768315+0000","last_clean":"2024-04-24T23:27:45.768315+0000","last_became_active":"2024-04-24T23:15:41.004648+0000","last_became_peered":"2024-04-24T23:15:41.004648+0000","last_unstale":"2024-04-24T23:27:45.768315+0000","last_undegraded":"2024-04-24T23:27:45.768315+0000","last_fullsized":"2024-04-24T23:27:45.768315+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":4762,"log_dups_size":0,"ondisk_log_size":4762,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:06:01.559760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33570085,"num_objects":14,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":77,"num_read_kb":0,"num_write":69570,"num_write_kb":1012990,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6,3],"acting":[4,1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.4","version":"30'4","reported_seq":41,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:26:06.045309+0000","last_change":"2024-04-24T23:15:36.967053+0000","last_active":"2024-04-24T23:26:06.045309+0000","last_peered":"2024-04-24T23:26:06.045309+0000","last_clean":"2024-04-24T23:26:06.045309+0000","last_became_active":"2024-04-24T23:15:36.966746+0000","last_became_peered":"2024-04-24T23:15:36.966746+0000","last_unstale":"2024-04-24T23:26:06.045309+0000","last_undegraded":"2024-04-24T23:26:06.045309+0000","last_fullsized":"2024-04-24T23:26:06.045309+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:16:08.779860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"26'4","reported_seq":46,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:25:42.781078+0000","last_change":"2024-04-24T23:15:35.965890+0000","last_active":"2024-04-24T23:25:42.781078+0000","last_peered":"2024-04-24T23:25:42.781078+0000","last_clean":"2024-04-24T23:25:42.781078+0000","last_became_active":"2024-04-24T23:15:35.965631+0000","last_became_peered":"2024-04-24T23:15:35.965631+0000","last_unstale":"2024-04-24T23:25:42.781078+0000","last_undegraded":"2024-04-24T23:25:42.781078+0000","last_fullsized":"2024-04-24T23:25:42.781078+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:17:35.650731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"30'4028","reported_seq":4119,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:45.686668+0000","last_change":"2024-04-24T23:15:41.011405+0000","last_active":"2024-04-24T23:27:45.686668+0000","last_peered":"2024-04-24T23:27:45.686668+0000","last_clean":"2024-04-24T23:27:45.686668+0000","last_became_active":"2024-04-24T23:15:41.010982+0000","last_became_peered":"2024-04-24T23:15:41.010982+0000","last_unstale":"2024-04-24T23:27:45.686668+0000","last_undegraded":"2024-04-24T23:27:45.686668+0000","last_fullsized":"2024-04-24T23:27:45.686668+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":4028,"log_dups_size":0,"ondisk_log_size":4028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:08:00.914296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58729331,"num_objects":18,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":60,"num_read_kb":0,"num_write":58636,"num_write_kb":828032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2,7],"acting":[6,0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"30'10","reported_seq":48,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:35.471656+0000","last_change":"2024-04-24T23:15:36.975244+0000","last_active":"2024-04-24T23:27:35.471656+0000","last_peered":"2024-04-24T23:27:35.471656+0000","last_clean":"2024-04-24T23:27:35.471656+0000","last_became_active":"2024-04-24T23:15:36.974872+0000","last_became_peered":"2024-04-24T23:15:36.974872+0000","last_unstale":"2024-04-24T23:27:35.471656+0000","last_undegraded":"2024-04-24T23:27:35.471656+0000","last_fullsized":"2024-04-24T23:27:35.471656+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"27'2","reported_seq":41,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:25:42.718435+0000","last_change":"2024-04-24T23:15:35.945792+0000","last_active":"2024-04-24T23:25:42.718435+0000","last_peered":"2024-04-24T23:25:42.718435+0000","last_clean":"2024-04-24T23:25:42.718435+0000","last_became_active":"2024-04-24T23:15:35.945094+0000","last_became_peered":"2024-04-24T23:15:35.945094+0000","last_unstale":"2024-04-24T23:25:42.718435+0000","last_undegraded":"2024-04-24T23:25:42.718435+0000","last_fullsized":"2024-04-24T23:25:42.718435+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:49:12.719127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"30'4941","reported_seq":5049,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:45.997411+0000","last_change":"2024-04-24T23:15:41.022105+0000","last_active":"2024-04-24T23:27:45.997411+0000","last_peered":"2024-04-24T23:27:45.997411+0000","last_clean":"2024-04-24T23:27:45.997411+0000","last_became_active":"2024-04-24T23:15:41.021645+0000","last_became_peered":"2024-04-24T23:15:41.021645+0000","last_unstale":"2024-04-24T23:27:45.997411+0000","last_undegraded":"2024-04-24T23:27:45.997411+0000","last_fullsized":"2024-04-24T23:27:45.997411+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":4941,"log_dups_size":0,"ondisk_log_size":4941,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:03:52.974835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75516331,"num_objects":21,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":77,"num_read_kb":0,"num_write":72415,"num_write_kb":1021329,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5,7],"acting":[6,3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.3","version":"30'8","reported_seq":46,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:32.415046+0000","last_change":"2024-04-24T23:15:36.991239+0000","last_active":"2024-04-24T23:27:32.415046+0000","last_peered":"2024-04-24T23:27:32.415046+0000","last_clean":"2024-04-24T23:27:32.415046+0000","last_became_active":"2024-04-24T23:15:36.990963+0000","last_became_peered":"2024-04-24T23:15:36.990963+0000","last_unstale":"2024-04-24T23:27:32.415046+0000","last_undegraded":"2024-04-24T23:27:32.415046+0000","last_fullsized":"2024-04-24T23:27:32.415046+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"29'42","reported_seq":91,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:26:59.824686+0000","last_change":"2024-04-24T23:15:35.957931+0000","last_active":"2024-04-24T23:26:59.824686+0000","last_peered":"2024-04-24T23:26:59.824686+0000","last_clean":"2024-04-24T23:26:59.824686+0000","last_became_active":"2024-04-24T23:15:35.957432+0000","last_became_peered":"2024-04-24T23:15:35.957432+0000","last_unstale":"2024-04-24T23:26:59.824686+0000","last_undegraded":"2024-04-24T23:26:59.824686+0000","last_fullsized":"2024-04-24T23:26:59.824686+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:11:38.773425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8389784,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":19,"num_read_kb":8206,"num_write":36,"num_write_kb":8211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.4","version":"30'4179","reported_seq":4275,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:45.664170+0000","last_change":"2024-04-24T23:15:41.007857+0000","last_active":"2024-04-24T23:27:45.664170+0000","last_peered":"2024-04-24T23:27:45.664170+0000","last_clean":"2024-04-24T23:27:45.664170+0000","last_became_active":"2024-04-24T23:15:41.007449+0000","last_became_peered":"2024-04-24T23:15:41.007449+0000","last_unstale":"2024-04-24T23:27:45.664170+0000","last_undegraded":"2024-04-24T23:27:45.664170+0000","last_fullsized":"2024-04-24T23:27:45.664170+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":4179,"log_dups_size":0,"ondisk_log_size":4179,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:05:06.145440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58797203,"num_objects":20,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":65,"num_read_kb":0,"num_write":60904,"num_write_kb":882911,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0,3],"acting":[6,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"27'11","reported_seq":49,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:25:42.717621+0000","last_change":"2024-04-24T23:15:36.970444+0000","last_active":"2024-04-24T23:25:42.717621+0000","last_peered":"2024-04-24T23:25:42.717621+0000","last_clean":"2024-04-24T23:25:42.717621+0000","last_became_active":"2024-04-24T23:15:36.970159+0000","last_became_peered":"2024-04-24T23:15:36.970159+0000","last_unstale":"2024-04-24T23:25:42.717621+0000","last_undegraded":"2024-04-24T23:25:42.717621+0000","last_fullsized":"2024-04-24T23:25:42.717621+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:37:02.956224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"30'43","reported_seq":115,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:33.304734+0000","last_change":"2024-04-24T23:15:35.945461+0000","last_active":"2024-04-24T23:27:33.304734+0000","last_peered":"2024-04-24T23:27:33.304734+0000","last_clean":"2024-04-24T23:27:33.304734+0000","last_became_active":"2024-04-24T23:15:35.945185+0000","last_became_peered":"2024-04-24T23:15:35.945185+0000","last_unstale":"2024-04-24T23:27:33.304734+0000","last_undegraded":"2024-04-24T23:27:33.304734+0000","last_fullsized":"2024-04-24T23:27:33.304734+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:37:34.175539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":94280,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":33,"num_read_kb":113,"num_write":40,"num_write_kb":118,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"30'4739","reported_seq":4844,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:45.961558+0000","last_change":"2024-04-24T23:15:41.014584+0000","last_active":"2024-04-24T23:27:45.961558+0000","last_peered":"2024-04-24T23:27:45.961558+0000","last_clean":"2024-04-24T23:27:45.961558+0000","last_became_active":"2024-04-24T23:15:41.014137+0000","last_became_peered":"2024-04-24T23:15:41.014137+0000","last_unstale":"2024-04-24T23:27:45.961558+0000","last_undegraded":"2024-04-24T23:27:45.961558+0000","last_fullsized":"2024-04-24T23:27:45.961558+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":4739,"log_dups_size":0,"ondisk_log_size":4739,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:23:50.387635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62965232,"num_objects":22,"num_object_clones":0,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":74,"num_read_kb":0,"num_write":69092,"num_write_kb":969120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,7],"acting":[6,0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"30'5","reported_seq":42,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:26:06.044607+0000","last_change":"2024-04-24T23:15:36.981783+0000","last_active":"2024-04-24T23:26:06.044607+0000","last_peered":"2024-04-24T23:26:06.044607+0000","last_clean":"2024-04-24T23:26:06.044607+0000","last_became_active":"2024-04-24T23:15:36.981502+0000","last_became_peered":"2024-04-24T23:15:36.981502+0000","last_unstale":"2024-04-24T23:26:06.044607+0000","last_undegraded":"2024-04-24T23:26:06.044607+0000","last_fullsized":"2024-04-24T23:26:06.044607+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:56:08.483410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"29'34","reported_seq":75,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:25:52.069108+0000","last_change":"2024-04-24T23:15:35.960519+0000","last_active":"2024-04-24T23:25:52.069108+0000","last_peered":"2024-04-24T23:25:52.069108+0000","last_clean":"2024-04-24T23:25:52.069108+0000","last_became_active":"2024-04-24T23:15:35.960180+0000","last_became_peered":"2024-04-24T23:15:35.960180+0000","last_unstale":"2024-04-24T23:25:52.069108+0000","last_undegraded":"2024-04-24T23:25:52.069108+0000","last_fullsized":"2024-04-24T23:25:52.069108+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:50:00.199120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":8192,"num_write":28,"num_write_kb":8204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"30'3847","reported_seq":3939,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:45.448604+0000","last_change":"2024-04-24T23:15:41.012876+0000","last_active":"2024-04-24T23:27:45.448604+0000","last_peered":"2024-04-24T23:27:45.448604+0000","last_clean":"2024-04-24T23:27:45.448604+0000","last_became_active":"2024-04-24T23:15:41.012493+0000","last_became_peered":"2024-04-24T23:15:41.012493+0000","last_unstale":"2024-04-24T23:27:45.448604+0000","last_undegraded":"2024-04-24T23:27:45.448604+0000","last_fullsized":"2024-04-24T23:27:45.448604+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":3847,"log_dups_size":0,"ondisk_log_size":3847,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:21:28.287059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46571186,"num_objects":17,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":61,"num_read_kb":307,"num_write":55950,"num_write_kb":780987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6,1],"acting":[5,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"30'7","reported_seq":44,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:45.654959+0000","last_change":"2024-04-24T23:15:36.981449+0000","last_active":"2024-04-24T23:27:45.654959+0000","last_peered":"2024-04-24T23:27:45.654959+0000","last_clean":"2024-04-24T23:27:45.654959+0000","last_became_active":"2024-04-24T23:15:36.980837+0000","last_became_peered":"2024-04-24T23:15:36.980837+0000","last_unstale":"2024-04-24T23:27:45.654959+0000","last_undegraded":"2024-04-24T23:27:45.654959+0000","last_fullsized":"2024-04-24T23:27:45.654959+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:15:13.513341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.3","version":"27'7","reported_seq":50,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:25:42.817265+0000","last_change":"2024-04-24T23:15:35.952947+0000","last_active":"2024-04-24T23:25:42.817265+0000","last_peered":"2024-04-24T23:25:42.817265+0000","last_clean":"2024-04-24T23:25:42.817265+0000","last_became_active":"2024-04-24T23:15:35.952655+0000","last_became_peered":"2024-04-24T23:15:35.952655+0000","last_unstale":"2024-04-24T23:25:42.817265+0000","last_undegraded":"2024-04-24T23:25:42.817265+0000","last_fullsized":"2024-04-24T23:25:42.817265+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:50:10.939396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":7,"num_read_kb":11,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"30'119","reported_seq":207,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:47.179514+0000","last_change":"2024-04-24T23:27:47.179576+0000","last_active":"2024-04-24T23:27:47.179514+0000","last_peered":"2024-04-24T23:27:47.179514+0000","last_clean":"2024-04-24T23:27:47.179514+0000","last_became_active":"2024-04-24T23:15:25.820030+0000","last_became_peered":"2024-04-24T23:15:25.820030+0000","last_unstale":"2024-04-24T23:27:47.179514+0000","last_undegraded":"2024-04-24T23:27:47.179514+0000","last_fullsized":"2024-04-24T23:27:47.179514+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'119","last_scrub_stamp":"2024-04-24T23:27:47.179391+0000","last_deep_scrub":"30'119","last_deep_scrub_stamp":"2024-04-24T23:27:47.179391+0000","last_clean_scrub_stamp":"2024-04-24T23:27:47.179391+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:06:58.842359+0000","scrub_duration":90,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"30'4271","reported_seq":4378,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:47.402700+0000","last_change":"2024-04-24T23:15:41.024712+0000","last_active":"2024-04-24T23:27:47.402700+0000","last_peered":"2024-04-24T23:27:47.402700+0000","last_clean":"2024-04-24T23:27:47.402700+0000","last_became_active":"2024-04-24T23:15:41.022697+0000","last_became_peered":"2024-04-24T23:15:41.022697+0000","last_unstale":"2024-04-24T23:27:47.402700+0000","last_undegraded":"2024-04-24T23:27:47.402700+0000","last_fullsized":"2024-04-24T23:27:47.402700+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":4271,"log_dups_size":0,"ondisk_log_size":4271,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:37:08.231105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41952772,"num_objects":15,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":76,"num_read_kb":0,"num_write":62639,"num_write_kb":898105,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2,0],"acting":[3,1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.d","version":"30'10","reported_seq":47,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:42.974365+0000","last_change":"2024-04-24T23:15:37.000489+0000","last_active":"2024-04-24T23:27:42.974365+0000","last_peered":"2024-04-24T23:27:42.974365+0000","last_clean":"2024-04-24T23:27:42.974365+0000","last_became_active":"2024-04-24T23:15:37.000179+0000","last_became_peered":"2024-04-24T23:15:37.000179+0000","last_unstale":"2024-04-24T23:27:42.974365+0000","last_undegraded":"2024-04-24T23:27:42.974365+0000","last_fullsized":"2024-04-24T23:27:42.974365+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:06:37.125573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.c","version":"28'10","reported_seq":49,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:25:42.744989+0000","last_change":"2024-04-24T23:15:35.966020+0000","last_active":"2024-04-24T23:25:42.744989+0000","last_peered":"2024-04-24T23:25:42.744989+0000","last_clean":"2024-04-24T23:25:42.744989+0000","last_became_active":"2024-04-24T23:15:35.965758+0000","last_became_peered":"2024-04-24T23:15:35.965758+0000","last_unstale":"2024-04-24T23:25:42.744989+0000","last_undegraded":"2024-04-24T23:25:42.744989+0000","last_fullsized":"2024-04-24T23:25:42.744989+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:39:21.233461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2157,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"30'4373","reported_seq":4475,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:48.278748+0000","last_change":"2024-04-24T23:15:41.024489+0000","last_active":"2024-04-24T23:27:48.278748+0000","last_peered":"2024-04-24T23:27:48.278748+0000","last_clean":"2024-04-24T23:27:48.278748+0000","last_became_active":"2024-04-24T23:15:41.024041+0000","last_became_peered":"2024-04-24T23:15:41.024041+0000","last_unstale":"2024-04-24T23:27:48.278748+0000","last_undegraded":"2024-04-24T23:27:48.278748+0000","last_fullsized":"2024-04-24T23:27:48.278748+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":4373,"log_dups_size":0,"ondisk_log_size":4373,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33561295,"num_objects":11,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":70,"num_read_kb":0,"num_write":63875,"num_write_kb":912579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7,4],"acting":[2,3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"30'8","reported_seq":46,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:45.746706+0000","last_change":"2024-04-24T23:15:36.979887+0000","last_active":"2024-04-24T23:27:45.746706+0000","last_peered":"2024-04-24T23:27:45.746706+0000","last_clean":"2024-04-24T23:27:45.746706+0000","last_became_active":"2024-04-24T23:15:36.979611+0000","last_became_peered":"2024-04-24T23:15:36.979611+0000","last_unstale":"2024-04-24T23:27:45.746706+0000","last_undegraded":"2024-04-24T23:27:45.746706+0000","last_fullsized":"2024-04-24T23:27:45.746706+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"28'25","reported_seq":66,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:26:56.337078+0000","last_change":"2024-04-24T23:15:35.965192+0000","last_active":"2024-04-24T23:26:56.337078+0000","last_peered":"2024-04-24T23:26:56.337078+0000","last_clean":"2024-04-24T23:26:56.337078+0000","last_became_active":"2024-04-24T23:15:35.964873+0000","last_became_peered":"2024-04-24T23:15:35.964873+0000","last_unstale":"2024-04-24T23:26:56.337078+0000","last_undegraded":"2024-04-24T23:26:56.337078+0000","last_fullsized":"2024-04-24T23:26:56.337078+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:14:27.884944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194374,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":4097,"num_write":20,"num_write_kb":4108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"30'4112","reported_seq":4214,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:45.295282+0000","last_change":"2024-04-24T23:15:41.004002+0000","last_active":"2024-04-24T23:27:45.295282+0000","last_peered":"2024-04-24T23:27:45.295282+0000","last_clean":"2024-04-24T23:27:45.295282+0000","last_became_active":"2024-04-24T23:15:41.003335+0000","last_became_peered":"2024-04-24T23:15:41.003335+0000","last_unstale":"2024-04-24T23:27:45.295282+0000","last_undegraded":"2024-04-24T23:27:45.295282+0000","last_fullsized":"2024-04-24T23:27:45.295282+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":4112,"log_dups_size":0,"ondisk_log_size":4112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:49:36.206903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54551044,"num_objects":22,"num_object_clones":0,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":71,"num_read_kb":0,"num_write":60130,"num_write_kb":843409,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1,3],"acting":[5,4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"27'8","reported_seq":46,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:02.756974+0000","last_change":"2024-04-24T23:15:37.002854+0000","last_active":"2024-04-24T23:27:02.756974+0000","last_peered":"2024-04-24T23:27:02.756974+0000","last_clean":"2024-04-24T23:27:02.756974+0000","last_became_active":"2024-04-24T23:15:37.002583+0000","last_became_peered":"2024-04-24T23:15:37.002583+0000","last_unstale":"2024-04-24T23:27:02.756974+0000","last_undegraded":"2024-04-24T23:27:02.756974+0000","last_fullsized":"2024-04-24T23:27:02.756974+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"30'166","reported_seq":213,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:28.158437+0000","last_change":"2024-04-24T23:15:35.948752+0000","last_active":"2024-04-24T23:27:28.158437+0000","last_peered":"2024-04-24T23:27:28.158437+0000","last_clean":"2024-04-24T23:27:28.158437+0000","last_became_active":"2024-04-24T23:15:35.948438+0000","last_became_peered":"2024-04-24T23:15:35.948438+0000","last_unstale":"2024-04-24T23:27:28.158437+0000","last_undegraded":"2024-04-24T23:27:28.158437+0000","last_fullsized":"2024-04-24T23:27:28.158437+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":166,"log_dups_size":0,"ondisk_log_size":166,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:45:56.006278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":21,"num_read_kb":2419,"num_write":164,"num_write_kb":4172,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.8","version":"30'4126","reported_seq":4225,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:47.386048+0000","last_change":"2024-04-24T23:15:41.017524+0000","last_active":"2024-04-24T23:27:47.386048+0000","last_peered":"2024-04-24T23:27:47.386048+0000","last_clean":"2024-04-24T23:27:47.386048+0000","last_became_active":"2024-04-24T23:15:41.017092+0000","last_became_peered":"2024-04-24T23:15:41.017092+0000","last_unstale":"2024-04-24T23:27:47.386048+0000","last_undegraded":"2024-04-24T23:27:47.386048+0000","last_fullsized":"2024-04-24T23:27:47.386048+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":4126,"log_dups_size":0,"ondisk_log_size":4126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:04:53.235038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50350428,"num_objects":16,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":68,"num_read_kb":0,"num_write":60414,"num_write_kb":856855,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"30'5","reported_seq":43,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:41.665192+0000","last_change":"2024-04-24T23:15:36.990441+0000","last_active":"2024-04-24T23:27:41.665192+0000","last_peered":"2024-04-24T23:27:41.665192+0000","last_clean":"2024-04-24T23:27:41.665192+0000","last_became_active":"2024-04-24T23:15:36.989863+0000","last_became_peered":"2024-04-24T23:15:36.989863+0000","last_unstale":"2024-04-24T23:27:41.665192+0000","last_undegraded":"2024-04-24T23:27:41.665192+0000","last_fullsized":"2024-04-24T23:27:41.665192+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.f","version":"30'6","reported_seq":47,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:02.756991+0000","last_change":"2024-04-24T23:15:35.969317+0000","last_active":"2024-04-24T23:27:02.756991+0000","last_peered":"2024-04-24T23:27:02.756991+0000","last_clean":"2024-04-24T23:27:02.756991+0000","last_became_active":"2024-04-24T23:15:35.968994+0000","last_became_peered":"2024-04-24T23:15:35.968994+0000","last_unstale":"2024-04-24T23:27:02.756991+0000","last_undegraded":"2024-04-24T23:27:02.756991+0000","last_fullsized":"2024-04-24T23:27:02.756991+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"30'4160","reported_seq":4265,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:47.472516+0000","last_change":"2024-04-24T23:15:41.010103+0000","last_active":"2024-04-24T23:27:47.472516+0000","last_peered":"2024-04-24T23:27:47.472516+0000","last_clean":"2024-04-24T23:27:47.472516+0000","last_became_active":"2024-04-24T23:15:41.009328+0000","last_became_peered":"2024-04-24T23:15:41.009328+0000","last_unstale":"2024-04-24T23:27:47.472516+0000","last_undegraded":"2024-04-24T23:27:47.472516+0000","last_fullsized":"2024-04-24T23:27:47.472516+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":4160,"log_dups_size":0,"ondisk_log_size":4160,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16856525,"num_objects":8,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":73,"num_read_kb":0,"num_write":60599,"num_write_kb":866216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,2],"acting":[1,5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.11","version":"27'10","reported_seq":47,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:25:42.747653+0000","last_change":"2024-04-24T23:15:36.996765+0000","last_active":"2024-04-24T23:25:42.747653+0000","last_peered":"2024-04-24T23:25:42.747653+0000","last_clean":"2024-04-24T23:25:42.747653+0000","last_became_active":"2024-04-24T23:15:36.995801+0000","last_became_peered":"2024-04-24T23:15:36.995801+0000","last_unstale":"2024-04-24T23:25:42.747653+0000","last_undegraded":"2024-04-24T23:25:42.747653+0000","last_fullsized":"2024-04-24T23:25:42.747653+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:25:19.421667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"28'19","reported_seq":66,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:23.188825+0000","last_change":"2024-04-24T23:15:35.944023+0000","last_active":"2024-04-24T23:27:23.188825+0000","last_peered":"2024-04-24T23:27:23.188825+0000","last_clean":"2024-04-24T23:27:23.188825+0000","last_became_active":"2024-04-24T23:15:35.943710+0000","last_became_peered":"2024-04-24T23:15:35.943710+0000","last_unstale":"2024-04-24T23:27:23.188825+0000","last_undegraded":"2024-04-24T23:27:23.188825+0000","last_fullsized":"2024-04-24T23:27:23.188825+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:46:48.090958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"30'4662","reported_seq":4762,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:45.996675+0000","last_change":"2024-04-24T23:15:41.013027+0000","last_active":"2024-04-24T23:27:45.996675+0000","last_peered":"2024-04-24T23:27:45.996675+0000","last_clean":"2024-04-24T23:27:45.996675+0000","last_became_active":"2024-04-24T23:15:41.012587+0000","last_became_peered":"2024-04-24T23:15:41.012587+0000","last_unstale":"2024-04-24T23:27:45.996675+0000","last_undegraded":"2024-04-24T23:27:45.996675+0000","last_fullsized":"2024-04-24T23:27:45.996675+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":4662,"log_dups_size":0,"ondisk_log_size":4662,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:27:04.276246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46391073,"num_objects":22,"num_object_clones":0,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":70,"num_read_kb":0,"num_write":67805,"num_write_kb":966739,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1,6],"acting":[5,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"30'9","reported_seq":47,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:44.024813+0000","last_change":"2024-04-24T23:15:36.979702+0000","last_active":"2024-04-24T23:27:44.024813+0000","last_peered":"2024-04-24T23:27:44.024813+0000","last_clean":"2024-04-24T23:27:44.024813+0000","last_became_active":"2024-04-24T23:15:36.979356+0000","last_became_peered":"2024-04-24T23:15:36.979356+0000","last_unstale":"2024-04-24T23:27:44.024813+0000","last_undegraded":"2024-04-24T23:27:44.024813+0000","last_fullsized":"2024-04-24T23:27:44.024813+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.11","version":"28'12","reported_seq":52,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:12.774973+0000","last_change":"2024-04-24T23:15:35.973026+0000","last_active":"2024-04-24T23:27:12.774973+0000","last_peered":"2024-04-24T23:27:12.774973+0000","last_clean":"2024-04-24T23:27:12.774973+0000","last_became_active":"2024-04-24T23:15:35.972655+0000","last_became_peered":"2024-04-24T23:15:35.972655+0000","last_unstale":"2024-04-24T23:27:12.774973+0000","last_undegraded":"2024-04-24T23:27:12.774973+0000","last_fullsized":"2024-04-24T23:27:12.774973+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1575,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"30'4502","reported_seq":4602,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:45.891267+0000","last_change":"2024-04-24T23:15:41.004221+0000","last_active":"2024-04-24T23:27:45.891267+0000","last_peered":"2024-04-24T23:27:45.891267+0000","last_clean":"2024-04-24T23:27:45.891267+0000","last_became_active":"2024-04-24T23:15:41.003790+0000","last_became_peered":"2024-04-24T23:15:41.003790+0000","last_unstale":"2024-04-24T23:27:45.891267+0000","last_undegraded":"2024-04-24T23:27:45.891267+0000","last_fullsized":"2024-04-24T23:27:45.891267+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":4502,"log_dups_size":0,"ondisk_log_size":4502,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:28:58.866024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62974182,"num_objects":21,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":70,"num_read_kb":0,"num_write":65456,"num_write_kb":956853,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6,4],"acting":[5,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"27'10","reported_seq":47,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:25:42.816663+0000","last_change":"2024-04-24T23:15:36.997389+0000","last_active":"2024-04-24T23:25:42.816663+0000","last_peered":"2024-04-24T23:25:42.816663+0000","last_clean":"2024-04-24T23:25:42.816663+0000","last_became_active":"2024-04-24T23:15:36.995923+0000","last_became_peered":"2024-04-24T23:15:36.995923+0000","last_unstale":"2024-04-24T23:25:42.816663+0000","last_undegraded":"2024-04-24T23:25:42.816663+0000","last_fullsized":"2024-04-24T23:25:42.816663+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:55:47.134320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"29'7","reported_seq":50,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:25:43.845951+0000","last_change":"2024-04-24T23:15:35.957715+0000","last_active":"2024-04-24T23:25:43.845951+0000","last_peered":"2024-04-24T23:25:43.845951+0000","last_clean":"2024-04-24T23:25:43.845951+0000","last_became_active":"2024-04-24T23:15:35.957088+0000","last_became_peered":"2024-04-24T23:15:35.957088+0000","last_unstale":"2024-04-24T23:25:43.845951+0000","last_undegraded":"2024-04-24T23:25:43.845951+0000","last_fullsized":"2024-04-24T23:25:43.845951+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:30:34.483669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":7,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"30'4661","reported_seq":4763,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:45.605559+0000","last_change":"2024-04-24T23:15:41.008165+0000","last_active":"2024-04-24T23:27:45.605559+0000","last_peered":"2024-04-24T23:27:45.605559+0000","last_clean":"2024-04-24T23:27:45.605559+0000","last_became_active":"2024-04-24T23:15:41.007555+0000","last_became_peered":"2024-04-24T23:15:41.007555+0000","last_unstale":"2024-04-24T23:27:45.605559+0000","last_undegraded":"2024-04-24T23:27:45.605559+0000","last_fullsized":"2024-04-24T23:27:45.605559+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":4661,"log_dups_size":0,"ondisk_log_size":4661,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:18:08.195747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46369483,"num_objects":17,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":71,"num_read_kb":0,"num_write":68031,"num_write_kb":979211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0,7],"acting":[6,4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.12","version":"30'13","reported_seq":50,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:37.783294+0000","last_change":"2024-04-24T23:15:36.991382+0000","last_active":"2024-04-24T23:27:37.783294+0000","last_peered":"2024-04-24T23:27:37.783294+0000","last_clean":"2024-04-24T23:27:37.783294+0000","last_became_active":"2024-04-24T23:15:36.991115+0000","last_became_peered":"2024-04-24T23:15:36.991115+0000","last_unstale":"2024-04-24T23:27:37.783294+0000","last_undegraded":"2024-04-24T23:27:37.783294+0000","last_fullsized":"2024-04-24T23:27:37.783294+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:21:51.863947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"28'40","reported_seq":90,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:22.419341+0000","last_change":"2024-04-24T23:15:35.964245+0000","last_active":"2024-04-24T23:27:22.419341+0000","last_peered":"2024-04-24T23:27:22.419341+0000","last_clean":"2024-04-24T23:27:22.419341+0000","last_became_active":"2024-04-24T23:15:35.963907+0000","last_became_peered":"2024-04-24T23:15:35.963907+0000","last_unstale":"2024-04-24T23:27:22.419341+0000","last_undegraded":"2024-04-24T23:27:22.419341+0000","last_fullsized":"2024-04-24T23:27:22.419341+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:15:00.913899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388676,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":26,"num_read_kb":12314,"num_write":42,"num_write_kb":8216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"30'4259","reported_seq":4363,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:47.622961+0000","last_change":"2024-04-24T23:15:41.021981+0000","last_active":"2024-04-24T23:27:47.622961+0000","last_peered":"2024-04-24T23:27:47.622961+0000","last_clean":"2024-04-24T23:27:47.622961+0000","last_became_active":"2024-04-24T23:15:41.019193+0000","last_became_peered":"2024-04-24T23:15:41.019193+0000","last_unstale":"2024-04-24T23:27:47.622961+0000","last_undegraded":"2024-04-24T23:27:47.622961+0000","last_fullsized":"2024-04-24T23:27:47.622961+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":4259,"log_dups_size":0,"ondisk_log_size":4259,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:04:39.828502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37748736,"num_objects":9,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":73,"num_read_kb":0,"num_write":62315,"num_write_kb":864711,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5,6],"acting":[3,0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"30'10","reported_seq":56,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:47.168889+0000","last_change":"2024-04-24T23:27:47.168936+0000","last_active":"2024-04-24T23:27:47.168889+0000","last_peered":"2024-04-24T23:27:47.168889+0000","last_clean":"2024-04-24T23:27:47.168889+0000","last_became_active":"2024-04-24T23:15:36.986484+0000","last_became_peered":"2024-04-24T23:15:36.986484+0000","last_unstale":"2024-04-24T23:27:47.168889+0000","last_undegraded":"2024-04-24T23:27:47.168889+0000","last_fullsized":"2024-04-24T23:27:47.168889+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'10","last_scrub_stamp":"2024-04-24T23:27:47.168801+0000","last_deep_scrub":"30'10","last_deep_scrub_stamp":"2024-04-24T23:27:47.168801+0000","last_clean_scrub_stamp":"2024-04-24T23:27:47.168801+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:37:32.843942+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.14","version":"30'241","reported_seq":300,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:48.044561+0000","last_change":"2024-04-24T23:27:48.044628+0000","last_active":"2024-04-24T23:27:48.044561+0000","last_peered":"2024-04-24T23:27:48.044561+0000","last_clean":"2024-04-24T23:27:48.044561+0000","last_became_active":"2024-04-24T23:15:35.951923+0000","last_became_peered":"2024-04-24T23:15:35.951923+0000","last_unstale":"2024-04-24T23:27:48.044561+0000","last_undegraded":"2024-04-24T23:27:48.044561+0000","last_fullsized":"2024-04-24T23:27:48.044561+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'241","last_scrub_stamp":"2024-04-24T23:27:48.044467+0000","last_deep_scrub":"30'241","last_deep_scrub_stamp":"2024-04-24T23:27:48.044467+0000","last_clean_scrub_stamp":"2024-04-24T23:27:48.044467+0000","objects_scrubbed":6,"log_size":241,"log_dups_size":0,"ondisk_log_size":241,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:24:37.933540+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194976,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":28,"num_read_kb":4122,"num_write":236,"num_write_kb":4244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"30'4365","reported_seq":4472,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:45.467387+0000","last_change":"2024-04-24T23:15:41.015494+0000","last_active":"2024-04-24T23:27:45.467387+0000","last_peered":"2024-04-24T23:27:45.467387+0000","last_clean":"2024-04-24T23:27:45.467387+0000","last_became_active":"2024-04-24T23:15:41.015179+0000","last_became_peered":"2024-04-24T23:15:41.015179+0000","last_unstale":"2024-04-24T23:27:45.467387+0000","last_undegraded":"2024-04-24T23:27:45.467387+0000","last_fullsized":"2024-04-24T23:27:45.467387+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":4365,"log_dups_size":0,"ondisk_log_size":4365,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:18:32.208506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88163170,"num_objects":26,"num_object_clones":0,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":76,"num_read_kb":0,"num_write":63916,"num_write_kb":903699,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5,0],"acting":[4,3,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"30'7","reported_seq":44,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:33.442182+0000","last_change":"2024-04-24T23:15:36.999334+0000","last_active":"2024-04-24T23:27:33.442182+0000","last_peered":"2024-04-24T23:27:33.442182+0000","last_clean":"2024-04-24T23:27:33.442182+0000","last_became_active":"2024-04-24T23:15:36.995470+0000","last_became_peered":"2024-04-24T23:15:36.995470+0000","last_unstale":"2024-04-24T23:27:33.442182+0000","last_undegraded":"2024-04-24T23:27:33.442182+0000","last_fullsized":"2024-04-24T23:27:33.442182+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:19:57.973402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.15","version":"30'20","reported_seq":62,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:41.283118+0000","last_change":"2024-04-24T23:15:35.946041+0000","last_active":"2024-04-24T23:27:41.283118+0000","last_peered":"2024-04-24T23:27:41.283118+0000","last_clean":"2024-04-24T23:27:41.283118+0000","last_became_active":"2024-04-24T23:15:35.944805+0000","last_became_peered":"2024-04-24T23:15:35.944805+0000","last_unstale":"2024-04-24T23:27:41.283118+0000","last_undegraded":"2024-04-24T23:27:41.283118+0000","last_fullsized":"2024-04-24T23:27:41.283118+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:52:19.261306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":22,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"30'4418","reported_seq":4524,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:48.620175+0000","last_change":"2024-04-24T23:15:41.024392+0000","last_active":"2024-04-24T23:27:48.620175+0000","last_peered":"2024-04-24T23:27:48.620175+0000","last_clean":"2024-04-24T23:27:48.620175+0000","last_became_active":"2024-04-24T23:15:41.023951+0000","last_became_peered":"2024-04-24T23:15:41.023951+0000","last_unstale":"2024-04-24T23:27:48.620175+0000","last_undegraded":"2024-04-24T23:27:48.620175+0000","last_fullsized":"2024-04-24T23:27:48.620175+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":4418,"log_dups_size":0,"ondisk_log_size":4418,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12690417,"num_objects":13,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":74,"num_read_kb":0,"num_write":64487,"num_write_kb":925608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1,7],"acting":[2,3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.17","version":"30'11","reported_seq":48,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:40.897196+0000","last_change":"2024-04-24T23:15:36.992243+0000","last_active":"2024-04-24T23:27:40.897196+0000","last_peered":"2024-04-24T23:27:40.897196+0000","last_clean":"2024-04-24T23:27:40.897196+0000","last_became_active":"2024-04-24T23:15:36.992002+0000","last_became_peered":"2024-04-24T23:15:36.992002+0000","last_unstale":"2024-04-24T23:27:40.897196+0000","last_undegraded":"2024-04-24T23:27:40.897196+0000","last_fullsized":"2024-04-24T23:27:40.897196+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:52:15.911009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"28'5","reported_seq":51,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:37.844845+0000","last_change":"2024-04-24T23:15:35.967408+0000","last_active":"2024-04-24T23:27:37.844845+0000","last_peered":"2024-04-24T23:27:37.844845+0000","last_clean":"2024-04-24T23:27:37.844845+0000","last_became_active":"2024-04-24T23:15:35.966646+0000","last_became_peered":"2024-04-24T23:15:35.966646+0000","last_unstale":"2024-04-24T23:27:37.844845+0000","last_undegraded":"2024-04-24T23:27:37.844845+0000","last_fullsized":"2024-04-24T23:27:37.844845+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":19,"num_read_kb":18,"num_write":7,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"30'4310","reported_seq":4409,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:47.329134+0000","last_change":"2024-04-24T23:15:41.021697+0000","last_active":"2024-04-24T23:27:47.329134+0000","last_peered":"2024-04-24T23:27:47.329134+0000","last_clean":"2024-04-24T23:27:47.329134+0000","last_became_active":"2024-04-24T23:15:41.021281+0000","last_became_peered":"2024-04-24T23:15:41.021281+0000","last_unstale":"2024-04-24T23:27:47.329134+0000","last_undegraded":"2024-04-24T23:27:47.329134+0000","last_fullsized":"2024-04-24T23:27:47.329134+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":4310,"log_dups_size":0,"ondisk_log_size":4310,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:59:37.306593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33622912,"num_objects":15,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":68,"num_read_kb":0,"num_write":62887,"num_write_kb":881151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3,2],"acting":[7,1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"30'7","reported_seq":45,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:44.521915+0000","last_change":"2024-04-24T23:15:36.990988+0000","last_active":"2024-04-24T23:27:44.521915+0000","last_peered":"2024-04-24T23:27:44.521915+0000","last_clean":"2024-04-24T23:27:44.521915+0000","last_became_active":"2024-04-24T23:15:36.990711+0000","last_became_peered":"2024-04-24T23:15:36.990711+0000","last_unstale":"2024-04-24T23:27:44.521915+0000","last_undegraded":"2024-04-24T23:27:44.521915+0000","last_fullsized":"2024-04-24T23:27:44.521915+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.17","version":"30'16","reported_seq":67,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:48.035149+0000","last_change":"2024-04-24T23:27:48.035149+0000","last_active":"2024-04-24T23:27:48.035149+0000","last_peered":"2024-04-24T23:27:48.035149+0000","last_clean":"2024-04-24T23:27:48.035149+0000","last_became_active":"2024-04-24T23:15:35.956870+0000","last_became_peered":"2024-04-24T23:15:35.956870+0000","last_unstale":"2024-04-24T23:27:48.035149+0000","last_undegraded":"2024-04-24T23:27:48.035149+0000","last_fullsized":"2024-04-24T23:27:48.035149+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'16","last_scrub_stamp":"2024-04-24T23:27:48.035025+0000","last_deep_scrub":"30'16","last_deep_scrub_stamp":"2024-04-24T23:27:48.035025+0000","last_clean_scrub_stamp":"2024-04-24T23:27:48.035025+0000","objects_scrubbed":3,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:12:15.713654+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":13,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"30'4390","reported_seq":4485,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:45.725642+0000","last_change":"2024-04-24T23:15:41.006098+0000","last_active":"2024-04-24T23:27:45.725642+0000","last_peered":"2024-04-24T23:27:45.725642+0000","last_clean":"2024-04-24T23:27:45.725642+0000","last_became_active":"2024-04-24T23:15:41.005578+0000","last_became_peered":"2024-04-24T23:15:41.005578+0000","last_unstale":"2024-04-24T23:27:45.725642+0000","last_undegraded":"2024-04-24T23:27:45.725642+0000","last_fullsized":"2024-04-24T23:27:45.725642+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":4390,"log_dups_size":0,"ondisk_log_size":4390,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:21:12.802112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50362001,"num_objects":18,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":64,"num_read_kb":0,"num_write":64099,"num_write_kb":908953,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3,7],"acting":[4,0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1e","version":"30'5155","reported_seq":5270,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:46.026163+0000","last_change":"2024-04-24T23:15:41.022122+0000","last_active":"2024-04-24T23:27:46.026163+0000","last_peered":"2024-04-24T23:27:46.026163+0000","last_clean":"2024-04-24T23:27:46.026163+0000","last_became_active":"2024-04-24T23:15:41.021698+0000","last_became_peered":"2024-04-24T23:15:41.021698+0000","last_unstale":"2024-04-24T23:27:46.026163+0000","last_undegraded":"2024-04-24T23:27:46.026163+0000","last_fullsized":"2024-04-24T23:27:46.026163+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":5155,"log_dups_size":0,"ondisk_log_size":5155,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:13:56.730050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92286090,"num_objects":25,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":83,"num_read_kb":0,"num_write":75863,"num_write_kb":1069592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5,3],"acting":[4,1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"27'3","reported_seq":40,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:25:42.710811+0000","last_change":"2024-04-24T23:15:36.979816+0000","last_active":"2024-04-24T23:25:42.710811+0000","last_peered":"2024-04-24T23:25:42.710811+0000","last_clean":"2024-04-24T23:25:42.710811+0000","last_became_active":"2024-04-24T23:15:36.979553+0000","last_became_peered":"2024-04-24T23:15:36.979553+0000","last_unstale":"2024-04-24T23:25:42.710811+0000","last_undegraded":"2024-04-24T23:25:42.710811+0000","last_fullsized":"2024-04-24T23:25:42.710811+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:55:47.134320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.18","version":"29'33","reported_seq":73,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:25:53.591106+0000","last_change":"2024-04-24T23:15:35.964744+0000","last_active":"2024-04-24T23:25:53.591106+0000","last_peered":"2024-04-24T23:25:53.591106+0000","last_clean":"2024-04-24T23:25:53.591106+0000","last_became_active":"2024-04-24T23:15:35.964434+0000","last_became_peered":"2024-04-24T23:15:35.964434+0000","last_unstale":"2024-04-24T23:25:53.591106+0000","last_undegraded":"2024-04-24T23:25:53.591106+0000","last_fullsized":"2024-04-24T23:25:53.591106+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:26:32.587197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8389190,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":8192,"num_write":27,"num_write_kb":8217,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"30'4537","reported_seq":4637,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:47.513121+0000","last_change":"2024-04-24T23:15:41.022032+0000","last_active":"2024-04-24T23:27:47.513121+0000","last_peered":"2024-04-24T23:27:47.513121+0000","last_clean":"2024-04-24T23:27:47.513121+0000","last_became_active":"2024-04-24T23:15:41.021616+0000","last_became_peered":"2024-04-24T23:15:41.021616+0000","last_unstale":"2024-04-24T23:27:47.513121+0000","last_undegraded":"2024-04-24T23:27:47.513121+0000","last_fullsized":"2024-04-24T23:27:47.513121+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":4537,"log_dups_size":0,"ondisk_log_size":4537,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:08:59.242392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50414796,"num_objects":23,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":69,"num_read_kb":0,"num_write":66254,"num_write_kb":941784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5,1],"acting":[3,2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"30'16","reported_seq":53,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:40.680874+0000","last_change":"2024-04-24T23:15:36.967198+0000","last_active":"2024-04-24T23:27:40.680874+0000","last_peered":"2024-04-24T23:27:40.680874+0000","last_clean":"2024-04-24T23:27:40.680874+0000","last_became_active":"2024-04-24T23:15:36.966910+0000","last_became_peered":"2024-04-24T23:15:36.966910+0000","last_unstale":"2024-04-24T23:27:40.680874+0000","last_undegraded":"2024-04-24T23:27:40.680874+0000","last_fullsized":"2024-04-24T23:27:40.680874+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:51:19.788437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"28'6","reported_seq":47,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:25:42.740697+0000","last_change":"2024-04-24T23:15:35.967169+0000","last_active":"2024-04-24T23:25:42.740697+0000","last_peered":"2024-04-24T23:25:42.740697+0000","last_clean":"2024-04-24T23:25:42.740697+0000","last_became_active":"2024-04-24T23:15:35.966939+0000","last_became_peered":"2024-04-24T23:15:35.966939+0000","last_unstale":"2024-04-24T23:25:42.740697+0000","last_undegraded":"2024-04-24T23:25:42.740697+0000","last_fullsized":"2024-04-24T23:25:42.740697+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:38:55.460569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":14,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"30'17","reported_seq":56,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:26:59.813995+0000","last_change":"2024-04-24T23:15:35.971236+0000","last_active":"2024-04-24T23:26:59.813995+0000","last_peered":"2024-04-24T23:26:59.813995+0000","last_clean":"2024-04-24T23:26:59.813995+0000","last_became_active":"2024-04-24T23:15:35.970908+0000","last_became_peered":"2024-04-24T23:15:35.970908+0000","last_unstale":"2024-04-24T23:26:59.813995+0000","last_undegraded":"2024-04-24T23:26:59.813995+0000","last_fullsized":"2024-04-24T23:26:59.813995+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:14:15.548024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":1,"num_read_kb":4096,"num_write":17,"num_write_kb":4106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"30'9","reported_seq":45,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:25:42.714442+0000","last_change":"2024-04-24T23:15:36.988156+0000","last_active":"2024-04-24T23:25:42.714442+0000","last_peered":"2024-04-24T23:25:42.714442+0000","last_clean":"2024-04-24T23:25:42.714442+0000","last_became_active":"2024-04-24T23:15:36.987896+0000","last_became_peered":"2024-04-24T23:15:36.987896+0000","last_unstale":"2024-04-24T23:25:42.714442+0000","last_undegraded":"2024-04-24T23:25:42.714442+0000","last_fullsized":"2024-04-24T23:25:42.714442+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:22:06.869644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"30'3634","reported_seq":3724,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:45.599146+0000","last_change":"2024-04-24T23:15:40.997559+0000","last_active":"2024-04-24T23:27:45.599146+0000","last_peered":"2024-04-24T23:27:45.599146+0000","last_clean":"2024-04-24T23:27:45.599146+0000","last_became_active":"2024-04-24T23:15:40.997102+0000","last_became_peered":"2024-04-24T23:15:40.997102+0000","last_unstale":"2024-04-24T23:27:45.599146+0000","last_undegraded":"2024-04-24T23:27:45.599146+0000","last_fullsized":"2024-04-24T23:27:45.599146+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":3634,"log_dups_size":0,"ondisk_log_size":3634,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:25:02.046761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33556159,"num_objects":9,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":60,"num_read_kb":0,"num_write":53460,"num_write_kb":749552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,7],"acting":[4,6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"30'1217","reported_seq":1259,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:25:51.681033+0000","last_change":"2024-04-24T23:15:35.944842+0000","last_active":"2024-04-24T23:25:51.681033+0000","last_peered":"2024-04-24T23:25:51.681033+0000","last_clean":"2024-04-24T23:25:51.681033+0000","last_became_active":"2024-04-24T23:15:35.944563+0000","last_became_peered":"2024-04-24T23:15:35.944563+0000","last_unstale":"2024-04-24T23:25:51.681033+0000","last_undegraded":"2024-04-24T23:25:51.681033+0000","last_fullsized":"2024-04-24T23:25:51.681033+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:34.909237+0000","last_clean_scrub_stamp":"2024-04-24T23:15:34.909237+0000","objects_scrubbed":0,"log_size":1217,"log_dups_size":0,"ondisk_log_size":1217,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:40:58.933384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4195441,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":4,"num_read_kb":4098,"num_write":1212,"num_write_kb":5208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"30'10","reported_seq":47,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:43.083552+0000","last_change":"2024-04-24T23:15:36.981103+0000","last_active":"2024-04-24T23:27:43.083552+0000","last_peered":"2024-04-24T23:27:43.083552+0000","last_clean":"2024-04-24T23:27:43.083552+0000","last_became_active":"2024-04-24T23:15:36.979637+0000","last_became_peered":"2024-04-24T23:15:36.979637+0000","last_unstale":"2024-04-24T23:27:43.083552+0000","last_undegraded":"2024-04-24T23:27:43.083552+0000","last_fullsized":"2024-04-24T23:27:43.083552+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"30'5055","reported_seq":5162,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:47.062225+0000","last_change":"2024-04-24T23:15:41.017087+0000","last_active":"2024-04-24T23:27:47.062225+0000","last_peered":"2024-04-24T23:27:47.062225+0000","last_clean":"2024-04-24T23:27:47.062225+0000","last_became_active":"2024-04-24T23:15:41.016696+0000","last_became_peered":"2024-04-24T23:15:41.016696+0000","last_unstale":"2024-04-24T23:27:47.062225+0000","last_undegraded":"2024-04-24T23:27:47.062225+0000","last_fullsized":"2024-04-24T23:27:47.062225+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":5055,"log_dups_size":0,"ondisk_log_size":5055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:05:57.823447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58829312,"num_objects":21,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":77,"num_read_kb":0,"num_write":74035,"num_write_kb":1014885,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1,0],"acting":[7,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"30'13","reported_seq":59,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:49.024713+0000","last_change":"2024-04-24T23:27:49.024754+0000","last_active":"2024-04-24T23:27:49.024713+0000","last_peered":"2024-04-24T23:27:49.024713+0000","last_clean":"2024-04-24T23:27:49.024713+0000","last_became_active":"2024-04-24T23:15:36.969090+0000","last_became_peered":"2024-04-24T23:15:36.969090+0000","last_unstale":"2024-04-24T23:27:49.024713+0000","last_undegraded":"2024-04-24T23:27:49.024713+0000","last_fullsized":"2024-04-24T23:27:49.024713+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'13","last_scrub_stamp":"2024-04-24T23:27:49.024601+0000","last_deep_scrub":"30'13","last_deep_scrub_stamp":"2024-04-24T23:27:49.024601+0000","last_clean_scrub_stamp":"2024-04-24T23:27:49.024601+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:10:37.734772+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.21","version":"30'13","reported_seq":50,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:35.220391+0000","last_change":"2024-04-24T23:15:36.982528+0000","last_active":"2024-04-24T23:27:35.220391+0000","last_peered":"2024-04-24T23:27:35.220391+0000","last_clean":"2024-04-24T23:27:35.220391+0000","last_became_active":"2024-04-24T23:15:36.982299+0000","last_became_peered":"2024-04-24T23:15:36.982299+0000","last_unstale":"2024-04-24T23:27:35.220391+0000","last_undegraded":"2024-04-24T23:27:35.220391+0000","last_fullsized":"2024-04-24T23:27:35.220391+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:02:17.732324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.22","version":"30'12","reported_seq":49,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:38.464079+0000","last_change":"2024-04-24T23:15:36.989899+0000","last_active":"2024-04-24T23:27:38.464079+0000","last_peered":"2024-04-24T23:27:38.464079+0000","last_clean":"2024-04-24T23:27:38.464079+0000","last_became_active":"2024-04-24T23:15:36.989557+0000","last_became_peered":"2024-04-24T23:15:36.989557+0000","last_unstale":"2024-04-24T23:27:38.464079+0000","last_undegraded":"2024-04-24T23:27:38.464079+0000","last_fullsized":"2024-04-24T23:27:38.464079+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:15:13.513341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"30'9","reported_seq":46,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:26:05.334192+0000","last_change":"2024-04-24T23:15:36.969628+0000","last_active":"2024-04-24T23:26:05.334192+0000","last_peered":"2024-04-24T23:26:05.334192+0000","last_clean":"2024-04-24T23:26:05.334192+0000","last_became_active":"2024-04-24T23:15:36.969327+0000","last_became_peered":"2024-04-24T23:15:36.969327+0000","last_unstale":"2024-04-24T23:26:05.334192+0000","last_undegraded":"2024-04-24T23:26:05.334192+0000","last_fullsized":"2024-04-24T23:26:05.334192+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:08:47.021999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.24","version":"30'8","reported_seq":45,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:26:31.416932+0000","last_change":"2024-04-24T23:15:36.986343+0000","last_active":"2024-04-24T23:26:31.416932+0000","last_peered":"2024-04-24T23:26:31.416932+0000","last_clean":"2024-04-24T23:26:31.416932+0000","last_became_active":"2024-04-24T23:15:36.986026+0000","last_became_peered":"2024-04-24T23:15:36.986026+0000","last_unstale":"2024-04-24T23:26:31.416932+0000","last_undegraded":"2024-04-24T23:26:31.416932+0000","last_fullsized":"2024-04-24T23:26:31.416932+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:10:00.347826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"30'8","reported_seq":45,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:28.112837+0000","last_change":"2024-04-24T23:15:36.968538+0000","last_active":"2024-04-24T23:27:28.112837+0000","last_peered":"2024-04-24T23:27:28.112837+0000","last_clean":"2024-04-24T23:27:28.112837+0000","last_became_active":"2024-04-24T23:15:36.968263+0000","last_became_peered":"2024-04-24T23:15:36.968263+0000","last_unstale":"2024-04-24T23:27:28.112837+0000","last_undegraded":"2024-04-24T23:27:28.112837+0000","last_fullsized":"2024-04-24T23:27:28.112837+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:07:43.731738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.26","version":"30'6","reported_seq":44,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:45.397404+0000","last_change":"2024-04-24T23:15:36.991008+0000","last_active":"2024-04-24T23:27:45.397404+0000","last_peered":"2024-04-24T23:27:45.397404+0000","last_clean":"2024-04-24T23:27:45.397404+0000","last_became_active":"2024-04-24T23:15:36.990764+0000","last_became_peered":"2024-04-24T23:15:36.990764+0000","last_unstale":"2024-04-24T23:27:45.397404+0000","last_undegraded":"2024-04-24T23:27:45.397404+0000","last_fullsized":"2024-04-24T23:27:45.397404+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"30'10","reported_seq":47,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:26:31.284444+0000","last_change":"2024-04-24T23:15:36.993550+0000","last_active":"2024-04-24T23:26:31.284444+0000","last_peered":"2024-04-24T23:26:31.284444+0000","last_clean":"2024-04-24T23:26:31.284444+0000","last_became_active":"2024-04-24T23:15:36.993310+0000","last_became_peered":"2024-04-24T23:15:36.993310+0000","last_unstale":"2024-04-24T23:26:31.284444+0000","last_undegraded":"2024-04-24T23:26:31.284444+0000","last_fullsized":"2024-04-24T23:26:31.284444+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:21:20.617940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.28","version":"30'9","reported_seq":46,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:30.082036+0000","last_change":"2024-04-24T23:15:36.987810+0000","last_active":"2024-04-24T23:27:30.082036+0000","last_peered":"2024-04-24T23:27:30.082036+0000","last_clean":"2024-04-24T23:27:30.082036+0000","last_became_active":"2024-04-24T23:15:36.987540+0000","last_became_peered":"2024-04-24T23:15:36.987540+0000","last_unstale":"2024-04-24T23:27:30.082036+0000","last_undegraded":"2024-04-24T23:27:30.082036+0000","last_fullsized":"2024-04-24T23:27:30.082036+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:49:56.176021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.29","version":"30'14","reported_seq":51,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:32.149953+0000","last_change":"2024-04-24T23:15:36.981187+0000","last_active":"2024-04-24T23:27:32.149953+0000","last_peered":"2024-04-24T23:27:32.149953+0000","last_clean":"2024-04-24T23:27:32.149953+0000","last_became_active":"2024-04-24T23:15:36.980838+0000","last_became_peered":"2024-04-24T23:15:36.980838+0000","last_unstale":"2024-04-24T23:27:32.149953+0000","last_undegraded":"2024-04-24T23:27:32.149953+0000","last_fullsized":"2024-04-24T23:27:32.149953+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:57:41.178367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2a","version":"30'12","reported_seq":50,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:43.627094+0000","last_change":"2024-04-24T23:15:36.999238+0000","last_active":"2024-04-24T23:27:43.627094+0000","last_peered":"2024-04-24T23:27:43.627094+0000","last_clean":"2024-04-24T23:27:43.627094+0000","last_became_active":"2024-04-24T23:15:36.996245+0000","last_became_peered":"2024-04-24T23:15:36.996245+0000","last_unstale":"2024-04-24T23:27:43.627094+0000","last_undegraded":"2024-04-24T23:27:43.627094+0000","last_fullsized":"2024-04-24T23:27:43.627094+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"30'10","reported_seq":47,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:30.193701+0000","last_change":"2024-04-24T23:15:36.981219+0000","last_active":"2024-04-24T23:27:30.193701+0000","last_peered":"2024-04-24T23:27:30.193701+0000","last_clean":"2024-04-24T23:27:30.193701+0000","last_became_active":"2024-04-24T23:15:36.980911+0000","last_became_peered":"2024-04-24T23:15:36.980911+0000","last_unstale":"2024-04-24T23:27:30.193701+0000","last_undegraded":"2024-04-24T23:27:30.193701+0000","last_fullsized":"2024-04-24T23:27:30.193701+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:19:57.973402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2c","version":"30'7","reported_seq":45,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:29.052763+0000","last_change":"2024-04-24T23:15:36.991418+0000","last_active":"2024-04-24T23:27:29.052763+0000","last_peered":"2024-04-24T23:27:29.052763+0000","last_clean":"2024-04-24T23:27:29.052763+0000","last_became_active":"2024-04-24T23:15:36.989792+0000","last_became_peered":"2024-04-24T23:15:36.989792+0000","last_unstale":"2024-04-24T23:27:29.052763+0000","last_undegraded":"2024-04-24T23:27:29.052763+0000","last_fullsized":"2024-04-24T23:27:29.052763+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2d","version":"30'12","reported_seq":49,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:28.000118+0000","last_change":"2024-04-24T23:15:36.987810+0000","last_active":"2024-04-24T23:27:28.000118+0000","last_peered":"2024-04-24T23:27:28.000118+0000","last_clean":"2024-04-24T23:27:28.000118+0000","last_became_active":"2024-04-24T23:15:36.987494+0000","last_became_peered":"2024-04-24T23:15:36.987494+0000","last_unstale":"2024-04-24T23:27:28.000118+0000","last_undegraded":"2024-04-24T23:27:28.000118+0000","last_fullsized":"2024-04-24T23:27:28.000118+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:43:21.141172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2e","version":"30'10","reported_seq":47,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:17.418750+0000","last_change":"2024-04-24T23:15:36.986576+0000","last_active":"2024-04-24T23:27:17.418750+0000","last_peered":"2024-04-24T23:27:17.418750+0000","last_clean":"2024-04-24T23:27:17.418750+0000","last_became_active":"2024-04-24T23:15:36.986272+0000","last_became_peered":"2024-04-24T23:15:36.986272+0000","last_unstale":"2024-04-24T23:27:17.418750+0000","last_undegraded":"2024-04-24T23:27:17.418750+0000","last_fullsized":"2024-04-24T23:27:17.418750+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:55:08.894945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2f","version":"30'10","reported_seq":52,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:41.130664+0000","last_change":"2024-04-24T23:16:18.294151+0000","last_active":"2024-04-24T23:27:41.130664+0000","last_peered":"2024-04-24T23:27:41.130664+0000","last_clean":"2024-04-24T23:27:41.130664+0000","last_became_active":"2024-04-24T23:16:18.293579+0000","last_became_peered":"2024-04-24T23:16:18.293579+0000","last_unstale":"2024-04-24T23:27:41.130664+0000","last_undegraded":"2024-04-24T23:27:41.130664+0000","last_fullsized":"2024-04-24T23:27:41.130664+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:22:20.603555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.30","version":"30'10","reported_seq":47,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:26.414660+0000","last_change":"2024-04-24T23:15:36.996521+0000","last_active":"2024-04-24T23:27:26.414660+0000","last_peered":"2024-04-24T23:27:26.414660+0000","last_clean":"2024-04-24T23:27:26.414660+0000","last_became_active":"2024-04-24T23:15:36.995507+0000","last_became_peered":"2024-04-24T23:15:36.995507+0000","last_unstale":"2024-04-24T23:27:26.414660+0000","last_undegraded":"2024-04-24T23:27:26.414660+0000","last_fullsized":"2024-04-24T23:27:26.414660+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:58:50.153471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.31","version":"30'12","reported_seq":49,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:35.692982+0000","last_change":"2024-04-24T23:15:36.984994+0000","last_active":"2024-04-24T23:27:35.692982+0000","last_peered":"2024-04-24T23:27:35.692982+0000","last_clean":"2024-04-24T23:27:35.692982+0000","last_became_active":"2024-04-24T23:15:36.984741+0000","last_became_peered":"2024-04-24T23:15:36.984741+0000","last_unstale":"2024-04-24T23:27:35.692982+0000","last_undegraded":"2024-04-24T23:27:35.692982+0000","last_fullsized":"2024-04-24T23:27:35.692982+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:45:23.429836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.32","version":"30'11","reported_seq":48,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:26.511382+0000","last_change":"2024-04-24T23:15:36.966165+0000","last_active":"2024-04-24T23:27:26.511382+0000","last_peered":"2024-04-24T23:27:26.511382+0000","last_clean":"2024-04-24T23:27:26.511382+0000","last_became_active":"2024-04-24T23:15:36.965877+0000","last_became_peered":"2024-04-24T23:15:36.965877+0000","last_unstale":"2024-04-24T23:27:26.511382+0000","last_undegraded":"2024-04-24T23:27:26.511382+0000","last_fullsized":"2024-04-24T23:27:26.511382+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:30:12.390978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.33","version":"30'9","reported_seq":46,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:35.584542+0000","last_change":"2024-04-24T23:15:36.990667+0000","last_active":"2024-04-24T23:27:35.584542+0000","last_peered":"2024-04-24T23:27:35.584542+0000","last_clean":"2024-04-24T23:27:35.584542+0000","last_became_active":"2024-04-24T23:15:36.990383+0000","last_became_peered":"2024-04-24T23:15:36.990383+0000","last_unstale":"2024-04-24T23:27:35.584542+0000","last_undegraded":"2024-04-24T23:27:35.584542+0000","last_fullsized":"2024-04-24T23:27:35.584542+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:36:43.850046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"30'7","reported_seq":45,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:38.854162+0000","last_change":"2024-04-24T23:15:36.991686+0000","last_active":"2024-04-24T23:27:38.854162+0000","last_peered":"2024-04-24T23:27:38.854162+0000","last_clean":"2024-04-24T23:27:38.854162+0000","last_became_active":"2024-04-24T23:15:36.990155+0000","last_became_peered":"2024-04-24T23:15:36.990155+0000","last_unstale":"2024-04-24T23:27:38.854162+0000","last_undegraded":"2024-04-24T23:27:38.854162+0000","last_fullsized":"2024-04-24T23:27:38.854162+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.35","version":"30'12","reported_seq":49,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:35.979063+0000","last_change":"2024-04-24T23:15:36.992958+0000","last_active":"2024-04-24T23:27:35.979063+0000","last_peered":"2024-04-24T23:27:35.979063+0000","last_clean":"2024-04-24T23:27:35.979063+0000","last_became_active":"2024-04-24T23:15:36.992608+0000","last_became_peered":"2024-04-24T23:15:36.992608+0000","last_unstale":"2024-04-24T23:27:35.979063+0000","last_undegraded":"2024-04-24T23:27:35.979063+0000","last_fullsized":"2024-04-24T23:27:35.979063+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.36","version":"30'10","reported_seq":47,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:30.771914+0000","last_change":"2024-04-24T23:15:36.992943+0000","last_active":"2024-04-24T23:27:30.771914+0000","last_peered":"2024-04-24T23:27:30.771914+0000","last_clean":"2024-04-24T23:27:30.771914+0000","last_became_active":"2024-04-24T23:15:36.992567+0000","last_became_peered":"2024-04-24T23:15:36.992567+0000","last_unstale":"2024-04-24T23:27:30.771914+0000","last_undegraded":"2024-04-24T23:27:30.771914+0000","last_fullsized":"2024-04-24T23:27:30.771914+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.37","version":"30'15","reported_seq":52,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:45.945908+0000","last_change":"2024-04-24T23:15:36.999697+0000","last_active":"2024-04-24T23:27:45.945908+0000","last_peered":"2024-04-24T23:27:45.945908+0000","last_clean":"2024-04-24T23:27:45.945908+0000","last_became_active":"2024-04-24T23:15:36.999413+0000","last_became_peered":"2024-04-24T23:15:36.999413+0000","last_unstale":"2024-04-24T23:27:45.945908+0000","last_undegraded":"2024-04-24T23:27:45.945908+0000","last_fullsized":"2024-04-24T23:27:45.945908+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"27'3","reported_seq":39,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:25:42.717468+0000","last_change":"2024-04-24T23:15:36.986742+0000","last_active":"2024-04-24T23:25:42.717468+0000","last_peered":"2024-04-24T23:25:42.717468+0000","last_clean":"2024-04-24T23:25:42.717468+0000","last_became_active":"2024-04-24T23:15:36.986013+0000","last_became_peered":"2024-04-24T23:15:36.986013+0000","last_unstale":"2024-04-24T23:25:42.717468+0000","last_undegraded":"2024-04-24T23:25:42.717468+0000","last_fullsized":"2024-04-24T23:25:42.717468+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:16:15.203151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"30'5","reported_seq":42,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:40.251498+0000","last_change":"2024-04-24T23:15:36.984336+0000","last_active":"2024-04-24T23:27:40.251498+0000","last_peered":"2024-04-24T23:27:40.251498+0000","last_clean":"2024-04-24T23:27:40.251498+0000","last_became_active":"2024-04-24T23:15:36.983966+0000","last_became_peered":"2024-04-24T23:15:36.983966+0000","last_unstale":"2024-04-24T23:27:40.251498+0000","last_undegraded":"2024-04-24T23:27:40.251498+0000","last_fullsized":"2024-04-24T23:27:40.251498+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3a","version":"30'9","reported_seq":46,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:38.669215+0000","last_change":"2024-04-24T23:15:36.979079+0000","last_active":"2024-04-24T23:27:38.669215+0000","last_peered":"2024-04-24T23:27:38.669215+0000","last_clean":"2024-04-24T23:27:38.669215+0000","last_became_active":"2024-04-24T23:15:36.978706+0000","last_became_peered":"2024-04-24T23:15:36.978706+0000","last_unstale":"2024-04-24T23:27:38.669215+0000","last_undegraded":"2024-04-24T23:27:38.669215+0000","last_fullsized":"2024-04-24T23:27:38.669215+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"30'7","reported_seq":44,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:42.248802+0000","last_change":"2024-04-24T23:15:36.986008+0000","last_active":"2024-04-24T23:27:42.248802+0000","last_peered":"2024-04-24T23:27:42.248802+0000","last_clean":"2024-04-24T23:27:42.248802+0000","last_became_active":"2024-04-24T23:15:36.985552+0000","last_became_peered":"2024-04-24T23:15:36.985552+0000","last_unstale":"2024-04-24T23:27:42.248802+0000","last_undegraded":"2024-04-24T23:27:42.248802+0000","last_fullsized":"2024-04-24T23:27:42.248802+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"30'7","reported_seq":46,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:47.981768+0000","last_change":"2024-04-24T23:15:36.978673+0000","last_active":"2024-04-24T23:27:47.981768+0000","last_peered":"2024-04-24T23:27:47.981768+0000","last_clean":"2024-04-24T23:27:47.981768+0000","last_became_active":"2024-04-24T23:15:36.972744+0000","last_became_peered":"2024-04-24T23:15:36.972744+0000","last_unstale":"2024-04-24T23:27:47.981768+0000","last_undegraded":"2024-04-24T23:27:47.981768+0000","last_fullsized":"2024-04-24T23:27:47.981768+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3d","version":"30'8","reported_seq":45,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:27.328493+0000","last_change":"2024-04-24T23:15:36.970238+0000","last_active":"2024-04-24T23:27:27.328493+0000","last_peered":"2024-04-24T23:27:27.328493+0000","last_clean":"2024-04-24T23:27:27.328493+0000","last_became_active":"2024-04-24T23:15:36.969751+0000","last_became_peered":"2024-04-24T23:15:36.969751+0000","last_unstale":"2024-04-24T23:27:27.328493+0000","last_undegraded":"2024-04-24T23:27:27.328493+0000","last_fullsized":"2024-04-24T23:27:27.328493+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:19:56.456801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3e","version":"26'2","reported_seq":39,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:25:42.719286+0000","last_change":"2024-04-24T23:15:36.968825+0000","last_active":"2024-04-24T23:25:42.719286+0000","last_peered":"2024-04-24T23:25:42.719286+0000","last_clean":"2024-04-24T23:25:42.719286+0000","last_became_active":"2024-04-24T23:15:36.968543+0000","last_became_peered":"2024-04-24T23:15:36.968543+0000","last_unstale":"2024-04-24T23:25:42.719286+0000","last_undegraded":"2024-04-24T23:25:42.719286+0000","last_fullsized":"2024-04-24T23:25:42.719286+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:36:48.795062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3f","version":"27'4","reported_seq":42,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:26:07.299333+0000","last_change":"2024-04-24T23:15:36.992800+0000","last_active":"2024-04-24T23:26:07.299333+0000","last_peered":"2024-04-24T23:26:07.299333+0000","last_clean":"2024-04-24T23:26:07.299333+0000","last_became_active":"2024-04-24T23:15:36.992476+0000","last_became_peered":"2024-04-24T23:15:36.992476+0000","last_unstale":"2024-04-24T23:26:07.299333+0000","last_undegraded":"2024-04-24T23:26:07.299333+0000","last_fullsized":"2024-04-24T23:26:07.299333+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":1378189189,"num_objects":519,"num_object_clones":0,"num_object_copies":2076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":519,"num_whiteouts":0,"num_read":2252,"num_read_kb":307,"num_write":2060223,"num_write_kb":29192446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2842279936,"data_stored":2842279936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":141077,"ondisk_log_size":141077,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":566,"ondisk_log_size":566,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":80027978,"num_objects":127,"num_object_clones":0,"num_object_copies":254,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":127,"num_whiteouts":0,"num_read":353,"num_read_kb":84742,"num_write":4104,"num_write_kb":81390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":74,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":160292864,"data_stored":160055956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3848,"ondisk_log_size":3848,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575000,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1428920,"kb_used_data":401008,"kb_used_omap":1,"kb_used_meta":1027902,"kb_avail":92942920,"statfs":{"total":96636764160,"available":95173550080,"internally_reserved":0,"allocated":410632192,"data_stored":410410944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":1052572107},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834575000,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1387380,"kb_used_data":437484,"kb_used_omap":1,"kb_used_meta":949886,"kb_avail":92984460,"statfs":{"total":96636764160,"available":95216087040,"internally_reserved":0,"allocated":447983616,"data_stored":447782208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":972683718},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834575000,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1430940,"kb_used_data":616276,"kb_used_omap":1,"kb_used_meta":814654,"kb_avail":92940900,"statfs":{"total":96636764160,"available":95171481600,"internally_reserved":0,"allocated":631066624,"data_stored":630854399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":834206149},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834575001,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1205208,"kb_used_data":408528,"kb_used_omap":1,"kb_used_meta":796670,"kb_avail":93166632,"statfs":{"total":96636764160,"available":95402631168,"internally_reserved":0,"allocated":418332672,"data_stored":418096536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":815790533},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574996,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1647520,"kb_used_data":597144,"kb_used_omap":1,"kb_used_meta":1050366,"kb_avail":92724320,"statfs":{"total":96636764160,"available":94949703680,"internally_reserved":0,"allocated":611475456,"data_stored":611249299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":1075575239},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574996,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":947892,"kb_used_data":201644,"kb_used_omap":1,"kb_used_meta":746238,"kb_avail":93423948,"statfs":{"total":96636764160,"available":95666122752,"internally_reserved":0,"allocated":206483456,"data_stored":206282414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":764148165},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574996,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1446044,"kb_used_data":387412,"kb_used_omap":1,"kb_used_meta":1058622,"kb_avail":92925796,"statfs":{"total":96636764160,"available":95156015104,"internally_reserved":0,"allocated":396709888,"data_stored":396500263,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1084029381},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574999,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":980256,"kb_used_data":308312,"kb_used_omap":1,"kb_used_meta":671934,"kb_avail":93391584,"statfs":{"total":96636764160,"available":95632982016,"internally_reserved":0,"allocated":315711488,"data_stored":315492133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":688060870},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":25436160,"data_stored":25402203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":21225472,"data_stored":21204797,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12599296,"data_stored":12583652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":37793792,"data_stored":37753033,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12636160,"data_stored":12585422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":25284608,"data_stored":25261333,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12603392,"data_stored":12587382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12713984,"data_stored":12678134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":224935936,"data_stored":224935936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":374562816,"data_stored":374562816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":157904896,"data_stored":157904896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":424456192,"data_stored":424456192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":388591616,"data_stored":388591616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":455962624,"data_stored":455962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":435052544,"data_stored":435052544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":380813312,"data_stored":380813312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T23:27:52.833 INFO:tasks.ceph:pgid 2.1f last_scrub_stamp 2024-04-24T23:15:34.909237+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=15, tm_sec=34, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=27, tm_sec=44, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T23:27:52.833 INFO:tasks.ceph:pgid 3.1e last_scrub_stamp 2024-04-24T23:15:35.921164+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=15, tm_sec=35, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=27, tm_sec=44, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T23:27:52.833 INFO:tasks.ceph:pgid 4.19 last_scrub_stamp 2024-04-24T23:15:39.973013+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=15, tm_sec=39, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=27, tm_sec=44, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T23:27:52.833 INFO:tasks.ceph:pgid 2.1e last_scrub_stamp 2024-04-24T23:15:34.909237+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=15, tm_sec=34, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=27, tm_sec=44, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T23:27:52.833 INFO:tasks.ceph:pgid 3.1f last_scrub_stamp 2024-04-24T23:15:35.921164+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=15, tm_sec=35, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=27, tm_sec=44, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T23:27:52.833 INFO:tasks.ceph:pgid 4.18 last_scrub_stamp 2024-04-24T23:15:39.973013+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=15, tm_sec=39, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=27, tm_sec=44, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T23:27:52.833 INFO:tasks.ceph:pgid 2.1d last_scrub_stamp 2024-04-24T23:15:34.909237+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=15, tm_sec=34, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=27, tm_sec=44, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T23:27:52.833 INFO:tasks.ceph:pgid 3.1c last_scrub_stamp 2024-04-24T23:15:35.921164+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=15, tm_sec=35, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=27, tm_sec=44, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T23:27:52.834 INFO:tasks.ceph:pgid 4.1b last_scrub_stamp 2024-04-24T23:15:39.973013+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=15, tm_sec=39, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=27, tm_sec=44, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T23:27:52.834 INFO:tasks.ceph:pgid 2.1c last_scrub_stamp 2024-04-24T23:15:34.909237+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=15, tm_sec=34, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=27, tm_sec=44, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T23:27:52.834 INFO:tasks.ceph:pgid 3.1d last_scrub_stamp 2024-04-24T23:15:35.921164+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=15, tm_sec=35, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=27, tm_sec=44, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T23:27:52.834 INFO:tasks.ceph:pgid 4.1a last_scrub_stamp 2024-04-24T23:15:39.973013+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=15, tm_sec=39, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=27, tm_sec=44, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T23:27:52.834 INFO:tasks.ceph:pgid 3.a last_scrub_stamp 2024-04-24T23:15:35.921164+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=15, tm_sec=35, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=27, tm_sec=44, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T23:27:52.834 INFO:tasks.ceph:pgid 2.b last_scrub_stamp 2024-04-24T23:15:34.909237+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=15, tm_sec=34, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=27, tm_sec=44, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T23:27:52.834 INFO:tasks.ceph:pgid 4.d last_scrub_stamp 2024-04-24T23:15:39.973013+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=15, tm_sec=39, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=27, tm_sec=44, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T23:27:52.834 INFO:tasks.ceph:pgid 3.b last_scrub_stamp 2024-04-24T23:15:35.921164+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=15, tm_sec=35, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=27, tm_sec=44, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T23:27:52.834 INFO:tasks.ceph:pgid 2.a last_scrub_stamp 2024-04-24T23:15:34.909237+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=15, tm_sec=34, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=27, tm_sec=44, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T23:27:52.835 INFO:tasks.ceph:pgid 4.c last_scrub_stamp 2024-04-24T23:15:39.973013+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=15, tm_sec=39, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=27, tm_sec=44, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T23:27:52.835 INFO:tasks.ceph:pgid 3.8 last_scrub_stamp 2024-04-24T23:15:35.921164+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=15, tm_sec=35, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=27, tm_sec=44, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T23:27:52.835 INFO:tasks.ceph:pgid 2.9 last_scrub_stamp 2024-04-24T23:15:34.909237+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=15, tm_sec=34, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=27, tm_sec=44, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T23:27:52.835 INFO:tasks.ceph:pgid 4.f last_scrub_stamp 2024-04-24T23:15:39.973013+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=15, tm_sec=39, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=27, tm_sec=44, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T23:27:52.835 INFO:tasks.ceph:pgid 3.9 last_scrub_stamp 2024-04-24T23:15:35.921164+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=15, tm_sec=35, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=27, tm_sec=44, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T23:27:52.835 INFO:tasks.ceph:pgid 2.8 last_scrub_stamp 2024-04-24T23:15:34.909237+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=15, tm_sec=34, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=27, tm_sec=44, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T23:27:52.835 INFO:tasks.ceph:pgid 4.e last_scrub_stamp 2024-04-24T23:15:39.973013+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=15, tm_sec=39, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=27, tm_sec=44, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T23:27:52.835 INFO:tasks.ceph:pgid 3.6 last_scrub_stamp 2024-04-24T23:15:35.921164+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=15, tm_sec=35, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=27, tm_sec=44, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T23:27:52.835 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-04-24T23:15:34.909237+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=15, tm_sec=34, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=27, tm_sec=44, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T23:27:52.836 INFO:tasks.ceph:pgid 4.1 last_scrub_stamp 2024-04-24T23:15:39.973013+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=15, tm_sec=39, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=27, tm_sec=44, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T23:27:52.836 INFO:tasks.ceph:pgid 3.7 last_scrub_stamp 2024-04-24T23:15:35.921164+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=15, tm_sec=35, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=27, tm_sec=44, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T23:27:52.836 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-04-24T23:15:34.909237+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=15, tm_sec=34, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=27, tm_sec=44, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T23:27:52.836 INFO:tasks.ceph:pgid 4.0 last_scrub_stamp 2024-04-24T23:15:39.973013+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=15, tm_sec=39, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=27, tm_sec=44, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T23:27:52.836 INFO:tasks.ceph:pgid 3.4 last_scrub_stamp 2024-04-24T23:15:35.921164+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=15, tm_sec=35, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=27, tm_sec=44, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T23:27:52.836 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-04-24T23:15:34.909237+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=15, tm_sec=34, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=27, tm_sec=44, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T23:27:52.836 INFO:tasks.ceph:pgid 4.3 last_scrub_stamp 2024-04-24T23:15:39.973013+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=15, tm_sec=39, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=27, tm_sec=44, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T23:27:52.836 INFO:tasks.ceph:pgid 3.5 last_scrub_stamp 2024-04-24T23:15:35.921164+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=15, tm_sec=35, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=27, tm_sec=44, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T23:27:52.836 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-04-24T23:15:34.909237+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=15, tm_sec=34, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=27, tm_sec=44, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T23:27:52.837 INFO:tasks.ceph:pgid 4.2 last_scrub_stamp 2024-04-24T23:15:39.973013+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=15, tm_sec=39, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=27, tm_sec=44, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T23:27:52.837 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2024-04-24T23:15:35.921164+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=15, tm_sec=35, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=27, tm_sec=44, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T23:27:52.837 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-04-24T23:15:34.909237+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=15, tm_sec=34, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=27, tm_sec=44, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T23:27:52.837 INFO:tasks.ceph:pgid 4.4 last_scrub_stamp 2024-04-24T23:15:39.973013+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=15, tm_sec=39, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=27, tm_sec=44, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T23:27:52.837 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2024-04-24T23:15:35.921164+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=15, tm_sec=35, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=27, tm_sec=44, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T23:27:52.837 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-04-24T23:15:34.909237+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=15, tm_sec=34, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=27, tm_sec=44, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T23:27:52.837 INFO:tasks.ceph:pgid 4.7 last_scrub_stamp 2024-04-24T23:15:39.973013+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=15, tm_sec=39, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=27, tm_sec=44, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T23:27:52.837 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2024-04-24T23:15:35.921164+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=15, tm_sec=35, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=27, tm_sec=44, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T23:27:52.837 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-04-24T23:15:34.909237+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=15, tm_sec=34, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=27, tm_sec=44, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T23:27:52.837 INFO:tasks.ceph:pgid 4.6 last_scrub_stamp 2024-04-24T23:15:39.973013+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=15, tm_sec=39, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=27, tm_sec=44, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T23:27:52.838 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2024-04-24T23:15:35.921164+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=15, tm_sec=35, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=27, tm_sec=44, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T23:27:52.838 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-04-24T23:15:34.909237+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=15, tm_sec=34, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=27, tm_sec=44, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T23:27:52.838 INFO:tasks.ceph:pgid 4.5 last_scrub_stamp 2024-04-24T23:15:39.973013+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=15, tm_sec=39, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=27, tm_sec=44, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T23:27:52.838 INFO:tasks.ceph:pgid 3.d last_scrub_stamp 2024-04-24T23:15:35.921164+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=15, tm_sec=35, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=27, tm_sec=44, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T23:27:52.838 INFO:tasks.ceph:pgid 2.c last_scrub_stamp 2024-04-24T23:15:34.909237+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=15, tm_sec=34, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=27, tm_sec=44, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T23:27:52.838 INFO:tasks.ceph:pgid 4.a last_scrub_stamp 2024-04-24T23:15:39.973013+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=15, tm_sec=39, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=27, tm_sec=44, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T23:27:52.838 INFO:tasks.ceph:pgid 3.c last_scrub_stamp 2024-04-24T23:15:35.921164+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=15, tm_sec=35, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=27, tm_sec=44, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T23:27:52.838 INFO:tasks.ceph:pgid 2.d last_scrub_stamp 2024-04-24T23:15:34.909237+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=15, tm_sec=34, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=27, tm_sec=44, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T23:27:52.838 INFO:tasks.ceph:pgid 4.b last_scrub_stamp 2024-04-24T23:15:39.973013+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=15, tm_sec=39, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=27, tm_sec=44, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T23:27:52.838 INFO:tasks.ceph:pgid 3.f last_scrub_stamp 2024-04-24T23:15:35.921164+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=15, tm_sec=35, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=27, tm_sec=44, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T23:27:52.839 INFO:tasks.ceph:pgid 2.e last_scrub_stamp 2024-04-24T23:15:34.909237+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=15, tm_sec=34, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=27, tm_sec=44, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T23:27:52.839 INFO:tasks.ceph:pgid 4.8 last_scrub_stamp 2024-04-24T23:15:39.973013+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=15, tm_sec=39, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=27, tm_sec=44, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T23:27:52.839 INFO:tasks.ceph:pgid 3.e last_scrub_stamp 2024-04-24T23:15:35.921164+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=15, tm_sec=35, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=27, tm_sec=44, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T23:27:52.839 INFO:tasks.ceph:pgid 2.f last_scrub_stamp 2024-04-24T23:15:34.909237+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=15, tm_sec=34, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=27, tm_sec=44, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T23:27:52.839 INFO:tasks.ceph:pgid 4.9 last_scrub_stamp 2024-04-24T23:15:39.973013+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=15, tm_sec=39, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=27, tm_sec=44, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T23:27:52.839 INFO:tasks.ceph:pgid 3.11 last_scrub_stamp 2024-04-24T23:15:35.921164+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=15, tm_sec=35, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=27, tm_sec=44, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T23:27:52.839 INFO:tasks.ceph:pgid 2.10 last_scrub_stamp 2024-04-24T23:15:34.909237+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=15, tm_sec=34, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=27, tm_sec=44, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T23:27:52.839 INFO:tasks.ceph:pgid 4.16 last_scrub_stamp 2024-04-24T23:15:39.973013+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=15, tm_sec=39, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=27, tm_sec=44, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T23:27:52.839 INFO:tasks.ceph:pgid 3.10 last_scrub_stamp 2024-04-24T23:15:35.921164+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=15, tm_sec=35, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=27, tm_sec=44, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T23:27:52.840 INFO:tasks.ceph:pgid 2.11 last_scrub_stamp 2024-04-24T23:15:34.909237+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=15, tm_sec=34, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=27, tm_sec=44, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T23:27:52.840 INFO:tasks.ceph:pgid 4.17 last_scrub_stamp 2024-04-24T23:15:39.973013+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=15, tm_sec=39, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=27, tm_sec=44, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T23:27:52.840 INFO:tasks.ceph:pgid 3.13 last_scrub_stamp 2024-04-24T23:15:35.921164+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=15, tm_sec=35, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=27, tm_sec=44, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T23:27:52.840 INFO:tasks.ceph:pgid 2.12 last_scrub_stamp 2024-04-24T23:15:34.909237+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=15, tm_sec=34, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=27, tm_sec=44, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T23:27:52.840 INFO:tasks.ceph:pgid 4.14 last_scrub_stamp 2024-04-24T23:15:39.973013+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=15, tm_sec=39, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=27, tm_sec=44, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T23:27:52.840 INFO:tasks.ceph:pgid 3.12 last_scrub_stamp 2024-04-24T23:15:35.921164+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=15, tm_sec=35, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=27, tm_sec=44, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T23:27:52.840 INFO:tasks.ceph:pgid 2.13 last_scrub_stamp 2024-04-24T23:15:34.909237+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=15, tm_sec=34, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=27, tm_sec=44, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T23:27:52.840 INFO:tasks.ceph:pgid 4.15 last_scrub_stamp 2024-04-24T23:15:39.973013+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=15, tm_sec=39, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=27, tm_sec=44, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T23:27:52.840 INFO:tasks.ceph:pgid 4.12 last_scrub_stamp 2024-04-24T23:15:39.973013+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=15, tm_sec=39, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=27, tm_sec=44, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T23:27:52.841 INFO:tasks.ceph:pgid 3.14 last_scrub_stamp 2024-04-24T23:15:35.921164+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=15, tm_sec=35, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=27, tm_sec=44, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T23:27:52.841 INFO:tasks.ceph:pgid 2.15 last_scrub_stamp 2024-04-24T23:15:34.909237+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=15, tm_sec=34, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=27, tm_sec=44, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T23:27:52.841 INFO:tasks.ceph:pgid 4.13 last_scrub_stamp 2024-04-24T23:15:39.973013+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=15, tm_sec=39, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=27, tm_sec=44, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T23:27:52.841 INFO:tasks.ceph:pgid 3.17 last_scrub_stamp 2024-04-24T23:15:35.921164+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=15, tm_sec=35, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=27, tm_sec=44, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T23:27:52.841 INFO:tasks.ceph:pgid 2.16 last_scrub_stamp 2024-04-24T23:15:34.909237+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=15, tm_sec=34, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=27, tm_sec=44, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T23:27:52.841 INFO:tasks.ceph:pgid 4.10 last_scrub_stamp 2024-04-24T23:15:39.973013+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=15, tm_sec=39, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=27, tm_sec=44, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T23:27:52.841 INFO:tasks.ceph:pgid 3.16 last_scrub_stamp 2024-04-24T23:15:35.921164+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=15, tm_sec=35, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=27, tm_sec=44, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T23:27:52.841 INFO:tasks.ceph:pgid 4.11 last_scrub_stamp 2024-04-24T23:15:39.973013+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=15, tm_sec=39, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=27, tm_sec=44, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T23:27:52.841 INFO:tasks.ceph:pgid 4.1e last_scrub_stamp 2024-04-24T23:15:39.973013+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=15, tm_sec=39, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=27, tm_sec=44, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T23:27:52.841 INFO:tasks.ceph:pgid 3.19 last_scrub_stamp 2024-04-24T23:15:35.921164+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=15, tm_sec=35, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=27, tm_sec=44, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T23:27:52.842 INFO:tasks.ceph:pgid 2.18 last_scrub_stamp 2024-04-24T23:15:34.909237+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=15, tm_sec=34, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=27, tm_sec=44, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T23:27:52.842 INFO:tasks.ceph:pgid 4.1f last_scrub_stamp 2024-04-24T23:15:39.973013+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=15, tm_sec=39, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=27, tm_sec=44, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T23:27:52.842 INFO:tasks.ceph:pgid 3.18 last_scrub_stamp 2024-04-24T23:15:35.921164+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=15, tm_sec=35, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=27, tm_sec=44, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T23:27:52.842 INFO:tasks.ceph:pgid 2.19 last_scrub_stamp 2024-04-24T23:15:34.909237+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=15, tm_sec=34, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=27, tm_sec=44, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T23:27:52.842 INFO:tasks.ceph:pgid 2.1a last_scrub_stamp 2024-04-24T23:15:34.909237+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=15, tm_sec=34, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=27, tm_sec=44, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T23:27:52.842 INFO:tasks.ceph:pgid 3.1b last_scrub_stamp 2024-04-24T23:15:35.921164+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=15, tm_sec=35, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=27, tm_sec=44, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T23:27:52.842 INFO:tasks.ceph:pgid 4.1c last_scrub_stamp 2024-04-24T23:15:39.973013+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=15, tm_sec=39, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=27, tm_sec=44, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T23:27:52.842 INFO:tasks.ceph:pgid 2.1b last_scrub_stamp 2024-04-24T23:15:34.909237+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=15, tm_sec=34, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=27, tm_sec=44, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T23:27:52.842 INFO:tasks.ceph:pgid 3.1a last_scrub_stamp 2024-04-24T23:15:35.921164+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=15, tm_sec=35, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=27, tm_sec=44, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T23:27:52.843 INFO:tasks.ceph:pgid 4.1d last_scrub_stamp 2024-04-24T23:15:39.973013+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=15, tm_sec=39, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=27, tm_sec=44, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T23:27:52.843 INFO:tasks.ceph:pgid 3.21 last_scrub_stamp 2024-04-24T23:15:35.921164+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=15, tm_sec=35, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=27, tm_sec=44, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T23:27:52.843 INFO:tasks.ceph:pgid 3.22 last_scrub_stamp 2024-04-24T23:15:35.921164+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=15, tm_sec=35, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=27, tm_sec=44, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T23:27:52.843 INFO:tasks.ceph:pgid 3.23 last_scrub_stamp 2024-04-24T23:15:35.921164+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=15, tm_sec=35, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=27, tm_sec=44, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T23:27:52.843 INFO:tasks.ceph:pgid 3.24 last_scrub_stamp 2024-04-24T23:15:35.921164+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=15, tm_sec=35, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=27, tm_sec=44, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T23:27:52.843 INFO:tasks.ceph:pgid 3.25 last_scrub_stamp 2024-04-24T23:15:35.921164+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=15, tm_sec=35, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=27, tm_sec=44, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T23:27:52.843 INFO:tasks.ceph:pgid 3.26 last_scrub_stamp 2024-04-24T23:15:35.921164+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=15, tm_sec=35, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=27, tm_sec=44, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T23:27:52.843 INFO:tasks.ceph:pgid 3.27 last_scrub_stamp 2024-04-24T23:15:35.921164+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=15, tm_sec=35, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=27, tm_sec=44, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T23:27:52.843 INFO:tasks.ceph:pgid 3.28 last_scrub_stamp 2024-04-24T23:15:35.921164+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=15, tm_sec=35, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=27, tm_sec=44, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T23:27:52.844 INFO:tasks.ceph:pgid 3.29 last_scrub_stamp 2024-04-24T23:15:35.921164+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=15, tm_sec=35, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=27, tm_sec=44, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T23:27:52.844 INFO:tasks.ceph:pgid 3.2a last_scrub_stamp 2024-04-24T23:15:35.921164+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=15, tm_sec=35, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=27, tm_sec=44, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T23:27:52.844 INFO:tasks.ceph:pgid 3.2b last_scrub_stamp 2024-04-24T23:15:35.921164+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=15, tm_sec=35, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=27, tm_sec=44, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T23:27:52.844 INFO:tasks.ceph:pgid 3.2c last_scrub_stamp 2024-04-24T23:15:35.921164+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=15, tm_sec=35, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=27, tm_sec=44, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T23:27:52.844 INFO:tasks.ceph:pgid 3.2d last_scrub_stamp 2024-04-24T23:15:35.921164+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=15, tm_sec=35, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=27, tm_sec=44, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T23:27:52.844 INFO:tasks.ceph:pgid 3.2e last_scrub_stamp 2024-04-24T23:15:35.921164+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=15, tm_sec=35, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=27, tm_sec=44, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T23:27:52.844 INFO:tasks.ceph:pgid 3.2f last_scrub_stamp 2024-04-24T23:15:35.921164+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=15, tm_sec=35, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=27, tm_sec=44, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T23:27:52.844 INFO:tasks.ceph:pgid 3.30 last_scrub_stamp 2024-04-24T23:15:35.921164+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=15, tm_sec=35, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=27, tm_sec=44, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T23:27:52.845 INFO:tasks.ceph:pgid 3.31 last_scrub_stamp 2024-04-24T23:15:35.921164+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=15, tm_sec=35, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=27, tm_sec=44, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T23:27:52.845 INFO:tasks.ceph:pgid 3.32 last_scrub_stamp 2024-04-24T23:15:35.921164+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=15, tm_sec=35, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=27, tm_sec=44, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T23:27:52.845 INFO:tasks.ceph:pgid 3.33 last_scrub_stamp 2024-04-24T23:15:35.921164+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=15, tm_sec=35, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=27, tm_sec=44, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T23:27:52.845 INFO:tasks.ceph:pgid 3.34 last_scrub_stamp 2024-04-24T23:15:35.921164+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=15, tm_sec=35, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=27, tm_sec=44, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T23:27:52.845 INFO:tasks.ceph:pgid 3.35 last_scrub_stamp 2024-04-24T23:15:35.921164+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=15, tm_sec=35, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=27, tm_sec=44, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T23:27:52.845 INFO:tasks.ceph:pgid 3.36 last_scrub_stamp 2024-04-24T23:15:35.921164+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=15, tm_sec=35, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=27, tm_sec=44, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T23:27:52.845 INFO:tasks.ceph:pgid 3.37 last_scrub_stamp 2024-04-24T23:15:35.921164+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=15, tm_sec=35, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=27, tm_sec=44, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T23:27:52.845 INFO:tasks.ceph:pgid 3.38 last_scrub_stamp 2024-04-24T23:15:35.921164+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=15, tm_sec=35, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=27, tm_sec=44, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T23:27:52.845 INFO:tasks.ceph:pgid 3.39 last_scrub_stamp 2024-04-24T23:15:35.921164+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=15, tm_sec=35, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=27, tm_sec=44, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T23:27:52.845 INFO:tasks.ceph:pgid 3.3a last_scrub_stamp 2024-04-24T23:15:35.921164+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=15, tm_sec=35, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=27, tm_sec=44, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T23:27:52.846 INFO:tasks.ceph:pgid 3.3b last_scrub_stamp 2024-04-24T23:15:35.921164+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=15, tm_sec=35, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=27, tm_sec=44, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T23:27:52.846 INFO:tasks.ceph:pgid 3.3c last_scrub_stamp 2024-04-24T23:15:35.921164+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=15, tm_sec=35, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=27, tm_sec=44, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T23:27:52.846 INFO:tasks.ceph:pgid 3.3d last_scrub_stamp 2024-04-24T23:15:35.921164+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=15, tm_sec=35, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=27, tm_sec=44, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T23:27:52.846 INFO:tasks.ceph:pgid 3.3e last_scrub_stamp 2024-04-24T23:15:35.921164+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=15, tm_sec=35, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=27, tm_sec=44, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T23:27:52.846 INFO:tasks.ceph:pgid 3.3f last_scrub_stamp 2024-04-24T23:15:35.921164+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=15, tm_sec=35, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=27, tm_sec=44, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T23:27:52.846 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-04-24T23:28:12.847 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T23:28:13.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:13.038+0000 7fadcd000640 1 Processor -- start 2024-04-24T23:28:13.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:13.038+0000 7fadcd000640 1 -- start start 2024-04-24T23:28:13.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:13.039+0000 7fadcd000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fadc80c7d00 0x7fadc8071fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:28:13.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:13.040+0000 7fadcd000640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fadc806ba60 0x7fadc8158d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:28:13.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:13.040+0000 7fadcd000640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fadc806c010 con 0x7fadc8152900 2024-04-24T23:28:13.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:13.040+0000 7fadcd000640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7fadc8158870 con 0x7fadc80c7d00 2024-04-24T23:28:13.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:13.040+0000 7fadcd000640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fadc80732c0 con 0x7fadc806ba60 2024-04-24T23:28:13.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:13.040+0000 7fadc7e00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fadc806ba60 0x7fadc8158d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:28:13.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:13.040+0000 7fadc7e00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fadc806ba60 0x7fadc8158d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.6:48180/0 (socket says 172.21.15.6:48180) 2024-04-24T23:28:13.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:13.040+0000 7fadc7400640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7fadc8152900 0x7fadc80c6a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.6:34554/0 (socket says 172.21.15.6:34554) 2024-04-24T23:28:13.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:13.040+0000 7fadc7e00640 1 -- 172.21.15.6:0/3413708558 learned_addr learned my addr 172.21.15.6:0/3413708558 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:28:13.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:13.040+0000 7fadc6a00640 1 --2- 172.21.15.6:0/3413708558 >> v2:172.21.15.6:3300/0 conn(0x7fadc80c7d00 0x7fadc8071fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:28:13.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:13.040+0000 7fadc7e00640 1 -- 172.21.15.6:0/3413708558 >> v2:172.21.15.6:3300/0 conn(0x7fadc80c7d00 msgr2=0x7fadc8071fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:28:13.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:13.040+0000 7fadc7e00640 1 --2- 172.21.15.6:0/3413708558 >> v2:172.21.15.6:3300/0 conn(0x7fadc80c7d00 0x7fadc8071fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:28:13.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:13.040+0000 7fadc7e00640 1 -- 172.21.15.6:0/3413708558 >> v1:172.21.15.190:6789/0 conn(0x7fadc8152900 legacy=0x7fadc80c6a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:28:13.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:13.040+0000 7fadc6000640 1 -- 172.21.15.6:0/3413708558 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2897475667 0 0) 0x7fadb8001110 con 0x7fadc8152900 2024-04-24T23:28:13.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:13.040+0000 7fadc7e00640 1 -- 172.21.15.6:0/3413708558 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fadc8072a20 con 0x7fadc806ba60 2024-04-24T23:28:13.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:13.040+0000 7fadc6a00640 1 --2- 172.21.15.6:0/3413708558 >> v2:172.21.15.6:3300/0 conn(0x7fadc80c7d00 0x7fadc8071fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:28:13.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:13.041+0000 7fadc7e00640 1 --2- 172.21.15.6:0/3413708558 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fadc806ba60 0x7fadc8158d00 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fadc0009e50 tx=0x7fadc002a680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=709f8b12b24a669d server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:28:13.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:13.042+0000 7fadc6000640 1 -- 172.21.15.6:0/3413708558 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadc0036070 con 0x7fadc806ba60 2024-04-24T23:28:13.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:13.042+0000 7fadc6000640 1 -- 172.21.15.6:0/3413708558 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fadc0031600 con 0x7fadc806ba60 2024-04-24T23:28:13.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:13.042+0000 7fadc6000640 1 -- 172.21.15.6:0/3413708558 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadc00319d0 con 0x7fadc806ba60 2024-04-24T23:28:13.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:13.042+0000 7fadcd000640 1 -- 172.21.15.6:0/3413708558 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fadc806ba60 msgr2=0x7fadc8158d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:28:13.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:13.042+0000 7fadcd000640 1 --2- 172.21.15.6:0/3413708558 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fadc806ba60 0x7fadc8158d00 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fadc0009e50 tx=0x7fadc002a680 comp rx=0 tx=0).stop 2024-04-24T23:28:13.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:13.042+0000 7fadcd000640 1 -- 172.21.15.6:0/3413708558 shutdown_connections 2024-04-24T23:28:13.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:13.042+0000 7fadcd000640 1 --2- 172.21.15.6:0/3413708558 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fadc806ba60 0x7fadc8158d00 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:28:13.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:13.042+0000 7fadcd000640 1 --2- 172.21.15.6:0/3413708558 >> v2:172.21.15.6:3300/0 conn(0x7fadc80c7d00 0x7fadc8071fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:28:13.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:13.042+0000 7fadcd000640 1 -- 172.21.15.6:0/3413708558 >> 172.21.15.6:0/3413708558 conn(0x7fadc806e590 msgr2=0x7fadc80cd300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:28:13.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:13.042+0000 7fadcd000640 1 -- 172.21.15.6:0/3413708558 shutdown_connections 2024-04-24T23:28:13.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:13.042+0000 7fadcd000640 1 -- 172.21.15.6:0/3413708558 wait complete. 2024-04-24T23:28:13.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:13.044+0000 7fadcd000640 1 Processor -- start 2024-04-24T23:28:13.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:13.044+0000 7fadcd000640 1 -- start start 2024-04-24T23:28:13.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:13.045+0000 7fadcd000640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fadc806ba60 0x7fadc81ef070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:28:13.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:13.045+0000 7fadcd000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fadc80c7d00 0x7fadc815bee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:28:13.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:13.045+0000 7fadcd000640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fadc8152900 0x7fadc8154990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:28:13.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:13.045+0000 7fadcd000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fadc8152ce0 con 0x7fadc80c7d00 2024-04-24T23:28:13.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:13.045+0000 7fadcd000640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fadc8072fe0 con 0x7fadc806ba60 2024-04-24T23:28:13.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:13.045+0000 7fadcd000640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fadc80c7120 con 0x7fadc8152900 2024-04-24T23:28:13.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:13.045+0000 7fadc7400640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fadc806ba60 0x7fadc81ef070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:28:13.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:13.045+0000 7fadc6a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fadc80c7d00 0x7fadc815bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:28:13.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:13.045+0000 7fadc7e00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fadc8152900 0x7fadc8154990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:28:13.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:13.045+0000 7fadc6a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fadc80c7d00 0x7fadc815bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:49994/0 (socket says 172.21.15.6:49994) 2024-04-24T23:28:13.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:13.045+0000 7fadc6a00640 1 -- 172.21.15.6:0/1723762996 learned_addr learned my addr 172.21.15.6:0/1723762996 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:28:13.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:13.046+0000 7fadc6a00640 1 -- 172.21.15.6:0/1723762996 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fadc8152900 msgr2=0x7fadc8154990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:28:13.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:13.046+0000 7fadc6a00640 1 --2- 172.21.15.6:0/1723762996 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fadc8152900 0x7fadc8154990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:28:13.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:13.046+0000 7fadc6a00640 1 -- 172.21.15.6:0/1723762996 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fadc806ba60 msgr2=0x7fadc81ef070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:28:13.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:13.046+0000 7fadc6a00640 1 --2- 172.21.15.6:0/1723762996 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fadc806ba60 0x7fadc81ef070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:28:13.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:13.046+0000 7fadc6a00640 1 -- 172.21.15.6:0/1723762996 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fadc8153060 con 0x7fadc80c7d00 2024-04-24T23:28:13.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:13.046+0000 7fadc7e00640 1 --2- 172.21.15.6:0/1723762996 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fadc8152900 0x7fadc8154990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:28:13.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:13.046+0000 7fadc6a00640 1 --2- 172.21.15.6:0/1723762996 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fadc80c7d00 0x7fadc815bee0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fadb400e810 tx=0x7fadb400ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:28:13.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:13.046+0000 7fadbf400640 1 -- 172.21.15.6:0/1723762996 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadb400f040 con 0x7fadc80c7d00 2024-04-24T23:28:13.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:13.046+0000 7fadbf400640 1 -- 172.21.15.6:0/1723762996 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fadb4004030 con 0x7fadc80c7d00 2024-04-24T23:28:13.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:13.046+0000 7fadbf400640 1 -- 172.21.15.6:0/1723762996 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadb4004400 con 0x7fadc80c7d00 2024-04-24T23:28:13.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:13.046+0000 7fadcd000640 1 -- 172.21.15.6:0/1723762996 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fadc8153340 con 0x7fadc80c7d00 2024-04-24T23:28:13.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:13.047+0000 7fadcd000640 1 -- 172.21.15.6:0/1723762996 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fadc8153900 con 0x7fadc80c7d00 2024-04-24T23:28:13.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:13.048+0000 7fadcd000640 1 -- 172.21.15.6:0/1723762996 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fadc80cd800 con 0x7fadc80c7d00 2024-04-24T23:28:13.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:13.049+0000 7fadbf400640 1 -- 172.21.15.6:0/1723762996 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fadb400c6d0 con 0x7fadc80c7d00 2024-04-24T23:28:13.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:13.049+0000 7fadbf400640 1 --2- 172.21.15.6:0/1723762996 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fada806be60 0x7fada806e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:28:13.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:13.050+0000 7fadbf400640 1 -- 172.21.15.6:0/1723762996 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6934+0+0 (secure 0 0 0) 0x7fadb408b150 con 0x7fadc80c7d00 2024-04-24T23:28:13.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:13.050+0000 7fadc7400640 1 --2- 172.21.15.6:0/1723762996 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fada806be60 0x7fada806e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:28:13.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:13.051+0000 7fadc7400640 1 --2- 172.21.15.6:0/1723762996 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fada806be60 0x7fada806e320 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fadb8007fd0 tx=0x7fadb8007de0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:28:13.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:13.054+0000 7fadbf400640 1 -- 172.21.15.6:0/1723762996 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fadc80cd800 con 0x7fadc80c7d00 2024-04-24T23:28:13.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:13.260+0000 7fadcd000640 1 -- 172.21.15.6:0/1723762996 --> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fadc8072540 con 0x7fada806be60 2024-04-24T23:28:13.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:13.270+0000 7fadbf400640 1 -- 172.21.15.6:0/1723762996 <== mgr.4105 v2:172.21.15.190:6832/806769663 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+334771 (secure 0 0 0) 0x7fadc8072540 con 0x7fada806be60 2024-04-24T23:28:13.274 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:28:13.276 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-04-24T23:28:13.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:13.275+0000 7fadcd000640 1 -- 172.21.15.6:0/1723762996 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fada806be60 msgr2=0x7fada806e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:28:13.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:13.275+0000 7fadcd000640 1 --2- 172.21.15.6:0/1723762996 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fada806be60 0x7fada806e320 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fadb8007fd0 tx=0x7fadb8007de0 comp rx=0 tx=0).stop 2024-04-24T23:28:13.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:13.275+0000 7fadcd000640 1 -- 172.21.15.6:0/1723762996 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fadc80c7d00 msgr2=0x7fadc815bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:28:13.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:13.275+0000 7fadcd000640 1 --2- 172.21.15.6:0/1723762996 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fadc80c7d00 0x7fadc815bee0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fadb400e810 tx=0x7fadb400ecd0 comp rx=0 tx=0).stop 2024-04-24T23:28:13.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:13.276+0000 7fadcd000640 1 -- 172.21.15.6:0/1723762996 shutdown_connections 2024-04-24T23:28:13.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:13.276+0000 7fadcd000640 1 --2- 172.21.15.6:0/1723762996 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fada806be60 0x7fada806e320 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:28:13.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:13.276+0000 7fadcd000640 1 --2- 172.21.15.6:0/1723762996 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fadc8152900 0x7fadc8154990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:28:13.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:13.276+0000 7fadcd000640 1 --2- 172.21.15.6:0/1723762996 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fadc80c7d00 0x7fadc815bee0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:28:13.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:13.276+0000 7fadcd000640 1 --2- 172.21.15.6:0/1723762996 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fadc806ba60 0x7fadc81ef070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:28:13.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:13.276+0000 7fadcd000640 1 -- 172.21.15.6:0/1723762996 >> 172.21.15.6:0/1723762996 conn(0x7fadc806e590 msgr2=0x7fadc8151430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:28:13.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:13.276+0000 7fadcd000640 1 -- 172.21.15.6:0/1723762996 shutdown_connections 2024-04-24T23:28:13.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:13.276+0000 7fadcd000640 1 -- 172.21.15.6:0/1723762996 wait complete. 2024-04-24T23:28:13.294 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":407,"stamp":"2024-04-24T23:28:12.879921+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":80618346,"num_objects":120,"num_object_clones":0,"num_object_copies":240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":120,"num_whiteouts":0,"num_read":2721,"num_read_kb":85149,"num_write":2065015,"num_write_kb":29276253,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25000,"num_omap_keys":50,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":146532,"ondisk_log_size":146532,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":7311296,"kb_used_data":160256,"kb_used_omap":12,"kb_used_meta":7150963,"kb_avail":747663424,"statfs":{"total":773094113280,"available":765607346176,"internally_reserved":0,"allocated":164102144,"data_stored":162374948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":7322586674},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002612"},"pg_stats":[{"pgid":"2.1f","version":"30'43","reported_seq":97,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:53.185318+0000","last_change":"2024-04-24T23:27:53.185318+0000","last_active":"2024-04-24T23:27:53.185318+0000","last_peered":"2024-04-24T23:27:53.185318+0000","last_clean":"2024-04-24T23:27:53.185318+0000","last_became_active":"2024-04-24T23:15:35.964357+0000","last_became_peered":"2024-04-24T23:15:35.964357+0000","last_unstale":"2024-04-24T23:27:53.185318+0000","last_undegraded":"2024-04-24T23:27:53.185318+0000","last_fullsized":"2024-04-24T23:27:53.185318+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'43","last_scrub_stamp":"2024-04-24T23:27:53.185145+0000","last_deep_scrub":"30'43","last_deep_scrub_stamp":"2024-04-24T23:27:53.185145+0000","last_clean_scrub_stamp":"2024-04-24T23:27:53.185145+0000","objects_scrubbed":7,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:37:01.920409+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194976,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":7,"num_read_kb":4102,"num_write":33,"num_write_kb":4110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"30'4","reported_seq":51,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:28:03.564026+0000","last_change":"2024-04-24T23:28:03.564062+0000","last_active":"2024-04-24T23:28:03.564026+0000","last_peered":"2024-04-24T23:28:03.564026+0000","last_clean":"2024-04-24T23:28:03.564026+0000","last_became_active":"2024-04-24T23:15:36.989543+0000","last_became_peered":"2024-04-24T23:15:36.989543+0000","last_unstale":"2024-04-24T23:28:03.564026+0000","last_undegraded":"2024-04-24T23:28:03.564026+0000","last_fullsized":"2024-04-24T23:28:03.564026+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'4","last_scrub_stamp":"2024-04-24T23:28:03.563929+0000","last_deep_scrub":"30'4","last_deep_scrub_stamp":"2024-04-24T23:28:03.563929+0000","last_clean_scrub_stamp":"2024-04-24T23:28:03.563929+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:11:47.175305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"30'4772","reported_seq":4875,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:28:05.082558+0000","last_change":"2024-04-24T23:28:05.082601+0000","last_active":"2024-04-24T23:28:05.082558+0000","last_peered":"2024-04-24T23:28:05.082558+0000","last_clean":"2024-04-24T23:28:05.082558+0000","last_became_active":"2024-04-24T23:16:18.299217+0000","last_became_peered":"2024-04-24T23:16:18.299217+0000","last_unstale":"2024-04-24T23:28:05.082558+0000","last_undegraded":"2024-04-24T23:28:05.082558+0000","last_fullsized":"2024-04-24T23:28:05.082558+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'4772","last_scrub_stamp":"2024-04-24T23:28:05.082434+0000","last_deep_scrub":"30'4772","last_deep_scrub_stamp":"2024-04-24T23:28:05.082434+0000","last_clean_scrub_stamp":"2024-04-24T23:28:05.082434+0000","objects_scrubbed":0,"log_size":4772,"log_dups_size":0,"ondisk_log_size":4772,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:17:51.312590+0000","scrub_duration":45,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":70,"num_read_kb":0,"num_write":69587,"num_write_kb":969009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4,1],"acting":[0,6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"30'15","reported_seq":65,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:57.556371+0000","last_change":"2024-04-24T23:27:57.556402+0000","last_active":"2024-04-24T23:27:57.556371+0000","last_peered":"2024-04-24T23:27:57.556371+0000","last_clean":"2024-04-24T23:27:57.556371+0000","last_became_active":"2024-04-24T23:15:35.945227+0000","last_became_peered":"2024-04-24T23:15:35.945227+0000","last_unstale":"2024-04-24T23:27:57.556371+0000","last_undegraded":"2024-04-24T23:27:57.556371+0000","last_fullsized":"2024-04-24T23:27:57.556371+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'15","last_scrub_stamp":"2024-04-24T23:27:57.556297+0000","last_deep_scrub":"30'15","last_deep_scrub_stamp":"2024-04-24T23:27:57.556297+0000","last_clean_scrub_stamp":"2024-04-24T23:27:57.556297+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:11:36.961216+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":4096,"num_write":8,"num_write_kb":4105,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1f","version":"30'15","reported_seq":62,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:28:06.488103+0000","last_change":"2024-04-24T23:28:06.488125+0000","last_active":"2024-04-24T23:28:06.488103+0000","last_peered":"2024-04-24T23:28:06.488103+0000","last_clean":"2024-04-24T23:28:06.488103+0000","last_became_active":"2024-04-24T23:15:36.998021+0000","last_became_peered":"2024-04-24T23:15:36.998021+0000","last_unstale":"2024-04-24T23:28:06.488103+0000","last_undegraded":"2024-04-24T23:28:06.488103+0000","last_fullsized":"2024-04-24T23:28:06.488103+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'15","last_scrub_stamp":"2024-04-24T23:28:06.488041+0000","last_deep_scrub":"30'15","last_deep_scrub_stamp":"2024-04-24T23:28:06.488041+0000","last_clean_scrub_stamp":"2024-04-24T23:28:06.488041+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:18:57.366081+0000","scrub_duration":45,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"30'4584","reported_seq":4699,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:56.164851+0000","last_change":"2024-04-24T23:27:56.164907+0000","last_active":"2024-04-24T23:27:56.164851+0000","last_peered":"2024-04-24T23:27:56.164851+0000","last_clean":"2024-04-24T23:27:56.164851+0000","last_became_active":"2024-04-24T23:15:41.007908+0000","last_became_peered":"2024-04-24T23:15:41.007908+0000","last_unstale":"2024-04-24T23:27:56.164851+0000","last_undegraded":"2024-04-24T23:27:56.164851+0000","last_fullsized":"2024-04-24T23:27:56.164851+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'4584","last_scrub_stamp":"2024-04-24T23:27:56.164679+0000","last_deep_scrub":"30'4584","last_deep_scrub_stamp":"2024-04-24T23:27:56.164679+0000","last_clean_scrub_stamp":"2024-04-24T23:27:56.164679+0000","objects_scrubbed":0,"log_size":4584,"log_dups_size":0,"ondisk_log_size":4584,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:30:37.102549+0000","scrub_duration":117,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":75,"num_read_kb":0,"num_write":66619,"num_write_kb":930809,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4,6],"acting":[2,7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1d","version":"30'53","reported_seq":114,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:53.103917+0000","last_change":"2024-04-24T23:27:53.104009+0000","last_active":"2024-04-24T23:27:53.103917+0000","last_peered":"2024-04-24T23:27:53.103917+0000","last_clean":"2024-04-24T23:27:53.103917+0000","last_became_active":"2024-04-24T23:15:35.967873+0000","last_became_peered":"2024-04-24T23:15:35.967873+0000","last_unstale":"2024-04-24T23:27:53.103917+0000","last_undegraded":"2024-04-24T23:27:53.103917+0000","last_fullsized":"2024-04-24T23:27:53.103917+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'53","last_scrub_stamp":"2024-04-24T23:27:53.103743+0000","last_deep_scrub":"30'53","last_deep_scrub_stamp":"2024-04-24T23:27:53.103743+0000","last_clean_scrub_stamp":"2024-04-24T23:27:53.103743+0000","objects_scrubbed":9,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:47:32.540188+0000","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":38,"num_read_kb":8226,"num_write":37,"num_write_kb":8205,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"30'8","reported_seq":53,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:50.990837+0000","last_change":"2024-04-24T23:27:50.990837+0000","last_active":"2024-04-24T23:27:50.990837+0000","last_peered":"2024-04-24T23:27:50.990837+0000","last_clean":"2024-04-24T23:27:50.990837+0000","last_became_active":"2024-04-24T23:15:36.997083+0000","last_became_peered":"2024-04-24T23:15:36.997083+0000","last_unstale":"2024-04-24T23:27:50.990837+0000","last_undegraded":"2024-04-24T23:27:50.990837+0000","last_fullsized":"2024-04-24T23:27:50.990837+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"27'7","last_scrub_stamp":"2024-04-24T23:27:50.990698+0000","last_deep_scrub":"27'7","last_deep_scrub_stamp":"2024-04-24T23:27:50.990698+0000","last_clean_scrub_stamp":"2024-04-24T23:27:50.990698+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:52:04.409030+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"30'4289","reported_seq":4398,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:28:00.134050+0000","last_change":"2024-04-24T23:28:00.134092+0000","last_active":"2024-04-24T23:28:00.134050+0000","last_peered":"2024-04-24T23:28:00.134050+0000","last_clean":"2024-04-24T23:28:00.134050+0000","last_became_active":"2024-04-24T23:15:41.024739+0000","last_became_peered":"2024-04-24T23:15:41.024739+0000","last_unstale":"2024-04-24T23:28:00.134050+0000","last_undegraded":"2024-04-24T23:28:00.134050+0000","last_fullsized":"2024-04-24T23:28:00.134050+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'4289","last_scrub_stamp":"2024-04-24T23:28:00.133912+0000","last_deep_scrub":"30'4289","last_deep_scrub_stamp":"2024-04-24T23:28:00.133912+0000","last_clean_scrub_stamp":"2024-04-24T23:28:00.133912+0000","objects_scrubbed":0,"log_size":4289,"log_dups_size":0,"ondisk_log_size":4289,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:46:24.969382+0000","scrub_duration":115,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":70,"num_read_kb":0,"num_write":62397,"num_write_kb":877947,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3,6],"acting":[2,5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1c","version":"30'21","reported_seq":77,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:51.356518+0000","last_change":"2024-04-24T23:27:51.145503+0000","last_active":"2024-04-24T23:27:51.356518+0000","last_peered":"2024-04-24T23:27:51.356518+0000","last_clean":"2024-04-24T23:27:51.356518+0000","last_became_active":"2024-04-24T23:15:35.966951+0000","last_became_peered":"2024-04-24T23:15:35.966951+0000","last_unstale":"2024-04-24T23:27:51.356518+0000","last_undegraded":"2024-04-24T23:27:51.356518+0000","last_fullsized":"2024-04-24T23:27:51.356518+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'20","last_scrub_stamp":"2024-04-24T23:27:51.145294+0000","last_deep_scrub":"30'20","last_deep_scrub_stamp":"2024-04-24T23:27:51.145294+0000","last_clean_scrub_stamp":"2024-04-24T23:27:51.145294+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:19:59.138191+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":232445,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":21,"num_read_kb":20,"num_write":17,"num_write_kb":234,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1d","version":"30'12","reported_seq":57,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:55.022971+0000","last_change":"2024-04-24T23:27:55.023031+0000","last_active":"2024-04-24T23:27:55.022971+0000","last_peered":"2024-04-24T23:27:55.022971+0000","last_clean":"2024-04-24T23:27:55.022971+0000","last_became_active":"2024-04-24T23:15:36.997735+0000","last_became_peered":"2024-04-24T23:15:36.997735+0000","last_unstale":"2024-04-24T23:27:55.022971+0000","last_undegraded":"2024-04-24T23:27:55.022971+0000","last_fullsized":"2024-04-24T23:27:55.022971+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'12","last_scrub_stamp":"2024-04-24T23:27:55.022792+0000","last_deep_scrub":"30'12","last_deep_scrub_stamp":"2024-04-24T23:27:55.022792+0000","last_clean_scrub_stamp":"2024-04-24T23:27:55.022792+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:20:58.930570+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"30'3772","reported_seq":3871,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:57.200767+0000","last_change":"2024-04-24T23:27:57.200767+0000","last_active":"2024-04-24T23:27:57.200767+0000","last_peered":"2024-04-24T23:27:57.200767+0000","last_clean":"2024-04-24T23:27:57.200767+0000","last_became_active":"2024-04-24T23:15:41.006199+0000","last_became_peered":"2024-04-24T23:15:41.006199+0000","last_unstale":"2024-04-24T23:27:57.200767+0000","last_undegraded":"2024-04-24T23:27:57.200767+0000","last_fullsized":"2024-04-24T23:27:57.200767+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3772","last_scrub_stamp":"2024-04-24T23:27:57.200437+0000","last_deep_scrub":"30'3772","last_deep_scrub_stamp":"2024-04-24T23:27:57.200437+0000","last_clean_scrub_stamp":"2024-04-24T23:27:57.200437+0000","objects_scrubbed":0,"log_size":3772,"log_dups_size":0,"ondisk_log_size":3772,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:28:20.543657+0000","scrub_duration":113,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":59,"num_read_kb":0,"num_write":54805,"num_write_kb":774507,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5,7],"acting":[2,6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"30'9","reported_seq":55,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:28:07.949858+0000","last_change":"2024-04-24T23:28:07.949896+0000","last_active":"2024-04-24T23:28:07.949858+0000","last_peered":"2024-04-24T23:28:07.949858+0000","last_clean":"2024-04-24T23:28:07.949858+0000","last_became_active":"2024-04-24T23:15:37.003544+0000","last_became_peered":"2024-04-24T23:15:37.003544+0000","last_unstale":"2024-04-24T23:28:07.949858+0000","last_undegraded":"2024-04-24T23:28:07.949858+0000","last_fullsized":"2024-04-24T23:28:07.949858+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'9","last_scrub_stamp":"2024-04-24T23:28:07.949765+0000","last_deep_scrub":"30'9","last_deep_scrub_stamp":"2024-04-24T23:28:07.949765+0000","last_clean_scrub_stamp":"2024-04-24T23:28:07.949765+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:24:58.985515+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"30'1465","reported_seq":1518,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:28:03.031075+0000","last_change":"2024-04-24T23:28:03.031123+0000","last_active":"2024-04-24T23:28:03.031075+0000","last_peered":"2024-04-24T23:28:03.031075+0000","last_clean":"2024-04-24T23:28:03.031075+0000","last_became_active":"2024-04-24T23:15:35.961960+0000","last_became_peered":"2024-04-24T23:15:35.961960+0000","last_unstale":"2024-04-24T23:28:03.031075+0000","last_undegraded":"2024-04-24T23:28:03.031075+0000","last_fullsized":"2024-04-24T23:28:03.031075+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'1465","last_scrub_stamp":"2024-04-24T23:28:03.030975+0000","last_deep_scrub":"30'1465","last_deep_scrub_stamp":"2024-04-24T23:28:03.030975+0000","last_clean_scrub_stamp":"2024-04-24T23:28:03.030975+0000","objects_scrubbed":3,"log_size":1465,"log_dups_size":0,"ondisk_log_size":1465,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:30:36.365621+0000","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":10,"num_read_kb":8200,"num_write":1452,"num_write_kb":5429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"30'4377","reported_seq":4482,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:28:11.986897+0000","last_change":"2024-04-24T23:28:11.987088+0000","last_active":"2024-04-24T23:28:11.986897+0000","last_peered":"2024-04-24T23:28:11.986897+0000","last_clean":"2024-04-24T23:28:11.986897+0000","last_became_active":"2024-04-24T23:15:41.020034+0000","last_became_peered":"2024-04-24T23:15:41.020034+0000","last_unstale":"2024-04-24T23:28:11.986897+0000","last_undegraded":"2024-04-24T23:28:11.986897+0000","last_fullsized":"2024-04-24T23:28:11.986897+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'4377","last_scrub_stamp":"2024-04-24T23:28:11.986573+0000","last_deep_scrub":"30'4377","last_deep_scrub_stamp":"2024-04-24T23:28:11.986573+0000","last_clean_scrub_stamp":"2024-04-24T23:28:11.986573+0000","objects_scrubbed":0,"log_size":4377,"log_dups_size":0,"ondisk_log_size":4377,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:33:35.191670+0000","scrub_duration":112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":65,"num_read_kb":0,"num_write":63662,"num_write_kb":904417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,5],"acting":[3,1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"30'9","reported_seq":55,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:55.457311+0000","last_change":"2024-04-24T23:27:55.457338+0000","last_active":"2024-04-24T23:27:55.457311+0000","last_peered":"2024-04-24T23:27:55.457311+0000","last_clean":"2024-04-24T23:27:55.457311+0000","last_became_active":"2024-04-24T23:15:36.971583+0000","last_became_peered":"2024-04-24T23:15:36.971583+0000","last_unstale":"2024-04-24T23:27:55.457311+0000","last_undegraded":"2024-04-24T23:27:55.457311+0000","last_fullsized":"2024-04-24T23:27:55.457311+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'9","last_scrub_stamp":"2024-04-24T23:27:55.457210+0000","last_deep_scrub":"30'9","last_deep_scrub_stamp":"2024-04-24T23:27:55.457210+0000","last_clean_scrub_stamp":"2024-04-24T23:27:55.457210+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:13:51.329870+0000","scrub_duration":57,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"30'12","reported_seq":61,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:52.425162+0000","last_change":"2024-04-24T23:27:52.425196+0000","last_active":"2024-04-24T23:27:52.425162+0000","last_peered":"2024-04-24T23:27:52.425162+0000","last_clean":"2024-04-24T23:27:52.425162+0000","last_became_active":"2024-04-24T23:15:35.945610+0000","last_became_peered":"2024-04-24T23:15:35.945610+0000","last_unstale":"2024-04-24T23:27:52.425162+0000","last_undegraded":"2024-04-24T23:27:52.425162+0000","last_fullsized":"2024-04-24T23:27:52.425162+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'12","last_scrub_stamp":"2024-04-24T23:27:52.425070+0000","last_deep_scrub":"30'12","last_deep_scrub_stamp":"2024-04-24T23:27:52.425070+0000","last_clean_scrub_stamp":"2024-04-24T23:27:52.425070+0000","objects_scrubbed":2,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:30:00.751396+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"30'4438","reported_seq":4548,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:28:04.149690+0000","last_change":"2024-04-24T23:28:04.149756+0000","last_active":"2024-04-24T23:28:04.149690+0000","last_peered":"2024-04-24T23:28:04.149690+0000","last_clean":"2024-04-24T23:28:04.149690+0000","last_became_active":"2024-04-24T23:15:41.029685+0000","last_became_peered":"2024-04-24T23:15:41.029685+0000","last_unstale":"2024-04-24T23:28:04.149690+0000","last_undegraded":"2024-04-24T23:28:04.149690+0000","last_fullsized":"2024-04-24T23:28:04.149690+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'4438","last_scrub_stamp":"2024-04-24T23:28:04.149507+0000","last_deep_scrub":"30'4438","last_deep_scrub_stamp":"2024-04-24T23:28:04.149507+0000","last_clean_scrub_stamp":"2024-04-24T23:28:04.149507+0000","objects_scrubbed":0,"log_size":4438,"log_dups_size":0,"ondisk_log_size":4438,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:08:29.255909+0000","scrub_duration":89,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":70,"num_read_kb":0,"num_write":64546,"num_write_kb":914755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6,1],"acting":[0,4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"30'9","reported_seq":55,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:57.213406+0000","last_change":"2024-04-24T23:27:57.213432+0000","last_active":"2024-04-24T23:27:57.213406+0000","last_peered":"2024-04-24T23:27:57.213406+0000","last_clean":"2024-04-24T23:27:57.213406+0000","last_became_active":"2024-04-24T23:15:36.965658+0000","last_became_peered":"2024-04-24T23:15:36.965658+0000","last_unstale":"2024-04-24T23:27:57.213406+0000","last_undegraded":"2024-04-24T23:27:57.213406+0000","last_fullsized":"2024-04-24T23:27:57.213406+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'9","last_scrub_stamp":"2024-04-24T23:27:57.213295+0000","last_deep_scrub":"30'9","last_deep_scrub_stamp":"2024-04-24T23:27:57.213295+0000","last_clean_scrub_stamp":"2024-04-24T23:27:57.213295+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:04:33.911020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"30'15","reported_seq":70,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:55.238261+0000","last_change":"2024-04-24T23:27:55.238323+0000","last_active":"2024-04-24T23:27:55.238261+0000","last_peered":"2024-04-24T23:27:55.238261+0000","last_clean":"2024-04-24T23:27:55.238261+0000","last_became_active":"2024-04-24T23:15:35.971972+0000","last_became_peered":"2024-04-24T23:15:35.971972+0000","last_unstale":"2024-04-24T23:27:55.238261+0000","last_undegraded":"2024-04-24T23:27:55.238261+0000","last_fullsized":"2024-04-24T23:27:55.238261+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'15","last_scrub_stamp":"2024-04-24T23:27:55.238155+0000","last_deep_scrub":"30'15","last_deep_scrub_stamp":"2024-04-24T23:27:55.238155+0000","last_clean_scrub_stamp":"2024-04-24T23:27:55.238155+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:39:46.964431+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":20,"num_read_kb":19,"num_write":11,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"30'5217","reported_seq":5338,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:59.524189+0000","last_change":"2024-04-24T23:27:59.524232+0000","last_active":"2024-04-24T23:27:59.524189+0000","last_peered":"2024-04-24T23:27:59.524189+0000","last_clean":"2024-04-24T23:27:59.524189+0000","last_became_active":"2024-04-24T23:15:41.019339+0000","last_became_peered":"2024-04-24T23:15:41.019339+0000","last_unstale":"2024-04-24T23:27:59.524189+0000","last_undegraded":"2024-04-24T23:27:59.524189+0000","last_fullsized":"2024-04-24T23:27:59.524189+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'5217","last_scrub_stamp":"2024-04-24T23:27:59.524092+0000","last_deep_scrub":"30'5217","last_deep_scrub_stamp":"2024-04-24T23:27:59.524092+0000","last_clean_scrub_stamp":"2024-04-24T23:27:59.524092+0000","objects_scrubbed":0,"log_size":5217,"log_dups_size":0,"ondisk_log_size":5217,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:40:34.204779+0000","scrub_duration":100,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":81,"num_read_kb":0,"num_write":75905,"num_write_kb":1074791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2,3],"acting":[7,4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"30'10","reported_seq":56,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:51.150860+0000","last_change":"2024-04-24T23:27:49.996036+0000","last_active":"2024-04-24T23:27:51.150860+0000","last_peered":"2024-04-24T23:27:51.150860+0000","last_clean":"2024-04-24T23:27:51.150860+0000","last_became_active":"2024-04-24T23:15:36.989336+0000","last_became_peered":"2024-04-24T23:15:36.989336+0000","last_unstale":"2024-04-24T23:27:51.150860+0000","last_undegraded":"2024-04-24T23:27:51.150860+0000","last_fullsized":"2024-04-24T23:27:51.150860+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'7","last_scrub_stamp":"2024-04-24T23:27:49.995954+0000","last_deep_scrub":"30'7","last_deep_scrub_stamp":"2024-04-24T23:27:49.995954+0000","last_clean_scrub_stamp":"2024-04-24T23:27:49.995954+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:03:50.327244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.8","version":"30'35","reported_seq":86,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:51.136208+0000","last_change":"2024-04-24T23:27:50.103062+0000","last_active":"2024-04-24T23:27:51.136208+0000","last_peered":"2024-04-24T23:27:51.136208+0000","last_clean":"2024-04-24T23:27:51.136208+0000","last_became_active":"2024-04-24T23:15:35.965043+0000","last_became_peered":"2024-04-24T23:15:35.965043+0000","last_unstale":"2024-04-24T23:27:51.136208+0000","last_undegraded":"2024-04-24T23:27:51.136208+0000","last_fullsized":"2024-04-24T23:27:51.136208+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'33","last_scrub_stamp":"2024-04-24T23:27:50.102956+0000","last_deep_scrub":"30'33","last_deep_scrub_stamp":"2024-04-24T23:27:50.102956+0000","last_clean_scrub_stamp":"2024-04-24T23:27:50.102956+0000","objects_scrubbed":2,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:06:47.021497+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":4102,"num_write":30,"num_write_kb":4111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"30'3837","reported_seq":3938,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:28:01.150193+0000","last_change":"2024-04-24T23:28:01.150257+0000","last_active":"2024-04-24T23:28:01.150193+0000","last_peered":"2024-04-24T23:28:01.150193+0000","last_clean":"2024-04-24T23:28:01.150193+0000","last_became_active":"2024-04-24T23:15:41.024731+0000","last_became_peered":"2024-04-24T23:15:41.024731+0000","last_unstale":"2024-04-24T23:28:01.150193+0000","last_undegraded":"2024-04-24T23:28:01.150193+0000","last_fullsized":"2024-04-24T23:28:01.150193+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3837","last_scrub_stamp":"2024-04-24T23:28:01.150009+0000","last_deep_scrub":"30'3837","last_deep_scrub_stamp":"2024-04-24T23:28:01.150009+0000","last_clean_scrub_stamp":"2024-04-24T23:28:01.150009+0000","objects_scrubbed":0,"log_size":3837,"log_dups_size":0,"ondisk_log_size":3837,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:30:36.770016+0000","scrub_duration":115,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":61,"num_read_kb":0,"num_write":55573,"num_write_kb":793811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,7],"acting":[2,1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"30'19","reported_seq":65,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:56.258479+0000","last_change":"2024-04-24T23:27:56.258504+0000","last_active":"2024-04-24T23:27:56.258479+0000","last_peered":"2024-04-24T23:27:56.258479+0000","last_clean":"2024-04-24T23:27:56.258479+0000","last_became_active":"2024-04-24T23:15:36.965547+0000","last_became_peered":"2024-04-24T23:15:36.965547+0000","last_unstale":"2024-04-24T23:27:56.258479+0000","last_undegraded":"2024-04-24T23:27:56.258479+0000","last_fullsized":"2024-04-24T23:27:56.258479+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'19","last_scrub_stamp":"2024-04-24T23:27:56.258383+0000","last_deep_scrub":"30'19","last_deep_scrub_stamp":"2024-04-24T23:27:56.258383+0000","last_clean_scrub_stamp":"2024-04-24T23:27:56.258383+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:09:34.691733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"30'15","reported_seq":71,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:28:05.986306+0000","last_change":"2024-04-24T23:28:05.986421+0000","last_active":"2024-04-24T23:28:05.986306+0000","last_peered":"2024-04-24T23:28:05.986306+0000","last_clean":"2024-04-24T23:28:05.986306+0000","last_became_active":"2024-04-24T23:15:35.970184+0000","last_became_peered":"2024-04-24T23:15:35.970184+0000","last_unstale":"2024-04-24T23:28:05.986306+0000","last_undegraded":"2024-04-24T23:28:05.986306+0000","last_fullsized":"2024-04-24T23:28:05.986306+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'15","last_scrub_stamp":"2024-04-24T23:28:05.986152+0000","last_deep_scrub":"30'15","last_deep_scrub_stamp":"2024-04-24T23:28:05.986152+0000","last_clean_scrub_stamp":"2024-04-24T23:28:05.986152+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:30:46.811111+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":24,"num_write":11,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"30'4423","reported_seq":4538,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:28:02.186881+0000","last_change":"2024-04-24T23:28:02.186881+0000","last_active":"2024-04-24T23:28:02.186881+0000","last_peered":"2024-04-24T23:28:02.186881+0000","last_clean":"2024-04-24T23:28:02.186881+0000","last_became_active":"2024-04-24T23:15:41.025137+0000","last_became_peered":"2024-04-24T23:15:41.025137+0000","last_unstale":"2024-04-24T23:28:02.186881+0000","last_undegraded":"2024-04-24T23:28:02.186881+0000","last_fullsized":"2024-04-24T23:28:02.186881+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'4423","last_scrub_stamp":"2024-04-24T23:28:02.186714+0000","last_deep_scrub":"30'4423","last_deep_scrub_stamp":"2024-04-24T23:28:02.186714+0000","last_clean_scrub_stamp":"2024-04-24T23:28:02.186714+0000","objects_scrubbed":0,"log_size":4423,"log_dups_size":0,"ondisk_log_size":4423,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:53:07.036567+0000","scrub_duration":106,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":74,"num_read_kb":0,"num_write":64413,"num_write_kb":917129,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,3,1],"acting":[2,6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"30'13","reported_seq":59,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:58.282620+0000","last_change":"2024-04-24T23:27:58.282646+0000","last_active":"2024-04-24T23:27:58.282620+0000","last_peered":"2024-04-24T23:27:58.282620+0000","last_clean":"2024-04-24T23:27:58.282620+0000","last_became_active":"2024-04-24T23:15:36.967098+0000","last_became_peered":"2024-04-24T23:15:36.967098+0000","last_unstale":"2024-04-24T23:27:58.282620+0000","last_undegraded":"2024-04-24T23:27:58.282620+0000","last_fullsized":"2024-04-24T23:27:58.282620+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'13","last_scrub_stamp":"2024-04-24T23:27:58.282516+0000","last_deep_scrub":"30'13","last_deep_scrub_stamp":"2024-04-24T23:27:58.282516+0000","last_clean_scrub_stamp":"2024-04-24T23:27:58.282516+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:49:04.125156+0000","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"30'271","reported_seq":320,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:54.195309+0000","last_change":"2024-04-24T23:27:54.195357+0000","last_active":"2024-04-24T23:27:54.195309+0000","last_peered":"2024-04-24T23:27:54.195309+0000","last_clean":"2024-04-24T23:27:54.195309+0000","last_became_active":"2024-04-24T23:15:35.970430+0000","last_became_peered":"2024-04-24T23:15:35.970430+0000","last_unstale":"2024-04-24T23:27:54.195309+0000","last_undegraded":"2024-04-24T23:27:54.195309+0000","last_fullsized":"2024-04-24T23:27:54.195309+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'271","last_scrub_stamp":"2024-04-24T23:27:54.195222+0000","last_deep_scrub":"30'271","last_deep_scrub_stamp":"2024-04-24T23:27:54.195222+0000","last_clean_scrub_stamp":"2024-04-24T23:27:54.195222+0000","objects_scrubbed":4,"log_size":271,"log_dups_size":0,"ondisk_log_size":271,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:49:13.561921+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":9,"num_write":563,"num_write_kb":230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"30'4776","reported_seq":4894,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:28:03.383173+0000","last_change":"2024-04-24T23:28:03.383200+0000","last_active":"2024-04-24T23:28:03.383173+0000","last_peered":"2024-04-24T23:28:03.383173+0000","last_clean":"2024-04-24T23:28:03.383173+0000","last_became_active":"2024-04-24T23:15:41.004648+0000","last_became_peered":"2024-04-24T23:15:41.004648+0000","last_unstale":"2024-04-24T23:28:03.383173+0000","last_undegraded":"2024-04-24T23:28:03.383173+0000","last_fullsized":"2024-04-24T23:28:03.383173+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'4776","last_scrub_stamp":"2024-04-24T23:28:03.383076+0000","last_deep_scrub":"30'4776","last_deep_scrub_stamp":"2024-04-24T23:28:03.383076+0000","last_clean_scrub_stamp":"2024-04-24T23:28:03.383076+0000","objects_scrubbed":0,"log_size":4776,"log_dups_size":0,"ondisk_log_size":4776,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:24:27.615324+0000","scrub_duration":84,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":77,"num_read_kb":0,"num_write":69584,"num_write_kb":1012990,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6,3],"acting":[4,1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.4","version":"30'9","reported_seq":56,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:56.350830+0000","last_change":"2024-04-24T23:27:56.350830+0000","last_active":"2024-04-24T23:27:56.350830+0000","last_peered":"2024-04-24T23:27:56.350830+0000","last_clean":"2024-04-24T23:27:56.350830+0000","last_became_active":"2024-04-24T23:15:36.966746+0000","last_became_peered":"2024-04-24T23:15:36.966746+0000","last_unstale":"2024-04-24T23:27:56.350830+0000","last_undegraded":"2024-04-24T23:27:56.350830+0000","last_fullsized":"2024-04-24T23:27:56.350830+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'9","last_scrub_stamp":"2024-04-24T23:27:56.350726+0000","last_deep_scrub":"30'9","last_deep_scrub_stamp":"2024-04-24T23:27:56.350726+0000","last_clean_scrub_stamp":"2024-04-24T23:27:56.350726+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:56:17.971553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"30'11","reported_seq":62,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:28:04.943826+0000","last_change":"2024-04-24T23:28:04.943896+0000","last_active":"2024-04-24T23:28:04.943826+0000","last_peered":"2024-04-24T23:28:04.943826+0000","last_clean":"2024-04-24T23:28:04.943826+0000","last_became_active":"2024-04-24T23:15:35.965631+0000","last_became_peered":"2024-04-24T23:15:35.965631+0000","last_unstale":"2024-04-24T23:28:04.943826+0000","last_undegraded":"2024-04-24T23:28:04.943826+0000","last_fullsized":"2024-04-24T23:28:04.943826+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'11","last_scrub_stamp":"2024-04-24T23:28:04.943646+0000","last_deep_scrub":"30'11","last_deep_scrub_stamp":"2024-04-24T23:28:04.943646+0000","last_clean_scrub_stamp":"2024-04-24T23:28:04.943646+0000","objects_scrubbed":3,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:43:42.634998+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"30'4046","reported_seq":4146,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:28:04.509070+0000","last_change":"2024-04-24T23:28:04.509094+0000","last_active":"2024-04-24T23:28:04.509070+0000","last_peered":"2024-04-24T23:28:04.509070+0000","last_clean":"2024-04-24T23:28:04.509070+0000","last_became_active":"2024-04-24T23:15:41.010982+0000","last_became_peered":"2024-04-24T23:15:41.010982+0000","last_unstale":"2024-04-24T23:28:04.509070+0000","last_undegraded":"2024-04-24T23:28:04.509070+0000","last_fullsized":"2024-04-24T23:28:04.509070+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'4046","last_scrub_stamp":"2024-04-24T23:28:04.508979+0000","last_deep_scrub":"30'4046","last_deep_scrub_stamp":"2024-04-24T23:28:04.508979+0000","last_clean_scrub_stamp":"2024-04-24T23:28:04.508979+0000","objects_scrubbed":0,"log_size":4046,"log_dups_size":0,"ondisk_log_size":4046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:56:10.599487+0000","scrub_duration":57,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":60,"num_read_kb":0,"num_write":58654,"num_write_kb":828032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2,7],"acting":[6,0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"30'14","reported_seq":60,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:51.095937+0000","last_change":"2024-04-24T23:27:50.043832+0000","last_active":"2024-04-24T23:27:51.095937+0000","last_peered":"2024-04-24T23:27:51.095937+0000","last_clean":"2024-04-24T23:27:51.095937+0000","last_became_active":"2024-04-24T23:15:36.974872+0000","last_became_peered":"2024-04-24T23:15:36.974872+0000","last_unstale":"2024-04-24T23:27:51.095937+0000","last_undegraded":"2024-04-24T23:27:51.095937+0000","last_fullsized":"2024-04-24T23:27:51.095937+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'10","last_scrub_stamp":"2024-04-24T23:27:50.043716+0000","last_deep_scrub":"30'10","last_deep_scrub_stamp":"2024-04-24T23:27:50.043716+0000","last_clean_scrub_stamp":"2024-04-24T23:27:50.043716+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:40:08.379839+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"30'6","reported_seq":55,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:56.509920+0000","last_change":"2024-04-24T23:27:56.509920+0000","last_active":"2024-04-24T23:27:56.509920+0000","last_peered":"2024-04-24T23:27:56.509920+0000","last_clean":"2024-04-24T23:27:56.509920+0000","last_became_active":"2024-04-24T23:15:35.945094+0000","last_became_peered":"2024-04-24T23:15:35.945094+0000","last_unstale":"2024-04-24T23:27:56.509920+0000","last_undegraded":"2024-04-24T23:27:56.509920+0000","last_fullsized":"2024-04-24T23:27:56.509920+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'6","last_scrub_stamp":"2024-04-24T23:27:56.509790+0000","last_deep_scrub":"30'6","last_deep_scrub_stamp":"2024-04-24T23:27:56.509790+0000","last_clean_scrub_stamp":"2024-04-24T23:27:56.509790+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:50.913465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"30'4962","reported_seq":5079,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:28:06.467549+0000","last_change":"2024-04-24T23:28:06.467609+0000","last_active":"2024-04-24T23:28:06.467549+0000","last_peered":"2024-04-24T23:28:06.467549+0000","last_clean":"2024-04-24T23:28:06.467549+0000","last_became_active":"2024-04-24T23:15:41.021645+0000","last_became_peered":"2024-04-24T23:15:41.021645+0000","last_unstale":"2024-04-24T23:28:06.467549+0000","last_undegraded":"2024-04-24T23:28:06.467549+0000","last_fullsized":"2024-04-24T23:28:06.467549+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'4962","last_scrub_stamp":"2024-04-24T23:28:06.467417+0000","last_deep_scrub":"30'4962","last_deep_scrub_stamp":"2024-04-24T23:28:06.467417+0000","last_clean_scrub_stamp":"2024-04-24T23:28:06.467417+0000","objects_scrubbed":0,"log_size":4962,"log_dups_size":0,"ondisk_log_size":4962,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:37:21.549128+0000","scrub_duration":59,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":77,"num_read_kb":0,"num_write":72436,"num_write_kb":1021329,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5,7],"acting":[6,3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.3","version":"30'10","reported_seq":57,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:51.409168+0000","last_change":"2024-04-24T23:27:48.119568+0000","last_active":"2024-04-24T23:27:51.409168+0000","last_peered":"2024-04-24T23:27:51.409168+0000","last_clean":"2024-04-24T23:27:51.409168+0000","last_became_active":"2024-04-24T23:15:36.990963+0000","last_became_peered":"2024-04-24T23:15:36.990963+0000","last_unstale":"2024-04-24T23:27:51.409168+0000","last_undegraded":"2024-04-24T23:27:51.409168+0000","last_fullsized":"2024-04-24T23:27:51.409168+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'8","last_scrub_stamp":"2024-04-24T23:27:48.119426+0000","last_deep_scrub":"30'8","last_deep_scrub_stamp":"2024-04-24T23:27:48.119426+0000","last_clean_scrub_stamp":"2024-04-24T23:27:48.119426+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:16:00.599432+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"30'50","reported_seq":109,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:28:01.006222+0000","last_change":"2024-04-24T23:28:01.006222+0000","last_active":"2024-04-24T23:28:01.006222+0000","last_peered":"2024-04-24T23:28:01.006222+0000","last_clean":"2024-04-24T23:28:01.006222+0000","last_became_active":"2024-04-24T23:15:35.957432+0000","last_became_peered":"2024-04-24T23:15:35.957432+0000","last_unstale":"2024-04-24T23:28:01.006222+0000","last_undegraded":"2024-04-24T23:28:01.006222+0000","last_fullsized":"2024-04-24T23:28:01.006222+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'50","last_scrub_stamp":"2024-04-24T23:28:01.005988+0000","last_deep_scrub":"30'50","last_deep_scrub_stamp":"2024-04-24T23:28:01.005988+0000","last_clean_scrub_stamp":"2024-04-24T23:28:01.005988+0000","objects_scrubbed":4,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:00:15.235585+0000","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8389784,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":19,"num_read_kb":8206,"num_write":36,"num_write_kb":8211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.4","version":"30'4199","reported_seq":4304,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:28:02.548031+0000","last_change":"2024-04-24T23:28:02.548075+0000","last_active":"2024-04-24T23:28:02.548031+0000","last_peered":"2024-04-24T23:28:02.548031+0000","last_clean":"2024-04-24T23:28:02.548031+0000","last_became_active":"2024-04-24T23:15:41.007449+0000","last_became_peered":"2024-04-24T23:15:41.007449+0000","last_unstale":"2024-04-24T23:28:02.548031+0000","last_undegraded":"2024-04-24T23:28:02.548031+0000","last_fullsized":"2024-04-24T23:28:02.548031+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'4199","last_scrub_stamp":"2024-04-24T23:28:02.547910+0000","last_deep_scrub":"30'4199","last_deep_scrub_stamp":"2024-04-24T23:28:02.547910+0000","last_clean_scrub_stamp":"2024-04-24T23:28:02.547910+0000","objects_scrubbed":0,"log_size":4199,"log_dups_size":0,"ondisk_log_size":4199,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:36:04.575953+0000","scrub_duration":66,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":65,"num_read_kb":0,"num_write":60924,"num_write_kb":882911,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0,3],"acting":[6,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"30'15","reported_seq":62,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:54.501709+0000","last_change":"2024-04-24T23:27:54.501736+0000","last_active":"2024-04-24T23:27:54.501709+0000","last_peered":"2024-04-24T23:27:54.501709+0000","last_clean":"2024-04-24T23:27:54.501709+0000","last_became_active":"2024-04-24T23:15:36.970159+0000","last_became_peered":"2024-04-24T23:15:36.970159+0000","last_unstale":"2024-04-24T23:27:54.501709+0000","last_undegraded":"2024-04-24T23:27:54.501709+0000","last_fullsized":"2024-04-24T23:27:54.501709+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'15","last_scrub_stamp":"2024-04-24T23:27:54.501618+0000","last_deep_scrub":"30'15","last_deep_scrub_stamp":"2024-04-24T23:27:54.501618+0000","last_clean_scrub_stamp":"2024-04-24T23:27:54.501618+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:13:00.741443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"30'47","reported_seq":129,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:52.378195+0000","last_change":"2024-04-24T23:27:52.378195+0000","last_active":"2024-04-24T23:27:52.378195+0000","last_peered":"2024-04-24T23:27:52.378195+0000","last_clean":"2024-04-24T23:27:52.378195+0000","last_became_active":"2024-04-24T23:15:35.945185+0000","last_became_peered":"2024-04-24T23:15:35.945185+0000","last_unstale":"2024-04-24T23:27:52.378195+0000","last_undegraded":"2024-04-24T23:27:52.378195+0000","last_fullsized":"2024-04-24T23:27:52.378195+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'47","last_scrub_stamp":"2024-04-24T23:27:52.378078+0000","last_deep_scrub":"30'47","last_deep_scrub_stamp":"2024-04-24T23:27:52.378078+0000","last_clean_scrub_stamp":"2024-04-24T23:27:52.378078+0000","objects_scrubbed":4,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:14:33.145761+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":94280,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":33,"num_read_kb":113,"num_write":40,"num_write_kb":118,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"30'4761","reported_seq":4875,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:28:05.513229+0000","last_change":"2024-04-24T23:28:05.513253+0000","last_active":"2024-04-24T23:28:05.513229+0000","last_peered":"2024-04-24T23:28:05.513229+0000","last_clean":"2024-04-24T23:28:05.513229+0000","last_became_active":"2024-04-24T23:15:41.014137+0000","last_became_peered":"2024-04-24T23:15:41.014137+0000","last_unstale":"2024-04-24T23:28:05.513229+0000","last_undegraded":"2024-04-24T23:28:05.513229+0000","last_fullsized":"2024-04-24T23:28:05.513229+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'4761","last_scrub_stamp":"2024-04-24T23:28:05.513152+0000","last_deep_scrub":"30'4761","last_deep_scrub_stamp":"2024-04-24T23:28:05.513152+0000","last_clean_scrub_stamp":"2024-04-24T23:28:05.513152+0000","objects_scrubbed":0,"log_size":4761,"log_dups_size":0,"ondisk_log_size":4761,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:38:14.069894+0000","scrub_duration":78,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":74,"num_read_kb":0,"num_write":69114,"num_write_kb":969120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,7],"acting":[6,0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"30'8","reported_seq":55,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:58.375786+0000","last_change":"2024-04-24T23:27:58.375786+0000","last_active":"2024-04-24T23:27:58.375786+0000","last_peered":"2024-04-24T23:27:58.375786+0000","last_clean":"2024-04-24T23:27:58.375786+0000","last_became_active":"2024-04-24T23:15:36.981502+0000","last_became_peered":"2024-04-24T23:15:36.981502+0000","last_unstale":"2024-04-24T23:27:58.375786+0000","last_undegraded":"2024-04-24T23:27:58.375786+0000","last_fullsized":"2024-04-24T23:27:58.375786+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'8","last_scrub_stamp":"2024-04-24T23:27:58.375627+0000","last_deep_scrub":"30'8","last_deep_scrub_stamp":"2024-04-24T23:27:58.375627+0000","last_clean_scrub_stamp":"2024-04-24T23:27:58.375627+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:54:33.616309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"30'41","reported_seq":92,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:28:01.976267+0000","last_change":"2024-04-24T23:28:01.976314+0000","last_active":"2024-04-24T23:28:01.976267+0000","last_peered":"2024-04-24T23:28:01.976267+0000","last_clean":"2024-04-24T23:28:01.976267+0000","last_became_active":"2024-04-24T23:15:35.960180+0000","last_became_peered":"2024-04-24T23:15:35.960180+0000","last_unstale":"2024-04-24T23:28:01.976267+0000","last_undegraded":"2024-04-24T23:28:01.976267+0000","last_fullsized":"2024-04-24T23:28:01.976267+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'41","last_scrub_stamp":"2024-04-24T23:28:01.976171+0000","last_deep_scrub":"30'41","last_deep_scrub_stamp":"2024-04-24T23:28:01.976171+0000","last_clean_scrub_stamp":"2024-04-24T23:28:01.976171+0000","objects_scrubbed":3,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:18:24.235567+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":8192,"num_write":28,"num_write_kb":8204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"30'3864","reported_seq":3965,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:28:09.559663+0000","last_change":"2024-04-24T23:28:09.559707+0000","last_active":"2024-04-24T23:28:09.559663+0000","last_peered":"2024-04-24T23:28:09.559663+0000","last_clean":"2024-04-24T23:28:09.559663+0000","last_became_active":"2024-04-24T23:15:41.012493+0000","last_became_peered":"2024-04-24T23:15:41.012493+0000","last_unstale":"2024-04-24T23:28:09.559663+0000","last_undegraded":"2024-04-24T23:28:09.559663+0000","last_fullsized":"2024-04-24T23:28:09.559663+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3864","last_scrub_stamp":"2024-04-24T23:28:09.559534+0000","last_deep_scrub":"30'3864","last_deep_scrub_stamp":"2024-04-24T23:28:09.559534+0000","last_clean_scrub_stamp":"2024-04-24T23:28:09.559534+0000","objects_scrubbed":0,"log_size":3864,"log_dups_size":0,"ondisk_log_size":3864,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:19:42.352711+0000","scrub_duration":57,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":61,"num_read_kb":307,"num_write":55967,"num_write_kb":780987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6,1],"acting":[5,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"30'8","reported_seq":55,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:28:00.306838+0000","last_change":"2024-04-24T23:28:00.306838+0000","last_active":"2024-04-24T23:28:00.306838+0000","last_peered":"2024-04-24T23:28:00.306838+0000","last_clean":"2024-04-24T23:28:00.306838+0000","last_became_active":"2024-04-24T23:15:36.980837+0000","last_became_peered":"2024-04-24T23:15:36.980837+0000","last_unstale":"2024-04-24T23:28:00.306838+0000","last_undegraded":"2024-04-24T23:28:00.306838+0000","last_fullsized":"2024-04-24T23:28:00.306838+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'8","last_scrub_stamp":"2024-04-24T23:28:00.306715+0000","last_deep_scrub":"30'8","last_deep_scrub_stamp":"2024-04-24T23:28:00.306715+0000","last_clean_scrub_stamp":"2024-04-24T23:28:00.306715+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:08:26.962757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.3","version":"30'12","reported_seq":65,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:58.974989+0000","last_change":"2024-04-24T23:27:58.974989+0000","last_active":"2024-04-24T23:27:58.974989+0000","last_peered":"2024-04-24T23:27:58.974989+0000","last_clean":"2024-04-24T23:27:58.974989+0000","last_became_active":"2024-04-24T23:15:35.952655+0000","last_became_peered":"2024-04-24T23:15:35.952655+0000","last_unstale":"2024-04-24T23:27:58.974989+0000","last_undegraded":"2024-04-24T23:27:58.974989+0000","last_fullsized":"2024-04-24T23:27:58.974989+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'12","last_scrub_stamp":"2024-04-24T23:27:58.974739+0000","last_deep_scrub":"30'12","last_deep_scrub_stamp":"2024-04-24T23:27:58.974739+0000","last_clean_scrub_stamp":"2024-04-24T23:27:58.974739+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:04:13.958405+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":7,"num_read_kb":11,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"30'119","reported_seq":207,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:47.179514+0000","last_change":"2024-04-24T23:27:47.179576+0000","last_active":"2024-04-24T23:27:47.179514+0000","last_peered":"2024-04-24T23:27:47.179514+0000","last_clean":"2024-04-24T23:27:47.179514+0000","last_became_active":"2024-04-24T23:15:25.820030+0000","last_became_peered":"2024-04-24T23:15:25.820030+0000","last_unstale":"2024-04-24T23:27:47.179514+0000","last_undegraded":"2024-04-24T23:27:47.179514+0000","last_fullsized":"2024-04-24T23:27:47.179514+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'119","last_scrub_stamp":"2024-04-24T23:27:47.179391+0000","last_deep_scrub":"30'119","last_deep_scrub_stamp":"2024-04-24T23:27:47.179391+0000","last_clean_scrub_stamp":"2024-04-24T23:27:47.179391+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:06:58.842359+0000","scrub_duration":90,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"30'4286","reported_seq":4394,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:51.310188+0000","last_change":"2024-04-24T23:15:41.024712+0000","last_active":"2024-04-24T23:27:51.310188+0000","last_peered":"2024-04-24T23:27:51.310188+0000","last_clean":"2024-04-24T23:27:51.310188+0000","last_became_active":"2024-04-24T23:15:41.022697+0000","last_became_peered":"2024-04-24T23:15:41.022697+0000","last_unstale":"2024-04-24T23:27:51.310188+0000","last_undegraded":"2024-04-24T23:27:51.310188+0000","last_fullsized":"2024-04-24T23:27:51.310188+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:39.973013+0000","last_clean_scrub_stamp":"2024-04-24T23:15:39.973013+0000","objects_scrubbed":0,"log_size":4286,"log_dups_size":0,"ondisk_log_size":4286,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":76,"num_read_kb":0,"num_write":62654,"num_write_kb":898105,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2,0],"acting":[3,1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.d","version":"30'15","reported_seq":61,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:28:06.947675+0000","last_change":"2024-04-24T23:28:06.947732+0000","last_active":"2024-04-24T23:28:06.947675+0000","last_peered":"2024-04-24T23:28:06.947675+0000","last_clean":"2024-04-24T23:28:06.947675+0000","last_became_active":"2024-04-24T23:15:37.000179+0000","last_became_peered":"2024-04-24T23:15:37.000179+0000","last_unstale":"2024-04-24T23:28:06.947675+0000","last_undegraded":"2024-04-24T23:28:06.947675+0000","last_fullsized":"2024-04-24T23:28:06.947675+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'15","last_scrub_stamp":"2024-04-24T23:28:06.947489+0000","last_deep_scrub":"30'15","last_deep_scrub_stamp":"2024-04-24T23:28:06.947489+0000","last_clean_scrub_stamp":"2024-04-24T23:28:06.947489+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:36:45.048151+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.c","version":"30'15","reported_seq":63,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:53.396776+0000","last_change":"2024-04-24T23:27:53.396809+0000","last_active":"2024-04-24T23:27:53.396776+0000","last_peered":"2024-04-24T23:27:53.396776+0000","last_clean":"2024-04-24T23:27:53.396776+0000","last_became_active":"2024-04-24T23:15:35.965758+0000","last_became_peered":"2024-04-24T23:15:35.965758+0000","last_unstale":"2024-04-24T23:27:53.396776+0000","last_undegraded":"2024-04-24T23:27:53.396776+0000","last_fullsized":"2024-04-24T23:27:53.396776+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'15","last_scrub_stamp":"2024-04-24T23:27:53.396692+0000","last_deep_scrub":"30'15","last_deep_scrub_stamp":"2024-04-24T23:27:53.396692+0000","last_clean_scrub_stamp":"2024-04-24T23:27:53.396692+0000","objects_scrubbed":3,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:03:53.462987+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2157,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"30'4384","reported_seq":4494,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:59.162578+0000","last_change":"2024-04-24T23:27:59.162644+0000","last_active":"2024-04-24T23:27:59.162578+0000","last_peered":"2024-04-24T23:27:59.162578+0000","last_clean":"2024-04-24T23:27:59.162578+0000","last_became_active":"2024-04-24T23:15:41.024041+0000","last_became_peered":"2024-04-24T23:15:41.024041+0000","last_unstale":"2024-04-24T23:27:59.162578+0000","last_undegraded":"2024-04-24T23:27:59.162578+0000","last_fullsized":"2024-04-24T23:27:59.162578+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'4384","last_scrub_stamp":"2024-04-24T23:27:59.162380+0000","last_deep_scrub":"30'4384","last_deep_scrub_stamp":"2024-04-24T23:27:59.162380+0000","last_clean_scrub_stamp":"2024-04-24T23:27:59.162380+0000","objects_scrubbed":0,"log_size":4384,"log_dups_size":0,"ondisk_log_size":4384,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:43:12.603815+0000","scrub_duration":106,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":70,"num_read_kb":0,"num_write":63886,"num_write_kb":912579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7,4],"acting":[2,3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"30'13","reported_seq":59,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:56.100904+0000","last_change":"2024-04-24T23:27:56.100963+0000","last_active":"2024-04-24T23:27:56.100904+0000","last_peered":"2024-04-24T23:27:56.100904+0000","last_clean":"2024-04-24T23:27:56.100904+0000","last_became_active":"2024-04-24T23:15:36.979611+0000","last_became_peered":"2024-04-24T23:15:36.979611+0000","last_unstale":"2024-04-24T23:27:56.100904+0000","last_undegraded":"2024-04-24T23:27:56.100904+0000","last_fullsized":"2024-04-24T23:27:56.100904+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'13","last_scrub_stamp":"2024-04-24T23:27:56.100727+0000","last_deep_scrub":"30'13","last_deep_scrub_stamp":"2024-04-24T23:27:56.100727+0000","last_clean_scrub_stamp":"2024-04-24T23:27:56.100727+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:31:52.409110+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"30'28","reported_seq":78,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:58.560132+0000","last_change":"2024-04-24T23:27:58.560160+0000","last_active":"2024-04-24T23:27:58.560132+0000","last_peered":"2024-04-24T23:27:58.560132+0000","last_clean":"2024-04-24T23:27:58.560132+0000","last_became_active":"2024-04-24T23:15:35.964873+0000","last_became_peered":"2024-04-24T23:15:35.964873+0000","last_unstale":"2024-04-24T23:27:58.560132+0000","last_undegraded":"2024-04-24T23:27:58.560132+0000","last_fullsized":"2024-04-24T23:27:58.560132+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'28","last_scrub_stamp":"2024-04-24T23:27:58.560042+0000","last_deep_scrub":"30'28","last_deep_scrub_stamp":"2024-04-24T23:27:58.560042+0000","last_clean_scrub_stamp":"2024-04-24T23:27:58.560042+0000","objects_scrubbed":2,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:13:43.031451+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194374,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":4097,"num_write":20,"num_write_kb":4108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"30'4134","reported_seq":4245,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:28:07.551932+0000","last_change":"2024-04-24T23:28:07.551988+0000","last_active":"2024-04-24T23:28:07.551932+0000","last_peered":"2024-04-24T23:28:07.551932+0000","last_clean":"2024-04-24T23:28:07.551932+0000","last_became_active":"2024-04-24T23:15:41.003335+0000","last_became_peered":"2024-04-24T23:15:41.003335+0000","last_unstale":"2024-04-24T23:28:07.551932+0000","last_undegraded":"2024-04-24T23:28:07.551932+0000","last_fullsized":"2024-04-24T23:28:07.551932+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'4134","last_scrub_stamp":"2024-04-24T23:28:07.551822+0000","last_deep_scrub":"30'4134","last_deep_scrub_stamp":"2024-04-24T23:28:07.551822+0000","last_clean_scrub_stamp":"2024-04-24T23:28:07.551822+0000","objects_scrubbed":0,"log_size":4134,"log_dups_size":0,"ondisk_log_size":4134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:44:02.913672+0000","scrub_duration":82,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":71,"num_read_kb":0,"num_write":60152,"num_write_kb":843409,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1,3],"acting":[5,4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"30'12","reported_seq":58,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:28:03.130420+0000","last_change":"2024-04-24T23:28:03.130460+0000","last_active":"2024-04-24T23:28:03.130420+0000","last_peered":"2024-04-24T23:28:03.130420+0000","last_clean":"2024-04-24T23:28:03.130420+0000","last_became_active":"2024-04-24T23:15:37.002583+0000","last_became_peered":"2024-04-24T23:15:37.002583+0000","last_unstale":"2024-04-24T23:28:03.130420+0000","last_undegraded":"2024-04-24T23:28:03.130420+0000","last_fullsized":"2024-04-24T23:28:03.130420+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'12","last_scrub_stamp":"2024-04-24T23:28:03.130281+0000","last_deep_scrub":"30'12","last_deep_scrub_stamp":"2024-04-24T23:28:03.130281+0000","last_clean_scrub_stamp":"2024-04-24T23:28:03.130281+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:38:07.799732+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"30'171","reported_seq":227,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:52.160463+0000","last_change":"2024-04-24T23:27:52.160500+0000","last_active":"2024-04-24T23:27:52.160463+0000","last_peered":"2024-04-24T23:27:52.160463+0000","last_clean":"2024-04-24T23:27:52.160463+0000","last_became_active":"2024-04-24T23:15:35.948438+0000","last_became_peered":"2024-04-24T23:15:35.948438+0000","last_unstale":"2024-04-24T23:27:52.160463+0000","last_undegraded":"2024-04-24T23:27:52.160463+0000","last_fullsized":"2024-04-24T23:27:52.160463+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'171","last_scrub_stamp":"2024-04-24T23:27:52.160374+0000","last_deep_scrub":"30'171","last_deep_scrub_stamp":"2024-04-24T23:27:52.160374+0000","last_clean_scrub_stamp":"2024-04-24T23:27:52.160374+0000","objects_scrubbed":3,"log_size":171,"log_dups_size":0,"ondisk_log_size":171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:54:22.224578+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":21,"num_read_kb":2419,"num_write":164,"num_write_kb":4172,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.8","version":"30'4142","reported_seq":4250,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:28:09.019589+0000","last_change":"2024-04-24T23:28:09.019624+0000","last_active":"2024-04-24T23:28:09.019589+0000","last_peered":"2024-04-24T23:28:09.019589+0000","last_clean":"2024-04-24T23:28:09.019589+0000","last_became_active":"2024-04-24T23:15:41.017092+0000","last_became_peered":"2024-04-24T23:15:41.017092+0000","last_unstale":"2024-04-24T23:28:09.019589+0000","last_undegraded":"2024-04-24T23:28:09.019589+0000","last_fullsized":"2024-04-24T23:28:09.019589+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'4142","last_scrub_stamp":"2024-04-24T23:28:09.019481+0000","last_deep_scrub":"30'4142","last_deep_scrub_stamp":"2024-04-24T23:28:09.019481+0000","last_clean_scrub_stamp":"2024-04-24T23:28:09.019481+0000","objects_scrubbed":0,"log_size":4142,"log_dups_size":0,"ondisk_log_size":4142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:24:29.735641+0000","scrub_duration":113,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":68,"num_read_kb":0,"num_write":60430,"num_write_kb":856855,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"30'7","reported_seq":53,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:52.933420+0000","last_change":"2024-04-24T23:27:52.933542+0000","last_active":"2024-04-24T23:27:52.933420+0000","last_peered":"2024-04-24T23:27:52.933420+0000","last_clean":"2024-04-24T23:27:52.933420+0000","last_became_active":"2024-04-24T23:15:36.989863+0000","last_became_peered":"2024-04-24T23:15:36.989863+0000","last_unstale":"2024-04-24T23:27:52.933420+0000","last_undegraded":"2024-04-24T23:27:52.933420+0000","last_fullsized":"2024-04-24T23:27:52.933420+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'7","last_scrub_stamp":"2024-04-24T23:27:52.933168+0000","last_deep_scrub":"30'7","last_deep_scrub_stamp":"2024-04-24T23:27:52.933168+0000","last_clean_scrub_stamp":"2024-04-24T23:27:52.933168+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:06:54.040616+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.f","version":"30'11","reported_seq":60,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:54.109447+0000","last_change":"2024-04-24T23:27:54.109515+0000","last_active":"2024-04-24T23:27:54.109447+0000","last_peered":"2024-04-24T23:27:54.109447+0000","last_clean":"2024-04-24T23:27:54.109447+0000","last_became_active":"2024-04-24T23:15:35.968994+0000","last_became_peered":"2024-04-24T23:15:35.968994+0000","last_unstale":"2024-04-24T23:27:54.109447+0000","last_undegraded":"2024-04-24T23:27:54.109447+0000","last_fullsized":"2024-04-24T23:27:54.109447+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'11","last_scrub_stamp":"2024-04-24T23:27:54.109225+0000","last_deep_scrub":"30'11","last_deep_scrub_stamp":"2024-04-24T23:27:54.109225+0000","last_clean_scrub_stamp":"2024-04-24T23:27:54.109225+0000","objects_scrubbed":2,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:28:04.145600+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"30'4168","reported_seq":4281,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:54.181383+0000","last_change":"2024-04-24T23:27:54.181500+0000","last_active":"2024-04-24T23:27:54.181383+0000","last_peered":"2024-04-24T23:27:54.181383+0000","last_clean":"2024-04-24T23:27:54.181383+0000","last_became_active":"2024-04-24T23:15:41.009328+0000","last_became_peered":"2024-04-24T23:15:41.009328+0000","last_unstale":"2024-04-24T23:27:54.181383+0000","last_undegraded":"2024-04-24T23:27:54.181383+0000","last_fullsized":"2024-04-24T23:27:54.181383+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'4168","last_scrub_stamp":"2024-04-24T23:27:54.181078+0000","last_deep_scrub":"30'4168","last_deep_scrub_stamp":"2024-04-24T23:27:54.181078+0000","last_clean_scrub_stamp":"2024-04-24T23:27:54.181078+0000","objects_scrubbed":0,"log_size":4168,"log_dups_size":0,"ondisk_log_size":4168,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:44:30.023840+0000","scrub_duration":142,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":73,"num_read_kb":0,"num_write":60607,"num_write_kb":866216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,2],"acting":[1,5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.11","version":"30'14","reported_seq":60,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:51.311338+0000","last_change":"2024-04-24T23:27:50.974130+0000","last_active":"2024-04-24T23:27:51.311338+0000","last_peered":"2024-04-24T23:27:51.311338+0000","last_clean":"2024-04-24T23:27:51.311338+0000","last_became_active":"2024-04-24T23:15:36.995801+0000","last_became_peered":"2024-04-24T23:15:36.995801+0000","last_unstale":"2024-04-24T23:27:51.311338+0000","last_undegraded":"2024-04-24T23:27:51.311338+0000","last_fullsized":"2024-04-24T23:27:51.311338+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"27'10","last_scrub_stamp":"2024-04-24T23:27:50.974001+0000","last_deep_scrub":"27'10","last_deep_scrub_stamp":"2024-04-24T23:27:50.974001+0000","last_clean_scrub_stamp":"2024-04-24T23:27:50.974001+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:15:32.521795+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"30'30","reported_seq":86,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:51.153587+0000","last_change":"2024-04-24T23:27:51.153587+0000","last_active":"2024-04-24T23:27:51.153587+0000","last_peered":"2024-04-24T23:27:51.153587+0000","last_clean":"2024-04-24T23:27:51.153587+0000","last_became_active":"2024-04-24T23:15:35.943710+0000","last_became_peered":"2024-04-24T23:15:35.943710+0000","last_unstale":"2024-04-24T23:27:51.153587+0000","last_undegraded":"2024-04-24T23:27:51.153587+0000","last_fullsized":"2024-04-24T23:27:51.153587+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'29","last_scrub_stamp":"2024-04-24T23:27:51.153469+0000","last_deep_scrub":"30'29","last_deep_scrub_stamp":"2024-04-24T23:27:51.153469+0000","last_clean_scrub_stamp":"2024-04-24T23:27:51.153469+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:41:24.647571+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"30'4684","reported_seq":4794,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:28:10.629258+0000","last_change":"2024-04-24T23:28:10.629299+0000","last_active":"2024-04-24T23:28:10.629258+0000","last_peered":"2024-04-24T23:28:10.629258+0000","last_clean":"2024-04-24T23:28:10.629258+0000","last_became_active":"2024-04-24T23:15:41.012587+0000","last_became_peered":"2024-04-24T23:15:41.012587+0000","last_unstale":"2024-04-24T23:28:10.629258+0000","last_undegraded":"2024-04-24T23:28:10.629258+0000","last_fullsized":"2024-04-24T23:28:10.629258+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'4684","last_scrub_stamp":"2024-04-24T23:28:10.629157+0000","last_deep_scrub":"30'4684","last_deep_scrub_stamp":"2024-04-24T23:28:10.629157+0000","last_clean_scrub_stamp":"2024-04-24T23:28:10.629157+0000","objects_scrubbed":0,"log_size":4684,"log_dups_size":0,"ondisk_log_size":4684,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:39:50.459623+0000","scrub_duration":104,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":70,"num_read_kb":0,"num_write":67827,"num_write_kb":966739,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1,6],"acting":[5,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"30'14","reported_seq":60,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:51.268761+0000","last_change":"2024-04-24T23:27:49.035576+0000","last_active":"2024-04-24T23:27:51.268761+0000","last_peered":"2024-04-24T23:27:51.268761+0000","last_clean":"2024-04-24T23:27:51.268761+0000","last_became_active":"2024-04-24T23:15:36.979356+0000","last_became_peered":"2024-04-24T23:15:36.979356+0000","last_unstale":"2024-04-24T23:27:51.268761+0000","last_undegraded":"2024-04-24T23:27:51.268761+0000","last_fullsized":"2024-04-24T23:27:51.268761+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'9","last_scrub_stamp":"2024-04-24T23:27:49.035212+0000","last_deep_scrub":"30'9","last_deep_scrub_stamp":"2024-04-24T23:27:49.035212+0000","last_clean_scrub_stamp":"2024-04-24T23:27:49.035212+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:34:14.717681+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.11","version":"30'19","reported_seq":67,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:55.071123+0000","last_change":"2024-04-24T23:27:55.071193+0000","last_active":"2024-04-24T23:27:55.071123+0000","last_peered":"2024-04-24T23:27:55.071123+0000","last_clean":"2024-04-24T23:27:55.071123+0000","last_became_active":"2024-04-24T23:15:35.972655+0000","last_became_peered":"2024-04-24T23:15:35.972655+0000","last_unstale":"2024-04-24T23:27:55.071123+0000","last_undegraded":"2024-04-24T23:27:55.071123+0000","last_fullsized":"2024-04-24T23:27:55.071123+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'19","last_scrub_stamp":"2024-04-24T23:27:55.070951+0000","last_deep_scrub":"30'19","last_deep_scrub_stamp":"2024-04-24T23:27:55.070951+0000","last_clean_scrub_stamp":"2024-04-24T23:27:55.070951+0000","objects_scrubbed":3,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:07:52.691776+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1575,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"30'4523","reported_seq":4633,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:28:08.592568+0000","last_change":"2024-04-24T23:28:08.592593+0000","last_active":"2024-04-24T23:28:08.592568+0000","last_peered":"2024-04-24T23:28:08.592568+0000","last_clean":"2024-04-24T23:28:08.592568+0000","last_became_active":"2024-04-24T23:15:41.003790+0000","last_became_peered":"2024-04-24T23:15:41.003790+0000","last_unstale":"2024-04-24T23:28:08.592568+0000","last_undegraded":"2024-04-24T23:28:08.592568+0000","last_fullsized":"2024-04-24T23:28:08.592568+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'4523","last_scrub_stamp":"2024-04-24T23:28:08.592484+0000","last_deep_scrub":"30'4523","last_deep_scrub_stamp":"2024-04-24T23:28:08.592484+0000","last_clean_scrub_stamp":"2024-04-24T23:28:08.592484+0000","objects_scrubbed":0,"log_size":4523,"log_dups_size":0,"ondisk_log_size":4523,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:05:21.509442+0000","scrub_duration":98,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":70,"num_read_kb":0,"num_write":65477,"num_write_kb":956853,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6,4],"acting":[5,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"30'14","reported_seq":60,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:53.014961+0000","last_change":"2024-04-24T23:27:53.015060+0000","last_active":"2024-04-24T23:27:53.014961+0000","last_peered":"2024-04-24T23:27:53.014961+0000","last_clean":"2024-04-24T23:27:53.014961+0000","last_became_active":"2024-04-24T23:15:36.995923+0000","last_became_peered":"2024-04-24T23:15:36.995923+0000","last_unstale":"2024-04-24T23:27:53.014961+0000","last_undegraded":"2024-04-24T23:27:53.014961+0000","last_fullsized":"2024-04-24T23:27:53.014961+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'14","last_scrub_stamp":"2024-04-24T23:27:53.014791+0000","last_deep_scrub":"30'14","last_deep_scrub_stamp":"2024-04-24T23:27:53.014791+0000","last_clean_scrub_stamp":"2024-04-24T23:27:53.014791+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:13:47.119786+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"30'14","reported_seq":66,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:59.996583+0000","last_change":"2024-04-24T23:27:59.996645+0000","last_active":"2024-04-24T23:27:59.996583+0000","last_peered":"2024-04-24T23:27:59.996583+0000","last_clean":"2024-04-24T23:27:59.996583+0000","last_became_active":"2024-04-24T23:15:35.957088+0000","last_became_peered":"2024-04-24T23:15:35.957088+0000","last_unstale":"2024-04-24T23:27:59.996583+0000","last_undegraded":"2024-04-24T23:27:59.996583+0000","last_fullsized":"2024-04-24T23:27:59.996583+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'14","last_scrub_stamp":"2024-04-24T23:27:59.996438+0000","last_deep_scrub":"30'14","last_deep_scrub_stamp":"2024-04-24T23:27:59.996438+0000","last_clean_scrub_stamp":"2024-04-24T23:27:59.996438+0000","objects_scrubbed":3,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:05:52.966855+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":8,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"30'4678","reported_seq":4789,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:28:03.508373+0000","last_change":"2024-04-24T23:28:03.508396+0000","last_active":"2024-04-24T23:28:03.508373+0000","last_peered":"2024-04-24T23:28:03.508373+0000","last_clean":"2024-04-24T23:28:03.508373+0000","last_became_active":"2024-04-24T23:15:41.007555+0000","last_became_peered":"2024-04-24T23:15:41.007555+0000","last_unstale":"2024-04-24T23:28:03.508373+0000","last_undegraded":"2024-04-24T23:28:03.508373+0000","last_fullsized":"2024-04-24T23:28:03.508373+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'4678","last_scrub_stamp":"2024-04-24T23:28:03.508282+0000","last_deep_scrub":"30'4678","last_deep_scrub_stamp":"2024-04-24T23:28:03.508282+0000","last_clean_scrub_stamp":"2024-04-24T23:28:03.508282+0000","objects_scrubbed":0,"log_size":4678,"log_dups_size":0,"ondisk_log_size":4678,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:26:36.436556+0000","scrub_duration":53,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":71,"num_read_kb":0,"num_write":68048,"num_write_kb":979211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0,7],"acting":[6,4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.12","version":"30'15","reported_seq":61,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:28:01.437112+0000","last_change":"2024-04-24T23:28:01.437135+0000","last_active":"2024-04-24T23:28:01.437112+0000","last_peered":"2024-04-24T23:28:01.437112+0000","last_clean":"2024-04-24T23:28:01.437112+0000","last_became_active":"2024-04-24T23:15:36.991115+0000","last_became_peered":"2024-04-24T23:15:36.991115+0000","last_unstale":"2024-04-24T23:28:01.437112+0000","last_undegraded":"2024-04-24T23:28:01.437112+0000","last_fullsized":"2024-04-24T23:28:01.437112+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'15","last_scrub_stamp":"2024-04-24T23:28:01.437043+0000","last_deep_scrub":"30'15","last_deep_scrub_stamp":"2024-04-24T23:28:01.437043+0000","last_clean_scrub_stamp":"2024-04-24T23:28:01.437043+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:26:54.984451+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"30'53","reported_seq":112,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:54.410603+0000","last_change":"2024-04-24T23:27:54.410662+0000","last_active":"2024-04-24T23:27:54.410603+0000","last_peered":"2024-04-24T23:27:54.410603+0000","last_clean":"2024-04-24T23:27:54.410603+0000","last_became_active":"2024-04-24T23:15:35.963907+0000","last_became_peered":"2024-04-24T23:15:35.963907+0000","last_unstale":"2024-04-24T23:27:54.410603+0000","last_undegraded":"2024-04-24T23:27:54.410603+0000","last_fullsized":"2024-04-24T23:27:54.410603+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'53","last_scrub_stamp":"2024-04-24T23:27:54.410494+0000","last_deep_scrub":"30'53","last_deep_scrub_stamp":"2024-04-24T23:27:54.410494+0000","last_clean_scrub_stamp":"2024-04-24T23:27:54.410494+0000","objects_scrubbed":8,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:11:12.638113+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388676,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":26,"num_read_kb":12314,"num_write":42,"num_write_kb":8216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"30'4268","reported_seq":4381,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:28:10.019842+0000","last_change":"2024-04-24T23:28:10.019908+0000","last_active":"2024-04-24T23:28:10.019842+0000","last_peered":"2024-04-24T23:28:10.019842+0000","last_clean":"2024-04-24T23:28:10.019842+0000","last_became_active":"2024-04-24T23:15:41.019193+0000","last_became_peered":"2024-04-24T23:15:41.019193+0000","last_unstale":"2024-04-24T23:28:10.019842+0000","last_undegraded":"2024-04-24T23:28:10.019842+0000","last_fullsized":"2024-04-24T23:28:10.019842+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'4268","last_scrub_stamp":"2024-04-24T23:28:10.019714+0000","last_deep_scrub":"30'4268","last_deep_scrub_stamp":"2024-04-24T23:28:10.019714+0000","last_clean_scrub_stamp":"2024-04-24T23:28:10.019714+0000","objects_scrubbed":0,"log_size":4268,"log_dups_size":0,"ondisk_log_size":4268,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:30:52.312608+0000","scrub_duration":106,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":73,"num_read_kb":0,"num_write":62324,"num_write_kb":864711,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5,6],"acting":[3,0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"30'11","reported_seq":57,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:47.168936+0000","last_change":"2024-04-24T23:27:47.168936+0000","last_active":"2024-04-24T23:27:47.168936+0000","last_peered":"2024-04-24T23:27:47.168936+0000","last_clean":"2024-04-24T23:27:47.168936+0000","last_became_active":"2024-04-24T23:15:36.986484+0000","last_became_peered":"2024-04-24T23:15:36.986484+0000","last_unstale":"2024-04-24T23:27:47.168936+0000","last_undegraded":"2024-04-24T23:27:47.168936+0000","last_fullsized":"2024-04-24T23:27:47.168936+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'10","last_scrub_stamp":"2024-04-24T23:27:47.168801+0000","last_deep_scrub":"30'10","last_deep_scrub_stamp":"2024-04-24T23:27:47.168801+0000","last_clean_scrub_stamp":"2024-04-24T23:27:47.168801+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:37:32.843942+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.14","version":"30'244","reported_seq":303,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:51.198912+0000","last_change":"2024-04-24T23:27:48.044628+0000","last_active":"2024-04-24T23:27:51.198912+0000","last_peered":"2024-04-24T23:27:51.198912+0000","last_clean":"2024-04-24T23:27:51.198912+0000","last_became_active":"2024-04-24T23:15:35.951923+0000","last_became_peered":"2024-04-24T23:15:35.951923+0000","last_unstale":"2024-04-24T23:27:51.198912+0000","last_undegraded":"2024-04-24T23:27:51.198912+0000","last_fullsized":"2024-04-24T23:27:51.198912+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'241","last_scrub_stamp":"2024-04-24T23:27:48.044467+0000","last_deep_scrub":"30'241","last_deep_scrub_stamp":"2024-04-24T23:27:48.044467+0000","last_clean_scrub_stamp":"2024-04-24T23:27:48.044467+0000","objects_scrubbed":6,"log_size":244,"log_dups_size":0,"ondisk_log_size":244,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:24:37.933540+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194976,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":28,"num_read_kb":4122,"num_write":236,"num_write_kb":4244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"30'4391","reported_seq":4507,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:28:05.422192+0000","last_change":"2024-04-24T23:28:05.422219+0000","last_active":"2024-04-24T23:28:05.422192+0000","last_peered":"2024-04-24T23:28:05.422192+0000","last_clean":"2024-04-24T23:28:05.422192+0000","last_became_active":"2024-04-24T23:15:41.015179+0000","last_became_peered":"2024-04-24T23:15:41.015179+0000","last_unstale":"2024-04-24T23:28:05.422192+0000","last_undegraded":"2024-04-24T23:28:05.422192+0000","last_fullsized":"2024-04-24T23:28:05.422192+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'4391","last_scrub_stamp":"2024-04-24T23:28:05.422097+0000","last_deep_scrub":"30'4391","last_deep_scrub_stamp":"2024-04-24T23:28:05.422097+0000","last_clean_scrub_stamp":"2024-04-24T23:28:05.422097+0000","objects_scrubbed":0,"log_size":4391,"log_dups_size":0,"ondisk_log_size":4391,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:05:26.306604+0000","scrub_duration":98,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":76,"num_read_kb":0,"num_write":63942,"num_write_kb":903699,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5,0],"acting":[4,3,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"30'10","reported_seq":56,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:56.976825+0000","last_change":"2024-04-24T23:27:56.976944+0000","last_active":"2024-04-24T23:27:56.976825+0000","last_peered":"2024-04-24T23:27:56.976825+0000","last_clean":"2024-04-24T23:27:56.976825+0000","last_became_active":"2024-04-24T23:15:36.995470+0000","last_became_peered":"2024-04-24T23:15:36.995470+0000","last_unstale":"2024-04-24T23:27:56.976825+0000","last_undegraded":"2024-04-24T23:27:56.976825+0000","last_fullsized":"2024-04-24T23:27:56.976825+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'10","last_scrub_stamp":"2024-04-24T23:27:56.976629+0000","last_deep_scrub":"30'10","last_deep_scrub_stamp":"2024-04-24T23:27:56.976629+0000","last_clean_scrub_stamp":"2024-04-24T23:27:56.976629+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:35:54.198302+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.15","version":"30'26","reported_seq":77,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:53.384833+0000","last_change":"2024-04-24T23:27:53.384886+0000","last_active":"2024-04-24T23:27:53.384833+0000","last_peered":"2024-04-24T23:27:53.384833+0000","last_clean":"2024-04-24T23:27:53.384833+0000","last_became_active":"2024-04-24T23:15:35.944805+0000","last_became_peered":"2024-04-24T23:15:35.944805+0000","last_unstale":"2024-04-24T23:27:53.384833+0000","last_undegraded":"2024-04-24T23:27:53.384833+0000","last_fullsized":"2024-04-24T23:27:53.384833+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'26","last_scrub_stamp":"2024-04-24T23:27:53.384737+0000","last_deep_scrub":"30'26","last_deep_scrub_stamp":"2024-04-24T23:27:53.384737+0000","last_clean_scrub_stamp":"2024-04-24T23:27:53.384737+0000","objects_scrubbed":4,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:04:05.494191+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":23,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"30'4431","reported_seq":4545,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:58.265245+0000","last_change":"2024-04-24T23:27:58.265294+0000","last_active":"2024-04-24T23:27:58.265245+0000","last_peered":"2024-04-24T23:27:58.265245+0000","last_clean":"2024-04-24T23:27:58.265245+0000","last_became_active":"2024-04-24T23:15:41.023951+0000","last_became_peered":"2024-04-24T23:15:41.023951+0000","last_unstale":"2024-04-24T23:27:58.265245+0000","last_undegraded":"2024-04-24T23:27:58.265245+0000","last_fullsized":"2024-04-24T23:27:58.265245+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'4431","last_scrub_stamp":"2024-04-24T23:27:58.265067+0000","last_deep_scrub":"30'4431","last_deep_scrub_stamp":"2024-04-24T23:27:58.265067+0000","last_clean_scrub_stamp":"2024-04-24T23:27:58.265067+0000","objects_scrubbed":0,"log_size":4431,"log_dups_size":0,"ondisk_log_size":4431,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:35:43.053036+0000","scrub_duration":179,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":74,"num_read_kb":0,"num_write":64500,"num_write_kb":925608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1,7],"acting":[2,3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.17","version":"30'16","reported_seq":62,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:28:04.535295+0000","last_change":"2024-04-24T23:28:04.535334+0000","last_active":"2024-04-24T23:28:04.535295+0000","last_peered":"2024-04-24T23:28:04.535295+0000","last_clean":"2024-04-24T23:28:04.535295+0000","last_became_active":"2024-04-24T23:15:36.992002+0000","last_became_peered":"2024-04-24T23:15:36.992002+0000","last_unstale":"2024-04-24T23:28:04.535295+0000","last_undegraded":"2024-04-24T23:28:04.535295+0000","last_fullsized":"2024-04-24T23:28:04.535295+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'16","last_scrub_stamp":"2024-04-24T23:28:04.535201+0000","last_deep_scrub":"30'16","last_deep_scrub_stamp":"2024-04-24T23:28:04.535201+0000","last_clean_scrub_stamp":"2024-04-24T23:28:04.535201+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:03:31.223318+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"30'9","reported_seq":63,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:52.095666+0000","last_change":"2024-04-24T23:27:52.095748+0000","last_active":"2024-04-24T23:27:52.095666+0000","last_peered":"2024-04-24T23:27:52.095666+0000","last_clean":"2024-04-24T23:27:52.095666+0000","last_became_active":"2024-04-24T23:15:35.966646+0000","last_became_peered":"2024-04-24T23:15:35.966646+0000","last_unstale":"2024-04-24T23:27:52.095666+0000","last_undegraded":"2024-04-24T23:27:52.095666+0000","last_fullsized":"2024-04-24T23:27:52.095666+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'9","last_scrub_stamp":"2024-04-24T23:27:52.095478+0000","last_deep_scrub":"30'9","last_deep_scrub_stamp":"2024-04-24T23:27:52.095478+0000","last_clean_scrub_stamp":"2024-04-24T23:27:52.095478+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:57:04.018915+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":19,"num_read_kb":18,"num_write":7,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"30'4325","reported_seq":4433,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:28:00.569065+0000","last_change":"2024-04-24T23:28:00.569091+0000","last_active":"2024-04-24T23:28:00.569065+0000","last_peered":"2024-04-24T23:28:00.569065+0000","last_clean":"2024-04-24T23:28:00.569065+0000","last_became_active":"2024-04-24T23:15:41.021281+0000","last_became_peered":"2024-04-24T23:15:41.021281+0000","last_unstale":"2024-04-24T23:28:00.569065+0000","last_undegraded":"2024-04-24T23:28:00.569065+0000","last_fullsized":"2024-04-24T23:28:00.569065+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'4325","last_scrub_stamp":"2024-04-24T23:28:00.568980+0000","last_deep_scrub":"30'4325","last_deep_scrub_stamp":"2024-04-24T23:28:00.568980+0000","last_clean_scrub_stamp":"2024-04-24T23:28:00.568980+0000","objects_scrubbed":0,"log_size":4325,"log_dups_size":0,"ondisk_log_size":4325,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:00:39.142058+0000","scrub_duration":109,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":68,"num_read_kb":0,"num_write":62902,"num_write_kb":881151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3,2],"acting":[7,1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"30'15","reported_seq":61,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:53.955159+0000","last_change":"2024-04-24T23:27:53.955219+0000","last_active":"2024-04-24T23:27:53.955159+0000","last_peered":"2024-04-24T23:27:53.955159+0000","last_clean":"2024-04-24T23:27:53.955159+0000","last_became_active":"2024-04-24T23:15:36.990711+0000","last_became_peered":"2024-04-24T23:15:36.990711+0000","last_unstale":"2024-04-24T23:27:53.955159+0000","last_undegraded":"2024-04-24T23:27:53.955159+0000","last_fullsized":"2024-04-24T23:27:53.955159+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'15","last_scrub_stamp":"2024-04-24T23:27:53.954973+0000","last_deep_scrub":"30'15","last_deep_scrub_stamp":"2024-04-24T23:27:53.954973+0000","last_clean_scrub_stamp":"2024-04-24T23:27:53.954973+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:47:09.263378+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.17","version":"30'17","reported_seq":68,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:48.130448+0000","last_change":"2024-04-24T23:27:48.035149+0000","last_active":"2024-04-24T23:27:48.130448+0000","last_peered":"2024-04-24T23:27:48.130448+0000","last_clean":"2024-04-24T23:27:48.130448+0000","last_became_active":"2024-04-24T23:15:35.956870+0000","last_became_peered":"2024-04-24T23:15:35.956870+0000","last_unstale":"2024-04-24T23:27:48.130448+0000","last_undegraded":"2024-04-24T23:27:48.130448+0000","last_fullsized":"2024-04-24T23:27:48.130448+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'16","last_scrub_stamp":"2024-04-24T23:27:48.035025+0000","last_deep_scrub":"30'16","last_deep_scrub_stamp":"2024-04-24T23:27:48.035025+0000","last_clean_scrub_stamp":"2024-04-24T23:27:48.035025+0000","objects_scrubbed":3,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:12:15.713654+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":13,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"30'4408","reported_seq":4512,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:28:04.386341+0000","last_change":"2024-04-24T23:28:04.386365+0000","last_active":"2024-04-24T23:28:04.386341+0000","last_peered":"2024-04-24T23:28:04.386341+0000","last_clean":"2024-04-24T23:28:04.386341+0000","last_became_active":"2024-04-24T23:15:41.005578+0000","last_became_peered":"2024-04-24T23:15:41.005578+0000","last_unstale":"2024-04-24T23:28:04.386341+0000","last_undegraded":"2024-04-24T23:28:04.386341+0000","last_fullsized":"2024-04-24T23:28:04.386341+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'4408","last_scrub_stamp":"2024-04-24T23:28:04.386247+0000","last_deep_scrub":"30'4408","last_deep_scrub_stamp":"2024-04-24T23:28:04.386247+0000","last_clean_scrub_stamp":"2024-04-24T23:28:04.386247+0000","objects_scrubbed":0,"log_size":4408,"log_dups_size":0,"ondisk_log_size":4408,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:53:59.287569+0000","scrub_duration":75,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":64,"num_read_kb":0,"num_write":64117,"num_write_kb":908953,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3,7],"acting":[4,0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1e","version":"30'5180","reported_seq":5304,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:28:06.409524+0000","last_change":"2024-04-24T23:28:06.409569+0000","last_active":"2024-04-24T23:28:06.409524+0000","last_peered":"2024-04-24T23:28:06.409524+0000","last_clean":"2024-04-24T23:28:06.409524+0000","last_became_active":"2024-04-24T23:15:41.021698+0000","last_became_peered":"2024-04-24T23:15:41.021698+0000","last_unstale":"2024-04-24T23:28:06.409524+0000","last_undegraded":"2024-04-24T23:28:06.409524+0000","last_fullsized":"2024-04-24T23:28:06.409524+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'5180","last_scrub_stamp":"2024-04-24T23:28:06.409403+0000","last_deep_scrub":"30'5180","last_deep_scrub_stamp":"2024-04-24T23:28:06.409403+0000","last_clean_scrub_stamp":"2024-04-24T23:28:06.409403+0000","objects_scrubbed":0,"log_size":5180,"log_dups_size":0,"ondisk_log_size":5180,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:44:54.992507+0000","scrub_duration":92,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":83,"num_read_kb":0,"num_write":75888,"num_write_kb":1069592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5,3],"acting":[4,1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"30'9","reported_seq":55,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:51.385170+0000","last_change":"2024-04-24T23:27:50.151067+0000","last_active":"2024-04-24T23:27:51.385170+0000","last_peered":"2024-04-24T23:27:51.385170+0000","last_clean":"2024-04-24T23:27:51.385170+0000","last_became_active":"2024-04-24T23:15:36.979553+0000","last_became_peered":"2024-04-24T23:15:36.979553+0000","last_unstale":"2024-04-24T23:27:51.385170+0000","last_undegraded":"2024-04-24T23:27:51.385170+0000","last_fullsized":"2024-04-24T23:27:51.385170+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"27'3","last_scrub_stamp":"2024-04-24T23:27:50.150961+0000","last_deep_scrub":"27'3","last_deep_scrub_stamp":"2024-04-24T23:27:50.150961+0000","last_clean_scrub_stamp":"2024-04-24T23:27:50.150961+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:45:08.759577+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.18","version":"30'38","reported_seq":88,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:28:04.012919+0000","last_change":"2024-04-24T23:28:04.012988+0000","last_active":"2024-04-24T23:28:04.012919+0000","last_peered":"2024-04-24T23:28:04.012919+0000","last_clean":"2024-04-24T23:28:04.012919+0000","last_became_active":"2024-04-24T23:15:35.964434+0000","last_became_peered":"2024-04-24T23:15:35.964434+0000","last_unstale":"2024-04-24T23:28:04.012919+0000","last_undegraded":"2024-04-24T23:28:04.012919+0000","last_fullsized":"2024-04-24T23:28:04.012919+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'38","last_scrub_stamp":"2024-04-24T23:28:04.012750+0000","last_deep_scrub":"30'38","last_deep_scrub_stamp":"2024-04-24T23:28:04.012750+0000","last_clean_scrub_stamp":"2024-04-24T23:28:04.012750+0000","objects_scrubbed":3,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:12:23.454965+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8389190,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":8192,"num_write":28,"num_write_kb":8217,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"30'4560","reported_seq":4670,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:28:10.959641+0000","last_change":"2024-04-24T23:28:10.959704+0000","last_active":"2024-04-24T23:28:10.959641+0000","last_peered":"2024-04-24T23:28:10.959641+0000","last_clean":"2024-04-24T23:28:10.959641+0000","last_became_active":"2024-04-24T23:15:41.021616+0000","last_became_peered":"2024-04-24T23:15:41.021616+0000","last_unstale":"2024-04-24T23:28:10.959641+0000","last_undegraded":"2024-04-24T23:28:10.959641+0000","last_fullsized":"2024-04-24T23:28:10.959641+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'4560","last_scrub_stamp":"2024-04-24T23:28:10.959520+0000","last_deep_scrub":"30'4560","last_deep_scrub_stamp":"2024-04-24T23:28:10.959520+0000","last_clean_scrub_stamp":"2024-04-24T23:28:10.959520+0000","objects_scrubbed":0,"log_size":4560,"log_dups_size":0,"ondisk_log_size":4560,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:23:22.285417+0000","scrub_duration":93,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":69,"num_read_kb":0,"num_write":66277,"num_write_kb":941784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5,1],"acting":[3,2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"30'19","reported_seq":65,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:59.517437+0000","last_change":"2024-04-24T23:27:59.517465+0000","last_active":"2024-04-24T23:27:59.517437+0000","last_peered":"2024-04-24T23:27:59.517437+0000","last_clean":"2024-04-24T23:27:59.517437+0000","last_became_active":"2024-04-24T23:15:36.966910+0000","last_became_peered":"2024-04-24T23:15:36.966910+0000","last_unstale":"2024-04-24T23:27:59.517437+0000","last_undegraded":"2024-04-24T23:27:59.517437+0000","last_fullsized":"2024-04-24T23:27:59.517437+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'19","last_scrub_stamp":"2024-04-24T23:27:59.517356+0000","last_deep_scrub":"30'19","last_deep_scrub_stamp":"2024-04-24T23:27:59.517356+0000","last_clean_scrub_stamp":"2024-04-24T23:27:59.517356+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:29:42.590508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"30'9","reported_seq":59,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:54.405362+0000","last_change":"2024-04-24T23:27:54.405410+0000","last_active":"2024-04-24T23:27:54.405362+0000","last_peered":"2024-04-24T23:27:54.405362+0000","last_clean":"2024-04-24T23:27:54.405362+0000","last_became_active":"2024-04-24T23:15:35.966939+0000","last_became_peered":"2024-04-24T23:15:35.966939+0000","last_unstale":"2024-04-24T23:27:54.405362+0000","last_undegraded":"2024-04-24T23:27:54.405362+0000","last_fullsized":"2024-04-24T23:27:54.405362+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'9","last_scrub_stamp":"2024-04-24T23:27:54.405278+0000","last_deep_scrub":"30'9","last_deep_scrub_stamp":"2024-04-24T23:27:54.405278+0000","last_clean_scrub_stamp":"2024-04-24T23:27:54.405278+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:08:13.932489+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":14,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"30'25","reported_seq":74,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:57.998293+0000","last_change":"2024-04-24T23:27:57.998293+0000","last_active":"2024-04-24T23:27:57.998293+0000","last_peered":"2024-04-24T23:27:57.998293+0000","last_clean":"2024-04-24T23:27:57.998293+0000","last_became_active":"2024-04-24T23:15:35.970908+0000","last_became_peered":"2024-04-24T23:15:35.970908+0000","last_unstale":"2024-04-24T23:27:57.998293+0000","last_undegraded":"2024-04-24T23:27:57.998293+0000","last_fullsized":"2024-04-24T23:27:57.998293+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'25","last_scrub_stamp":"2024-04-24T23:27:57.998040+0000","last_deep_scrub":"30'25","last_deep_scrub_stamp":"2024-04-24T23:27:57.998040+0000","last_clean_scrub_stamp":"2024-04-24T23:27:57.998040+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:50:58.232284+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":4096,"num_write":18,"num_write_kb":4106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"30'11","reported_seq":57,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:28:01.269124+0000","last_change":"2024-04-24T23:28:01.269124+0000","last_active":"2024-04-24T23:28:01.269124+0000","last_peered":"2024-04-24T23:28:01.269124+0000","last_clean":"2024-04-24T23:28:01.269124+0000","last_became_active":"2024-04-24T23:15:36.987896+0000","last_became_peered":"2024-04-24T23:15:36.987896+0000","last_unstale":"2024-04-24T23:28:01.269124+0000","last_undegraded":"2024-04-24T23:28:01.269124+0000","last_fullsized":"2024-04-24T23:28:01.269124+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'11","last_scrub_stamp":"2024-04-24T23:28:01.269005+0000","last_deep_scrub":"30'11","last_deep_scrub_stamp":"2024-04-24T23:28:01.269005+0000","last_clean_scrub_stamp":"2024-04-24T23:28:01.269005+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:00:09.366001+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"30'3643","reported_seq":3742,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:28:02.313698+0000","last_change":"2024-04-24T23:28:02.313723+0000","last_active":"2024-04-24T23:28:02.313698+0000","last_peered":"2024-04-24T23:28:02.313698+0000","last_clean":"2024-04-24T23:28:02.313698+0000","last_became_active":"2024-04-24T23:15:40.997102+0000","last_became_peered":"2024-04-24T23:15:40.997102+0000","last_unstale":"2024-04-24T23:28:02.313698+0000","last_undegraded":"2024-04-24T23:28:02.313698+0000","last_fullsized":"2024-04-24T23:28:02.313698+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3643","last_scrub_stamp":"2024-04-24T23:28:02.313604+0000","last_deep_scrub":"30'3643","last_deep_scrub_stamp":"2024-04-24T23:28:02.313604+0000","last_clean_scrub_stamp":"2024-04-24T23:28:02.313604+0000","objects_scrubbed":0,"log_size":3643,"log_dups_size":0,"ondisk_log_size":3643,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:34:58.163112+0000","scrub_duration":54,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":60,"num_read_kb":0,"num_write":53469,"num_write_kb":749552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,7],"acting":[4,6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"30'1220","reported_seq":1272,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:55.541225+0000","last_change":"2024-04-24T23:27:55.541225+0000","last_active":"2024-04-24T23:27:55.541225+0000","last_peered":"2024-04-24T23:27:55.541225+0000","last_clean":"2024-04-24T23:27:55.541225+0000","last_became_active":"2024-04-24T23:15:35.944563+0000","last_became_peered":"2024-04-24T23:15:35.944563+0000","last_unstale":"2024-04-24T23:27:55.541225+0000","last_undegraded":"2024-04-24T23:27:55.541225+0000","last_fullsized":"2024-04-24T23:27:55.541225+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'1220","last_scrub_stamp":"2024-04-24T23:27:55.541114+0000","last_deep_scrub":"30'1220","last_deep_scrub_stamp":"2024-04-24T23:27:55.541114+0000","last_clean_scrub_stamp":"2024-04-24T23:27:55.541114+0000","objects_scrubbed":3,"log_size":1220,"log_dups_size":0,"ondisk_log_size":1220,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:06:45.381398+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4195441,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":4,"num_read_kb":4098,"num_write":1212,"num_write_kb":5208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"30'13","reported_seq":59,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:51.339661+0000","last_change":"2024-04-24T23:27:50.019919+0000","last_active":"2024-04-24T23:27:51.339661+0000","last_peered":"2024-04-24T23:27:51.339661+0000","last_clean":"2024-04-24T23:27:51.339661+0000","last_became_active":"2024-04-24T23:15:36.979637+0000","last_became_peered":"2024-04-24T23:15:36.979637+0000","last_unstale":"2024-04-24T23:27:51.339661+0000","last_undegraded":"2024-04-24T23:27:51.339661+0000","last_fullsized":"2024-04-24T23:27:51.339661+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'10","last_scrub_stamp":"2024-04-24T23:27:50.019778+0000","last_deep_scrub":"30'10","last_deep_scrub_stamp":"2024-04-24T23:27:50.019778+0000","last_clean_scrub_stamp":"2024-04-24T23:27:50.019778+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:28:39.922324+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"30'5076","reported_seq":5192,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:58.570048+0000","last_change":"2024-04-24T23:27:58.570095+0000","last_active":"2024-04-24T23:27:58.570048+0000","last_peered":"2024-04-24T23:27:58.570048+0000","last_clean":"2024-04-24T23:27:58.570048+0000","last_became_active":"2024-04-24T23:15:41.016696+0000","last_became_peered":"2024-04-24T23:15:41.016696+0000","last_unstale":"2024-04-24T23:27:58.570048+0000","last_undegraded":"2024-04-24T23:27:58.570048+0000","last_fullsized":"2024-04-24T23:27:58.570048+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'5076","last_scrub_stamp":"2024-04-24T23:27:58.569923+0000","last_deep_scrub":"30'5076","last_deep_scrub_stamp":"2024-04-24T23:27:58.569923+0000","last_clean_scrub_stamp":"2024-04-24T23:27:58.569923+0000","objects_scrubbed":0,"log_size":5076,"log_dups_size":0,"ondisk_log_size":5076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:56:31.012399+0000","scrub_duration":95,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":77,"num_read_kb":0,"num_write":74056,"num_write_kb":1014885,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1,0],"acting":[7,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"30'15","reported_seq":62,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:51.375476+0000","last_change":"2024-04-24T23:27:49.024754+0000","last_active":"2024-04-24T23:27:51.375476+0000","last_peered":"2024-04-24T23:27:51.375476+0000","last_clean":"2024-04-24T23:27:51.375476+0000","last_became_active":"2024-04-24T23:15:36.969090+0000","last_became_peered":"2024-04-24T23:15:36.969090+0000","last_unstale":"2024-04-24T23:27:51.375476+0000","last_undegraded":"2024-04-24T23:27:51.375476+0000","last_fullsized":"2024-04-24T23:27:51.375476+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'13","last_scrub_stamp":"2024-04-24T23:27:49.024601+0000","last_deep_scrub":"30'13","last_deep_scrub_stamp":"2024-04-24T23:27:49.024601+0000","last_clean_scrub_stamp":"2024-04-24T23:27:49.024601+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:10:37.734772+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.21","version":"30'13","reported_seq":60,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:56.449870+0000","last_change":"2024-04-24T23:27:56.449870+0000","last_active":"2024-04-24T23:27:56.449870+0000","last_peered":"2024-04-24T23:27:56.449870+0000","last_clean":"2024-04-24T23:27:56.449870+0000","last_became_active":"2024-04-24T23:15:36.982299+0000","last_became_peered":"2024-04-24T23:15:36.982299+0000","last_unstale":"2024-04-24T23:27:56.449870+0000","last_undegraded":"2024-04-24T23:27:56.449870+0000","last_fullsized":"2024-04-24T23:27:56.449870+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'13","last_scrub_stamp":"2024-04-24T23:27:56.449768+0000","last_deep_scrub":"30'13","last_deep_scrub_stamp":"2024-04-24T23:27:56.449768+0000","last_clean_scrub_stamp":"2024-04-24T23:27:56.449768+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:46:11.098469+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.22","version":"30'15","reported_seq":62,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:51.968530+0000","last_change":"2024-04-24T23:27:51.968530+0000","last_active":"2024-04-24T23:27:51.968530+0000","last_peered":"2024-04-24T23:27:51.968530+0000","last_clean":"2024-04-24T23:27:51.968530+0000","last_became_active":"2024-04-24T23:15:36.989557+0000","last_became_peered":"2024-04-24T23:15:36.989557+0000","last_unstale":"2024-04-24T23:27:51.968530+0000","last_undegraded":"2024-04-24T23:27:51.968530+0000","last_fullsized":"2024-04-24T23:27:51.968530+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'15","last_scrub_stamp":"2024-04-24T23:27:51.968386+0000","last_deep_scrub":"30'15","last_deep_scrub_stamp":"2024-04-24T23:27:51.968386+0000","last_clean_scrub_stamp":"2024-04-24T23:27:51.968386+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:48:35.495811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"30'11","reported_seq":58,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:28:01.563693+0000","last_change":"2024-04-24T23:28:01.563693+0000","last_active":"2024-04-24T23:28:01.563693+0000","last_peered":"2024-04-24T23:28:01.563693+0000","last_clean":"2024-04-24T23:28:01.563693+0000","last_became_active":"2024-04-24T23:15:36.969327+0000","last_became_peered":"2024-04-24T23:15:36.969327+0000","last_unstale":"2024-04-24T23:28:01.563693+0000","last_undegraded":"2024-04-24T23:28:01.563693+0000","last_fullsized":"2024-04-24T23:28:01.563693+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'11","last_scrub_stamp":"2024-04-24T23:28:01.563598+0000","last_deep_scrub":"30'11","last_deep_scrub_stamp":"2024-04-24T23:28:01.563598+0000","last_clean_scrub_stamp":"2024-04-24T23:28:01.563598+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:35:28.681060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.24","version":"30'10","reported_seq":56,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:59.420563+0000","last_change":"2024-04-24T23:27:59.420603+0000","last_active":"2024-04-24T23:27:59.420563+0000","last_peered":"2024-04-24T23:27:59.420563+0000","last_clean":"2024-04-24T23:27:59.420563+0000","last_became_active":"2024-04-24T23:15:36.986026+0000","last_became_peered":"2024-04-24T23:15:36.986026+0000","last_unstale":"2024-04-24T23:27:59.420563+0000","last_undegraded":"2024-04-24T23:27:59.420563+0000","last_fullsized":"2024-04-24T23:27:59.420563+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'10","last_scrub_stamp":"2024-04-24T23:27:59.420442+0000","last_deep_scrub":"30'10","last_deep_scrub_stamp":"2024-04-24T23:27:59.420442+0000","last_clean_scrub_stamp":"2024-04-24T23:27:59.420442+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:12:37.084902+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"30'9","reported_seq":55,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:57.359077+0000","last_change":"2024-04-24T23:27:57.359102+0000","last_active":"2024-04-24T23:27:57.359077+0000","last_peered":"2024-04-24T23:27:57.359077+0000","last_clean":"2024-04-24T23:27:57.359077+0000","last_became_active":"2024-04-24T23:15:36.968263+0000","last_became_peered":"2024-04-24T23:15:36.968263+0000","last_unstale":"2024-04-24T23:27:57.359077+0000","last_undegraded":"2024-04-24T23:27:57.359077+0000","last_fullsized":"2024-04-24T23:27:57.359077+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'9","last_scrub_stamp":"2024-04-24T23:27:57.358983+0000","last_deep_scrub":"30'9","last_deep_scrub_stamp":"2024-04-24T23:27:57.358983+0000","last_clean_scrub_stamp":"2024-04-24T23:27:57.358983+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:27:31.870033+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.26","version":"30'9","reported_seq":55,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:55.001407+0000","last_change":"2024-04-24T23:27:55.001471+0000","last_active":"2024-04-24T23:27:55.001407+0000","last_peered":"2024-04-24T23:27:55.001407+0000","last_clean":"2024-04-24T23:27:55.001407+0000","last_became_active":"2024-04-24T23:15:36.990764+0000","last_became_peered":"2024-04-24T23:15:36.990764+0000","last_unstale":"2024-04-24T23:27:55.001407+0000","last_undegraded":"2024-04-24T23:27:55.001407+0000","last_fullsized":"2024-04-24T23:27:55.001407+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'9","last_scrub_stamp":"2024-04-24T23:27:55.001180+0000","last_deep_scrub":"30'9","last_deep_scrub_stamp":"2024-04-24T23:27:55.001180+0000","last_clean_scrub_stamp":"2024-04-24T23:27:55.001180+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:39:17.437466+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"30'15","reported_seq":61,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:28:05.491295+0000","last_change":"2024-04-24T23:28:05.491317+0000","last_active":"2024-04-24T23:28:05.491295+0000","last_peered":"2024-04-24T23:28:05.491295+0000","last_clean":"2024-04-24T23:28:05.491295+0000","last_became_active":"2024-04-24T23:15:36.993310+0000","last_became_peered":"2024-04-24T23:15:36.993310+0000","last_unstale":"2024-04-24T23:28:05.491295+0000","last_undegraded":"2024-04-24T23:28:05.491295+0000","last_fullsized":"2024-04-24T23:28:05.491295+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'15","last_scrub_stamp":"2024-04-24T23:28:05.491221+0000","last_deep_scrub":"30'15","last_deep_scrub_stamp":"2024-04-24T23:28:05.491221+0000","last_clean_scrub_stamp":"2024-04-24T23:28:05.491221+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:47:41.696547+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.28","version":"30'12","reported_seq":58,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:54.039883+0000","last_change":"2024-04-24T23:27:54.039940+0000","last_active":"2024-04-24T23:27:54.039883+0000","last_peered":"2024-04-24T23:27:54.039883+0000","last_clean":"2024-04-24T23:27:54.039883+0000","last_became_active":"2024-04-24T23:15:36.987540+0000","last_became_peered":"2024-04-24T23:15:36.987540+0000","last_unstale":"2024-04-24T23:27:54.039883+0000","last_undegraded":"2024-04-24T23:27:54.039883+0000","last_fullsized":"2024-04-24T23:27:54.039883+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'12","last_scrub_stamp":"2024-04-24T23:27:54.039716+0000","last_deep_scrub":"30'12","last_deep_scrub_stamp":"2024-04-24T23:27:54.039716+0000","last_clean_scrub_stamp":"2024-04-24T23:27:54.039716+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:44:14.768880+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.29","version":"30'15","reported_seq":61,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:52.543383+0000","last_change":"2024-04-24T23:27:52.543408+0000","last_active":"2024-04-24T23:27:52.543383+0000","last_peered":"2024-04-24T23:27:52.543383+0000","last_clean":"2024-04-24T23:27:52.543383+0000","last_became_active":"2024-04-24T23:15:36.980838+0000","last_became_peered":"2024-04-24T23:15:36.980838+0000","last_unstale":"2024-04-24T23:27:52.543383+0000","last_undegraded":"2024-04-24T23:27:52.543383+0000","last_fullsized":"2024-04-24T23:27:52.543383+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'15","last_scrub_stamp":"2024-04-24T23:27:52.543282+0000","last_deep_scrub":"30'15","last_deep_scrub_stamp":"2024-04-24T23:27:52.543282+0000","last_clean_scrub_stamp":"2024-04-24T23:27:52.543282+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:26:38.143381+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2a","version":"30'14","reported_seq":60,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:53.022299+0000","last_change":"2024-04-24T23:27:53.022404+0000","last_active":"2024-04-24T23:27:53.022299+0000","last_peered":"2024-04-24T23:27:53.022299+0000","last_clean":"2024-04-24T23:27:53.022299+0000","last_became_active":"2024-04-24T23:15:36.996245+0000","last_became_peered":"2024-04-24T23:15:36.996245+0000","last_unstale":"2024-04-24T23:27:53.022299+0000","last_undegraded":"2024-04-24T23:27:53.022299+0000","last_fullsized":"2024-04-24T23:27:53.022299+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'14","last_scrub_stamp":"2024-04-24T23:27:53.022132+0000","last_deep_scrub":"30'14","last_deep_scrub_stamp":"2024-04-24T23:27:53.022132+0000","last_clean_scrub_stamp":"2024-04-24T23:27:53.022132+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:23:59.313058+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"30'17","reported_seq":63,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:59.291616+0000","last_change":"2024-04-24T23:27:59.291654+0000","last_active":"2024-04-24T23:27:59.291616+0000","last_peered":"2024-04-24T23:27:59.291616+0000","last_clean":"2024-04-24T23:27:59.291616+0000","last_became_active":"2024-04-24T23:15:36.980911+0000","last_became_peered":"2024-04-24T23:15:36.980911+0000","last_unstale":"2024-04-24T23:27:59.291616+0000","last_undegraded":"2024-04-24T23:27:59.291616+0000","last_fullsized":"2024-04-24T23:27:59.291616+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'17","last_scrub_stamp":"2024-04-24T23:27:59.291508+0000","last_deep_scrub":"30'17","last_deep_scrub_stamp":"2024-04-24T23:27:59.291508+0000","last_clean_scrub_stamp":"2024-04-24T23:27:59.291508+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:23:04.607632+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2c","version":"30'11","reported_seq":57,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:58.144877+0000","last_change":"2024-04-24T23:27:58.145002+0000","last_active":"2024-04-24T23:27:58.144877+0000","last_peered":"2024-04-24T23:27:58.144877+0000","last_clean":"2024-04-24T23:27:58.144877+0000","last_became_active":"2024-04-24T23:15:36.989792+0000","last_became_peered":"2024-04-24T23:15:36.989792+0000","last_unstale":"2024-04-24T23:27:58.144877+0000","last_undegraded":"2024-04-24T23:27:58.144877+0000","last_fullsized":"2024-04-24T23:27:58.144877+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'11","last_scrub_stamp":"2024-04-24T23:27:58.144584+0000","last_deep_scrub":"30'11","last_deep_scrub_stamp":"2024-04-24T23:27:58.144584+0000","last_clean_scrub_stamp":"2024-04-24T23:27:58.144584+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:01:47.513619+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2d","version":"30'15","reported_seq":61,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:55.978639+0000","last_change":"2024-04-24T23:27:55.978759+0000","last_active":"2024-04-24T23:27:55.978639+0000","last_peered":"2024-04-24T23:27:55.978639+0000","last_clean":"2024-04-24T23:27:55.978639+0000","last_became_active":"2024-04-24T23:15:36.987494+0000","last_became_peered":"2024-04-24T23:15:36.987494+0000","last_unstale":"2024-04-24T23:27:55.978639+0000","last_undegraded":"2024-04-24T23:27:55.978639+0000","last_fullsized":"2024-04-24T23:27:55.978639+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'15","last_scrub_stamp":"2024-04-24T23:27:55.978421+0000","last_deep_scrub":"30'15","last_deep_scrub_stamp":"2024-04-24T23:27:55.978421+0000","last_clean_scrub_stamp":"2024-04-24T23:27:55.978421+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:23:51.808207+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2e","version":"30'19","reported_seq":65,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:57.434308+0000","last_change":"2024-04-24T23:27:57.434336+0000","last_active":"2024-04-24T23:27:57.434308+0000","last_peered":"2024-04-24T23:27:57.434308+0000","last_clean":"2024-04-24T23:27:57.434308+0000","last_became_active":"2024-04-24T23:15:36.986272+0000","last_became_peered":"2024-04-24T23:15:36.986272+0000","last_unstale":"2024-04-24T23:27:57.434308+0000","last_undegraded":"2024-04-24T23:27:57.434308+0000","last_fullsized":"2024-04-24T23:27:57.434308+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'19","last_scrub_stamp":"2024-04-24T23:27:57.434230+0000","last_deep_scrub":"30'19","last_deep_scrub_stamp":"2024-04-24T23:27:57.434230+0000","last_clean_scrub_stamp":"2024-04-24T23:27:57.434230+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:20:00.881173+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2f","version":"30'13","reported_seq":56,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:51.009831+0000","last_change":"2024-04-24T23:16:18.294151+0000","last_active":"2024-04-24T23:27:51.009831+0000","last_peered":"2024-04-24T23:27:51.009831+0000","last_clean":"2024-04-24T23:27:51.009831+0000","last_became_active":"2024-04-24T23:16:18.293579+0000","last_became_peered":"2024-04-24T23:16:18.293579+0000","last_unstale":"2024-04-24T23:27:51.009831+0000","last_undegraded":"2024-04-24T23:27:51.009831+0000","last_fullsized":"2024-04-24T23:27:51.009831+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:15:35.921164+0000","last_clean_scrub_stamp":"2024-04-24T23:15:35.921164+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.30","version":"30'12","reported_seq":58,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:51.096354+0000","last_change":"2024-04-24T23:27:49.032530+0000","last_active":"2024-04-24T23:27:51.096354+0000","last_peered":"2024-04-24T23:27:51.096354+0000","last_clean":"2024-04-24T23:27:51.096354+0000","last_became_active":"2024-04-24T23:15:36.995507+0000","last_became_peered":"2024-04-24T23:15:36.995507+0000","last_unstale":"2024-04-24T23:27:51.096354+0000","last_undegraded":"2024-04-24T23:27:51.096354+0000","last_fullsized":"2024-04-24T23:27:51.096354+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'10","last_scrub_stamp":"2024-04-24T23:27:49.032001+0000","last_deep_scrub":"30'10","last_deep_scrub_stamp":"2024-04-24T23:27:49.032001+0000","last_clean_scrub_stamp":"2024-04-24T23:27:49.032001+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:45:06.602301+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.31","version":"30'14","reported_seq":60,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:51.494563+0000","last_change":"2024-04-24T23:27:51.494602+0000","last_active":"2024-04-24T23:27:51.494563+0000","last_peered":"2024-04-24T23:27:51.494563+0000","last_clean":"2024-04-24T23:27:51.494563+0000","last_became_active":"2024-04-24T23:15:36.984741+0000","last_became_peered":"2024-04-24T23:15:36.984741+0000","last_unstale":"2024-04-24T23:27:51.494563+0000","last_undegraded":"2024-04-24T23:27:51.494563+0000","last_fullsized":"2024-04-24T23:27:51.494563+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'14","last_scrub_stamp":"2024-04-24T23:27:51.494487+0000","last_deep_scrub":"30'14","last_deep_scrub_stamp":"2024-04-24T23:27:51.494487+0000","last_clean_scrub_stamp":"2024-04-24T23:27:51.494487+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:32:26.711735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.32","version":"30'17","reported_seq":63,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:55.358923+0000","last_change":"2024-04-24T23:27:55.358975+0000","last_active":"2024-04-24T23:27:55.358923+0000","last_peered":"2024-04-24T23:27:55.358923+0000","last_clean":"2024-04-24T23:27:55.358923+0000","last_became_active":"2024-04-24T23:15:36.965877+0000","last_became_peered":"2024-04-24T23:15:36.965877+0000","last_unstale":"2024-04-24T23:27:55.358923+0000","last_undegraded":"2024-04-24T23:27:55.358923+0000","last_fullsized":"2024-04-24T23:27:55.358923+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'17","last_scrub_stamp":"2024-04-24T23:27:55.358801+0000","last_deep_scrub":"30'17","last_deep_scrub_stamp":"2024-04-24T23:27:55.358801+0000","last_clean_scrub_stamp":"2024-04-24T23:27:55.358801+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:22:40.397575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.33","version":"30'15","reported_seq":61,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:28:00.399535+0000","last_change":"2024-04-24T23:28:00.399573+0000","last_active":"2024-04-24T23:28:00.399535+0000","last_peered":"2024-04-24T23:28:00.399535+0000","last_clean":"2024-04-24T23:28:00.399535+0000","last_became_active":"2024-04-24T23:15:36.990383+0000","last_became_peered":"2024-04-24T23:15:36.990383+0000","last_unstale":"2024-04-24T23:28:00.399535+0000","last_undegraded":"2024-04-24T23:28:00.399535+0000","last_fullsized":"2024-04-24T23:28:00.399535+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'15","last_scrub_stamp":"2024-04-24T23:28:00.399434+0000","last_deep_scrub":"30'15","last_deep_scrub_stamp":"2024-04-24T23:28:00.399434+0000","last_clean_scrub_stamp":"2024-04-24T23:28:00.399434+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:06:35.983668+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"30'10","reported_seq":56,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:59.156655+0000","last_change":"2024-04-24T23:27:59.156766+0000","last_active":"2024-04-24T23:27:59.156655+0000","last_peered":"2024-04-24T23:27:59.156655+0000","last_clean":"2024-04-24T23:27:59.156655+0000","last_became_active":"2024-04-24T23:15:36.990155+0000","last_became_peered":"2024-04-24T23:15:36.990155+0000","last_unstale":"2024-04-24T23:27:59.156655+0000","last_undegraded":"2024-04-24T23:27:59.156655+0000","last_fullsized":"2024-04-24T23:27:59.156655+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'10","last_scrub_stamp":"2024-04-24T23:27:59.156441+0000","last_deep_scrub":"30'10","last_deep_scrub_stamp":"2024-04-24T23:27:59.156441+0000","last_clean_scrub_stamp":"2024-04-24T23:27:59.156441+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:42:30.613654+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.35","version":"30'12","reported_seq":58,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:28:02.113006+0000","last_change":"2024-04-24T23:28:02.113043+0000","last_active":"2024-04-24T23:28:02.113006+0000","last_peered":"2024-04-24T23:28:02.113006+0000","last_clean":"2024-04-24T23:28:02.113006+0000","last_became_active":"2024-04-24T23:15:36.992608+0000","last_became_peered":"2024-04-24T23:15:36.992608+0000","last_unstale":"2024-04-24T23:28:02.113006+0000","last_undegraded":"2024-04-24T23:28:02.113006+0000","last_fullsized":"2024-04-24T23:28:02.113006+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'12","last_scrub_stamp":"2024-04-24T23:28:02.112907+0000","last_deep_scrub":"30'12","last_deep_scrub_stamp":"2024-04-24T23:28:02.112907+0000","last_clean_scrub_stamp":"2024-04-24T23:28:02.112907+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:18:42.489669+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.36","version":"30'13","reported_seq":59,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:28:01.112289+0000","last_change":"2024-04-24T23:28:01.112380+0000","last_active":"2024-04-24T23:28:01.112289+0000","last_peered":"2024-04-24T23:28:01.112289+0000","last_clean":"2024-04-24T23:28:01.112289+0000","last_became_active":"2024-04-24T23:15:36.992567+0000","last_became_peered":"2024-04-24T23:15:36.992567+0000","last_unstale":"2024-04-24T23:28:01.112289+0000","last_undegraded":"2024-04-24T23:28:01.112289+0000","last_fullsized":"2024-04-24T23:28:01.112289+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'13","last_scrub_stamp":"2024-04-24T23:28:01.112111+0000","last_deep_scrub":"30'13","last_deep_scrub_stamp":"2024-04-24T23:28:01.112111+0000","last_clean_scrub_stamp":"2024-04-24T23:28:01.112111+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:01:38.544642+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.37","version":"30'16","reported_seq":62,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:52.041925+0000","last_change":"2024-04-24T23:27:52.041925+0000","last_active":"2024-04-24T23:27:52.041925+0000","last_peered":"2024-04-24T23:27:52.041925+0000","last_clean":"2024-04-24T23:27:52.041925+0000","last_became_active":"2024-04-24T23:15:36.999413+0000","last_became_peered":"2024-04-24T23:15:36.999413+0000","last_unstale":"2024-04-24T23:27:52.041925+0000","last_undegraded":"2024-04-24T23:27:52.041925+0000","last_fullsized":"2024-04-24T23:27:52.041925+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'16","last_scrub_stamp":"2024-04-24T23:27:52.041753+0000","last_deep_scrub":"30'16","last_deep_scrub_stamp":"2024-04-24T23:27:52.041753+0000","last_clean_scrub_stamp":"2024-04-24T23:27:52.041753+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:44:52.887221+0000","scrub_duration":66,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"30'5","reported_seq":51,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:53.489217+0000","last_change":"2024-04-24T23:27:53.489217+0000","last_active":"2024-04-24T23:27:53.489217+0000","last_peered":"2024-04-24T23:27:53.489217+0000","last_clean":"2024-04-24T23:27:53.489217+0000","last_became_active":"2024-04-24T23:15:36.986013+0000","last_became_peered":"2024-04-24T23:15:36.986013+0000","last_unstale":"2024-04-24T23:27:53.489217+0000","last_undegraded":"2024-04-24T23:27:53.489217+0000","last_fullsized":"2024-04-24T23:27:53.489217+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'5","last_scrub_stamp":"2024-04-24T23:27:53.489094+0000","last_deep_scrub":"30'5","last_deep_scrub_stamp":"2024-04-24T23:27:53.489094+0000","last_clean_scrub_stamp":"2024-04-24T23:27:53.489094+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:33:58.795317+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"30'9","reported_seq":55,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:57.134449+0000","last_change":"2024-04-24T23:27:57.134449+0000","last_active":"2024-04-24T23:27:57.134449+0000","last_peered":"2024-04-24T23:27:57.134449+0000","last_clean":"2024-04-24T23:27:57.134449+0000","last_became_active":"2024-04-24T23:15:36.983966+0000","last_became_peered":"2024-04-24T23:15:36.983966+0000","last_unstale":"2024-04-24T23:27:57.134449+0000","last_undegraded":"2024-04-24T23:27:57.134449+0000","last_fullsized":"2024-04-24T23:27:57.134449+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'9","last_scrub_stamp":"2024-04-24T23:27:57.133962+0000","last_deep_scrub":"30'9","last_deep_scrub_stamp":"2024-04-24T23:27:57.133962+0000","last_clean_scrub_stamp":"2024-04-24T23:27:57.133962+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:21:53.198234+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3a","version":"30'10","reported_seq":55,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:51.978763+0000","last_change":"2024-04-24T23:27:51.978816+0000","last_active":"2024-04-24T23:27:51.978763+0000","last_peered":"2024-04-24T23:27:51.978763+0000","last_clean":"2024-04-24T23:27:51.978763+0000","last_became_active":"2024-04-24T23:15:36.978706+0000","last_became_peered":"2024-04-24T23:15:36.978706+0000","last_unstale":"2024-04-24T23:27:51.978763+0000","last_undegraded":"2024-04-24T23:27:51.978763+0000","last_fullsized":"2024-04-24T23:27:51.978763+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'10","last_scrub_stamp":"2024-04-24T23:27:51.978624+0000","last_deep_scrub":"30'10","last_deep_scrub_stamp":"2024-04-24T23:27:51.978624+0000","last_clean_scrub_stamp":"2024-04-24T23:27:51.978624+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:36:36.555148+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"30'15","reported_seq":60,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:51.440631+0000","last_change":"2024-04-24T23:27:49.134562+0000","last_active":"2024-04-24T23:27:51.440631+0000","last_peered":"2024-04-24T23:27:51.440631+0000","last_clean":"2024-04-24T23:27:51.440631+0000","last_became_active":"2024-04-24T23:15:36.985552+0000","last_became_peered":"2024-04-24T23:15:36.985552+0000","last_unstale":"2024-04-24T23:27:51.440631+0000","last_undegraded":"2024-04-24T23:27:51.440631+0000","last_fullsized":"2024-04-24T23:27:51.440631+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'7","last_scrub_stamp":"2024-04-24T23:27:49.134436+0000","last_deep_scrub":"30'7","last_deep_scrub_stamp":"2024-04-24T23:27:49.134436+0000","last_clean_scrub_stamp":"2024-04-24T23:27:49.134436+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:18:17.906068+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"30'7","reported_seq":54,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:51.035733+0000","last_change":"2024-04-24T23:27:51.035772+0000","last_active":"2024-04-24T23:27:51.035733+0000","last_peered":"2024-04-24T23:27:51.035733+0000","last_clean":"2024-04-24T23:27:51.035733+0000","last_became_active":"2024-04-24T23:15:36.972744+0000","last_became_peered":"2024-04-24T23:15:36.972744+0000","last_unstale":"2024-04-24T23:27:51.035733+0000","last_undegraded":"2024-04-24T23:27:51.035733+0000","last_fullsized":"2024-04-24T23:27:51.035733+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'7","last_scrub_stamp":"2024-04-24T23:27:51.035634+0000","last_deep_scrub":"30'7","last_deep_scrub_stamp":"2024-04-24T23:27:51.035634+0000","last_clean_scrub_stamp":"2024-04-24T23:27:51.035634+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:47:30.700734+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3d","version":"30'12","reported_seq":59,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:28:02.523052+0000","last_change":"2024-04-24T23:28:02.523052+0000","last_active":"2024-04-24T23:28:02.523052+0000","last_peered":"2024-04-24T23:28:02.523052+0000","last_clean":"2024-04-24T23:28:02.523052+0000","last_became_active":"2024-04-24T23:15:36.969751+0000","last_became_peered":"2024-04-24T23:15:36.969751+0000","last_unstale":"2024-04-24T23:28:02.523052+0000","last_undegraded":"2024-04-24T23:28:02.523052+0000","last_fullsized":"2024-04-24T23:28:02.523052+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'12","last_scrub_stamp":"2024-04-24T23:28:02.522909+0000","last_deep_scrub":"30'12","last_deep_scrub_stamp":"2024-04-24T23:28:02.522909+0000","last_clean_scrub_stamp":"2024-04-24T23:28:02.522909+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:50:33.757457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3e","version":"30'6","reported_seq":53,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:28:00.515256+0000","last_change":"2024-04-24T23:28:00.515256+0000","last_active":"2024-04-24T23:28:00.515256+0000","last_peered":"2024-04-24T23:28:00.515256+0000","last_clean":"2024-04-24T23:28:00.515256+0000","last_became_active":"2024-04-24T23:15:36.968543+0000","last_became_peered":"2024-04-24T23:15:36.968543+0000","last_unstale":"2024-04-24T23:28:00.515256+0000","last_undegraded":"2024-04-24T23:28:00.515256+0000","last_fullsized":"2024-04-24T23:28:00.515256+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'6","last_scrub_stamp":"2024-04-24T23:28:00.515165+0000","last_deep_scrub":"30'6","last_deep_scrub_stamp":"2024-04-24T23:28:00.515165+0000","last_clean_scrub_stamp":"2024-04-24T23:28:00.515165+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:16:52.463392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3f","version":"30'8","reported_seq":55,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:28:00.191194+0000","last_change":"2024-04-24T23:28:00.191194+0000","last_active":"2024-04-24T23:28:00.191194+0000","last_peered":"2024-04-24T23:28:00.191194+0000","last_clean":"2024-04-24T23:28:00.191194+0000","last_became_active":"2024-04-24T23:15:36.992476+0000","last_became_peered":"2024-04-24T23:15:36.992476+0000","last_unstale":"2024-04-24T23:28:00.191194+0000","last_undegraded":"2024-04-24T23:28:00.191194+0000","last_fullsized":"2024-04-24T23:28:00.191194+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'8","last_scrub_stamp":"2024-04-24T23:28:00.191018+0000","last_deep_scrub":"30'8","last_deep_scrub_stamp":"2024-04-24T23:28:00.191018+0000","last_clean_scrub_stamp":"2024-04-24T23:28:00.191018+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:41:02.972561+0000","scrub_duration":81,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2252,"num_read_kb":307,"num_write":2060742,"num_write_kb":29192446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":141598,"ondisk_log_size":141598,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":774,"ondisk_log_size":774,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":80027978,"num_objects":118,"num_object_clones":0,"num_object_copies":236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":118,"num_whiteouts":0,"num_read":353,"num_read_kb":84742,"num_write":4114,"num_write_kb":81391,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25000,"num_omap_keys":50,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":160292864,"data_stored":160055956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4041,"ondisk_log_size":4041,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575005,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1044808,"kb_used_data":12736,"kb_used_omap":1,"kb_used_meta":1032062,"kb_avail":93327032,"statfs":{"total":96636764160,"available":95566880768,"internally_reserved":0,"allocated":13041664,"data_stored":12820416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":1056831947},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834575005,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":962524,"kb_used_data":12628,"kb_used_omap":1,"kb_used_meta":949886,"kb_avail":93409316,"statfs":{"total":96636764160,"available":95651139584,"internally_reserved":0,"allocated":12931072,"data_stored":12729664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":972683718},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834575005,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":844416,"kb_used_data":25592,"kb_used_omap":1,"kb_used_meta":818814,"kb_avail":93527424,"statfs":{"total":96636764160,"available":95772082176,"internally_reserved":0,"allocated":26206208,"data_stored":25993983,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":838465989},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834575006,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":813500,"kb_used_data":12660,"kb_used_omap":1,"kb_used_meta":800830,"kb_avail":93558340,"statfs":{"total":96636764160,"available":95803740160,"internally_reserved":0,"allocated":12963840,"data_stored":12727704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":820050373},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834575001,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1091764,"kb_used_data":37228,"kb_used_omap":1,"kb_used_meta":1054526,"kb_avail":93280076,"statfs":{"total":96636764160,"available":95518797824,"internally_reserved":0,"allocated":38121472,"data_stored":37895315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":1079835079},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575000,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":758872,"kb_used_data":12624,"kb_used_omap":1,"kb_used_meta":746238,"kb_avail":93612968,"statfs":{"total":96636764160,"available":95859679232,"internally_reserved":0,"allocated":12926976,"data_stored":12725934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":764148165},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575000,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1098308,"kb_used_data":21628,"kb_used_omap":1,"kb_used_meta":1076670,"kb_avail":93273532,"statfs":{"total":96636764160,"available":95512096768,"internally_reserved":0,"allocated":22147072,"data_stored":21937447,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1102510533},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834575004,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":697104,"kb_used_data":25160,"kb_used_omap":1,"kb_used_meta":671934,"kb_avail":93674736,"statfs":{"total":96636764160,"available":95922929664,"internally_reserved":0,"allocated":25763840,"data_stored":25544485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":688060870},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":25436160,"data_stored":25402203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":21225472,"data_stored":21204797,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12599296,"data_stored":12583652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":37793792,"data_stored":37753033,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12636160,"data_stored":12585422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":25284608,"data_stored":25261333,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12603392,"data_stored":12587382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12713984,"data_stored":12678134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T23:28:13.300 INFO:tasks.ceph:pgid 4.5 last_scrub_stamp 2024-04-24T23:15:39.973013+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=15, tm_sec=39, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=27, tm_sec=44, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T23:28:13.301 INFO:tasks.ceph:pgid 3.2f last_scrub_stamp 2024-04-24T23:15:35.921164+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=15, tm_sec=35, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=27, tm_sec=44, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T23:28:13.301 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-04-24T23:28:33.303 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T23:28:33.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:33.502+0000 7faa50a00640 1 Processor -- start 2024-04-24T23:28:33.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:33.502+0000 7faa50a00640 1 -- start start 2024-04-24T23:28:33.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:33.504+0000 7faa50a00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faa4c0ca520 0x7faa4c0cb810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:28:33.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:33.504+0000 7faa50a00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa4c0ca900 con 0x7faa4c15c1b0 2024-04-24T23:28:33.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:33.504+0000 7faa49e00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faa4c0ca520 0x7faa4c0cb810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:28:33.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:33.504+0000 7faa43e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7faa4c15c1b0 0x7faa4c0cd400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:46636/0 (socket says 172.21.15.6:46636) 2024-04-24T23:28:33.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:33.504+0000 7faa43e00640 1 -- 172.21.15.6:0/4284640533 learned_addr learned my addr 172.21.15.6:0/4284640533 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:28:33.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:33.504+0000 7faa4a800640 1 --1- 172.21.15.6:0/4284640533 >> v1:172.21.15.190:6789/0 conn(0x7faa4c15e2b0 0x7faa4c15e760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.6:46318/0 (socket says 172.21.15.6:46318) 2024-04-24T23:28:33.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:33.504+0000 7faa50a00640 1 -- 172.21.15.6:0/4284640533 --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa4c0cc360 con 0x7faa4c15e2b0 2024-04-24T23:28:33.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:33.504+0000 7faa50a00640 1 -- 172.21.15.6:0/4284640533 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7faa4c0ccf70 con 0x7faa4c0ca520 2024-04-24T23:28:33.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:33.505+0000 7faa49400640 1 -- 172.21.15.6:0/4284640533 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3864863853 0 0) 0x7faa4c0cc360 con 0x7faa4c15e2b0 2024-04-24T23:28:33.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:33.505+0000 7faa49e00640 1 -- 172.21.15.6:0/4284640533 >> v1:172.21.15.190:6789/0 conn(0x7faa4c15e2b0 legacy=0x7faa4c15e760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:28:33.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:33.505+0000 7faa49e00640 1 -- 172.21.15.6:0/4284640533 >> v1:172.21.15.6:6789/0 conn(0x7faa4c15c1b0 legacy=0x7faa4c0cd400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:28:33.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:33.505+0000 7faa49e00640 1 -- 172.21.15.6:0/4284640533 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa4c0cc640 con 0x7faa4c0ca520 2024-04-24T23:28:33.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:33.506+0000 7faa49e00640 1 --2- 172.21.15.6:0/4284640533 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faa4c0ca520 0x7faa4c0cb810 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7faa3c0097d0 tx=0x7faa3c02a1a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5dd2f3cd11be08be server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:28:33.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:33.506+0000 7faa49400640 1 -- 172.21.15.6:0/4284640533 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa3c037070 con 0x7faa4c0ca520 2024-04-24T23:28:33.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:33.506+0000 7faa49400640 1 -- 172.21.15.6:0/4284640533 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa3c02ae30 con 0x7faa4c0ca520 2024-04-24T23:28:33.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:33.506+0000 7faa49400640 1 -- 172.21.15.6:0/4284640533 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa3c032640 con 0x7faa4c0ca520 2024-04-24T23:28:33.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:33.507+0000 7faa50a00640 1 -- 172.21.15.6:0/4284640533 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faa4c0ca520 msgr2=0x7faa4c0cb810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:28:33.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:33.507+0000 7faa50a00640 1 --2- 172.21.15.6:0/4284640533 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faa4c0ca520 0x7faa4c0cb810 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7faa3c0097d0 tx=0x7faa3c02a1a0 comp rx=0 tx=0).stop 2024-04-24T23:28:33.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:33.507+0000 7faa50a00640 1 -- 172.21.15.6:0/4284640533 shutdown_connections 2024-04-24T23:28:33.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:33.507+0000 7faa50a00640 1 --2- 172.21.15.6:0/4284640533 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faa4c0ca520 0x7faa4c0cb810 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:28:33.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:33.507+0000 7faa50a00640 1 -- 172.21.15.6:0/4284640533 >> 172.21.15.6:0/4284640533 conn(0x7faa4c06c0f0 msgr2=0x7faa4c066ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:28:33.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:33.507+0000 7faa50a00640 1 -- 172.21.15.6:0/4284640533 shutdown_connections 2024-04-24T23:28:33.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:33.507+0000 7faa50a00640 1 -- 172.21.15.6:0/4284640533 wait complete. 2024-04-24T23:28:33.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:33.508+0000 7faa50a00640 1 Processor -- start 2024-04-24T23:28:33.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:33.509+0000 7faa50a00640 1 -- start start 2024-04-24T23:28:33.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:33.509+0000 7faa50a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faa4c0ca520 0x7faa4c0cfd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:28:33.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:33.509+0000 7faa50a00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faa4c15c1b0 0x7faa4c155ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:28:33.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:33.510+0000 7faa50a00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faa4c15e2b0 0x7faa4c156010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:28:33.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:33.510+0000 7faa50a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7faa4c067890 con 0x7faa4c0ca520 2024-04-24T23:28:33.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:33.510+0000 7faa50a00640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7faa4c0ccc00 con 0x7faa4c15e2b0 2024-04-24T23:28:33.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:33.510+0000 7faa50a00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7faa4c067bc0 con 0x7faa4c15c1b0 2024-04-24T23:28:33.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:33.510+0000 7faa49e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faa4c0ca520 0x7faa4c0cfd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:28:33.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:33.510+0000 7faa49e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faa4c0ca520 0x7faa4c0cfd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:49232/0 (socket says 172.21.15.6:49232) 2024-04-24T23:28:33.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:33.510+0000 7faa4a800640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faa4c15e2b0 0x7faa4c156010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:28:33.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:33.510+0000 7faa49e00640 1 -- 172.21.15.6:0/1014283496 learned_addr learned my addr 172.21.15.6:0/1014283496 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:28:33.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:33.510+0000 7faa43e00640 1 --2- 172.21.15.6:0/1014283496 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faa4c15c1b0 0x7faa4c155ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:28:33.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:33.510+0000 7faa4a800640 1 -- 172.21.15.6:0/1014283496 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faa4c15c1b0 msgr2=0x7faa4c155ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:28:33.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:33.510+0000 7faa4a800640 1 --2- 172.21.15.6:0/1014283496 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faa4c15c1b0 0x7faa4c155ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:28:33.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:33.510+0000 7faa4a800640 1 -- 172.21.15.6:0/1014283496 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faa4c0ca520 msgr2=0x7faa4c0cfd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:28:33.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:33.510+0000 7faa4a800640 1 --2- 172.21.15.6:0/1014283496 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faa4c0ca520 0x7faa4c0cfd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:28:33.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:33.510+0000 7faa4a800640 1 -- 172.21.15.6:0/1014283496 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa4c157e00 con 0x7faa4c15e2b0 2024-04-24T23:28:33.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:33.511+0000 7faa4a800640 1 --2- 172.21.15.6:0/1014283496 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faa4c15e2b0 0x7faa4c156010 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7faa4400c780 tx=0x7faa4400cc40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:28:33.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:33.511+0000 7faa42000640 1 -- 172.21.15.6:0/1014283496 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa44018070 con 0x7faa4c15e2b0 2024-04-24T23:28:33.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:33.511+0000 7faa42000640 1 -- 172.21.15.6:0/1014283496 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa44004d60 con 0x7faa4c15e2b0 2024-04-24T23:28:33.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:33.511+0000 7faa50a00640 1 -- 172.21.15.6:0/1014283496 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa4c1580e0 con 0x7faa4c15e2b0 2024-04-24T23:28:33.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:33.511+0000 7faa42000640 1 -- 172.21.15.6:0/1014283496 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa4400a880 con 0x7faa4c15e2b0 2024-04-24T23:28:33.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:33.511+0000 7faa50a00640 1 -- 172.21.15.6:0/1014283496 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa4c1586a0 con 0x7faa4c15e2b0 2024-04-24T23:28:33.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:33.512+0000 7faa50a00640 1 -- 172.21.15.6:0/1014283496 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa4c0767b0 con 0x7faa4c15e2b0 2024-04-24T23:28:33.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:33.513+0000 7faa42000640 1 -- 172.21.15.6:0/1014283496 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7faa4401b070 con 0x7faa4c15e2b0 2024-04-24T23:28:33.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:33.514+0000 7faa42000640 1 --2- 172.21.15.6:0/1014283496 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7faa2806be60 0x7faa2806e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:28:33.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:33.514+0000 7faa42000640 1 -- 172.21.15.6:0/1014283496 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6934+0+0 (secure 0 0 0) 0x7faa4408a7a0 con 0x7faa4c15e2b0 2024-04-24T23:28:33.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:33.515+0000 7faa49e00640 1 --2- 172.21.15.6:0/1014283496 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7faa2806be60 0x7faa2806e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:28:33.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:33.515+0000 7faa49e00640 1 --2- 172.21.15.6:0/1014283496 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7faa2806be60 0x7faa2806e320 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7faa3c02a1e0 tx=0x7faa3c02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:28:33.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:33.519+0000 7faa42000640 1 -- 172.21.15.6:0/1014283496 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7faa44058590 con 0x7faa4c15e2b0 2024-04-24T23:28:33.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:33.716+0000 7faa50a00640 1 -- 172.21.15.6:0/1014283496 --> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7faa4c15e020 con 0x7faa2806be60 2024-04-24T23:28:33.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:33.726+0000 7faa42000640 1 -- 172.21.15.6:0/1014283496 <== mgr.4105 v2:172.21.15.190:6832/806769663 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+334806 (secure 0 0 0) 0x7faa4c15e020 con 0x7faa2806be60 2024-04-24T23:28:33.732 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:28:33.734 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-04-24T23:28:33.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:33.733+0000 7faa50a00640 1 -- 172.21.15.6:0/1014283496 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7faa2806be60 msgr2=0x7faa2806e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:28:33.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:33.733+0000 7faa50a00640 1 --2- 172.21.15.6:0/1014283496 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7faa2806be60 0x7faa2806e320 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7faa3c02a1e0 tx=0x7faa3c02e000 comp rx=0 tx=0).stop 2024-04-24T23:28:33.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:33.733+0000 7faa50a00640 1 -- 172.21.15.6:0/1014283496 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faa4c15e2b0 msgr2=0x7faa4c156010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:28:33.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:33.733+0000 7faa50a00640 1 --2- 172.21.15.6:0/1014283496 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faa4c15e2b0 0x7faa4c156010 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7faa4400c780 tx=0x7faa4400cc40 comp rx=0 tx=0).stop 2024-04-24T23:28:33.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:33.733+0000 7faa50a00640 1 -- 172.21.15.6:0/1014283496 shutdown_connections 2024-04-24T23:28:33.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:33.733+0000 7faa50a00640 1 --2- 172.21.15.6:0/1014283496 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7faa2806be60 0x7faa2806e320 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:28:33.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:33.733+0000 7faa50a00640 1 --2- 172.21.15.6:0/1014283496 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7faa4c15c1b0 0x7faa4c155ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:28:33.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:33.733+0000 7faa50a00640 1 --2- 172.21.15.6:0/1014283496 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faa4c15e2b0 0x7faa4c156010 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:28:33.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:33.733+0000 7faa50a00640 1 --2- 172.21.15.6:0/1014283496 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faa4c0ca520 0x7faa4c0cfd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:28:33.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:33.733+0000 7faa50a00640 1 -- 172.21.15.6:0/1014283496 >> 172.21.15.6:0/1014283496 conn(0x7faa4c06c0f0 msgr2=0x7faa4c15d490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:28:33.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:33.733+0000 7faa50a00640 1 -- 172.21.15.6:0/1014283496 shutdown_connections 2024-04-24T23:28:33.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:33.733+0000 7faa50a00640 1 -- 172.21.15.6:0/1014283496 wait complete. 2024-04-24T23:28:33.754 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":417,"stamp":"2024-04-24T23:28:32.883928+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":80618346,"num_objects":120,"num_object_clones":0,"num_object_copies":240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":120,"num_whiteouts":0,"num_read":2721,"num_read_kb":85149,"num_write":2065015,"num_write_kb":29276253,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25000,"num_omap_keys":50,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":146532,"ondisk_log_size":146532,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":7311296,"kb_used_data":160256,"kb_used_omap":12,"kb_used_meta":7150963,"kb_avail":747663424,"statfs":{"total":773094113280,"available":765607346176,"internally_reserved":0,"allocated":164102144,"data_stored":162374948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":7322586674},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002390"},"pg_stats":[{"pgid":"2.1f","version":"30'43","reported_seq":97,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:53.185318+0000","last_change":"2024-04-24T23:27:53.185318+0000","last_active":"2024-04-24T23:27:53.185318+0000","last_peered":"2024-04-24T23:27:53.185318+0000","last_clean":"2024-04-24T23:27:53.185318+0000","last_became_active":"2024-04-24T23:15:35.964357+0000","last_became_peered":"2024-04-24T23:15:35.964357+0000","last_unstale":"2024-04-24T23:27:53.185318+0000","last_undegraded":"2024-04-24T23:27:53.185318+0000","last_fullsized":"2024-04-24T23:27:53.185318+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'43","last_scrub_stamp":"2024-04-24T23:27:53.185145+0000","last_deep_scrub":"30'43","last_deep_scrub_stamp":"2024-04-24T23:27:53.185145+0000","last_clean_scrub_stamp":"2024-04-24T23:27:53.185145+0000","objects_scrubbed":7,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:37:01.920409+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194976,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":7,"num_read_kb":4102,"num_write":33,"num_write_kb":4110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"30'4","reported_seq":51,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:28:03.564026+0000","last_change":"2024-04-24T23:28:03.564062+0000","last_active":"2024-04-24T23:28:03.564026+0000","last_peered":"2024-04-24T23:28:03.564026+0000","last_clean":"2024-04-24T23:28:03.564026+0000","last_became_active":"2024-04-24T23:15:36.989543+0000","last_became_peered":"2024-04-24T23:15:36.989543+0000","last_unstale":"2024-04-24T23:28:03.564026+0000","last_undegraded":"2024-04-24T23:28:03.564026+0000","last_fullsized":"2024-04-24T23:28:03.564026+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'4","last_scrub_stamp":"2024-04-24T23:28:03.563929+0000","last_deep_scrub":"30'4","last_deep_scrub_stamp":"2024-04-24T23:28:03.563929+0000","last_clean_scrub_stamp":"2024-04-24T23:28:03.563929+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:11:47.175305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"30'4772","reported_seq":4876,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:28:05.082601+0000","last_change":"2024-04-24T23:28:05.082601+0000","last_active":"2024-04-24T23:28:05.082601+0000","last_peered":"2024-04-24T23:28:05.082601+0000","last_clean":"2024-04-24T23:28:05.082601+0000","last_became_active":"2024-04-24T23:16:18.299217+0000","last_became_peered":"2024-04-24T23:16:18.299217+0000","last_unstale":"2024-04-24T23:28:05.082601+0000","last_undegraded":"2024-04-24T23:28:05.082601+0000","last_fullsized":"2024-04-24T23:28:05.082601+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'4772","last_scrub_stamp":"2024-04-24T23:28:05.082434+0000","last_deep_scrub":"30'4772","last_deep_scrub_stamp":"2024-04-24T23:28:05.082434+0000","last_clean_scrub_stamp":"2024-04-24T23:28:05.082434+0000","objects_scrubbed":0,"log_size":4772,"log_dups_size":0,"ondisk_log_size":4772,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:17:51.312590+0000","scrub_duration":45,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":70,"num_read_kb":0,"num_write":69587,"num_write_kb":969009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4,1],"acting":[0,6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"30'15","reported_seq":65,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:57.556371+0000","last_change":"2024-04-24T23:27:57.556402+0000","last_active":"2024-04-24T23:27:57.556371+0000","last_peered":"2024-04-24T23:27:57.556371+0000","last_clean":"2024-04-24T23:27:57.556371+0000","last_became_active":"2024-04-24T23:15:35.945227+0000","last_became_peered":"2024-04-24T23:15:35.945227+0000","last_unstale":"2024-04-24T23:27:57.556371+0000","last_undegraded":"2024-04-24T23:27:57.556371+0000","last_fullsized":"2024-04-24T23:27:57.556371+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'15","last_scrub_stamp":"2024-04-24T23:27:57.556297+0000","last_deep_scrub":"30'15","last_deep_scrub_stamp":"2024-04-24T23:27:57.556297+0000","last_clean_scrub_stamp":"2024-04-24T23:27:57.556297+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:11:36.961216+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":4096,"num_write":8,"num_write_kb":4105,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1f","version":"30'15","reported_seq":62,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:28:06.488103+0000","last_change":"2024-04-24T23:28:06.488125+0000","last_active":"2024-04-24T23:28:06.488103+0000","last_peered":"2024-04-24T23:28:06.488103+0000","last_clean":"2024-04-24T23:28:06.488103+0000","last_became_active":"2024-04-24T23:15:36.998021+0000","last_became_peered":"2024-04-24T23:15:36.998021+0000","last_unstale":"2024-04-24T23:28:06.488103+0000","last_undegraded":"2024-04-24T23:28:06.488103+0000","last_fullsized":"2024-04-24T23:28:06.488103+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'15","last_scrub_stamp":"2024-04-24T23:28:06.488041+0000","last_deep_scrub":"30'15","last_deep_scrub_stamp":"2024-04-24T23:28:06.488041+0000","last_clean_scrub_stamp":"2024-04-24T23:28:06.488041+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:18:57.366081+0000","scrub_duration":45,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"30'4584","reported_seq":4699,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:56.164851+0000","last_change":"2024-04-24T23:27:56.164907+0000","last_active":"2024-04-24T23:27:56.164851+0000","last_peered":"2024-04-24T23:27:56.164851+0000","last_clean":"2024-04-24T23:27:56.164851+0000","last_became_active":"2024-04-24T23:15:41.007908+0000","last_became_peered":"2024-04-24T23:15:41.007908+0000","last_unstale":"2024-04-24T23:27:56.164851+0000","last_undegraded":"2024-04-24T23:27:56.164851+0000","last_fullsized":"2024-04-24T23:27:56.164851+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'4584","last_scrub_stamp":"2024-04-24T23:27:56.164679+0000","last_deep_scrub":"30'4584","last_deep_scrub_stamp":"2024-04-24T23:27:56.164679+0000","last_clean_scrub_stamp":"2024-04-24T23:27:56.164679+0000","objects_scrubbed":0,"log_size":4584,"log_dups_size":0,"ondisk_log_size":4584,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:30:37.102549+0000","scrub_duration":117,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":75,"num_read_kb":0,"num_write":66619,"num_write_kb":930809,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4,6],"acting":[2,7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1d","version":"30'53","reported_seq":115,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:53.104009+0000","last_change":"2024-04-24T23:27:53.104009+0000","last_active":"2024-04-24T23:27:53.104009+0000","last_peered":"2024-04-24T23:27:53.104009+0000","last_clean":"2024-04-24T23:27:53.104009+0000","last_became_active":"2024-04-24T23:15:35.967873+0000","last_became_peered":"2024-04-24T23:15:35.967873+0000","last_unstale":"2024-04-24T23:27:53.104009+0000","last_undegraded":"2024-04-24T23:27:53.104009+0000","last_fullsized":"2024-04-24T23:27:53.104009+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'53","last_scrub_stamp":"2024-04-24T23:27:53.103743+0000","last_deep_scrub":"30'53","last_deep_scrub_stamp":"2024-04-24T23:27:53.103743+0000","last_clean_scrub_stamp":"2024-04-24T23:27:53.103743+0000","objects_scrubbed":9,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:47:32.540188+0000","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":38,"num_read_kb":8226,"num_write":37,"num_write_kb":8205,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"30'8","reported_seq":54,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:51.455664+0000","last_change":"2024-04-24T23:27:50.990837+0000","last_active":"2024-04-24T23:27:51.455664+0000","last_peered":"2024-04-24T23:27:51.455664+0000","last_clean":"2024-04-24T23:27:51.455664+0000","last_became_active":"2024-04-24T23:15:36.997083+0000","last_became_peered":"2024-04-24T23:15:36.997083+0000","last_unstale":"2024-04-24T23:27:51.455664+0000","last_undegraded":"2024-04-24T23:27:51.455664+0000","last_fullsized":"2024-04-24T23:27:51.455664+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"27'7","last_scrub_stamp":"2024-04-24T23:27:50.990698+0000","last_deep_scrub":"27'7","last_deep_scrub_stamp":"2024-04-24T23:27:50.990698+0000","last_clean_scrub_stamp":"2024-04-24T23:27:50.990698+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:52:04.409030+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"30'4289","reported_seq":4399,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:28:00.134092+0000","last_change":"2024-04-24T23:28:00.134092+0000","last_active":"2024-04-24T23:28:00.134092+0000","last_peered":"2024-04-24T23:28:00.134092+0000","last_clean":"2024-04-24T23:28:00.134092+0000","last_became_active":"2024-04-24T23:15:41.024739+0000","last_became_peered":"2024-04-24T23:15:41.024739+0000","last_unstale":"2024-04-24T23:28:00.134092+0000","last_undegraded":"2024-04-24T23:28:00.134092+0000","last_fullsized":"2024-04-24T23:28:00.134092+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'4289","last_scrub_stamp":"2024-04-24T23:28:00.133912+0000","last_deep_scrub":"30'4289","last_deep_scrub_stamp":"2024-04-24T23:28:00.133912+0000","last_clean_scrub_stamp":"2024-04-24T23:28:00.133912+0000","objects_scrubbed":0,"log_size":4289,"log_dups_size":0,"ondisk_log_size":4289,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:46:24.969382+0000","scrub_duration":115,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":70,"num_read_kb":0,"num_write":62397,"num_write_kb":877947,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3,6],"acting":[2,5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1c","version":"30'21","reported_seq":77,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:51.356518+0000","last_change":"2024-04-24T23:27:51.145503+0000","last_active":"2024-04-24T23:27:51.356518+0000","last_peered":"2024-04-24T23:27:51.356518+0000","last_clean":"2024-04-24T23:27:51.356518+0000","last_became_active":"2024-04-24T23:15:35.966951+0000","last_became_peered":"2024-04-24T23:15:35.966951+0000","last_unstale":"2024-04-24T23:27:51.356518+0000","last_undegraded":"2024-04-24T23:27:51.356518+0000","last_fullsized":"2024-04-24T23:27:51.356518+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'20","last_scrub_stamp":"2024-04-24T23:27:51.145294+0000","last_deep_scrub":"30'20","last_deep_scrub_stamp":"2024-04-24T23:27:51.145294+0000","last_clean_scrub_stamp":"2024-04-24T23:27:51.145294+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:19:59.138191+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":232445,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":21,"num_read_kb":20,"num_write":17,"num_write_kb":234,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1d","version":"30'12","reported_seq":58,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:55.023031+0000","last_change":"2024-04-24T23:27:55.023031+0000","last_active":"2024-04-24T23:27:55.023031+0000","last_peered":"2024-04-24T23:27:55.023031+0000","last_clean":"2024-04-24T23:27:55.023031+0000","last_became_active":"2024-04-24T23:15:36.997735+0000","last_became_peered":"2024-04-24T23:15:36.997735+0000","last_unstale":"2024-04-24T23:27:55.023031+0000","last_undegraded":"2024-04-24T23:27:55.023031+0000","last_fullsized":"2024-04-24T23:27:55.023031+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'12","last_scrub_stamp":"2024-04-24T23:27:55.022792+0000","last_deep_scrub":"30'12","last_deep_scrub_stamp":"2024-04-24T23:27:55.022792+0000","last_clean_scrub_stamp":"2024-04-24T23:27:55.022792+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:20:58.930570+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"30'3772","reported_seq":3871,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:57.200767+0000","last_change":"2024-04-24T23:27:57.200767+0000","last_active":"2024-04-24T23:27:57.200767+0000","last_peered":"2024-04-24T23:27:57.200767+0000","last_clean":"2024-04-24T23:27:57.200767+0000","last_became_active":"2024-04-24T23:15:41.006199+0000","last_became_peered":"2024-04-24T23:15:41.006199+0000","last_unstale":"2024-04-24T23:27:57.200767+0000","last_undegraded":"2024-04-24T23:27:57.200767+0000","last_fullsized":"2024-04-24T23:27:57.200767+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3772","last_scrub_stamp":"2024-04-24T23:27:57.200437+0000","last_deep_scrub":"30'3772","last_deep_scrub_stamp":"2024-04-24T23:27:57.200437+0000","last_clean_scrub_stamp":"2024-04-24T23:27:57.200437+0000","objects_scrubbed":0,"log_size":3772,"log_dups_size":0,"ondisk_log_size":3772,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:28:20.543657+0000","scrub_duration":113,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":59,"num_read_kb":0,"num_write":54805,"num_write_kb":774507,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5,7],"acting":[2,6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"30'9","reported_seq":55,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:28:07.949858+0000","last_change":"2024-04-24T23:28:07.949896+0000","last_active":"2024-04-24T23:28:07.949858+0000","last_peered":"2024-04-24T23:28:07.949858+0000","last_clean":"2024-04-24T23:28:07.949858+0000","last_became_active":"2024-04-24T23:15:37.003544+0000","last_became_peered":"2024-04-24T23:15:37.003544+0000","last_unstale":"2024-04-24T23:28:07.949858+0000","last_undegraded":"2024-04-24T23:28:07.949858+0000","last_fullsized":"2024-04-24T23:28:07.949858+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'9","last_scrub_stamp":"2024-04-24T23:28:07.949765+0000","last_deep_scrub":"30'9","last_deep_scrub_stamp":"2024-04-24T23:28:07.949765+0000","last_clean_scrub_stamp":"2024-04-24T23:28:07.949765+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:24:58.985515+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"30'1465","reported_seq":1518,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:28:03.031075+0000","last_change":"2024-04-24T23:28:03.031123+0000","last_active":"2024-04-24T23:28:03.031075+0000","last_peered":"2024-04-24T23:28:03.031075+0000","last_clean":"2024-04-24T23:28:03.031075+0000","last_became_active":"2024-04-24T23:15:35.961960+0000","last_became_peered":"2024-04-24T23:15:35.961960+0000","last_unstale":"2024-04-24T23:28:03.031075+0000","last_undegraded":"2024-04-24T23:28:03.031075+0000","last_fullsized":"2024-04-24T23:28:03.031075+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'1465","last_scrub_stamp":"2024-04-24T23:28:03.030975+0000","last_deep_scrub":"30'1465","last_deep_scrub_stamp":"2024-04-24T23:28:03.030975+0000","last_clean_scrub_stamp":"2024-04-24T23:28:03.030975+0000","objects_scrubbed":3,"log_size":1465,"log_dups_size":0,"ondisk_log_size":1465,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:30:36.365621+0000","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":10,"num_read_kb":8200,"num_write":1452,"num_write_kb":5429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"30'4377","reported_seq":4482,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:28:11.986897+0000","last_change":"2024-04-24T23:28:11.987088+0000","last_active":"2024-04-24T23:28:11.986897+0000","last_peered":"2024-04-24T23:28:11.986897+0000","last_clean":"2024-04-24T23:28:11.986897+0000","last_became_active":"2024-04-24T23:15:41.020034+0000","last_became_peered":"2024-04-24T23:15:41.020034+0000","last_unstale":"2024-04-24T23:28:11.986897+0000","last_undegraded":"2024-04-24T23:28:11.986897+0000","last_fullsized":"2024-04-24T23:28:11.986897+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'4377","last_scrub_stamp":"2024-04-24T23:28:11.986573+0000","last_deep_scrub":"30'4377","last_deep_scrub_stamp":"2024-04-24T23:28:11.986573+0000","last_clean_scrub_stamp":"2024-04-24T23:28:11.986573+0000","objects_scrubbed":0,"log_size":4377,"log_dups_size":0,"ondisk_log_size":4377,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:33:35.191670+0000","scrub_duration":112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":65,"num_read_kb":0,"num_write":63662,"num_write_kb":904417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,5],"acting":[3,1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"30'9","reported_seq":55,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:55.457311+0000","last_change":"2024-04-24T23:27:55.457338+0000","last_active":"2024-04-24T23:27:55.457311+0000","last_peered":"2024-04-24T23:27:55.457311+0000","last_clean":"2024-04-24T23:27:55.457311+0000","last_became_active":"2024-04-24T23:15:36.971583+0000","last_became_peered":"2024-04-24T23:15:36.971583+0000","last_unstale":"2024-04-24T23:27:55.457311+0000","last_undegraded":"2024-04-24T23:27:55.457311+0000","last_fullsized":"2024-04-24T23:27:55.457311+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'9","last_scrub_stamp":"2024-04-24T23:27:55.457210+0000","last_deep_scrub":"30'9","last_deep_scrub_stamp":"2024-04-24T23:27:55.457210+0000","last_clean_scrub_stamp":"2024-04-24T23:27:55.457210+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:13:51.329870+0000","scrub_duration":57,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"30'12","reported_seq":61,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:52.425162+0000","last_change":"2024-04-24T23:27:52.425196+0000","last_active":"2024-04-24T23:27:52.425162+0000","last_peered":"2024-04-24T23:27:52.425162+0000","last_clean":"2024-04-24T23:27:52.425162+0000","last_became_active":"2024-04-24T23:15:35.945610+0000","last_became_peered":"2024-04-24T23:15:35.945610+0000","last_unstale":"2024-04-24T23:27:52.425162+0000","last_undegraded":"2024-04-24T23:27:52.425162+0000","last_fullsized":"2024-04-24T23:27:52.425162+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'12","last_scrub_stamp":"2024-04-24T23:27:52.425070+0000","last_deep_scrub":"30'12","last_deep_scrub_stamp":"2024-04-24T23:27:52.425070+0000","last_clean_scrub_stamp":"2024-04-24T23:27:52.425070+0000","objects_scrubbed":2,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:30:00.751396+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"30'4438","reported_seq":4548,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:28:04.149690+0000","last_change":"2024-04-24T23:28:04.149756+0000","last_active":"2024-04-24T23:28:04.149690+0000","last_peered":"2024-04-24T23:28:04.149690+0000","last_clean":"2024-04-24T23:28:04.149690+0000","last_became_active":"2024-04-24T23:15:41.029685+0000","last_became_peered":"2024-04-24T23:15:41.029685+0000","last_unstale":"2024-04-24T23:28:04.149690+0000","last_undegraded":"2024-04-24T23:28:04.149690+0000","last_fullsized":"2024-04-24T23:28:04.149690+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'4438","last_scrub_stamp":"2024-04-24T23:28:04.149507+0000","last_deep_scrub":"30'4438","last_deep_scrub_stamp":"2024-04-24T23:28:04.149507+0000","last_clean_scrub_stamp":"2024-04-24T23:28:04.149507+0000","objects_scrubbed":0,"log_size":4438,"log_dups_size":0,"ondisk_log_size":4438,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:08:29.255909+0000","scrub_duration":89,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":70,"num_read_kb":0,"num_write":64546,"num_write_kb":914755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6,1],"acting":[0,4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"30'9","reported_seq":55,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:57.213406+0000","last_change":"2024-04-24T23:27:57.213432+0000","last_active":"2024-04-24T23:27:57.213406+0000","last_peered":"2024-04-24T23:27:57.213406+0000","last_clean":"2024-04-24T23:27:57.213406+0000","last_became_active":"2024-04-24T23:15:36.965658+0000","last_became_peered":"2024-04-24T23:15:36.965658+0000","last_unstale":"2024-04-24T23:27:57.213406+0000","last_undegraded":"2024-04-24T23:27:57.213406+0000","last_fullsized":"2024-04-24T23:27:57.213406+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'9","last_scrub_stamp":"2024-04-24T23:27:57.213295+0000","last_deep_scrub":"30'9","last_deep_scrub_stamp":"2024-04-24T23:27:57.213295+0000","last_clean_scrub_stamp":"2024-04-24T23:27:57.213295+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:04:33.911020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"30'15","reported_seq":70,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:55.238261+0000","last_change":"2024-04-24T23:27:55.238323+0000","last_active":"2024-04-24T23:27:55.238261+0000","last_peered":"2024-04-24T23:27:55.238261+0000","last_clean":"2024-04-24T23:27:55.238261+0000","last_became_active":"2024-04-24T23:15:35.971972+0000","last_became_peered":"2024-04-24T23:15:35.971972+0000","last_unstale":"2024-04-24T23:27:55.238261+0000","last_undegraded":"2024-04-24T23:27:55.238261+0000","last_fullsized":"2024-04-24T23:27:55.238261+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'15","last_scrub_stamp":"2024-04-24T23:27:55.238155+0000","last_deep_scrub":"30'15","last_deep_scrub_stamp":"2024-04-24T23:27:55.238155+0000","last_clean_scrub_stamp":"2024-04-24T23:27:55.238155+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:39:46.964431+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":20,"num_read_kb":19,"num_write":11,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"30'5217","reported_seq":5338,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:59.524189+0000","last_change":"2024-04-24T23:27:59.524232+0000","last_active":"2024-04-24T23:27:59.524189+0000","last_peered":"2024-04-24T23:27:59.524189+0000","last_clean":"2024-04-24T23:27:59.524189+0000","last_became_active":"2024-04-24T23:15:41.019339+0000","last_became_peered":"2024-04-24T23:15:41.019339+0000","last_unstale":"2024-04-24T23:27:59.524189+0000","last_undegraded":"2024-04-24T23:27:59.524189+0000","last_fullsized":"2024-04-24T23:27:59.524189+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'5217","last_scrub_stamp":"2024-04-24T23:27:59.524092+0000","last_deep_scrub":"30'5217","last_deep_scrub_stamp":"2024-04-24T23:27:59.524092+0000","last_clean_scrub_stamp":"2024-04-24T23:27:59.524092+0000","objects_scrubbed":0,"log_size":5217,"log_dups_size":0,"ondisk_log_size":5217,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:40:34.204779+0000","scrub_duration":100,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":81,"num_read_kb":0,"num_write":75905,"num_write_kb":1074791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2,3],"acting":[7,4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"30'10","reported_seq":56,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:51.150860+0000","last_change":"2024-04-24T23:27:49.996036+0000","last_active":"2024-04-24T23:27:51.150860+0000","last_peered":"2024-04-24T23:27:51.150860+0000","last_clean":"2024-04-24T23:27:51.150860+0000","last_became_active":"2024-04-24T23:15:36.989336+0000","last_became_peered":"2024-04-24T23:15:36.989336+0000","last_unstale":"2024-04-24T23:27:51.150860+0000","last_undegraded":"2024-04-24T23:27:51.150860+0000","last_fullsized":"2024-04-24T23:27:51.150860+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'7","last_scrub_stamp":"2024-04-24T23:27:49.995954+0000","last_deep_scrub":"30'7","last_deep_scrub_stamp":"2024-04-24T23:27:49.995954+0000","last_clean_scrub_stamp":"2024-04-24T23:27:49.995954+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:03:50.327244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.8","version":"30'35","reported_seq":86,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:51.136208+0000","last_change":"2024-04-24T23:27:50.103062+0000","last_active":"2024-04-24T23:27:51.136208+0000","last_peered":"2024-04-24T23:27:51.136208+0000","last_clean":"2024-04-24T23:27:51.136208+0000","last_became_active":"2024-04-24T23:15:35.965043+0000","last_became_peered":"2024-04-24T23:15:35.965043+0000","last_unstale":"2024-04-24T23:27:51.136208+0000","last_undegraded":"2024-04-24T23:27:51.136208+0000","last_fullsized":"2024-04-24T23:27:51.136208+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'33","last_scrub_stamp":"2024-04-24T23:27:50.102956+0000","last_deep_scrub":"30'33","last_deep_scrub_stamp":"2024-04-24T23:27:50.102956+0000","last_clean_scrub_stamp":"2024-04-24T23:27:50.102956+0000","objects_scrubbed":2,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:06:47.021497+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":4102,"num_write":30,"num_write_kb":4111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"30'3837","reported_seq":3938,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:28:01.150193+0000","last_change":"2024-04-24T23:28:01.150257+0000","last_active":"2024-04-24T23:28:01.150193+0000","last_peered":"2024-04-24T23:28:01.150193+0000","last_clean":"2024-04-24T23:28:01.150193+0000","last_became_active":"2024-04-24T23:15:41.024731+0000","last_became_peered":"2024-04-24T23:15:41.024731+0000","last_unstale":"2024-04-24T23:28:01.150193+0000","last_undegraded":"2024-04-24T23:28:01.150193+0000","last_fullsized":"2024-04-24T23:28:01.150193+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3837","last_scrub_stamp":"2024-04-24T23:28:01.150009+0000","last_deep_scrub":"30'3837","last_deep_scrub_stamp":"2024-04-24T23:28:01.150009+0000","last_clean_scrub_stamp":"2024-04-24T23:28:01.150009+0000","objects_scrubbed":0,"log_size":3837,"log_dups_size":0,"ondisk_log_size":3837,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:30:36.770016+0000","scrub_duration":115,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":61,"num_read_kb":0,"num_write":55573,"num_write_kb":793811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,7],"acting":[2,1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"30'19","reported_seq":65,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:56.258479+0000","last_change":"2024-04-24T23:27:56.258504+0000","last_active":"2024-04-24T23:27:56.258479+0000","last_peered":"2024-04-24T23:27:56.258479+0000","last_clean":"2024-04-24T23:27:56.258479+0000","last_became_active":"2024-04-24T23:15:36.965547+0000","last_became_peered":"2024-04-24T23:15:36.965547+0000","last_unstale":"2024-04-24T23:27:56.258479+0000","last_undegraded":"2024-04-24T23:27:56.258479+0000","last_fullsized":"2024-04-24T23:27:56.258479+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'19","last_scrub_stamp":"2024-04-24T23:27:56.258383+0000","last_deep_scrub":"30'19","last_deep_scrub_stamp":"2024-04-24T23:27:56.258383+0000","last_clean_scrub_stamp":"2024-04-24T23:27:56.258383+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:09:34.691733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"30'15","reported_seq":71,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:28:05.986306+0000","last_change":"2024-04-24T23:28:05.986421+0000","last_active":"2024-04-24T23:28:05.986306+0000","last_peered":"2024-04-24T23:28:05.986306+0000","last_clean":"2024-04-24T23:28:05.986306+0000","last_became_active":"2024-04-24T23:15:35.970184+0000","last_became_peered":"2024-04-24T23:15:35.970184+0000","last_unstale":"2024-04-24T23:28:05.986306+0000","last_undegraded":"2024-04-24T23:28:05.986306+0000","last_fullsized":"2024-04-24T23:28:05.986306+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'15","last_scrub_stamp":"2024-04-24T23:28:05.986152+0000","last_deep_scrub":"30'15","last_deep_scrub_stamp":"2024-04-24T23:28:05.986152+0000","last_clean_scrub_stamp":"2024-04-24T23:28:05.986152+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:30:46.811111+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":24,"num_write":11,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"30'4423","reported_seq":4538,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:28:02.186881+0000","last_change":"2024-04-24T23:28:02.186881+0000","last_active":"2024-04-24T23:28:02.186881+0000","last_peered":"2024-04-24T23:28:02.186881+0000","last_clean":"2024-04-24T23:28:02.186881+0000","last_became_active":"2024-04-24T23:15:41.025137+0000","last_became_peered":"2024-04-24T23:15:41.025137+0000","last_unstale":"2024-04-24T23:28:02.186881+0000","last_undegraded":"2024-04-24T23:28:02.186881+0000","last_fullsized":"2024-04-24T23:28:02.186881+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'4423","last_scrub_stamp":"2024-04-24T23:28:02.186714+0000","last_deep_scrub":"30'4423","last_deep_scrub_stamp":"2024-04-24T23:28:02.186714+0000","last_clean_scrub_stamp":"2024-04-24T23:28:02.186714+0000","objects_scrubbed":0,"log_size":4423,"log_dups_size":0,"ondisk_log_size":4423,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:53:07.036567+0000","scrub_duration":106,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":74,"num_read_kb":0,"num_write":64413,"num_write_kb":917129,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,3,1],"acting":[2,6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"30'13","reported_seq":59,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:58.282620+0000","last_change":"2024-04-24T23:27:58.282646+0000","last_active":"2024-04-24T23:27:58.282620+0000","last_peered":"2024-04-24T23:27:58.282620+0000","last_clean":"2024-04-24T23:27:58.282620+0000","last_became_active":"2024-04-24T23:15:36.967098+0000","last_became_peered":"2024-04-24T23:15:36.967098+0000","last_unstale":"2024-04-24T23:27:58.282620+0000","last_undegraded":"2024-04-24T23:27:58.282620+0000","last_fullsized":"2024-04-24T23:27:58.282620+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'13","last_scrub_stamp":"2024-04-24T23:27:58.282516+0000","last_deep_scrub":"30'13","last_deep_scrub_stamp":"2024-04-24T23:27:58.282516+0000","last_clean_scrub_stamp":"2024-04-24T23:27:58.282516+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:49:04.125156+0000","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"30'271","reported_seq":320,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:54.195309+0000","last_change":"2024-04-24T23:27:54.195357+0000","last_active":"2024-04-24T23:27:54.195309+0000","last_peered":"2024-04-24T23:27:54.195309+0000","last_clean":"2024-04-24T23:27:54.195309+0000","last_became_active":"2024-04-24T23:15:35.970430+0000","last_became_peered":"2024-04-24T23:15:35.970430+0000","last_unstale":"2024-04-24T23:27:54.195309+0000","last_undegraded":"2024-04-24T23:27:54.195309+0000","last_fullsized":"2024-04-24T23:27:54.195309+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'271","last_scrub_stamp":"2024-04-24T23:27:54.195222+0000","last_deep_scrub":"30'271","last_deep_scrub_stamp":"2024-04-24T23:27:54.195222+0000","last_clean_scrub_stamp":"2024-04-24T23:27:54.195222+0000","objects_scrubbed":4,"log_size":271,"log_dups_size":0,"ondisk_log_size":271,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:49:13.561921+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":9,"num_write":563,"num_write_kb":230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"30'4776","reported_seq":4894,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:28:03.383173+0000","last_change":"2024-04-24T23:28:03.383200+0000","last_active":"2024-04-24T23:28:03.383173+0000","last_peered":"2024-04-24T23:28:03.383173+0000","last_clean":"2024-04-24T23:28:03.383173+0000","last_became_active":"2024-04-24T23:15:41.004648+0000","last_became_peered":"2024-04-24T23:15:41.004648+0000","last_unstale":"2024-04-24T23:28:03.383173+0000","last_undegraded":"2024-04-24T23:28:03.383173+0000","last_fullsized":"2024-04-24T23:28:03.383173+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'4776","last_scrub_stamp":"2024-04-24T23:28:03.383076+0000","last_deep_scrub":"30'4776","last_deep_scrub_stamp":"2024-04-24T23:28:03.383076+0000","last_clean_scrub_stamp":"2024-04-24T23:28:03.383076+0000","objects_scrubbed":0,"log_size":4776,"log_dups_size":0,"ondisk_log_size":4776,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:24:27.615324+0000","scrub_duration":84,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":77,"num_read_kb":0,"num_write":69584,"num_write_kb":1012990,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6,3],"acting":[4,1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.4","version":"30'9","reported_seq":56,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:56.350830+0000","last_change":"2024-04-24T23:27:56.350830+0000","last_active":"2024-04-24T23:27:56.350830+0000","last_peered":"2024-04-24T23:27:56.350830+0000","last_clean":"2024-04-24T23:27:56.350830+0000","last_became_active":"2024-04-24T23:15:36.966746+0000","last_became_peered":"2024-04-24T23:15:36.966746+0000","last_unstale":"2024-04-24T23:27:56.350830+0000","last_undegraded":"2024-04-24T23:27:56.350830+0000","last_fullsized":"2024-04-24T23:27:56.350830+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'9","last_scrub_stamp":"2024-04-24T23:27:56.350726+0000","last_deep_scrub":"30'9","last_deep_scrub_stamp":"2024-04-24T23:27:56.350726+0000","last_clean_scrub_stamp":"2024-04-24T23:27:56.350726+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:56:17.971553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"30'11","reported_seq":63,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:28:04.943896+0000","last_change":"2024-04-24T23:28:04.943896+0000","last_active":"2024-04-24T23:28:04.943896+0000","last_peered":"2024-04-24T23:28:04.943896+0000","last_clean":"2024-04-24T23:28:04.943896+0000","last_became_active":"2024-04-24T23:15:35.965631+0000","last_became_peered":"2024-04-24T23:15:35.965631+0000","last_unstale":"2024-04-24T23:28:04.943896+0000","last_undegraded":"2024-04-24T23:28:04.943896+0000","last_fullsized":"2024-04-24T23:28:04.943896+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'11","last_scrub_stamp":"2024-04-24T23:28:04.943646+0000","last_deep_scrub":"30'11","last_deep_scrub_stamp":"2024-04-24T23:28:04.943646+0000","last_clean_scrub_stamp":"2024-04-24T23:28:04.943646+0000","objects_scrubbed":3,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:43:42.634998+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"30'4046","reported_seq":4147,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:28:04.509094+0000","last_change":"2024-04-24T23:28:04.509094+0000","last_active":"2024-04-24T23:28:04.509094+0000","last_peered":"2024-04-24T23:28:04.509094+0000","last_clean":"2024-04-24T23:28:04.509094+0000","last_became_active":"2024-04-24T23:15:41.010982+0000","last_became_peered":"2024-04-24T23:15:41.010982+0000","last_unstale":"2024-04-24T23:28:04.509094+0000","last_undegraded":"2024-04-24T23:28:04.509094+0000","last_fullsized":"2024-04-24T23:28:04.509094+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'4046","last_scrub_stamp":"2024-04-24T23:28:04.508979+0000","last_deep_scrub":"30'4046","last_deep_scrub_stamp":"2024-04-24T23:28:04.508979+0000","last_clean_scrub_stamp":"2024-04-24T23:28:04.508979+0000","objects_scrubbed":0,"log_size":4046,"log_dups_size":0,"ondisk_log_size":4046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:56:10.599487+0000","scrub_duration":57,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":60,"num_read_kb":0,"num_write":58654,"num_write_kb":828032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2,7],"acting":[6,0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"30'14","reported_seq":61,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:51.305046+0000","last_change":"2024-04-24T23:27:50.043832+0000","last_active":"2024-04-24T23:27:51.305046+0000","last_peered":"2024-04-24T23:27:51.305046+0000","last_clean":"2024-04-24T23:27:51.305046+0000","last_became_active":"2024-04-24T23:15:36.974872+0000","last_became_peered":"2024-04-24T23:15:36.974872+0000","last_unstale":"2024-04-24T23:27:51.305046+0000","last_undegraded":"2024-04-24T23:27:51.305046+0000","last_fullsized":"2024-04-24T23:27:51.305046+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'10","last_scrub_stamp":"2024-04-24T23:27:50.043716+0000","last_deep_scrub":"30'10","last_deep_scrub_stamp":"2024-04-24T23:27:50.043716+0000","last_clean_scrub_stamp":"2024-04-24T23:27:50.043716+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:40:08.379839+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"30'6","reported_seq":55,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:56.509920+0000","last_change":"2024-04-24T23:27:56.509920+0000","last_active":"2024-04-24T23:27:56.509920+0000","last_peered":"2024-04-24T23:27:56.509920+0000","last_clean":"2024-04-24T23:27:56.509920+0000","last_became_active":"2024-04-24T23:15:35.945094+0000","last_became_peered":"2024-04-24T23:15:35.945094+0000","last_unstale":"2024-04-24T23:27:56.509920+0000","last_undegraded":"2024-04-24T23:27:56.509920+0000","last_fullsized":"2024-04-24T23:27:56.509920+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'6","last_scrub_stamp":"2024-04-24T23:27:56.509790+0000","last_deep_scrub":"30'6","last_deep_scrub_stamp":"2024-04-24T23:27:56.509790+0000","last_clean_scrub_stamp":"2024-04-24T23:27:56.509790+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:50.913465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"30'4962","reported_seq":5080,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:28:06.467609+0000","last_change":"2024-04-24T23:28:06.467609+0000","last_active":"2024-04-24T23:28:06.467609+0000","last_peered":"2024-04-24T23:28:06.467609+0000","last_clean":"2024-04-24T23:28:06.467609+0000","last_became_active":"2024-04-24T23:15:41.021645+0000","last_became_peered":"2024-04-24T23:15:41.021645+0000","last_unstale":"2024-04-24T23:28:06.467609+0000","last_undegraded":"2024-04-24T23:28:06.467609+0000","last_fullsized":"2024-04-24T23:28:06.467609+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'4962","last_scrub_stamp":"2024-04-24T23:28:06.467417+0000","last_deep_scrub":"30'4962","last_deep_scrub_stamp":"2024-04-24T23:28:06.467417+0000","last_clean_scrub_stamp":"2024-04-24T23:28:06.467417+0000","objects_scrubbed":0,"log_size":4962,"log_dups_size":0,"ondisk_log_size":4962,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:37:21.549128+0000","scrub_duration":59,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":77,"num_read_kb":0,"num_write":72436,"num_write_kb":1021329,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5,7],"acting":[6,3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.3","version":"30'10","reported_seq":57,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:51.409168+0000","last_change":"2024-04-24T23:27:48.119568+0000","last_active":"2024-04-24T23:27:51.409168+0000","last_peered":"2024-04-24T23:27:51.409168+0000","last_clean":"2024-04-24T23:27:51.409168+0000","last_became_active":"2024-04-24T23:15:36.990963+0000","last_became_peered":"2024-04-24T23:15:36.990963+0000","last_unstale":"2024-04-24T23:27:51.409168+0000","last_undegraded":"2024-04-24T23:27:51.409168+0000","last_fullsized":"2024-04-24T23:27:51.409168+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'8","last_scrub_stamp":"2024-04-24T23:27:48.119426+0000","last_deep_scrub":"30'8","last_deep_scrub_stamp":"2024-04-24T23:27:48.119426+0000","last_clean_scrub_stamp":"2024-04-24T23:27:48.119426+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:16:00.599432+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"30'50","reported_seq":109,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:28:01.006222+0000","last_change":"2024-04-24T23:28:01.006222+0000","last_active":"2024-04-24T23:28:01.006222+0000","last_peered":"2024-04-24T23:28:01.006222+0000","last_clean":"2024-04-24T23:28:01.006222+0000","last_became_active":"2024-04-24T23:15:35.957432+0000","last_became_peered":"2024-04-24T23:15:35.957432+0000","last_unstale":"2024-04-24T23:28:01.006222+0000","last_undegraded":"2024-04-24T23:28:01.006222+0000","last_fullsized":"2024-04-24T23:28:01.006222+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'50","last_scrub_stamp":"2024-04-24T23:28:01.005988+0000","last_deep_scrub":"30'50","last_deep_scrub_stamp":"2024-04-24T23:28:01.005988+0000","last_clean_scrub_stamp":"2024-04-24T23:28:01.005988+0000","objects_scrubbed":4,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:00:15.235585+0000","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8389784,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":19,"num_read_kb":8206,"num_write":36,"num_write_kb":8211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.4","version":"30'4199","reported_seq":4305,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:28:02.548075+0000","last_change":"2024-04-24T23:28:02.548075+0000","last_active":"2024-04-24T23:28:02.548075+0000","last_peered":"2024-04-24T23:28:02.548075+0000","last_clean":"2024-04-24T23:28:02.548075+0000","last_became_active":"2024-04-24T23:15:41.007449+0000","last_became_peered":"2024-04-24T23:15:41.007449+0000","last_unstale":"2024-04-24T23:28:02.548075+0000","last_undegraded":"2024-04-24T23:28:02.548075+0000","last_fullsized":"2024-04-24T23:28:02.548075+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'4199","last_scrub_stamp":"2024-04-24T23:28:02.547910+0000","last_deep_scrub":"30'4199","last_deep_scrub_stamp":"2024-04-24T23:28:02.547910+0000","last_clean_scrub_stamp":"2024-04-24T23:28:02.547910+0000","objects_scrubbed":0,"log_size":4199,"log_dups_size":0,"ondisk_log_size":4199,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:36:04.575953+0000","scrub_duration":66,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":65,"num_read_kb":0,"num_write":60924,"num_write_kb":882911,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0,3],"acting":[6,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"30'15","reported_seq":62,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:54.501709+0000","last_change":"2024-04-24T23:27:54.501736+0000","last_active":"2024-04-24T23:27:54.501709+0000","last_peered":"2024-04-24T23:27:54.501709+0000","last_clean":"2024-04-24T23:27:54.501709+0000","last_became_active":"2024-04-24T23:15:36.970159+0000","last_became_peered":"2024-04-24T23:15:36.970159+0000","last_unstale":"2024-04-24T23:27:54.501709+0000","last_undegraded":"2024-04-24T23:27:54.501709+0000","last_fullsized":"2024-04-24T23:27:54.501709+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'15","last_scrub_stamp":"2024-04-24T23:27:54.501618+0000","last_deep_scrub":"30'15","last_deep_scrub_stamp":"2024-04-24T23:27:54.501618+0000","last_clean_scrub_stamp":"2024-04-24T23:27:54.501618+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:13:00.741443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"30'47","reported_seq":129,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:52.378195+0000","last_change":"2024-04-24T23:27:52.378195+0000","last_active":"2024-04-24T23:27:52.378195+0000","last_peered":"2024-04-24T23:27:52.378195+0000","last_clean":"2024-04-24T23:27:52.378195+0000","last_became_active":"2024-04-24T23:15:35.945185+0000","last_became_peered":"2024-04-24T23:15:35.945185+0000","last_unstale":"2024-04-24T23:27:52.378195+0000","last_undegraded":"2024-04-24T23:27:52.378195+0000","last_fullsized":"2024-04-24T23:27:52.378195+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'47","last_scrub_stamp":"2024-04-24T23:27:52.378078+0000","last_deep_scrub":"30'47","last_deep_scrub_stamp":"2024-04-24T23:27:52.378078+0000","last_clean_scrub_stamp":"2024-04-24T23:27:52.378078+0000","objects_scrubbed":4,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:14:33.145761+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":94280,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":33,"num_read_kb":113,"num_write":40,"num_write_kb":118,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"30'4761","reported_seq":4875,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:28:05.513229+0000","last_change":"2024-04-24T23:28:05.513253+0000","last_active":"2024-04-24T23:28:05.513229+0000","last_peered":"2024-04-24T23:28:05.513229+0000","last_clean":"2024-04-24T23:28:05.513229+0000","last_became_active":"2024-04-24T23:15:41.014137+0000","last_became_peered":"2024-04-24T23:15:41.014137+0000","last_unstale":"2024-04-24T23:28:05.513229+0000","last_undegraded":"2024-04-24T23:28:05.513229+0000","last_fullsized":"2024-04-24T23:28:05.513229+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'4761","last_scrub_stamp":"2024-04-24T23:28:05.513152+0000","last_deep_scrub":"30'4761","last_deep_scrub_stamp":"2024-04-24T23:28:05.513152+0000","last_clean_scrub_stamp":"2024-04-24T23:28:05.513152+0000","objects_scrubbed":0,"log_size":4761,"log_dups_size":0,"ondisk_log_size":4761,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:38:14.069894+0000","scrub_duration":78,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":74,"num_read_kb":0,"num_write":69114,"num_write_kb":969120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,7],"acting":[6,0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"30'8","reported_seq":55,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:58.375786+0000","last_change":"2024-04-24T23:27:58.375786+0000","last_active":"2024-04-24T23:27:58.375786+0000","last_peered":"2024-04-24T23:27:58.375786+0000","last_clean":"2024-04-24T23:27:58.375786+0000","last_became_active":"2024-04-24T23:15:36.981502+0000","last_became_peered":"2024-04-24T23:15:36.981502+0000","last_unstale":"2024-04-24T23:27:58.375786+0000","last_undegraded":"2024-04-24T23:27:58.375786+0000","last_fullsized":"2024-04-24T23:27:58.375786+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'8","last_scrub_stamp":"2024-04-24T23:27:58.375627+0000","last_deep_scrub":"30'8","last_deep_scrub_stamp":"2024-04-24T23:27:58.375627+0000","last_clean_scrub_stamp":"2024-04-24T23:27:58.375627+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:54:33.616309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"30'41","reported_seq":92,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:28:01.976267+0000","last_change":"2024-04-24T23:28:01.976314+0000","last_active":"2024-04-24T23:28:01.976267+0000","last_peered":"2024-04-24T23:28:01.976267+0000","last_clean":"2024-04-24T23:28:01.976267+0000","last_became_active":"2024-04-24T23:15:35.960180+0000","last_became_peered":"2024-04-24T23:15:35.960180+0000","last_unstale":"2024-04-24T23:28:01.976267+0000","last_undegraded":"2024-04-24T23:28:01.976267+0000","last_fullsized":"2024-04-24T23:28:01.976267+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'41","last_scrub_stamp":"2024-04-24T23:28:01.976171+0000","last_deep_scrub":"30'41","last_deep_scrub_stamp":"2024-04-24T23:28:01.976171+0000","last_clean_scrub_stamp":"2024-04-24T23:28:01.976171+0000","objects_scrubbed":3,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:18:24.235567+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":8192,"num_write":28,"num_write_kb":8204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"30'3864","reported_seq":3965,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:28:09.559663+0000","last_change":"2024-04-24T23:28:09.559707+0000","last_active":"2024-04-24T23:28:09.559663+0000","last_peered":"2024-04-24T23:28:09.559663+0000","last_clean":"2024-04-24T23:28:09.559663+0000","last_became_active":"2024-04-24T23:15:41.012493+0000","last_became_peered":"2024-04-24T23:15:41.012493+0000","last_unstale":"2024-04-24T23:28:09.559663+0000","last_undegraded":"2024-04-24T23:28:09.559663+0000","last_fullsized":"2024-04-24T23:28:09.559663+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3864","last_scrub_stamp":"2024-04-24T23:28:09.559534+0000","last_deep_scrub":"30'3864","last_deep_scrub_stamp":"2024-04-24T23:28:09.559534+0000","last_clean_scrub_stamp":"2024-04-24T23:28:09.559534+0000","objects_scrubbed":0,"log_size":3864,"log_dups_size":0,"ondisk_log_size":3864,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:19:42.352711+0000","scrub_duration":57,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":61,"num_read_kb":307,"num_write":55967,"num_write_kb":780987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6,1],"acting":[5,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"30'8","reported_seq":55,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:28:00.306838+0000","last_change":"2024-04-24T23:28:00.306838+0000","last_active":"2024-04-24T23:28:00.306838+0000","last_peered":"2024-04-24T23:28:00.306838+0000","last_clean":"2024-04-24T23:28:00.306838+0000","last_became_active":"2024-04-24T23:15:36.980837+0000","last_became_peered":"2024-04-24T23:15:36.980837+0000","last_unstale":"2024-04-24T23:28:00.306838+0000","last_undegraded":"2024-04-24T23:28:00.306838+0000","last_fullsized":"2024-04-24T23:28:00.306838+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'8","last_scrub_stamp":"2024-04-24T23:28:00.306715+0000","last_deep_scrub":"30'8","last_deep_scrub_stamp":"2024-04-24T23:28:00.306715+0000","last_clean_scrub_stamp":"2024-04-24T23:28:00.306715+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:08:26.962757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.3","version":"30'12","reported_seq":65,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:58.974989+0000","last_change":"2024-04-24T23:27:58.974989+0000","last_active":"2024-04-24T23:27:58.974989+0000","last_peered":"2024-04-24T23:27:58.974989+0000","last_clean":"2024-04-24T23:27:58.974989+0000","last_became_active":"2024-04-24T23:15:35.952655+0000","last_became_peered":"2024-04-24T23:15:35.952655+0000","last_unstale":"2024-04-24T23:27:58.974989+0000","last_undegraded":"2024-04-24T23:27:58.974989+0000","last_fullsized":"2024-04-24T23:27:58.974989+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'12","last_scrub_stamp":"2024-04-24T23:27:58.974739+0000","last_deep_scrub":"30'12","last_deep_scrub_stamp":"2024-04-24T23:27:58.974739+0000","last_clean_scrub_stamp":"2024-04-24T23:27:58.974739+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:04:13.958405+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":7,"num_read_kb":11,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"30'119","reported_seq":207,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:47.179514+0000","last_change":"2024-04-24T23:27:47.179576+0000","last_active":"2024-04-24T23:27:47.179514+0000","last_peered":"2024-04-24T23:27:47.179514+0000","last_clean":"2024-04-24T23:27:47.179514+0000","last_became_active":"2024-04-24T23:15:25.820030+0000","last_became_peered":"2024-04-24T23:15:25.820030+0000","last_unstale":"2024-04-24T23:27:47.179514+0000","last_undegraded":"2024-04-24T23:27:47.179514+0000","last_fullsized":"2024-04-24T23:27:47.179514+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'119","last_scrub_stamp":"2024-04-24T23:27:47.179391+0000","last_deep_scrub":"30'119","last_deep_scrub_stamp":"2024-04-24T23:27:47.179391+0000","last_clean_scrub_stamp":"2024-04-24T23:27:47.179391+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:06:58.842359+0000","scrub_duration":90,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"30'4286","reported_seq":4403,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:28:12.917561+0000","last_change":"2024-04-24T23:28:12.917561+0000","last_active":"2024-04-24T23:28:12.917561+0000","last_peered":"2024-04-24T23:28:12.917561+0000","last_clean":"2024-04-24T23:28:12.917561+0000","last_became_active":"2024-04-24T23:15:41.022697+0000","last_became_peered":"2024-04-24T23:15:41.022697+0000","last_unstale":"2024-04-24T23:28:12.917561+0000","last_undegraded":"2024-04-24T23:28:12.917561+0000","last_fullsized":"2024-04-24T23:28:12.917561+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'4286","last_scrub_stamp":"2024-04-24T23:28:12.917446+0000","last_deep_scrub":"30'4286","last_deep_scrub_stamp":"2024-04-24T23:28:12.917446+0000","last_clean_scrub_stamp":"2024-04-24T23:28:12.917446+0000","objects_scrubbed":0,"log_size":4286,"log_dups_size":0,"ondisk_log_size":4286,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:36:07.323496+0000","scrub_duration":87,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":76,"num_read_kb":0,"num_write":62654,"num_write_kb":898105,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2,0],"acting":[3,1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.d","version":"30'15","reported_seq":61,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:28:06.947675+0000","last_change":"2024-04-24T23:28:06.947732+0000","last_active":"2024-04-24T23:28:06.947675+0000","last_peered":"2024-04-24T23:28:06.947675+0000","last_clean":"2024-04-24T23:28:06.947675+0000","last_became_active":"2024-04-24T23:15:37.000179+0000","last_became_peered":"2024-04-24T23:15:37.000179+0000","last_unstale":"2024-04-24T23:28:06.947675+0000","last_undegraded":"2024-04-24T23:28:06.947675+0000","last_fullsized":"2024-04-24T23:28:06.947675+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'15","last_scrub_stamp":"2024-04-24T23:28:06.947489+0000","last_deep_scrub":"30'15","last_deep_scrub_stamp":"2024-04-24T23:28:06.947489+0000","last_clean_scrub_stamp":"2024-04-24T23:28:06.947489+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:36:45.048151+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.c","version":"30'15","reported_seq":63,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:53.396776+0000","last_change":"2024-04-24T23:27:53.396809+0000","last_active":"2024-04-24T23:27:53.396776+0000","last_peered":"2024-04-24T23:27:53.396776+0000","last_clean":"2024-04-24T23:27:53.396776+0000","last_became_active":"2024-04-24T23:15:35.965758+0000","last_became_peered":"2024-04-24T23:15:35.965758+0000","last_unstale":"2024-04-24T23:27:53.396776+0000","last_undegraded":"2024-04-24T23:27:53.396776+0000","last_fullsized":"2024-04-24T23:27:53.396776+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'15","last_scrub_stamp":"2024-04-24T23:27:53.396692+0000","last_deep_scrub":"30'15","last_deep_scrub_stamp":"2024-04-24T23:27:53.396692+0000","last_clean_scrub_stamp":"2024-04-24T23:27:53.396692+0000","objects_scrubbed":3,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:03:53.462987+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2157,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"30'4384","reported_seq":4494,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:59.162578+0000","last_change":"2024-04-24T23:27:59.162644+0000","last_active":"2024-04-24T23:27:59.162578+0000","last_peered":"2024-04-24T23:27:59.162578+0000","last_clean":"2024-04-24T23:27:59.162578+0000","last_became_active":"2024-04-24T23:15:41.024041+0000","last_became_peered":"2024-04-24T23:15:41.024041+0000","last_unstale":"2024-04-24T23:27:59.162578+0000","last_undegraded":"2024-04-24T23:27:59.162578+0000","last_fullsized":"2024-04-24T23:27:59.162578+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'4384","last_scrub_stamp":"2024-04-24T23:27:59.162380+0000","last_deep_scrub":"30'4384","last_deep_scrub_stamp":"2024-04-24T23:27:59.162380+0000","last_clean_scrub_stamp":"2024-04-24T23:27:59.162380+0000","objects_scrubbed":0,"log_size":4384,"log_dups_size":0,"ondisk_log_size":4384,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:43:12.603815+0000","scrub_duration":106,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":70,"num_read_kb":0,"num_write":63886,"num_write_kb":912579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7,4],"acting":[2,3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"30'13","reported_seq":59,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:56.100904+0000","last_change":"2024-04-24T23:27:56.100963+0000","last_active":"2024-04-24T23:27:56.100904+0000","last_peered":"2024-04-24T23:27:56.100904+0000","last_clean":"2024-04-24T23:27:56.100904+0000","last_became_active":"2024-04-24T23:15:36.979611+0000","last_became_peered":"2024-04-24T23:15:36.979611+0000","last_unstale":"2024-04-24T23:27:56.100904+0000","last_undegraded":"2024-04-24T23:27:56.100904+0000","last_fullsized":"2024-04-24T23:27:56.100904+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'13","last_scrub_stamp":"2024-04-24T23:27:56.100727+0000","last_deep_scrub":"30'13","last_deep_scrub_stamp":"2024-04-24T23:27:56.100727+0000","last_clean_scrub_stamp":"2024-04-24T23:27:56.100727+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:31:52.409110+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"30'28","reported_seq":78,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:58.560132+0000","last_change":"2024-04-24T23:27:58.560160+0000","last_active":"2024-04-24T23:27:58.560132+0000","last_peered":"2024-04-24T23:27:58.560132+0000","last_clean":"2024-04-24T23:27:58.560132+0000","last_became_active":"2024-04-24T23:15:35.964873+0000","last_became_peered":"2024-04-24T23:15:35.964873+0000","last_unstale":"2024-04-24T23:27:58.560132+0000","last_undegraded":"2024-04-24T23:27:58.560132+0000","last_fullsized":"2024-04-24T23:27:58.560132+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'28","last_scrub_stamp":"2024-04-24T23:27:58.560042+0000","last_deep_scrub":"30'28","last_deep_scrub_stamp":"2024-04-24T23:27:58.560042+0000","last_clean_scrub_stamp":"2024-04-24T23:27:58.560042+0000","objects_scrubbed":2,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:13:43.031451+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194374,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":4097,"num_write":20,"num_write_kb":4108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"30'4134","reported_seq":4245,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:28:07.551932+0000","last_change":"2024-04-24T23:28:07.551988+0000","last_active":"2024-04-24T23:28:07.551932+0000","last_peered":"2024-04-24T23:28:07.551932+0000","last_clean":"2024-04-24T23:28:07.551932+0000","last_became_active":"2024-04-24T23:15:41.003335+0000","last_became_peered":"2024-04-24T23:15:41.003335+0000","last_unstale":"2024-04-24T23:28:07.551932+0000","last_undegraded":"2024-04-24T23:28:07.551932+0000","last_fullsized":"2024-04-24T23:28:07.551932+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'4134","last_scrub_stamp":"2024-04-24T23:28:07.551822+0000","last_deep_scrub":"30'4134","last_deep_scrub_stamp":"2024-04-24T23:28:07.551822+0000","last_clean_scrub_stamp":"2024-04-24T23:28:07.551822+0000","objects_scrubbed":0,"log_size":4134,"log_dups_size":0,"ondisk_log_size":4134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:44:02.913672+0000","scrub_duration":82,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":71,"num_read_kb":0,"num_write":60152,"num_write_kb":843409,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1,3],"acting":[5,4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"30'12","reported_seq":58,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:28:03.130420+0000","last_change":"2024-04-24T23:28:03.130460+0000","last_active":"2024-04-24T23:28:03.130420+0000","last_peered":"2024-04-24T23:28:03.130420+0000","last_clean":"2024-04-24T23:28:03.130420+0000","last_became_active":"2024-04-24T23:15:37.002583+0000","last_became_peered":"2024-04-24T23:15:37.002583+0000","last_unstale":"2024-04-24T23:28:03.130420+0000","last_undegraded":"2024-04-24T23:28:03.130420+0000","last_fullsized":"2024-04-24T23:28:03.130420+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'12","last_scrub_stamp":"2024-04-24T23:28:03.130281+0000","last_deep_scrub":"30'12","last_deep_scrub_stamp":"2024-04-24T23:28:03.130281+0000","last_clean_scrub_stamp":"2024-04-24T23:28:03.130281+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:38:07.799732+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"30'171","reported_seq":227,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:52.160463+0000","last_change":"2024-04-24T23:27:52.160500+0000","last_active":"2024-04-24T23:27:52.160463+0000","last_peered":"2024-04-24T23:27:52.160463+0000","last_clean":"2024-04-24T23:27:52.160463+0000","last_became_active":"2024-04-24T23:15:35.948438+0000","last_became_peered":"2024-04-24T23:15:35.948438+0000","last_unstale":"2024-04-24T23:27:52.160463+0000","last_undegraded":"2024-04-24T23:27:52.160463+0000","last_fullsized":"2024-04-24T23:27:52.160463+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'171","last_scrub_stamp":"2024-04-24T23:27:52.160374+0000","last_deep_scrub":"30'171","last_deep_scrub_stamp":"2024-04-24T23:27:52.160374+0000","last_clean_scrub_stamp":"2024-04-24T23:27:52.160374+0000","objects_scrubbed":3,"log_size":171,"log_dups_size":0,"ondisk_log_size":171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:54:22.224578+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":21,"num_read_kb":2419,"num_write":164,"num_write_kb":4172,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.8","version":"30'4142","reported_seq":4250,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:28:09.019589+0000","last_change":"2024-04-24T23:28:09.019624+0000","last_active":"2024-04-24T23:28:09.019589+0000","last_peered":"2024-04-24T23:28:09.019589+0000","last_clean":"2024-04-24T23:28:09.019589+0000","last_became_active":"2024-04-24T23:15:41.017092+0000","last_became_peered":"2024-04-24T23:15:41.017092+0000","last_unstale":"2024-04-24T23:28:09.019589+0000","last_undegraded":"2024-04-24T23:28:09.019589+0000","last_fullsized":"2024-04-24T23:28:09.019589+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'4142","last_scrub_stamp":"2024-04-24T23:28:09.019481+0000","last_deep_scrub":"30'4142","last_deep_scrub_stamp":"2024-04-24T23:28:09.019481+0000","last_clean_scrub_stamp":"2024-04-24T23:28:09.019481+0000","objects_scrubbed":0,"log_size":4142,"log_dups_size":0,"ondisk_log_size":4142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:24:29.735641+0000","scrub_duration":113,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":68,"num_read_kb":0,"num_write":60430,"num_write_kb":856855,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"30'7","reported_seq":53,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:52.933420+0000","last_change":"2024-04-24T23:27:52.933542+0000","last_active":"2024-04-24T23:27:52.933420+0000","last_peered":"2024-04-24T23:27:52.933420+0000","last_clean":"2024-04-24T23:27:52.933420+0000","last_became_active":"2024-04-24T23:15:36.989863+0000","last_became_peered":"2024-04-24T23:15:36.989863+0000","last_unstale":"2024-04-24T23:27:52.933420+0000","last_undegraded":"2024-04-24T23:27:52.933420+0000","last_fullsized":"2024-04-24T23:27:52.933420+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'7","last_scrub_stamp":"2024-04-24T23:27:52.933168+0000","last_deep_scrub":"30'7","last_deep_scrub_stamp":"2024-04-24T23:27:52.933168+0000","last_clean_scrub_stamp":"2024-04-24T23:27:52.933168+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:06:54.040616+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.f","version":"30'11","reported_seq":60,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:54.109447+0000","last_change":"2024-04-24T23:27:54.109515+0000","last_active":"2024-04-24T23:27:54.109447+0000","last_peered":"2024-04-24T23:27:54.109447+0000","last_clean":"2024-04-24T23:27:54.109447+0000","last_became_active":"2024-04-24T23:15:35.968994+0000","last_became_peered":"2024-04-24T23:15:35.968994+0000","last_unstale":"2024-04-24T23:27:54.109447+0000","last_undegraded":"2024-04-24T23:27:54.109447+0000","last_fullsized":"2024-04-24T23:27:54.109447+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'11","last_scrub_stamp":"2024-04-24T23:27:54.109225+0000","last_deep_scrub":"30'11","last_deep_scrub_stamp":"2024-04-24T23:27:54.109225+0000","last_clean_scrub_stamp":"2024-04-24T23:27:54.109225+0000","objects_scrubbed":2,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:28:04.145600+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"30'4168","reported_seq":4281,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:54.181383+0000","last_change":"2024-04-24T23:27:54.181500+0000","last_active":"2024-04-24T23:27:54.181383+0000","last_peered":"2024-04-24T23:27:54.181383+0000","last_clean":"2024-04-24T23:27:54.181383+0000","last_became_active":"2024-04-24T23:15:41.009328+0000","last_became_peered":"2024-04-24T23:15:41.009328+0000","last_unstale":"2024-04-24T23:27:54.181383+0000","last_undegraded":"2024-04-24T23:27:54.181383+0000","last_fullsized":"2024-04-24T23:27:54.181383+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'4168","last_scrub_stamp":"2024-04-24T23:27:54.181078+0000","last_deep_scrub":"30'4168","last_deep_scrub_stamp":"2024-04-24T23:27:54.181078+0000","last_clean_scrub_stamp":"2024-04-24T23:27:54.181078+0000","objects_scrubbed":0,"log_size":4168,"log_dups_size":0,"ondisk_log_size":4168,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:44:30.023840+0000","scrub_duration":142,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":73,"num_read_kb":0,"num_write":60607,"num_write_kb":866216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,2],"acting":[1,5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.11","version":"30'14","reported_seq":60,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:51.311338+0000","last_change":"2024-04-24T23:27:50.974130+0000","last_active":"2024-04-24T23:27:51.311338+0000","last_peered":"2024-04-24T23:27:51.311338+0000","last_clean":"2024-04-24T23:27:51.311338+0000","last_became_active":"2024-04-24T23:15:36.995801+0000","last_became_peered":"2024-04-24T23:15:36.995801+0000","last_unstale":"2024-04-24T23:27:51.311338+0000","last_undegraded":"2024-04-24T23:27:51.311338+0000","last_fullsized":"2024-04-24T23:27:51.311338+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"27'10","last_scrub_stamp":"2024-04-24T23:27:50.974001+0000","last_deep_scrub":"27'10","last_deep_scrub_stamp":"2024-04-24T23:27:50.974001+0000","last_clean_scrub_stamp":"2024-04-24T23:27:50.974001+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:15:32.521795+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"30'30","reported_seq":86,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:51.153587+0000","last_change":"2024-04-24T23:27:51.153587+0000","last_active":"2024-04-24T23:27:51.153587+0000","last_peered":"2024-04-24T23:27:51.153587+0000","last_clean":"2024-04-24T23:27:51.153587+0000","last_became_active":"2024-04-24T23:15:35.943710+0000","last_became_peered":"2024-04-24T23:15:35.943710+0000","last_unstale":"2024-04-24T23:27:51.153587+0000","last_undegraded":"2024-04-24T23:27:51.153587+0000","last_fullsized":"2024-04-24T23:27:51.153587+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'29","last_scrub_stamp":"2024-04-24T23:27:51.153469+0000","last_deep_scrub":"30'29","last_deep_scrub_stamp":"2024-04-24T23:27:51.153469+0000","last_clean_scrub_stamp":"2024-04-24T23:27:51.153469+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:41:24.647571+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"30'4684","reported_seq":4794,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:28:10.629258+0000","last_change":"2024-04-24T23:28:10.629299+0000","last_active":"2024-04-24T23:28:10.629258+0000","last_peered":"2024-04-24T23:28:10.629258+0000","last_clean":"2024-04-24T23:28:10.629258+0000","last_became_active":"2024-04-24T23:15:41.012587+0000","last_became_peered":"2024-04-24T23:15:41.012587+0000","last_unstale":"2024-04-24T23:28:10.629258+0000","last_undegraded":"2024-04-24T23:28:10.629258+0000","last_fullsized":"2024-04-24T23:28:10.629258+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'4684","last_scrub_stamp":"2024-04-24T23:28:10.629157+0000","last_deep_scrub":"30'4684","last_deep_scrub_stamp":"2024-04-24T23:28:10.629157+0000","last_clean_scrub_stamp":"2024-04-24T23:28:10.629157+0000","objects_scrubbed":0,"log_size":4684,"log_dups_size":0,"ondisk_log_size":4684,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:39:50.459623+0000","scrub_duration":104,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":70,"num_read_kb":0,"num_write":67827,"num_write_kb":966739,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1,6],"acting":[5,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"30'14","reported_seq":60,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:51.268761+0000","last_change":"2024-04-24T23:27:49.035576+0000","last_active":"2024-04-24T23:27:51.268761+0000","last_peered":"2024-04-24T23:27:51.268761+0000","last_clean":"2024-04-24T23:27:51.268761+0000","last_became_active":"2024-04-24T23:15:36.979356+0000","last_became_peered":"2024-04-24T23:15:36.979356+0000","last_unstale":"2024-04-24T23:27:51.268761+0000","last_undegraded":"2024-04-24T23:27:51.268761+0000","last_fullsized":"2024-04-24T23:27:51.268761+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'9","last_scrub_stamp":"2024-04-24T23:27:49.035212+0000","last_deep_scrub":"30'9","last_deep_scrub_stamp":"2024-04-24T23:27:49.035212+0000","last_clean_scrub_stamp":"2024-04-24T23:27:49.035212+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:34:14.717681+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.11","version":"30'19","reported_seq":67,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:55.071123+0000","last_change":"2024-04-24T23:27:55.071193+0000","last_active":"2024-04-24T23:27:55.071123+0000","last_peered":"2024-04-24T23:27:55.071123+0000","last_clean":"2024-04-24T23:27:55.071123+0000","last_became_active":"2024-04-24T23:15:35.972655+0000","last_became_peered":"2024-04-24T23:15:35.972655+0000","last_unstale":"2024-04-24T23:27:55.071123+0000","last_undegraded":"2024-04-24T23:27:55.071123+0000","last_fullsized":"2024-04-24T23:27:55.071123+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'19","last_scrub_stamp":"2024-04-24T23:27:55.070951+0000","last_deep_scrub":"30'19","last_deep_scrub_stamp":"2024-04-24T23:27:55.070951+0000","last_clean_scrub_stamp":"2024-04-24T23:27:55.070951+0000","objects_scrubbed":3,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:07:52.691776+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1575,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"30'4523","reported_seq":4633,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:28:08.592568+0000","last_change":"2024-04-24T23:28:08.592593+0000","last_active":"2024-04-24T23:28:08.592568+0000","last_peered":"2024-04-24T23:28:08.592568+0000","last_clean":"2024-04-24T23:28:08.592568+0000","last_became_active":"2024-04-24T23:15:41.003790+0000","last_became_peered":"2024-04-24T23:15:41.003790+0000","last_unstale":"2024-04-24T23:28:08.592568+0000","last_undegraded":"2024-04-24T23:28:08.592568+0000","last_fullsized":"2024-04-24T23:28:08.592568+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'4523","last_scrub_stamp":"2024-04-24T23:28:08.592484+0000","last_deep_scrub":"30'4523","last_deep_scrub_stamp":"2024-04-24T23:28:08.592484+0000","last_clean_scrub_stamp":"2024-04-24T23:28:08.592484+0000","objects_scrubbed":0,"log_size":4523,"log_dups_size":0,"ondisk_log_size":4523,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:05:21.509442+0000","scrub_duration":98,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":70,"num_read_kb":0,"num_write":65477,"num_write_kb":956853,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6,4],"acting":[5,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"30'14","reported_seq":60,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:53.014961+0000","last_change":"2024-04-24T23:27:53.015060+0000","last_active":"2024-04-24T23:27:53.014961+0000","last_peered":"2024-04-24T23:27:53.014961+0000","last_clean":"2024-04-24T23:27:53.014961+0000","last_became_active":"2024-04-24T23:15:36.995923+0000","last_became_peered":"2024-04-24T23:15:36.995923+0000","last_unstale":"2024-04-24T23:27:53.014961+0000","last_undegraded":"2024-04-24T23:27:53.014961+0000","last_fullsized":"2024-04-24T23:27:53.014961+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'14","last_scrub_stamp":"2024-04-24T23:27:53.014791+0000","last_deep_scrub":"30'14","last_deep_scrub_stamp":"2024-04-24T23:27:53.014791+0000","last_clean_scrub_stamp":"2024-04-24T23:27:53.014791+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:13:47.119786+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"30'14","reported_seq":66,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:59.996583+0000","last_change":"2024-04-24T23:27:59.996645+0000","last_active":"2024-04-24T23:27:59.996583+0000","last_peered":"2024-04-24T23:27:59.996583+0000","last_clean":"2024-04-24T23:27:59.996583+0000","last_became_active":"2024-04-24T23:15:35.957088+0000","last_became_peered":"2024-04-24T23:15:35.957088+0000","last_unstale":"2024-04-24T23:27:59.996583+0000","last_undegraded":"2024-04-24T23:27:59.996583+0000","last_fullsized":"2024-04-24T23:27:59.996583+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'14","last_scrub_stamp":"2024-04-24T23:27:59.996438+0000","last_deep_scrub":"30'14","last_deep_scrub_stamp":"2024-04-24T23:27:59.996438+0000","last_clean_scrub_stamp":"2024-04-24T23:27:59.996438+0000","objects_scrubbed":3,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:05:52.966855+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":8,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"30'4678","reported_seq":4789,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:28:03.508373+0000","last_change":"2024-04-24T23:28:03.508396+0000","last_active":"2024-04-24T23:28:03.508373+0000","last_peered":"2024-04-24T23:28:03.508373+0000","last_clean":"2024-04-24T23:28:03.508373+0000","last_became_active":"2024-04-24T23:15:41.007555+0000","last_became_peered":"2024-04-24T23:15:41.007555+0000","last_unstale":"2024-04-24T23:28:03.508373+0000","last_undegraded":"2024-04-24T23:28:03.508373+0000","last_fullsized":"2024-04-24T23:28:03.508373+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'4678","last_scrub_stamp":"2024-04-24T23:28:03.508282+0000","last_deep_scrub":"30'4678","last_deep_scrub_stamp":"2024-04-24T23:28:03.508282+0000","last_clean_scrub_stamp":"2024-04-24T23:28:03.508282+0000","objects_scrubbed":0,"log_size":4678,"log_dups_size":0,"ondisk_log_size":4678,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:26:36.436556+0000","scrub_duration":53,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":71,"num_read_kb":0,"num_write":68048,"num_write_kb":979211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0,7],"acting":[6,4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.12","version":"30'15","reported_seq":61,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:28:01.437112+0000","last_change":"2024-04-24T23:28:01.437135+0000","last_active":"2024-04-24T23:28:01.437112+0000","last_peered":"2024-04-24T23:28:01.437112+0000","last_clean":"2024-04-24T23:28:01.437112+0000","last_became_active":"2024-04-24T23:15:36.991115+0000","last_became_peered":"2024-04-24T23:15:36.991115+0000","last_unstale":"2024-04-24T23:28:01.437112+0000","last_undegraded":"2024-04-24T23:28:01.437112+0000","last_fullsized":"2024-04-24T23:28:01.437112+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'15","last_scrub_stamp":"2024-04-24T23:28:01.437043+0000","last_deep_scrub":"30'15","last_deep_scrub_stamp":"2024-04-24T23:28:01.437043+0000","last_clean_scrub_stamp":"2024-04-24T23:28:01.437043+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:26:54.984451+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"30'53","reported_seq":112,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:54.410603+0000","last_change":"2024-04-24T23:27:54.410662+0000","last_active":"2024-04-24T23:27:54.410603+0000","last_peered":"2024-04-24T23:27:54.410603+0000","last_clean":"2024-04-24T23:27:54.410603+0000","last_became_active":"2024-04-24T23:15:35.963907+0000","last_became_peered":"2024-04-24T23:15:35.963907+0000","last_unstale":"2024-04-24T23:27:54.410603+0000","last_undegraded":"2024-04-24T23:27:54.410603+0000","last_fullsized":"2024-04-24T23:27:54.410603+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'53","last_scrub_stamp":"2024-04-24T23:27:54.410494+0000","last_deep_scrub":"30'53","last_deep_scrub_stamp":"2024-04-24T23:27:54.410494+0000","last_clean_scrub_stamp":"2024-04-24T23:27:54.410494+0000","objects_scrubbed":8,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:11:12.638113+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388676,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":26,"num_read_kb":12314,"num_write":42,"num_write_kb":8216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"30'4268","reported_seq":4381,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:28:10.019842+0000","last_change":"2024-04-24T23:28:10.019908+0000","last_active":"2024-04-24T23:28:10.019842+0000","last_peered":"2024-04-24T23:28:10.019842+0000","last_clean":"2024-04-24T23:28:10.019842+0000","last_became_active":"2024-04-24T23:15:41.019193+0000","last_became_peered":"2024-04-24T23:15:41.019193+0000","last_unstale":"2024-04-24T23:28:10.019842+0000","last_undegraded":"2024-04-24T23:28:10.019842+0000","last_fullsized":"2024-04-24T23:28:10.019842+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'4268","last_scrub_stamp":"2024-04-24T23:28:10.019714+0000","last_deep_scrub":"30'4268","last_deep_scrub_stamp":"2024-04-24T23:28:10.019714+0000","last_clean_scrub_stamp":"2024-04-24T23:28:10.019714+0000","objects_scrubbed":0,"log_size":4268,"log_dups_size":0,"ondisk_log_size":4268,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:30:52.312608+0000","scrub_duration":106,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":73,"num_read_kb":0,"num_write":62324,"num_write_kb":864711,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5,6],"acting":[3,0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"30'11","reported_seq":57,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:47.168936+0000","last_change":"2024-04-24T23:27:47.168936+0000","last_active":"2024-04-24T23:27:47.168936+0000","last_peered":"2024-04-24T23:27:47.168936+0000","last_clean":"2024-04-24T23:27:47.168936+0000","last_became_active":"2024-04-24T23:15:36.986484+0000","last_became_peered":"2024-04-24T23:15:36.986484+0000","last_unstale":"2024-04-24T23:27:47.168936+0000","last_undegraded":"2024-04-24T23:27:47.168936+0000","last_fullsized":"2024-04-24T23:27:47.168936+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'10","last_scrub_stamp":"2024-04-24T23:27:47.168801+0000","last_deep_scrub":"30'10","last_deep_scrub_stamp":"2024-04-24T23:27:47.168801+0000","last_clean_scrub_stamp":"2024-04-24T23:27:47.168801+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:37:32.843942+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.14","version":"30'244","reported_seq":303,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:51.198912+0000","last_change":"2024-04-24T23:27:48.044628+0000","last_active":"2024-04-24T23:27:51.198912+0000","last_peered":"2024-04-24T23:27:51.198912+0000","last_clean":"2024-04-24T23:27:51.198912+0000","last_became_active":"2024-04-24T23:15:35.951923+0000","last_became_peered":"2024-04-24T23:15:35.951923+0000","last_unstale":"2024-04-24T23:27:51.198912+0000","last_undegraded":"2024-04-24T23:27:51.198912+0000","last_fullsized":"2024-04-24T23:27:51.198912+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'241","last_scrub_stamp":"2024-04-24T23:27:48.044467+0000","last_deep_scrub":"30'241","last_deep_scrub_stamp":"2024-04-24T23:27:48.044467+0000","last_clean_scrub_stamp":"2024-04-24T23:27:48.044467+0000","objects_scrubbed":6,"log_size":244,"log_dups_size":0,"ondisk_log_size":244,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:24:37.933540+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194976,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":28,"num_read_kb":4122,"num_write":236,"num_write_kb":4244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"30'4391","reported_seq":4507,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:28:05.422192+0000","last_change":"2024-04-24T23:28:05.422219+0000","last_active":"2024-04-24T23:28:05.422192+0000","last_peered":"2024-04-24T23:28:05.422192+0000","last_clean":"2024-04-24T23:28:05.422192+0000","last_became_active":"2024-04-24T23:15:41.015179+0000","last_became_peered":"2024-04-24T23:15:41.015179+0000","last_unstale":"2024-04-24T23:28:05.422192+0000","last_undegraded":"2024-04-24T23:28:05.422192+0000","last_fullsized":"2024-04-24T23:28:05.422192+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'4391","last_scrub_stamp":"2024-04-24T23:28:05.422097+0000","last_deep_scrub":"30'4391","last_deep_scrub_stamp":"2024-04-24T23:28:05.422097+0000","last_clean_scrub_stamp":"2024-04-24T23:28:05.422097+0000","objects_scrubbed":0,"log_size":4391,"log_dups_size":0,"ondisk_log_size":4391,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:05:26.306604+0000","scrub_duration":98,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":76,"num_read_kb":0,"num_write":63942,"num_write_kb":903699,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5,0],"acting":[4,3,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"30'10","reported_seq":56,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:56.976825+0000","last_change":"2024-04-24T23:27:56.976944+0000","last_active":"2024-04-24T23:27:56.976825+0000","last_peered":"2024-04-24T23:27:56.976825+0000","last_clean":"2024-04-24T23:27:56.976825+0000","last_became_active":"2024-04-24T23:15:36.995470+0000","last_became_peered":"2024-04-24T23:15:36.995470+0000","last_unstale":"2024-04-24T23:27:56.976825+0000","last_undegraded":"2024-04-24T23:27:56.976825+0000","last_fullsized":"2024-04-24T23:27:56.976825+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'10","last_scrub_stamp":"2024-04-24T23:27:56.976629+0000","last_deep_scrub":"30'10","last_deep_scrub_stamp":"2024-04-24T23:27:56.976629+0000","last_clean_scrub_stamp":"2024-04-24T23:27:56.976629+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:35:54.198302+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.15","version":"30'26","reported_seq":77,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:53.384833+0000","last_change":"2024-04-24T23:27:53.384886+0000","last_active":"2024-04-24T23:27:53.384833+0000","last_peered":"2024-04-24T23:27:53.384833+0000","last_clean":"2024-04-24T23:27:53.384833+0000","last_became_active":"2024-04-24T23:15:35.944805+0000","last_became_peered":"2024-04-24T23:15:35.944805+0000","last_unstale":"2024-04-24T23:27:53.384833+0000","last_undegraded":"2024-04-24T23:27:53.384833+0000","last_fullsized":"2024-04-24T23:27:53.384833+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'26","last_scrub_stamp":"2024-04-24T23:27:53.384737+0000","last_deep_scrub":"30'26","last_deep_scrub_stamp":"2024-04-24T23:27:53.384737+0000","last_clean_scrub_stamp":"2024-04-24T23:27:53.384737+0000","objects_scrubbed":4,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:04:05.494191+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":23,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"30'4431","reported_seq":4545,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:58.265245+0000","last_change":"2024-04-24T23:27:58.265294+0000","last_active":"2024-04-24T23:27:58.265245+0000","last_peered":"2024-04-24T23:27:58.265245+0000","last_clean":"2024-04-24T23:27:58.265245+0000","last_became_active":"2024-04-24T23:15:41.023951+0000","last_became_peered":"2024-04-24T23:15:41.023951+0000","last_unstale":"2024-04-24T23:27:58.265245+0000","last_undegraded":"2024-04-24T23:27:58.265245+0000","last_fullsized":"2024-04-24T23:27:58.265245+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'4431","last_scrub_stamp":"2024-04-24T23:27:58.265067+0000","last_deep_scrub":"30'4431","last_deep_scrub_stamp":"2024-04-24T23:27:58.265067+0000","last_clean_scrub_stamp":"2024-04-24T23:27:58.265067+0000","objects_scrubbed":0,"log_size":4431,"log_dups_size":0,"ondisk_log_size":4431,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:35:43.053036+0000","scrub_duration":179,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":74,"num_read_kb":0,"num_write":64500,"num_write_kb":925608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1,7],"acting":[2,3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.17","version":"30'16","reported_seq":62,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:28:04.535295+0000","last_change":"2024-04-24T23:28:04.535334+0000","last_active":"2024-04-24T23:28:04.535295+0000","last_peered":"2024-04-24T23:28:04.535295+0000","last_clean":"2024-04-24T23:28:04.535295+0000","last_became_active":"2024-04-24T23:15:36.992002+0000","last_became_peered":"2024-04-24T23:15:36.992002+0000","last_unstale":"2024-04-24T23:28:04.535295+0000","last_undegraded":"2024-04-24T23:28:04.535295+0000","last_fullsized":"2024-04-24T23:28:04.535295+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'16","last_scrub_stamp":"2024-04-24T23:28:04.535201+0000","last_deep_scrub":"30'16","last_deep_scrub_stamp":"2024-04-24T23:28:04.535201+0000","last_clean_scrub_stamp":"2024-04-24T23:28:04.535201+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:03:31.223318+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"30'9","reported_seq":63,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:52.095666+0000","last_change":"2024-04-24T23:27:52.095748+0000","last_active":"2024-04-24T23:27:52.095666+0000","last_peered":"2024-04-24T23:27:52.095666+0000","last_clean":"2024-04-24T23:27:52.095666+0000","last_became_active":"2024-04-24T23:15:35.966646+0000","last_became_peered":"2024-04-24T23:15:35.966646+0000","last_unstale":"2024-04-24T23:27:52.095666+0000","last_undegraded":"2024-04-24T23:27:52.095666+0000","last_fullsized":"2024-04-24T23:27:52.095666+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'9","last_scrub_stamp":"2024-04-24T23:27:52.095478+0000","last_deep_scrub":"30'9","last_deep_scrub_stamp":"2024-04-24T23:27:52.095478+0000","last_clean_scrub_stamp":"2024-04-24T23:27:52.095478+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:57:04.018915+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":19,"num_read_kb":18,"num_write":7,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"30'4325","reported_seq":4433,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:28:00.569065+0000","last_change":"2024-04-24T23:28:00.569091+0000","last_active":"2024-04-24T23:28:00.569065+0000","last_peered":"2024-04-24T23:28:00.569065+0000","last_clean":"2024-04-24T23:28:00.569065+0000","last_became_active":"2024-04-24T23:15:41.021281+0000","last_became_peered":"2024-04-24T23:15:41.021281+0000","last_unstale":"2024-04-24T23:28:00.569065+0000","last_undegraded":"2024-04-24T23:28:00.569065+0000","last_fullsized":"2024-04-24T23:28:00.569065+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'4325","last_scrub_stamp":"2024-04-24T23:28:00.568980+0000","last_deep_scrub":"30'4325","last_deep_scrub_stamp":"2024-04-24T23:28:00.568980+0000","last_clean_scrub_stamp":"2024-04-24T23:28:00.568980+0000","objects_scrubbed":0,"log_size":4325,"log_dups_size":0,"ondisk_log_size":4325,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:00:39.142058+0000","scrub_duration":109,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":68,"num_read_kb":0,"num_write":62902,"num_write_kb":881151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3,2],"acting":[7,1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"30'15","reported_seq":61,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:53.955159+0000","last_change":"2024-04-24T23:27:53.955219+0000","last_active":"2024-04-24T23:27:53.955159+0000","last_peered":"2024-04-24T23:27:53.955159+0000","last_clean":"2024-04-24T23:27:53.955159+0000","last_became_active":"2024-04-24T23:15:36.990711+0000","last_became_peered":"2024-04-24T23:15:36.990711+0000","last_unstale":"2024-04-24T23:27:53.955159+0000","last_undegraded":"2024-04-24T23:27:53.955159+0000","last_fullsized":"2024-04-24T23:27:53.955159+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'15","last_scrub_stamp":"2024-04-24T23:27:53.954973+0000","last_deep_scrub":"30'15","last_deep_scrub_stamp":"2024-04-24T23:27:53.954973+0000","last_clean_scrub_stamp":"2024-04-24T23:27:53.954973+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:47:09.263378+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.17","version":"30'17","reported_seq":68,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:48.130448+0000","last_change":"2024-04-24T23:27:48.035149+0000","last_active":"2024-04-24T23:27:48.130448+0000","last_peered":"2024-04-24T23:27:48.130448+0000","last_clean":"2024-04-24T23:27:48.130448+0000","last_became_active":"2024-04-24T23:15:35.956870+0000","last_became_peered":"2024-04-24T23:15:35.956870+0000","last_unstale":"2024-04-24T23:27:48.130448+0000","last_undegraded":"2024-04-24T23:27:48.130448+0000","last_fullsized":"2024-04-24T23:27:48.130448+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'16","last_scrub_stamp":"2024-04-24T23:27:48.035025+0000","last_deep_scrub":"30'16","last_deep_scrub_stamp":"2024-04-24T23:27:48.035025+0000","last_clean_scrub_stamp":"2024-04-24T23:27:48.035025+0000","objects_scrubbed":3,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:12:15.713654+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":13,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"30'4408","reported_seq":4512,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:28:04.386341+0000","last_change":"2024-04-24T23:28:04.386365+0000","last_active":"2024-04-24T23:28:04.386341+0000","last_peered":"2024-04-24T23:28:04.386341+0000","last_clean":"2024-04-24T23:28:04.386341+0000","last_became_active":"2024-04-24T23:15:41.005578+0000","last_became_peered":"2024-04-24T23:15:41.005578+0000","last_unstale":"2024-04-24T23:28:04.386341+0000","last_undegraded":"2024-04-24T23:28:04.386341+0000","last_fullsized":"2024-04-24T23:28:04.386341+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'4408","last_scrub_stamp":"2024-04-24T23:28:04.386247+0000","last_deep_scrub":"30'4408","last_deep_scrub_stamp":"2024-04-24T23:28:04.386247+0000","last_clean_scrub_stamp":"2024-04-24T23:28:04.386247+0000","objects_scrubbed":0,"log_size":4408,"log_dups_size":0,"ondisk_log_size":4408,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:53:59.287569+0000","scrub_duration":75,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":64,"num_read_kb":0,"num_write":64117,"num_write_kb":908953,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3,7],"acting":[4,0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1e","version":"30'5180","reported_seq":5304,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:28:06.409524+0000","last_change":"2024-04-24T23:28:06.409569+0000","last_active":"2024-04-24T23:28:06.409524+0000","last_peered":"2024-04-24T23:28:06.409524+0000","last_clean":"2024-04-24T23:28:06.409524+0000","last_became_active":"2024-04-24T23:15:41.021698+0000","last_became_peered":"2024-04-24T23:15:41.021698+0000","last_unstale":"2024-04-24T23:28:06.409524+0000","last_undegraded":"2024-04-24T23:28:06.409524+0000","last_fullsized":"2024-04-24T23:28:06.409524+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'5180","last_scrub_stamp":"2024-04-24T23:28:06.409403+0000","last_deep_scrub":"30'5180","last_deep_scrub_stamp":"2024-04-24T23:28:06.409403+0000","last_clean_scrub_stamp":"2024-04-24T23:28:06.409403+0000","objects_scrubbed":0,"log_size":5180,"log_dups_size":0,"ondisk_log_size":5180,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:44:54.992507+0000","scrub_duration":92,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":83,"num_read_kb":0,"num_write":75888,"num_write_kb":1069592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5,3],"acting":[4,1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"30'9","reported_seq":55,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:51.385170+0000","last_change":"2024-04-24T23:27:50.151067+0000","last_active":"2024-04-24T23:27:51.385170+0000","last_peered":"2024-04-24T23:27:51.385170+0000","last_clean":"2024-04-24T23:27:51.385170+0000","last_became_active":"2024-04-24T23:15:36.979553+0000","last_became_peered":"2024-04-24T23:15:36.979553+0000","last_unstale":"2024-04-24T23:27:51.385170+0000","last_undegraded":"2024-04-24T23:27:51.385170+0000","last_fullsized":"2024-04-24T23:27:51.385170+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"27'3","last_scrub_stamp":"2024-04-24T23:27:50.150961+0000","last_deep_scrub":"27'3","last_deep_scrub_stamp":"2024-04-24T23:27:50.150961+0000","last_clean_scrub_stamp":"2024-04-24T23:27:50.150961+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:45:08.759577+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.18","version":"30'38","reported_seq":88,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:28:04.012919+0000","last_change":"2024-04-24T23:28:04.012988+0000","last_active":"2024-04-24T23:28:04.012919+0000","last_peered":"2024-04-24T23:28:04.012919+0000","last_clean":"2024-04-24T23:28:04.012919+0000","last_became_active":"2024-04-24T23:15:35.964434+0000","last_became_peered":"2024-04-24T23:15:35.964434+0000","last_unstale":"2024-04-24T23:28:04.012919+0000","last_undegraded":"2024-04-24T23:28:04.012919+0000","last_fullsized":"2024-04-24T23:28:04.012919+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'38","last_scrub_stamp":"2024-04-24T23:28:04.012750+0000","last_deep_scrub":"30'38","last_deep_scrub_stamp":"2024-04-24T23:28:04.012750+0000","last_clean_scrub_stamp":"2024-04-24T23:28:04.012750+0000","objects_scrubbed":3,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:12:23.454965+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8389190,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":8192,"num_write":28,"num_write_kb":8217,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"30'4560","reported_seq":4670,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:28:10.959641+0000","last_change":"2024-04-24T23:28:10.959704+0000","last_active":"2024-04-24T23:28:10.959641+0000","last_peered":"2024-04-24T23:28:10.959641+0000","last_clean":"2024-04-24T23:28:10.959641+0000","last_became_active":"2024-04-24T23:15:41.021616+0000","last_became_peered":"2024-04-24T23:15:41.021616+0000","last_unstale":"2024-04-24T23:28:10.959641+0000","last_undegraded":"2024-04-24T23:28:10.959641+0000","last_fullsized":"2024-04-24T23:28:10.959641+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'4560","last_scrub_stamp":"2024-04-24T23:28:10.959520+0000","last_deep_scrub":"30'4560","last_deep_scrub_stamp":"2024-04-24T23:28:10.959520+0000","last_clean_scrub_stamp":"2024-04-24T23:28:10.959520+0000","objects_scrubbed":0,"log_size":4560,"log_dups_size":0,"ondisk_log_size":4560,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:23:22.285417+0000","scrub_duration":93,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":69,"num_read_kb":0,"num_write":66277,"num_write_kb":941784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5,1],"acting":[3,2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"30'19","reported_seq":65,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:59.517437+0000","last_change":"2024-04-24T23:27:59.517465+0000","last_active":"2024-04-24T23:27:59.517437+0000","last_peered":"2024-04-24T23:27:59.517437+0000","last_clean":"2024-04-24T23:27:59.517437+0000","last_became_active":"2024-04-24T23:15:36.966910+0000","last_became_peered":"2024-04-24T23:15:36.966910+0000","last_unstale":"2024-04-24T23:27:59.517437+0000","last_undegraded":"2024-04-24T23:27:59.517437+0000","last_fullsized":"2024-04-24T23:27:59.517437+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'19","last_scrub_stamp":"2024-04-24T23:27:59.517356+0000","last_deep_scrub":"30'19","last_deep_scrub_stamp":"2024-04-24T23:27:59.517356+0000","last_clean_scrub_stamp":"2024-04-24T23:27:59.517356+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:29:42.590508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"30'9","reported_seq":59,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:54.405362+0000","last_change":"2024-04-24T23:27:54.405410+0000","last_active":"2024-04-24T23:27:54.405362+0000","last_peered":"2024-04-24T23:27:54.405362+0000","last_clean":"2024-04-24T23:27:54.405362+0000","last_became_active":"2024-04-24T23:15:35.966939+0000","last_became_peered":"2024-04-24T23:15:35.966939+0000","last_unstale":"2024-04-24T23:27:54.405362+0000","last_undegraded":"2024-04-24T23:27:54.405362+0000","last_fullsized":"2024-04-24T23:27:54.405362+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'9","last_scrub_stamp":"2024-04-24T23:27:54.405278+0000","last_deep_scrub":"30'9","last_deep_scrub_stamp":"2024-04-24T23:27:54.405278+0000","last_clean_scrub_stamp":"2024-04-24T23:27:54.405278+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:08:13.932489+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":14,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"30'25","reported_seq":74,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:57.998293+0000","last_change":"2024-04-24T23:27:57.998293+0000","last_active":"2024-04-24T23:27:57.998293+0000","last_peered":"2024-04-24T23:27:57.998293+0000","last_clean":"2024-04-24T23:27:57.998293+0000","last_became_active":"2024-04-24T23:15:35.970908+0000","last_became_peered":"2024-04-24T23:15:35.970908+0000","last_unstale":"2024-04-24T23:27:57.998293+0000","last_undegraded":"2024-04-24T23:27:57.998293+0000","last_fullsized":"2024-04-24T23:27:57.998293+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'25","last_scrub_stamp":"2024-04-24T23:27:57.998040+0000","last_deep_scrub":"30'25","last_deep_scrub_stamp":"2024-04-24T23:27:57.998040+0000","last_clean_scrub_stamp":"2024-04-24T23:27:57.998040+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:50:58.232284+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":4096,"num_write":18,"num_write_kb":4106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"30'11","reported_seq":57,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:28:01.269124+0000","last_change":"2024-04-24T23:28:01.269124+0000","last_active":"2024-04-24T23:28:01.269124+0000","last_peered":"2024-04-24T23:28:01.269124+0000","last_clean":"2024-04-24T23:28:01.269124+0000","last_became_active":"2024-04-24T23:15:36.987896+0000","last_became_peered":"2024-04-24T23:15:36.987896+0000","last_unstale":"2024-04-24T23:28:01.269124+0000","last_undegraded":"2024-04-24T23:28:01.269124+0000","last_fullsized":"2024-04-24T23:28:01.269124+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'11","last_scrub_stamp":"2024-04-24T23:28:01.269005+0000","last_deep_scrub":"30'11","last_deep_scrub_stamp":"2024-04-24T23:28:01.269005+0000","last_clean_scrub_stamp":"2024-04-24T23:28:01.269005+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:00:09.366001+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"30'3643","reported_seq":3743,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:28:02.313723+0000","last_change":"2024-04-24T23:28:02.313723+0000","last_active":"2024-04-24T23:28:02.313723+0000","last_peered":"2024-04-24T23:28:02.313723+0000","last_clean":"2024-04-24T23:28:02.313723+0000","last_became_active":"2024-04-24T23:15:40.997102+0000","last_became_peered":"2024-04-24T23:15:40.997102+0000","last_unstale":"2024-04-24T23:28:02.313723+0000","last_undegraded":"2024-04-24T23:28:02.313723+0000","last_fullsized":"2024-04-24T23:28:02.313723+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3643","last_scrub_stamp":"2024-04-24T23:28:02.313604+0000","last_deep_scrub":"30'3643","last_deep_scrub_stamp":"2024-04-24T23:28:02.313604+0000","last_clean_scrub_stamp":"2024-04-24T23:28:02.313604+0000","objects_scrubbed":0,"log_size":3643,"log_dups_size":0,"ondisk_log_size":3643,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:34:58.163112+0000","scrub_duration":54,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":60,"num_read_kb":0,"num_write":53469,"num_write_kb":749552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,7],"acting":[4,6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"30'1220","reported_seq":1272,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:55.541225+0000","last_change":"2024-04-24T23:27:55.541225+0000","last_active":"2024-04-24T23:27:55.541225+0000","last_peered":"2024-04-24T23:27:55.541225+0000","last_clean":"2024-04-24T23:27:55.541225+0000","last_became_active":"2024-04-24T23:15:35.944563+0000","last_became_peered":"2024-04-24T23:15:35.944563+0000","last_unstale":"2024-04-24T23:27:55.541225+0000","last_undegraded":"2024-04-24T23:27:55.541225+0000","last_fullsized":"2024-04-24T23:27:55.541225+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'1220","last_scrub_stamp":"2024-04-24T23:27:55.541114+0000","last_deep_scrub":"30'1220","last_deep_scrub_stamp":"2024-04-24T23:27:55.541114+0000","last_clean_scrub_stamp":"2024-04-24T23:27:55.541114+0000","objects_scrubbed":3,"log_size":1220,"log_dups_size":0,"ondisk_log_size":1220,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:06:45.381398+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4195441,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":4,"num_read_kb":4098,"num_write":1212,"num_write_kb":5208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"30'13","reported_seq":59,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:51.339661+0000","last_change":"2024-04-24T23:27:50.019919+0000","last_active":"2024-04-24T23:27:51.339661+0000","last_peered":"2024-04-24T23:27:51.339661+0000","last_clean":"2024-04-24T23:27:51.339661+0000","last_became_active":"2024-04-24T23:15:36.979637+0000","last_became_peered":"2024-04-24T23:15:36.979637+0000","last_unstale":"2024-04-24T23:27:51.339661+0000","last_undegraded":"2024-04-24T23:27:51.339661+0000","last_fullsized":"2024-04-24T23:27:51.339661+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'10","last_scrub_stamp":"2024-04-24T23:27:50.019778+0000","last_deep_scrub":"30'10","last_deep_scrub_stamp":"2024-04-24T23:27:50.019778+0000","last_clean_scrub_stamp":"2024-04-24T23:27:50.019778+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:28:39.922324+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"30'5076","reported_seq":5193,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:58.570095+0000","last_change":"2024-04-24T23:27:58.570095+0000","last_active":"2024-04-24T23:27:58.570095+0000","last_peered":"2024-04-24T23:27:58.570095+0000","last_clean":"2024-04-24T23:27:58.570095+0000","last_became_active":"2024-04-24T23:15:41.016696+0000","last_became_peered":"2024-04-24T23:15:41.016696+0000","last_unstale":"2024-04-24T23:27:58.570095+0000","last_undegraded":"2024-04-24T23:27:58.570095+0000","last_fullsized":"2024-04-24T23:27:58.570095+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'5076","last_scrub_stamp":"2024-04-24T23:27:58.569923+0000","last_deep_scrub":"30'5076","last_deep_scrub_stamp":"2024-04-24T23:27:58.569923+0000","last_clean_scrub_stamp":"2024-04-24T23:27:58.569923+0000","objects_scrubbed":0,"log_size":5076,"log_dups_size":0,"ondisk_log_size":5076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:56:31.012399+0000","scrub_duration":95,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":77,"num_read_kb":0,"num_write":74056,"num_write_kb":1014885,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1,0],"acting":[7,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"30'15","reported_seq":62,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:51.375476+0000","last_change":"2024-04-24T23:27:49.024754+0000","last_active":"2024-04-24T23:27:51.375476+0000","last_peered":"2024-04-24T23:27:51.375476+0000","last_clean":"2024-04-24T23:27:51.375476+0000","last_became_active":"2024-04-24T23:15:36.969090+0000","last_became_peered":"2024-04-24T23:15:36.969090+0000","last_unstale":"2024-04-24T23:27:51.375476+0000","last_undegraded":"2024-04-24T23:27:51.375476+0000","last_fullsized":"2024-04-24T23:27:51.375476+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'13","last_scrub_stamp":"2024-04-24T23:27:49.024601+0000","last_deep_scrub":"30'13","last_deep_scrub_stamp":"2024-04-24T23:27:49.024601+0000","last_clean_scrub_stamp":"2024-04-24T23:27:49.024601+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:10:37.734772+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.21","version":"30'13","reported_seq":60,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:56.449870+0000","last_change":"2024-04-24T23:27:56.449870+0000","last_active":"2024-04-24T23:27:56.449870+0000","last_peered":"2024-04-24T23:27:56.449870+0000","last_clean":"2024-04-24T23:27:56.449870+0000","last_became_active":"2024-04-24T23:15:36.982299+0000","last_became_peered":"2024-04-24T23:15:36.982299+0000","last_unstale":"2024-04-24T23:27:56.449870+0000","last_undegraded":"2024-04-24T23:27:56.449870+0000","last_fullsized":"2024-04-24T23:27:56.449870+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'13","last_scrub_stamp":"2024-04-24T23:27:56.449768+0000","last_deep_scrub":"30'13","last_deep_scrub_stamp":"2024-04-24T23:27:56.449768+0000","last_clean_scrub_stamp":"2024-04-24T23:27:56.449768+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:46:11.098469+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.22","version":"30'15","reported_seq":62,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:51.968530+0000","last_change":"2024-04-24T23:27:51.968530+0000","last_active":"2024-04-24T23:27:51.968530+0000","last_peered":"2024-04-24T23:27:51.968530+0000","last_clean":"2024-04-24T23:27:51.968530+0000","last_became_active":"2024-04-24T23:15:36.989557+0000","last_became_peered":"2024-04-24T23:15:36.989557+0000","last_unstale":"2024-04-24T23:27:51.968530+0000","last_undegraded":"2024-04-24T23:27:51.968530+0000","last_fullsized":"2024-04-24T23:27:51.968530+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'15","last_scrub_stamp":"2024-04-24T23:27:51.968386+0000","last_deep_scrub":"30'15","last_deep_scrub_stamp":"2024-04-24T23:27:51.968386+0000","last_clean_scrub_stamp":"2024-04-24T23:27:51.968386+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:48:35.495811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"30'11","reported_seq":58,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:28:01.563693+0000","last_change":"2024-04-24T23:28:01.563693+0000","last_active":"2024-04-24T23:28:01.563693+0000","last_peered":"2024-04-24T23:28:01.563693+0000","last_clean":"2024-04-24T23:28:01.563693+0000","last_became_active":"2024-04-24T23:15:36.969327+0000","last_became_peered":"2024-04-24T23:15:36.969327+0000","last_unstale":"2024-04-24T23:28:01.563693+0000","last_undegraded":"2024-04-24T23:28:01.563693+0000","last_fullsized":"2024-04-24T23:28:01.563693+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'11","last_scrub_stamp":"2024-04-24T23:28:01.563598+0000","last_deep_scrub":"30'11","last_deep_scrub_stamp":"2024-04-24T23:28:01.563598+0000","last_clean_scrub_stamp":"2024-04-24T23:28:01.563598+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:35:28.681060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.24","version":"30'10","reported_seq":56,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:59.420563+0000","last_change":"2024-04-24T23:27:59.420603+0000","last_active":"2024-04-24T23:27:59.420563+0000","last_peered":"2024-04-24T23:27:59.420563+0000","last_clean":"2024-04-24T23:27:59.420563+0000","last_became_active":"2024-04-24T23:15:36.986026+0000","last_became_peered":"2024-04-24T23:15:36.986026+0000","last_unstale":"2024-04-24T23:27:59.420563+0000","last_undegraded":"2024-04-24T23:27:59.420563+0000","last_fullsized":"2024-04-24T23:27:59.420563+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'10","last_scrub_stamp":"2024-04-24T23:27:59.420442+0000","last_deep_scrub":"30'10","last_deep_scrub_stamp":"2024-04-24T23:27:59.420442+0000","last_clean_scrub_stamp":"2024-04-24T23:27:59.420442+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:12:37.084902+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"30'9","reported_seq":55,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:57.359077+0000","last_change":"2024-04-24T23:27:57.359102+0000","last_active":"2024-04-24T23:27:57.359077+0000","last_peered":"2024-04-24T23:27:57.359077+0000","last_clean":"2024-04-24T23:27:57.359077+0000","last_became_active":"2024-04-24T23:15:36.968263+0000","last_became_peered":"2024-04-24T23:15:36.968263+0000","last_unstale":"2024-04-24T23:27:57.359077+0000","last_undegraded":"2024-04-24T23:27:57.359077+0000","last_fullsized":"2024-04-24T23:27:57.359077+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'9","last_scrub_stamp":"2024-04-24T23:27:57.358983+0000","last_deep_scrub":"30'9","last_deep_scrub_stamp":"2024-04-24T23:27:57.358983+0000","last_clean_scrub_stamp":"2024-04-24T23:27:57.358983+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:27:31.870033+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.26","version":"30'9","reported_seq":55,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:55.001407+0000","last_change":"2024-04-24T23:27:55.001471+0000","last_active":"2024-04-24T23:27:55.001407+0000","last_peered":"2024-04-24T23:27:55.001407+0000","last_clean":"2024-04-24T23:27:55.001407+0000","last_became_active":"2024-04-24T23:15:36.990764+0000","last_became_peered":"2024-04-24T23:15:36.990764+0000","last_unstale":"2024-04-24T23:27:55.001407+0000","last_undegraded":"2024-04-24T23:27:55.001407+0000","last_fullsized":"2024-04-24T23:27:55.001407+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'9","last_scrub_stamp":"2024-04-24T23:27:55.001180+0000","last_deep_scrub":"30'9","last_deep_scrub_stamp":"2024-04-24T23:27:55.001180+0000","last_clean_scrub_stamp":"2024-04-24T23:27:55.001180+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:39:17.437466+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"30'15","reported_seq":61,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:28:05.491295+0000","last_change":"2024-04-24T23:28:05.491317+0000","last_active":"2024-04-24T23:28:05.491295+0000","last_peered":"2024-04-24T23:28:05.491295+0000","last_clean":"2024-04-24T23:28:05.491295+0000","last_became_active":"2024-04-24T23:15:36.993310+0000","last_became_peered":"2024-04-24T23:15:36.993310+0000","last_unstale":"2024-04-24T23:28:05.491295+0000","last_undegraded":"2024-04-24T23:28:05.491295+0000","last_fullsized":"2024-04-24T23:28:05.491295+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'15","last_scrub_stamp":"2024-04-24T23:28:05.491221+0000","last_deep_scrub":"30'15","last_deep_scrub_stamp":"2024-04-24T23:28:05.491221+0000","last_clean_scrub_stamp":"2024-04-24T23:28:05.491221+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:47:41.696547+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.28","version":"30'12","reported_seq":58,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:54.039883+0000","last_change":"2024-04-24T23:27:54.039940+0000","last_active":"2024-04-24T23:27:54.039883+0000","last_peered":"2024-04-24T23:27:54.039883+0000","last_clean":"2024-04-24T23:27:54.039883+0000","last_became_active":"2024-04-24T23:15:36.987540+0000","last_became_peered":"2024-04-24T23:15:36.987540+0000","last_unstale":"2024-04-24T23:27:54.039883+0000","last_undegraded":"2024-04-24T23:27:54.039883+0000","last_fullsized":"2024-04-24T23:27:54.039883+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'12","last_scrub_stamp":"2024-04-24T23:27:54.039716+0000","last_deep_scrub":"30'12","last_deep_scrub_stamp":"2024-04-24T23:27:54.039716+0000","last_clean_scrub_stamp":"2024-04-24T23:27:54.039716+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:44:14.768880+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.29","version":"30'15","reported_seq":61,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:52.543383+0000","last_change":"2024-04-24T23:27:52.543408+0000","last_active":"2024-04-24T23:27:52.543383+0000","last_peered":"2024-04-24T23:27:52.543383+0000","last_clean":"2024-04-24T23:27:52.543383+0000","last_became_active":"2024-04-24T23:15:36.980838+0000","last_became_peered":"2024-04-24T23:15:36.980838+0000","last_unstale":"2024-04-24T23:27:52.543383+0000","last_undegraded":"2024-04-24T23:27:52.543383+0000","last_fullsized":"2024-04-24T23:27:52.543383+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'15","last_scrub_stamp":"2024-04-24T23:27:52.543282+0000","last_deep_scrub":"30'15","last_deep_scrub_stamp":"2024-04-24T23:27:52.543282+0000","last_clean_scrub_stamp":"2024-04-24T23:27:52.543282+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:26:38.143381+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2a","version":"30'14","reported_seq":60,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:53.022299+0000","last_change":"2024-04-24T23:27:53.022404+0000","last_active":"2024-04-24T23:27:53.022299+0000","last_peered":"2024-04-24T23:27:53.022299+0000","last_clean":"2024-04-24T23:27:53.022299+0000","last_became_active":"2024-04-24T23:15:36.996245+0000","last_became_peered":"2024-04-24T23:15:36.996245+0000","last_unstale":"2024-04-24T23:27:53.022299+0000","last_undegraded":"2024-04-24T23:27:53.022299+0000","last_fullsized":"2024-04-24T23:27:53.022299+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'14","last_scrub_stamp":"2024-04-24T23:27:53.022132+0000","last_deep_scrub":"30'14","last_deep_scrub_stamp":"2024-04-24T23:27:53.022132+0000","last_clean_scrub_stamp":"2024-04-24T23:27:53.022132+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:23:59.313058+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"30'17","reported_seq":63,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:59.291616+0000","last_change":"2024-04-24T23:27:59.291654+0000","last_active":"2024-04-24T23:27:59.291616+0000","last_peered":"2024-04-24T23:27:59.291616+0000","last_clean":"2024-04-24T23:27:59.291616+0000","last_became_active":"2024-04-24T23:15:36.980911+0000","last_became_peered":"2024-04-24T23:15:36.980911+0000","last_unstale":"2024-04-24T23:27:59.291616+0000","last_undegraded":"2024-04-24T23:27:59.291616+0000","last_fullsized":"2024-04-24T23:27:59.291616+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'17","last_scrub_stamp":"2024-04-24T23:27:59.291508+0000","last_deep_scrub":"30'17","last_deep_scrub_stamp":"2024-04-24T23:27:59.291508+0000","last_clean_scrub_stamp":"2024-04-24T23:27:59.291508+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:23:04.607632+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2c","version":"30'11","reported_seq":57,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:58.144877+0000","last_change":"2024-04-24T23:27:58.145002+0000","last_active":"2024-04-24T23:27:58.144877+0000","last_peered":"2024-04-24T23:27:58.144877+0000","last_clean":"2024-04-24T23:27:58.144877+0000","last_became_active":"2024-04-24T23:15:36.989792+0000","last_became_peered":"2024-04-24T23:15:36.989792+0000","last_unstale":"2024-04-24T23:27:58.144877+0000","last_undegraded":"2024-04-24T23:27:58.144877+0000","last_fullsized":"2024-04-24T23:27:58.144877+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'11","last_scrub_stamp":"2024-04-24T23:27:58.144584+0000","last_deep_scrub":"30'11","last_deep_scrub_stamp":"2024-04-24T23:27:58.144584+0000","last_clean_scrub_stamp":"2024-04-24T23:27:58.144584+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:01:47.513619+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2d","version":"30'15","reported_seq":61,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:55.978639+0000","last_change":"2024-04-24T23:27:55.978759+0000","last_active":"2024-04-24T23:27:55.978639+0000","last_peered":"2024-04-24T23:27:55.978639+0000","last_clean":"2024-04-24T23:27:55.978639+0000","last_became_active":"2024-04-24T23:15:36.987494+0000","last_became_peered":"2024-04-24T23:15:36.987494+0000","last_unstale":"2024-04-24T23:27:55.978639+0000","last_undegraded":"2024-04-24T23:27:55.978639+0000","last_fullsized":"2024-04-24T23:27:55.978639+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'15","last_scrub_stamp":"2024-04-24T23:27:55.978421+0000","last_deep_scrub":"30'15","last_deep_scrub_stamp":"2024-04-24T23:27:55.978421+0000","last_clean_scrub_stamp":"2024-04-24T23:27:55.978421+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:23:51.808207+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2e","version":"30'19","reported_seq":65,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:57.434308+0000","last_change":"2024-04-24T23:27:57.434336+0000","last_active":"2024-04-24T23:27:57.434308+0000","last_peered":"2024-04-24T23:27:57.434308+0000","last_clean":"2024-04-24T23:27:57.434308+0000","last_became_active":"2024-04-24T23:15:36.986272+0000","last_became_peered":"2024-04-24T23:15:36.986272+0000","last_unstale":"2024-04-24T23:27:57.434308+0000","last_undegraded":"2024-04-24T23:27:57.434308+0000","last_fullsized":"2024-04-24T23:27:57.434308+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'19","last_scrub_stamp":"2024-04-24T23:27:57.434230+0000","last_deep_scrub":"30'19","last_deep_scrub_stamp":"2024-04-24T23:27:57.434230+0000","last_clean_scrub_stamp":"2024-04-24T23:27:57.434230+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:20:00.881173+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2f","version":"30'13","reported_seq":64,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:28:13.831272+0000","last_change":"2024-04-24T23:28:13.831329+0000","last_active":"2024-04-24T23:28:13.831272+0000","last_peered":"2024-04-24T23:28:13.831272+0000","last_clean":"2024-04-24T23:28:13.831272+0000","last_became_active":"2024-04-24T23:16:18.293579+0000","last_became_peered":"2024-04-24T23:16:18.293579+0000","last_unstale":"2024-04-24T23:28:13.831272+0000","last_undegraded":"2024-04-24T23:28:13.831272+0000","last_fullsized":"2024-04-24T23:28:13.831272+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'13","last_scrub_stamp":"2024-04-24T23:28:13.831115+0000","last_deep_scrub":"30'13","last_deep_scrub_stamp":"2024-04-24T23:28:13.831115+0000","last_clean_scrub_stamp":"2024-04-24T23:28:13.831115+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:37:40.017007+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.30","version":"30'12","reported_seq":58,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:51.096354+0000","last_change":"2024-04-24T23:27:49.032530+0000","last_active":"2024-04-24T23:27:51.096354+0000","last_peered":"2024-04-24T23:27:51.096354+0000","last_clean":"2024-04-24T23:27:51.096354+0000","last_became_active":"2024-04-24T23:15:36.995507+0000","last_became_peered":"2024-04-24T23:15:36.995507+0000","last_unstale":"2024-04-24T23:27:51.096354+0000","last_undegraded":"2024-04-24T23:27:51.096354+0000","last_fullsized":"2024-04-24T23:27:51.096354+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'10","last_scrub_stamp":"2024-04-24T23:27:49.032001+0000","last_deep_scrub":"30'10","last_deep_scrub_stamp":"2024-04-24T23:27:49.032001+0000","last_clean_scrub_stamp":"2024-04-24T23:27:49.032001+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:45:06.602301+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.31","version":"30'14","reported_seq":60,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:51.494563+0000","last_change":"2024-04-24T23:27:51.494602+0000","last_active":"2024-04-24T23:27:51.494563+0000","last_peered":"2024-04-24T23:27:51.494563+0000","last_clean":"2024-04-24T23:27:51.494563+0000","last_became_active":"2024-04-24T23:15:36.984741+0000","last_became_peered":"2024-04-24T23:15:36.984741+0000","last_unstale":"2024-04-24T23:27:51.494563+0000","last_undegraded":"2024-04-24T23:27:51.494563+0000","last_fullsized":"2024-04-24T23:27:51.494563+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'14","last_scrub_stamp":"2024-04-24T23:27:51.494487+0000","last_deep_scrub":"30'14","last_deep_scrub_stamp":"2024-04-24T23:27:51.494487+0000","last_clean_scrub_stamp":"2024-04-24T23:27:51.494487+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:32:26.711735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.32","version":"30'17","reported_seq":63,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:55.358923+0000","last_change":"2024-04-24T23:27:55.358975+0000","last_active":"2024-04-24T23:27:55.358923+0000","last_peered":"2024-04-24T23:27:55.358923+0000","last_clean":"2024-04-24T23:27:55.358923+0000","last_became_active":"2024-04-24T23:15:36.965877+0000","last_became_peered":"2024-04-24T23:15:36.965877+0000","last_unstale":"2024-04-24T23:27:55.358923+0000","last_undegraded":"2024-04-24T23:27:55.358923+0000","last_fullsized":"2024-04-24T23:27:55.358923+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'17","last_scrub_stamp":"2024-04-24T23:27:55.358801+0000","last_deep_scrub":"30'17","last_deep_scrub_stamp":"2024-04-24T23:27:55.358801+0000","last_clean_scrub_stamp":"2024-04-24T23:27:55.358801+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:22:40.397575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.33","version":"30'15","reported_seq":61,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:28:00.399535+0000","last_change":"2024-04-24T23:28:00.399573+0000","last_active":"2024-04-24T23:28:00.399535+0000","last_peered":"2024-04-24T23:28:00.399535+0000","last_clean":"2024-04-24T23:28:00.399535+0000","last_became_active":"2024-04-24T23:15:36.990383+0000","last_became_peered":"2024-04-24T23:15:36.990383+0000","last_unstale":"2024-04-24T23:28:00.399535+0000","last_undegraded":"2024-04-24T23:28:00.399535+0000","last_fullsized":"2024-04-24T23:28:00.399535+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'15","last_scrub_stamp":"2024-04-24T23:28:00.399434+0000","last_deep_scrub":"30'15","last_deep_scrub_stamp":"2024-04-24T23:28:00.399434+0000","last_clean_scrub_stamp":"2024-04-24T23:28:00.399434+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:06:35.983668+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"30'10","reported_seq":56,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:59.156655+0000","last_change":"2024-04-24T23:27:59.156766+0000","last_active":"2024-04-24T23:27:59.156655+0000","last_peered":"2024-04-24T23:27:59.156655+0000","last_clean":"2024-04-24T23:27:59.156655+0000","last_became_active":"2024-04-24T23:15:36.990155+0000","last_became_peered":"2024-04-24T23:15:36.990155+0000","last_unstale":"2024-04-24T23:27:59.156655+0000","last_undegraded":"2024-04-24T23:27:59.156655+0000","last_fullsized":"2024-04-24T23:27:59.156655+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'10","last_scrub_stamp":"2024-04-24T23:27:59.156441+0000","last_deep_scrub":"30'10","last_deep_scrub_stamp":"2024-04-24T23:27:59.156441+0000","last_clean_scrub_stamp":"2024-04-24T23:27:59.156441+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:42:30.613654+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.35","version":"30'12","reported_seq":58,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:28:02.113006+0000","last_change":"2024-04-24T23:28:02.113043+0000","last_active":"2024-04-24T23:28:02.113006+0000","last_peered":"2024-04-24T23:28:02.113006+0000","last_clean":"2024-04-24T23:28:02.113006+0000","last_became_active":"2024-04-24T23:15:36.992608+0000","last_became_peered":"2024-04-24T23:15:36.992608+0000","last_unstale":"2024-04-24T23:28:02.113006+0000","last_undegraded":"2024-04-24T23:28:02.113006+0000","last_fullsized":"2024-04-24T23:28:02.113006+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'12","last_scrub_stamp":"2024-04-24T23:28:02.112907+0000","last_deep_scrub":"30'12","last_deep_scrub_stamp":"2024-04-24T23:28:02.112907+0000","last_clean_scrub_stamp":"2024-04-24T23:28:02.112907+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:18:42.489669+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.36","version":"30'13","reported_seq":59,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:28:01.112289+0000","last_change":"2024-04-24T23:28:01.112380+0000","last_active":"2024-04-24T23:28:01.112289+0000","last_peered":"2024-04-24T23:28:01.112289+0000","last_clean":"2024-04-24T23:28:01.112289+0000","last_became_active":"2024-04-24T23:15:36.992567+0000","last_became_peered":"2024-04-24T23:15:36.992567+0000","last_unstale":"2024-04-24T23:28:01.112289+0000","last_undegraded":"2024-04-24T23:28:01.112289+0000","last_fullsized":"2024-04-24T23:28:01.112289+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'13","last_scrub_stamp":"2024-04-24T23:28:01.112111+0000","last_deep_scrub":"30'13","last_deep_scrub_stamp":"2024-04-24T23:28:01.112111+0000","last_clean_scrub_stamp":"2024-04-24T23:28:01.112111+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:01:38.544642+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.37","version":"30'16","reported_seq":62,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:52.041925+0000","last_change":"2024-04-24T23:27:52.041925+0000","last_active":"2024-04-24T23:27:52.041925+0000","last_peered":"2024-04-24T23:27:52.041925+0000","last_clean":"2024-04-24T23:27:52.041925+0000","last_became_active":"2024-04-24T23:15:36.999413+0000","last_became_peered":"2024-04-24T23:15:36.999413+0000","last_unstale":"2024-04-24T23:27:52.041925+0000","last_undegraded":"2024-04-24T23:27:52.041925+0000","last_fullsized":"2024-04-24T23:27:52.041925+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'16","last_scrub_stamp":"2024-04-24T23:27:52.041753+0000","last_deep_scrub":"30'16","last_deep_scrub_stamp":"2024-04-24T23:27:52.041753+0000","last_clean_scrub_stamp":"2024-04-24T23:27:52.041753+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:44:52.887221+0000","scrub_duration":66,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"30'5","reported_seq":51,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:53.489217+0000","last_change":"2024-04-24T23:27:53.489217+0000","last_active":"2024-04-24T23:27:53.489217+0000","last_peered":"2024-04-24T23:27:53.489217+0000","last_clean":"2024-04-24T23:27:53.489217+0000","last_became_active":"2024-04-24T23:15:36.986013+0000","last_became_peered":"2024-04-24T23:15:36.986013+0000","last_unstale":"2024-04-24T23:27:53.489217+0000","last_undegraded":"2024-04-24T23:27:53.489217+0000","last_fullsized":"2024-04-24T23:27:53.489217+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'5","last_scrub_stamp":"2024-04-24T23:27:53.489094+0000","last_deep_scrub":"30'5","last_deep_scrub_stamp":"2024-04-24T23:27:53.489094+0000","last_clean_scrub_stamp":"2024-04-24T23:27:53.489094+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:33:58.795317+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"30'9","reported_seq":55,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:57.134449+0000","last_change":"2024-04-24T23:27:57.134449+0000","last_active":"2024-04-24T23:27:57.134449+0000","last_peered":"2024-04-24T23:27:57.134449+0000","last_clean":"2024-04-24T23:27:57.134449+0000","last_became_active":"2024-04-24T23:15:36.983966+0000","last_became_peered":"2024-04-24T23:15:36.983966+0000","last_unstale":"2024-04-24T23:27:57.134449+0000","last_undegraded":"2024-04-24T23:27:57.134449+0000","last_fullsized":"2024-04-24T23:27:57.134449+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'9","last_scrub_stamp":"2024-04-24T23:27:57.133962+0000","last_deep_scrub":"30'9","last_deep_scrub_stamp":"2024-04-24T23:27:57.133962+0000","last_clean_scrub_stamp":"2024-04-24T23:27:57.133962+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:21:53.198234+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3a","version":"30'10","reported_seq":56,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:51.978816+0000","last_change":"2024-04-24T23:27:51.978816+0000","last_active":"2024-04-24T23:27:51.978816+0000","last_peered":"2024-04-24T23:27:51.978816+0000","last_clean":"2024-04-24T23:27:51.978816+0000","last_became_active":"2024-04-24T23:15:36.978706+0000","last_became_peered":"2024-04-24T23:15:36.978706+0000","last_unstale":"2024-04-24T23:27:51.978816+0000","last_undegraded":"2024-04-24T23:27:51.978816+0000","last_fullsized":"2024-04-24T23:27:51.978816+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'10","last_scrub_stamp":"2024-04-24T23:27:51.978624+0000","last_deep_scrub":"30'10","last_deep_scrub_stamp":"2024-04-24T23:27:51.978624+0000","last_clean_scrub_stamp":"2024-04-24T23:27:51.978624+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:36:36.555148+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"30'15","reported_seq":61,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:51.450316+0000","last_change":"2024-04-24T23:27:49.134562+0000","last_active":"2024-04-24T23:27:51.450316+0000","last_peered":"2024-04-24T23:27:51.450316+0000","last_clean":"2024-04-24T23:27:51.450316+0000","last_became_active":"2024-04-24T23:15:36.985552+0000","last_became_peered":"2024-04-24T23:15:36.985552+0000","last_unstale":"2024-04-24T23:27:51.450316+0000","last_undegraded":"2024-04-24T23:27:51.450316+0000","last_fullsized":"2024-04-24T23:27:51.450316+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'7","last_scrub_stamp":"2024-04-24T23:27:49.134436+0000","last_deep_scrub":"30'7","last_deep_scrub_stamp":"2024-04-24T23:27:49.134436+0000","last_clean_scrub_stamp":"2024-04-24T23:27:49.134436+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:18:17.906068+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"30'7","reported_seq":54,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:27:51.035733+0000","last_change":"2024-04-24T23:27:51.035772+0000","last_active":"2024-04-24T23:27:51.035733+0000","last_peered":"2024-04-24T23:27:51.035733+0000","last_clean":"2024-04-24T23:27:51.035733+0000","last_became_active":"2024-04-24T23:15:36.972744+0000","last_became_peered":"2024-04-24T23:15:36.972744+0000","last_unstale":"2024-04-24T23:27:51.035733+0000","last_undegraded":"2024-04-24T23:27:51.035733+0000","last_fullsized":"2024-04-24T23:27:51.035733+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'7","last_scrub_stamp":"2024-04-24T23:27:51.035634+0000","last_deep_scrub":"30'7","last_deep_scrub_stamp":"2024-04-24T23:27:51.035634+0000","last_clean_scrub_stamp":"2024-04-24T23:27:51.035634+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:47:30.700734+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3d","version":"30'12","reported_seq":59,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:28:02.523052+0000","last_change":"2024-04-24T23:28:02.523052+0000","last_active":"2024-04-24T23:28:02.523052+0000","last_peered":"2024-04-24T23:28:02.523052+0000","last_clean":"2024-04-24T23:28:02.523052+0000","last_became_active":"2024-04-24T23:15:36.969751+0000","last_became_peered":"2024-04-24T23:15:36.969751+0000","last_unstale":"2024-04-24T23:28:02.523052+0000","last_undegraded":"2024-04-24T23:28:02.523052+0000","last_fullsized":"2024-04-24T23:28:02.523052+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'12","last_scrub_stamp":"2024-04-24T23:28:02.522909+0000","last_deep_scrub":"30'12","last_deep_scrub_stamp":"2024-04-24T23:28:02.522909+0000","last_clean_scrub_stamp":"2024-04-24T23:28:02.522909+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:50:33.757457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3e","version":"30'6","reported_seq":53,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:28:00.515256+0000","last_change":"2024-04-24T23:28:00.515256+0000","last_active":"2024-04-24T23:28:00.515256+0000","last_peered":"2024-04-24T23:28:00.515256+0000","last_clean":"2024-04-24T23:28:00.515256+0000","last_became_active":"2024-04-24T23:15:36.968543+0000","last_became_peered":"2024-04-24T23:15:36.968543+0000","last_unstale":"2024-04-24T23:28:00.515256+0000","last_undegraded":"2024-04-24T23:28:00.515256+0000","last_fullsized":"2024-04-24T23:28:00.515256+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'6","last_scrub_stamp":"2024-04-24T23:28:00.515165+0000","last_deep_scrub":"30'6","last_deep_scrub_stamp":"2024-04-24T23:28:00.515165+0000","last_clean_scrub_stamp":"2024-04-24T23:28:00.515165+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:16:52.463392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3f","version":"30'8","reported_seq":55,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-24T23:28:00.191194+0000","last_change":"2024-04-24T23:28:00.191194+0000","last_active":"2024-04-24T23:28:00.191194+0000","last_peered":"2024-04-24T23:28:00.191194+0000","last_clean":"2024-04-24T23:28:00.191194+0000","last_became_active":"2024-04-24T23:15:36.992476+0000","last_became_peered":"2024-04-24T23:15:36.992476+0000","last_unstale":"2024-04-24T23:28:00.191194+0000","last_undegraded":"2024-04-24T23:28:00.191194+0000","last_fullsized":"2024-04-24T23:28:00.191194+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'8","last_scrub_stamp":"2024-04-24T23:28:00.191018+0000","last_deep_scrub":"30'8","last_deep_scrub_stamp":"2024-04-24T23:28:00.191018+0000","last_clean_scrub_stamp":"2024-04-24T23:28:00.191018+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:41:02.972561+0000","scrub_duration":81,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2252,"num_read_kb":307,"num_write":2060742,"num_write_kb":29192446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":141598,"ondisk_log_size":141598,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":774,"ondisk_log_size":774,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":80027978,"num_objects":118,"num_object_clones":0,"num_object_copies":236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":118,"num_whiteouts":0,"num_read":353,"num_read_kb":84742,"num_write":4114,"num_write_kb":81391,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25000,"num_omap_keys":50,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":160292864,"data_stored":160055956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4041,"ondisk_log_size":4041,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575009,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1044808,"kb_used_data":12736,"kb_used_omap":1,"kb_used_meta":1032062,"kb_avail":93327032,"statfs":{"total":96636764160,"available":95566880768,"internally_reserved":0,"allocated":13041664,"data_stored":12820416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":1056831947},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834575009,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":962524,"kb_used_data":12628,"kb_used_omap":1,"kb_used_meta":949886,"kb_avail":93409316,"statfs":{"total":96636764160,"available":95651139584,"internally_reserved":0,"allocated":12931072,"data_stored":12729664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":972683718},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834575009,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":844416,"kb_used_data":25592,"kb_used_omap":1,"kb_used_meta":818814,"kb_avail":93527424,"statfs":{"total":96636764160,"available":95772082176,"internally_reserved":0,"allocated":26206208,"data_stored":25993983,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":838465989},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834575010,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":813500,"kb_used_data":12660,"kb_used_omap":1,"kb_used_meta":800830,"kb_avail":93558340,"statfs":{"total":96636764160,"available":95803740160,"internally_reserved":0,"allocated":12963840,"data_stored":12727704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":820050373},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834575005,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1091764,"kb_used_data":37228,"kb_used_omap":1,"kb_used_meta":1054526,"kb_avail":93280076,"statfs":{"total":96636764160,"available":95518797824,"internally_reserved":0,"allocated":38121472,"data_stored":37895315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":1079835079},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575004,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":758872,"kb_used_data":12624,"kb_used_omap":1,"kb_used_meta":746238,"kb_avail":93612968,"statfs":{"total":96636764160,"available":95859679232,"internally_reserved":0,"allocated":12926976,"data_stored":12725934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":764148165},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575004,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1098308,"kb_used_data":21628,"kb_used_omap":1,"kb_used_meta":1076670,"kb_avail":93273532,"statfs":{"total":96636764160,"available":95512096768,"internally_reserved":0,"allocated":22147072,"data_stored":21937447,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1102510533},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834575008,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":697104,"kb_used_data":25160,"kb_used_omap":1,"kb_used_meta":671934,"kb_avail":93674736,"statfs":{"total":96636764160,"available":95922929664,"internally_reserved":0,"allocated":25763840,"data_stored":25544485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":688060870},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":25436160,"data_stored":25402203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":21225472,"data_stored":21204797,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12599296,"data_stored":12583652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":37793792,"data_stored":37753033,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12636160,"data_stored":12585422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":25284608,"data_stored":25261333,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12603392,"data_stored":12587382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12713984,"data_stored":12678134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T23:28:33.763 DEBUG:teuthology.orchestra.run.smithi006:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-04-24T23:28:33.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:33.986+0000 7febe2800640 1 Processor -- start 2024-04-24T23:28:33.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:33.986+0000 7febe2800640 1 -- start start 2024-04-24T23:28:33.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:33.988+0000 7febe2800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7febdc070550 0x7febdc072a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:28:33.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:33.988+0000 7febe2800640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7febdc072fc0 0x7febdc15e740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:28:33.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:33.988+0000 7febe2800640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7febdc15bab0 con 0x7febdc1550c0 2024-04-24T23:28:33.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:33.988+0000 7febe2800640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7febdc0723b0 con 0x7febdc070550 2024-04-24T23:28:33.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:33.988+0000 7febe2800640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7febdc15e2b0 con 0x7febdc072fc0 2024-04-24T23:28:33.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:33.988+0000 7febdb400640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7febdc1550c0 0x7febdc06fd20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.6:46324/0 (socket says 172.21.15.6:46324) 2024-04-24T23:28:33.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:33.988+0000 7febdb400640 1 -- 172.21.15.6:0/411201404 learned_addr learned my addr 172.21.15.6:0/411201404 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:28:33.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:33.988+0000 7febdaa00640 1 --2- 172.21.15.6:0/411201404 >> v2:172.21.15.6:3300/0 conn(0x7febdc070550 0x7febdc072a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:28:33.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:33.988+0000 7febdbe00640 1 --2- 172.21.15.6:0/411201404 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7febdc072fc0 0x7febdc15e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:28:33.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:33.988+0000 7febda000640 1 -- 172.21.15.6:0/411201404 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1187243074 0 0) 0x7febdc15bab0 con 0x7febdc1550c0 2024-04-24T23:28:33.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:33.988+0000 7febda000640 1 -- 172.21.15.6:0/411201404 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7febc0003520 con 0x7febdc1550c0 2024-04-24T23:28:33.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:33.989+0000 7febdbe00640 1 -- 172.21.15.6:0/411201404 >> v2:172.21.15.6:3300/0 conn(0x7febdc070550 msgr2=0x7febdc072a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:28:33.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:33.989+0000 7febdbe00640 1 --2- 172.21.15.6:0/411201404 >> v2:172.21.15.6:3300/0 conn(0x7febdc070550 0x7febdc072a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:28:33.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:33.989+0000 7febdbe00640 1 -- 172.21.15.6:0/411201404 >> v1:172.21.15.190:6789/0 conn(0x7febdc1550c0 legacy=0x7febdc06fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:28:33.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:33.989+0000 7febdbe00640 1 -- 172.21.15.6:0/411201404 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febdc15af30 con 0x7febdc072fc0 2024-04-24T23:28:33.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:33.990+0000 7febdbe00640 1 --2- 172.21.15.6:0/411201404 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7febdc072fc0 0x7febdc15e740 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7febd4009d90 tx=0x7febd402a660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ea6bff3386d2e1fb server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:28:33.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:33.990+0000 7febda000640 1 -- 172.21.15.6:0/411201404 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febd4007740 con 0x7febdc072fc0 2024-04-24T23:28:33.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:33.990+0000 7febda000640 1 -- 172.21.15.6:0/411201404 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7febd40078e0 con 0x7febdc072fc0 2024-04-24T23:28:33.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:33.990+0000 7febda000640 1 -- 172.21.15.6:0/411201404 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febd4007cb0 con 0x7febdc072fc0 2024-04-24T23:28:33.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:33.991+0000 7febe2800640 1 -- 172.21.15.6:0/411201404 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7febdc072fc0 msgr2=0x7febdc15e740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:28:33.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:33.991+0000 7febe2800640 1 --2- 172.21.15.6:0/411201404 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7febdc072fc0 0x7febdc15e740 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7febd4009d90 tx=0x7febd402a660 comp rx=0 tx=0).stop 2024-04-24T23:28:33.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:33.991+0000 7febe2800640 1 -- 172.21.15.6:0/411201404 shutdown_connections 2024-04-24T23:28:33.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:33.991+0000 7febe2800640 1 --2- 172.21.15.6:0/411201404 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7febdc072fc0 0x7febdc15e740 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:28:33.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:33.991+0000 7febe2800640 1 --2- 172.21.15.6:0/411201404 >> v2:172.21.15.6:3300/0 conn(0x7febdc070550 0x7febdc072a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:28:33.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:33.991+0000 7febe2800640 1 -- 172.21.15.6:0/411201404 >> 172.21.15.6:0/411201404 conn(0x7febdc06c1f0 msgr2=0x7febdc06c600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:28:33.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:33.991+0000 7febe2800640 1 -- 172.21.15.6:0/411201404 shutdown_connections 2024-04-24T23:28:33.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:33.991+0000 7febe2800640 1 -- 172.21.15.6:0/411201404 wait complete. 2024-04-24T23:28:33.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:33.993+0000 7febe2800640 1 Processor -- start 2024-04-24T23:28:33.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:33.993+0000 7febe2800640 1 -- start start 2024-04-24T23:28:33.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:33.994+0000 7febe2800640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7febdc070550 0x7febdc1e35d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:28:33.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:33.994+0000 7febe2800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7febdc072fc0 0x7febdc153a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:28:33.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:33.994+0000 7febe2800640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7febdc1550c0 0x7febdc153f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:28:33.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:33.994+0000 7febe2800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7febdc15b7d0 con 0x7febdc072fc0 2024-04-24T23:28:33.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:33.994+0000 7febe2800640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7febdc15b4f0 con 0x7febdc1550c0 2024-04-24T23:28:33.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:33.994+0000 7febe2800640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7febdc15ab60 con 0x7febdc070550 2024-04-24T23:28:33.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:33.994+0000 7febdb400640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7febdc070550 0x7febdc1e35d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:28:33.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:33.994+0000 7febdb400640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7febdc070550 0x7febdc1e35d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.6:42850/0 (socket says 172.21.15.6:42850) 2024-04-24T23:28:33.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:33.994+0000 7febdaa00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7febdc072fc0 0x7febdc153a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:28:33.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:33.994+0000 7febdb400640 1 -- 172.21.15.6:0/784366528 learned_addr learned my addr 172.21.15.6:0/784366528 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:28:33.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:33.994+0000 7febdbe00640 1 --2- 172.21.15.6:0/784366528 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7febdc1550c0 0x7febdc153f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:28:33.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:33.994+0000 7febdb400640 1 -- 172.21.15.6:0/784366528 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7febdc1550c0 msgr2=0x7febdc153f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:28:33.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:33.994+0000 7febdb400640 1 --2- 172.21.15.6:0/784366528 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7febdc1550c0 0x7febdc153f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:28:33.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:33.994+0000 7febdb400640 1 -- 172.21.15.6:0/784366528 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7febdc072fc0 msgr2=0x7febdc153a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:28:33.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:33.994+0000 7febdb400640 1 --2- 172.21.15.6:0/784366528 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7febdc072fc0 0x7febdc153a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:28:33.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:33.994+0000 7febdb400640 1 -- 172.21.15.6:0/784366528 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febdc1f4a60 con 0x7febdc070550 2024-04-24T23:28:33.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:33.995+0000 7febdb400640 1 --2- 172.21.15.6:0/784366528 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7febdc070550 0x7febdc1e35d0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7febcc00c530 tx=0x7febcc00c9f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:28:33.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:33.995+0000 7febd3400640 1 -- 172.21.15.6:0/784366528 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febcc017070 con 0x7febdc070550 2024-04-24T23:28:33.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:33.995+0000 7febd3400640 1 -- 172.21.15.6:0/784366528 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7febcc005ce0 con 0x7febdc070550 2024-04-24T23:28:33.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:33.995+0000 7febd3400640 1 -- 172.21.15.6:0/784366528 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febcc0072b0 con 0x7febdc070550 2024-04-24T23:28:33.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:33.995+0000 7febe2800640 1 -- 172.21.15.6:0/784366528 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7febdc14f400 con 0x7febdc070550 2024-04-24T23:28:33.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:33.995+0000 7febe2800640 1 -- 172.21.15.6:0/784366528 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7febdc14f9c0 con 0x7febdc070550 2024-04-24T23:28:33.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:33.996+0000 7febe2800640 1 -- 172.21.15.6:0/784366528 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7febdc070b10 con 0x7febdc070550 2024-04-24T23:28:33.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:33.997+0000 7febd3400640 1 -- 172.21.15.6:0/784366528 <== mon.2 v2:172.21.15.190:3301/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7febcc01a070 con 0x7febdc070550 2024-04-24T23:28:33.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:33.998+0000 7febd3400640 1 --2- 172.21.15.6:0/784366528 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7febb806be60 0x7febb806e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:28:33.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:33.998+0000 7febd3400640 1 -- 172.21.15.6:0/784366528 <== mon.2 v2:172.21.15.190:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6934+0+0 (secure 0 0 0) 0x7febcc08a6e0 con 0x7febdc070550 2024-04-24T23:28:33.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:33.998+0000 7febdaa00640 1 --2- 172.21.15.6:0/784366528 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7febb806be60 0x7febb806e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:28:33.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:33.999+0000 7febdaa00640 1 --2- 172.21.15.6:0/784366528 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7febb806be60 0x7febb806e320 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7febdc154ad0 tx=0x7febc4009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:28:34.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:34.006+0000 7febd3400640 1 -- 172.21.15.6:0/784366528 <== mon.2 v2:172.21.15.190:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7febcc058590 con 0x7febdc070550 2024-04-24T23:28:34.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:34.203+0000 7febe2800640 1 -- 172.21.15.6:0/784366528 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) -- 0x7febdc14b1d0 con 0x7febdc070550 2024-04-24T23:28:34.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:34.245+0000 7febd3400640 1 -- 172.21.15.6:0/784366528 <== mon.2 v2:172.21.15.190:3301/0 7 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7febcc05c890 con 0x7febdc070550 2024-04-24T23:28:34.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:34.260+0000 7febd3400640 1 -- 172.21.15.6:0/784366528 <== mon.2 v2:172.21.15.190:3301/0 8 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v10)=0 v10) ==== 123+0+0 (secure 0 0 0) 0x7febcc0610c0 con 0x7febdc070550 2024-04-24T23:28:34.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:34.263+0000 7febe2800640 1 -- 172.21.15.6:0/784366528 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7febb806be60 msgr2=0x7febb806e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:28:34.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:34.263+0000 7febe2800640 1 --2- 172.21.15.6:0/784366528 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7febb806be60 0x7febb806e320 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7febdc154ad0 tx=0x7febc4009190 comp rx=0 tx=0).stop 2024-04-24T23:28:34.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:34.263+0000 7febe2800640 1 -- 172.21.15.6:0/784366528 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7febdc070550 msgr2=0x7febdc1e35d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:28:34.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:34.263+0000 7febe2800640 1 --2- 172.21.15.6:0/784366528 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7febdc070550 0x7febdc1e35d0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7febcc00c530 tx=0x7febcc00c9f0 comp rx=0 tx=0).stop 2024-04-24T23:28:34.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:34.264+0000 7febe2800640 1 -- 172.21.15.6:0/784366528 shutdown_connections 2024-04-24T23:28:34.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:34.264+0000 7febe2800640 1 --2- 172.21.15.6:0/784366528 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7febb806be60 0x7febb806e320 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:28:34.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:34.264+0000 7febe2800640 1 --2- 172.21.15.6:0/784366528 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7febdc1550c0 0x7febdc153f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:28:34.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:34.264+0000 7febe2800640 1 --2- 172.21.15.6:0/784366528 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7febdc072fc0 0x7febdc153a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:28:34.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:34.264+0000 7febe2800640 1 --2- 172.21.15.6:0/784366528 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7febdc070550 0x7febdc1e35d0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:28:34.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:34.264+0000 7febe2800640 1 -- 172.21.15.6:0/784366528 >> 172.21.15.6:0/784366528 conn(0x7febdc06c1f0 msgr2=0x7febdc1590c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:28:34.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:34.264+0000 7febe2800640 1 -- 172.21.15.6:0/784366528 shutdown_connections 2024-04-24T23:28:34.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:34.264+0000 7febe2800640 1 -- 172.21.15.6:0/784366528 wait complete. 2024-04-24T23:28:34.282 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-24T23:28:34.282 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:28:34.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:34.479+0000 7f59b6e00640 1 Processor -- start 2024-04-24T23:28:34.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:34.479+0000 7f59b6e00640 1 -- start start 2024-04-24T23:28:34.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:34.481+0000 7f59b6e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f59b00c53d0 0x7f59b00cb0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:28:34.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:34.481+0000 7f59b6e00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f59b00cb5f0 0x7f59b015e020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:28:34.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:34.481+0000 7f59b6e00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59b00c6e20 con 0x7f59b0154c40 2024-04-24T23:28:34.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:34.481+0000 7f59b6e00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f59b015ef20 con 0x7f59b00c53d0 2024-04-24T23:28:34.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:34.481+0000 7f59b6e00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f59b015ec40 con 0x7f59b00cb5f0 2024-04-24T23:28:34.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:34.481+0000 7f59b5a00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f59b0154c40 0x7f59b00c6710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:46644/0 (socket says 172.21.15.6:46644) 2024-04-24T23:28:34.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:34.481+0000 7f59b5a00640 1 -- 172.21.15.6:0/322472381 learned_addr learned my addr 172.21.15.6:0/322472381 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:28:34.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:34.481+0000 7f59b5000640 1 --2- 172.21.15.6:0/322472381 >> v2:172.21.15.6:3300/0 conn(0x7f59b00c53d0 0x7f59b00cb0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:28:34.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:34.481+0000 7f59b6400640 1 --2- 172.21.15.6:0/322472381 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f59b00cb5f0 0x7f59b015e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:28:34.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:34.482+0000 7f59b5000640 1 -- 172.21.15.6:0/322472381 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f59b00cb5f0 msgr2=0x7f59b015e020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:28:34.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:34.482+0000 7f59b5000640 1 --2- 172.21.15.6:0/322472381 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f59b00cb5f0 0x7f59b015e020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:28:34.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:34.482+0000 7f59b5000640 1 -- 172.21.15.6:0/322472381 >> v1:172.21.15.6:6789/0 conn(0x7f59b0154c40 legacy=0x7f59b00c6710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:28:34.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:34.482+0000 7f59b5000640 1 -- 172.21.15.6:0/322472381 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59b00c5950 con 0x7f59b00c53d0 2024-04-24T23:28:34.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:34.483+0000 7f59b5000640 1 --2- 172.21.15.6:0/322472381 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f59b00c53d0 0x7f59b00cb0b0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f599c0097a0 tx=0x7f599c02a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49fff8205ff7ad66 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:28:34.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:34.483+0000 7f59afe00640 1 -- 172.21.15.6:0/322472381 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f599c037070 con 0x7f59b00c53d0 2024-04-24T23:28:34.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:34.483+0000 7f59afe00640 1 -- 172.21.15.6:0/322472381 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f599c032440 con 0x7f59b00c53d0 2024-04-24T23:28:34.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:34.483+0000 7f59afe00640 1 -- 172.21.15.6:0/322472381 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f599c032830 con 0x7f59b00c53d0 2024-04-24T23:28:34.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:34.484+0000 7f59b6e00640 1 -- 172.21.15.6:0/322472381 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f59b00c53d0 msgr2=0x7f59b00cb0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:28:34.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:34.484+0000 7f59b6e00640 1 --2- 172.21.15.6:0/322472381 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f59b00c53d0 0x7f59b00cb0b0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f599c0097a0 tx=0x7f599c02a230 comp rx=0 tx=0).stop 2024-04-24T23:28:34.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:34.484+0000 7f59b6e00640 1 -- 172.21.15.6:0/322472381 shutdown_connections 2024-04-24T23:28:34.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:34.484+0000 7f59b6e00640 1 --2- 172.21.15.6:0/322472381 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f59b00cb5f0 0x7f59b015e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:28:34.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:34.484+0000 7f59b6e00640 1 --2- 172.21.15.6:0/322472381 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f59b00c53d0 0x7f59b00cb0b0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:28:34.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:34.484+0000 7f59b6e00640 1 -- 172.21.15.6:0/322472381 >> 172.21.15.6:0/322472381 conn(0x7f59b006cd70 msgr2=0x7f59b006d870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:28:34.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:34.484+0000 7f59b6e00640 1 -- 172.21.15.6:0/322472381 shutdown_connections 2024-04-24T23:28:34.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:34.484+0000 7f59b6e00640 1 -- 172.21.15.6:0/322472381 wait complete. 2024-04-24T23:28:34.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:34.486+0000 7f59b6e00640 1 Processor -- start 2024-04-24T23:28:34.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:34.486+0000 7f59b6e00640 1 -- start start 2024-04-24T23:28:34.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:34.487+0000 7f59b6e00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f59b00c53d0 0x7f59b01f0c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:28:34.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:34.487+0000 7f59b6e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f59b00cb5f0 0x7f59b015a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:28:34.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:34.487+0000 7f59b6e00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f59b0154c40 0x7f59b015aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:28:34.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:34.487+0000 7f59b6e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f59b00cc470 con 0x7f59b00cb5f0 2024-04-24T23:28:34.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:34.487+0000 7f59b6e00640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f59b00c5f10 con 0x7f59b0154c40 2024-04-24T23:28:34.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:34.487+0000 7f59b6e00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f59b0155020 con 0x7f59b00c53d0 2024-04-24T23:28:34.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:34.487+0000 7f59b5000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f59b00cb5f0 0x7f59b015a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:28:34.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:34.487+0000 7f59b5a00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f59b00c53d0 0x7f59b01f0c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:28:34.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:34.487+0000 7f59b5000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f59b00cb5f0 0x7f59b015a050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:49268/0 (socket says 172.21.15.6:49268) 2024-04-24T23:28:34.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:34.487+0000 7f59b5000640 1 -- 172.21.15.6:0/1645036900 learned_addr learned my addr 172.21.15.6:0/1645036900 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:28:34.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:34.487+0000 7f59b6400640 1 --2- 172.21.15.6:0/1645036900 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f59b0154c40 0x7f59b015aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:28:34.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:34.487+0000 7f59b5000640 1 -- 172.21.15.6:0/1645036900 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f59b00c53d0 msgr2=0x7f59b01f0c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:28:34.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:34.487+0000 7f59b5000640 1 --2- 172.21.15.6:0/1645036900 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f59b00c53d0 0x7f59b01f0c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:28:34.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:34.487+0000 7f59b5000640 1 -- 172.21.15.6:0/1645036900 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f59b0154c40 msgr2=0x7f59b015aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:28:34.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:34.487+0000 7f59b5000640 1 --2- 172.21.15.6:0/1645036900 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f59b0154c40 0x7f59b015aad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:28:34.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:34.487+0000 7f59b5000640 1 -- 172.21.15.6:0/1645036900 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59b0145dc0 con 0x7f59b00cb5f0 2024-04-24T23:28:34.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:34.488+0000 7f59b5000640 1 --2- 172.21.15.6:0/1645036900 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f59b00cb5f0 0x7f59b015a050 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f599c035040 tx=0x7f599c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:28:34.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:34.488+0000 7f59ad600640 1 -- 172.21.15.6:0/1645036900 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f599c041030 con 0x7f59b00cb5f0 2024-04-24T23:28:34.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:34.488+0000 7f59ad600640 1 -- 172.21.15.6:0/1645036900 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f599c030540 con 0x7f59b00cb5f0 2024-04-24T23:28:34.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:34.488+0000 7f59b6e00640 1 -- 172.21.15.6:0/1645036900 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59b01460a0 con 0x7f59b00cb5f0 2024-04-24T23:28:34.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:34.488+0000 7f59ad600640 1 -- 172.21.15.6:0/1645036900 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f599c037040 con 0x7f59b00cb5f0 2024-04-24T23:28:34.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:34.488+0000 7f59b6e00640 1 -- 172.21.15.6:0/1645036900 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59b0146660 con 0x7f59b00cb5f0 2024-04-24T23:28:34.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:34.489+0000 7f59b6e00640 1 -- 172.21.15.6:0/1645036900 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f59b0075b30 con 0x7f59b00cb5f0 2024-04-24T23:28:34.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:34.490+0000 7f59ad600640 1 -- 172.21.15.6:0/1645036900 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f599c044070 con 0x7f59b00cb5f0 2024-04-24T23:28:34.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:34.492+0000 7f59ad600640 1 --2- 172.21.15.6:0/1645036900 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f599006be30 0x7f599006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:28:34.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:34.492+0000 7f59ad600640 1 -- 172.21.15.6:0/1645036900 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6934+0+0 (secure 0 0 0) 0x7f599c0abac0 con 0x7f59b00cb5f0 2024-04-24T23:28:34.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:34.492+0000 7f59b5a00640 1 --2- 172.21.15.6:0/1645036900 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f599006be30 0x7f599006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:28:34.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:34.493+0000 7f59b5a00640 1 --2- 172.21.15.6:0/1645036900 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f599006be30 0x7f599006e2f0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f59a4007fd0 tx=0x7f59a4007e80 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:28:34.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:34.497+0000 7f59ad600640 1 -- 172.21.15.6:0/1645036900 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f599c0798b0 con 0x7f59b00cb5f0 2024-04-24T23:28:34.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:34.732+0000 7f59b6e00640 1 -- 172.21.15.6:0/1645036900 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f59b00cbc60 con 0x7f59b00cb5f0 2024-04-24T23:28:34.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:34.733+0000 7f59ad600640 1 -- 172.21.15.6:0/1645036900 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) ==== 94+0+6861 (secure 0 0 0) 0x7f599c07dbb0 con 0x7f59b00cb5f0 2024-04-24T23:28:34.734 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:28:34.734 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":54,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5081,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/1190759626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1190759626},{"type":"v1","addr":"172.21.15.6:6834","nonce":1190759626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":5249,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6838/2416336040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2416336040},{"type":"v1","addr":"172.21.15.190:6838","nonce":2416336040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":42},{"gid":5503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/3928731059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":3928731059},{"type":"v1","addr":"172.21.15.6:6837","nonce":3928731059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":35},{"gid":5902,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/4245754886","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":4245754886},{"type":"v1","addr":"172.21.15.6:6835","nonce":4245754886}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":54}],"filesystems":[{"mdsmap":{"epoch":53,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:27:18.122830+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5725,"mds_1":5599},"failed":[],"damaged":[],"stopped":[2,3,4],"info":{"gid_5599":{"gid":5599,"name":"d","rank":1,"incarnation":43,"state":"up:active","state_seq":25,"addr":"172.21.15.190:6839/2643233757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":2643233757},{"type":"v1","addr":"172.21.15.190:6839","nonce":2643233757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5725":{"gid":5725,"name":"f","rank":0,"incarnation":49,"state":"up:active","state_seq":36,"addr":"172.21.15.190:6837/1151527362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":1151527362},{"type":"v1","addr":"172.21.15.190:6837","nonce":1151527362}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5725,"qdb_cluster":[5725,5599]},"id":1}]} 2024-04-24T23:28:34.735 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 54 2024-04-24T23:28:34.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:34.735+0000 7f59b6e00640 1 -- 172.21.15.6:0/1645036900 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f599006be30 msgr2=0x7f599006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:28:34.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:34.735+0000 7f59b6e00640 1 --2- 172.21.15.6:0/1645036900 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f599006be30 0x7f599006e2f0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f59a4007fd0 tx=0x7f59a4007e80 comp rx=0 tx=0).stop 2024-04-24T23:28:34.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:34.735+0000 7f59b6e00640 1 -- 172.21.15.6:0/1645036900 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f59b00cb5f0 msgr2=0x7f59b015a050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:28:34.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:34.735+0000 7f59b6e00640 1 --2- 172.21.15.6:0/1645036900 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f59b00cb5f0 0x7f59b015a050 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f599c035040 tx=0x7f599c0046c0 comp rx=0 tx=0).stop 2024-04-24T23:28:34.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:34.735+0000 7f59b6e00640 1 -- 172.21.15.6:0/1645036900 shutdown_connections 2024-04-24T23:28:34.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:34.735+0000 7f59b6e00640 1 --2- 172.21.15.6:0/1645036900 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f599006be30 0x7f599006e2f0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:28:34.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:34.735+0000 7f59b6e00640 1 --2- 172.21.15.6:0/1645036900 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f59b0154c40 0x7f59b015aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:28:34.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:34.735+0000 7f59b6e00640 1 --2- 172.21.15.6:0/1645036900 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f59b00cb5f0 0x7f59b015a050 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:28:34.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:34.735+0000 7f59b6e00640 1 --2- 172.21.15.6:0/1645036900 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f59b00c53d0 0x7f59b01f0c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:28:34.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:34.735+0000 7f59b6e00640 1 -- 172.21.15.6:0/1645036900 >> 172.21.15.6:0/1645036900 conn(0x7f59b006cd70 msgr2=0x7f59b00cd480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:28:34.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:34.736+0000 7f59b6e00640 1 -- 172.21.15.6:0/1645036900 shutdown_connections 2024-04-24T23:28:34.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:34.736+0000 7f59b6e00640 1 -- 172.21.15.6:0/1645036900 wait complete. 2024-04-24T23:28:34.755 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:28:34.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:34.943+0000 7fe5ab000640 1 Processor -- start 2024-04-24T23:28:34.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:34.943+0000 7fe5ab000640 1 -- start start 2024-04-24T23:28:34.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:34.945+0000 7fe5ab000640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fe5a40c9ff0 0x7fe5a40caaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:28:34.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:34.945+0000 7fe5ab000640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe5a40ca4e0 con 0x7fe5a4152b30 2024-04-24T23:28:34.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:34.945+0000 7fe5ab000640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe5a40ca7c0 con 0x7fe5a40c50b0 2024-04-24T23:28:34.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:34.945+0000 7fe5ab000640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7fe5a40c3bd0 con 0x7fe5a40c9ff0 2024-04-24T23:28:34.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:34.945+0000 7fe5a9c00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fe5a4152b30 0x7fe5a406f5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:35852/0 (socket says 172.21.15.6:35852) 2024-04-24T23:28:34.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:34.945+0000 7fe5a9c00640 1 -- 172.21.15.6:0/4213822417 learned_addr learned my addr 172.21.15.6:0/4213822417 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:28:34.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:34.945+0000 7fe5a9200640 1 --2- 172.21.15.6:0/4213822417 >> v2:172.21.15.190:3300/0 conn(0x7fe5a40c9ff0 0x7fe5a40caaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:28:34.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:34.946+0000 7fe5a3e00640 1 -- 172.21.15.6:0/4213822417 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1906612411 0 0) 0x7fe5a40ca7c0 con 0x7fe5a40c50b0 2024-04-24T23:28:34.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:34.946+0000 7fe5a9200640 1 -- 172.21.15.6:0/4213822417 >> v1:172.21.15.190:6789/0 conn(0x7fe5a40c50b0 legacy=0x7fe5a40cafe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:28:34.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:34.946+0000 7fe5a9200640 1 -- 172.21.15.6:0/4213822417 >> v1:172.21.15.6:6789/0 conn(0x7fe5a4152b30 legacy=0x7fe5a406f5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:28:34.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:34.946+0000 7fe5a9200640 1 -- 172.21.15.6:0/4213822417 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5a40c33d0 con 0x7fe5a40c9ff0 2024-04-24T23:28:34.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:34.947+0000 7fe5a9200640 1 --2- 172.21.15.6:0/4213822417 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe5a40c9ff0 0x7fe5a40caaa0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fe5900097a0 tx=0x7fe59002a250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=16030d43ecabf9e3 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:28:34.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:34.947+0000 7fe5a3e00640 1 -- 172.21.15.6:0/4213822417 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe590037070 con 0x7fe5a40c9ff0 2024-04-24T23:28:34.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:34.947+0000 7fe5a3e00640 1 -- 172.21.15.6:0/4213822417 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7fe590032440 con 0x7fe5a40c9ff0 2024-04-24T23:28:34.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:34.947+0000 7fe5a3e00640 1 -- 172.21.15.6:0/4213822417 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe590032830 con 0x7fe5a40c9ff0 2024-04-24T23:28:34.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:34.948+0000 7fe5ab000640 1 -- 172.21.15.6:0/4213822417 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe5a40c9ff0 msgr2=0x7fe5a40caaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:28:34.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:34.948+0000 7fe5ab000640 1 --2- 172.21.15.6:0/4213822417 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe5a40c9ff0 0x7fe5a40caaa0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fe5900097a0 tx=0x7fe59002a250 comp rx=0 tx=0).stop 2024-04-24T23:28:34.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:34.948+0000 7fe5ab000640 1 -- 172.21.15.6:0/4213822417 shutdown_connections 2024-04-24T23:28:34.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:34.948+0000 7fe5ab000640 1 --2- 172.21.15.6:0/4213822417 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe5a40c9ff0 0x7fe5a40caaa0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:28:34.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:34.948+0000 7fe5ab000640 1 -- 172.21.15.6:0/4213822417 >> 172.21.15.6:0/4213822417 conn(0x7fe5a406b500 msgr2=0x7fe5a4066ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:28:34.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:34.948+0000 7fe5ab000640 1 -- 172.21.15.6:0/4213822417 shutdown_connections 2024-04-24T23:28:34.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:34.948+0000 7fe5ab000640 1 -- 172.21.15.6:0/4213822417 wait complete. 2024-04-24T23:28:34.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:34.949+0000 7fe5ab000640 1 Processor -- start 2024-04-24T23:28:34.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:34.950+0000 7fe5ab000640 1 -- start start 2024-04-24T23:28:34.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:34.951+0000 7fe5ab000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe5a40c50b0 0x7fe5a41f4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:28:34.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:34.951+0000 7fe5ab000640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe5a40c9ff0 0x7fe5a40afb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:28:34.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:34.951+0000 7fe5ab000640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe5a4152b30 0x7fe5a40b13c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:28:34.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:34.951+0000 7fe5ab000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fe5a40c90e0 con 0x7fe5a40c50b0 2024-04-24T23:28:34.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:34.951+0000 7fe5ab000640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fe5a40c3740 con 0x7fe5a4152b30 2024-04-24T23:28:34.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:34.951+0000 7fe5ab000640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fe5a40c8d10 con 0x7fe5a40c9ff0 2024-04-24T23:28:34.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:34.951+0000 7fe5aa600640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe5a4152b30 0x7fe5a40b13c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:28:34.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:34.951+0000 7fe5aa600640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe5a4152b30 0x7fe5a40b13c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:46726/0 (socket says 172.21.15.6:46726) 2024-04-24T23:28:34.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:34.951+0000 7fe5aa600640 1 -- 172.21.15.6:0/53893426 learned_addr learned my addr 172.21.15.6:0/53893426 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:28:34.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:34.951+0000 7fe5a9200640 1 --2- 172.21.15.6:0/53893426 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe5a40c9ff0 0x7fe5a40afb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:28:34.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:34.951+0000 7fe5a9c00640 1 --2- 172.21.15.6:0/53893426 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe5a40c50b0 0x7fe5a41f4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:28:34.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:34.951+0000 7fe5aa600640 1 -- 172.21.15.6:0/53893426 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe5a40c9ff0 msgr2=0x7fe5a40afb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:28:34.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:34.951+0000 7fe5aa600640 1 --2- 172.21.15.6:0/53893426 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe5a40c9ff0 0x7fe5a40afb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:28:34.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:34.951+0000 7fe5aa600640 1 -- 172.21.15.6:0/53893426 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe5a40c50b0 msgr2=0x7fe5a41f4010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:28:34.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:34.951+0000 7fe5aa600640 1 --2- 172.21.15.6:0/53893426 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe5a40c50b0 0x7fe5a41f4010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:28:34.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:34.951+0000 7fe5aa600640 1 -- 172.21.15.6:0/53893426 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5a41f4550 con 0x7fe5a4152b30 2024-04-24T23:28:34.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:34.952+0000 7fe5aa600640 1 --2- 172.21.15.6:0/53893426 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe5a4152b30 0x7fe5a40b13c0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fe59c00c760 tx=0x7fe59c00cc20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:28:34.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:34.952+0000 7fe5a1600640 1 -- 172.21.15.6:0/53893426 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe59c003da0 con 0x7fe5a4152b30 2024-04-24T23:28:34.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:34.952+0000 7fe5a1600640 1 -- 172.21.15.6:0/53893426 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7fe59c005070 con 0x7fe5a4152b30 2024-04-24T23:28:34.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:34.952+0000 7fe5ab000640 1 -- 172.21.15.6:0/53893426 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5a41537f0 con 0x7fe5a4152b30 2024-04-24T23:28:34.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:34.952+0000 7fe5a1600640 1 -- 172.21.15.6:0/53893426 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe59c005480 con 0x7fe5a4152b30 2024-04-24T23:28:34.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:34.952+0000 7fe5ab000640 1 -- 172.21.15.6:0/53893426 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5a4153db0 con 0x7fe5a4152b30 2024-04-24T23:28:34.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:34.953+0000 7fe5ab000640 1 -- 172.21.15.6:0/53893426 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe5a4153ff0 con 0x7fe5a4152b30 2024-04-24T23:28:34.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:34.954+0000 7fe5a1600640 1 -- 172.21.15.6:0/53893426 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fe59c005620 con 0x7fe5a4152b30 2024-04-24T23:28:34.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:34.955+0000 7fe5a1600640 1 --2- 172.21.15.6:0/53893426 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fe58406be60 0x7fe58406e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:28:34.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:34.955+0000 7fe5a1600640 1 -- 172.21.15.6:0/53893426 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6934+0+0 (secure 0 0 0) 0x7fe59c08b220 con 0x7fe5a4152b30 2024-04-24T23:28:34.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:34.955+0000 7fe5a9c00640 1 --2- 172.21.15.6:0/53893426 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fe58406be60 0x7fe58406e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:28:34.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:34.956+0000 7fe5a9c00640 1 --2- 172.21.15.6:0/53893426 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fe58406be60 0x7fe58406e320 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fe598005880 tx=0x7fe598009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:28:34.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:34.959+0000 7fe5a1600640 1 -- 172.21.15.6:0/53893426 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fe5a4153ff0 con 0x7fe5a4152b30 2024-04-24T23:28:35.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.194+0000 7fe5ab000640 1 -- 172.21.15.6:0/53893426 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe5a4075b30 con 0x7fe5a4152b30 2024-04-24T23:28:35.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.195+0000 7fe5a1600640 1 -- 172.21.15.6:0/53893426 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) ==== 94+0+6861 (secure 0 0 0) 0x7fe59c058f80 con 0x7fe5a4152b30 2024-04-24T23:28:35.196 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:28:35.196 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":54,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5081,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/1190759626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1190759626},{"type":"v1","addr":"172.21.15.6:6834","nonce":1190759626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":5249,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6838/2416336040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6835","nonce":2416336040},{"type":"v1","addr":"172.21.15.190:6838","nonce":2416336040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":42},{"gid":5503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6837/3928731059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6836","nonce":3928731059},{"type":"v1","addr":"172.21.15.6:6837","nonce":3928731059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":35},{"gid":5902,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/4245754886","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":4245754886},{"type":"v1","addr":"172.21.15.6:6835","nonce":4245754886}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":54}],"filesystems":[{"mdsmap":{"epoch":53,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:15:36.400716+0000","modified":"2024-04-24T23:27:18.122830+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5725,"mds_1":5599},"failed":[],"damaged":[],"stopped":[2,3,4],"info":{"gid_5599":{"gid":5599,"name":"d","rank":1,"incarnation":43,"state":"up:active","state_seq":25,"addr":"172.21.15.190:6839/2643233757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":2643233757},{"type":"v1","addr":"172.21.15.190:6839","nonce":2643233757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5725":{"gid":5725,"name":"f","rank":0,"incarnation":49,"state":"up:active","state_seq":36,"addr":"172.21.15.190:6837/1151527362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":1151527362},{"type":"v1","addr":"172.21.15.190:6837","nonce":1151527362}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5725,"qdb_cluster":[5725,5599]},"id":1}]} 2024-04-24T23:28:35.196 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 54 2024-04-24T23:28:35.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.197+0000 7fe5ab000640 1 -- 172.21.15.6:0/53893426 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fe58406be60 msgr2=0x7fe58406e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:28:35.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.197+0000 7fe5ab000640 1 --2- 172.21.15.6:0/53893426 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fe58406be60 0x7fe58406e320 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fe598005880 tx=0x7fe598009040 comp rx=0 tx=0).stop 2024-04-24T23:28:35.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.198+0000 7fe5ab000640 1 -- 172.21.15.6:0/53893426 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe5a4152b30 msgr2=0x7fe5a40b13c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:28:35.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.198+0000 7fe5ab000640 1 --2- 172.21.15.6:0/53893426 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe5a4152b30 0x7fe5a40b13c0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fe59c00c760 tx=0x7fe59c00cc20 comp rx=0 tx=0).stop 2024-04-24T23:28:35.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.198+0000 7fe5ab000640 1 -- 172.21.15.6:0/53893426 shutdown_connections 2024-04-24T23:28:35.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.198+0000 7fe5ab000640 1 --2- 172.21.15.6:0/53893426 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fe58406be60 0x7fe58406e320 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:28:35.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.198+0000 7fe5ab000640 1 --2- 172.21.15.6:0/53893426 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe5a40c9ff0 0x7fe5a40afb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:28:35.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.198+0000 7fe5ab000640 1 --2- 172.21.15.6:0/53893426 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe5a4152b30 0x7fe5a40b13c0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:28:35.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.198+0000 7fe5ab000640 1 --2- 172.21.15.6:0/53893426 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe5a40c50b0 0x7fe5a41f4010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:28:35.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.198+0000 7fe5ab000640 1 -- 172.21.15.6:0/53893426 >> 172.21.15.6:0/53893426 conn(0x7fe5a406b500 msgr2=0x7fe5a40c9420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:28:35.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.198+0000 7fe5ab000640 1 -- 172.21.15.6:0/53893426 shutdown_connections 2024-04-24T23:28:35.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.198+0000 7fe5ab000640 1 -- 172.21.15.6:0/53893426 wait complete. 2024-04-24T23:28:35.217 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T23:28:35.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.394+0000 7f5936600640 1 Processor -- start 2024-04-24T23:28:35.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.394+0000 7f5936600640 1 -- start start 2024-04-24T23:28:35.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.396+0000 7f5936600640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f59300711e0 0x7f59300ca8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:28:35.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.396+0000 7f5936600640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f59300c90f0 0x7f5930154790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:28:35.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.396+0000 7f5936600640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f593015abc0 con 0x7f5930070e00 2024-04-24T23:28:35.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.396+0000 7f5936600640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f593015b9e0 con 0x7f59300c90f0 2024-04-24T23:28:35.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.396+0000 7f5936600640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f593015ae60 con 0x7f59300711e0 2024-04-24T23:28:35.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.396+0000 7f5935200640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f5930070e00 0x7f593015e360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.6:51864/0 (socket says 172.21.15.6:51864) 2024-04-24T23:28:35.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.396+0000 7f592fe00640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f59300711e0 0x7f59300ca8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:28:35.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.396+0000 7f5935200640 1 -- 172.21.15.6:0/3464618744 learned_addr learned my addr 172.21.15.6:0/3464618744 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:28:35.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.396+0000 7f5935c00640 1 --2- 172.21.15.6:0/3464618744 >> v2:172.21.15.6:3300/0 conn(0x7f59300c90f0 0x7f5930154790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:28:35.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.397+0000 7f592fe00640 1 -- 172.21.15.6:0/3464618744 >> v2:172.21.15.6:3300/0 conn(0x7f59300c90f0 msgr2=0x7f5930154790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:28:35.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.397+0000 7f592fe00640 1 --2- 172.21.15.6:0/3464618744 >> v2:172.21.15.6:3300/0 conn(0x7f59300c90f0 0x7f5930154790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:28:35.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.397+0000 7f592fe00640 1 -- 172.21.15.6:0/3464618744 >> v1:172.21.15.190:6789/0 conn(0x7f5930070e00 legacy=0x7f593015e360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:28:35.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.397+0000 7f592fe00640 1 -- 172.21.15.6:0/3464618744 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f593015b140 con 0x7f59300711e0 2024-04-24T23:28:35.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.397+0000 7f592f400640 1 -- 172.21.15.6:0/3464618744 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4103617255 0 0) 0x7f593015abc0 con 0x7f5930070e00 2024-04-24T23:28:35.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.398+0000 7f592fe00640 1 --2- 172.21.15.6:0/3464618744 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f59300711e0 0x7f59300ca8f0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f5930071af0 tx=0x7f592002a250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5fedb1335c63fc62 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:28:35.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.398+0000 7f592f400640 1 -- 172.21.15.6:0/3464618744 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5920037070 con 0x7f59300711e0 2024-04-24T23:28:35.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.398+0000 7f592f400640 1 -- 172.21.15.6:0/3464618744 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f5920032440 con 0x7f59300711e0 2024-04-24T23:28:35.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.398+0000 7f592f400640 1 -- 172.21.15.6:0/3464618744 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5920032830 con 0x7f59300711e0 2024-04-24T23:28:35.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.399+0000 7f5936600640 1 -- 172.21.15.6:0/3464618744 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f59300711e0 msgr2=0x7f59300ca8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:28:35.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.399+0000 7f5936600640 1 --2- 172.21.15.6:0/3464618744 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f59300711e0 0x7f59300ca8f0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f5930071af0 tx=0x7f592002a250 comp rx=0 tx=0).stop 2024-04-24T23:28:35.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.399+0000 7f5936600640 1 -- 172.21.15.6:0/3464618744 shutdown_connections 2024-04-24T23:28:35.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.399+0000 7f5936600640 1 --2- 172.21.15.6:0/3464618744 >> v2:172.21.15.6:3300/0 conn(0x7f59300c90f0 0x7f5930154790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:28:35.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.399+0000 7f5936600640 1 --2- 172.21.15.6:0/3464618744 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f59300711e0 0x7f59300ca8f0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:28:35.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.399+0000 7f5936600640 1 -- 172.21.15.6:0/3464618744 >> 172.21.15.6:0/3464618744 conn(0x7f593006b0e0 msgr2=0x7f593006b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:28:35.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.399+0000 7f5936600640 1 -- 172.21.15.6:0/3464618744 shutdown_connections 2024-04-24T23:28:35.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.399+0000 7f5936600640 1 -- 172.21.15.6:0/3464618744 wait complete. 2024-04-24T23:28:35.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.400+0000 7f5936600640 1 Processor -- start 2024-04-24T23:28:35.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.401+0000 7f5936600640 1 -- start start 2024-04-24T23:28:35.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.401+0000 7f5936600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5930070e00 0x7f593014de90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:28:35.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.401+0000 7f5936600640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f59300711e0 0x7f59301f09b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:28:35.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.401+0000 7f5936600640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f59300c90f0 0x7f5930151230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:28:35.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.401+0000 7f5936600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f593015ea70 con 0x7f5930070e00 2024-04-24T23:28:35.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.401+0000 7f5936600640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f593015b700 con 0x7f59300711e0 2024-04-24T23:28:35.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.401+0000 7f5936600640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f593015ebf0 con 0x7f59300c90f0 2024-04-24T23:28:35.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.402+0000 7f5935200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5930070e00 0x7f593014de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:28:35.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.402+0000 7f592fe00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f59300711e0 0x7f59301f09b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:28:35.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.402+0000 7f5935c00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f59300c90f0 0x7f5930151230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:28:35.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.402+0000 7f5935200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5930070e00 0x7f593014de90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:48786/0 (socket says 172.21.15.6:48786) 2024-04-24T23:28:35.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.402+0000 7f5935200640 1 -- 172.21.15.6:0/2836074406 learned_addr learned my addr 172.21.15.6:0/2836074406 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:28:35.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.402+0000 7f592fe00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f59300711e0 0x7f59301f09b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:46742/0 (socket says 172.21.15.6:46742) 2024-04-24T23:28:35.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.402+0000 7f5935c00640 1 -- 172.21.15.6:0/2836074406 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f59300711e0 msgr2=0x7f59301f09b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:28:35.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.402+0000 7f5935c00640 1 --2- 172.21.15.6:0/2836074406 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f59300711e0 0x7f59301f09b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:28:35.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.402+0000 7f5935c00640 1 -- 172.21.15.6:0/2836074406 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5930070e00 msgr2=0x7f593014de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:28:35.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.402+0000 7f5935c00640 1 --2- 172.21.15.6:0/2836074406 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5930070e00 0x7f593014de90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:28:35.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.402+0000 7f5935c00640 1 -- 172.21.15.6:0/2836074406 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f593014e560 con 0x7f59300c90f0 2024-04-24T23:28:35.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.403+0000 7f5935c00640 1 --2- 172.21.15.6:0/2836074406 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f59300c90f0 0x7f5930151230 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f5928007d60 tx=0x7f592800e4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:28:35.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.403+0000 7f592cc00640 1 -- 172.21.15.6:0/2836074406 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f592800edd0 con 0x7f59300c90f0 2024-04-24T23:28:35.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.403+0000 7f592cc00640 1 -- 172.21.15.6:0/2836074406 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f5928003990 con 0x7f59300c90f0 2024-04-24T23:28:35.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.403+0000 7f592cc00640 1 -- 172.21.15.6:0/2836074406 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5928003da0 con 0x7f59300c90f0 2024-04-24T23:28:35.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.403+0000 7f5936600640 1 -- 172.21.15.6:0/2836074406 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59301f0ef0 con 0x7f59300c90f0 2024-04-24T23:28:35.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.403+0000 7f5936600640 1 -- 172.21.15.6:0/2836074406 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f593014f670 con 0x7f59300c90f0 2024-04-24T23:28:35.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.404+0000 7f5936600640 1 -- 172.21.15.6:0/2836074406 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5930075b30 con 0x7f59300c90f0 2024-04-24T23:28:35.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.405+0000 7f592cc00640 1 -- 172.21.15.6:0/2836074406 <== mon.2 v2:172.21.15.190:3301/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f592801b020 con 0x7f59300c90f0 2024-04-24T23:28:35.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.406+0000 7f592cc00640 1 --2- 172.21.15.6:0/2836074406 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f591406be30 0x7f591406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:28:35.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.406+0000 7f592cc00640 1 -- 172.21.15.6:0/2836074406 <== mon.2 v2:172.21.15.190:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6934+0+0 (secure 0 0 0) 0x7f592808b4e0 con 0x7f59300c90f0 2024-04-24T23:28:35.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.406+0000 7f5935200640 1 --2- 172.21.15.6:0/2836074406 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f591406be30 0x7f591406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:28:35.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.407+0000 7f5935200640 1 --2- 172.21.15.6:0/2836074406 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f591406be30 0x7f591406e2f0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f5924000f60 tx=0x7f5924007240 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:28:35.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.410+0000 7f592cc00640 1 -- 172.21.15.6:0/2836074406 <== mon.2 v2:172.21.15.190:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f5928059240 con 0x7f59300c90f0 2024-04-24T23:28:35.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.617+0000 7f5936600640 1 -- 172.21.15.6:0/2836074406 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f59300cb220 con 0x7f59300c90f0 2024-04-24T23:28:35.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.618+0000 7f592cc00640 1 -- 172.21.15.6:0/2836074406 <== mon.2 v2:172.21.15.190:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v30) ==== 74+0+19063 (secure 0 0 0) 0x7f592805d540 con 0x7f59300c90f0 2024-04-24T23:28:35.620 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-24T23:28:35.620 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":30,"fsid":"18bef68d-5e6d-4ec5-862b-b6f206e2588d","created":"2024-04-24T23:15:12.535072+0000","modified":"2024-04-24T23:25:42.694801+0000","last_up_change":"2024-04-24T23:15:23.773223+0000","last_in_change":"2024-04-24T23:15:14.905328+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T23:15:24.747441+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-24T23:15:34.051800+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-24T23:15:35.436870+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-24T23:15:39.034257+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"3b7a212d-d498-462e-90bf-ed493c803a09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":3022569437},{"type":"v1","addr":"172.21.15.6:6809","nonce":3022569437}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":3022569437},{"type":"v1","addr":"172.21.15.6:6811","nonce":3022569437}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":3022569437},{"type":"v1","addr":"172.21.15.6:6815","nonce":3022569437}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":3022569437},{"type":"v1","addr":"172.21.15.6:6813","nonce":3022569437}]},"public_addr":"172.21.15.6:6809/3022569437","cluster_addr":"172.21.15.6:6811/3022569437","heartbeat_back_addr":"172.21.15.6:6815/3022569437","heartbeat_front_addr":"172.21.15.6:6813/3022569437","state":["exists","up"]},{"osd":1,"uuid":"7c97f5a2-1735-4775-9f15-eb50099b19d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":3687836888},{"type":"v1","addr":"172.21.15.6:6817","nonce":3687836888}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":3687836888},{"type":"v1","addr":"172.21.15.6:6819","nonce":3687836888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":3687836888},{"type":"v1","addr":"172.21.15.6:6823","nonce":3687836888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":3687836888},{"type":"v1","addr":"172.21.15.6:6821","nonce":3687836888}]},"public_addr":"172.21.15.6:6817/3687836888","cluster_addr":"172.21.15.6:6819/3687836888","heartbeat_back_addr":"172.21.15.6:6823/3687836888","heartbeat_front_addr":"172.21.15.6:6821/3687836888","state":["exists","up"]},{"osd":2,"uuid":"d86bbc18-438c-4ec2-8018-ba7dd78f790b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":1020213724},{"type":"v1","addr":"172.21.15.6:6801","nonce":1020213724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":1020213724},{"type":"v1","addr":"172.21.15.6:6803","nonce":1020213724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":1020213724},{"type":"v1","addr":"172.21.15.6:6807","nonce":1020213724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":1020213724},{"type":"v1","addr":"172.21.15.6:6805","nonce":1020213724}]},"public_addr":"172.21.15.6:6801/1020213724","cluster_addr":"172.21.15.6:6803/1020213724","heartbeat_back_addr":"172.21.15.6:6807/1020213724","heartbeat_front_addr":"172.21.15.6:6805/1020213724","state":["exists","up"]},{"osd":3,"uuid":"3754ae16-4248-40dd-95f9-0f59c0eb5426","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6824","nonce":3163883468},{"type":"v1","addr":"172.21.15.6:6825","nonce":3163883468}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":3163883468},{"type":"v1","addr":"172.21.15.6:6827","nonce":3163883468}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":3163883468},{"type":"v1","addr":"172.21.15.6:6831","nonce":3163883468}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":3163883468},{"type":"v1","addr":"172.21.15.6:6829","nonce":3163883468}]},"public_addr":"172.21.15.6:6825/3163883468","cluster_addr":"172.21.15.6:6827/3163883468","heartbeat_back_addr":"172.21.15.6:6831/3163883468","heartbeat_front_addr":"172.21.15.6:6829/3163883468","state":["exists","up"]},{"osd":4,"uuid":"8b2913a0-d006-479c-84ed-5b21e0e6c392","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":1989500934},{"type":"v1","addr":"172.21.15.190:6809","nonce":1989500934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":1989500934},{"type":"v1","addr":"172.21.15.190:6811","nonce":1989500934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":1989500934},{"type":"v1","addr":"172.21.15.190:6815","nonce":1989500934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":1989500934},{"type":"v1","addr":"172.21.15.190:6813","nonce":1989500934}]},"public_addr":"172.21.15.190:6809/1989500934","cluster_addr":"172.21.15.190:6811/1989500934","heartbeat_back_addr":"172.21.15.190:6815/1989500934","heartbeat_front_addr":"172.21.15.190:6813/1989500934","state":["exists","up"]},{"osd":5,"uuid":"499b79d0-0585-4dae-ab69-196a551793da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":3687548412},{"type":"v1","addr":"172.21.15.190:6801","nonce":3687548412}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":3687548412},{"type":"v1","addr":"172.21.15.190:6803","nonce":3687548412}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":3687548412},{"type":"v1","addr":"172.21.15.190:6807","nonce":3687548412}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":3687548412},{"type":"v1","addr":"172.21.15.190:6805","nonce":3687548412}]},"public_addr":"172.21.15.190:6801/3687548412","cluster_addr":"172.21.15.190:6803/3687548412","heartbeat_back_addr":"172.21.15.190:6807/3687548412","heartbeat_front_addr":"172.21.15.190:6805/3687548412","state":["exists","up"]},{"osd":6,"uuid":"6ba6df3b-4a6b-4a89-a890-0f79f387d4d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":870571564},{"type":"v1","addr":"172.21.15.190:6817","nonce":870571564}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":870571564},{"type":"v1","addr":"172.21.15.190:6819","nonce":870571564}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":870571564},{"type":"v1","addr":"172.21.15.190:6823","nonce":870571564}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":870571564},{"type":"v1","addr":"172.21.15.190:6821","nonce":870571564}]},"public_addr":"172.21.15.190:6817/870571564","cluster_addr":"172.21.15.190:6819/870571564","heartbeat_back_addr":"172.21.15.190:6823/870571564","heartbeat_front_addr":"172.21.15.190:6821/870571564","state":["exists","up"]},{"osd":7,"uuid":"e9668b1c-a000-4332-86e8-4efd73b02409","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":123639197},{"type":"v1","addr":"172.21.15.190:6825","nonce":123639197}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":123639197},{"type":"v1","addr":"172.21.15.190:6827","nonce":123639197}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":123639197},{"type":"v1","addr":"172.21.15.190:6831","nonce":123639197}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":123639197},{"type":"v1","addr":"172.21.15.190:6829","nonce":123639197}]},"public_addr":"172.21.15.190:6825/123639197","cluster_addr":"172.21.15.190:6827/123639197","heartbeat_back_addr":"172.21.15.190:6831/123639197","heartbeat_front_addr":"172.21.15.190:6829/123639197","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.321646+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.363873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.466963+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.531989+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:19.990707+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.043543+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.032492+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:15:20.115004+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.2f","mappings":[{"from":0,"to":7}]},{"pgid":"4.19","mappings":[{"from":7,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.6:6835/489092393":"2024-04-25T23:25:42.694416+0000","172.21.15.190:6837/580315269":"2024-04-25T23:24:32.678217+0000","172.21.15.6:6834/105267896":"2024-04-25T23:20:17.585424+0000","172.21.15.6:6833/489092393":"2024-04-25T23:25:42.694416+0000","172.21.15.6:6832/105267896":"2024-04-25T23:20:17.585424+0000","172.21.15.190:6834/580315269":"2024-04-25T23:24:32.678217+0000","172.21.15.6:6837/2590421094":"2024-04-25T23:18:27.556337+0000","172.21.15.6:6836/2590421094":"2024-04-25T23:18:27.556337+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T23:28:35.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.621+0000 7f5936600640 1 -- 172.21.15.6:0/2836074406 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f591406be30 msgr2=0x7f591406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:28:35.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.621+0000 7f5936600640 1 --2- 172.21.15.6:0/2836074406 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f591406be30 0x7f591406e2f0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f5924000f60 tx=0x7f5924007240 comp rx=0 tx=0).stop 2024-04-24T23:28:35.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.621+0000 7f5936600640 1 -- 172.21.15.6:0/2836074406 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f59300c90f0 msgr2=0x7f5930151230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:28:35.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.621+0000 7f5936600640 1 --2- 172.21.15.6:0/2836074406 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f59300c90f0 0x7f5930151230 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f5928007d60 tx=0x7f592800e4b0 comp rx=0 tx=0).stop 2024-04-24T23:28:35.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.621+0000 7f5936600640 1 -- 172.21.15.6:0/2836074406 shutdown_connections 2024-04-24T23:28:35.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.621+0000 7f5936600640 1 --2- 172.21.15.6:0/2836074406 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f591406be30 0x7f591406e2f0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:28:35.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.621+0000 7f5936600640 1 --2- 172.21.15.6:0/2836074406 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f59300c90f0 0x7f5930151230 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:28:35.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.621+0000 7f5936600640 1 --2- 172.21.15.6:0/2836074406 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f59300711e0 0x7f59301f09b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:28:35.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.621+0000 7f5936600640 1 --2- 172.21.15.6:0/2836074406 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5930070e00 0x7f593014de90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:28:35.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.621+0000 7f5936600640 1 -- 172.21.15.6:0/2836074406 >> 172.21.15.6:0/2836074406 conn(0x7f593006b0e0 msgr2=0x7f5930158ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:28:35.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.621+0000 7f5936600640 1 -- 172.21.15.6:0/2836074406 shutdown_connections 2024-04-24T23:28:35.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.621+0000 7f5936600640 1 -- 172.21.15.6:0/2836074406 wait complete. 2024-04-24T23:28:35.642 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-24T23:28:35.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.836+0000 7fc42a800640 1 Processor -- start 2024-04-24T23:28:35.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.836+0000 7fc42a800640 1 -- start start 2024-04-24T23:28:35.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.838+0000 7fc42a800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc4240c6be0 0x7fc424158870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:28:35.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.838+0000 7fc42a800640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fc424158db0 0x7fc4240c3d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:28:35.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.838+0000 7fc42a800640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc4240c3a90 con 0x7fc4241529b0 2024-04-24T23:28:35.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.838+0000 7fc42a800640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7fc4240c7e70 con 0x7fc4240c6be0 2024-04-24T23:28:35.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.838+0000 7fc42a800640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7fc4240c8150 con 0x7fc424158db0 2024-04-24T23:28:35.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.838+0000 7fc429400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fc4241529b0 0x7fc4240c60a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:35860/0 (socket says 172.21.15.6:35860) 2024-04-24T23:28:35.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.838+0000 7fc429400640 1 -- 172.21.15.6:0/2110982516 learned_addr learned my addr 172.21.15.6:0/2110982516 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:28:35.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.838+0000 7fc429e00640 1 --2- 172.21.15.6:0/2110982516 >> v2:172.21.15.190:3300/0 conn(0x7fc424158db0 0x7fc4240c3d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:28:35.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.838+0000 7fc428a00640 1 --2- 172.21.15.6:0/2110982516 >> v2:172.21.15.6:3300/0 conn(0x7fc4240c6be0 0x7fc424158870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:28:35.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.839+0000 7fc423e00640 1 -- 172.21.15.6:0/2110982516 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1329066553 0 0) 0x7fc4240c3a90 con 0x7fc4241529b0 2024-04-24T23:28:35.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.839+0000 7fc429e00640 1 -- 172.21.15.6:0/2110982516 >> v2:172.21.15.6:3300/0 conn(0x7fc4240c6be0 msgr2=0x7fc424158870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:28:35.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.839+0000 7fc429e00640 1 --2- 172.21.15.6:0/2110982516 >> v2:172.21.15.6:3300/0 conn(0x7fc4240c6be0 0x7fc424158870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:28:35.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.839+0000 7fc429e00640 1 -- 172.21.15.6:0/2110982516 >> v1:172.21.15.6:6789/0 conn(0x7fc4241529b0 legacy=0x7fc4240c60a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:28:35.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.839+0000 7fc429e00640 1 -- 172.21.15.6:0/2110982516 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4240c8430 con 0x7fc424158db0 2024-04-24T23:28:35.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.840+0000 7fc429e00640 1 --2- 172.21.15.6:0/2110982516 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc424158db0 0x7fc4240c3d50 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fc42406f3b0 tx=0x7fc41c02a680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e03ea67b6150c8b5 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:28:35.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.840+0000 7fc423e00640 1 -- 172.21.15.6:0/2110982516 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc41c007740 con 0x7fc424158db0 2024-04-24T23:28:35.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.840+0000 7fc423e00640 1 -- 172.21.15.6:0/2110982516 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7fc41c0078e0 con 0x7fc424158db0 2024-04-24T23:28:35.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.840+0000 7fc423e00640 1 -- 172.21.15.6:0/2110982516 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc41c007cd0 con 0x7fc424158db0 2024-04-24T23:28:35.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.841+0000 7fc42a800640 1 -- 172.21.15.6:0/2110982516 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc424158db0 msgr2=0x7fc4240c3d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:28:35.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.841+0000 7fc42a800640 1 --2- 172.21.15.6:0/2110982516 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc424158db0 0x7fc4240c3d50 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fc42406f3b0 tx=0x7fc41c02a680 comp rx=0 tx=0).stop 2024-04-24T23:28:35.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.841+0000 7fc42a800640 1 -- 172.21.15.6:0/2110982516 shutdown_connections 2024-04-24T23:28:35.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.841+0000 7fc42a800640 1 --2- 172.21.15.6:0/2110982516 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc424158db0 0x7fc4240c3d50 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:28:35.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.841+0000 7fc42a800640 1 --2- 172.21.15.6:0/2110982516 >> v2:172.21.15.6:3300/0 conn(0x7fc4240c6be0 0x7fc424158870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:28:35.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.841+0000 7fc42a800640 1 -- 172.21.15.6:0/2110982516 >> 172.21.15.6:0/2110982516 conn(0x7fc4240cc1e0 msgr2=0x7fc4240cc5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:28:35.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.841+0000 7fc42a800640 1 -- 172.21.15.6:0/2110982516 shutdown_connections 2024-04-24T23:28:35.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.841+0000 7fc42a800640 1 -- 172.21.15.6:0/2110982516 wait complete. 2024-04-24T23:28:35.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.843+0000 7fc42a800640 1 Processor -- start 2024-04-24T23:28:35.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.843+0000 7fc42a800640 1 -- start start 2024-04-24T23:28:35.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.844+0000 7fc42a800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc4240c6be0 0x7fc4241e1ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:28:35.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.844+0000 7fc42a800640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc4241529b0 0x7fc424149e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:28:35.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.844+0000 7fc42a800640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc424158db0 0x7fc4240c9fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:28:35.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.844+0000 7fc42a800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7fc4240c36e0 con 0x7fc4240c6be0 2024-04-24T23:28:35.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.844+0000 7fc42a800640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fc424152d90 con 0x7fc4241529b0 2024-04-24T23:28:35.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.844+0000 7fc42a800640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fc4241ef2f0 con 0x7fc424158db0 2024-04-24T23:28:35.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.844+0000 7fc429400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc4240c6be0 0x7fc4241e1ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:28:35.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.844+0000 7fc428a00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc4241529b0 0x7fc424149e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:28:35.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.844+0000 7fc429400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc4240c6be0 0x7fc4241e1ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:48808/0 (socket says 172.21.15.6:48808) 2024-04-24T23:28:35.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.844+0000 7fc429400640 1 -- 172.21.15.6:0/1372352089 learned_addr learned my addr 172.21.15.6:0/1372352089 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:28:35.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.844+0000 7fc429e00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc424158db0 0x7fc4240c9fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:28:35.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.844+0000 7fc428a00640 1 -- 172.21.15.6:0/1372352089 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc424158db0 msgr2=0x7fc4240c9fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:28:35.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.844+0000 7fc428a00640 1 --2- 172.21.15.6:0/1372352089 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc424158db0 0x7fc4240c9fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:28:35.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.844+0000 7fc428a00640 1 -- 172.21.15.6:0/1372352089 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc4240c6be0 msgr2=0x7fc4241e1ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:28:35.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.844+0000 7fc428a00640 1 --2- 172.21.15.6:0/1372352089 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc4240c6be0 0x7fc4241e1ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:28:35.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.844+0000 7fc428a00640 1 -- 172.21.15.6:0/1372352089 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4240cbed0 con 0x7fc4241529b0 2024-04-24T23:28:35.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.845+0000 7fc428a00640 1 --2- 172.21.15.6:0/1372352089 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc4241529b0 0x7fc424149e30 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fc41000e850 tx=0x7fc41000ed10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:28:35.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.845+0000 7fc421600640 1 -- 172.21.15.6:0/1372352089 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc410017070 con 0x7fc4241529b0 2024-04-24T23:28:35.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.845+0000 7fc421600640 1 -- 172.21.15.6:0/1372352089 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7fc41000c940 con 0x7fc4241529b0 2024-04-24T23:28:35.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.845+0000 7fc421600640 1 -- 172.21.15.6:0/1372352089 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc41000cd30 con 0x7fc4241529b0 2024-04-24T23:28:35.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.845+0000 7fc42a800640 1 -- 172.21.15.6:0/1372352089 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4241ee9c0 con 0x7fc4241529b0 2024-04-24T23:28:35.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.845+0000 7fc42a800640 1 -- 172.21.15.6:0/1372352089 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4241eef80 con 0x7fc4241529b0 2024-04-24T23:28:35.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.846+0000 7fc42a800640 1 -- 172.21.15.6:0/1372352089 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc424075ba0 con 0x7fc4241529b0 2024-04-24T23:28:35.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.847+0000 7fc421600640 1 -- 172.21.15.6:0/1372352089 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fc41001a070 con 0x7fc4241529b0 2024-04-24T23:28:35.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.848+0000 7fc421600640 1 --2- 172.21.15.6:0/1372352089 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fc40406be30 0x7fc40406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:28:35.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.848+0000 7fc421600640 1 -- 172.21.15.6:0/1372352089 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6934+0+0 (secure 0 0 0) 0x7fc41008b800 con 0x7fc4241529b0 2024-04-24T23:28:35.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.848+0000 7fc429400640 1 --2- 172.21.15.6:0/1372352089 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fc40406be30 0x7fc40406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:28:35.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.849+0000 7fc429400640 1 --2- 172.21.15.6:0/1372352089 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fc40406be30 0x7fc40406e2f0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fc418000f60 tx=0x7fc418007e10 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:28:35.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:35.852+0000 7fc421600640 1 -- 172.21.15.6:0/1372352089 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fc410059560 con 0x7fc4241529b0 2024-04-24T23:28:36.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:36.091+0000 7fc42a800640 1 -- 172.21.15.6:0/1372352089 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fc424156af0 con 0x7fc4241529b0 2024-04-24T23:28:36.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:36.109+0000 7fc421600640 1 -- 172.21.15.6:0/1372352089 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v55) ==== 161+0+0 (secure 0 0 0) 0x7fc41005d860 con 0x7fc4241529b0 2024-04-24T23:28:36.111 INFO:teuthology.orchestra.run.smithi006.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-24T23:28:36.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:36.113+0000 7fc42a800640 1 -- 172.21.15.6:0/1372352089 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fc40406be30 msgr2=0x7fc40406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:28:36.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:36.113+0000 7fc42a800640 1 --2- 172.21.15.6:0/1372352089 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fc40406be30 0x7fc40406e2f0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fc418000f60 tx=0x7fc418007e10 comp rx=0 tx=0).stop 2024-04-24T23:28:36.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:36.113+0000 7fc42a800640 1 -- 172.21.15.6:0/1372352089 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc4241529b0 msgr2=0x7fc424149e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:28:36.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:36.113+0000 7fc42a800640 1 --2- 172.21.15.6:0/1372352089 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc4241529b0 0x7fc424149e30 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fc41000e850 tx=0x7fc41000ed10 comp rx=0 tx=0).stop 2024-04-24T23:28:36.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:36.114+0000 7fc42a800640 1 -- 172.21.15.6:0/1372352089 shutdown_connections 2024-04-24T23:28:36.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:36.114+0000 7fc42a800640 1 --2- 172.21.15.6:0/1372352089 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7fc40406be30 0x7fc40406e2f0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:28:36.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:36.114+0000 7fc42a800640 1 --2- 172.21.15.6:0/1372352089 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc424158db0 0x7fc4240c9fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:28:36.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:36.114+0000 7fc42a800640 1 --2- 172.21.15.6:0/1372352089 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc4241529b0 0x7fc424149e30 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:28:36.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:36.114+0000 7fc42a800640 1 --2- 172.21.15.6:0/1372352089 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc4240c6be0 0x7fc4241e1ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:28:36.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:36.114+0000 7fc42a800640 1 -- 172.21.15.6:0/1372352089 >> 172.21.15.6:0/1372352089 conn(0x7fc4240cc1e0 msgr2=0x7fc42406df90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:28:36.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:36.114+0000 7fc42a800640 1 -- 172.21.15.6:0/1372352089 shutdown_connections 2024-04-24T23:28:36.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:36.114+0000 7fc42a800640 1 -- 172.21.15.6:0/1372352089 wait complete. 2024-04-24T23:28:36.133 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-24T23:28:36.148 INFO:tasks.ceph.mds.d.smithi190.stdout:starting mds.d at 2024-04-24T23:28:36.148 INFO:tasks.ceph.mds.f.smithi190.stdout:starting mds.f at 2024-04-24T23:28:36.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:36.324+0000 7f89d5200640 1 Processor -- start 2024-04-24T23:28:36.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:36.324+0000 7f89d5200640 1 -- start start 2024-04-24T23:28:36.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:36.326+0000 7f89d5200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f89d00c5e40 0x7f89d00c6ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:28:36.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:36.326+0000 7f89d5200640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f89d00c8570 0x7f89d01524a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:28:36.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:36.326+0000 7f89d5200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f89d00cca80 con 0x7f89d00c5a60 2024-04-24T23:28:36.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:36.326+0000 7f89d5200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f89d015c440 con 0x7f89d00c5e40 2024-04-24T23:28:36.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:36.326+0000 7f89d5200640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f89d00ccd60 con 0x7f89d00c8570 2024-04-24T23:28:36.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:36.326+0000 7f89cf400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f89d00c5a60 0x7f89d015bd30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:35862/0 (socket says 172.21.15.6:35862) 2024-04-24T23:28:36.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:36.326+0000 7f89cf400640 1 -- 172.21.15.6:0/3914016323 learned_addr learned my addr 172.21.15.6:0/3914016323 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:28:36.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:36.326+0000 7f89cea00640 1 --2- 172.21.15.6:0/3914016323 >> v2:172.21.15.6:3300/0 conn(0x7f89d00c5e40 0x7f89d00c6ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:28:36.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:36.326+0000 7f89cfe00640 1 --2- 172.21.15.6:0/3914016323 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f89d00c8570 0x7f89d01524a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:28:36.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:36.326+0000 7f89cfe00640 1 -- 172.21.15.6:0/3914016323 >> v2:172.21.15.6:3300/0 conn(0x7f89d00c5e40 msgr2=0x7f89d00c6ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:28:36.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:36.327+0000 7f89cfe00640 1 --2- 172.21.15.6:0/3914016323 >> v2:172.21.15.6:3300/0 conn(0x7f89d00c5e40 0x7f89d00c6ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:28:36.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:36.327+0000 7f89cfe00640 1 -- 172.21.15.6:0/3914016323 >> v1:172.21.15.6:6789/0 conn(0x7f89d00c5a60 legacy=0x7f89d015bd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:28:36.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:36.327+0000 7f89cfe00640 1 -- 172.21.15.6:0/3914016323 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89d006eab0 con 0x7f89d00c8570 2024-04-24T23:28:36.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:36.326+0000 7f89ce000640 1 -- 172.21.15.6:0/3914016323 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2261636890 0 0) 0x7f89d00cca80 con 0x7f89d00c5a60 2024-04-24T23:28:36.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:36.328+0000 7f89cfe00640 1 --2- 172.21.15.6:0/3914016323 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f89d00c8570 0x7f89d01524a0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f89c8009d90 tx=0x7f89c802a5d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=628f3bdbc90b53f server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:28:36.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:36.328+0000 7f89ce000640 1 -- 172.21.15.6:0/3914016323 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89c8031600 con 0x7f89d00c8570 2024-04-24T23:28:36.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:36.328+0000 7f89ce000640 1 -- 172.21.15.6:0/3914016323 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f89c80317a0 con 0x7f89d00c8570 2024-04-24T23:28:36.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:36.328+0000 7f89ce000640 1 -- 172.21.15.6:0/3914016323 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89c8031b90 con 0x7f89d00c8570 2024-04-24T23:28:36.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:36.328+0000 7f89d5200640 1 -- 172.21.15.6:0/3914016323 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f89d00c8570 msgr2=0x7f89d01524a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:28:36.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:36.328+0000 7f89d5200640 1 --2- 172.21.15.6:0/3914016323 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f89d00c8570 0x7f89d01524a0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f89c8009d90 tx=0x7f89c802a5d0 comp rx=0 tx=0).stop 2024-04-24T23:28:36.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:36.329+0000 7f89d5200640 1 -- 172.21.15.6:0/3914016323 shutdown_connections 2024-04-24T23:28:36.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:36.329+0000 7f89d5200640 1 --2- 172.21.15.6:0/3914016323 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f89d00c8570 0x7f89d01524a0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:28:36.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:36.329+0000 7f89d5200640 1 --2- 172.21.15.6:0/3914016323 >> v2:172.21.15.6:3300/0 conn(0x7f89d00c5e40 0x7f89d00c6ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:28:36.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:36.329+0000 7f89d5200640 1 -- 172.21.15.6:0/3914016323 >> 172.21.15.6:0/3914016323 conn(0x7f89d006cc40 msgr2=0x7f89d006d050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:28:36.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:36.329+0000 7f89d5200640 1 -- 172.21.15.6:0/3914016323 shutdown_connections 2024-04-24T23:28:36.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:36.329+0000 7f89d5200640 1 -- 172.21.15.6:0/3914016323 wait complete. 2024-04-24T23:28:36.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:36.330+0000 7f89d5200640 1 Processor -- start 2024-04-24T23:28:36.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:36.331+0000 7f89d5200640 1 -- start start 2024-04-24T23:28:36.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:36.332+0000 7f89d5200640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f89d00c5a60 0x7f89d0147720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:28:36.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:36.332+0000 7f89d5200640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f89d00c5e40 0x7f89d0147c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:28:36.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:36.332+0000 7f89d5200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f89d00c8570 0x7f89d0142d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:28:36.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:36.332+0000 7f89d5200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f89d006f590 con 0x7f89d00c8570 2024-04-24T23:28:36.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:36.332+0000 7f89d5200640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f89d006f070 con 0x7f89d00c5a60 2024-04-24T23:28:36.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:36.332+0000 7f89d5200640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f89d014cf10 con 0x7f89d00c5e40 2024-04-24T23:28:36.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:36.332+0000 7f89cf400640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f89d00c5a60 0x7f89d0147720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:28:36.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:36.332+0000 7f89cf400640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f89d00c5a60 0x7f89d0147720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:46772/0 (socket says 172.21.15.6:46772) 2024-04-24T23:28:36.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:36.332+0000 7f89cf400640 1 -- 172.21.15.6:0/1037136699 learned_addr learned my addr 172.21.15.6:0/1037136699 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:28:36.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:36.332+0000 7f89cea00640 1 --2- 172.21.15.6:0/1037136699 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f89d00c5e40 0x7f89d0147c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:28:36.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:36.332+0000 7f89cfe00640 1 --2- 172.21.15.6:0/1037136699 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f89d00c8570 0x7f89d0142d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:28:36.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:36.332+0000 7f89cf400640 1 -- 172.21.15.6:0/1037136699 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f89d00c5e40 msgr2=0x7f89d0147c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:28:36.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:36.332+0000 7f89cf400640 1 --2- 172.21.15.6:0/1037136699 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f89d00c5e40 0x7f89d0147c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:28:36.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:36.332+0000 7f89cf400640 1 -- 172.21.15.6:0/1037136699 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f89d00c8570 msgr2=0x7f89d0142d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:28:36.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:36.332+0000 7f89cf400640 1 --2- 172.21.15.6:0/1037136699 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f89d00c8570 0x7f89d0142d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:28:36.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:36.332+0000 7f89cf400640 1 -- 172.21.15.6:0/1037136699 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89d0150f30 con 0x7f89d00c5a60 2024-04-24T23:28:36.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:36.333+0000 7f89cf400640 1 --2- 172.21.15.6:0/1037136699 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f89d00c5a60 0x7f89d0147720 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f89c000c4d0 tx=0x7f89c000c990 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:28:36.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:36.333+0000 7f89c7400640 1 -- 172.21.15.6:0/1037136699 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89c0018070 con 0x7f89d00c5a60 2024-04-24T23:28:36.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:36.333+0000 7f89c7400640 1 -- 172.21.15.6:0/1037136699 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f89c0007500 con 0x7f89d00c5a60 2024-04-24T23:28:36.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:36.333+0000 7f89c7400640 1 -- 172.21.15.6:0/1037136699 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89c0004a40 con 0x7f89d00c5a60 2024-04-24T23:28:36.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:36.333+0000 7f89d5200640 1 -- 172.21.15.6:0/1037136699 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89d0151210 con 0x7f89d00c5a60 2024-04-24T23:28:36.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:36.333+0000 7f89d5200640 1 -- 172.21.15.6:0/1037136699 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89d01517d0 con 0x7f89d00c5a60 2024-04-24T23:28:36.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:36.334+0000 7f89d5200640 1 -- 172.21.15.6:0/1037136699 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f89d0075b30 con 0x7f89d00c5a60 2024-04-24T23:28:36.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:36.335+0000 7f89c7400640 1 -- 172.21.15.6:0/1037136699 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f89c001b070 con 0x7f89d00c5a60 2024-04-24T23:28:36.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:36.336+0000 7f89c7400640 1 --2- 172.21.15.6:0/1037136699 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f89b006be60 0x7f89b006e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:28:36.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:36.336+0000 7f89c7400640 1 -- 172.21.15.6:0/1037136699 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 7106+0+0 (secure 0 0 0) 0x7f89c008a7b0 con 0x7f89d00c5a60 2024-04-24T23:28:36.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:36.336+0000 7f89cea00640 1 --2- 172.21.15.6:0/1037136699 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f89b006be60 0x7f89b006e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:28:36.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:36.337+0000 7f89cea00640 1 --2- 172.21.15.6:0/1037136699 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f89b006be60 0x7f89b006e320 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f89d0148b40 tx=0x7f89bc009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:28:36.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:36.341+0000 7f89c7400640 1 -- 172.21.15.6:0/1037136699 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f89c0058460 con 0x7f89d00c5a60 2024-04-24T23:28:36.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:36.575+0000 7f89d5200640 1 -- 172.21.15.6:0/1037136699 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f89d006d1c0 con 0x7f89d00c5a60 2024-04-24T23:28:37.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:37.110+0000 7f89c7400640 1 -- 172.21.15.6:0/1037136699 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v56) ==== 104+0+0 (secure 0 0 0) 0x7f89c005c760 con 0x7f89d00c5a60 2024-04-24T23:28:37.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:37.113+0000 7f89d5200640 1 -- 172.21.15.6:0/1037136699 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f89b006be60 msgr2=0x7f89b006e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:28:37.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:37.113+0000 7f89d5200640 1 --2- 172.21.15.6:0/1037136699 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f89b006be60 0x7f89b006e320 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f89d0148b40 tx=0x7f89bc009190 comp rx=0 tx=0).stop 2024-04-24T23:28:37.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:37.113+0000 7f89d5200640 1 -- 172.21.15.6:0/1037136699 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f89d00c5a60 msgr2=0x7f89d0147720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:28:37.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:37.113+0000 7f89d5200640 1 --2- 172.21.15.6:0/1037136699 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f89d00c5a60 0x7f89d0147720 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f89c000c4d0 tx=0x7f89c000c990 comp rx=0 tx=0).stop 2024-04-24T23:28:37.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:37.113+0000 7f89d5200640 1 -- 172.21.15.6:0/1037136699 shutdown_connections 2024-04-24T23:28:37.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:37.113+0000 7f89d5200640 1 --2- 172.21.15.6:0/1037136699 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f89b006be60 0x7f89b006e320 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:28:37.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:37.113+0000 7f89d5200640 1 --2- 172.21.15.6:0/1037136699 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f89d00c5e40 0x7f89d0147c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:28:37.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:37.113+0000 7f89d5200640 1 --2- 172.21.15.6:0/1037136699 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f89d00c8570 0x7f89d0142d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:28:37.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:37.113+0000 7f89d5200640 1 --2- 172.21.15.6:0/1037136699 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f89d00c5a60 0x7f89d0147720 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:28:37.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:37.113+0000 7f89d5200640 1 -- 172.21.15.6:0/1037136699 >> 172.21.15.6:0/1037136699 conn(0x7f89d006cc40 msgr2=0x7f89d0156900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:28:37.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:37.113+0000 7f89d5200640 1 -- 172.21.15.6:0/1037136699 shutdown_connections 2024-04-24T23:28:37.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:37.113+0000 7f89d5200640 1 -- 172.21.15.6:0/1037136699 wait complete. 2024-04-24T23:28:37.132 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-24T23:28:37.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:37.338+0000 7f06efe00640 1 Processor -- start 2024-04-24T23:28:37.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:37.339+0000 7f06efe00640 1 -- start start 2024-04-24T23:28:37.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:37.340+0000 7f06efe00640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f06f00c7310 0x7f06f006cce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:28:37.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:37.340+0000 7f06efe00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f06f015ea70 0x7f06f006f180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:28:37.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:37.340+0000 7f06efe00640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f06f006bb90 con 0x7f06f0154a60 2024-04-24T23:28:37.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:37.340+0000 7f06efe00640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f06f015e5e0 con 0x7f06f00c7310 2024-04-24T23:28:37.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:37.340+0000 7f06efe00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f06f006be70 con 0x7f06f015ea70 2024-04-24T23:28:37.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:37.341+0000 7f06eea00640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f06f0154a60 0x7f06f015a800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.6:51878/0 (socket says 172.21.15.6:51878) 2024-04-24T23:28:37.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:37.341+0000 7f06eea00640 1 -- 172.21.15.6:0/2493486956 learned_addr learned my addr 172.21.15.6:0/2493486956 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:28:37.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:37.341+0000 7f06e7e00640 1 --2- 172.21.15.6:0/2493486956 >> v2:172.21.15.190:3300/0 conn(0x7f06f00c7310 0x7f06f006cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:28:37.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:37.341+0000 7f06ef400640 1 --2- 172.21.15.6:0/2493486956 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f06f015ea70 0x7f06f006f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:28:37.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:37.341+0000 7f06e7e00640 1 -- 172.21.15.6:0/2493486956 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f06f015ea70 msgr2=0x7f06f006f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:28:37.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:37.341+0000 7f06e7e00640 1 --2- 172.21.15.6:0/2493486956 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f06f015ea70 0x7f06f006f180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:28:37.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:37.341+0000 7f06e7e00640 1 -- 172.21.15.6:0/2493486956 >> v1:172.21.15.190:6789/0 conn(0x7f06f0154a60 legacy=0x7f06f015a800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:28:37.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:37.341+0000 7f06e7e00640 1 -- 172.21.15.6:0/2493486956 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06f015df90 con 0x7f06f00c7310 2024-04-24T23:28:37.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:37.342+0000 7f06e7e00640 1 --2- 172.21.15.6:0/2493486956 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f06f00c7310 0x7f06f006cce0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f06d80097a0 tx=0x7f06d802a180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ee5dac89566202e3 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:28:37.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:37.342+0000 7f06ee000640 1 -- 172.21.15.6:0/2493486956 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06d8037070 con 0x7f06f00c7310 2024-04-24T23:28:37.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:37.342+0000 7f06ee000640 1 -- 172.21.15.6:0/2493486956 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f06d8032440 con 0x7f06f00c7310 2024-04-24T23:28:37.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:37.342+0000 7f06ee000640 1 -- 172.21.15.6:0/2493486956 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06d8032830 con 0x7f06f00c7310 2024-04-24T23:28:37.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:37.343+0000 7f06efe00640 1 -- 172.21.15.6:0/2493486956 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f06f00c7310 msgr2=0x7f06f006cce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:28:37.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:37.343+0000 7f06efe00640 1 --2- 172.21.15.6:0/2493486956 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f06f00c7310 0x7f06f006cce0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f06d80097a0 tx=0x7f06d802a180 comp rx=0 tx=0).stop 2024-04-24T23:28:37.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:37.343+0000 7f06efe00640 1 -- 172.21.15.6:0/2493486956 shutdown_connections 2024-04-24T23:28:37.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:37.343+0000 7f06efe00640 1 --2- 172.21.15.6:0/2493486956 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f06f015ea70 0x7f06f006f180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:28:37.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:37.343+0000 7f06efe00640 1 --2- 172.21.15.6:0/2493486956 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f06f00c7310 0x7f06f006cce0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:28:37.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:37.343+0000 7f06efe00640 1 -- 172.21.15.6:0/2493486956 >> 172.21.15.6:0/2493486956 conn(0x7f06f006c1f0 msgr2=0x7f06f006c5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:28:37.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:37.343+0000 7f06efe00640 1 -- 172.21.15.6:0/2493486956 shutdown_connections 2024-04-24T23:28:37.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:37.343+0000 7f06efe00640 1 -- 172.21.15.6:0/2493486956 wait complete. 2024-04-24T23:28:37.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:37.345+0000 7f06efe00640 1 Processor -- start 2024-04-24T23:28:37.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:37.345+0000 7f06efe00640 1 -- start start 2024-04-24T23:28:37.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:37.347+0000 7f06efe00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f06f00c7310 0x7f06f01533e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:28:37.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:37.347+0000 7f06efe00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f06f0154a60 0x7f06f01f4210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:28:37.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:37.347+0000 7f06efe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f06f015ea70 0x7f06f014fb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:28:37.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:37.347+0000 7f06efe00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f06f0154fc0 con 0x7f06f015ea70 2024-04-24T23:28:37.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:37.347+0000 7f06efe00640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f06f0154e40 con 0x7f06f0154a60 2024-04-24T23:28:37.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:37.347+0000 7f06efe00640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f06f00c6dc0 con 0x7f06f00c7310 2024-04-24T23:28:37.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:37.347+0000 7f06eea00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f06f00c7310 0x7f06f01533e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:28:37.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:37.347+0000 7f06eea00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f06f00c7310 0x7f06f01533e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.6:45764/0 (socket says 172.21.15.6:45764) 2024-04-24T23:28:37.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:37.347+0000 7f06e7e00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f06f0154a60 0x7f06f01f4210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:28:37.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:37.347+0000 7f06eea00640 1 -- 172.21.15.6:0/2927785746 learned_addr learned my addr 172.21.15.6:0/2927785746 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:28:37.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:37.347+0000 7f06ef400640 1 --2- 172.21.15.6:0/2927785746 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f06f015ea70 0x7f06f014fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:28:37.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:37.347+0000 7f06eea00640 1 -- 172.21.15.6:0/2927785746 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f06f0154a60 msgr2=0x7f06f01f4210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:28:37.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:37.347+0000 7f06eea00640 1 --2- 172.21.15.6:0/2927785746 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f06f0154a60 0x7f06f01f4210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:28:37.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:37.347+0000 7f06eea00640 1 -- 172.21.15.6:0/2927785746 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f06f015ea70 msgr2=0x7f06f014fb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:28:37.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:37.347+0000 7f06eea00640 1 --2- 172.21.15.6:0/2927785746 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f06f015ea70 0x7f06f014fb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:28:37.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:37.347+0000 7f06eea00640 1 -- 172.21.15.6:0/2927785746 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06f0159980 con 0x7f06f00c7310 2024-04-24T23:28:37.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:37.348+0000 7f06eea00640 1 --2- 172.21.15.6:0/2927785746 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f06f00c7310 0x7f06f01533e0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f06e000c4d0 tx=0x7f06e000c990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:28:37.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:37.348+0000 7f06e6a00640 1 -- 172.21.15.6:0/2927785746 <== mon.2 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06e0017070 con 0x7f06f00c7310 2024-04-24T23:28:37.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:37.348+0000 7f06e6a00640 1 -- 172.21.15.6:0/2927785746 <== mon.2 v2:172.21.15.190:3301/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f06e0005ce0 con 0x7f06f00c7310 2024-04-24T23:28:37.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:37.348+0000 7f06e6a00640 1 -- 172.21.15.6:0/2927785746 <== mon.2 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06e0007290 con 0x7f06f00c7310 2024-04-24T23:28:37.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:37.348+0000 7f06efe00640 1 -- 172.21.15.6:0/2927785746 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06f0159c60 con 0x7f06f00c7310 2024-04-24T23:28:37.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:37.348+0000 7f06efe00640 1 -- 172.21.15.6:0/2927785746 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f06f015a220 con 0x7f06f00c7310 2024-04-24T23:28:37.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:37.349+0000 7f06efe00640 1 -- 172.21.15.6:0/2927785746 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f06f015a460 con 0x7f06f00c7310 2024-04-24T23:28:37.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:37.350+0000 7f06e6a00640 1 -- 172.21.15.6:0/2927785746 <== mon.2 v2:172.21.15.190:3301/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f06e001a070 con 0x7f06f00c7310 2024-04-24T23:28:37.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:37.351+0000 7f06e6a00640 1 --2- 172.21.15.6:0/2927785746 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f06cc06be60 0x7f06cc06e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:28:37.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:37.351+0000 7f06e6a00640 1 -- 172.21.15.6:0/2927785746 <== mon.2 v2:172.21.15.190:3301/0 5 ==== osd_map(31..31 src has 1..31) ==== 7106+0+0 (secure 0 0 0) 0x7f06e008a7a0 con 0x7f06f00c7310 2024-04-24T23:28:37.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:37.351+0000 7f06e7e00640 1 --2- 172.21.15.6:0/2927785746 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f06cc06be60 0x7f06cc06e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:28:37.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:37.352+0000 7f06e7e00640 1 --2- 172.21.15.6:0/2927785746 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f06cc06be60 0x7f06cc06e320 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f06d8034000 tx=0x7f06d802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:28:37.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:37.355+0000 7f06e6a00640 1 -- 172.21.15.6:0/2927785746 <== mon.2 v2:172.21.15.190:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f06f015a460 con 0x7f06f00c7310 2024-04-24T23:28:37.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:37.562+0000 7f06efe00640 1 -- 172.21.15.6:0/2927785746 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f06f0075b30 con 0x7f06f00c7310 2024-04-24T23:28:38.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:38.547+0000 7f06e6a00640 1 -- 172.21.15.6:0/2927785746 <== mon.2 v2:172.21.15.190:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v32) ==== 181+0+0 (secure 0 0 0) 0x7f06e00584e0 con 0x7f06f00c7310 2024-04-24T23:28:38.549 INFO:teuthology.orchestra.run.smithi006.stderr:pool 'cephfs_metadata' removed 2024-04-24T23:28:38.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:38.551+0000 7f06efe00640 1 -- 172.21.15.6:0/2927785746 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f06cc06be60 msgr2=0x7f06cc06e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:28:38.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:38.551+0000 7f06efe00640 1 --2- 172.21.15.6:0/2927785746 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f06cc06be60 0x7f06cc06e320 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f06d8034000 tx=0x7f06d802e000 comp rx=0 tx=0).stop 2024-04-24T23:28:38.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:38.551+0000 7f06efe00640 1 -- 172.21.15.6:0/2927785746 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f06f00c7310 msgr2=0x7f06f01533e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:28:38.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:38.551+0000 7f06efe00640 1 --2- 172.21.15.6:0/2927785746 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f06f00c7310 0x7f06f01533e0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f06e000c4d0 tx=0x7f06e000c990 comp rx=0 tx=0).stop 2024-04-24T23:28:38.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:38.551+0000 7f06efe00640 1 -- 172.21.15.6:0/2927785746 shutdown_connections 2024-04-24T23:28:38.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:38.551+0000 7f06efe00640 1 --2- 172.21.15.6:0/2927785746 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f06cc06be60 0x7f06cc06e320 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:28:38.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:38.551+0000 7f06efe00640 1 --2- 172.21.15.6:0/2927785746 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f06f015ea70 0x7f06f014fb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:28:38.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:38.551+0000 7f06efe00640 1 --2- 172.21.15.6:0/2927785746 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f06f0154a60 0x7f06f01f4210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:28:38.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:38.551+0000 7f06efe00640 1 --2- 172.21.15.6:0/2927785746 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f06f00c7310 0x7f06f01533e0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:28:38.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:38.551+0000 7f06efe00640 1 -- 172.21.15.6:0/2927785746 >> 172.21.15.6:0/2927785746 conn(0x7f06f006c1f0 msgr2=0x7f06f00c5a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:28:38.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:38.551+0000 7f06efe00640 1 -- 172.21.15.6:0/2927785746 shutdown_connections 2024-04-24T23:28:38.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:38.551+0000 7f06efe00640 1 -- 172.21.15.6:0/2927785746 wait complete. 2024-04-24T23:28:38.570 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-24T23:28:38.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:38.776+0000 7f7db1200640 1 Processor -- start 2024-04-24T23:28:38.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:38.776+0000 7f7db1200640 1 -- start start 2024-04-24T23:28:38.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:38.778+0000 7f7db1200640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f7dac0c7310 0x7f7dac06cce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:28:38.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:38.778+0000 7f7db1200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7dac06bb90 con 0x7f7dac15ea70 2024-04-24T23:28:38.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:38.778+0000 7f7db1200640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7dac06be70 con 0x7f7dac154a60 2024-04-24T23:28:38.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:38.778+0000 7f7db1200640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f7dac15e5e0 con 0x7f7dac0c7310 2024-04-24T23:28:38.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:38.778+0000 7f7daaa00640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f7dac0c7310 0x7f7dac06cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:28:38.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:38.778+0000 7f7daaa00640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f7dac0c7310 0x7f7dac06cce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:46802/0 (socket says 172.21.15.6:46802) 2024-04-24T23:28:38.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:38.778+0000 7f7daaa00640 1 -- 172.21.15.6:0/963011467 learned_addr learned my addr 172.21.15.6:0/963011467 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:28:38.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:38.778+0000 7f7dab400640 1 --1- 172.21.15.6:0/963011467 >> v1:172.21.15.190:6789/0 conn(0x7f7dac154a60 0x7f7dac15a800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.6:51880/0 (socket says 172.21.15.6:51880) 2024-04-24T23:28:38.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:38.779+0000 7f7daaa00640 1 -- 172.21.15.6:0/963011467 >> v1:172.21.15.190:6789/0 conn(0x7f7dac154a60 legacy=0x7f7dac15a800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:28:38.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:38.779+0000 7f7daaa00640 1 -- 172.21.15.6:0/963011467 >> v1:172.21.15.6:6789/0 conn(0x7f7dac15ea70 legacy=0x7f7dac06f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:28:38.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:38.779+0000 7f7daaa00640 1 -- 172.21.15.6:0/963011467 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7dac15df90 con 0x7f7dac0c7310 2024-04-24T23:28:38.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:38.779+0000 7f7daa000640 1 -- 172.21.15.6:0/963011467 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 403386712 0 0) 0x7f7dac06be70 con 0x7f7dac154a60 2024-04-24T23:28:38.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:38.780+0000 7f7daaa00640 1 --2- 172.21.15.6:0/963011467 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7dac0c7310 0x7f7dac06cce0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f7d98009870 tx=0x7f7d9802a300 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5615b45bfd92957e server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:28:38.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:38.780+0000 7f7daa000640 1 -- 172.21.15.6:0/963011467 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d98037070 con 0x7f7dac0c7310 2024-04-24T23:28:38.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:38.780+0000 7f7daa000640 1 -- 172.21.15.6:0/963011467 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f7d9802ae50 con 0x7f7dac0c7310 2024-04-24T23:28:38.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:38.780+0000 7f7daa000640 1 -- 172.21.15.6:0/963011467 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d98032880 con 0x7f7dac0c7310 2024-04-24T23:28:38.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:38.781+0000 7f7db1200640 1 -- 172.21.15.6:0/963011467 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7dac0c7310 msgr2=0x7f7dac06cce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:28:38.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:38.781+0000 7f7db1200640 1 --2- 172.21.15.6:0/963011467 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7dac0c7310 0x7f7dac06cce0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f7d98009870 tx=0x7f7d9802a300 comp rx=0 tx=0).stop 2024-04-24T23:28:38.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:38.781+0000 7f7db1200640 1 -- 172.21.15.6:0/963011467 shutdown_connections 2024-04-24T23:28:38.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:38.781+0000 7f7db1200640 1 --2- 172.21.15.6:0/963011467 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7dac0c7310 0x7f7dac06cce0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:28:38.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:38.781+0000 7f7db1200640 1 -- 172.21.15.6:0/963011467 >> 172.21.15.6:0/963011467 conn(0x7f7dac06c1f0 msgr2=0x7f7dac06c5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:28:38.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:38.781+0000 7f7db1200640 1 -- 172.21.15.6:0/963011467 shutdown_connections 2024-04-24T23:28:38.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:38.781+0000 7f7db1200640 1 -- 172.21.15.6:0/963011467 wait complete. 2024-04-24T23:28:38.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:38.782+0000 7f7db1200640 1 Processor -- start 2024-04-24T23:28:38.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:38.782+0000 7f7db1200640 1 -- start start 2024-04-24T23:28:38.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:38.783+0000 7f7db1200640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7dac0c7310 0x7f7dac15b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:28:38.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:38.783+0000 7f7db1200640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7dac154a60 0x7f7dac146dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:28:38.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:38.783+0000 7f7db1200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7dac15ea70 0x7f7dac1f0a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:28:38.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:38.783+0000 7f7db1200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f7dac154fc0 con 0x7f7dac15ea70 2024-04-24T23:28:38.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:38.783+0000 7f7db1200640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f7dac154e40 con 0x7f7dac154a60 2024-04-24T23:28:38.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:38.783+0000 7f7db1200640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f7dac06f890 con 0x7f7dac0c7310 2024-04-24T23:28:38.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:38.784+0000 7f7daaa00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7dac154a60 0x7f7dac146dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:28:38.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:38.784+0000 7f7dabe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7dac15ea70 0x7f7dac1f0a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:28:38.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:38.784+0000 7f7daaa00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7dac154a60 0x7f7dac146dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.6:46810/0 (socket says 172.21.15.6:46810) 2024-04-24T23:28:38.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:38.784+0000 7f7daaa00640 1 -- 172.21.15.6:0/620270510 learned_addr learned my addr 172.21.15.6:0/620270510 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:28:38.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:38.784+0000 7f7dab400640 1 --2- 172.21.15.6:0/620270510 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7dac0c7310 0x7f7dac15b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:28:38.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:38.784+0000 7f7daaa00640 1 -- 172.21.15.6:0/620270510 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7dac0c7310 msgr2=0x7f7dac15b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:28:38.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:38.784+0000 7f7daaa00640 1 --2- 172.21.15.6:0/620270510 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7dac0c7310 0x7f7dac15b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:28:38.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:38.784+0000 7f7daaa00640 1 -- 172.21.15.6:0/620270510 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7dac15ea70 msgr2=0x7f7dac1f0a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:28:38.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:38.784+0000 7f7daaa00640 1 --2- 172.21.15.6:0/620270510 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7dac15ea70 0x7f7dac1f0a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:28:38.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:38.784+0000 7f7daaa00640 1 -- 172.21.15.6:0/620270510 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7dac15c000 con 0x7f7dac154a60 2024-04-24T23:28:38.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:38.784+0000 7f7dabe00640 1 --2- 172.21.15.6:0/620270510 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7dac15ea70 0x7f7dac1f0a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:28:38.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:38.784+0000 7f7daaa00640 1 --2- 172.21.15.6:0/620270510 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7dac154a60 0x7f7dac146dd0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f7d9802a340 tx=0x7f7d980046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:28:38.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:38.785+0000 7f7da3400640 1 -- 172.21.15.6:0/620270510 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d98041020 con 0x7f7dac154a60 2024-04-24T23:28:38.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:38.785+0000 7f7da3400640 1 -- 172.21.15.6:0/620270510 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f7d9803b830 con 0x7f7dac154a60 2024-04-24T23:28:38.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:38.785+0000 7f7da3400640 1 -- 172.21.15.6:0/620270510 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d98037040 con 0x7f7dac154a60 2024-04-24T23:28:38.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:38.785+0000 7f7db1200640 1 -- 172.21.15.6:0/620270510 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7dac15d760 con 0x7f7dac154a60 2024-04-24T23:28:38.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:38.785+0000 7f7db1200640 1 -- 172.21.15.6:0/620270510 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7dac15dd20 con 0x7f7dac154a60 2024-04-24T23:28:38.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:38.786+0000 7f7db1200640 1 -- 172.21.15.6:0/620270510 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7dac075b30 con 0x7f7dac154a60 2024-04-24T23:28:38.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:38.787+0000 7f7da3400640 1 -- 172.21.15.6:0/620270510 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f7d98044020 con 0x7f7dac154a60 2024-04-24T23:28:38.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:38.788+0000 7f7da3400640 1 --2- 172.21.15.6:0/620270510 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f7d8c06be30 0x7f7d8c06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:28:38.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:38.788+0000 7f7da3400640 1 -- 172.21.15.6:0/620270510 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 6668+0+0 (secure 0 0 0) 0x7f7d980ab4f0 con 0x7f7dac154a60 2024-04-24T23:28:38.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:38.788+0000 7f7dab400640 1 --2- 172.21.15.6:0/620270510 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f7d8c06be30 0x7f7d8c06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:28:38.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:38.789+0000 7f7dab400640 1 --2- 172.21.15.6:0/620270510 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f7d8c06be30 0x7f7d8c06e2f0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f7d9c007fd0 tx=0x7f7d9c007d80 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:28:38.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:38.792+0000 7f7da3400640 1 -- 172.21.15.6:0/620270510 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f7d9804b030 con 0x7f7dac154a60 2024-04-24T23:28:39.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:39.004+0000 7f7db1200640 1 -- 172.21.15.6:0/620270510 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f7dac06b8d0 con 0x7f7dac154a60 2024-04-24T23:28:39.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:39.550+0000 7f7da3400640 1 -- 172.21.15.6:0/620270510 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v33) ==== 169+0+0 (secure 0 0 0) 0x7f7d98079360 con 0x7f7dac154a60 2024-04-24T23:28:39.551 INFO:teuthology.orchestra.run.smithi006.stderr:pool 'cephfs_data' removed 2024-04-24T23:28:39.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:39.552+0000 7f7db1200640 1 -- 172.21.15.6:0/620270510 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f7d8c06be30 msgr2=0x7f7d8c06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:28:39.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:39.552+0000 7f7db1200640 1 --2- 172.21.15.6:0/620270510 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f7d8c06be30 0x7f7d8c06e2f0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f7d9c007fd0 tx=0x7f7d9c007d80 comp rx=0 tx=0).stop 2024-04-24T23:28:39.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:39.552+0000 7f7db1200640 1 -- 172.21.15.6:0/620270510 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7dac154a60 msgr2=0x7f7dac146dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:28:39.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:39.552+0000 7f7db1200640 1 --2- 172.21.15.6:0/620270510 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7dac154a60 0x7f7dac146dd0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f7d9802a340 tx=0x7f7d980046c0 comp rx=0 tx=0).stop 2024-04-24T23:28:39.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:39.553+0000 7f7db1200640 1 -- 172.21.15.6:0/620270510 shutdown_connections 2024-04-24T23:28:39.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:39.553+0000 7f7db1200640 1 --2- 172.21.15.6:0/620270510 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f7d8c06be30 0x7f7d8c06e2f0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:28:39.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:39.553+0000 7f7db1200640 1 --2- 172.21.15.6:0/620270510 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7dac15ea70 0x7f7dac1f0a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:28:39.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:39.553+0000 7f7db1200640 1 --2- 172.21.15.6:0/620270510 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7dac154a60 0x7f7dac146dd0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:28:39.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:39.553+0000 7f7db1200640 1 --2- 172.21.15.6:0/620270510 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7dac0c7310 0x7f7dac15b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:28:39.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:39.553+0000 7f7db1200640 1 -- 172.21.15.6:0/620270510 >> 172.21.15.6:0/620270510 conn(0x7f7dac06c1f0 msgr2=0x7f7dac152cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:28:39.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:39.553+0000 7f7db1200640 1 -- 172.21.15.6:0/620270510 shutdown_connections 2024-04-24T23:28:39.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:39.553+0000 7f7db1200640 1 -- 172.21.15.6:0/620270510 wait complete. 2024-04-24T23:28:39.586 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data_ec cephfs_data_ec --yes-i-really-really-mean-it 2024-04-24T23:28:39.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:39.767+0000 7f756e600640 1 Processor -- start 2024-04-24T23:28:39.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:39.767+0000 7f756e600640 1 -- start start 2024-04-24T23:28:39.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:39.768+0000 7f756e600640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f7568152c60 0x7f75681589a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:28:39.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:39.769+0000 7f756e600640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f756815cad0 con 0x7f75680c2f10 2024-04-24T23:28:39.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:39.769+0000 7f756e600640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f75680c6c80 con 0x7f75680c4720 2024-04-24T23:28:39.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:39.769+0000 7f756e600640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 -- 0x7f75680c66c0 con 0x7f7568152c60 2024-04-24T23:28:39.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:39.769+0000 7f7567400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f7568152c60 0x7f75681589a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:28:39.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:39.769+0000 7f755fe00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f75680c2f10 0x7f75680c4010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:35882/0 (socket says 172.21.15.6:35882) 2024-04-24T23:28:39.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:39.769+0000 7f755fe00640 1 -- 172.21.15.6:0/1970543236 learned_addr learned my addr 172.21.15.6:0/1970543236 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-24T23:28:39.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:39.769+0000 7f7566a00640 1 -- 172.21.15.6:0/1970543236 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2839569635 0 0) 0x7f75680c6c80 con 0x7f75680c4720 2024-04-24T23:28:39.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:39.769+0000 7f7566a00640 1 -- 172.21.15.6:0/1970543236 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7548003520 con 0x7f75680c4720 2024-04-24T23:28:39.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:39.769+0000 7f7567400640 1 -- 172.21.15.6:0/1970543236 >> v1:172.21.15.190:6789/0 conn(0x7f75680c4720 legacy=0x7f7568163320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:28:39.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:39.769+0000 7f7567400640 1 -- 172.21.15.6:0/1970543236 >> v1:172.21.15.6:6789/0 conn(0x7f75680c2f10 legacy=0x7f75680c4010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:28:39.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:39.769+0000 7f7567400640 1 -- 172.21.15.6:0/1970543236 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75680c6f60 con 0x7f7568152c60 2024-04-24T23:28:39.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:39.770+0000 7f7567400640 1 --2- 172.21.15.6:0/1970543236 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7568152c60 0x7f75681589a0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f75580097a0 tx=0x7f755802a2c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca1c7d8d007ed190 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:28:39.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:39.771+0000 7f7566a00640 1 -- 172.21.15.6:0/1970543236 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7558037070 con 0x7f7568152c60 2024-04-24T23:28:39.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:39.771+0000 7f7566a00640 1 -- 172.21.15.6:0/1970543236 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f7558032640 con 0x7f7568152c60 2024-04-24T23:28:39.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:39.771+0000 7f7566a00640 1 -- 172.21.15.6:0/1970543236 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75580329f0 con 0x7f7568152c60 2024-04-24T23:28:39.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:39.771+0000 7f756e600640 1 -- 172.21.15.6:0/1970543236 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7568152c60 msgr2=0x7f75681589a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:28:39.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:39.771+0000 7f756e600640 1 --2- 172.21.15.6:0/1970543236 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7568152c60 0x7f75681589a0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f75580097a0 tx=0x7f755802a2c0 comp rx=0 tx=0).stop 2024-04-24T23:28:39.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:39.771+0000 7f756e600640 1 -- 172.21.15.6:0/1970543236 shutdown_connections 2024-04-24T23:28:39.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:39.771+0000 7f756e600640 1 --2- 172.21.15.6:0/1970543236 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7568152c60 0x7f75681589a0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:28:39.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:39.771+0000 7f756e600640 1 -- 172.21.15.6:0/1970543236 >> 172.21.15.6:0/1970543236 conn(0x7f756806b0e0 msgr2=0x7f756806b4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:28:39.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:39.771+0000 7f756e600640 1 -- 172.21.15.6:0/1970543236 shutdown_connections 2024-04-24T23:28:39.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:39.772+0000 7f756e600640 1 -- 172.21.15.6:0/1970543236 wait complete. 2024-04-24T23:28:39.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:39.773+0000 7f756e600640 1 Processor -- start 2024-04-24T23:28:39.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:39.773+0000 7f756e600640 1 -- start start 2024-04-24T23:28:39.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:39.774+0000 7f756e600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f75680c2f10 0x7f75681f23f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:28:39.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:39.774+0000 7f756e600640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7568152c60 0x7f756814d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:28:39.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:39.774+0000 7f756e600640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7568154100 0x7f75681eeaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:28:39.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:39.774+0000 7f756e600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 -- 0x7f75680c7240 con 0x7f75680c2f10 2024-04-24T23:28:39.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:39.774+0000 7f756e600640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f756815cdb0 con 0x7f7568154100 2024-04-24T23:28:39.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:39.774+0000 7f756e600640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f75680c4bf0 con 0x7f7568152c60 2024-04-24T23:28:39.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:39.774+0000 7f7567400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f75680c2f10 0x7f75681f23f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:28:39.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:39.774+0000 7f755fe00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7568152c60 0x7f756814d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:28:39.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:39.774+0000 7f7567e00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7568154100 0x7f75681eeaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:28:39.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:39.774+0000 7f755fe00640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7568152c60 0x7f756814d350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.6:45780/0 (socket says 172.21.15.6:45780) 2024-04-24T23:28:39.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:39.774+0000 7f755fe00640 1 -- 172.21.15.6:0/2632412855 learned_addr learned my addr 172.21.15.6:0/2632412855 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-04-24T23:28:39.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:39.775+0000 7f7567e00640 1 -- 172.21.15.6:0/2632412855 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7568152c60 msgr2=0x7f756814d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:28:39.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:39.775+0000 7f7567e00640 1 --2- 172.21.15.6:0/2632412855 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7568152c60 0x7f756814d350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:28:39.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:39.775+0000 7f7567e00640 1 -- 172.21.15.6:0/2632412855 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f75680c2f10 msgr2=0x7f75681f23f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:28:39.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:39.775+0000 7f7567e00640 1 --2- 172.21.15.6:0/2632412855 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f75680c2f10 0x7f75681f23f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:28:39.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:39.775+0000 7f7567e00640 1 -- 172.21.15.6:0/2632412855 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7568158030 con 0x7f7568154100 2024-04-24T23:28:39.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:39.775+0000 7f755fe00640 1 --2- 172.21.15.6:0/2632412855 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7568152c60 0x7f756814d350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:28:39.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:39.775+0000 7f7567e00640 1 --2- 172.21.15.6:0/2632412855 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7568154100 0x7f75681eeaa0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f756000c6c0 tx=0x7f756000cb80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:28:39.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:39.775+0000 7f755f400640 1 -- 172.21.15.6:0/2632412855 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7560003b70 con 0x7f7568154100 2024-04-24T23:28:39.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:39.775+0000 7f755f400640 1 -- 172.21.15.6:0/2632412855 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f7560003d10 con 0x7f7568154100 2024-04-24T23:28:39.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:39.775+0000 7f755f400640 1 -- 172.21.15.6:0/2632412855 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7560005330 con 0x7f7568154100 2024-04-24T23:28:39.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:39.775+0000 7f756e600640 1 -- 172.21.15.6:0/2632412855 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7568158310 con 0x7f7568154100 2024-04-24T23:28:39.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:39.776+0000 7f756e600640 1 -- 172.21.15.6:0/2632412855 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75681ee580 con 0x7f7568154100 2024-04-24T23:28:39.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:39.777+0000 7f756e600640 1 -- 172.21.15.6:0/2632412855 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f75681ee780 con 0x7f7568154100 2024-04-24T23:28:39.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:39.777+0000 7f755f400640 1 -- 172.21.15.6:0/2632412855 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f75600054d0 con 0x7f7568154100 2024-04-24T23:28:39.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:39.778+0000 7f755f400640 1 --2- 172.21.15.6:0/2632412855 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f754406be30 0x7f754406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:28:39.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:39.778+0000 7f755f400640 1 -- 172.21.15.6:0/2632412855 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 6225+0+0 (secure 0 0 0) 0x7f756008afb0 con 0x7f7568154100 2024-04-24T23:28:39.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:39.779+0000 7f7567400640 1 --2- 172.21.15.6:0/2632412855 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f754406be30 0x7f754406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:28:39.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:39.779+0000 7f7567400640 1 --2- 172.21.15.6:0/2632412855 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f754406be30 0x7f754406e2f0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f7558034000 tx=0x7f755802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:28:39.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:39.783+0000 7f755f400640 1 -- 172.21.15.6:0/2632412855 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f75681ee780 con 0x7f7568154100 2024-04-24T23:28:39.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:39.983+0000 7f756e600640 1 -- 172.21.15.6:0/2632412855 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true} v 0) -- 0x7f7568075c10 con 0x7f7568154100 2024-04-24T23:28:40.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:40.565+0000 7f755f400640 1 -- 172.21.15.6:0/2632412855 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data_ec' removed v34) ==== 178+0+0 (secure 0 0 0) 0x7f7560058fd0 con 0x7f7568154100 2024-04-24T23:28:40.566 INFO:teuthology.orchestra.run.smithi006.stderr:pool 'cephfs_data_ec' removed 2024-04-24T23:28:40.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:40.568+0000 7f756e600640 1 -- 172.21.15.6:0/2632412855 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f754406be30 msgr2=0x7f754406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:28:40.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:40.568+0000 7f756e600640 1 --2- 172.21.15.6:0/2632412855 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f754406be30 0x7f754406e2f0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f7558034000 tx=0x7f755802e000 comp rx=0 tx=0).stop 2024-04-24T23:28:40.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:40.568+0000 7f756e600640 1 -- 172.21.15.6:0/2632412855 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7568154100 msgr2=0x7f75681eeaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:28:40.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:40.568+0000 7f756e600640 1 --2- 172.21.15.6:0/2632412855 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7568154100 0x7f75681eeaa0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f756000c6c0 tx=0x7f756000cb80 comp rx=0 tx=0).stop 2024-04-24T23:28:40.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:40.568+0000 7f756e600640 1 -- 172.21.15.6:0/2632412855 shutdown_connections 2024-04-24T23:28:40.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:40.568+0000 7f756e600640 1 --2- 172.21.15.6:0/2632412855 >> [v2:172.21.15.190:6832/806769663,v1:172.21.15.190:6833/806769663] conn(0x7f754406be30 0x7f754406e2f0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:28:40.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:40.568+0000 7f756e600640 1 --2- 172.21.15.6:0/2632412855 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7568152c60 0x7f756814d350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:28:40.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:40.568+0000 7f756e600640 1 --2- 172.21.15.6:0/2632412855 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7568154100 0x7f75681eeaa0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:28:40.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:40.568+0000 7f756e600640 1 --2- 172.21.15.6:0/2632412855 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f75680c2f10 0x7f75681f23f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:28:40.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:40.568+0000 7f756e600640 1 -- 172.21.15.6:0/2632412855 >> 172.21.15.6:0/2632412855 conn(0x7f756806b0e0 msgr2=0x7f75680c7700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:28:40.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:40.568+0000 7f756e600640 1 -- 172.21.15.6:0/2632412855 shutdown_connections 2024-04-24T23:28:40.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-24T23:28:40.568+0000 7f756e600640 1 -- 172.21.15.6:0/2632412855 wait complete. 2024-04-24T23:28:40.587 INFO:teuthology.misc:Shutting down mds daemons... 2024-04-24T23:28:40.588 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-24T23:28:40.588 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T23:28:40.614 INFO:tasks.ceph.mds.a:Stopped 2024-04-24T23:28:40.615 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-24T23:28:40.615 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T23:28:40.705 INFO:tasks.ceph.mds.c:Stopped 2024-04-24T23:28:40.706 DEBUG:tasks.ceph.mds.e:waiting for process to exit 2024-04-24T23:28:40.706 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T23:28:40.751 INFO:tasks.ceph.mds.e:Stopped 2024-04-24T23:28:40.752 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-24T23:28:40.752 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T23:28:40.765 INFO:tasks.ceph.mds.b:Stopped 2024-04-24T23:28:40.766 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-24T23:28:40.766 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T23:28:40.825 INFO:tasks.ceph.mds.d:Stopped 2024-04-24T23:28:40.825 DEBUG:tasks.ceph.mds.f:waiting for process to exit 2024-04-24T23:28:40.826 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T23:28:40.881 INFO:tasks.ceph.mds.f:Stopped 2024-04-24T23:28:40.881 INFO:teuthology.misc:Shutting down osd daemons... 2024-04-24T23:28:40.881 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-04-24T23:28:40.881 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T23:28:42.389 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~0s 2024-04-24T23:28:46.983 INFO:tasks.ceph.osd.0:Stopped 2024-04-24T23:28:46.984 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-04-24T23:28:46.984 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T23:28:48.499 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2024-04-24T23:28:48.500 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 has been restored 2024-04-24T23:28:53.086 INFO:tasks.ceph.osd.1:Stopped 2024-04-24T23:28:53.089 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-04-24T23:28:53.089 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T23:28:54.508 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2024-04-24T23:28:54.509 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 has been restored 2024-04-24T23:28:59.191 INFO:tasks.ceph.osd.2:Stopped 2024-04-24T23:28:59.191 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-04-24T23:28:59.191 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T23:29:00.416 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~0s 2024-04-24T23:29:00.416 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2024-04-24T23:29:05.293 INFO:tasks.ceph.osd.3:Stopped 2024-04-24T23:29:05.294 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-04-24T23:29:05.294 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T23:29:05.351 INFO:tasks.ceph.osd.4:Stopped 2024-04-24T23:29:05.351 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-04-24T23:29:05.351 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T23:29:05.421 INFO:tasks.ceph.osd.5:Stopped 2024-04-24T23:29:05.421 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-04-24T23:29:05.421 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T23:29:05.490 INFO:tasks.ceph.osd.6:Stopped 2024-04-24T23:29:05.490 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-04-24T23:29:05.490 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T23:29:05.563 INFO:tasks.ceph.osd.7:Stopped 2024-04-24T23:29:05.564 INFO:teuthology.misc:Shutting down mgr daemons... 2024-04-24T23:29:05.564 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-04-24T23:29:05.564 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T23:29:05.636 INFO:tasks.ceph.mgr.y:Stopped 2024-04-24T23:29:05.636 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-04-24T23:29:05.637 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T23:29:05.663 INFO:tasks.ceph.mgr.x:Stopped 2024-04-24T23:29:05.663 INFO:teuthology.misc:Shutting down mon daemons... 2024-04-24T23:29:05.663 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-04-24T23:29:05.663 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T23:29:05.695 INFO:tasks.ceph.mon.a:Stopped 2024-04-24T23:29:05.695 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-04-24T23:29:05.695 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T23:29:05.714 INFO:tasks.ceph.mon.b:Stopped 2024-04-24T23:29:05.714 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-04-24T23:29:05.714 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T23:29:05.765 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~0s 2024-04-24T23:29:05.765 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~0s 2024-04-24T23:29:05.765 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~0s 2024-04-24T23:29:05.765 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~0s 2024-04-24T23:29:05.765 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~0s 2024-04-24T23:29:05.765 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 has been restored 2024-04-24T23:29:05.765 INFO:tasks.ceph.mon.c:Stopped 2024-04-24T23:29:05.766 INFO:tasks.ceph:Checking cluster log for badness... 2024-04-24T23:29:05.766 DEBUG:teuthology.orchestra.run.smithi006:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'Replacing daemon mds' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-04-24T23:29:05.999 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi006.front.sepia.ceph.com 2024-04-24T23:29:06.000 DEBUG:teuthology.orchestra.run.smithi006:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-04-24T23:29:06.919 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi006.front.sepia.ceph.com 2024-04-24T23:29:06.919 DEBUG:teuthology.orchestra.run.smithi006:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-04-24T23:29:07.232 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi006.front.sepia.ceph.com 2024-04-24T23:29:07.233 DEBUG:teuthology.orchestra.run.smithi006:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-04-24T23:29:07.491 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi006.front.sepia.ceph.com 2024-04-24T23:29:07.492 DEBUG:teuthology.orchestra.run.smithi006:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-04-24T23:29:07.775 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi190.front.sepia.ceph.com 2024-04-24T23:29:07.775 DEBUG:teuthology.orchestra.run.smithi190:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-04-24T23:29:08.760 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi190.front.sepia.ceph.com 2024-04-24T23:29:08.760 DEBUG:teuthology.orchestra.run.smithi190:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-04-24T23:29:08.926 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi190.front.sepia.ceph.com 2024-04-24T23:29:08.926 DEBUG:teuthology.orchestra.run.smithi190:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-04-24T23:29:09.053 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi190.front.sepia.ceph.com 2024-04-24T23:29:09.053 DEBUG:teuthology.orchestra.run.smithi190:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-04-24T23:29:09.191 INFO:tasks.ceph:Archiving mon data... 2024-04-24T23:29:09.193 DEBUG:teuthology.misc:Transferring archived files from smithi006:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672187/data/mon.a.tgz 2024-04-24T23:29:09.193 DEBUG:teuthology.orchestra.run.smithi006:> mktemp 2024-04-24T23:29:09.295 INFO:teuthology.orchestra.run.smithi006.stdout:/tmp/tmp.mbD5EghZWI 2024-04-24T23:29:09.295 DEBUG:teuthology.orchestra.run.smithi006:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.mbD5EghZWI 2024-04-24T23:29:09.539 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 0666 /tmp/tmp.mbD5EghZWI 2024-04-24T23:29:09.796 DEBUG:teuthology.orchestra.remote:smithi006:/tmp/tmp.mbD5EghZWI is 363KB 2024-04-24T23:29:09.906 DEBUG:teuthology.orchestra.run.smithi006:> rm -fr /tmp/tmp.mbD5EghZWI 2024-04-24T23:29:09.960 DEBUG:teuthology.misc:Transferring archived files from smithi190:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672187/data/mon.b.tgz 2024-04-24T23:29:09.961 DEBUG:teuthology.orchestra.run.smithi190:> mktemp 2024-04-24T23:29:09.981 INFO:teuthology.orchestra.run.smithi190.stdout:/tmp/tmp.u4W6s9addt 2024-04-24T23:29:09.982 DEBUG:teuthology.orchestra.run.smithi190:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.u4W6s9addt 2024-04-24T23:29:10.184 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod 0666 /tmp/tmp.u4W6s9addt 2024-04-24T23:29:10.292 DEBUG:teuthology.orchestra.remote:smithi190:/tmp/tmp.u4W6s9addt is 370KB 2024-04-24T23:29:10.387 DEBUG:teuthology.orchestra.run.smithi190:> rm -fr /tmp/tmp.u4W6s9addt 2024-04-24T23:29:10.401 DEBUG:teuthology.misc:Transferring archived files from smithi190:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672187/data/mon.c.tgz 2024-04-24T23:29:10.401 DEBUG:teuthology.orchestra.run.smithi190:> mktemp 2024-04-24T23:29:10.455 INFO:teuthology.orchestra.run.smithi190.stdout:/tmp/tmp.5Qc17BZr8A 2024-04-24T23:29:10.455 DEBUG:teuthology.orchestra.run.smithi190:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.5Qc17BZr8A 2024-04-24T23:29:10.588 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod 0666 /tmp/tmp.5Qc17BZr8A 2024-04-24T23:29:10.667 DEBUG:teuthology.orchestra.remote:smithi190:/tmp/tmp.5Qc17BZr8A is 370KB 2024-04-24T23:29:10.764 DEBUG:teuthology.orchestra.run.smithi190:> rm -fr /tmp/tmp.5Qc17BZr8A 2024-04-24T23:29:10.766 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 has been restored 2024-04-24T23:29:10.766 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 has been restored 2024-04-24T23:29:10.766 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 has been restored 2024-04-24T23:29:10.766 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b has been restored 2024-04-24T23:29:10.766 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c has been restored 2024-04-24T23:29:10.779 INFO:tasks.ceph:Cleaning ceph cluster... 2024-04-24T23:29:10.779 DEBUG:teuthology.orchestra.run.smithi006:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-04-24T23:29:10.787 DEBUG:teuthology.orchestra.run.smithi190:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-04-24T23:29:10.963 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-04-24T23:29:10.992 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-04-24T23:29:10.992 INFO:tasks.ceph:Archiving crash dumps... 2024-04-24T23:29:10.993 DEBUG:teuthology.misc:Transferring archived files from smithi006:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672187/remote/smithi006/crash 2024-04-24T23:29:10.994 DEBUG:teuthology.orchestra.run.smithi006:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-24T23:29:11.072 DEBUG:teuthology.misc:Transferring archived files from smithi190:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672187/remote/smithi190/crash 2024-04-24T23:29:11.073 DEBUG:teuthology.orchestra.run.smithi190:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-24T23:29:11.095 INFO:tasks.ceph:Shutting down logrotate 2024-04-24T23:32:06.481 DEBUG:teuthology.orchestra.run.smithi006:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-04-24T23:32:06.586 DEBUG:teuthology.orchestra.run.smithi190:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-04-24T23:32:06.653 INFO:tasks.ceph:Compressing logs... 2024-04-24T23:32:06.653 DEBUG:teuthology.orchestra.run.smithi006:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-24T23:32:06.660 DEBUG:teuthology.orchestra.run.smithi190:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-24T23:32:06.707 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-04-24T23:32:06.707 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.f.log 2024-04-24T23:32:06.707 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-04-24T23:32:06.707 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph.log: /var/log/ceph/ceph-mds.f.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-04-24T23:32:06.707 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.b.log 2024-04-24T23:32:06.707 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-04-24T23:32:06.711 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-04-24T23:32:06.712 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34640.log 2024-04-24T23:32:06.712 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34600.log 2024-04-24T23:32:06.712 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.34640.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34349.log 2024-04-24T23:32:06.712 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.34600.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34438.log 2024-04-24T23:32:06.712 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.34349.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34389.log 2024-04-24T23:32:06.712 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.34438.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-04-24T23:32:06.712 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.34389.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34560.log 2024-04-24T23:32:06.712 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-04-24T23:32:06.712 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.34560.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-04-24T23:32:06.712 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34480.log 2024-04-24T23:32:06.712 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34520.log 2024-04-24T23:32:06.713 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.34480.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.d.log 2024-04-24T23:32:06.713 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.34520.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-04-24T23:32:06.713 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-mds.d.log: /var/log/ceph/ceph-mon.b.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.34600.log.gz 2024-04-24T23:32:06.714 INFO:teuthology.orchestra.run.smithi190.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.34640.log.gz 2024-04-24T23:32:06.717 INFO:teuthology.orchestra.run.smithi190.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.34349.log.gz 2024-04-24T23:32:06.718 INFO:teuthology.orchestra.run.smithi190.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.34389.log.gz 2024-04-24T23:32:06.719 INFO:teuthology.orchestra.run.smithi190.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34438.log.gz 2024-04-24T23:32:06.721 INFO:teuthology.orchestra.run.smithi190.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.34480.log.gz 2024-04-24T23:32:06.721 INFO:teuthology.orchestra.run.smithi190.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.34520.log.gz 2024-04-24T23:32:06.721 INFO:teuthology.orchestra.run.smithi190.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34560.log.gz 2024-04-24T23:32:06.770 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44913.log 2024-04-24T23:32:06.772 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53874.log 2024-04-24T23:32:06.773 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.44913.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49647.log 2024-04-24T23:32:06.775 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.53874.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44451.log 2024-04-24T23:32:06.775 INFO:teuthology.orchestra.run.smithi006.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.53874.log.gz 2024-04-24T23:32:06.776 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.49647.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38930.log 2024-04-24T23:32:06.777 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.44451.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51751.log 2024-04-24T23:32:06.779 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.38930.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48875.log 2024-04-24T23:32:06.780 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.51751.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48833.log 2024-04-24T23:32:06.781 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.48875.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43165.log 2024-04-24T23:32:06.783 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.48833.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43207.log 2024-04-24T23:32:06.783 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.43165.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.44913.log.gz 2024-04-24T23:32:06.784 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43021.log 2024-04-24T23:32:06.785 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.43207.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55112.log 2024-04-24T23:32:06.787 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.43021.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50543.log 2024-04-24T23:32:06.788 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.55112.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.55112.log.gz 2024-04-24T23:32:06.788 INFO:teuthology.orchestra.run.smithi006.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.38930.log.gz 2024-04-24T23:32:06.788 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40368.log 2024-04-24T23:32:06.789 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.50543.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43123.log 2024-04-24T23:32:06.790 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.40368.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51793.log 2024-04-24T23:32:06.791 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.43123.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41278.log 2024-04-24T23:32:06.792 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.51793.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43534.log 2024-04-24T23:32:06.794 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.41278.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48611.log 2024-04-24T23:32:06.794 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.43534.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42851.log 2024-04-24T23:32:06.795 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.48611.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41567.log 2024-04-24T23:32:06.796 INFO:teuthology.orchestra.run.smithi006.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.50543.log.gz 2024-04-24T23:32:06.796 INFO:teuthology.orchestra.run.smithi006.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.51751.log.gz 2024-04-24T23:32:06.797 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.42851.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54300.log 2024-04-24T23:32:06.797 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.41567.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44661.log 2024-04-24T23:32:06.799 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.54300.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40452.log 2024-04-24T23:32:06.799 INFO:teuthology.orchestra.run.smithi006.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.44451.log.gz 2024-04-24T23:32:06.799 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.44661.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.54300.log.gz 2024-04-24T23:32:06.800 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48936.log 2024-04-24T23:32:06.800 INFO:teuthology.orchestra.run.smithi006.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.40368.log.gz 2024-04-24T23:32:06.800 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.40452.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.41278.log.gz 2024-04-24T23:32:06.801 INFO:teuthology.orchestra.run.smithi006.stderr:gzip 88.2% -- replaced with /var/log/ceph/ceph-client.admin.41567.log.gz 2024-04-24T23:32:06.801 INFO:teuthology.orchestra.run.smithi006.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.44493.log 2024-04-24T23:32:06.802 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.48936.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52860.log 2024-04-24T23:32:06.803 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.44493.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56202.log 2024-04-24T23:32:06.804 INFO:teuthology.orchestra.run.smithi006.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.42851.log.gz 2024-04-24T23:32:06.805 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.52860.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.43021.log.gz 2024-04-24T23:32:06.805 INFO:teuthology.orchestra.run.smithi006.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.43165.log.gz 2024-04-24T23:32:06.805 INFO:teuthology.orchestra.run.smithi006.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.43123.log.gz 2024-04-24T23:32:06.805 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47793.log 2024-04-24T23:32:06.805 INFO:teuthology.orchestra.run.smithi006.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.43207.log.gz 2024-04-24T23:32:06.806 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.56202.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38843.log 2024-04-24T23:32:06.806 INFO:teuthology.orchestra.run.smithi006.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.43534.log.gz 2024-04-24T23:32:06.806 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.47793.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.56202.log.gz 2024-04-24T23:32:06.807 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49707.log 2024-04-24T23:32:06.807 INFO:teuthology.orchestra.run.smithi006.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.44493.log.gz 2024-04-24T23:32:06.807 INFO:teuthology.orchestra.run.smithi006.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.44661.log.gz 2024-04-24T23:32:06.808 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.38843.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54426.log 2024-04-24T23:32:06.809 INFO:teuthology.orchestra.run.smithi006.stderr: 88.4%/var/log/ceph/ceph-client.admin.49707.log: -- replaced with /var/log/ceph/ceph-client.admin.38843.log.gz 2024-04-24T23:32:06.809 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53452.log 2024-04-24T23:32:06.810 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.54426.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-04-24T23:32:06.810 INFO:teuthology.orchestra.run.smithi006.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.54426.log.gz 2024-04-24T23:32:06.811 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.53452.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.40452.log.gz 2024-04-24T23:32:06.811 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43333.log 2024-04-24T23:32:06.812 INFO:teuthology.orchestra.run.smithi006.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.53452.log.gz 2024-04-24T23:32:06.812 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53578.log 2024-04-24T23:32:06.813 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.43333.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45039.log 2024-04-24T23:32:06.815 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.53578.log: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.43333.log.gz 2024-04-24T23:32:06.815 INFO:teuthology.orchestra.run.smithi006.stderr:gzip 86.2% -- replaced with /var/log/ceph/ceph.log.gz -5 2024-04-24T23:32:06.815 INFO:teuthology.orchestra.run.smithi006.stderr: --verbose -- /var/log/ceph/ceph-client.admin.52335.log 2024-04-24T23:32:06.815 INFO:teuthology.orchestra.run.smithi006.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.53578.log.gz 2024-04-24T23:32:06.816 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.45039.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48341.log 2024-04-24T23:32:06.816 INFO:teuthology.orchestra.run.smithi006.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.49647.log.gz 2024-04-24T23:32:06.816 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.52335.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.45039.log.gz 2024-04-24T23:32:06.816 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41362.log 2024-04-24T23:32:06.817 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.48341.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52629.log 2024-04-24T23:32:06.818 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.41362.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52503.log 2024-04-24T23:32:06.819 INFO:teuthology.orchestra.run.smithi006.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.41362.log.gz 2024-04-24T23:32:06.819 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.52629.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49082.log 2024-04-24T23:32:06.820 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.52503.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51313.log 2024-04-24T23:32:06.821 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.49082.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53368.log 2024-04-24T23:32:06.822 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.51313.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52275.log 2024-04-24T23:32:06.823 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.53368.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50230.log 2024-04-24T23:32:06.823 INFO:teuthology.orchestra.run.smithi006.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.53368.log.gz 2024-04-24T23:32:06.825 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.52275.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41889.log 2024-04-24T23:32:06.825 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.50230.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52986.log 2024-04-24T23:32:06.826 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.41889.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-04-24T23:32:06.827 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.52986.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56328.log 2024-04-24T23:32:06.827 INFO:teuthology.orchestra.run.smithi006.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.52335.log.gz 2024-04-24T23:32:06.827 INFO:teuthology.orchestra.run.smithi006.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.52629.log.gz 2024-04-24T23:32:06.829 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39551.log 2024-04-24T23:32:06.829 INFO:teuthology.orchestra.run.smithi006.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.52275.log.gz 2024-04-24T23:32:06.829 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.56328.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41584.log 2024-04-24T23:32:06.829 INFO:teuthology.orchestra.run.smithi006.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.56328.log.gz 2024-04-24T23:32:06.829 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.39551.log: 87.3% 87.7% -- replaced with /var/log/ceph/ceph-client.admin.48875.log.gz 2024-04-24T23:32:06.830 INFO:teuthology.orchestra.run.smithi006.stderr: -- replaced with /var/log/ceph/ceph-client.admin.48936.log.gz 2024-04-24T23:32:06.831 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51066.log 2024-04-24T23:32:06.831 INFO:teuthology.orchestra.run.smithi006.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.49707.log.gz 2024-04-24T23:32:06.831 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.41584.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39032.log 2024-04-24T23:32:06.832 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.51066.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51541.log 2024-04-24T23:32:06.833 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.39032.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42807.log 2024-04-24T23:32:06.833 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.51541.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44199.log 2024-04-24T23:32:06.834 INFO:teuthology.orchestra.run.smithi006.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.50230.log.gz 2024-04-24T23:32:06.835 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.42807.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50940.log 2024-04-24T23:32:06.835 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.44199.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40326.log 2024-04-24T23:32:06.836 INFO:teuthology.orchestra.run.smithi006.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.52986.log.gz 2024-04-24T23:32:06.836 INFO:teuthology.orchestra.run.smithi006.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.39032.log.gz 2024-04-24T23:32:06.836 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.50940.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42580.log 2024-04-24T23:32:06.837 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.40326.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.44066.log 2024-04-24T23:32:06.837 INFO:teuthology.orchestra.run.smithi006.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.42807.log.gz 2024-04-24T23:32:06.838 INFO:teuthology.orchestra.run.smithi006.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.39551.log.gz 2024-04-24T23:32:06.838 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.42580.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43063.log 2024-04-24T23:32:06.839 INFO:teuthology.orchestra.run.smithi006.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.42580.log.gz 2024-04-24T23:32:06.839 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.0.44066.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.44066.log.gz 2024-04-24T23:32:06.839 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54131.log 2024-04-24T23:32:06.840 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.43063.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54762.log 2024-04-24T23:32:06.841 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.54131.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42894.log 2024-04-24T23:32:06.842 INFO:teuthology.orchestra.run.smithi006.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.43063.log.gz 2024-04-24T23:32:06.842 INFO:teuthology.orchestra.run.smithi006.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.54131.log.gz 2024-04-24T23:32:06.842 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.54762.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-04-24T23:32:06.842 INFO:teuthology.orchestra.run.smithi006.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.54762.log.gz 2024-04-24T23:32:06.843 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.42894.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54258.log 2024-04-24T23:32:06.843 INFO:teuthology.orchestra.run.smithi006.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.41584.log.gz 2024-04-24T23:32:06.843 INFO:teuthology.orchestra.run.smithi006.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.41889.log.gz 2024-04-24T23:32:06.844 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54888.log 2024-04-24T23:32:06.845 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.54258.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.42894.log.gz 2024-04-24T23:32:06.845 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50754.log 2024-04-24T23:32:06.845 INFO:teuthology.orchestra.run.smithi006.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.54258.log.gz 2024-04-24T23:32:06.846 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.54888.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50856.log 2024-04-24T23:32:06.846 INFO:teuthology.orchestra.run.smithi006.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.54888.log.gz 2024-04-24T23:32:06.846 INFO:teuthology.orchestra.run.smithi006.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.48833.log.gz 2024-04-24T23:32:06.847 INFO:teuthology.orchestra.run.smithi006.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.48611.log.gz 2024-04-24T23:32:06.847 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.50754.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39425.log 2024-04-24T23:32:06.848 INFO:teuthology.orchestra.run.smithi006.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.50940.log.gz 2024-04-24T23:32:06.848 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.50856.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55700.log 2024-04-24T23:32:06.849 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.39425.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41893.log 2024-04-24T23:32:06.849 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.55700.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54846.log 2024-04-24T23:32:06.850 INFO:teuthology.orchestra.run.smithi006.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.55700.log.gz 2024-04-24T23:32:06.851 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.41893.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52191.log 2024-04-24T23:32:06.851 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.54846.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39663.log 2024-04-24T23:32:06.851 INFO:teuthology.orchestra.run.smithi006.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.54846.log.gz 2024-04-24T23:32:06.853 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.52191.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.47793.log.gz 2024-04-24T23:32:06.853 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38885.log 2024-04-24T23:32:06.853 INFO:teuthology.orchestra.run.smithi006.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.48341.log.gz 2024-04-24T23:32:06.853 INFO:teuthology.orchestra.run.smithi006.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.50754.log.gz 2024-04-24T23:32:06.854 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.39663.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42496.log 2024-04-24T23:32:06.854 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.38885.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.40326.log.gz 2024-04-24T23:32:06.855 INFO:teuthology.orchestra.run.smithi006.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.39663.log.gz 2024-04-24T23:32:06.855 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44871.log 2024-04-24T23:32:06.855 INFO:teuthology.orchestra.run.smithi006.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.41893.log.gz 2024-04-24T23:32:06.856 INFO:teuthology.orchestra.run.smithi006.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.38885.log.gz 2024-04-24T23:32:06.856 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.42496.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48124.log 2024-04-24T23:32:06.856 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.44871.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.42496.log.gz 2024-04-24T23:32:06.856 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56034.log 2024-04-24T23:32:06.857 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.48124.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41899.log 2024-04-24T23:32:06.858 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.56034.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53032.log 2024-04-24T23:32:06.858 INFO:teuthology.orchestra.run.smithi006.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.56034.log.gz 2024-04-24T23:32:06.858 INFO:teuthology.orchestra.run.smithi006.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.48124.log.gz 2024-04-24T23:32:06.859 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.41899.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39298.log 2024-04-24T23:32:06.860 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.53032.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56286.log 2024-04-24T23:32:06.860 INFO:teuthology.orchestra.run.smithi006.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.50856.log.gz 2024-04-24T23:32:06.860 INFO:teuthology.orchestra.run.smithi006.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.51066.log.gz 2024-04-24T23:32:06.861 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.39298.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51878.log 2024-04-24T23:32:06.862 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.56286.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54720.log 2024-04-24T23:32:06.862 INFO:teuthology.orchestra.run.smithi006.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.56286.log.gz 2024-04-24T23:32:06.863 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.51878.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43468.log 2024-04-24T23:32:06.864 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.54720.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-04-24T23:32:06.864 INFO:teuthology.orchestra.run.smithi006.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.54720.log.gz 2024-04-24T23:32:06.865 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55826.log 2024-04-24T23:32:06.866 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41320.log 2024-04-24T23:32:06.867 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54384.log 2024-04-24T23:32:06.867 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40494.log 2024-04-24T23:32:06.868 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40720.log 2024-04-24T23:32:06.869 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54586.log 2024-04-24T23:32:06.870 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44997.log 2024-04-24T23:32:06.871 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52902.log 2024-04-24T23:32:06.871 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42225.log 2024-04-24T23:32:06.872 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49001.log 2024-04-24T23:32:06.873 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54804.log 2024-04-24T23:32:06.874 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52587.log 2024-04-24T23:32:06.875 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50314.log 2024-04-24T23:32:06.876 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47310.log 2024-04-24T23:32:06.877 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49833.log 2024-04-24T23:32:06.878 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55154.log 2024-04-24T23:32:06.879 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.c.log 2024-04-24T23:32:06.880 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54342.log 2024-04-24T23:32:06.880 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54494.log 2024-04-24T23:32:06.881 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38744.log 2024-04-24T23:32:06.886 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54678.log 2024-04-24T23:32:06.887 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55406.log 2024-04-24T23:32:06.887 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49435.log 2024-04-24T23:32:06.887 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44703.log 2024-04-24T23:32:06.887 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53494.log 2024-04-24T23:32:06.887 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50982.log 2024-04-24T23:32:06.888 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42234.log 2024-04-24T23:32:06.889 INFO:teuthology.orchestra.run.smithi190.stderr: 93.8% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-04-24T23:32:06.889 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.50982.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33493.log/var/log/ceph/ceph-client.admin.43468.log: 2024-04-24T23:32:06.889 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-osd.3.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.44871.log.gz 2024-04-24T23:32:06.892 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.55826.log: /var/log/ceph/ceph-client.admin.41320.log: /var/log/ceph/ceph-client.admin.54384.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.41899.log.gz 2024-04-24T23:32:06.892 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.40494.log: /var/log/ceph/ceph-client.admin.40720.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.55826.log.gz 2024-04-24T23:32:06.892 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.42234.log: /var/log/ceph/ceph-client.admin.54586.log: /var/log/ceph/ceph-client.admin.44997.log: 89.3% 88.0% -- replaced with /var/log/ceph/ceph-client.admin.51313.log.gz 2024-04-24T23:32:06.892 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39162.log 2024-04-24T23:32:06.892 INFO:teuthology.orchestra.run.smithi006.stderr: -- replaced with /var/log/ceph/ceph-client.admin.54384.log.gz 2024-04-24T23:32:06.892 INFO:teuthology.orchestra.run.smithi006.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.54586.log.gz 2024-04-24T23:32:06.893 INFO:teuthology.orchestra.run.smithi006.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.50982.log.gz 2024-04-24T23:32:06.893 INFO:teuthology.orchestra.run.smithi006.stderr: 86.9% 87.7% -- replaced with /var/log/ceph/ceph-client.admin.41320.log.gz 2024-04-24T23:32:06.893 INFO:teuthology.orchestra.run.smithi006.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.51793.log.gz 2024-04-24T23:32:06.893 INFO:teuthology.orchestra.run.smithi006.stderr: -- replaced with /var/log/ceph/ceph-client.admin.51541.log.gz/var/log/ceph/ceph-client.admin.33493.log: 2024-04-24T23:32:06.898 INFO:teuthology.orchestra.run.smithi006.stderr: 87.6% 90.9% 87.6% -- replaced with /var/log/ceph/ceph-client.admin.42234.log.gz -- replaced with /var/log/ceph/ceph-client.admin.43468.log.gz -- replaced with /var/log/ceph/ceph-client.admin.44997.log.gz 2024-04-24T23:32:06.898 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-24T23:32:06.898 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-24T23:32:06.899 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5/var/log/ceph/ceph-client.admin.52902.log: --verbose -- /var/log/ceph/ceph-osd.1.log 2024-04-24T23:32:06.899 INFO:teuthology.orchestra.run.smithi006.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.52503.log.gz 2024-04-24T23:32:06.899 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.42225.log: /var/log/ceph/ceph-client.admin.49001.log: /var/log/ceph/ceph-client.admin.54804.log: /var/log/ceph/ceph-client.admin.52587.log: /var/log/ceph/ceph-client.admin.50314.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.52860.log.gz 2024-04-24T23:32:06.899 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.47310.log: /var/log/ceph/ceph-client.admin.49833.log: /var/log/ceph/ceph-client.admin.39162.log: /var/log/ceph/ceph-client.admin.55154.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.54804.log.gz 2024-04-24T23:32:06.899 INFO:teuthology.orchestra.run.smithi006.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.49001.log.gz 2024-04-24T23:32:06.899 INFO:teuthology.orchestra.run.smithi006.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.39298.log.gz 2024-04-24T23:32:06.899 INFO:teuthology.orchestra.run.smithi006.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.39425.log.gz 2024-04-24T23:32:06.899 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 89.0% --verbose -- replaced with /var/log/ceph/ceph-client.admin.42225.log.gz -- 2024-04-24T23:32:06.899 INFO:teuthology.orchestra.run.smithi006.stderr: /var/log/ceph/ceph-client.admin.47548.log 2024-04-24T23:32:06.899 INFO:teuthology.orchestra.run.smithi006.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.55154.log.gz 2024-04-24T23:32:06.899 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-mds.c.log: /var/log/ceph/ceph-client.admin.54342.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.39162.log.gz 2024-04-24T23:32:06.899 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-osd.1.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.50314.log.gz 2024-04-24T23:32:06.899 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.54494.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.51878.log.gz 2024-04-24T23:32:06.899 INFO:teuthology.orchestra.run.smithi006.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.52587.log.gz 2024-04-24T23:32:06.900 INFO:teuthology.orchestra.run.smithi006.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.52191.log.gz 2024-04-24T23:32:06.900 INFO:teuthology.orchestra.run.smithi006.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.53032.log.gz 2024-04-24T23:32:06.900 INFO:teuthology.orchestra.run.smithi006.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.54342.log.gz 2024-04-24T23:32:06.900 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-04-24T23:32:06.900 INFO:teuthology.orchestra.run.smithi006.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.54494.log.gz 2024-04-24T23:32:06.900 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.38744.log: /var/log/ceph/ceph-client.admin.54678.log: /var/log/ceph/ceph-client.admin.55406.log: 88.9%/var/log/ceph/ceph-client.admin.49435.log: -- replaced with /var/log/ceph/ceph-client.admin.49082.log.gz 2024-04-24T23:32:06.900 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.44703.log: 89.1% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-04-24T23:32:06.900 INFO:teuthology.orchestra.run.smithi006.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.54678.log.gz 2024-04-24T23:32:06.900 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.53494.log: /var/log/ceph/ceph-client.admin.47548.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.55406.log.gz 2024-04-24T23:32:06.900 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53664.log 2024-04-24T23:32:06.900 INFO:teuthology.orchestra.run.smithi006.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.53494.log.gz 2024-04-24T23:32:06.900 INFO:teuthology.orchestra.run.smithi006.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.38744.log.gz 2024-04-24T23:32:06.900 INFO:teuthology.orchestra.run.smithi006.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.49435.log.gz 2024-04-24T23:32:06.901 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52419.log 2024-04-24T23:32:06.901 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.53664.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40942.log 2024-04-24T23:32:06.902 INFO:teuthology.orchestra.run.smithi006.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.53664.log.gz 2024-04-24T23:32:06.903 INFO:teuthology.orchestra.run.smithi006.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.49833.log.gz 2024-04-24T23:32:06.903 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52377.log 2024-04-24T23:32:06.903 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.52419.log: /var/log/ceph/ceph-client.admin.40942.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-04-24T23:32:06.903 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.52377.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55020.log 2024-04-24T23:32:06.903 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40100.log 2024-04-24T23:32:06.904 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.55020.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43402.log 2024-04-24T23:32:06.904 INFO:teuthology.orchestra.run.smithi006.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.55020.log.gz 2024-04-24T23:32:06.904 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.40100.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40762.log 2024-04-24T23:32:06.905 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.43402.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53706.log 2024-04-24T23:32:06.906 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.40762.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53200.log 2024-04-24T23:32:06.907 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.53706.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54000.log 2024-04-24T23:32:06.907 INFO:teuthology.orchestra.run.smithi006.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.53706.log.gz 2024-04-24T23:32:06.907 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.53200.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53790.log 2024-04-24T23:32:06.908 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.54000.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48744.log 2024-04-24T23:32:06.909 INFO:teuthology.orchestra.run.smithi006.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.54000.log.gz 2024-04-24T23:32:06.910 INFO:teuthology.orchestra.run.smithi006.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.43402.log.gz 2024-04-24T23:32:06.910 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.53790.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.40762.log.gz 2024-04-24T23:32:06.910 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41573.log 2024-04-24T23:32:06.910 INFO:teuthology.orchestra.run.smithi006.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.53200.log.gz 2024-04-24T23:32:06.910 INFO:teuthology.orchestra.run.smithi006.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.53790.log.gz 2024-04-24T23:32:06.911 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.48744.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.44047.log 2024-04-24T23:32:06.911 INFO:teuthology.orchestra.run.smithi006.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.40494.log.gz 2024-04-24T23:32:06.912 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.41573.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53326.log 2024-04-24T23:32:06.912 INFO:teuthology.orchestra.run.smithi006.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.40720.log.gz 2024-04-24T23:32:06.912 INFO:teuthology.orchestra.run.smithi006.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.48744.log.gz 2024-04-24T23:32:06.912 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.0.44047.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44535.log 2024-04-24T23:32:06.912 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.44047.log.gz 2024-04-24T23:32:06.913 INFO:teuthology.orchestra.run.smithi006.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.41573.log.gz 2024-04-24T23:32:06.913 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.53326.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45380.log 2024-04-24T23:32:06.913 INFO:teuthology.orchestra.run.smithi006.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.53326.log.gz 2024-04-24T23:32:06.914 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.44535.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52713.log 2024-04-24T23:32:06.914 INFO:teuthology.orchestra.run.smithi006.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.44199.log.gz 2024-04-24T23:32:06.915 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.45380.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51229.log 2024-04-24T23:32:06.916 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.52713.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42227.log 2024-04-24T23:32:06.916 INFO:teuthology.orchestra.run.smithi006.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.44703.log.gz 2024-04-24T23:32:06.916 INFO:teuthology.orchestra.run.smithi006.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.44535.log.gz 2024-04-24T23:32:06.917 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.51229.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50398.log 2024-04-24T23:32:06.918 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.42227.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53536.log 2024-04-24T23:32:06.919 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.50398.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.52419.log.gz 2024-04-24T23:32:06.919 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56244.log 2024-04-24T23:32:06.919 INFO:teuthology.orchestra.run.smithi006.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.52713.log.gz 2024-04-24T23:32:06.919 INFO:teuthology.orchestra.run.smithi006.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.52902.log.gz 2024-04-24T23:32:06.919 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.53536.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.42227.log.gz 2024-04-24T23:32:06.920 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49791.log 2024-04-24T23:32:06.920 INFO:teuthology.orchestra.run.smithi006.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.53536.log.gz 2024-04-24T23:32:06.920 INFO:teuthology.orchestra.run.smithi006.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.45380.log.gz 2024-04-24T23:32:06.921 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.56244.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53284.log 2024-04-24T23:32:06.921 INFO:teuthology.orchestra.run.smithi006.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.47310.log.gz 2024-04-24T23:32:06.921 INFO:teuthology.orchestra.run.smithi006.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.47548.log.gz 2024-04-24T23:32:06.922 INFO:teuthology.orchestra.run.smithi006.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.56244.log.gz 2024-04-24T23:32:06.922 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.49791.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51625.log 2024-04-24T23:32:06.922 INFO:teuthology.orchestra.run.smithi006.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.50398.log.gz 2024-04-24T23:32:06.923 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.53284.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52233.log 2024-04-24T23:32:06.924 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.51625.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38990.log 2024-04-24T23:32:06.924 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.52233.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55238.log 2024-04-24T23:32:06.925 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.38990.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50356.log 2024-04-24T23:32:06.926 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.55238.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41895.log 2024-04-24T23:32:06.926 INFO:teuthology.orchestra.run.smithi006.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.55238.log.gz 2024-04-24T23:32:06.927 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.50356.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55490.log 2024-04-24T23:32:06.927 INFO:teuthology.orchestra.run.smithi006.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.53284.log.gz 2024-04-24T23:32:06.928 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.41895.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51481.log 2024-04-24T23:32:06.929 INFO:teuthology.orchestra.run.smithi006.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.51625.log.gz 2024-04-24T23:32:06.929 INFO:teuthology.orchestra.run.smithi006.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.52233.log.gz 2024-04-24T23:32:06.929 INFO:teuthology.orchestra.run.smithi006.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.38990.log.gz 2024-04-24T23:32:06.929 INFO:teuthology.orchestra.run.smithi006.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.50356.log.gz 2024-04-24T23:32:06.930 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.55490.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49226.log 2024-04-24T23:32:06.930 INFO:teuthology.orchestra.run.smithi006.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.55490.log.gz 2024-04-24T23:32:06.930 INFO:teuthology.orchestra.run.smithi006.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.41895.log.gz 2024-04-24T23:32:06.931 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.51481.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53074.log 2024-04-24T23:32:06.931 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.49226.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.51481.log.gz 2024-04-24T23:32:06.931 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-04-24T23:32:06.932 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.53074.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33390.log 2024-04-24T23:32:06.932 INFO:teuthology.orchestra.run.smithi006.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.49226.log.gz 2024-04-24T23:32:06.932 INFO:teuthology.orchestra.run.smithi006.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.49791.log.gz 2024-04-24T23:32:06.933 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51187.log 2024-04-24T23:32:06.934 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.33390.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42454.log 2024-04-24T23:32:06.935 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.51187.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51397.log 2024-04-24T23:32:06.936 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.42454.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44955.log 2024-04-24T23:32:06.936 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.51397.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41068.log 2024-04-24T23:32:06.937 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.44955.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55574.log 2024-04-24T23:32:06.938 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.41068.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.25262.log 2024-04-24T23:32:06.939 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.55574.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33125.log 2024-04-24T23:32:06.940 INFO:teuthology.orchestra.run.smithi006.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.55574.log.gz 2024-04-24T23:32:06.940 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph.tmp-client.admin.25262.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.51187.log.gz 2024-04-24T23:32:06.940 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.25262.log.gz 2024-04-24T23:32:06.940 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50626.log 2024-04-24T23:32:06.941 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.33125.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.40942.log.gz 2024-04-24T23:32:06.941 INFO:teuthology.orchestra.run.smithi006.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.42454.log.gz 2024-04-24T23:32:06.941 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55742.log 88.3% 2024-04-24T23:32:06.942 INFO:teuthology.orchestra.run.smithi006.stderr: -- replaced with /var/log/ceph/ceph-client.admin.51397.log.gz 2024-04-24T23:32:06.942 INFO:teuthology.orchestra.run.smithi006.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.44955.log.gz 2024-04-24T23:32:06.942 INFO:teuthology.orchestra.run.smithi006.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.41068.log.gz 2024-04-24T23:32:06.943 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.50626.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44787.log 2024-04-24T23:32:06.943 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.55742.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52461.log 2024-04-24T23:32:06.943 INFO:teuthology.orchestra.run.smithi006.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.55742.log.gz 2024-04-24T23:32:06.944 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.44787.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51583.log 2024-04-24T23:32:06.944 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.52461.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50272.log 2024-04-24T23:32:06.945 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.51583.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.33390.log.gz 2024-04-24T23:32:06.946 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49749.log 2024-04-24T23:32:06.946 INFO:teuthology.orchestra.run.smithi006.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.33493.log.gz 2024-04-24T23:32:06.946 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.50272.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.50626.log.gz 2024-04-24T23:32:06.946 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51145.log 2024-04-24T23:32:06.948 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.49749.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50440.log 2024-04-24T23:32:06.948 INFO:teuthology.orchestra.run.smithi006.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.44787.log.gz 2024-04-24T23:32:06.948 INFO:teuthology.orchestra.run.smithi006.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.50272.log.gz 2024-04-24T23:32:06.948 INFO:teuthology.orchestra.run.smithi006.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.49749.log.gz 2024-04-24T23:32:06.948 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.51145.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49521.log 2024-04-24T23:32:06.949 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.50440.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55196.log 2024-04-24T23:32:06.950 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.49521.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41585.log 2024-04-24T23:32:06.950 INFO:teuthology.orchestra.run.smithi006.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.51145.log.gz 2024-04-24T23:32:06.951 INFO:teuthology.orchestra.run.smithi006.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.50440.log.gz 2024-04-24T23:32:06.951 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.55196.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44619.log 2024-04-24T23:32:06.951 INFO:teuthology.orchestra.run.smithi006.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.55196.log.gz 2024-04-24T23:32:06.952 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.41585.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.49521.log.gz 2024-04-24T23:32:06.952 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52818.log 2024-04-24T23:32:06.953 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.44619.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41586.log 2024-04-24T23:32:06.953 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.52818.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50001.log 2024-04-24T23:32:06.954 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.41586.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41026.log 2024-04-24T23:32:06.956 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.50001.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53158.log 2024-04-24T23:32:06.956 INFO:teuthology.orchestra.run.smithi006.stderr: 92.1% -- replaced with /var/log/ceph/ceph-client.admin.40100.log.gz 2024-04-24T23:32:06.956 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.41026.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55322.log 2024-04-24T23:32:06.957 INFO:teuthology.orchestra.run.smithi006.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.41585.log.gz 2024-04-24T23:32:06.957 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.53158.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39340.log 2024-04-24T23:32:06.959 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.55322.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.51229.log.gz 2024-04-24T23:32:06.959 INFO:teuthology.orchestra.run.smithi006.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.51583.log.gz 2024-04-24T23:32:06.959 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42224.log 2024-04-24T23:32:06.959 INFO:teuthology.orchestra.run.smithi006.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.55322.log.gz 2024-04-24T23:32:06.959 INFO:teuthology.orchestra.run.smithi006.stderr: 88.3% 88.2% -- replaced with /var/log/ceph/ceph-client.admin.52461.log.gz -- replaced with /var/log/ceph/ceph-client.admin.52377.log.gz 2024-04-24T23:32:06.959 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-24T23:32:06.959 INFO:teuthology.orchestra.run.smithi006.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.53074.log.gz 2024-04-24T23:32:06.961 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.39340.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.44619.log.gz 2024-04-24T23:32:06.961 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44283.log 2024-04-24T23:32:06.961 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.42224.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.41586.log.gz 2024-04-24T23:32:06.961 INFO:teuthology.orchestra.run.smithi006.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.52818.log.gz 2024-04-24T23:32:06.961 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51439.log 2024-04-24T23:32:06.961 INFO:teuthology.orchestra.run.smithi006.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.50001.log.gz 2024-04-24T23:32:06.961 INFO:teuthology.orchestra.run.smithi006.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.41026.log.gz 2024-04-24T23:32:06.961 INFO:teuthology.orchestra.run.smithi006.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.53158.log.gz 2024-04-24T23:32:06.962 INFO:teuthology.orchestra.run.smithi006.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.42224.log.gz 2024-04-24T23:32:06.962 INFO:teuthology.orchestra.run.smithi006.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.39340.log.gz 2024-04-24T23:32:06.962 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.44283.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42715.log 2024-04-24T23:32:06.963 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.51439.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42233.log 2024-04-24T23:32:06.964 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.42715.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.51439.log.gz 2024-04-24T23:32:06.964 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55280.log 2024-04-24T23:32:06.965 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.42233.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50814.log 2024-04-24T23:32:06.965 INFO:teuthology.orchestra.run.smithi006.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.42715.log.gz 2024-04-24T23:32:06.966 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.55280.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.42233.log.gz 2024-04-24T23:32:06.966 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 87.9% --verbose -- replaced with /var/log/ceph/ceph-client.admin.55280.log.gz 2024-04-24T23:32:06.966 INFO:teuthology.orchestra.run.smithi006.stderr: -- /var/log/ceph/ceph-client.admin.55952.log 2024-04-24T23:32:06.967 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.50814.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.33125.log.gz 2024-04-24T23:32:06.967 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44577.log 2024-04-24T23:32:06.968 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.55952.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.50814.log.gz 2024-04-24T23:32:06.968 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40900.log 2024-04-24T23:32:06.968 INFO:teuthology.orchestra.run.smithi006.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.55952.log.gz 2024-04-24T23:32:06.968 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.44577.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51709.log 2024-04-24T23:32:06.969 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.40900.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49385.log 2024-04-24T23:32:06.969 INFO:teuthology.orchestra.run.smithi006.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.44577.log.gz 2024-04-24T23:32:06.970 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.51709.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46789.log 2024-04-24T23:32:06.971 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.49385.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39118.log 2024-04-24T23:32:06.971 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.46789.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40804.log 2024-04-24T23:32:06.973 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51355.log 2024-04-24T23:32:06.973 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.39118.log: /var/log/ceph/ceph-client.admin.40804.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.e.log 2024-04-24T23:32:06.973 INFO:teuthology.orchestra.run.smithi006.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.40900.log.gz 2024-04-24T23:32:06.974 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.51355.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40594.log 2024-04-24T23:32:06.974 INFO:teuthology.orchestra.run.smithi006.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.51709.log.gz 2024-04-24T23:32:06.975 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-mds.e.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53958.log 2024-04-24T23:32:06.976 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.40594.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51920.log 2024-04-24T23:32:06.977 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.53958.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49184.log 2024-04-24T23:32:06.977 INFO:teuthology.orchestra.run.smithi006.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.46789.log.gz 2024-04-24T23:32:06.977 INFO:teuthology.orchestra.run.smithi006.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.53958.log.gz 2024-04-24T23:32:06.979 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.51920.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46385.log 2024-04-24T23:32:06.979 INFO:teuthology.orchestra.run.smithi006.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.49385.log.gz 2024-04-24T23:32:06.979 INFO:teuthology.orchestra.run.smithi006.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.39118.log.gz 2024-04-24T23:32:06.979 INFO:teuthology.orchestra.run.smithi006.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.51355.log.gz 2024-04-24T23:32:06.980 INFO:teuthology.orchestra.run.smithi006.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.40594.log.gz 2024-04-24T23:32:06.980 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.49184.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.51920.log.gz 2024-04-24T23:32:06.980 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49563.log 2024-04-24T23:32:06.980 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.46385.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50188.log 2024-04-24T23:32:06.981 INFO:teuthology.orchestra.run.smithi006.stderr: 87.8%/var/log/ceph/ceph-client.admin.49563.log: -- replaced with /var/log/ceph/ceph-client.admin.49184.log.gz 2024-04-24T23:32:06.981 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40678.log 2024-04-24T23:32:06.982 INFO:teuthology.orchestra.run.smithi006.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.46385.log.gz 2024-04-24T23:32:06.982 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5/var/log/ceph/ceph-client.admin.50188.log: --verbose -- /var/log/ceph/ceph-client.admin.55616.log 2024-04-24T23:32:06.982 INFO:teuthology.orchestra.run.smithi006.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.49563.log.gz 2024-04-24T23:32:06.983 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.40678.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51024.log 2024-04-24T23:32:06.983 INFO:teuthology.orchestra.run.smithi006.stderr: 94.5% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-04-24T23:32:06.984 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.55616.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41900.log 2024-04-24T23:32:06.985 INFO:teuthology.orchestra.run.smithi006.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.55616.log.gz 2024-04-24T23:32:06.985 INFO:teuthology.orchestra.run.smithi006.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.50188.log.gz 2024-04-24T23:32:06.985 INFO:teuthology.orchestra.run.smithi006.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.40678.log.gz 2024-04-24T23:32:06.985 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.51024.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52944.log 2024-04-24T23:32:06.986 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.41900.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39593.log 2024-04-24T23:32:06.986 INFO:teuthology.orchestra.run.smithi006.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.51024.log.gz 2024-04-24T23:32:06.987 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.52944.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43291.log 2024-04-24T23:32:06.988 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.39593.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53410.log 2024-04-24T23:32:06.989 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.43291.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54174.log 2024-04-24T23:32:06.989 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.53410.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51271.log 2024-04-24T23:32:06.990 INFO:teuthology.orchestra.run.smithi006.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.53410.log.gz 2024-04-24T23:32:06.991 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.54174.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40984.log 2024-04-24T23:32:06.991 INFO:teuthology.orchestra.run.smithi006.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.41900.log.gz 2024-04-24T23:32:06.991 INFO:teuthology.orchestra.run.smithi006.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.54174.log.gz 2024-04-24T23:32:06.991 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.51271.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.52944.log.gz 2024-04-24T23:32:06.992 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41569.log 2024-04-24T23:32:06.992 INFO:teuthology.orchestra.run.smithi006.stderr: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.39593.log.gz 2024-04-24T23:32:06.993 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.40984.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42979.log 2024-04-24T23:32:06.993 INFO:teuthology.orchestra.run.smithi006.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.40804.log.gz 2024-04-24T23:32:06.993 INFO:teuthology.orchestra.run.smithi006.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.43291.log.gz 2024-04-24T23:32:06.994 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.41569.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.51271.log.gz 2024-04-24T23:32:06.994 INFO:teuthology.orchestra.run.smithi006.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.40984.log.gz 2024-04-24T23:32:06.994 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51667.log 2024-04-24T23:32:06.995 INFO:teuthology.orchestra.run.smithi006.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.41569.log.gz 2024-04-24T23:32:06.995 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.42979.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44241.log 2024-04-24T23:32:06.995 INFO:teuthology.orchestra.run.smithi006.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.42979.log.gz 2024-04-24T23:32:06.995 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.51667.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41110.log 2024-04-24T23:32:06.996 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.44241.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54045.log 2024-04-24T23:32:06.997 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.41110.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52755.log 2024-04-24T23:32:06.998 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.54045.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42622.log 2024-04-24T23:32:06.999 INFO:teuthology.orchestra.run.smithi006.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.54045.log.gz 2024-04-24T23:32:06.999 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.52755.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49605.log 2024-04-24T23:32:07.000 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.42622.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49875.log 2024-04-24T23:32:07.000 INFO:teuthology.orchestra.run.smithi006.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.51667.log.gz 2024-04-24T23:32:07.001 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.49605.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44829.log 2024-04-24T23:32:07.001 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.49875.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41898.log 2024-04-24T23:32:07.002 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.44829.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41152.log 2024-04-24T23:32:07.004 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.41898.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.41110.log.gz 2024-04-24T23:32:07.004 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54216.log 2024-04-24T23:32:07.004 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.41152.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53748.log 2024-04-24T23:32:07.005 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.54216.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51835.log 2024-04-24T23:32:07.006 INFO:teuthology.orchestra.run.smithi006.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.54216.log.gz 2024-04-24T23:32:07.006 INFO:teuthology.orchestra.run.smithi006.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.42622.log.gz 2024-04-24T23:32:07.006 INFO:teuthology.orchestra.run.smithi006.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.49605.log.gz 2024-04-24T23:32:07.007 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.53748.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55364.log 2024-04-24T23:32:07.007 INFO:teuthology.orchestra.run.smithi006.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.53748.log.gz 2024-04-24T23:32:07.007 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.51835.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44409.log 2024-04-24T23:32:07.008 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.55364.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52022.log 2024-04-24T23:32:07.008 INFO:teuthology.orchestra.run.smithi006.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.55364.log.gz 2024-04-24T23:32:07.009 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.44409.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42672.log 2024-04-24T23:32:07.009 INFO:teuthology.orchestra.run.smithi006.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.49875.log.gz 2024-04-24T23:32:07.010 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.52022.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50712.log 2024-04-24T23:32:07.011 INFO:teuthology.orchestra.run.smithi006.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.41898.log.gz 2024-04-24T23:32:07.011 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.42672.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47174.log 2024-04-24T23:32:07.012 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.50712.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.41152.log.gz 2024-04-24T23:32:07.012 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50043.log 2024-04-24T23:32:07.013 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.47174.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50145.log 2024-04-24T23:32:07.014 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.50043.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45615.log 2024-04-24T23:32:07.014 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.50145.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53242.log 2024-04-24T23:32:07.016 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.45615.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.51835.log.gzgzip 2024-04-24T23:32:07.016 INFO:teuthology.orchestra.run.smithi006.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.39074.log 2024-04-24T23:32:07.016 INFO:teuthology.orchestra.run.smithi006.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.44409.log.gz 2024-04-24T23:32:07.017 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.53242.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43728.log 2024-04-24T23:32:07.018 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.39074.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41886.log 2024-04-24T23:32:07.019 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.43728.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.42672.log.gz 2024-04-24T23:32:07.019 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40410.log 2024-04-24T23:32:07.019 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.41886.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39509.log 2024-04-24T23:32:07.021 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.40410.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.50712.log.gz 2024-04-24T23:32:07.021 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50898.log 2024-04-24T23:32:07.021 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.39509.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40636.log 2024-04-24T23:32:07.022 INFO:teuthology.orchestra.run.smithi006.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.47174.log.gz 2024-04-24T23:32:07.022 INFO:teuthology.orchestra.run.smithi006.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.50043.log.gz 2024-04-24T23:32:07.022 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.50898.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42388.log 2024-04-24T23:32:07.023 INFO:teuthology.orchestra.run.smithi006.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.50145.log.gz 2024-04-24T23:32:07.023 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.40636.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51962.log 2024-04-24T23:32:07.023 INFO:teuthology.orchestra.run.smithi006.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.45615.log.gz 2024-04-24T23:32:07.024 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.42388.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38629.log 2024-04-24T23:32:07.024 INFO:teuthology.orchestra.run.smithi006.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.44829.log.gz 2024-04-24T23:32:07.025 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.51962.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.53242.log.gz 2024-04-24T23:32:07.025 INFO:teuthology.orchestra.run.smithi006.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.39074.log.gz 2024-04-24T23:32:07.025 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41404.log 2024-04-24T23:32:07.026 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.38629.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.43728.log.gz 2024-04-24T23:32:07.026 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38548.log 2024-04-24T23:32:07.027 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.41404.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49343.log 2024-04-24T23:32:07.028 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.38548.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55658.log 2024-04-24T23:32:07.028 INFO:teuthology.orchestra.run.smithi006.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.41886.log.gz 2024-04-24T23:32:07.028 INFO:teuthology.orchestra.run.smithi006.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.40410.log.gz 2024-04-24T23:32:07.029 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.49343.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.39509.log.gz 2024-04-24T23:32:07.029 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49917.log 2024-04-24T23:32:07.030 INFO:teuthology.orchestra.run.smithi006.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.50898.log.gz 2024-04-24T23:32:07.030 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.55658.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53916.log 2024-04-24T23:32:07.030 INFO:teuthology.orchestra.run.smithi006.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.55658.log.gz 2024-04-24T23:32:07.031 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.49917.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49477.log 2024-04-24T23:32:07.032 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.53916.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.40636.log.gz 2024-04-24T23:32:07.032 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40536.log 2024-04-24T23:32:07.032 INFO:teuthology.orchestra.run.smithi006.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.53916.log.gz 2024-04-24T23:32:07.032 INFO:teuthology.orchestra.run.smithi006.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.42388.log.gz 2024-04-24T23:32:07.033 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.49477.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41236.log 2024-04-24T23:32:07.034 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.40536.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41194.log 2024-04-24T23:32:07.035 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.41236.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49959.log 2024-04-24T23:32:07.035 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.41194.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42170.log 2024-04-24T23:32:07.037 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.49959.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53832.log 2024-04-24T23:32:07.037 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.42170.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.51962.log.gz 2024-04-24T23:32:07.038 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39212.log 2024-04-24T23:32:07.039 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.53832.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52148.log 2024-04-24T23:32:07.039 INFO:teuthology.orchestra.run.smithi006.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.38629.log.gz 2024-04-24T23:32:07.039 INFO:teuthology.orchestra.run.smithi006.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.53832.log.gz 2024-04-24T23:32:07.039 INFO:teuthology.orchestra.run.smithi190.stderr: 92.3% -- replaced with /var/log/ceph/ceph.log.gz 2024-04-24T23:32:07.040 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.39212.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.44241.log.gz 2024-04-24T23:32:07.040 INFO:teuthology.orchestra.run.smithi006.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.44283.log.gz 2024-04-24T23:32:07.040 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43249.log 2024-04-24T23:32:07.042 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.52148.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.49343.log.gz 2024-04-24T23:32:07.042 INFO:teuthology.orchestra.run.smithi006.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.41404.log.gz 2024-04-24T23:32:07.042 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43600.log 2024-04-24T23:32:07.042 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.43249.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52671.log 2024-04-24T23:32:07.043 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.43600.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50482.log 2024-04-24T23:32:07.043 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.52671.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.49917.log.gz 2024-04-24T23:32:07.043 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52064.log 2024-04-24T23:32:07.044 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.50482.log: gzip 87.5% -- replaced with /var/log/ceph/ceph-client.admin.49477.log.gz -5 2024-04-24T23:32:07.044 INFO:teuthology.orchestra.run.smithi006.stderr: --verbose -- /var/log/ceph/ceph-mds.a.log 2024-04-24T23:32:07.045 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.52064.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39467.log 2024-04-24T23:32:07.046 INFO:teuthology.orchestra.run.smithi006.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.40536.log.gz 2024-04-24T23:32:07.047 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-mds.a.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.41236.log.gz 2024-04-24T23:32:07.047 INFO:teuthology.orchestra.run.smithi006.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.41194.log.gz 2024-04-24T23:32:07.047 INFO:teuthology.orchestra.run.smithi006.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.49959.log.gz 2024-04-24T23:32:07.047 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54088.log 2024-04-24T23:32:07.047 INFO:teuthology.orchestra.run.smithi006.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.42170.log.gz 2024-04-24T23:32:07.048 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.39467.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55784.log 2024-04-24T23:32:07.048 INFO:teuthology.orchestra.run.smithi006.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.39212.log.gz 2024-04-24T23:32:07.048 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.54088.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56160.log 2024-04-24T23:32:07.049 INFO:teuthology.orchestra.run.smithi006.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.54088.log.gz 2024-04-24T23:32:07.049 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.55784.log: gzip -5 87.5% --verbose -- replaced with /var/log/ceph/ceph-client.admin.55784.log.gz -- 2024-04-24T23:32:07.049 INFO:teuthology.orchestra.run.smithi006.stderr: /var/log/ceph/ceph-client.admin.52545.log 2024-04-24T23:32:07.051 INFO:teuthology.orchestra.run.smithi006.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.52148.log.gz 2024-04-24T23:32:07.051 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.56160.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.43249.log.gz 2024-04-24T23:32:07.051 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44325.log 2024-04-24T23:32:07.051 INFO:teuthology.orchestra.run.smithi006.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.56160.log.gz 2024-04-24T23:32:07.052 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.52545.log: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.43600.log.gz 2024-04-24T23:32:07.052 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49301.log 2024-04-24T23:32:07.052 INFO:teuthology.orchestra.run.smithi006.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.50482.log.gz 2024-04-24T23:32:07.052 INFO:teuthology.orchestra.run.smithi006.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.52064.log.gz 2024-04-24T23:32:07.053 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.44325.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55448.log 2024-04-24T23:32:07.054 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.49301.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45292.log 2024-04-24T23:32:07.054 INFO:teuthology.orchestra.run.smithi006.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.39467.log.gz 2024-04-24T23:32:07.055 INFO:teuthology.orchestra.run.smithi006.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.44325.log.gz 2024-04-24T23:32:07.055 INFO:teuthology.orchestra.run.smithi006.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.52545.log.gz 2024-04-24T23:32:07.055 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.55448.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.49301.log.gz 2024-04-24T23:32:07.056 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42936.log 2024-04-24T23:32:07.056 INFO:teuthology.orchestra.run.smithi006.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.55448.log.gz 2024-04-24T23:32:07.056 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.45292.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53116.log 2024-04-24T23:32:07.057 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.42936.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45423.log 2024-04-24T23:32:07.057 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.53116.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55070.log 2024-04-24T23:32:07.058 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.45423.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49124.log 2024-04-24T23:32:07.058 INFO:teuthology.orchestra.run.smithi006.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.42936.log.gz 2024-04-24T23:32:07.059 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.55070.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45995.log 2024-04-24T23:32:07.059 INFO:teuthology.orchestra.run.smithi006.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.55070.log.gz 2024-04-24T23:32:07.060 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.49124.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39254.log 2024-04-24T23:32:07.061 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.45995.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55532.log 2024-04-24T23:32:07.062 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.39254.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50669.log 2024-04-24T23:32:07.062 INFO:teuthology.orchestra.run.smithi006.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.45292.log.gz 2024-04-24T23:32:07.062 INFO:teuthology.orchestra.run.smithi006.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.45423.log.gz 2024-04-24T23:32:07.063 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.55532.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42538.log 2024-04-24T23:32:07.063 INFO:teuthology.orchestra.run.smithi006.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.49124.log.gz 2024-04-24T23:32:07.063 INFO:teuthology.orchestra.run.smithi006.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.55532.log.gz 2024-04-24T23:32:07.064 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.50669.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50085.log 2024-04-24T23:32:07.064 INFO:teuthology.orchestra.run.smithi006.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.39254.log.gz 2024-04-24T23:32:07.064 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.42538.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42238.log 2024-04-24T23:32:07.065 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.50085.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56076.log 2024-04-24T23:32:07.066 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.42238.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42062.log 2024-04-24T23:32:07.067 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.56076.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56118.log 2024-04-24T23:32:07.067 INFO:teuthology.orchestra.run.smithi006.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.56076.log.gz 2024-04-24T23:32:07.068 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.42062.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52106.log 2024-04-24T23:32:07.069 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.56118.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55910.log 2024-04-24T23:32:07.069 INFO:teuthology.orchestra.run.smithi006.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.50669.log.gz 2024-04-24T23:32:07.070 INFO:teuthology.orchestra.run.smithi006.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.42538.log.gz 2024-04-24T23:32:07.070 INFO:teuthology.orchestra.run.smithi006.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.56118.log.gz 2024-04-24T23:32:07.070 INFO:teuthology.orchestra.run.smithi006.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.50085.log.gz 2024-04-24T23:32:07.070 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.52106.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.42238.log.gz 2024-04-24T23:32:07.070 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55992.log 2024-04-24T23:32:07.071 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.55910.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44745.log 2024-04-24T23:32:07.071 INFO:teuthology.orchestra.run.smithi006.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.55910.log.gz 2024-04-24T23:32:07.072 INFO:teuthology.orchestra.run.smithi006.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.42062.log.gz 2024-04-24T23:32:07.073 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.55992.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55868.log 2024-04-24T23:32:07.073 INFO:teuthology.orchestra.run.smithi006.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.55992.log.gz 2024-04-24T23:32:07.073 INFO:teuthology.orchestra.run.smithi006.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.52106.log.gz 2024-04-24T23:32:07.073 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.44745.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39383.log 2024-04-24T23:32:07.074 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.55868.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41581.log 2024-04-24T23:32:07.074 INFO:teuthology.orchestra.run.smithi006.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.55868.log.gz 2024-04-24T23:32:07.075 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.39383.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.44745.log.gz 2024-04-24T23:32:07.075 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44367.log 2024-04-24T23:32:07.075 INFO:teuthology.orchestra.run.smithi006.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.39383.log.gz 2024-04-24T23:32:07.076 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.41581.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41583.log 2024-04-24T23:32:07.076 INFO:teuthology.orchestra.run.smithi006.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.41581.log.gz 2024-04-24T23:32:07.077 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.44367.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53622.log 2024-04-24T23:32:07.077 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.41583.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42760.log 2024-04-24T23:32:07.078 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.53622.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.53622.log.gz 2024-04-24T23:32:07.081 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.42760.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.44367.log.gz 2024-04-24T23:32:07.084 INFO:teuthology.orchestra.run.smithi006.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.41583.log.gz 2024-04-24T23:32:07.085 INFO:teuthology.orchestra.run.smithi006.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.42760.log.gz 2024-04-24T23:32:07.086 INFO:teuthology.orchestra.run.smithi006.stderr: 92.6% -- replaced with /var/log/ceph/ceph-mds.a.log.gz 2024-04-24T23:32:07.087 INFO:teuthology.orchestra.run.smithi006.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.45995.log.gz 2024-04-24T23:32:07.096 INFO:teuthology.orchestra.run.smithi006.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.52022.log.gz 2024-04-24T23:32:07.099 INFO:teuthology.orchestra.run.smithi006.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.53116.log.gz 2024-04-24T23:32:07.109 INFO:teuthology.orchestra.run.smithi006.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.52671.log.gz 2024-04-24T23:32:07.109 INFO:teuthology.orchestra.run.smithi006.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.52755.log.gz 2024-04-24T23:32:07.112 INFO:teuthology.orchestra.run.smithi006.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.38548.log.gz 2024-04-24T23:32:07.474 INFO:teuthology.orchestra.run.smithi006.stderr: 91.3% -- replaced with /var/log/ceph/ceph-mds.e.log.gz 2024-04-24T23:32:07.588 INFO:teuthology.orchestra.run.smithi190.stderr: 92.5% -- replaced with /var/log/ceph/ceph-mds.b.log.gz 2024-04-24T23:32:07.994 INFO:teuthology.orchestra.run.smithi190.stderr: 91.4% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-04-24T23:32:08.165 INFO:teuthology.orchestra.run.smithi190.stderr: 92.3% -- replaced with /var/log/ceph/ceph-mds.d.log.gz 2024-04-24T23:32:08.820 INFO:teuthology.orchestra.run.smithi190.stderr: 91.2% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-04-24T23:32:09.274 INFO:teuthology.orchestra.run.smithi190.stderr: 91.3% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-04-24T23:32:09.537 INFO:teuthology.orchestra.run.smithi190.stderr: 95.4% -- replaced with /var/log/ceph/ceph-mds.f.log.gz 2024-04-24T23:32:13.270 INFO:teuthology.orchestra.run.smithi006.stderr: 90.1% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-04-24T23:34:14.134 INFO:teuthology.orchestra.run.smithi190.stderr: 92.2% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-04-24T23:34:20.103 INFO:teuthology.orchestra.run.smithi190.stderr: 92.2% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-04-24T23:36:01.068 INFO:teuthology.orchestra.run.smithi006.stderr: 95.3% -- replaced with /var/log/ceph/ceph-mds.c.log.gz 2024-04-24T23:38:06.050 INFO:teuthology.orchestra.run.smithi006.stderr: 92.4% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-04-24T23:43:59.641 INFO:teuthology.orchestra.run.smithi190.stderr: 92.3% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-04-24T23:44:09.649 INFO:teuthology.orchestra.run.smithi190.stderr: 92.1% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-04-24T23:44:09.651 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-04-24T23:44:09.651 INFO:teuthology.orchestra.run.smithi190.stderr:real 12m2.976s 2024-04-24T23:44:09.651 INFO:teuthology.orchestra.run.smithi190.stderr:user 6m22.061s 2024-04-24T23:44:09.651 INFO:teuthology.orchestra.run.smithi190.stderr:sys 0m16.590s 2024-04-24T23:47:38.521 INFO:teuthology.orchestra.run.smithi006.stderr: 92.6% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-04-24T23:49:12.617 INFO:teuthology.orchestra.run.smithi006.stderr: 92.2% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-04-24T23:49:30.029 INFO:teuthology.orchestra.run.smithi006.stderr: 92.5% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-04-24T23:49:30.033 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-24T23:49:30.033 INFO:teuthology.orchestra.run.smithi006.stderr:real 17m23.324s 2024-04-24T23:49:30.033 INFO:teuthology.orchestra.run.smithi006.stderr:user 20m4.703s 2024-04-24T23:49:30.033 INFO:teuthology.orchestra.run.smithi006.stderr:sys 2m41.570s 2024-04-24T23:49:30.034 INFO:tasks.ceph:Archiving logs... 2024-04-24T23:49:30.035 DEBUG:teuthology.misc:Transferring archived files from smithi006:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672187/remote/smithi006/log 2024-04-24T23:49:30.036 DEBUG:teuthology.orchestra.run.smithi006:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-24T23:51:00.805 DEBUG:teuthology.misc:Transferring archived files from smithi190:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672187/remote/smithi190/log 2024-04-24T23:51:00.829 DEBUG:teuthology.orchestra.run.smithi190:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-24T23:52:34.151 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-24T23:52:34.161 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-24T23:52:34.161 DEBUG:teuthology.orchestra.run.smithi006:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-24T23:52:34.170 DEBUG:teuthology.orchestra.run.smithi190:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-24T23:52:34.325 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-24T23:52:34.335 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-24T23:52:34.335 DEBUG:teuthology.orchestra.run.smithi006:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-24T23:52:34.343 DEBUG:teuthology.orchestra.run.smithi190:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-24T23:52:34.360 INFO:teuthology.orchestra.run.smithi190.stderr:bash: line 1: ntpq: command not found 2024-04-24T23:52:34.386 INFO:teuthology.orchestra.run.smithi190.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-24T23:52:34.386 INFO:teuthology.orchestra.run.smithi190.stdout:=============================================================================== 2024-04-24T23:52:34.387 INFO:teuthology.orchestra.run.smithi190.stdout:^- hv01.front.sepia.ceph.com 2 6 377 61 +943us[ +943us] +/- 572ms 2024-04-24T23:52:34.387 INFO:teuthology.orchestra.run.smithi190.stdout:^* hv02.front.sepia.ceph.com 3 6 377 64 +110us[ +136us] +/- 63ms 2024-04-24T23:52:34.387 INFO:teuthology.orchestra.run.smithi190.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 63 -143us[ -143us] +/- 66ms 2024-04-24T23:52:34.387 INFO:teuthology.orchestra.run.smithi190.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-24T23:52:34.392 INFO:teuthology.orchestra.run.smithi006.stderr:bash: line 1: ntpq: command not found 2024-04-24T23:52:34.492 INFO:teuthology.orchestra.run.smithi006.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-24T23:52:34.492 INFO:teuthology.orchestra.run.smithi006.stdout:=============================================================================== 2024-04-24T23:52:34.492 INFO:teuthology.orchestra.run.smithi006.stdout:^- hv01.front.sepia.ceph.com 2 6 377 63 +943us[ +947us] +/- 572ms 2024-04-24T23:52:34.492 INFO:teuthology.orchestra.run.smithi006.stdout:^* hv02.front.sepia.ceph.com 3 6 377 0 +115us[ +122us] +/- 64ms 2024-04-24T23:52:34.493 INFO:teuthology.orchestra.run.smithi006.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 63 -96us[ -92us] +/- 66ms 2024-04-24T23:52:34.493 INFO:teuthology.orchestra.run.smithi006.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-24T23:52:34.497 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-24T23:52:34.509 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-24T23:52:34.511 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-24T23:52:34.565 DEBUG:teuthology.orchestra.run.smithi006:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-24T23:52:34.895 DEBUG:teuthology.orchestra.run.smithi190:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-24T23:52:34.980 DEBUG:teuthology.orchestra.run.smithi006:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-24T23:52:35.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T23:52:35.079 DEBUG:teuthology.orchestra.run.smithi190:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-24T23:52:35.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T23:52:35.109 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-24T23:52:35.120 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-24T23:52:35.177 INFO:teuthology.task.internal:Duration was 2710.231516 seconds 2024-04-24T23:52:35.177 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-24T23:52:35.244 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-24T23:52:35.245 DEBUG:teuthology.orchestra.run.smithi006:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-24T23:52:35.252 DEBUG:teuthology.orchestra.run.smithi190:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-24T23:52:35.322 INFO:teuthology.orchestra.run.smithi190.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-24T23:52:35.494 INFO:teuthology.orchestra.run.smithi006.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-24T23:52:35.798 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-24T23:52:35.798 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi006.front.sepia.ceph.com 2024-04-24T23:52:35.798 DEBUG:teuthology.orchestra.run.smithi006:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-24T23:52:35.870 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi190.front.sepia.ceph.com 2024-04-24T23:52:35.870 DEBUG:teuthology.orchestra.run.smithi190:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-24T23:52:35.899 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-24T23:52:35.899 DEBUG:teuthology.orchestra.run.smithi006:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-24T23:52:35.907 DEBUG:teuthology.orchestra.run.smithi190:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-24T23:52:35.990 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-24T23:52:35.991 DEBUG:teuthology.orchestra.run.smithi006:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-24T23:52:35.996 DEBUG:teuthology.orchestra.run.smithi190:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-24T23:52:36.221 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-24T23:52:36.232 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-24T23:52:36.232 DEBUG:teuthology.orchestra.run.smithi006:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-24T23:52:36.334 DEBUG:teuthology.orchestra.run.smithi190:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-24T23:52:36.392 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-24T23:52:36.402 DEBUG:teuthology.orchestra.run.smithi006:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-24T23:52:36.410 DEBUG:teuthology.orchestra.run.smithi190:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-24T23:52:36.445 INFO:teuthology.orchestra.run.smithi190.stdout:kernel.core_pattern = core 2024-04-24T23:52:36.521 INFO:teuthology.orchestra.run.smithi006.stdout:kernel.core_pattern = core 2024-04-24T23:52:36.624 DEBUG:teuthology.orchestra.run.smithi006:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-24T23:52:36.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T23:52:36.680 DEBUG:teuthology.orchestra.run.smithi190:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-24T23:52:36.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T23:52:36.695 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-24T23:52:36.705 INFO:teuthology.task.internal:Transferring archived files... 2024-04-24T23:52:36.706 DEBUG:teuthology.misc:Transferring archived files from smithi006:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672187/remote/smithi006 2024-04-24T23:52:36.707 DEBUG:teuthology.orchestra.run.smithi006:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-24T23:52:36.809 DEBUG:teuthology.misc:Transferring archived files from smithi190:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672187/remote/smithi190 2024-04-24T23:52:36.809 DEBUG:teuthology.orchestra.run.smithi190:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-24T23:52:36.843 INFO:teuthology.task.internal:Removing archive directory... 2024-04-24T23:52:36.844 DEBUG:teuthology.orchestra.run.smithi006:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-24T23:52:36.852 DEBUG:teuthology.orchestra.run.smithi190:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-24T23:52:36.902 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-24T23:52:36.917 INFO:teuthology.task.internal:Not uploading archives. 2024-04-24T23:52:36.918 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-24T23:52:36.945 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-24T23:52:36.946 DEBUG:teuthology.orchestra.run.smithi006:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-24T23:52:36.953 DEBUG:teuthology.orchestra.run.smithi190:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-24T23:52:36.967 INFO:teuthology.orchestra.run.smithi190.stdout: 262161 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 24 23:52 /home/ubuntu/cephtest 2024-04-24T23:52:37.005 INFO:teuthology.orchestra.run.smithi006.stdout: 393249 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 24 23:52 /home/ubuntu/cephtest 2024-04-24T23:52:37.008 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-24T23:52:37.020 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-24T23:52:37.156 INFO:teuthology.run:Summary data: client.0-kernel-sha1: 806ec36b21129f113ce356c007dd59694d340b37 description: fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/ffsb}} duration: 2710.2315158843994 flavor: default owner: scheduled_teuthology@teuthology success: true 2024-04-24T23:52:37.156 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-24T23:52:37.238 INFO:teuthology.run:pass